jghoman/haivvreo — Issue #14: Schema mismatch when running INSERT OVERWRITE from one Avro-backed table to another
Author: elibingham
Created: 2012-02-21 18:14:30+00:00
State: open

Hi there,

I have two tables defined roughly as:

CREATE EXTERNAL TABLE a
ROW FORMAT SERDE
'com.linkedin.haivvreo.AvroSerDe'
WITH SERDEPROPERTIES (
'schema.url'='hdfs://schema/a.avsc')
STORED AS INPUTFORMAT
'com.linkedin.haivvreo.AvroContainerInputFormat'
OUTPUTFORMAT
'com.linkedin.haivvreo.AvroContainerOutputFormat'
LOCATION
'/foo/a';

CREATE EXTERNAL TABLE b
ROW FORMAT SERDE
'com.linkedin.haivvreo.AvroSerDe'
WITH SERDEPROPERTIES (
'schema.literal'='{ ... }')
STORED AS INPUTFORMAT
'com.linkedin.haivvreo.AvroContainerInputFormat'
OUTPUTFORMAT
'com.linkedin.haivvreo.AvroContainerOutputFormat'
LOCATION
'/foo/b';

When I do:

INSERT OVERWRITE TABLE b
SELECT a, b, c FROM a;

(where b's schema is equivalent to a, b, c)

This fails with an AvroTypeException like this:

2012-02-21 11:31:58,587 INFO org.apache.hadoop.mapred.TaskStatus: task-diagnostic-info for task attempt_201202091933_0059_m_000000_1 : java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable com.linkedin.haivvreo.AvroGenericRecordWritable@92696c2
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable com.linkedin.haivvreo.AvroGenericRecordWritable@92696c2
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:520)
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
... 8 more
Caused by: org.apache.avro.AvroTypeException: Found {
"type" : "record",
"name" : "b",
"fields" : [ { … /* schema of b */ … } ]
}, expecting {
"type" : "record",
"name" : "a",
"fields" : [ { … /* schema of a */ … } ],
...
}
at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:231)
at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
at org.apache.avro.io.ResolvingDecoder.readFieldOrder(ResolvingDecoder.java:127)
at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:162)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129)
at com.linkedin.haivvreo.AvroDeserializer$SchemaReEncoder.reencode(AvroDeserializer.java:79)
at com.linkedin.haivvreo.AvroDeserializer.deserialize(AvroDeserializer.java:121)
at com.linkedin.haivvreo.AvroSerDe.deserialize(AvroSerDe.java:81)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:504)
... 9 more

Any help would be appreciated.