kevinweil/elephant-bird — Issue #161: Pig Complex thrift structures
Author: calmera
Created: 2012-03-07 14:40:07+00:00
State: open

I'm using the ThriftWritableConverter to convert a pig tuple into a Thrift object. I get the following exception when trying to create the converter:

ERROR 2999: Unexpected internal error. could not instantiate 'com.twitter.elephantbird.pig.store.SequenceFileStorage' with arguments '[-c com.twitter.elephantbird.pig.util.ThriftWritableConverter AdvertisementId, -c com.twitter.elephantbird.pig.util.ThriftWritableConverter Advertisement]'

java.lang.RuntimeException: could not instantiate 'com.twitter.elephantbird.pig.store.SequenceFileStorage' with arguments '[-c com.twitter.elephantbird.pig.util.ThriftWritableConverter AdvertisementId, -c com.twitter.elephantbird.pig.util.ThriftWritableConverter Advertisement]'
at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:502)
at org.apache.pig.impl.logicalLayer.parser.QueryParser.NonEvalFuncSpec(QueryParser.java:5660)
at org.apache.pig.impl.logicalLayer.parser.QueryParser.StoreClause(QueryParser.java:4034)
at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:1501)
at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:1013)
at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:825)
at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1612)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1562)
at org.apache.pig.PigServer.registerQuery(PigServer.java:534)
at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:871)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:388)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:168)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:144)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
at org.apache.pig.Main.run(Main.java:500)
at org.apache.pig.Main.main(Main.java:107)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:470)
... 21 more
Caused by: java.lang.RuntimeException: Failed to create WritableConverter instance
at com.twitter.elephantbird.pig.load.SequenceFileLoader.getWritableConverter(SequenceFileLoader.java:247)
at com.twitter.elephantbird.pig.load.SequenceFileLoader.(SequenceFileLoader.java:140)
at com.twitter.elephantbird.pig.store.SequenceFileStorage.(SequenceFileStorage.java:103)
... 26 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.twitter.elephantbird.pig.load.SequenceFileLoader.getWritableConverter(SequenceFileLoader.java:222)
... 28 more
Caused by: java.lang.RuntimeException: while trying to find id in class AdvertisementId
at com.twitter.elephantbird.util.ThriftUtils.getFiedlType(ThriftUtils.java:110)
at com.twitter.elephantbird.thrift.TStructDescriptor$Field.(TStructDescriptor.java:201)
at com.twitter.elephantbird.thrift.TStructDescriptor$Field.(TStructDescriptor.java:128)
at com.twitter.elephantbird.thrift.TStructDescriptor.build(TStructDescriptor.java:104)
at com.twitter.elephantbird.thrift.TStructDescriptor.getInstance(TStructDescriptor.java:88)
at com.twitter.elephantbird.pig.util.ThriftToPig.(ThriftToPig.java:56)
at com.twitter.elephantbird.pig.util.ThriftToPig.newInstance(ThriftToPig.java:52)
at com.twitter.elephantbird.pig.util.ThriftWritableConverter.(ThriftWritableConverter.java:61)
... 33 more
Caused by: java.lang.NoSuchFieldException: id
at java.lang.Class.getDeclaredField(Class.java:1882)
at com.twitter.elephantbird.util.ThriftUtils.getFiedlType(ThriftUtils.java:107)
... 40 more

This is the part of my schema:

union AdvertisementId {
1: string id;
}

struct Advertisement {
1: required AdvertisementId id;
}

And this is the structure of the pig relation:
ads: {advertisementId: (id: chararray)}

the TStructDescriptor tries to search for a field but in case of a union you won't have actual fields within the generated classes.