apache/incubator-parquet-mr — pull request #36 by dcw-netflix (2014-08-11 21:37:02+00:00)
Title: Fixed storing pig schema to udfcontext for non projection case and moved...

... column index access setting to udfcontext so as not to affect other loaders.

I found an problem that affects both the Column name access and column index access due to the way the pig schema is stored by the loader.

##Column Name Access:
The ParquetLoader was only storing the pig schema in the UDFContext when push projection is applied. In the full load case, the schema was not stored which triggered a full reload of the schema during task execution. You can see in initSchema references the UDFContext for the schema, but that is only set in push projection. However, the schema needs to be set in both the job context (so the TupleReadSupport can access the schema) and the UDFContext (so the task side loader can access it), which is why it is set in both locations. This also meant the requested schema was never set to the task side either, which could cause other problems as well.

##Column Index Access:
For index based access, the problem was that the column index access setting and the requested schema were not stored in the udfcontext and sent to the task side (unless pushProjection was called). The schema was stored in the job context, but this would be overwritten if another loader was executed first. Also, the property to use column index access was only being set at the job context level, so subsequent loaders would use column index access even if they didn't request it.

This fix now ensures that both the schema and column index access are set in the udfcontext and loaded in the initSchema method.

-Dan