kevinweil/elephant-bird — pull request #180 by sagemintblue (2012-04-09 22:26:48+00:00)
Title: Improves SequenceFileStorage schema validation

SequenceFileStorage currently delegates validation of key and value field schemata to the configured WritableConverters. SequenceFileStorage itself does not require input data to contain only two fields-- Fields beyond the first two are silently ignored. This can cause confusion for new users who may have forgotten to properly organize key and value data before passing data to this storefunc.

This patch increases strictness of validation by requiring input schema to have two and only two fields. This may break existing client usage (e.g. someone passes in a relation which contains more than two input fields), but would reduce confusion regarding SequenceFileStorage input requirements and catch user errors sooner.

Additionally, I've extended input schema validation for `Thrift...` and `ProtobufWritableConverter` classes. These converters now test input schema to ensure it matches the expected pig tuple schema of the configured message type. This allows fail-fast behavior in the front-end when a user tries to store data whose schema doesn't accurately reflect the intended message format. Currently, an error isn't seen until back-end task nodes fail to convert input pig data to thrift / proto messages.