linkedin/camus — pull request #99 by henrylinkedin (2014-08-14 23:48:53+00:00)
Title: Make camus-daily-sweeper able to parse the schema generated from avro-1.4 (from hourly/ files)

...nts

Currently camus-daily-sweeper cannot loop through all events because the avro files from hourly has avro schemas generated from avro-1.4 which allows (-, @, ' ') in field names but avro 1.7 (which daily job was using) is much strict in field names. Ask the producer to change all those field names is going to be a long migration process.

Fortunately, avro-1.7 parser does allow skip-the-name-validation if you this flag. The problem is we don't control all the places in jobLauncher/MapTask/ReduceTask where the parser is created.

I took a look at the source code for MapTask and ReduceTask and added a few subclasses from AvroSerialization which knows to set the skip-name-validation flag.

This behavior is now controlled by conf property:
camus.sweeper.skip.name.validation=true
If we don't want this behavior, we can turn off this flag (which is the default)