linkedin/camus — pull request #5 by criccomini (2013-01-27 06:44:01+00:00)
Title: Removing Avro dependency for Schema Registry

I've done the following:

##### Refactored SchemaRegistry to be a typed interface

Previously, SchemaRegistry was an abstract class that used Hadoop's Configuration, and Avro's Schema for types. I've changed SchemaRegistry to have a developer specified type, rather than hard-coding Avro's schema. I've also removed dependency on Configuration, though this is a bit of a hack right now. The ETL jobs still Class.forName and call the constructor with a Hadoop Configuration. I've left the AvroRestSchemaRegistry alone, so it should still work, but the other SchemaRegistries won't work with the ETL job yet.

##### Added File and Memory SchemaRegistry implementations

They are a bit hacky, but they work. The FileSchemaRegistry just stores schemas in the structure //.schema<.latest>, and uses Serde to go to/from bytes on disk. The memory schema registry just uses HashMaps.

##### Added tests

I've added JUnit tests for all registries (including AvroRestSchemaRegistry).

##### Removed resolvers

There were previously 4 interfaces that had to do with resolvers. As far as I could tell, they just wrapped the SchemaRegistry for specific topics, and would cache results. Instead of this complexity, I removed them all, and just added a CachingSchemaRegistry implementation.

##### Removed Avro from camus-api

I moved the TrackingMonitoringEvent to camus-etl-kafka, which is where it belongs. The benefit is that developers can now implement SchemaRegistry (in camus-api) without pulling in Avro.

##### Removed ExampleEnumSchemaRegistry

This is functionally equivalent to using MemorySchemaRegistry, and calling register(). See the DummySchemaRegistry in camus-example for an example.

##### Proposed work

* What is LatestSchemaKafkaAvroMessageDecoder used for?
* Docs: How do I use kafka-etl-pull.job and camus.properties? What's the difference?
* Add an init(Properties props) method to SchemaRegistry interface.
* Make KafkaAvroMessage[Encoder|Decoder] call init(properties) instead of passing Configuration to schema registry constructors.

This should get us on our way toward supporting non-Avro serialization in camus-etl-kafka. The rest of the project is now agnostic to serialization, so this is the remaining work.