nathanmarz/cascalog — pull request #65 by mlimotte (2012-04-03 18:32:45+00:00)
Title: Support custom serializers for defop stateful params

As far as I can tell, there are two methods to acquire the job-conf to configure the SeriailzationFactory. And each has it's limitations.

1) cascalog.conf/project-conf

This method does not see the dynamic binding done by with-job-conf (since deser is done by a different thread); and it won't see any JobConf configured on the cluster itself either (e.g. the hadoop/conf directory).

2) Cast FlowProcess, so you can call HadoopFlowProcess#getJobConf()

This method can see Hadoop config (and by extension, the conf via with-job-conf). But

a) it does not get the special treatment of project-conf for serializations (ClojureKryoSerialization and the default-serializations).

b) The FlowProcess object is not always available to us (e.g. KryoInsert#hashcode), or is not always a HadoopFlowProcess (e.g. bridge_test.clj).

Due to these limitations, I had to use a combination of both methods. At least this is the only way I could come up with to get all the unit tests to pass. The combined methods attempt to get the conf out of the FlowProcess if available, and will pass it on to project-conf for special treatment.