yieldbot/flambo — pull request #27 by dajac (2014-11-05 08:57:36+00:00)
Title: Another take on serialization of functions

First of all, I would like to thank you for Flambo. It is really good library to deal with Spark!

I have been using it for few weeks and I have noticed that our big jobs often failed due to `java.lang.OutOfMemoryError: PermGen space` in Spark's executors. This is due to the fact that 'serialize-fn' uses eval for deserialising functions which results in a lot of classes created in Spark's executors. Basically, classes are re-created for every tasks.

I looked a bit more deeply at how stuff are serialized in Flambo/Spark and I have tried another approach with this patch. I don't expect to have this PR merged in Flambo but I would like to have your opinion about it.

In short, this PR changes the way functions are serialized in Flambo. Instead of using "serialize-fn", it relies on out-of-the-box java serialization for serializing functions with theirs bindings and it also ensures that namespaces are loaded in executors. This approach works well but requires to AOT namespaces used in executors so it is not super convenient in the REPL. Another thing that I have noticed is that **my jobs are 2x faster** with this mechanism.

In my opinion, the best solution would be to have both approaches. I mean, having the ability to define functions without AOT for developing and using AOT for running apps in production.

WDYT?

PS: Midje tests can not be AOT'ed so I switched them to clojure.test.