mesos/spark — pull request #195 by dennybritz (2012-09-10 22:58:19+00:00)
Title: Spark HTTP FileServer

A general fileserver for serving both regular as well as JAR files.

- I modified the existing JAR file server code to put everything into one fileserver.
- Modified the LocalScheduler to mimic the ClusterScheduler/Executor in terms of getting files. That's useful for testing.
- Includes test with local-cluster and local mode.
- Modified serialization of ShuffleMapTask
- I wasn't quite sure about the caching in ShuffleMapTask, is using the hashCode safe? There may be a better way to do this. Also, it doesn't seem like the serializedInfoCache (and fileCache) is cleared except when the SparkContext shuts down? What if it's a really long running task?