mesos/spark — pull request #347 by shivaram (2012-12-30 02:57:49+00:00)
Title: Changes to make Spark run using IBM J9

The changeset contains the following changes:
a. Fix WorkerArguments to use reflection to get total physical memory using the appropriate MBean

b. While modifying the SizeEstimator to work with IBM J9, I found that the project `java-sizeof` already has a RamUsageEstimator class which uses very similar techniques to what we have. Additionally it is pulled from the Apache Lucene/Solr source code and is available on maven. To avoid having to maintain SizeEstimator in spark, I have replaced it with java-sizeof from https://github.com/dweiss/java-sizeof.
More details on what it does can be found at
http://www.slideshare.net/DawidWeiss/sizeofobject-how-much-memory-objects-take-on-jvms-and-when-this-may-matter (Also the slides show why it is tricky to estimate size across JVMs)

c. Upgrade default Hadoop version to 1.0.3. This is because Hadoop 0.20 has Sun-specific dependencies. Also fix a bug in saving SequenceFiles to infer the correct Writable type

d. Remove unused BoundedMemoryCache and associated test file. These tests fail on IBM J9 as string size varies across JVMs. The class was unused in the current version of Spark as it has been replaced by MemoryStore.

Tested using OpenJDK 6,7 on 32 and 64-bit arch and using IBM J9 1.7 on 32-bit arch.