Added a number of very fast, memory-efficient data structures: BitSet, OpenHashSet, OpenHashMap, PrimitiveKeyOpenHashMap.
Handle ConcurrentModificationExceptions in SparkContext init.

System.getProperties.toMap will fail-fast when concurrently modified,
and it seems like some other thread started by SparkContext does
a System.setProperty during it's initialization.

Handle this by just looping on ConcurrentModificationException, which
seems the safest, since the non-fail-fast methods (Hastable.entrySet)
have undefined behavior under concurrent modification.
Remove SparkHadoopUtil stuff from SparkEnv
Use Properties.clone() instead.
fix sparkhdfs lr test
Merge remote-tracking branch 'upstream/master' into sparkHadoopUtilFix
move the hadoopJobMetadata back into SparkEnv
Avoid match errors when filtering for spark.hadoop settings.
Fixed incorrect log message in local scheduler
removing unused commented code
Merge pull request #124 from tgravescs/sparkHadoopUtilFix

Pull SparkHadoopUtil out of SparkEnv (jira SPARK-886)

Having the logic to initialize the correct SparkHadoopUtil in SparkEnv prevents it from being used until after the SparkContext is initialized.   This causes issues like https://spark-project.atlassian.net/browse/SPARK-886.  It also makes it hard to use in singleton objects.  For instance I want to use it in the security code.
Merge pull request #126 from kayousterhout/local_fix

Fixed incorrect log message in local scheduler

This change is especially relevant at the moment, because some users are seeing this failure, and the log message is misleading/incorrect (because for the tests, the max failures is set to 0, not 4)
added lineage tracking code
Removing potential additional shuffle dependency where an already partitioned RDD[(Vid, VD)] is repartitioned.
removing completely unnecessary map operation.
Merge pull request #117 from stephenh/avoid_concurrent_modification_exception

Handle ConcurrentModificationExceptions in SparkContext init.

System.getProperties.toMap will fail-fast when concurrently modified,
and it seems like some other thread started by SparkContext does
a System.setProperty during it's initialization.

Handle this by just looping on ConcurrentModificationException, which
seems the safest, since the non-fail-fast methods (Hastable.entrySet)
have undefined behavior under concurrent modification.
Adding additional type constraints to VertexSetRDD to help diagnose issues with recent benchmarks.
Merge branch 'hash1' of https://github.com/rxin/incubator-spark into rxinBitSet
Added bitset to kryo register
Adding logical AND/OR, setUntil, and iterators to the BitSet.