Added a number of very fast, memory-efficient data structures: BitSet, OpenHashSet, OpenHashMap, PrimitiveKeyOpenHashMap.
Adding improved error message when multiple assembly jars are present.

This can happen easily if building different hadoop versions.
Merge pull request #114 from soulmachine/master

A little revise for the document
Merge pull request #113 from pwendell/master

Improve error message when multiple assembly jars are present.

This can happen easily if building different hadoop versions. Right now it gives a class not found exception.
Display both task ID and task index in UI
Eliminate extra memory usage when shuffle file consolidation is disabled

Otherwise, we see SPARK-946 even when shuffle file consolidation is disabled.
Fixing SPARK-946 is still forthcoming.
Use flag instead of name check.
Handle ConcurrentModificationExceptions in SparkContext init.

System.getProperties.toMap will fail-fast when concurrently modified,
and it seems like some other thread started by SparkContext does
a System.setProperty during it's initialization.

Handle this by just looping on ConcurrentModificationException, which
seems the safest, since the non-fail-fast methods (Hastable.entrySet)
have undefined behavior under concurrent modification.
Merge pull request #115 from aarondav/shuffle-fix

Eliminate extra memory usage when shuffle file consolidation is disabled

Otherwise, we see SPARK-946 even when shuffle file consolidation is disabled.
Fixing SPARK-946 is still forthcoming.
Merge pull request #112 from kayousterhout/ui_task_attempt_id

Display both task ID and task attempt ID in UI, and rename taskId to taskAttemptId

Previously only the task attempt ID was shown in the UI; this was confusing because the job can be shown as complete while there are tasks still running.  Showing the task ID in addition to the attempt ID makes it clear which tasks are redundant.

This commit also renames taskId to taskAttemptId in TaskInfo and in the local/cluster schedulers.  This identifier was used to uniquely identify attempts, not tasks, so the current naming was confusing.  The new naming is also more consistent with map reduce.
Restructure BlockInfo fields to reduce memory use.
Merge pull request #119 from soulmachine/master

A little revise for the document
Remove SparkHadoopUtil stuff from SparkEnv
Store fewer BlockInfo fields for shuffle blocks.
Use Properties.clone() instead.
Extract BlockInfo classes from BlockManager.

This saves space, since the inner classes needed
to keep a reference to the enclosing BlockManager.
fix sparkhdfs lr test
Merge pull request #118 from JoshRosen/blockinfo-memory-usage

Reduce the memory footprint of BlockInfo objects

This pull request reduces the memory footprint of all BlockInfo objects and makes additional optimizations for shuffle blocks.  For all BlockInfo objects, these changes remove two boolean fields and one Object field.  For shuffle blocks, we additionally remove an Object field and a boolean field.

When storing tens of thousands of these objects, this may add up to significant memory savings.  A ShuffleBlockInfo now only needs to wrap a single long.

This was motivated by a [report of high blockInfo memory usage during shuffles](https://mail-archives.apache.org/mod_mbox/incubator-spark-user/201310.mbox/%3C20131026134353.202b2b9b%40sh9%3E).

I haven't run benchmarks to measure the exact memory savings.

/cc @aarondav
Merge remote-tracking branch 'upstream/master' into sparkHadoopUtilFix
Add support for local:// URI scheme for addJars()

This indicates that a jar is available locally on each worker node.