Adding algorithm for implicit feedback data to ALS
Initial commit for job killing.
Properly handle job failure when the job gets killed.
Added a submitJob interface that returns a Future of the result.
Allow SparkContext.submitJob to submit a job for only a subset of the partitions.
Added a hack to kill all active jobs in SparkContext.
Fixed a bug for zero partition in JobWaiter.
Made task killing work for standalone cluster schedulers.
More logging changes (task killing for local cluster doesn't work yet).
Added task killing iterator to RDDs that take inputs.
Added comment to InterruptibleIterator.
Wrap around cached data to InterruptibleIterator.
Made output of CoGroup and aggregations interruptible.
Split HadoopRDD into one for general Hadoop datasets and one tailored to Hadoop files, which is a common case.

This is the first step to avoiding unnecessary Configuration broadcasts per HadoopRDD instantiation.
Add a cache for HadoopRDD metadata needed during computation.

Currently, the cache is in SparkHadoopUtils, since it's conveniently a member of the SparkEnv.
Move Configuration broadcasts to SparkContext.
Send Task results through the block manager when larger than Akka frame size.

This change requires adding an extra failure mode: tasks can complete
successfully, but the result gets lost or flushed from the block manager
before it's been fetched.
Merge remote-tracking branch 'upstream/master' into implicit-als
Support distributed cache files and archives on spark on yarn and attempt to cleanup the staging directory on exit
Standalone Scheduler fault recovery

Implements a basic form of Standalone Scheduler fault recovery. In particular,
this allows faults to be manually recovered from by means of restarting the
Master process on the same machine. This is the majority of the code necessary
for general fault tolerance, which will first elect a leader and then recover
the Master state.

In order to enable fault recovery, the Master will persist a small amount of state related
to the registration of Workers and Applications to disk. If the Master is started and
sees that this state is still around, it will enter Recovery mode, during which time it
will not schedule any new Executors on Workers (but it does accept the registration of
new Clients and Workers).

At this point, the Master attempts to reconnect to all Workers and Client applications
that were registered at the time of failure. After confirming either the existence
or nonexistence of all such nodes (within a certain timeout), the Master will exit
Recovery mode and resume normal scheduling.