Added setter method setGenerator to JavaRDD.
Added mapPartitions method to JavaRDD.
Added foreachPartition method to JavaRDD.
Remove classtag from mapPartitions.
Remove default param from mapPartitions
Modifications as suggested in PR feedback-

- mapPartitions, foreachPartition moved to JavaRDDLike
- call scala rdd's setGenerator instead of setting directly in JavaRDD
Modifications as suggested in PR feedback-

- more variants of mapPartitions added to JavaRDDLike
- move setGenerator to JavaRDDLike
- clean up
Allow files added through SparkContext.addFile() to be overwritten

This is useful for the cases when a file needs to be refreshed and downloaded
by the executors periodically.

Signed-off-by: Yinan Li <liyinan926@gmail.com>
Addressed comments from Reynold

Signed-off-by: Yinan Li <liyinan926@gmail.com>
executor creation failed should not make the worker restart
fix for SPARK-1027

change TestClient & Worker to Some("xxx")

kill manager if it is started

remove unnecessary .get when fetch "SPARK_HOME" values
Replace the code to check for Option != None with Option.isDefined call in Scala code.

This hopefully will make the code cleaner.
refactor sparkHome to val

clean code
Fix bug in worker clean-up in UI

Introduced in d5a96fec. This should be picked into 0.8 and 0.9 as well.
Merge pull request #447 from CodingCat/SPARK-1027

fix for SPARK-1027

fix for SPARK-1027  (https://spark-project.atlassian.net/browse/SPARK-1027)

FIXES

1. change sparkhome from String to Option(String) in ApplicationDesc

2. remove sparkhome parameter in LaunchExecutor message

3. adjust involved files
Merge pull request #496 from pwendell/master

Fix bug in worker clean-up in UI

Introduced in d5a96fec (/cc @aarondav).

This should be picked into 0.8 and 0.9 as well. The bug causes old (zombie) workers on a node to not disappear immediately from the UI when a new one registers.
Replace commons-math with jblas
Add jblas dependency
Add jblas dependency
fixed ClassTag in mapPartitions