Added setter method setGenerator to JavaRDD.
Added mapPartitions method to JavaRDD.
Added foreachPartition method to JavaRDD.
Remove classtag from mapPartitions.
Remove default param from mapPartitions
Modifications as suggested in PR feedback-

- mapPartitions, foreachPartition moved to JavaRDDLike
- call scala rdd's setGenerator instead of setting directly in JavaRDD
Modifications as suggested in PR feedback-

- more variants of mapPartitions added to JavaRDDLike
- move setGenerator to JavaRDDLike
- clean up
Allow files added through SparkContext.addFile() to be overwritten

This is useful for the cases when a file needs to be refreshed and downloaded
by the executors periodically.

Signed-off-by: Yinan Li <liyinan926@gmail.com>
Addressed comments from Reynold

Signed-off-by: Yinan Li <liyinan926@gmail.com>
executor creation failed should not make the worker restart
fix for SPARK-1027

change TestClient & Worker to Some("xxx")

kill manager if it is started

remove unnecessary .get when fetch "SPARK_HOME" values
Replace the code to check for Option != None with Option.isDefined call in Scala code.

This hopefully will make the code cleaner.
Depend on Commons Math explicitly instead of accidentally getting it from Hadoop (which stops working in 2.2.x) and also use the newer commons-math3
Also add graphx commons-math3 dependeny in sbt build
Merge pull request #495 from srowen/GraphXCommonsMathDependency

Fix graphx Commons Math dependency

`graphx` depends on Commons Math (2.x) in `SVDPlusPlus.scala`. However the module doesn't declare this dependency. It happens to work because it is included by Hadoop artifacts. But, I can tell you this isn't true as of a month or so ago. Building versus recent Hadoop would fail. (That's how we noticed.)

The simple fix is to declare the dependency, as it should be. But it's also worth noting that `commons-math` is the old-ish 2.x line, while `commons-math3` is where newer 3.x releases are. Drop-in replacement, but different artifact and package name. Changing this only usage to `commons-math3` works, tests pass, and isn't surprising that it does, so is probably also worth changing. (A comment in some test code also references `commons-math3`, FWIW.)

It does raise another question though: `mllib` looks like it uses the `jblas` `DoubleMatrix` for general purpose vector/matrix stuff. Should `graphx` really use Commons Math for this? Beyond the tiny scope here but worth asking.
refactor sparkHome to val

clean code
Fix bug in worker clean-up in UI

Introduced in d5a96fec. This should be picked into 0.8 and 0.9 as well.
Merge pull request #447 from CodingCat/SPARK-1027

fix for SPARK-1027

fix for SPARK-1027  (https://spark-project.atlassian.net/browse/SPARK-1027)

FIXES

1. change sparkhome from String to Option(String) in ApplicationDesc

2. remove sparkhome parameter in LaunchExecutor message

3. adjust involved files
Merge pull request #496 from pwendell/master

Fix bug in worker clean-up in UI

Introduced in d5a96fec (/cc @aarondav).

This should be picked into 0.8 and 0.9 as well. The bug causes old (zombie) workers on a node to not disappear immediately from the UI when a new one registers.
Replace commons-math with jblas