Merge pull request #490 from hsaputra/modify_checkoption_with_isdefined

Replace the check for None Option with isDefined and isEmpty in Scala code

Propose to replace the Scala check for Option "!= None" with Option.isDefined and "=== None" with Option.isEmpty.

I think this, using method call if possible then operator function plus argument, will make the Scala code easier to read and understand.

Pass compile and tests.
(cherry picked from commit f16c21e22f706b268419fefce44e9905db3ee485)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #516 from sarutak/master

modified SparkPluginBuild.scala to use https protocol for accessing gith...

We cannot build Spark behind a proxy although we execute sbt with -Dhttp(s).proxyHost -Dhttp(s).proxyPort -Dhttp(s).proxyUser -Dhttp(s).proxyPassword options.
It's because of using git protocol to clone junit_xml_listener.git.
I could build after modifying SparkPluginBuild.scala.

I reported this issue to JIRA.
https://spark-project.atlassian.net/browse/SPARK-1046
(cherry picked from commit 3d5c03e2305777b8a32f2e196e3b73ab221b3e79)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #504 from JoshRosen/SPARK-1025

Fix PySpark hang when input files are deleted (SPARK-1025)

This pull request addresses [SPARK-1025](https://spark-project.atlassian.net/browse/SPARK-1025), an issue where PySpark could hang if its input files were deleted.
(cherry picked from commit c40619d4873f36ffb96a2e6292b32d5b64eab153)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #511 from JoshRosen/SPARK-1040

Fix ClassCastException in JavaPairRDD.collectAsMap() (SPARK-1040)

This fixes [SPARK-1040](https://spark-project.atlassian.net/browse/SPARK-1040), an issue where JavaPairRDD.collectAsMap() could sometimes fail with ClassCastException.  I applied the same fix to the Spark Streaming Java APIs.  The commit message describes the fix in more detail.

I also increased the verbosity of JUnit test output under SBT to make it easier to verify that the Java tests are actually running.
(cherry picked from commit c66a2ef1c2dc9c218069b3ce8c39a49e5b92fc16)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #493 from kayousterhout/double_add

Fixed bug where task set managers are added to queue twice

@mateiz can you verify that this is a bug and wasn't intentional? (https://github.com/apache/incubator-spark/commit/90a04dab8d9a2a9a372cea7cdf46cc0fd0f2f76c#diff-7fa4f84a961750c374f2120ca70e96edR551)

This bug leads to a small performance hit because task
set managers will get offered each rejected resource
offer twice, but doesn't lead to any incorrect functionality.

Thanks to @hdc1112 for pointing this out.
(cherry picked from commit 5bcfd798117e8617e604c1dd3b5c9b67e83100bb)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #489 from ash211/patch-6

Clarify spark.default.parallelism

It's the task count across the cluster, not per worker, per machine, per core, or anything else.
(cherry picked from commit 749f842827c7e7766a342b6b0a437803044a9f90)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #445 from kayousterhout/exec_lost

Fail rather than hanging if a task crashes the JVM.

Prior to this commit, if a task crashes the JVM, the task (and
all other tasks running on that executor) is marked at KILLED rather
than FAILED.  As a result, the TaskSetManager will retry the task
indefinitely rather than failing the job after maxFailures. Eventually,
this makes the job hang, because the Standalone Scheduler removes
the application after 10 works have failed, and then the app is left
in a state where it's disconnected from the master and waiting to reconnect.
This commit fixes that problem by marking tasks as FAILED rather than
killed when an executor is lost.

The downside of this commit is that if task A fails because another
task running on the same executor caused the VM to crash, the failure
will incorrectly be counted as a failure of task A. This should not
be an issue because we typically set maxFailures to 3, and it is
unlikely that a task will be co-located with a JVM-crashing task
multiple times.
(cherry picked from commit c06a307ca22901839df00d25fe623f6faa6af17e)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #414 from soulmachine/code-style

Code clean up for mllib

* Removed unnecessary parentheses
* Removed unused imports
* Simplified `filter...size()` to `count ...`
* Removed obsoleted parameters' comments
(cherry picked from commit 84595ea3e25d2f9578b3de34704da14eb02330fa)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #449 from CrazyJvm/master

SPARK-1028 : fix "set MASTER automatically fails" bug.

spark-shell intends to set MASTER automatically if we do not provide the option when we start the shell , but there's a problem.
The condition is "if [[ "x" != "x$SPARK_MASTER_IP" && "y" != "y$SPARK_MASTER_PORT" ]];" we sure will set SPARK_MASTER_IP explicitly, the SPARK_MASTER_PORT option, however, we probably do not set just using spark default port 7077. So if we do not set SPARK_MASTER_PORT, the condition will never be true. We should just use default port if users do not set port explicitly I think.
(cherry picked from commit 6b4eed779bd1889543ac2c058745bd0864f02b2a)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>