[Hot Fix #469] Fix flaky test in SparkListenerSuite

The two modified tests may fail if the race condition does not bid in our favor...

Author: Andrew Or <andrewor14@gmail.com>

Closes #516 from andrewor14/stage-info-test-fix and squashes the following commits:

b4b6100 [Andrew Or] Add/replace missing waitUntilEmpty() calls to listener bus

(cherry picked from commit 4b2bab1d08a6b790be94717bbdd643d896d85c16)
Signed-off-by: Reynold Xin <rxin@apache.org>
Update Java api for setJobGroup with interruptOnCancel

Also adds a unit test.

Author: Aaron Davidson <aaron@databricks.com>

Closes #522 from aarondav/cancel2 and squashes the following commits:

565c253 [Aaron Davidson] Update Java api for setJobGroup with interruptOnCancel
65b33d8 [Aaron Davidson] Add unit test for Thread interruption on cancellation

(cherry picked from commit d485eecb7233dd339ae85a6f58f1c0686dd2037d)
Signed-off-by: Reynold Xin <rxin@apache.org>
Mark all fields of EdgePartition, Graph, and GraphOps transient

These classes are only serializable to work around closure capture, so their fields should all be marked `@transient` to avoid wasteful serialization.

This PR supersedes apache/spark#519 and fixes the same bug.

Author: Ankur Dave <ankurdave@gmail.com>

Closes #520 from ankurdave/graphx-transient and squashes the following commits:

6431760 [Ankur Dave] Mark all fields of EdgePartition, Graph, and GraphOps `@transient`

(cherry picked from commit 1d6abe3a4b58f28fc4e0e690e02c19b2568ce1ee)
Signed-off-by: Reynold Xin <rxin@apache.org>
SPARK-1589: Fix the incorrect compare

JIRA: https://issues.apache.org/jira/browse/SPARK-1589

Author: zsxwing <zsxwing@gmail.com>

Closes #508 from zsxwing/SPARK-1589 and squashes the following commits:

570c67a [zsxwing] SPARK-1589: Fix the incorrect compare

(cherry picked from commit 6ab7578067e3bb78b64f99fd67c97e9607050ffe)
Signed-off-by: Reynold Xin <rxin@apache.org>
[Fix #79] Replace Breakable For Loops By While Loops

Author: Sandeep <sandeep@techaddict.me>

Closes #503 from techaddict/fix-79 and squashes the following commits:

e3f6746 [Sandeep] Style changes
07a4f6b [Sandeep] for loop to While loop
0a6d8e9 [Sandeep] Breakable for loop to While loop

(cherry picked from commit bb68f47745eec2954814d3da277a672d5cf89980)
Signed-off-by: Reynold Xin <rxin@apache.org>
SPARK-1587 Fix thread leak

mvn test fails (intermittently) due to thread leak - since scalatest runs all tests in same vm.

Author: Mridul Muralidharan <mridulm80@apache.org>

Closes #504 from mridulm/resource_leak_fixes and squashes the following commits:

a5d10d0 [Mridul Muralidharan] Prevent thread leaks while running tests : cleanup all threads when SparkContext.stop is invoked. Causes tests to fail
7b5e19c [Mridul Muralidharan] Prevent NPE while running tests

(cherry picked from commit dd681f502eafe39cfb8a5a62ea2d28016ac6013d)
Signed-off-by: Aaron Davidson <aaron@databricks.com>
SPARK-1601 & SPARK-1602: two bug fixes related to cancellation

This should go into 1.0 since it would return wrong data when the bug happens (which is pretty likely if cancellation is used). Test case attached.

1. Do not put partially executed partitions into cache (in task killing).

2. Iterator returned by CacheManager#getOrCompute was not an InterruptibleIterator, and was thus leading to uninterruptible jobs.

Thanks @aarondav and @ahirreddy for reporting and helping debug.

Author: Reynold Xin <rxin@apache.org>

Closes #521 from rxin/kill and squashes the following commits:

401033f [Reynold Xin] Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/spark into kill
7a7bdd2 [Reynold Xin] Add a new line in the end of JobCancellationSuite.scala.
35cd9f7 [Reynold Xin] Fixed a bug that partially executed partitions can be put into cache (in task killing).

(cherry picked from commit 1fdf659d2fdf23c5562e5dc646d05083062281ed)
Signed-off-by: Reynold Xin <rxin@apache.org>