mesos/spark — pull request #568 by jey (2013-04-16 02:53:59+00:00)
Title: Fix sporadic hangs in DistributedSuite

This patch fixes sporadic hangs in DistributedSuite by using Runtime.halt when simulating a JVM crash to avoid running the shutdown hooks. The workers hang while executing the Akka termination handlers due to some kind of deadlock, but I haven't traced it further to figure out which termination handler is causing the deadlock.

When the main process hangs the thread running the test has this trace:
```
"pool-8-thread-11" prio=10 tid=0x00007fee02278000 nid=0x7811 in Object.wait() [0x00007fee8d30e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007e7ccc5a0> (a spark.scheduler.JobWaiter)
at java.lang.Object.wait(Object.java:503)
at spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:45)
- locked <0x00000007e7ccc5a0> (a spark.scheduler.JobWaiter)
at spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:248)
at spark.SparkContext.runJob(SparkContext.scala:581)
at spark.SparkContext.runJob(SparkContext.scala:599)
at spark.SparkContext.runJob(SparkContext.scala:613)
at spark.SparkContext.runJob(SparkContext.scala:627)
at spark.RDD.collect(RDD.scala:461)
at spark.DistributedSuite$$anonfun$23$$anonfun$apply$mcV$sp$5.apply$mcVI$sp(DistributedSuite.scala:247)
at spark.DistributedSuite$$anonfun$23.apply$mcV$sp(DistributedSuite.scala:241)
at spark.DistributedSuite$$anonfun$23.apply(DistributedSuite.scala:235)
at spark.DistributedSuite$$anonfun$23.apply(DistributedSuite.scala:235)
at org.scalatest.FunSuite$$anon$1.apply(FunSuite.scala:1265)
at org.scalatest.Suite$class.withFixture(Suite.scala:1974)
at spark.DistributedSuite.withFixture(DistributedSuite.scala:19)
...
```

And the workers have a main thread with a trace of:

```
"main" prio=10 tid=0x00007fb97800e800 nid=0x1e71 waiting on condition [0x00007fb981dda000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000fd588118> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:736)
at akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:705)
at akka.dispatch.Await$.ready(Future.scala:64)
at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:604)
at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:605)
at spark.executor.StandaloneExecutorBackend$.run(StandaloneExecutorBackend.scala:72)
at spark.executor.StandaloneExecutorBackend$.main(StandaloneExecutorBackend.scala:81)
at spark.executor.StandaloneExecutorBackend.main(StandaloneExecutorBackend.scala)
```