mesos/spark — Issue #102: Why "./run spark.examples.SparkPi 1@localhost:5050" can't get result of Pi
Author: shawny
Created: 2011-12-16 15:10:09+00:00
State: open

I setup mesos on local machine:

{
1000 29724 0.0 0.4 49556 9728 pts/0 Sl 21:44 0:02 ./bin mesos-master
1000 29774 0.0 0.4 48864 8756 pts/0 Sl 21:44 0:01 ./bin mesos-slave --resources=cpus:4;mem:916 -m mesos://master@localhost:5050
}

When i run "./run spark.examples.SparkPi 1@localhost:5050", it will stop at the following last line:

11/12/16 23:04:45 INFO CacheTrackerActor: Registered actor on port 50501
11/12/16 23:04:45 INFO MapOutputTrackerActor: Registered actor on port 50501
11/12/16 23:04:45 INFO MesosScheduler: Temp directory for JARs: /tmp/spark-d10f5f39-4d21-4062-ac67-7e27ebd15794
11/12/16 23:04:46 INFO MesosScheduler: JAR server started at http://127.0.0.1:40518
11/12/16 23:04:46 INFO SparkContext: Starting job...
11/12/16 23:04:46 INFO CacheTracker: Registering RDD ID 0 with cache
11/12/16 23:04:46 INFO CacheTrackerActor: Registering RDD 0 with 2 partitions
11/12/16 23:04:46 INFO CacheTrackerActor: Asked for current cache locations
11/12/16 23:04:46 INFO MesosScheduler: Final stage: Stage 0
11/12/16 23:04:46 INFO MesosScheduler: Parents of final stage: List()
11/12/16 23:04:46 INFO MesosScheduler: Missing parents: List()
11/12/16 23:04:46 INFO MesosScheduler: Submitting Stage 0, which has no missing parents
11/12/16 23:04:46 INFO MesosScheduler: Got a job with 2 tasks

But when i run "./run spark.examples.SparkPi mesos://master@localhost:5050", i can get the result of Pi:

11/12/16 22:57:44 INFO MapOutputTrackerActor: Registered actor on port 50501
11/12/16 22:57:44 INFO CacheTrackerActor: Registered actor on port 50501
11/12/16 22:57:44 INFO MesosScheduler: Temp directory for JARs: /tmp/spark-f173b70d-bf22-4dc7-a1c5-c241a67af3dc
11/12/16 22:57:45 INFO MesosScheduler: JAR server started at http://127.0.0.1:40327
11/12/16 22:57:45 INFO SparkContext: Starting job...
11/12/16 22:57:45 INFO CacheTracker: Registering RDD ID 0 with cache
11/12/16 22:57:45 INFO CacheTrackerActor: Registering RDD 0 with 2 partitions
11/12/16 22:57:45 INFO CacheTrackerActor: Asked for current cache locations
11/12/16 22:57:45 INFO MesosScheduler: Final stage: Stage 0
11/12/16 22:57:45 INFO MesosScheduler: Parents of final stage: List()
11/12/16 22:57:45 INFO MesosScheduler: Missing parents: List()
11/12/16 22:57:45 INFO MesosScheduler: Submitting Stage 0, which has no missing parents
11/12/16 22:57:45 INFO MesosScheduler: Got a job with 2 tasks
11/12/16 22:57:45 INFO MesosScheduler: Registered as framework ID 201112162144-0-0002
11/12/16 22:57:45 INFO MesosScheduler: Adding job with ID 0
11/12/16 22:57:45 INFO SimpleJob: Starting task 0:0 as TID 0 on slave 201112162144-0-0: localhost (preferred)
11/12/16 22:57:45 INFO SimpleJob: Starting task 0:1 as TID 1 on slave 201112162144-0-0: localhost (preferred)
11/12/16 22:57:46 INFO SimpleJob: Finished TID 1 (progress: 1/2)
11/12/16 22:57:47 INFO MesosScheduler: Completed ResultTask(0, 1)
11/12/16 22:57:47 INFO SimpleJob: Finished TID 0 (progress: 2/2)
11/12/16 22:57:47 INFO MesosScheduler: Completed ResultTask(0, 0)
11/12/16 22:57:47 INFO SparkContext: Job finished in 1.827198029 s
Pi is roughly 3.14944

==> but SparkPi still as active frameworks from UI
(
Active Frameworks

ID	User	Name	Running Tasks	CPUs	MEM	Max Share	Connected
201112162144-0-0002	xuanhouysh	SparkPi	0	0	200.0 MB	0.22	2011-12-16 22:57:45
)

My Spark config file as follows:

export MESOS_HOME=/home/xuanhouysh/software/mesos
export SCALA_HOME=/home/xuanhouysh/software/scala-2.9.1.final
export SPARK_MEM=200m