[Fix #204] Eliminate delay between binding and log checking

**Bug**: In the existing history server, there is a `spark.history.updateInterval` seconds delay before application logs show up on the UI.

**Cause**: This is because the following events happen in this order: (1) The background thread that checks for logs starts, but realizes the server has not yet bound and so waits for N seconds, (2) server binds, (3) N seconds later the background thread finds that the server has finally bound to a port, and so finally checks for application logs.

**Fix**: This PR forces the log checking thread to start immediately after binding. It also documents two relevant environment variables that are currently missing.

Author: Andrew Or <andrewor14@gmail.com>

Closes #441 from andrewor14/history-server-fix and squashes the following commits:

b2eb46e [Andrew Or] Document SPARK_PUBLIC_DNS and SPARK_HISTORY_OPTS for the history server
e8d1fbc [Andrew Or] Eliminate delay between binding and checking for logs
(cherry picked from commit 745e496c59cfece2fcd6120ecc366dcab07b293a)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Fix compilation on Hadoop 2.4.x.

Author: Marcelo Vanzin <vanzin@cloudera.com>

Closes #483 from vanzin/yarn-2.4 and squashes the following commits:

0fc57d8 [Marcelo Vanzin] Fix compilation on Hadoop 2.4.x.
(cherry picked from commit 0ea0b1a2d66044af97ab84bf3014ff8dfe91fee4)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
[SPARK-1570] Fix classloading in JavaSQLContext.applySchema

I think I hit a class loading issue when running JavaSparkSQL example using spark-submit in local mode.

Author: Kan Zhang <kzhang@apache.org>

Closes #484 from kanzhang/SPARK-1570 and squashes the following commits:

feaaeba [Kan Zhang] [SPARK-1570] Fix classloading in JavaSQLContext.applySchema
(cherry picked from commit ea8cea82a02099bb66f1e77b757e4d96cc31d6e2)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Assorted clean-up for Spark-on-YARN.

In particular when the HADOOP_CONF_DIR is not not specified.

Author: Patrick Wendell <pwendell@gmail.com>

Closes #488 from pwendell/hadoop-cleanup and squashes the following commits:

fe95f13 [Patrick Wendell] Changes based on Andrew's feeback
18d09c1 [Patrick Wendell] Review comments from Andrew
17929cc [Patrick Wendell] Assorted clean-up for Spark-on-YARN.
(cherry picked from commit 995fdc96bcd2c540804401eaab009a777d7d7aa9)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
[Spark-1538] Fix SparkUI incorrectly hiding persisted RDDs

**Bug**: After the following command `sc.parallelize(1 to 1000).persist.map(_ + 1).count()` is run, the the persisted RDD is missing from the storage tab of the SparkUI.

**Cause**: The command creates two RDDs in one stage, a `ParallelCollectionRDD` and a `MappedRDD`. However, the existing StageInfo only keeps the RDDInfo of the last RDD associated with the stage (`MappedRDD`), and so all RDD information regarding the first RDD (`ParallelCollectionRDD`) is discarded. In this case, we persist the first RDD,  but the StorageTab doesn't know about this RDD because it is not encoded in the StageInfo.

**Fix**: Record information of all RDDs in StageInfo, instead of just the last RDD (i.e. `stage.rdd`). Since stage boundaries are marked by shuffle dependencies, the solution is to traverse the last RDD's dependency tree, visiting only ancestor RDDs related through a sequence of narrow dependencies.

---

This PR also moves RDDInfo to its own file, includes a few style fixes, and adds a unit test for constructing StageInfos.

Author: Andrew Or <andrewor14@gmail.com>

Closes #469 from andrewor14/storage-ui-fix and squashes the following commits:

07fc7f0 [Andrew Or] Add back comment that was accidentally removed (minor)
5d799fe [Andrew Or] Add comment to justify testing of getNarrowAncestors with cycles
9d0e2b8 [Andrew Or] Hide details of getNarrowAncestors from outsiders
d2bac8a [Andrew Or] Deal with cycles in RDD dependency graph + add extensive tests
2acb177 [Andrew Or] Move getNarrowAncestors to RDD.scala
bfe83f0 [Andrew Or] Backtrace RDD dependency tree to find all RDDs that belong to a Stage
(cherry picked from commit 2de573877fbed20092f1b3af20b603b30ba9a940)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
[streaming][SPARK-1578] Removed requirement for TTL in StreamingContext.

Since shuffles and RDDs that are out of context are automatically cleaned by Spark core (using ContextCleaner) there is no need for setting the cleaner TTL while creating a StreamingContext.

Author: Tathagata Das <tathagata.das1565@gmail.com>

Closes #491 from tdas/ttl-fix and squashes the following commits:

cf01dc7 [Tathagata Das] Removed requirement for TTL in StreamingContext.
(cherry picked from commit f3d19a9f1a4e38ff9fb5bf78e04ed5d321219bf6)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
[FIX: SPARK-1376] use --arg instead of --args in SparkSubmit to avoid warning messages

Even if users use `--arg`, `SparkSubmit` still uses `--args` for child args internally, which triggers a warning message that may confuse users:

~~~
--args is deprecated. Use --arg instead.
~~~

@sryza Does it look good to you?

Author: Xiangrui Meng <meng@databricks.com>

Closes #485 from mengxr/submit-arg and squashes the following commits:

5e1b9fe [Xiangrui Meng] update test
cebbeb7 [Xiangrui Meng] use --arg instead of --args in SparkSubmit to avoid warning messages
(cherry picked from commit 662c860ebcec5565493a7dff4812e6b7a37b1d7d)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-1562 Fix visibility / annotation of Spark SQL APIs

Author: Michael Armbrust <michael@databricks.com>

Closes #489 from marmbrus/sqlDocFixes and squashes the following commits:

acee4f3 [Michael Armbrust] Fix visibility / annotation of Spark SQL APIs
(cherry picked from commit aa77f8a6a604efe0d02bc8412b3f1ba3903b7a57)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-1494 Don't initialize classes loaded by MIMA excludes.

[WIP]  Just seeing how Jenkins likes this...

Author: Michael Armbrust <michael@databricks.com>

Closes #494 from marmbrus/mima and squashes the following commits:

6eec616 [Michael Armbrust] Force hive tests to run.
acaf682 [Michael Armbrust] Don't initialize loaded classes.
(cherry picked from commit 8e950813334dd8545357f59f4ed77c39ee4180b3)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
[SQL] SPARK-1571 Mistake in java example code

Author: Michael Armbrust <michael@databricks.com>

Closes #496 from marmbrus/javaBeanBug and squashes the following commits:

644fedd [Michael Armbrust] Bean methods must be public.

(cherry picked from commit 39f85e0322cfecefbc30e7d5a30356cfab1e9640)
Signed-off-by: Reynold Xin <rxin@apache.org>
SPARK-1119 and other build improvements

1. Makes assembly and examples jar naming consistent in maven/sbt.
2. Updates make-distribution.sh to use Maven and fixes some bugs.
3. Updates the create-release script to call make-distribution script.

Author: Patrick Wendell <pwendell@gmail.com>

Closes #502 from pwendell/make-distribution and squashes the following commits:

1a97f0d [Patrick Wendell] SPARK-1119 and other build improvements
(cherry picked from commit cd4ed293262e2349794c13467d1737974385c019)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-1583: Fix a bug that using java.util.HashMap by mistake

JIRA: https://issues.apache.org/jira/browse/SPARK-1583

Does anyone know why using `java.util.HashMap` rather than `mutable.HashMap`? Some methods of `java.util.HashMap` are not generics and compiler can not help us find similar problems.

Author: zsxwing <zsxwing@gmail.com>

Closes #500 from zsxwing/SPARK-1583 and squashes the following commits:

7bfd74d [zsxwing] SPARK-1583: Fix a bug that using java.util.HashMap by mistake

(cherry picked from commit a66460661303bd652b5c5d32860216cc3ceb7f98)
Signed-off-by: Reynold Xin <rxin@apache.org>
SPARK-1572 Don't kill Executor if PythonRDD fails while computing parent

Previously, the behavior was that if the parent RDD threw any exception other than IOException or FileNotFoundException (which is quite possible for Hadoop input sources), the entire Executor would crash, because the default thread a uncaught exception handler calls System.exit().

This patch avoids two related issues:

  1. Always catch exceptions in this reader thread.
  2. Don't mask readerException when Python throws an EOFError
     after worker.shutdownOutput() is called.

Author: Aaron Davidson <aaron@databricks.com>

Closes #486 from aarondav/pyspark and squashes the following commits:

fbb11e9 [Aaron Davidson] Make sure FileNotFoundExceptions are handled same as before
b9acb3e [Aaron Davidson] SPARK-1572 Don't kill Executor if PythonRDD fails while computing parent
(cherry picked from commit a967b005c8937a3053e215c952d2172ee3dc300d)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Honor default fs name when initializing event logger.

This is related to SPARK-1459 / PR #375. Without this fix,
FileLogger.createLogDir() may try to create the log dir on
HDFS, while createWriter() will try to open the log file on
the local file system, leading to interesting errors and
confusion.

Author: Marcelo Vanzin <vanzin@cloudera.com>

Closes #450 from vanzin/event-file-2 and squashes the following commits:

592cdb3 [Marcelo Vanzin] Honor default fs name when initializing event logger.
(cherry picked from commit dd1b7a61d9193c93ab95ab550622259f4bc26f53)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-1582 Invoke Thread.interrupt() when cancelling jobs

Sometimes executor threads are blocked waiting for IO or monitors, and the current implementation of job cancellation may never recover these threads. By simply invoking Thread.interrupt() during cancellation, we can often safely unblock the threads and use them for subsequent work.

Note that this feature must remain optional for now because of a bug in HDFS where Thread.interrupt() may cause nodes to be marked as permanently dead (as the InterruptedException is reinterpreted as an IOException during communication with some node).

Author: Aaron Davidson <aaron@databricks.com>

Closes #498 from aarondav/cancel and squashes the following commits:

e52b829 [Aaron Davidson] Don't use job.properties when null
82f78bb [Aaron Davidson] Update DAGSchedulerSuite
b67f472 [Aaron Davidson] Add comment on why interruptOnCancel is in setJobGroup
4cb9fd6 [Aaron Davidson] SPARK-1582 Invoke Thread.interrupt() when cancelling jobs

(cherry picked from commit 432201c7ee9e1ea1d70a6418cbad1c5ad2653ed3)
Signed-off-by: Reynold Xin <rxin@apache.org>
[SPARK-1540] Add an optional Ordering parameter to PairRDDFunctions.

In https://issues.apache.org/jira/browse/SPARK-1540 we'd like to look at Spark's API to see if we can take advantage of Comparable keys in more places, which will make external spilling more efficient. This PR is a first step towards that that shows how to pass an Ordering when available and still continue functioning otherwise. It does this using a new implicit parameter with a default value of null.

The API is currently only in Scala -- in Java we'd have to add new versions of mapToPair and such that take a Comparator, or a new method to add a "type hint" to an RDD. We can address those later though.

Unfortunately requiring all keys to be Comparable would not work without requiring RDDs in general to contain only Comparable types. The reason is that methods such as distinct() and intersection() do a shuffle, but should be usable on RDDs of any type. So ordering will have to remain an optimization for the types that can be ordered. I think this isn't a horrible outcome though because one of the nice things about Spark's API is that it works on objects of *any* type, without requiring you to specify a schema or implement Writable or stuff like that.

Author: Matei Zaharia <matei@databricks.com>

This patch had conflicts when merged, resolved by
Committer: Reynold Xin <rxin@apache.org>

Closes #487 from mateiz/ordered-keys and squashes the following commits:

bd565f6 [Matei Zaharia] Pass an Ordering to only one version of groupBy because the Scala language spec doesn't allow having an optional parameter on all of them (this was only compiling in Scala 2.10 due to a bug).
4629965 [Matei Zaharia] Add tests for other versions of groupBy
3beae85 [Matei Zaharia] Added a test for implicit orderings
80b7a3b [Matei Zaharia] Add an optional Ordering parameter to PairRDDFunctions.

(cherry picked from commit 640f9a0efefd42cff86aecd4878a3a57f5ae85fa)
Signed-off-by: Reynold Xin <rxin@apache.org>
[maven-release-plugin] prepare release v1.0.0-rc2
[maven-release-plugin] prepare for next development iteration
[Hot Fix #469] Fix flaky test in SparkListenerSuite

The two modified tests may fail if the race condition does not bid in our favor...

Author: Andrew Or <andrewor14@gmail.com>

Closes #516 from andrewor14/stage-info-test-fix and squashes the following commits:

b4b6100 [Andrew Or] Add/replace missing waitUntilEmpty() calls to listener bus

(cherry picked from commit 4b2bab1d08a6b790be94717bbdd643d896d85c16)
Signed-off-by: Reynold Xin <rxin@apache.org>
Update Java api for setJobGroup with interruptOnCancel

Also adds a unit test.

Author: Aaron Davidson <aaron@databricks.com>

Closes #522 from aarondav/cancel2 and squashes the following commits:

565c253 [Aaron Davidson] Update Java api for setJobGroup with interruptOnCancel
65b33d8 [Aaron Davidson] Add unit test for Thread interruption on cancellation

(cherry picked from commit d485eecb7233dd339ae85a6f58f1c0686dd2037d)
Signed-off-by: Reynold Xin <rxin@apache.org>