Update versions to 1.1.2-SNAPSHOT
[SPARK-4196][SPARK-4602][Streaming] Fix serialization issue in PairDStreamFunctions.saveAsNewAPIHadoopFiles

Solves two JIRAs in one shot
- Makes the ForechDStream created by saveAsNewAPIHadoopFiles serializable for checkpoints
- Makes the default configuration object used saveAsNewAPIHadoopFiles be the Spark's hadoop configuration

Author: Tathagata Das <tathagata.das1565@gmail.com>

Closes #3457 from tdas/savefiles-fix and squashes the following commits:

bb4729a [Tathagata Das] Same treatment for saveAsHadoopFiles
b382ea9 [Tathagata Das] Fix serialization issue in PairDStreamFunctions.saveAsNewAPIHadoopFiles.

(cherry picked from commit 8838ad7c135a585cde015dc38b5cb23314502dd9)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
[HOTFIX] Fixing broken build due to missing imports.
[Release] Automate generation of contributors list

This commit provides a script that computes the contributors list
by linking the github commits with JIRA issues. Automatically
translating github usernames remains a TODO at this point.
[BRANCH-1.1][SPARK-4626] Kill a task only if the executorId is (still) registered with the scheduler

v1.1 backport for #3483

Author: roxchkplusony <roxchkplusony@gmail.com>

Closes #3503 from roxchkplusony/bugfix/4626-1.1 and squashes the following commits:

234d350 [roxchkplusony] [SPARK-4626] Kill a task only if the executorId is (still) registered with the scheduler
[SPARK-4597] Use proper exception and reset variable in Utils.createTempDir()

`File.exists()` and `File.mkdirs()` only throw `SecurityException` instead of `IOException`. Then, when an exception is thrown, `dir` should be reset too.

Author: Liang-Chi Hsieh <viirya@gmail.com>

Closes #3449 from viirya/fix_createtempdir and squashes the following commits:

36cacbd [Liang-Chi Hsieh] Use proper exception and reset variable.

(cherry picked from commit 49fe8797e64f10c574e0790b32a8c3fdc7e594a0)
Signed-off-by: Josh Rosen <joshrosen@databricks.com>
SPARK-2143 [WEB UI] Add Spark version to UI footer

This PR adds the Spark version number to the UI footer; this is how it looks:

![screen shot 2014-11-21 at 22 58 40](https://cloud.githubusercontent.com/assets/822522/5157738/f4822094-7316-11e4-98f1-333a535fdcfa.png)

Author: Sean Owen <sowen@cloudera.com>

Closes #3410 from srowen/SPARK-2143 and squashes the following commits:

e9b3a7a [Sean Owen] Add Spark version to footer
[HOTFIX] Fix build break in 1a2508b73f6d46a0faf7740b85a5c216c925c25a

org.apache.spark.SPARK_VERSION is new in 1.2; in earlier versions,
we have to use SparkContext.SPARK_VERSION.
[DOC] Fixes formatting typo in SQL programming guide

<!-- Reviewable:start -->
[<img src="https://reviewable.io/review_button.png" height=40 alt="Review on Reviewable"/>](https://reviewable.io/reviews/apache/spark/3498)
<!-- Reviewable:end -->

Author: Cheng Lian <lian@databricks.com>

Closes #3498 from liancheng/fix-sql-doc-typo and squashes the following commits:

865ecd7 [Cheng Lian] Fixes formatting typo in SQL programming guide

(cherry picked from commit 2a4d389f70b2066b1ac32b081bef44e61fefb03c)
Signed-off-by: Josh Rosen <joshrosen@databricks.com>
[SPARK-4686] Link to allowed master URLs is broken

The link points to the old scala programming guide; it should point to the submitting applications page.

This should be backported to 1.1.2 (it's been broken as of 1.0).

Author: Kay Ousterhout <kayousterhout@gmail.com>

Closes #3542 from kayousterhout/SPARK-4686 and squashes the following commits:

a8fc43b [Kay Ousterhout] [SPARK-4686] Link to allowed master URLs is broken

(cherry picked from commit d9a148ba6a67a01e4bf77c35c41dd4cbc8918c82)
Signed-off-by: Kay Ousterhout <kayousterhout@gmail.com>
[Release] Translate unknown author names automatically
[SPARK-4701] Typo in sbt/sbt

Modified typo.

Author: Masayoshi TSUZUKI <tsudukim@oss.nttdata.co.jp>

Closes #3560 from tsudukim/feature/SPARK-4701 and squashes the following commits:

ed2a3f1 [Masayoshi TSUZUKI] Another whitespace position error.
1af3a35 [Masayoshi TSUZUKI] [SPARK-4701] Typo in sbt/sbt

(cherry picked from commit 96786e3ee53a13a57463b74bec0e77b172f719a3)
Signed-off-by: Andrew Or <andrew@databricks.com>
[SPARK-4715][Core] Make sure tryToAcquire won't return a negative value

ShuffleMemoryManager.tryToAcquire may return a negative value. The unit test demonstrates this bug. It will output `0 did not equal -200 granted is negative`.

Author: zsxwing <zsxwing@gmail.com>

Closes #3575 from zsxwing/SPARK-4715 and squashes the following commits:

a193ae6 [zsxwing] Make sure tryToAcquire won't return a negative value
[SPARK-4642] Add description about spark.yarn.queue to running-on-YARN document.

Added descriptions about these parameters.
- spark.yarn.queue

Modified description about the defalut value of this parameter.
- spark.yarn.submit.file.replication

Author: Masayoshi TSUZUKI <tsudukim@oss.nttdata.co.jp>

Closes #3500 from tsudukim/feature/SPARK-4642 and squashes the following commits:

ce99655 [Masayoshi TSUZUKI] better gramatically.
21cf624 [Masayoshi TSUZUKI] Removed intentionally undocumented properties.
88cac9b [Masayoshi TSUZUKI] [SPARK-4642] Documents about running-on-YARN needs update
[SPARK-4498][core] Don't transition ExecutorInfo to RUNNING until Driver adds Executor

The ExecutorInfo only reaches the RUNNING state if the Driver is alive to send the ExecutorStateChanged message to master.  Else, appInfo.resetRetryCount() is never called and failing Executors will eventually exceed ApplicationState.MAX_NUM_RETRY, resulting in the application being removed from the master's accounting.

Author: Mark Hamstra <markhamstra@gmail.com>

Closes #3550 from markhamstra/SPARK-4498 and squashes the following commits:

8f543b1 [Mark Hamstra] Don't transition ExecutorInfo to RUNNING until Executor is added by Driver
[Release] Correctly translate contributors name in release notes

This commit involves three main changes:

(1) It separates the translation of contributor names from the
generation of the contributors list. This is largely motivated
by the Github API limit; even if we exceed this limit, we should
at least be able to proceed manually as before. This is why the
translation logic is abstracted into its own script
translate-contributors.py.

(2) When we look for candidate replacements for invalid author
names, we should look for the assignees of the associated JIRAs
too. As a result, the intermediate file must keep track of these.

(3) This provides an interactive mode with which the user can
sit at the terminal and manually pick the candidate replacement
that he/she thinks makes the most sense. As before, there is a
non-interactive mode that picks the first candidate that the
script considers "valid."

TODO: We should have a known_contributors file that stores
known mappings so we don't have to go through all of this
translation every time. This is also valuable because some
contributors simply cannot be automatically translated.

Conflicts:
	.gitignore
[SPARK-4253] Ignore spark.driver.host in yarn-cluster and standalone-cluster modes

In yarn-cluster and standalone-cluster modes, we don't know where driver will run until it is launched.  If the `spark.driver.host` property is set on the submitting machine and propagated to the driver through SparkConf then this will lead to errors when the driver launches.

This patch fixes this issue by dropping the `spark.driver.host` property in SparkSubmit when running in a cluster deploy mode.

Author: WangTaoTheTonic <barneystinson@aliyun.com>
Author: WangTao <barneystinson@aliyun.com>

Closes #3112 from WangTaoTheTonic/SPARK4253 and squashes the following commits:

ed1a25c [WangTaoTheTonic] revert unrelated formatting issue
02c4e49 [WangTao] add comment
32a3f3f [WangTaoTheTonic] ingore it in SparkSubmit instead of SparkContext
667cf24 [WangTaoTheTonic] document fix
ff8d5f7 [WangTaoTheTonic] also ignore it in standalone cluster mode
2286e6b [WangTao] ignore spark.driver.host in yarn-cluster mode

(cherry picked from commit 8106b1e36b2c2b9f5dc5d7252540e48cc3fc96d5)
Signed-off-by: Josh Rosen <joshrosen@databricks.com>

Conflicts:
	core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
[SPARK-4745] Fix get_existing_cluster() function with multiple security groups

The current get_existing_cluster() function would only find an instance belonged to a cluster if the instance's security groups == cluster_name + "-master" (or "-slaves"). This fix allows for multiple security groups by checking if the cluster_name + "-master" security group is in the list of groups for a particular instance.

Author: alexdebrie <alexdebrie1@gmail.com>

Closes #3596 from alexdebrie/master and squashes the following commits:

9d51232 [alexdebrie] Fix get_existing_cluster() function with multiple security groups

(cherry picked from commit 794f3aec24acb578e258532ad0590554d07958ba)
Signed-off-by: Josh Rosen <joshrosen@databricks.com>
[SPARK-4459] Change groupBy type parameter from K to U

Please see https://issues.apache.org/jira/browse/SPARK-4459

Author: Saldanha <saldaal1@phusca-l24858.wlan.na.novartis.net>

Closes #3327 from alokito/master and squashes the following commits:

54b1095 [Saldanha] [SPARK-4459] changed type parameter for keyBy from K to U
d5f73c3 [Saldanha] [SPARK-4459] added keyBy test
316ad77 [Saldanha] SPARK-4459 changed type parameter for groupBy from K to U.
62ddd4b [Saldanha] SPARK-4459 added failing unit test
(cherry picked from commit 743a889d2778f797aabc3b1e8146e7aa32b62a48)

Signed-off-by: Josh Rosen <joshrosen@databricks.com>
[SPARK-4652][DOCS] Add docs about spark-git-repo option

There might be some cases when WIPS spark version need to be run
on EC2 cluster. In order to setup this type of cluster more easily,
add --spark-git-repo option description to ec2 documentation.

Author: lewuathe <lewuathe@me.com>
Author: Josh Rosen <joshrosen@databricks.com>

Closes #3513 from Lewuathe/doc-for-development-spark-cluster and squashes the following commits:

6dae8ee [lewuathe] Wrap consistent with other descriptions
cfaf9be [lewuathe] Add docs about spark-git-repo option

(Editing / cleanup by Josh Rosen)

(cherry picked from commit ab8177da2defab1ecd8bc0cd5a21f07be5b8d2c5)
Signed-off-by: Josh Rosen <joshrosen@databricks.com>