[SPARK-1468] Modify the partition function used by partitionBy.

Make partitionBy use a tweaked version of hash as its default partition function
since the python hash function does not consistently assign the same value
to None across python processes.

Associated JIRA at https://issues.apache.org/jira/browse/SPARK-1468

Author: Erik Selin <erik.selin@jadedpixel.com>

Closes #371 from tyro89/consistent_hashing and squashes the following commits:

201c301 [Erik Selin] Make partitionBy use a tweaked version of hash as its default partition function since the python hash function does not consistently assign the same value to None across python processes.

(cherry picked from commit 8edc9d0330c94b50e01956ae88693cff4e0977b2)
Signed-off-by: Matei Zaharia <matei@databricks.com>
SPARK-1790: Update EC2 scripts to support r3 instance types

Author: Varakhedi Sujeet <svarakhedi@gopivotal.com>

Closes #960 from sujeetv/ec2-r3 and squashes the following commits:

3cb9fd5 [Varakhedi Sujeet] SPARK-1790: Update EC2 scripts to support r3 instance
(cherry picked from commit 11ded3f66f178e4d8d2b23491dd5e0ea23bcf719)

Conflicts:

	ec2/spark_ec2.py
SPARK-2043: ExternalAppendOnlyMap doesn't always find matching keys

The current implementation reads one key with the next hash code as it finishes reading the keys with the current hash code, which may cause it to miss some matches of the next key. This can cause operations like join to give the wrong result when reduce tasks spill to disk and there are hash collisions, as values won't be matched together. This PR fixes it by not reading in that next key, using a peeking iterator instead.

Author: Matei Zaharia <matei@databricks.com>

Closes #986 from mateiz/spark-2043 and squashes the following commits:

0959514 [Matei Zaharia] Added unit test for having many hash collisions
892debb [Matei Zaharia] SPARK-2043: don't read a key with the next hash code in ExternalAppendOnlyMap, instead use a buffered iterator to only read values with the current hash code.

(cherry picked from commit b45c13e7d798f97b92f1a6329528191b8d779c4f)
Signed-off-by: Matei Zaharia <matei@databricks.com>
[SPARK-1870] Made deployment with --jars work in yarn-standalone mode.

Ported from 1.0 branch to 0.9 branch. Sent secondary jars to distributed cache of all containers and add the cached jars to classpath before executors start.

Author: DB Tsai <dbtsai@dbtsai.com>

Closes #1013 from dbtsai/branch-0.9 and squashes the following commits:

c5696f4 [DB Tsai] fix line too long
b085f10 [DB Tsai] Make sure that empty string is filtered out when we get secondary jars
3cc1085 [DB Tsai] changed from var to val
ab94aa1 [DB Tsai] Code formatting.
0956af9 [DB Tsai] Ported SPARK-1870 from 1.0 branch to 0.9 branch
Spark 1384 - Fix spark-shell on yarn access to secure hdfs - branch-0.9 only

Author: Thomas Graves <tgraves@apache.org>

Closes #287 from tgravescs/SPARK-1384 and squashes the following commits:

ae9162a [Thomas Graves] SPARK-1384 - fix spark-shell on yarn access to secure HDFS
[SPARK-1998] SparkFlumeEvent with body bigger than 1020 bytes are not re...

flume event sent to Spark will fail if the body is too large and numHeaders is greater than zero

Author: joyyoj <sunshch@gmail.com>

Closes #951 from joyyoj/master and squashes the following commits:

f4660c5 [joyyoj] [SPARK-1998] SparkFlumeEvent with body bigger than 1020 bytes are not read properly
(cherry picked from commit 29660443077619ee854025b8d0d3d64181724054)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-1990: added compatibility for python 2.6 for ssh_read command

https://issues.apache.org/jira/browse/SPARK-1990

There were some posts on the lists that spark-ec2 does not work with Python 2.6. In addition, we should check the Python version at the top of the script and exit if it's too old

Author: Anant <anant.asty@gmail.com>

Closes #941 from anantasty/SPARK-1990 and squashes the following commits:

4ca441d [Anant] Implmented check_optput withinthe module to work with python 2.6
c6ed85c [Anant] added compatibility for python 2.6 for ssh_read command

Conflicts:

	ec2/spark_ec2.py
HOTFIX: bug caused by #941

This patch should have qualified the use of PIPE. This needs to be back ported into 0.9 and 1.0.

Author: Patrick Wendell <pwendell@gmail.com>

Closes #1108 from pwendell/hotfix and squashes the following commits:

711c58d [Patrick Wendell] HOTFIX: bug caused by #941

(cherry picked from commit b2ebf429e24566c29850c570f8d76943151ad78c)
Signed-off-by: Xiangrui Meng <meng@databricks.com>
SPARK-2241: quote command line args in ec2 script

To preserve quoted command line args (in case options have space in them).

Author: Ori Kremer <ori.kremer@gmail.com>

Closes #1169 from orikremer/quote_cmd_line_args and squashes the following commits:

67e2aa1 [Ori Kremer] quote command line args
(cherry picked from commit 9fc373e3a9a8ba7bea9df0950775f48918f63a8a)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
[SPARK-1912] fix compress memory issue during reduce

When we need to read a compressed block, we will first create a compress stream instance(LZF or Snappy) and use it to wrap that block.
Let's say a reducer task need to read 1000 local shuffle blocks, it will first prepare to read that 1000 blocks, which means create 1000 compression stream instance to wrap them. But the initialization of compression instance will allocate some memory and when we have many compression instance at the same time, it is a problem.
Actually reducer reads the shuffle blocks one by one, so we can do the compression instance initialization lazily.

Author: Wenchen Fan(Cloud) <cloud0fan@gmail.com>

Closes #860 from cloud-fan/fix-compress and squashes the following commits:

0924a6b [Wenchen Fan(Cloud)] rename 'doWork' into 'getIterator'
07f32c2 [Wenchen Fan(Cloud)] move the LazyProxyIterator to dataDeserialize
d80c426 [Wenchen Fan(Cloud)] remove empty lines in short class
2c8adb2 [Wenchen Fan(Cloud)] add inline comment
8ebff77 [Wenchen Fan(Cloud)] fix compress memory issue during reduce
HOTFIX: Removing out dated python path in testing tool.
[SPARK-1516]Throw exception in yarn client instead of run system.exit

    [SPARK-1516]Throw exception in yarn client instead of run system.exit directly.

    All the changes is in  the package of "org.apache.spark.deploy.yarn":
        1) Throw IllegalArgumentException in ClinetArguments  instead of exit directly.
        2) In Client's main method, if exception is caught, it will exit with code 1, otherwise exit with code 0.
        3) In YarnClientSchedulerBackend's start method, if IllegalArgumentException is caught, it will exit with code 1, otherwise throw that exception.
        4) Fix some message typo in the Client.scala
    After the fix, if user integrate the spark yarn client into their applications,
    when the argument is wrong or the running is finished, the application won't be terminated.

+CC dbtsai mengxr

Author: John Zhao <codeboyyong@gmail.com>

Closes #1099 from codeboyyong/branch-0.9 and squashes the following commits:

00144b5 [John Zhao] use e.printStackTrace() to replace "Console.err.println(e.getMessage)" so that client console can  get more useful information when somthing is wrong.
addcecb [John Zhao]   [SPARK-1516]Throw exception in yarn client instead of run system.exit directly.
[SPARK-2350] Don't NPE while launching drivers

Prior to this change, we could throw a NPE if we launch a driver while another one is waiting, because removing from an iterator while iterating over it is not safe.

Author: Aaron Davidson <aaron@databricks.com>

Closes #1289 from aarondav/master-fail and squashes the following commits:

1cf1cf4 [Aaron Davidson] SPARK-2350: Don't NPE while launching drivers
(cherry picked from commit 586feb5c9528042420f678f78bacb6c254a5eaf8)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-2282: Reuse PySpark Accumulator sockets to avoid crashing Spark

JIRA: https://issues.apache.org/jira/browse/SPARK-2282

This issue is caused by a buildup of sockets in the TIME_WAIT stage of TCP, which is a stage that lasts for some period of time after the communication closes.

This solution simply allows us to reuse sockets that are in TIME_WAIT, to avoid issues with the buildup of the rapid creation of these sockets.

Author: Aaron Davidson <aaron@databricks.com>

Closes #1220 from aarondav/SPARK-2282 and squashes the following commits:

2e5cab3 [Aaron Davidson] SPARK-2282: Reuse PySpark Accumulator sockets to avoid crashing Spark
(cherry picked from commit 97a0bfe1c0261384f09d53f9350de52fb6446d59)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
[SPARK-2362] Fix for newFilesOnly logic in file DStream

The newFilesOnly logic should be inverted: the logic should be that if the flag newFilesOnly==true then only start reading files older than current time. As the code is now if newFilesOnly==true then it will start to read files that are older than 0L (that is: every file in the directory).

Author: Gabriele Nizzoli <mail@nizzoli.net>

Closes #1077 from gabrielenizzoli/master and squashes the following commits:

4f1d261 [Gabriele Nizzoli] Fix for newFilesOnly logic in file DStream

(cherry picked from commit e6f7bfcfbf6aff7a9f8cd8e0a2166d0bf62b0912)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
[SPARK-2433][MLLIB] fix NaiveBayesModel.predict

This is the same as https://github.com/apache/spark/pull/463 , which I forgot to merge into branch-0.9.

Author: Xiangrui Meng <meng@databricks.com>

Closes #1453 from mengxr/nb-transpose-0.9 and squashes the following commits:

bc53ce8 [Xiangrui Meng] fix NaiveBayes
[SPARK-1112, 2156] (0.9 edition) Use correct akka frame size and overhead amounts.

backport #1172 to branch-0.9.

Author: Patrick Wendell <pwendell@gmail.com>

Closes #1455 from mengxr/akka-fix-0.9 and squashes the following commits:

a99f201 [Patrick Wendell] backport PR #1172 to branch-0.9
[branch-0.9] Fix github links in docs

We moved example code in v1.0. The links are no longer valid if still pointing to `tree/master`.

Author: Xiangrui Meng <meng@databricks.com>

Closes #1456 from mengxr/links-0.9 and squashes the following commits:

b7b9260 [Xiangrui Meng] change tree/master to tree/branch-0.9 in docs
[branch-0.9] bump versions for v0.9.2 release candidate

Manually update some version numbers.

Author: Xiangrui Meng <meng@databricks.com>

Closes #1458 from mengxr/v0.9.2-rc and squashes the following commits:

2c38419 [Xiangrui Meng] Merge remote-tracking branch 'apache/branch-0.9' into v0.9.2-rc
7d0fb76 [Xiangrui Meng] change tree/master to tree/branch-0.9 in docs
ea2b205 [Xiangrui Meng] update version in SparkBuild
162af66 [Xiangrui Meng] Merge remote-tracking branch 'apache/branch-0.9' into v0.9.2-rc
bc87035 [Xiangrui Meng] bump version numbers to 0.9.2
[branch-0.9] Update CHANGES.txt

Author: Xiangrui Meng <meng@databricks.com>

Closes #1459 from mengxr/v0.9.2-rc and squashes the following commits:

6fa5a65 [Xiangrui Meng] update CHANGES.txt