SPARK-1990: added compatibility for python 2.6 for ssh_read command

https://issues.apache.org/jira/browse/SPARK-1990

There were some posts on the lists that spark-ec2 does not work with Python 2.6. In addition, we should check the Python version at the top of the script and exit if it's too old

Author: Anant <anant.asty@gmail.com>

Closes #941 from anantasty/SPARK-1990 and squashes the following commits:

4ca441d [Anant] Implmented check_optput withinthe module to work with python 2.6
c6ed85c [Anant] added compatibility for python 2.6 for ssh_read command

Conflicts:

	ec2/spark_ec2.py
HOTFIX: bug caused by #941

This patch should have qualified the use of PIPE. This needs to be back ported into 0.9 and 1.0.

Author: Patrick Wendell <pwendell@gmail.com>

Closes #1108 from pwendell/hotfix and squashes the following commits:

711c58d [Patrick Wendell] HOTFIX: bug caused by #941

(cherry picked from commit b2ebf429e24566c29850c570f8d76943151ad78c)
Signed-off-by: Xiangrui Meng <meng@databricks.com>
SPARK-2241: quote command line args in ec2 script

To preserve quoted command line args (in case options have space in them).

Author: Ori Kremer <ori.kremer@gmail.com>

Closes #1169 from orikremer/quote_cmd_line_args and squashes the following commits:

67e2aa1 [Ori Kremer] quote command line args
(cherry picked from commit 9fc373e3a9a8ba7bea9df0950775f48918f63a8a)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
[SPARK-1912] fix compress memory issue during reduce

When we need to read a compressed block, we will first create a compress stream instance(LZF or Snappy) and use it to wrap that block.
Let's say a reducer task need to read 1000 local shuffle blocks, it will first prepare to read that 1000 blocks, which means create 1000 compression stream instance to wrap them. But the initialization of compression instance will allocate some memory and when we have many compression instance at the same time, it is a problem.
Actually reducer reads the shuffle blocks one by one, so we can do the compression instance initialization lazily.

Author: Wenchen Fan(Cloud) <cloud0fan@gmail.com>

Closes #860 from cloud-fan/fix-compress and squashes the following commits:

0924a6b [Wenchen Fan(Cloud)] rename 'doWork' into 'getIterator'
07f32c2 [Wenchen Fan(Cloud)] move the LazyProxyIterator to dataDeserialize
d80c426 [Wenchen Fan(Cloud)] remove empty lines in short class
2c8adb2 [Wenchen Fan(Cloud)] add inline comment
8ebff77 [Wenchen Fan(Cloud)] fix compress memory issue during reduce
HOTFIX: Removing out dated python path in testing tool.
[SPARK-1516]Throw exception in yarn client instead of run system.exit

    [SPARK-1516]Throw exception in yarn client instead of run system.exit directly.

    All the changes is in  the package of "org.apache.spark.deploy.yarn":
        1) Throw IllegalArgumentException in ClinetArguments  instead of exit directly.
        2) In Client's main method, if exception is caught, it will exit with code 1, otherwise exit with code 0.
        3) In YarnClientSchedulerBackend's start method, if IllegalArgumentException is caught, it will exit with code 1, otherwise throw that exception.
        4) Fix some message typo in the Client.scala
    After the fix, if user integrate the spark yarn client into their applications,
    when the argument is wrong or the running is finished, the application won't be terminated.

+CC dbtsai mengxr

Author: John Zhao <codeboyyong@gmail.com>

Closes #1099 from codeboyyong/branch-0.9 and squashes the following commits:

00144b5 [John Zhao] use e.printStackTrace() to replace "Console.err.println(e.getMessage)" so that client console can  get more useful information when somthing is wrong.
addcecb [John Zhao]   [SPARK-1516]Throw exception in yarn client instead of run system.exit directly.
[SPARK-2350] Don't NPE while launching drivers

Prior to this change, we could throw a NPE if we launch a driver while another one is waiting, because removing from an iterator while iterating over it is not safe.

Author: Aaron Davidson <aaron@databricks.com>

Closes #1289 from aarondav/master-fail and squashes the following commits:

1cf1cf4 [Aaron Davidson] SPARK-2350: Don't NPE while launching drivers
(cherry picked from commit 586feb5c9528042420f678f78bacb6c254a5eaf8)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-2282: Reuse PySpark Accumulator sockets to avoid crashing Spark

JIRA: https://issues.apache.org/jira/browse/SPARK-2282

This issue is caused by a buildup of sockets in the TIME_WAIT stage of TCP, which is a stage that lasts for some period of time after the communication closes.

This solution simply allows us to reuse sockets that are in TIME_WAIT, to avoid issues with the buildup of the rapid creation of these sockets.

Author: Aaron Davidson <aaron@databricks.com>

Closes #1220 from aarondav/SPARK-2282 and squashes the following commits:

2e5cab3 [Aaron Davidson] SPARK-2282: Reuse PySpark Accumulator sockets to avoid crashing Spark
(cherry picked from commit 97a0bfe1c0261384f09d53f9350de52fb6446d59)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
[SPARK-2362] Fix for newFilesOnly logic in file DStream

The newFilesOnly logic should be inverted: the logic should be that if the flag newFilesOnly==true then only start reading files older than current time. As the code is now if newFilesOnly==true then it will start to read files that are older than 0L (that is: every file in the directory).

Author: Gabriele Nizzoli <mail@nizzoli.net>

Closes #1077 from gabrielenizzoli/master and squashes the following commits:

4f1d261 [Gabriele Nizzoli] Fix for newFilesOnly logic in file DStream

(cherry picked from commit e6f7bfcfbf6aff7a9f8cd8e0a2166d0bf62b0912)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
[SPARK-2433][MLLIB] fix NaiveBayesModel.predict

This is the same as https://github.com/apache/spark/pull/463 , which I forgot to merge into branch-0.9.

Author: Xiangrui Meng <meng@databricks.com>

Closes #1453 from mengxr/nb-transpose-0.9 and squashes the following commits:

bc53ce8 [Xiangrui Meng] fix NaiveBayes
[SPARK-1112, 2156] (0.9 edition) Use correct akka frame size and overhead amounts.

backport #1172 to branch-0.9.

Author: Patrick Wendell <pwendell@gmail.com>

Closes #1455 from mengxr/akka-fix-0.9 and squashes the following commits:

a99f201 [Patrick Wendell] backport PR #1172 to branch-0.9
[branch-0.9] Fix github links in docs

We moved example code in v1.0. The links are no longer valid if still pointing to `tree/master`.

Author: Xiangrui Meng <meng@databricks.com>

Closes #1456 from mengxr/links-0.9 and squashes the following commits:

b7b9260 [Xiangrui Meng] change tree/master to tree/branch-0.9 in docs
[branch-0.9] bump versions for v0.9.2 release candidate

Manually update some version numbers.

Author: Xiangrui Meng <meng@databricks.com>

Closes #1458 from mengxr/v0.9.2-rc and squashes the following commits:

2c38419 [Xiangrui Meng] Merge remote-tracking branch 'apache/branch-0.9' into v0.9.2-rc
7d0fb76 [Xiangrui Meng] change tree/master to tree/branch-0.9 in docs
ea2b205 [Xiangrui Meng] update version in SparkBuild
162af66 [Xiangrui Meng] Merge remote-tracking branch 'apache/branch-0.9' into v0.9.2-rc
bc87035 [Xiangrui Meng] bump version numbers to 0.9.2
[branch-0.9] Update CHANGES.txt

Author: Xiangrui Meng <meng@databricks.com>

Closes #1459 from mengxr/v0.9.2-rc and squashes the following commits:

6fa5a65 [Xiangrui Meng] update CHANGES.txt
[maven-release-plugin] prepare release v0.9.2-rc1
[maven-release-plugin] prepare for next development iteration
[SPARK-2494] [PySpark] make hash of None consistant cross machines

In CPython, hash of None is different cross machines, it will cause wrong result during shuffle. This PR will fix this.

Author: Davies Liu <davies.liu@gmail.com>

Closes #1371 from davies/hash_of_none and squashes the following commits:

d01745f [Davies Liu] add comments, remove outdated unit tests
5467141 [Davies Liu] disable hijack of hash, use it only for partitionBy()
b7118aa [Davies Liu] use __builtin__ instead of __builtins__
839e417 [Davies Liu] hijack hash to make hash of None consistant cross machines

(cherry picked from commit 872538c600a452ead52638c1ccba90643a9fa41c)
Signed-off-by: Matei Zaharia <matei@databricks.com>
[branch-0.9] manually update some version numbers to 0.9.3-SNAPSHOT

Author: Xiangrui Meng <meng@databricks.com>

Closes #1543 from mengxr/after-release-0.9.2 and squashes the following commits:

a74c730 [Xiangrui Meng] manually update some version numbers to 0.9.3-SNAPSHOT
[SPARK-2547]:The clustering documentaion example provided for spark 0.9....

I modified a trivial mistake in the MLlib documentation.
I checked that the python sample code for a k-means clustering can correctly run on a EC2 instance.

Author: Yuu ISHIKAWA <yuu.ishikawa@gmail.com>

Closes #1590 from yu-iskw/branch-0.9 and squashes the following commits:

06eeb94 [Yuu ISHIKAWA] [SPARK-2547]:The clustering documentaion example provided for spark 0.9.1/docs is having a error
[SPARK-791] [PySpark] fix pickle itemgetter with cloudpickle

fix the problem with pickle operator.itemgetter with multiple index.

Author: Davies Liu <davies.liu@gmail.com>

Closes #1627 from davies/itemgetter and squashes the following commits:

aabd7fa [Davies Liu] fix pickle itemgetter with cloudpickle

(cherry picked from commit 92ef02626e793ea853cced4cbfee316f0b748ed7)

Conflicts:
	python/pyspark/tests.py