Spark 1162 Implemented takeOrdered in pyspark.

Since python does not have a library for max heap and usual tricks like inverting values etc.. does not work for all cases.

We have our own implementation of max heap.

Author: Prashant Sharma <prashant.s@imaginea.com>

Closes #97 from ScrapCodes/SPARK-1162/pyspark-top-takeOrdered2 and squashes the following commits:

35f86ba [Prashant Sharma] code review
2b1124d [Prashant Sharma] fixed tests
e8a08e2 [Prashant Sharma] Code review comments.
49e6ba7 [Prashant Sharma] SPARK-1162 added takeOrdered to pyspark

(cherry picked from commit c1ea3afb516c204925259f0928dfb17d0fa89621)
Signed-off-by: Matei Zaharia <matei@databricks.com>
[SPARK-1134] Fix and document passing of arguments to IPython

This is based on @dianacarroll's previous pull request https://github.com/apache/spark/pull/227, and @joshrosen's comments on https://github.com/apache/spark/pull/38. Since we do want to allow passing arguments to IPython, this does the following:
* It documents that IPython can't be used with standalone jobs for now. (Later versions of IPython will deal with PYTHONSTARTUP properly and enable this, see https://github.com/ipython/ipython/pull/5226, but no released version has that fix.)
* If you run `pyspark` with `IPYTHON=1`, it passes your command-line arguments to it. This way you can do stuff like `IPYTHON=1 bin/pyspark notebook`.
* The old `IPYTHON_OPTS` remains, but I've removed it from the documentation. This is in case people read an old tutorial that uses it.

This is not a perfect solution and I'd also be okay with keeping things as they are today (ignoring `$@` for IPython and using IPYTHON_OPTS), and only doing the doc change. With this change though, when IPython fixes https://github.com/ipython/ipython/pull/5226, people will immediately be able to do `IPYTHON=1 bin/pyspark myscript.py` to run a standalone script and get all the benefits of running scripts in IPython (presumably better debugging and such). Without it, there will be no way to run scripts in IPython.

@joshrosen you should probably take the final call on this.

Author: Diana Carroll <dcarroll@cloudera.com>

Closes #294 from mateiz/spark-1134 and squashes the following commits:

747bb13 [Diana Carroll] SPARK-1134 bug with ipython prevents non-interactive use with spark; only call ipython if no command line arguments were supplied

(cherry picked from commit a599e43d6e0950f6b6b32150ce264a8c2711470c)
Signed-off-by: Matei Zaharia <matei@databricks.com>
SPARK-1337: Application web UI garbage collects newest stages

Simple fix...

Author: Patrick Wendell <pwendell@gmail.com>

Closes #320 from pwendell/stage-clean-up and squashes the following commits:

29be62e [Patrick Wendell] SPARK-1337: Application web UI garbage collects newest stages instead old ones
(cherry picked from commit ee6e9e7d863022304ac9ced405b353b63accb6ab)

Conflicts:

	core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala
	core/src/test/scala/org/apache/spark/ui/jobs/JobProgressListenerSuite.scala
Update documentation for work around for SPARK-1384

This is to workaround accessing secure hdfs from spark-shell in yarn-client mode.  Note this only applies to branch-0.9 and is intended to be included in the documentation for 0.9.1. The real fix after 0.9.1 is included in https://github.com/apache/spark/pull/287

Author: Thomas Graves <tgraves@apache.org>

Closes #314 from tgravescs/docFix09rc3 and squashes the following commits:

222e848 [Thomas Graves] Update documentation for work around for SPARK-1384
SPARK-1421. Make MLlib work on Python 2.6

The reason it wasn't working was passing a bytearray to stream.write(), which is not supported in Python 2.6 but is in 2.7. (This array came from NumPy when we converted data to send it over to Java). Now we just convert those bytearrays to strings of bytes, which preserves nonprintable characters as well.

Author: Matei Zaharia <matei@databricks.com>

Closes #335 from mateiz/mllib-python-2.6 and squashes the following commits:

f26c59f [Matei Zaharia] Update docs to no longer say we need Python 2.7
a84d6af [Matei Zaharia] SPARK-1421. Make MLlib work on Python 2.6

(cherry picked from commit 0b855167818b9afd2d2aa9f617b9861d77b2425d)
Signed-off-by: Matei Zaharia <matei@databricks.com>
SPARK-1432: Make sure that all metadata fields are properly cleaned

While working on spark-1337 with @pwendell, we noticed that not all of the metadata maps in JobProgessListener were being properly cleaned. This could lead to a (hypothetical) memory leak issue should a job run long enough. This patch aims to address the issue.

Author: Davis Shepherd <davis@conviva.com>

Closes #338 from dgshep/master and squashes the following commits:

a77b65c [Davis Shepherd] In the contex of SPARK-1337: Make sure that all metadata fields are properly cleaned
(cherry picked from commit a3c51c6ea2320efdeb2a6a5c1cd11d714f8994aa)

Conflicts:

	core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala
Fixed typo on Spark quick-start docs.
Update WindowedDStream.scala

update the content of Exception when windowDuration is not multiple of parent.slideDuration

Author: baishuo(白硕) <vc_java@hotmail.com>

Closes #390 from baishuo/windowdstream and squashes the following commits:

533c968 [baishuo(白硕)] Update WindowedDStream.scala

Conflicts:

	streaming/src/main/scala/org/apache/spark/streaming/dstream/WindowedDStream.scala
Small syntax error from previous backport
version number fix

self-explain

Author: Nan Zhu <CodingCat@users.noreply.github.com>

Closes #467 from CodingCat/branch-0.9 and squashes the following commits:

ba36109 [Nan Zhu] remove out-dated comments
9a8810e [Nan Zhu] version number fix