[maven-release-plugin] prepare for next development iteration
Fix PEP8 violations in examples/src/main/python.

Author: Reynold Xin <rxin@apache.org>

Closes #870 from rxin/examples-python-pep8 and squashes the following commits:

2829e84 [Reynold Xin] Fix PEP8 violations in examples/src/main/python.

(cherry picked from commit d79c2b28e17ec0b15198aaedd2e1f403d81f717e)
Signed-off-by: Reynold Xin <rxin@apache.org>
Python docstring update for sql.py.

Mostly related to the following two rules in PEP8 and PEP257:
- Line length < 72 chars.
- First line should be a concise description of the function/class.

Author: Reynold Xin <rxin@apache.org>

Closes #869 from rxin/docstring-schemardd and squashes the following commits:

7cf0cbc [Reynold Xin] Updated sql.py for pep8 docstring.
0a4aef9 [Reynold Xin] Merge branch 'master' into docstring-schemardd
6678937 [Reynold Xin] Python docstring update for sql.py.

(cherry picked from commit 14f0358b2a0a9b92526bdad6d501ab753459eaa0)
Signed-off-by: Reynold Xin <rxin@apache.org>
Fix PEP8 violations in Python mllib.

Author: Reynold Xin <rxin@apache.org>

Closes #871 from rxin/mllib-pep8 and squashes the following commits:

848416f [Reynold Xin] Fixed a typo in the previous cleanup (c -> sc).
a8db4cd [Reynold Xin] Fix PEP8 violations in Python mllib.

(cherry picked from commit d33d3c61ae9e4551aed0217e525a109e678298f2)
Signed-off-by: Reynold Xin <rxin@apache.org>
SPARK-1903 Document Spark's network connections

https://issues.apache.org/jira/browse/SPARK-1903

Author: Andrew Ash <andrew@andrewash.com>

Closes #856 from ash211/SPARK-1903 and squashes the following commits:

6e7782a [Andrew Ash] Add the technology used on each port
1d9b5d3 [Andrew Ash] Document port for history server
56193ee [Andrew Ash] spark.ui.port becomes worker.ui.port and master.ui.port
a774c07 [Andrew Ash] Wording in network section
90e8237 [Andrew Ash] Use real :toc instead of the hand-written one
edaa337 [Andrew Ash] Master -> Standalone Cluster Master
57e8869 [Andrew Ash] Port -> Default Port
3d4d289 [Andrew Ash] Title to title case
c7d42d9 [Andrew Ash] [WIP] SPARK-1903 Add initial port listing for documentation
a416ae9 [Andrew Ash] Word wrap to 100 lines

(cherry picked from commit 0659529614c804e0c04efc59cb67dab3a6cdc9d9)
Signed-off-by: Reynold Xin <rxin@apache.org>
[SQL] Minor: Introduce SchemaRDD#aggregate() for simple aggregations

```scala
rdd.aggregate(Sum('val))
```
is just shorthand for

```scala
rdd.groupBy()(Sum('val))
```

but seems be more natural than doing a groupBy with no grouping expressions when you really just want an aggregation over all rows.

Did not add a JavaSchemaRDD or Python API, as these seem to be lacking several other methods like groupBy() already -- leaving that cleanup for future patches.

Author: Aaron Davidson <aaron@databricks.com>

Closes #874 from aarondav/schemardd and squashes the following commits:

e9e68ee [Aaron Davidson] Add comment
db6afe2 [Aaron Davidson] Introduce SchemaRDD#aggregate() for simple aggregations

(cherry picked from commit c3576ffcd7910e38928f233a824dd9e037cde05f)
Signed-off-by: Reynold Xin <rxin@apache.org>
HOTFIX: Add no-arg SparkContext constructor in Java

Self explanatory.

Author: Patrick Wendell <pwendell@gmail.com>

Closes #878 from pwendell/java-constructor and squashes the following commits:

2cc1605 [Patrick Wendell] HOTFIX: Add no-arg SparkContext constructor in Java

(cherry picked from commit b6d22af040073cd611b0fcfdf8a5259c0dfd854c)
Signed-off-by: Aaron Davidson <aaron@databricks.com>
Revert "[maven-release-plugin] prepare for next development iteration"

This reverts commit 832dc594e7666f1d402334f8015ce29917d9c888.
Revert "[maven-release-plugin] prepare release v1.0.0-rc11"

This reverts commit 2f1dc868e5714882cf40d2633fb66772baf34789.
Updated CHANGES.txt
[maven-release-plugin] prepare release v1.0.0-rc11
[maven-release-plugin] prepare for next development iteration
[SPARK-1914] [SQL] Simplify CountFunction not to traverse to evaluate all child expressions.

`CountFunction` should count up only if the child's evaluated value is not null.

Because it traverses to evaluate all child expressions, even if the child is null, it counts up if one of the all children is not null.

Author: Takuya UESHIN <ueshin@happy-camper.st>

Closes #861 from ueshin/issues/SPARK-1914 and squashes the following commits:

3b37315 [Takuya UESHIN] Merge branch 'master' into issues/SPARK-1914
2afa238 [Takuya UESHIN] Simplify CountFunction not to traverse to evaluate all child expressions.

(cherry picked from commit d6395d86f90d1c47c5b6ad17c618b56e00b7fc85)
Signed-off-by: Reynold Xin <rxin@apache.org>
SPARK-1925: Replace '&' with '&&'

JIRA: https://issues.apache.org/jira/browse/SPARK-1925

Author: zsxwing <zsxwing@gmail.com>

Closes #879 from zsxwing/SPARK-1925 and squashes the following commits:

5cf5a6d [zsxwing] SPARK-1925: Replace '&' with '&&'

(cherry picked from commit cb7fe5034826844f1b50fbe8b92646317b66f21c)
Signed-off-by: Reynold Xin <rxin@apache.org>
[SPARK-1931] Reconstruct routing tables in Graph.partitionBy

905173df57b90f90ebafb22e43f55164445330e6 introduced a bug in partitionBy where, after repartitioning the edges, it reuses the VertexRDD without updating the routing tables to reflect the new edge layout. Subsequent accesses of the triplets contain nulls for many vertex properties.

This commit adds a test for this bug and fixes it by introducing `VertexRDD#withEdges` and calling it in `partitionBy`.

Author: Ankur Dave <ankurdave@gmail.com>

Closes #885 from ankurdave/SPARK-1931 and squashes the following commits:

3930cdd [Ankur Dave] Note how to set up VertexRDD for efficient joins
9bdbaa4 [Ankur Dave] [SPARK-1931] Reconstruct routing tables in Graph.partitionBy

(cherry picked from commit 56c771cb2d00a5843c391ae6561536ee46e535d4)
Signed-off-by: Reynold Xin <rxin@apache.org>
Updated dev Python scripts to make them PEP8 compliant.

Author: Reynold Xin <rxin@apache.org>

Closes #875 from rxin/pep8-dev-scripts and squashes the following commits:

04b084f [Reynold Xin] Made dev Python scripts PEP8 compliant.

(cherry picked from commit 9ed37190f45fd9e6aa0f2c73b66d317732a53eb8)
Signed-off-by: Reynold Xin <rxin@apache.org>
SPARK-1933: Throw a more meaningful exception when a directory is passed to addJar/addFile.

https://issues.apache.org/jira/browse/SPARK-1933

Author: Reynold Xin <rxin@apache.org>

Closes #888 from rxin/addfile and squashes the following commits:

8c402a3 [Reynold Xin] Updated comment.
ff6c162 [Reynold Xin] SPARK-1933: Throw a more meaningful exception when a directory is passed to addJar/addFile.

(cherry picked from commit 90e281b55aecbfbe4431ac582311d5790fe7aad3)
Signed-off-by: Reynold Xin <rxin@apache.org>
SPARK-1932: Fix race conditions in onReceiveCallback and cachedPeers

`var cachedPeers: Seq[BlockManagerId] = null` is used in `def replicate(blockId: BlockId, data: ByteBuffer, level: StorageLevel)` without proper protection.

There are two place will call `replicate(blockId, bytesAfterPut, level)`
* https://github.com/apache/spark/blob/17f3075bc4aa8cbed165f7b367f70e84b1bc8db9/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L644 runs in `connectionManager.futureExecContext`
* https://github.com/apache/spark/blob/17f3075bc4aa8cbed165f7b367f70e84b1bc8db9/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L752 `doPut` runs in `connectionManager.handleMessageExecutor`. `org.apache.spark.storage.BlockManagerWorker` calls `blockManager.putBytes` in `connectionManager.handleMessageExecutor`.

As they run in different `Executor`s, this is a race condition which may cause the memory pointed by `cachedPeers` is not correct even if `cachedPeers != null`.

The race condition of `onReceiveCallback` is that it's set in `BlockManagerWorker` but read in a different thread in `ConnectionManager.handleMessageExecutor`.

Author: zsxwing <zsxwing@gmail.com>

Closes #887 from zsxwing/SPARK-1932 and squashes the following commits:

524f69c [zsxwing] SPARK-1932: Fix race conditions in onReceiveCallback and cachedPeers

(cherry picked from commit 549830b0db2c8b069391224f3a73bb0d7f397f71)
Signed-off-by: Aaron Davidson <aaron@databricks.com>
bugfix worker DriverStateChanged state should match DriverState.FAILED

bugfix worker DriverStateChanged state should match DriverState.FAILED

Author: lianhuiwang <lianhuiwang09@gmail.com>

Closes #864 from lianhuiwang/master and squashes the following commits:

480ce94 [lianhuiwang] address aarondav comments
f2b5970 [lianhuiwang] bugfix worker DriverStateChanged state should match DriverState.FAILED
[SPARK-1926] [SQL] Nullability of Max/Min/First should be true.

Nullability of `Max`/`Min`/`First` should be `true` because they return `null` if there are no rows.

Author: Takuya UESHIN <ueshin@happy-camper.st>

Closes #881 from ueshin/issues/SPARK-1926 and squashes the following commits:

322610f [Takuya UESHIN] Fix nullability of Min/Max/First.

(cherry picked from commit d1375a2bff846f2c4274e14545924646852895f9)
Signed-off-by: Reynold Xin <rxin@apache.org>