[maven-release-plugin] prepare for next development iteration
Merge pull request #478 from sryza/sandy-spark-1033

SPARK-1033. Ask for cores in Yarn container requests

Tested on a pseudo-distributed cluster against the Fair Scheduler and observed a worker taking more than a single core.
(cherry picked from commit 576c4a4c502ccca5fcd6b3552dd93cc2f3c50666)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #492 from skicavs/master

fixed job name and usage information for the JavaSparkPi example
(cherry picked from commit a1238bb5fcab763d32c729ea7ed99cb3c05c896f)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #495 from srowen/GraphXCommonsMathDependency

Fix graphx Commons Math dependency

`graphx` depends on Commons Math (2.x) in `SVDPlusPlus.scala`. However the module doesn't declare this dependency. It happens to work because it is included by Hadoop artifacts. But, I can tell you this isn't true as of a month or so ago. Building versus recent Hadoop would fail. (That's how we noticed.)

The simple fix is to declare the dependency, as it should be. But it's also worth noting that `commons-math` is the old-ish 2.x line, while `commons-math3` is where newer 3.x releases are. Drop-in replacement, but different artifact and package name. Changing this only usage to `commons-math3` works, tests pass, and isn't surprising that it does, so is probably also worth changing. (A comment in some test code also references `commons-math3`, FWIW.)

It does raise another question though: `mllib` looks like it uses the `jblas` `DoubleMatrix` for general purpose vector/matrix stuff. Should `graphx` really use Commons Math for this? Beyond the tiny scope here but worth asking.
(cherry picked from commit 3184facdc5b1e9ded89133f9b1e4985c9ac78c55)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #496 from pwendell/master

Fix bug in worker clean-up in UI

Introduced in d5a96fec (/cc @aarondav).

This should be picked into 0.8 and 0.9 as well. The bug causes old (zombie) workers on a node to not disappear immediately from the UI when a new one registers.
(cherry picked from commit a1cd185122602c96fb8ae16c0b506702283bf6e2)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #501 from JoshRosen/cartesian-rdd-fixes

Fix two bugs in PySpark cartesian(): SPARK-978 and SPARK-1034

This pull request fixes two bugs in PySpark's `cartesian()` method:

- [SPARK-978](https://spark-project.atlassian.net/browse/SPARK-978): PySpark's cartesian method throws ClassCastException exception
- [SPARK-1034](https://spark-project.atlassian.net/browse/SPARK-1034): Py4JException on PySpark Cartesian Result

The JIRAs have more details describing the fixes.
(cherry picked from commit cad3002fead89d3c9a8de4fa989e88f367bc0b05)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #502 from pwendell/clone-1

Remove Hadoop object cloning and warn users making Hadoop RDD's.

The code introduced in #359 used Hadoop's WritableUtils.clone() to
duplicate objects when reading from Hadoop files. Some users have
reported exceptions when cloning data in various file formats,
including Avro and another custom format.

This patch removes that functionality to ensure stability for the
0.9 release. Instead, it puts a clear warning in the documentation
that copying may be necessary for Hadoop data sets.
(cherry picked from commit c3196171f3dffde6c9e67e3d35c398a01fbba846)

Conflicts:

	core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala
Merge pull request #503 from pwendell/master

Fix bug on read-side of external sort when using Snappy.

This case wasn't handled correctly and this patch fixes it.
(cherry picked from commit 3d6e75419330d27435becfdf8cfb0b6d20d56cf8)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #505 from JoshRosen/SPARK-1026

Deprecate mapPartitionsWithSplit in PySpark (SPARK-1026)

This commit deprecates `mapPartitionsWithSplit` in PySpark (see [SPARK-1026](https://spark-project.atlassian.net/browse/SPARK-1026) and removes the remaining references to it from the docs.
(cherry picked from commit 05be7047744c88e64e7e6bd973f9bcfacd00da5f)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Revert "[maven-release-plugin] prepare for next development iteration"

This reverts commit e1dc5bedb48d2ac9b9e1c9b3b1a15c41b7d90ad8.
Revert "[maven-release-plugin] prepare release v0.9.0-incubating"

This reverts commit 0771df675363c69622404cb514bd751bc90526af.
Updating changes file
[maven-release-plugin] prepare release v0.9.0-incubating
Merge tag 'v0.9.0-incubating' of https://github.com/apache/incubator-spark into csd-2.10

[maven-release-plugin]  copy for tag v0.9.0-incubating