Merge pull request #441 from pwendell/graphx-build

GraphX shouldn't list Spark as provided.

I noticed this when building an application against GraphX to audit the released artifacts.
(cherry picked from commit 5fecd2516dc8de28b76fe6e0fbdca7922cc28d1c)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #443 from tdas/filestream-fix

Made some classes private[stremaing] and deprecated a method in JavaStreamingContext.

Classes `RawTextHelper`, `RawTextSender` and `RateLimitedOutputStream` are not useful in the streaming API. There are not used by the core functionality and was there as a support classes for an obscure example. One of the classes is RawTextSender has a main function which can be executed using bin/spark-class even if it is made private[streaming]. In future, I will probably completely remove these classes. For the time being, I am just converting them to private[streaming].

Accessing underlying JavaSparkContext in JavaStreamingContext was through `JavaStreamingContext.sc` . This is deprecated and preferred method is `JavaStreamingContext.sparkContext` to keep it consistent with the `StreamingContext.sparkContext`.
(cherry picked from commit 2a05403a7ced4ecf6084c96f582ee3a24f3cc874)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #442 from pwendell/standalone

Workers should use working directory as spark home if it's not specified

If users don't set SPARK_HOME in their environment file when launching an application, the standalone cluster should default to the spark home of the worker.
(cherry picked from commit 59f475c79fc8fd6d3485e4d0adf6768b6a9225a4)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #444 from mateiz/py-version

Clarify that Python 2.7 is only needed for MLlib
(cherry picked from commit 4f0c361b0e140f5f6879f019b2e1a16c683c705c)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Change log for release 0.9.0-incubating
[maven-release-plugin] prepare release v0.9.0-incubating
[maven-release-plugin] prepare for next development iteration
Merge pull request #451 from Qiuzhuang/master

Fixed Window spark shell launch script error.

 JIRA SPARK-1029:https://spark-project.atlassian.net/browse/SPARK-1029
(cherry picked from commit d749d472b37448edb322bc7208a3db925c9a4fc2)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Remove Typesafe Config usage and conf files to fix nested property names

With Typesafe Config we had the subtle problem of no longer allowing
nested property names, which are used for a few of our properties:
http://apache-spark-developers-list.1001551.n3.nabble.com/Config-properties-broken-in-master-td208.html
Merge pull request #461 from pwendell/master

Use renamed shuffle spill config in CoGroupedRDD.scala

This one got missed when it was renamed.
(cherry picked from commit aa981e4e97a11dbd5a4d012bfbdb395982968372)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #462 from mateiz/conf-file-fix

Remove Typesafe Config usage and conf files to fix nested property names

With Typesafe Config we had the subtle problem of no longer allowing
nested property names, which are used for a few of our properties:
http://apache-spark-developers-list.1001551.n3.nabble.com/Config-properties-broken-in-master-td208.html

This PR is for branch 0.9 but should be added into master too.
Merge pull request #426 from mateiz/py-ml-tests

Re-enable Python MLlib tests (require Python 2.7 and NumPy 1.7+)

We disabled these earlier because Jenkins didn't have these versions.
(cherry picked from commit 4c16f79ce45a68ee613a3d565b0e8676b724f867)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #437 from mridulm/master

Minor api usability changes

- Expose checkpoint directory - since it is autogenerated now
- null check for jars
- Expose SparkHadoopUtil : so that configuration creation is abstracted even from user code to avoid duplication of functionality already in spark.
(cherry picked from commit 73dfd42fba5e526cc57e2a2ed78be323b63cb8fa)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #459 from srowen/UpdaterL2Regularization

Correct L2 regularized weight update with canonical form

Per thread on the user@ mailing list, and comments from Ameet, I believe the weight update for L2 regularization needs to be corrected. See http://mail-archives.apache.org/mod_mbox/spark-user/201401.mbox/%3CCAH3_EVMetuQuhj3__NdUniDLc4P-FMmmrmxw9TS14or8nT4BNQ%40mail.gmail.com%3E
(cherry picked from commit fe8a3546f40394466a41fc750cb60f6fc73d8bbb)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Rolling back versions for 0.9.0 release
Typo fix in build versions
[maven-release-plugin] prepare release v0.9.0-incubating
[maven-release-plugin] prepare for next development iteration
Revert "[maven-release-plugin] prepare for next development iteration"

This reverts commit 4f8f86c2c66dc2f6a17d5b0e4fdeeb06a71ba52f.
Revert "[maven-release-plugin] prepare release v0.9.0-incubating"

This reverts commit 77c32470a1b02d6f1475bda2cfb9ae5bd4b53dde.