Rename testExecutorEnvs --> executorEnvs.

This was changed, but in fact, it's used for things other than tests.
So I've changed it back.

Author: Patrick Wendell <pwendell@gmail.com>

Closes #747 from pwendell/executor-env and squashes the following commits:

36a60a5 [Patrick Wendell] Rename testExecutorEnvs --> executorEnvs.
(cherry picked from commit 3ce526b168050c572a1feee8e0121e1426f7d9ee)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-1623: Use File objects instead of String's in HTTPBroadcast

This seems strictly better, and I think it's justified only the grounds of
clean-up. It might also fix issues with path conversions, but I haven't
yet isolated any instance of that happening.

/cc @srowen @tdas

Author: Patrick Wendell <pwendell@gmail.com>

Closes #749 from pwendell/broadcast-cleanup and squashes the following commits:

d6d54f2 [Patrick Wendell] SPARK-1623: Use File objects instead of string's in HTTPBroadcast
(cherry picked from commit 925d8b249b84d2706c52f0d1e29fb8dcd6de452e)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-1802. (Addendium) Audit dependency graph when Spark is built with -Pyarn

Following on a few more items from SPARK-1802 --

The first commit touches up a few similar problems remaining with the YARN profile. I think this is worth cherry-picking.

The second commit is more of the same for hadoop-client, although the fix is a little more complex. It may or may not be worth bothering with.

Author: Sean Owen <sowen@cloudera.com>

Closes #746 from srowen/SPARK-1802.2 and squashes the following commits:

52aeb41 [Sean Owen] Add more commons-logging, servlet excludes to avoid conflicts in assembly when building for YARN
(cherry picked from commit 4b31f4ec7efab8eabf956284a99bfd96a58b79f7)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
[SPARK-1736] Spark submit for Windows

Tested on Windows 7.

Author: Andrew Or <andrewor14@gmail.com>

Closes #745 from andrewor14/windows-submit and squashes the following commits:

c0b58fb [Andrew Or] Allow spaces in parameters
162e54d [Andrew Or] Merge branch 'master' of github.com:apache/spark into windows-submit
91597ce [Andrew Or] Make spark-shell.cmd use spark-submit.cmd
af6fd29 [Andrew Or] Add spark submit for Windows
[SQL] Make Hive Metastore conversion functions publicly visible.

I need this to be public for the implementation of SharkServer2.  However, I think this functionality is generally useful and should be pretty stable.

Author: Michael Armbrust <michael@databricks.com>

Closes #750 from marmbrus/metastoreTypes and squashes the following commits:

f51b62e [Michael Armbrust] Make Hive Metastore conversion functions publicly visible.

(cherry picked from commit 2f1a3373583f9b34a121236c25f5142ba8729546)
Signed-off-by: Reynold Xin <rxin@apache.org>
Typo: resond -> respond

Author: Andrew Ash <andrew@andrewash.com>

Closes #743 from ash211/patch-4 and squashes the following commits:

c959f3b [Andrew Ash] Typo: resond -> respond

(cherry picked from commit a5150d199ca97ab2992bc2bb221a3ebf3d3450ba)
Signed-off-by: Reynold Xin <rxin@apache.org>
L-BFGS Documentation

Documentation for L-BFGS, and an example of training binary L2 logistic regression using L-BFGS.

Author: DB Tsai <dbtsai@alpinenow.com>

Closes #702 from dbtsai/dbtsai-lbfgs-doc and squashes the following commits:

0712215 [DB Tsai] Update
38fdfa1 [DB Tsai] Removed extra empty line
5745b64 [DB Tsai] Update again
e9e418e [DB Tsai] Update
7381521 [DB Tsai] L-BFGS Documentation

(cherry picked from commit 5c2275d6e4639946fd11ff6403338c8a9ade3d1e)
Signed-off-by: Reynold Xin <rxin@apache.org>
Modify a typo in monitoring.md

As I mentioned in SPARK-1765, there is a word 'JXM' in monitoring.md.
I think it's typo for 'JMX'.

Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>

Closes #698 from sarutak/SPARK-1765 and squashes the following commits:

bae9843 [Kousuke Saruta] modified a typoe in monitoring.md

(cherry picked from commit 9cf9f18973840f7287f7cfa5ce90efed3225bb30)
Signed-off-by: Reynold Xin <rxin@apache.org>
SPARK-1757 Failing test for saving null primitives with .saveAsParquetFile()

https://issues.apache.org/jira/browse/SPARK-1757

The first test succeeds, but the second test fails with exception:

```
[info] - save and load case class RDD with Nones as parquet *** FAILED *** (14 milliseconds)
[info]   java.lang.RuntimeException: Unsupported datatype StructType(List())
[info]   at scala.sys.package$.error(package.scala:27)
[info]   at org.apache.spark.sql.parquet.ParquetTypesConverter$.fromDataType(ParquetRelation.scala:201)
[info]   at org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$1.apply(ParquetRelation.scala:235)
[info]   at org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$1.apply(ParquetRelation.scala:235)
[info]   at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
[info]   at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
[info]   at scala.collection.immutable.List.foreach(List.scala:318)
[info]   at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
[info]   at scala.collection.AbstractTraversable.map(Traversable.scala:105)
[info]   at org.apache.spark.sql.parquet.ParquetTypesConverter$.convertFromAttributes(ParquetRelation.scala:234)
[info]   at org.apache.spark.sql.parquet.ParquetTypesConverter$.writeMetaData(ParquetRelation.scala:267)
[info]   at org.apache.spark.sql.parquet.ParquetRelation$.createEmpty(ParquetRelation.scala:143)
[info]   at org.apache.spark.sql.parquet.ParquetRelation$.create(ParquetRelation.scala:122)
[info]   at org.apache.spark.sql.execution.SparkStrategies$ParquetOperations$.apply(SparkStrategies.scala:139)
[info]   at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58)
[info]   at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58)
[info]   at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
[info]   at org.apache.spark.sql.catalyst.planning.QueryPlanner.apply(QueryPlanner.scala:59)
[info]   at org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan$lzycompute(SQLContext.scala:264)
[info]   at org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan(SQLContext.scala:264)
[info]   at org.apache.spark.sql.SQLContext$QueryExecution.executedPlan$lzycompute(SQLContext.scala:265)
[info]   at org.apache.spark.sql.SQLContext$QueryExecution.executedPlan(SQLContext.scala:265)
[info]   at org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:268)
[info]   at org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:268)
[info]   at org.apache.spark.sql.SchemaRDDLike$class.saveAsParquetFile(SchemaRDDLike.scala:66)
[info]   at org.apache.spark.sql.SchemaRDD.saveAsParquetFile(SchemaRDD.scala:98)
```

Author: Andrew Ash <andrew@andrewash.com>
Author: Michael Armbrust <michael@databricks.com>

Closes #690 from ash211/rdd-parquet-save and squashes the following commits:

747a0b9 [Andrew Ash] Merge pull request #1 from marmbrus/pr/690
54bd00e [Michael Armbrust] Need to put Option first since Option <: Seq.
8f3f281 [Andrew Ash] SPARK-1757 Add failing test for saving SparkSQL Schemas with Option[?] fields as parquet

(cherry picked from commit 156df87e7ca0e6cda2cc970ecd1466ce06f7576f)
Signed-off-by: Reynold Xin <rxin@apache.org>
[SPARK-1780] Non-existent SPARK_DAEMON_OPTS is lurking around

What they really mean is SPARK_DAEMON_***JAVA***_OPTS

Author: Andrew Or <andrewor14@gmail.com>

Closes #751 from andrewor14/spark-daemon-opts and squashes the following commits:

70c41f9 [Andrew Or] SPARK_DAEMON_OPTS -> SPARK_DAEMON_JAVA_OPTS
(cherry picked from commit ba96bb3d591130075763706526f86fb2aaffa3ae)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
[SPARK-1753 / 1773 / 1814] Update outdated docs for spark-submit, YARN, standalone etc.

YARN
- SparkPi was updated to not take in master as an argument; we should update the docs to reflect that.
- The default YARN build guide should be in maven, not sbt.
- This PR also adds a paragraph on steps to debug a YARN application.

Standalone
- Emphasize spark-submit more. Right now it's one small paragraph preceding the legacy way of launching through `org.apache.spark.deploy.Client`.
- The way we set configurations / environment variables according to the old docs is outdated. This needs to reflect changes introduced by the Spark configuration changes we made.

In general, this PR also adds a little more documentation on the new spark-shell, spark-submit, spark-defaults.conf etc here and there.

Author: Andrew Or <andrewor14@gmail.com>

Closes #701 from andrewor14/yarn-docs and squashes the following commits:

e2c2312 [Andrew Or] Merge in changes in #752 (SPARK-1814)
25cfe7b [Andrew Or] Merge in the warning from SPARK-1753
a8c39c5 [Andrew Or] Minor changes
336bbd9 [Andrew Or] Tabs -> spaces
4d9d8f7 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs
041017a [Andrew Or] Abstract Spark submit documentation to cluster-overview.html
3cc0649 [Andrew Or] Detail how to set configurations + remove legacy instructions
5b7140a [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs
85a51fc [Andrew Or] Update run-example, spark-shell, configuration etc.
c10e8c7 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs
381fe32 [Andrew Or] Update docs for standalone mode
757c184 [Andrew Or] Add a note about the requirements for the debugging trick
f8ca990 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs
924f04c [Andrew Or] Revert addition of --deploy-mode
d5fe17b [Andrew Or] Update the YARN docs
(cherry picked from commit 2ffd1eafd28635dcecc0ac738d4a62c05d740925)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-1815. SparkContext should not be marked DeveloperApi

Author: Sandy Ryza <sandy@cloudera.com>

Closes #753 from sryza/sandy-spark-1815 and squashes the following commits:

957a8ac [Sandy Ryza] SPARK-1815. SparkContext should not be marked DeveloperApi
(cherry picked from commit 2792bd016af2a67848e6f403c4e1e05e9f3e3c2a)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Adding CHANGES.txt file and removing YARN support for now
[maven-release-plugin] prepare release v1.0.0-rc4
[maven-release-plugin] prepare for next development iteration