[SPARK-959] Updated SBT from 0.13.1 to 0.13.2

JIRA issue: [SPARK-959](https://spark-project.atlassian.net/browse/SPARK-959)

SBT 0.13.2 has been officially released. This version updated Ivy 2.0 to Ivy 2.3, which fixes [IVY-899](https://issues.apache.org/jira/browse/IVY-899). This PR also removed previous workaround.

Author: Cheng Lian <lian.cs.zju@gmail.com>

Closes #426 from liancheng/updateSbt and squashes the following commits:

95e3dc8 [Cheng Lian] Updated SBT from 0.13.1 to 0.13.2 to fix SPARK-959
(cherry picked from commit 6a10d801626f1513b1b349b54ba0e2e6bf55c7e2)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Make "spark logo" link refer to "/".

This is not an issue with the driver UI, but when you fire
up the history server, there's currently no way to go back to
the app listing page without editing the browser's location
field (since the logo's link points to the root of the
application's own UI - i.e. the "stages" tab).

The change just points the logo link to "/", which is the app
listing for the history server, and the stages tab for the
driver's UI.

Tested with both history server and live driver.

Author: Marcelo Vanzin <vanzin@cloudera.com>

Closes #408 from vanzin/web-ui-root and squashes the following commits:

1b60cb6 [Marcelo Vanzin] Make "spark logo" link refer to "/".
(cherry picked from commit c0273d806ea9b83dd8585039f2a18c2cc795dad2)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Loads test tables when running "sbt hive/console" without HIVE_DEV_HOME

When running Hive tests, the working directory is `$SPARK_HOME/sql/hive`, while when running `sbt hive/console`, it becomes `$SPARK_HOME`, and test tables are not loaded if `HIVE_DEV_HOME` is not defined.

Author: Cheng Lian <lian.cs.zju@gmail.com>

Closes #417 from liancheng/loadTestTables and squashes the following commits:

7cea8d6 [Cheng Lian] Loads test tables when running "sbt hive/console" without HIVE_DEV_HOME
(cherry picked from commit fec462c15321fa138d9654d64cc08468b6ab5ad1)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
update spark.default.parallelism

actually, the value 8 is only valid in mesos fine-grained mode :
<code>
  override def defaultParallelism() = sc.conf.getInt("spark.default.parallelism", 8)
</code>

while in coarse-grained model including mesos coares-grained, the value of the property depending on core numbers!
<code>
override def defaultParallelism(): Int = {
   conf.getInt("spark.default.parallelism", math.max(totalCoreCount.get(), 2))
  }
</code>

Author: Chen Chao <crazyjvm@gmail.com>

Closes #389 from CrazyJvm/patch-2 and squashes the following commits:

84a7fe4 [Chen Chao] miss </li> at the end of every single line
04a9796 [Chen Chao] change format
ee0fae0 [Chen Chao] update spark.default.parallelism
(cherry picked from commit 9edd88782e0268439c5ab57400d6a7ab432fc269)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-1310: Start adding k-fold cross validation to MLLib [adds kFold to MLUtils & fixes bug in BernoulliSampler]

Author: Holden Karau <holden@pigscanfly.ca>

Closes #18 from holdenk/addkfoldcrossvalidation and squashes the following commits:

208db9b [Holden Karau] Fix a bad space
e84f2fc [Holden Karau] Fix the test, we should be looking at the second element instead
6ddbf05 [Holden Karau] swap training and validation order
7157ae9 [Holden Karau] CR feedback
90896c7 [Holden Karau] New line
150889c [Holden Karau] Fix up error messages in the MLUtilsSuite
2cb90b3 [Holden Karau] Fix the names in kFold
c702a96 [Holden Karau] Fix imports in MLUtils
e187e35 [Holden Karau] Move { up to same line as whenExecuting(random) in RandomSamplerSuite.scala
c5b723f [Holden Karau] clean up
7ebe4d5 [Holden Karau] CR feedback, remove unecessary learners (came back during merge mistake) and insert an empty line
bb5fa56 [Holden Karau] extra line sadness
163c5b1 [Holden Karau] code review feedback 1.to -> 1 to and folds -> numFolds
5a33f1d [Holden Karau] Code review follow up.
e8741a7 [Holden Karau] CR feedback
b78804e [Holden Karau] Remove cross validation [TODO in another pull request]
91eae64 [Holden Karau] Consolidate things in mlutils
264502a [Holden Karau] Add a test for the bug that was found with BernoulliSampler not copying the complement param
dd0b737 [Holden Karau] Wrap long lines (oops)
c0b7fa4 [Holden Karau] Switch FoldedRDD to use BernoulliSampler and PartitionwiseSampledRDD
08f8e4d [Holden Karau] Fix BernoulliSampler to respect complement
a751ec6 [Holden Karau] Add k-fold cross validation to MLLib
(cherry picked from commit c3527a333a0877f4b49614f3fd1f041b01749651)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-1497. Fix scalastyle warnings in YARN, Hive code

(I wasn't sure how to automatically set `SPARK_YARN=true` and `SPARK_HIVE=true` when running scalastyle, but these are the errors that turn up.)

Author: Sean Owen <sowen@cloudera.com>

Closes #413 from srowen/SPARK-1497 and squashes the following commits:

f0c9318 [Sean Owen] Fix more scalastyle warnings in yarn
80bf4c3 [Sean Owen] Add YARN alpha / YARN profile to scalastyle check
026319c [Sean Owen] Fix scalastyle warnings in YARN, Hive code
(cherry picked from commit 77f836799639ea939a1773cef2f4828b381f5ca2)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Minor addition to SPARK-1497
SPARK-1469: Scheduler mode should accept lower-case definitions and have...

... nicer error messages

There are  two improvements to Scheduler Mode:
1. Made the built in ones case insensitive (fair/FAIR, fifo/FIFO).
2. If an invalid mode is given we should print a better error message.

Author: Sandeep <sandeep@techaddict.me>

Closes #388 from techaddict/1469 and squashes the following commits:

a31bbd5 [Sandeep] SPARK-1469: Scheduler mode should accept lower-case definitions and have nicer error messages There are  two improvements to Scheduler Mode: 1. Made the built in ones case insensitive (fair/FAIR, fifo/FIFO). 2. If an invalid mode is given we should print a better error message.
(cherry picked from commit e269c24db7882ba05b26eff8fc6e1869103517f8)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-1465: Spark compilation is broken with the latest hadoop-2.4.0 release

YARN-1824 changes the APIs (addToEnvironment, setEnvFromInputString) in Apps, which causes the spark build to break if built against a version 2.4.0. To fix this, create the spark own function to do that functionality which will not break compiling against 2.3 and other 2.x versions.

Author: xuan <xuan@MacBook-Pro.local>
Author: xuan <xuan@macbook-pro.home>

Closes #396 from xgong/master and squashes the following commits:

42b5984 [xuan] Remove two extra imports
bc0926f [xuan] Remove usage of org.apache.hadoop.util.Shell
be89fa7 [xuan] fix Spark compilation is broken with the latest hadoop-2.4.0 release

(cherry picked from commit 725925cf2120e998651f7d1406fdb34fc2405b9f)
Signed-off-by: Thomas Graves <tgraves@apache.org>
[SPARK-1511] use Files.move instead of renameTo in TestUtils.scala

JIRA issue:[SPARK-1511](https://issues.apache.org/jira/browse/SPARK-1511)

TestUtils.createCompiledClass method use renameTo() to move files which fails when the src and dest files are in different disks or partitions. This pr uses Files.move() instead. The move method will try to use renameTo() and then fall back to copy() and delete(). I think this should handle this issue.

I didn't found a test suite for this file, so I add file existence detection after file moving.

Author: Ye Xianjin <advancedxy@gmail.com>

Closes #427 from advancedxy/SPARK-1511 and squashes the following commits:

a2b97c7 [Ye Xianjin] Based on @srowen's comment, assert file existence.
6f95550 [Ye Xianjin] use Files.move instead of renameTo to handle the src and dest files are in different disks or partitions.
(cherry picked from commit 10b1c59dcc9ca2c1dafa02cb3ea298f3b33fc914)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>