Add YARN/Stable compiled classes to the CLASSPATH.

The change adds the `./yarn/stable/target/<scala-version>/classes` to
the _Classpath_ when a _dependencies_ assembly is available at the
assembly directory.

Why is this change necessary?
Ease the development features and bug-fixes for Spark-YARN.

[ticket: X] : NA

Author      : bernardo.gomezpalacio@gmail.com
Reviewer    : ?
Testing     : ?
The `spark-class` shell now ignores non jar files in the assembly directory.

Why is this change necessary?

While developing in Spark I found myself rebuilding either the
dependencies assembly or the full spark assembly. I kept running into
the case of having both the dep-assembly and full-assembly in the same
directory and getting an error when I called either `spark-shell` or
`spark-submit`.

Quick fix: move either of them as a .bkp file depending on
the development work flow you are executing at the moment and enabling
the `spark-class` to ignore non-jar files. An other option could be to
move the "offending" jar to a different directory but in my opinion
keeping them in there is a bit tidier.

e.g.

```
ll ./assembly/target/scala-2.10
spark-assembly-1.0.0-SNAPSHOT-hadoop2.3.0-deps.jar
spark-assembly-1.0.0-SNAPSHOT-hadoop2.3.0.jar.bkp
```

[ticket: X] : ?
The `spark-shell` option `--log-conf` also enables the SPARK_PRINT_LAUNCH_COMMAND .

Why is this change necessary?
Most likely when enabling the `--log-conf` through the `spark-shell` you
are also interested on the full invocation of the java command including the
_classpath_ and extended options. e.g.

```
INFO: Base Directory set to /Users/bernardo/work/github/berngp/spark
INFO: Spark Master is yarn-client
INFO: Spark REPL options   -Dspark.logConf=true
Spark Command: /Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/Home/bin/java -cp :/Users/bernardo/work/github/berngp/spark/conf:/Users/bernardo/work/github/berngp/spark/core/target/scala-2.10/classes:/Users/bernardo/work/github/berngp/spark/repl/target/scala-2.10/classes:/Users/bernardo/work/github/berngp/spark/mllib/target/scala-2.10/classes:/Users/bernardo/work/github/berngp/spark/bagel/target/scala-2.10/classes:/Users/bernardo/work/github/berngp/spark/graphx/target/scala-2.10/classes:/Users/bernardo/work/github/berngp/spark/streaming/target/scala-2.10/classes:/Users/bernardo/work/github/berngp/spark/tools/target/scala-2.10/classes:/Users/bernardo/work/github/berngp/spark/sql/catalyst/target/scala-2.10/classes:/Users/bernardo/work/github/berngp/spark/sql/core/target/scala-2.10/classes:/Users/bernardo/work/github/berngp/spark/sql/hive/target/scala-2.10/classes:/Users/bernardo/work/github/berngp/spark/yarn/stable/target/scala-2.10/classes:/Users/bernardo/work/github/berngp/spark/assembly/target/scala-2.10/spark-assembly-1.0.0-SNAPSHOT-hadoop2.3.0-deps.jar:/usr/local/Cellar/hadoop/2.2.0/libexec/etc/hadoop -XX:ErrorFile=/tmp/spark-shell-hs_err_pid.log -XX:HeapDumpPath=/tmp/spark-shell-java_pid.hprof -XX:-HeapDumpOnOutOfMemoryError -XX:-PrintGC -XX:-PrintGCDetails -XX:-PrintGCTimeStamps -XX:-PrintTenuringDistribution -XX:-PrintAdaptiveSizePolicy -XX:GCLogFileSize=1024K -XX:-UseGCLogFileRotation -Xloggc:/tmp/spark-shell-gc.log -XX:+UseConcMarkSweepGC -Dspark.cleaner.ttl=10000 -Dspark.driver.host=33.33.33.1 -Dspark.logConf=true -Djava.library.path= -Xms400M -Xmx400M org.apache.spark.repl.Main
```

[ticket: X] : ?
Root is now Spark and qualify the assembly if it was built with YARN.

Why is this change necessary?
Renamed the SBT "root" project to "spark" to enhance readability.

Currently the assembly is qualified with the Hadoop Version but not if
YARN has been enabled or not. This change qualifies the assembly such
that it is easy to identify if YARN was enabled.

e.g

```
./make-distribution.sh --hadoop 2.3.0 --with-yarn

ls -l ./assembly/target/scala-2.10
    spark-assembly-1.0.0-SNAPSHOT-hadoop2.3.0-yarn.jar
```

vs

```
./make-distribution.sh --hadoop 2.3.0

ls -l ./assembly/target/scala-2.10
    spark-assembly-1.0.0-SNAPSHOT-hadoop2.3.0.jar
```

[ticket: X] : ?
Upgrade the Maven Build to YARN 2.3.0.

Upgraded to YARN 2.3.0, removed unnecessary `relativePath` values and
removed incorrect version for the "org.apache.hadoop:hadoop-client"
dependency at yarn/pom.xml.
merge https://github.com/berngp/spark/commits/feature/small-shell-changes
Improved build configuration Ⅱ
review commit
review commit
Merge branch 'master' of https://github.com/apache/spark into improved_build
improve travis tests coverage
missing ","
add the dependency of commons-lang
SPARK-1693: Most of the tests throw a java.lang.SecurityException when spark built for hadoop 2.3.0 , 2.4.0
revert .travis.yml
Add the missing yarn dependencies
Fix SPARK-1693: Dependent on multiple versions of servlet-api jars lead to throw an SecurityException when Spark built for hadoop 2.3.0 , 2.4.0
review commit
review commit
review commit