[SPARK-1900 / 1918] PySpark on YARN is broken

If I run the following on a YARN cluster
```
bin/spark-submit sheep.py --master yarn-client
```
it fails because of a mismatch in paths: `spark-submit` thinks that `sheep.py` resides on HDFS, and balks when it can't find the file there. A natural workaround is to add the `file:` prefix to the file:
```
bin/spark-submit file:/path/to/sheep.py --master yarn-client
```
However, this also fails. This time it is because python does not understand URI schemes.

This PR fixes this by automatically resolving all paths passed as command line argument to `spark-submit` properly. This has the added benefit of keeping file and jar paths consistent across different cluster modes. For python, we strip the URI scheme before we actually try to run it.

Much of the code is originally written by @mengxr. Tested on YARN cluster. More tests pending.

Author: Andrew Or <andrewor14@gmail.com>

Closes #853 from andrewor14/submit-paths and squashes the following commits:

0bb097a [Andrew Or] Format path correctly before adding it to PYTHONPATH
323b45c [Andrew Or] Include --py-files on PYTHONPATH for pyspark shell
3c36587 [Andrew Or] Improve error messages (minor)
854aa6a [Andrew Or] Guard against NPE if user gives pathological paths
6638a6b [Andrew Or] Fix spark-shell jar paths after #849 went in
3bb0359 [Andrew Or] Update more comments (minor)
2a1f8a0 [Andrew Or] Update comments (minor)
6af2c77 [Andrew Or] Merge branch 'master' of github.com:apache/spark into submit-paths
a68c4d1 [Andrew Or] Handle Windows python file path correctly
427a250 [Andrew Or] Resolve paths properly for Windows
a591a4a [Andrew Or] Update tests for resolving URIs
6c8621c [Andrew Or] Move resolveURIs to Utils
db8255e [Andrew Or] Merge branch 'master' of github.com:apache/spark into submit-paths
f542dce [Andrew Or] Fix outdated tests
691c4ce [Andrew Or] Ignore special primary resource names
5342ac7 [Andrew Or] Add missing space in error message
02f77f3 [Andrew Or] Resolve command line arguments to spark-submit properly

(cherry picked from commit 5081a0a9d47ca31900ea4de570de2cbb0e063105)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
SPARK-1911: Emphasize that Spark jars should be built with Java 6.

This commit requires the user to manually say "yes" when buiding Spark
without Java 6. The prompt can be bypassed with a flag (e.g. if the user
is scripting around make-distribution).

Author: Patrick Wendell <pwendell@gmail.com>

Closes #859 from pwendell/java6 and squashes the following commits:

4921133 [Patrick Wendell] Adding Pyspark Notice
fee8c9e [Patrick Wendell] SPARK-1911: Emphasize that Spark jars should be built with Java 6.

(cherry picked from commit 75a03277704f8618a0f1c41aecfb1ebd24a8ac1a)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
Updated CHANGES.txt
Revert "[maven-release-plugin] prepare for next development iteration"

This reverts commit 67dd53d2556f03ce292e6889128cf441f1aa48f8.
Revert "[maven-release-plugin] prepare release v1.0.0-rc10"

This reverts commit d807023479ce10aec28ef3c1ab646ddefc2e663c.
[SPARK-1886] check executor id existence when executor exit

Author: Zhen Peng <zhenpeng01@baidu.com>

Closes #827 from zhpengg/bugfix-executor-id-not-found and squashes the following commits:

cd8bb65 [Zhen Peng] bugfix: check executor id existence when executor exit

(cherry picked from commit 4e4831b8facc186cda6ef31040ccdeab48acbbb7)
Signed-off-by: Aaron Davidson <aaron@databricks.com>
spark-submit: add exec at the end of the script

Add an 'exec' at the end of the spark-submit script, to avoid keeping a
bash process hanging around while it runs.  This makes ps look a little
bit nicer.

Author: Colin Patrick Mccabe <cmccabe@cloudera.com>

Closes #858 from cmccabe/SPARK-1907 and squashes the following commits:

7023b64 [Colin Patrick Mccabe] spark-submit: add exec at the end of the script

(cherry picked from commit 6e9fb6320bec3371bc9c010ccbc1b915f500486b)
Signed-off-by: Reynold Xin <rxin@apache.org>
[SPARK-1822] SchemaRDD.count() should use query optimizer

Author: Kan Zhang <kzhang@apache.org>

Closes #841 from kanzhang/SPARK-1822 and squashes the following commits:

2f8072a [Kan Zhang] [SPARK-1822] Minor style update
cf4baa4 [Kan Zhang] [SPARK-1822] Adding Scaladoc
e67c910 [Kan Zhang] [SPARK-1822] SchemaRDD.count() should use optimizer

(cherry picked from commit 6052db9dc10c996215658485e805200e4f0cf549)
Signed-off-by: Reynold Xin <rxin@apache.org>
Added PEP8 style configuration file.

This sets the max line length to 100 as a PEP8 exception.

Author: Reynold Xin <rxin@apache.org>

Closes #872 from rxin/pep8 and squashes the following commits:

2f26029 [Reynold Xin] Added PEP8 style configuration file.

(cherry picked from commit 5c7faecd75ea59454ad3209390ac078e6cf6e4a6)
Signed-off-by: Reynold Xin <rxin@apache.org>