[SPARK-1395] Fix "local:" URI support in Yarn mode (again).

Recent changes ignored the fact that path may be defined with "local:"
URIs, which means they need to be explicitly added to the classpath
everywhere a remote process is started. This change fixes that by:

- Using the correct methods to add paths to the classpath
- Creating SparkConf settings for the Spark jar itself and for the
  user's jar
- Propagating those two settings to the remote processes where needed

This ensures that both in client and in cluster mode, the driver has
the necessary info to build the executor's classpath and have things
still work when they contain "local:" references.

On the cleanup front, I removed the hacky way that log4j configuration
was being propagated to handle the "local:" case. It's much more cleanly
(and generically) handled by using spark-submit arguments (--files to
upload a config file, or setting spark.executor.extraJavaOptions to pass
JVM arguments and use a local file).

export SPARK_JAVA_OPTS="-Dspark...=foo"   .  This needs to work for backwards compatibility but it doesn't with this code change because SparkConf throws an error saying you aren't supposed to do this.

I agree it is cleaner the way you have it but its going to require more investigation and a bunch of testing to make sure it works properly and I would rather have it split off into another jira/pr.  There are already alot of things in this pr.