If the memory allocated to a container reserves more than one cpu, then increase workerCores to reflect the higher count.
Unfortunately, this hits a hadoop bug : https://issues.apache.org/jira/browse/HADOOP-8406
Attempt(s) to workaround this hadoop bug with incorrect use of ServiceLoader ... did not work well consistently.
Disabled for now, re-enable as soon as a hadoop distro which has it fixed comes out.
When there is a usage error, print out the incorrect arg along with the usage message to aid debugging.
1) Add SPARK_LOG4J_CONF env variable to specify the log4j.properties to be pushed to cluster.

2) Specify appropriate Xmx for spark container in application master

3) Fix bug with incorrect use of JAVA_OPTS
Ensure that Xmx specified for workers is below the memory overhead.
Increase frame size to 100Mb using akka.remote.netty.message-frame-size
min, not max
Fix an NPE, overlooked it
Hand merge changes from spark trunk for spark.akka.frameSize
Merge pull request #383 from mridulm/yarn

Bugfixes to yarn branch