apache/spark — pull request #1845 by andrewor14 (2014-08-08 02:09:48+00:00)
Title: [SPARK-2849 / 2914] Handle certain Spark configs in bash correctly

We currently rely on the Java properties file to parse `spark-defaults.conf` file. However, certain Spark configs need to be processed independently of the JVM. These fall into two general categories:

**1) spark.driver.***

In client deploy mode, the driver is launched from within `SparkSubmit`'s JVM. This means by the time we parse Spark configs from `spark-defaults.conf`, it is already too late to control certain properties of the driver's JVM. We currently ignore these configs in client mode altogether.
```
spark.driver.memory
spark.driver.extraJavaOptions
spark.driver.extraClassPath
spark.driver.extraLibraryPath
```
**2) spark.*.extraJavaOptions**

These configs involve a list of java options squeezed into a string. Currently, if any of the options include escaped double quotes or backslashes, the splitting of these options will likely be incorrect and the Java command to start the executors may be corrupted. The relevant configs here include the following.

```
spark.driver.extraJavaOptions
spark.executor.extraJavaOptions
```

For both categories, we need to preemptively parse the Spark configs in bash. There is a lot of trickery involved in performing heavy-duty parsing in bash, and I have moved much of the complexity to a new file `bin/util.sh`. I have tested this locally with escaped double quotes, backslashes, whitespace, and a combination of the above, and the behavior is as expected.

The changes build directly on top of my old PR at #1770.