apache/spark — pull request #1699 by liancheng (2014-07-31 19:17:04+00:00)
Title: [SPARK-2678][Core] Prevents `spark-submit` from shadowing application options

JIRA issue: [SPARK-2678](https://issues.apache.org/jira/browse/SPARK-2678)

Currently, `spark-submit` shadows user application options. Namely, options like `--help` are always captured by `spark-submit` and won't be passed to the user application. A negative impact of this is that, every time we add a new option to `spark-submit`, we may potentially break some existing user scripts if the new option name happens to shadow an existing option in the application.

This PR introduced an **incompatible** change to fix this issue: `--` is used as a separator between `spark-submit` options and user application options. All arguments after `--` are passed to the application as is. Thus,

```bash
./bin/spark-submit --class Foo user.jar arg1 -arg2 --arg3 x
# or
./bin/spark-submit user.jar --class Foo arg1 -arg2 --arg3 x
```

must be rewritten to

```bash
./bin/spark-submit --class Foo user.jar -- arg1 -arg2 --arg3 x
# or
./bin/spark-submit user.jar --class Foo -- arg1 -arg2 --arg3 x
```

@pwendell Please help review, thanks. Especially, maybe we need a vote here to decide whether fixing this issue worth an incompatible change.