apache/spark — pull request #1107 by ash211 (2014-06-17 18:00:19+00:00)
Title: [WIP] SPARK-2157 Ability to write tight firewall rules for Spark

https://issues.apache.org/jira/browse/SPARK-2157

This pull request adds the ability to specify every port opened by Spark that I could find to configure. Specifically, it adds the following four configuration options:

- `spark.fileserver.port`
- `spark.broadcast.port`
- `spark.replClassServer.port`
- `spark.blockManager.port`

Of those options, most will attempt to fall back to the n+1 incremental port if the port is currently in use. This incremental fallback logic is done for :

- `spark.broadcast.port`
- `spark.blockManager.port`
- `spark.replClassServer.port`

But not for `spark.fileserver.port`

Things still thinking about for this PR:

- I'm still observing instances of `CoarseGrainedExecutorBackend` listening on ephemeral ports
- In earlier documentation `spark.driver.port` is described as being the way to configure :7077 on Spark Master, and I'm not sure this is the case. If so the same config item is used for multiple things, as it's also the port on which the applications starts listening on Akka.

Many thanks to @epahomov whose work this is based off of.