apache/spark — pull request #85 by CodingCat (2014-03-06 00:30:06+00:00)
Title: SPARK-1192: The document for most of the parameters used in core component

I grep the code in core component, I found that around 30 parameters in the implementation is actually used but undocumented. By reading the source code, I found that some of them are actually very useful for the user.

I suggest to make a complete document on the parameters.

Also some parameters are having confusing names

1. spark.shuffle.copier.threads - this parameters is to control how many threads you will use when you start a Netty-based shuffle service....but from the name, we cannot get this information

2. spark.shuffle.sender.port - the similar problem with the above one, when you use Netty-based shuffle receiver, you will have to setup a Netty-based sender...this parameter is to setup the port used by the Netty sender, but the name cannot convey this information.

---

To facilitate the review, I mostly make a commit for each parameter, but some highly related parameters are combined together. The format of the commit title is like

" %s - %s L%d" % parameter name, File/Class name, line-number

---

spark.deploy.retainedApplications - Master.scala L49

spark.dead.worker.persistence - Master.scala L50