[SPARK-2075][Core] backport for branch-1.2

backport #3740 for branch-1.2

Author: zsxwing <zsxwing@gmail.com>

Closes #3758 from zsxwing/SPARK-2075-branch-1.2 and squashes the following commits:

b57d440 [zsxwing] SPARK-2075 backport for branch-1.2
[SPARK-4915][YARN] Fix classname to be specified for external shuffle service.

Author: Tsuyoshi Ozawa <ozawa.tsuyoshi@lab.ntt.co.jp>

Closes #3757 from oza/SPARK-4915 and squashes the following commits:

3b0d6d6 [Tsuyoshi Ozawa] Fix classname to be specified for external shuffle service.

(cherry picked from commit 96606f69b7cd88edea5d4b6427f598a91ccb4c33)
Signed-off-by: Andrew Or <andrew@databricks.com>
[SPARK-4883][Shuffle] Add a name to the directoryCleaner thread

Author: zsxwing <zsxwing@gmail.com>

Closes #3734 from zsxwing/SPARK-4883 and squashes the following commits:

e6f2b61 [zsxwing] Fix the name
cc74727 [zsxwing] Add a name to the directoryCleaner thread

(cherry picked from commit 8773705fd4ab167bb06bee7f274b337560e3c293)
Signed-off-by: Andrew Or <andrew@databricks.com>
[Minor] Improve some code in BroadcastTest for short

Using
    val arr1 = (0 until num).toArray
instead of
    val arr1 = new Array[Int](num)
    for (i <- 0 until arr1.length) {
      arr1(i) = i
    }
for short.

Author: carlmartin <carlmartinmax@gmail.com>

Closes #3750 from SaintBacchus/BroadcastTest and squashes the following commits:

43adb70 [carlmartin] Improve some code in BroadcastTest for short
[SPARK-4864] Add documentation to Netty-based configs

Author: Aaron Davidson <aaron@databricks.com>

Closes #3713 from aarondav/netty-configs and squashes the following commits:

8a8b373 [Aaron Davidson] Address Patrick's comments
3b1f84e [Aaron Davidson] [SPARK-4864] Add documentation to Netty-based configs

(cherry picked from commit fbca6b6ce293b1997b40abeb9ab77b8a969a5fc9)
Signed-off-by: Patrick Wendell <pwendell@gmail.com>
[SPARK-4920][UI]:current spark version in UI is not striking.

It is not convenient to see the Spark version. We can keep the same style with Spark website.

![spark_version](https://cloud.githubusercontent.com/assets/7402327/5527025/1c8c721c-8a35-11e4-8d6a-2734f3c6bdf8.jpg)

Author: genmao.ygm <genmao.ygm@alibaba-inc.com>

Closes #3763 from uncleGen/master-clean-141222 and squashes the following commits:

0dcb9a9 [genmao.ygm] [SPARK-4920][UI]:current spark version in UI is not striking.

(cherry picked from commit de9d7d2b5b6d80963505571700e83779fd98f850)
Signed-off-by: Andrew Or <andrew@databricks.com>
[SPARK-4818][Core] Add 'iterator' to reduce memory consumed by join

In Scala, `map` and `flatMap` of `Iterable` will copy the contents of `Iterable` to a new `Seq`. Such as,
```Scala
  val iterable = Seq(1, 2, 3).map(v => {
    println(v)
    v
  })
  println("Iterable map done")

  val iterator = Seq(1, 2, 3).iterator.map(v => {
    println(v)
    v
  })
  println("Iterator map done")
```
outputed
```
1
2
3
Iterable map done
Iterator map done
```
So we should use 'iterator' to reduce memory consumed by join.

Found by Johannes Simon in http://mail-archives.apache.org/mod_mbox/spark-user/201412.mbox/%3C5BE70814-9D03-4F61-AE2C-0D63F2DE4446%40mail.de%3E

Author: zsxwing <zsxwing@gmail.com>

Closes #3671 from zsxwing/SPARK-4824 and squashes the following commits:

48ee7b9 [zsxwing] Remove the explicit types
95d59d6 [zsxwing] Add 'iterator' to reduce memory consumed by join

(cherry picked from commit c233ab3d8d75a33495298964fe73dbf7dd8fe305)
Signed-off-by: Josh Rosen <joshrosen@databricks.com>