Merge branch 'branch-1.1' of github.com:apache/spark into csd-1.1
Merge branch 'csd-1.1' of github.com:markhamstra/spark into csd-1.1
[SPARK-3518] Remove wasted statement in JsonProtocol

Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>

Closes #2380 from sarutak/SPARK-3518 and squashes the following commits:

8a1464e [Kousuke Saruta] Replaced a variable with simple field reference
c660fbc [Kousuke Saruta] Removed useless statement in JsonProtocol.scala

(cherry picked from commit e59fac1f97c3fbeeb6defd12625a49763a353156)
Signed-off-by: Patrick Wendell <pwendell@gmail.com>
[SQL][DOCS] Improve section on thrift-server

Taken from liancheng's updates. Merged conflicts with #2316.

Author: Michael Armbrust <michael@databricks.com>

Closes #2384 from marmbrus/sqlDocUpdate and squashes the following commits:

2db6319 [Michael Armbrust] @liancheng's updates

(cherry picked from commit 84073eb1172dc959936149265378f6e24d303685)
Signed-off-by: Michael Armbrust <michael@databricks.com>
Merge branch 'csd-1.1' of github.com:clearstorydata/spark into csd-1.1
Bumped thriftserver version
More messaging from SQL NotImplementedError
[SPARK-3555] Fix UISuite race condition

The test "jetty selects different port under contention" is flaky.

If another process binds to 4040 before the test starts, then the first server we start there will fail, and the subsequent servers we start thereafter may successfully bind to 4040 if it was released between the servers starting. Instead, we should just let Java find a random free port for us and hold onto it for the duration of the test.

Author: Andrew Or <andrewor14@gmail.com>

Closes #2418 from andrewor14/fix-port-contention and squashes the following commits:

0cd4974 [Andrew Or] Stop them servers
a7071fe [Andrew Or] Pick random port instead of 4040

(cherry picked from commit 0a7091e689a4c8b1e7b61e9f0873e6557f40d952)
Signed-off-by: Andrew Or <andrewor14@gmail.com>
[SPARK-3490] Disable SparkUI for tests (backport into 1.1)

Original PR: #2363

Author: Andrew Or <andrewor14@gmail.com>

Closes #2415 from andrewor14/disable-ui-for-tests-1.1 and squashes the following commits:

8d9df5a [Andrew Or] Oops, missed one.
509507d [Andrew Or] Backport #2363 (SPARK-3490) into branch-1.1
[SQL][DOCS] Improve table caching section

Author: Michael Armbrust <michael@databricks.com>

Closes #2434 from marmbrus/patch-1 and squashes the following commits:

67215be [Michael Armbrust] [SQL][DOCS] Improve table caching section

(cherry picked from commit cbf983bb4a550ff26756ed7308fb03db42cffcff)
Signed-off-by: Michael Armbrust <michael@databricks.com>
Docs: move HA subsections to a deeper indentation level

Makes the table of contents read better

Author: Andrew Ash <andrew@andrewash.com>

Closes #2402 from ash211/docs/better-indentation and squashes the following commits:

ea0e130 [Andrew Ash] Move HA subsections to a deeper indentation level

(cherry picked from commit b3830b28f8a70224d87c89d8491c514c4c191d23)
Signed-off-by: Andrew Or <andrewor14@gmail.com>
[SPARK-3564][WebUI] Display App ID on HistoryPage

Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>

Closes #2424 from sarutak/display-appid-on-webui and squashes the following commits:

417fe90 [Kousuke Saruta] Added "App ID column" to HistoryPage

(cherry picked from commit 6688a266f2cb84c2d43b8e4d27f710718c4cc4a0)
Signed-off-by: Andrew Or <andrewor14@gmail.com>
[SPARK-3565]Fix configuration item not consistent with document

https://issues.apache.org/jira/browse/SPARK-3565

"spark.ports.maxRetries" should be "spark.port.maxRetries". Make the configuration keys in document and code consistent.

Author: WangTaoTheTonic <barneystinson@aliyun.com>

Closes #2427 from WangTaoTheTonic/fixPortRetries and squashes the following commits:

c178813 [WangTaoTheTonic] Use blank lines trigger Jenkins
646f3fe [WangTaoTheTonic] also in SparkBuild.scala
3700dba [WangTaoTheTonic] Fix configuration item not consistent with document

(cherry picked from commit 3f169bfe3c322bf4344e13276dbbe34279b59ad0)
Signed-off-by: Patrick Wendell <pwendell@gmail.com>
[SPARK-3589][Minor]remove redundant code

https://issues.apache.org/jira/browse/SPARK-3589

"export CLASSPATH" in spark-class is redundant since same variable is exported before.
We could reuse defined value "isYarnCluster" in SparkSubmit.scala.

Author: WangTaoTheTonic <barneystinson@aliyun.com>

Closes #2445 from WangTaoTheTonic/removeRedundant and squashes the following commits:

6fb6872 [WangTaoTheTonic] remove redundant code

(cherry picked from commit 471e6a3a47bd4b94878798f6f6fc93e2e672efff)
Signed-off-by: Patrick Wendell <pwendell@gmail.com>
[SPARK-3560] Fixed setting spark.jars system property in yarn-cluster mode

Author: Victsm <victor.nju@gmail.com>
Author: Min Shen <mshen@linkedin.com>

Closes #2449 from Victsm/SPARK-3560 and squashes the following commits:

918405a [Victsm] Removed the additional space
4502a2a [Min Shen] [SPARK-3560] Fixed setting spark.jars system property in yarn-cluster mode.
[Minor Hot Fix] Move a line in SparkSubmit to the right place

This was introduced in #2449

Author: Andrew Or <andrewor14@gmail.com>

Closes #2452 from andrewor14/standalone-hot-fix and squashes the following commits:

d5190ca [Andrew Or] Put that line in the right place

(cherry picked from commit 9306297d1d888d0430f79b2133ee7377871a3a18)
Signed-off-by: Andrew Or <andrewor14@gmail.com>
[SPARK-3518] Remove wasted statement in JsonProtocol

Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>

Closes #2380 from sarutak/SPARK-3518 and squashes the following commits:

8a1464e [Kousuke Saruta] Replaced a variable with simple field reference
c660fbc [Kousuke Saruta] Removed useless statement in JsonProtocol.scala

(cherry picked from commit e59fac1f97c3fbeeb6defd12625a49763a353156)
Signed-off-by: Patrick Wendell <pwendell@gmail.com>
[SQL][DOCS] Improve section on thrift-server

Taken from liancheng's updates. Merged conflicts with #2316.

Author: Michael Armbrust <michael@databricks.com>

Closes #2384 from marmbrus/sqlDocUpdate and squashes the following commits:

2db6319 [Michael Armbrust] @liancheng's updates

(cherry picked from commit 84073eb1172dc959936149265378f6e24d303685)
Signed-off-by: Michael Armbrust <michael@databricks.com>
[SPARK-3555] Fix UISuite race condition

The test "jetty selects different port under contention" is flaky.

If another process binds to 4040 before the test starts, then the first server we start there will fail, and the subsequent servers we start thereafter may successfully bind to 4040 if it was released between the servers starting. Instead, we should just let Java find a random free port for us and hold onto it for the duration of the test.

Author: Andrew Or <andrewor14@gmail.com>

Closes #2418 from andrewor14/fix-port-contention and squashes the following commits:

0cd4974 [Andrew Or] Stop them servers
a7071fe [Andrew Or] Pick random port instead of 4040

(cherry picked from commit 0a7091e689a4c8b1e7b61e9f0873e6557f40d952)
Signed-off-by: Andrew Or <andrewor14@gmail.com>
[SPARK-3490] Disable SparkUI for tests (backport into 1.1)

Original PR: #2363

Author: Andrew Or <andrewor14@gmail.com>

Closes #2415 from andrewor14/disable-ui-for-tests-1.1 and squashes the following commits:

8d9df5a [Andrew Or] Oops, missed one.
509507d [Andrew Or] Backport #2363 (SPARK-3490) into branch-1.1