Merge branch 'branch-1.1' of github.com:apache/spark into csd-1.1
Merge branch 'branch-1.1' of github.com:apache/spark into csd-1.1
Merge branch 'csd-1.1' of github.com:markhamstra/spark into csd-1.1
Merge branch 'csd-1.1' of github.com:clearstorydata/spark into csd-1.1
[SPARK-4169] [Core] Accommodate non-English Locales in unit tests

For me the core tests failed because there are two locale dependent parts in the code.
Look at the Jira ticket for details.

Why is it necessary to check the exception message in isBindCollision in
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/Utils.scala#L1686
?

Author: Niklas Wilcke <1wilcke@informatik.uni-hamburg.de>

Closes #3036 from numbnut/core-test-fix and squashes the following commits:

1fb0d04 [Niklas Wilcke] Fixing locale dependend code and tests

(cherry picked from commit ed8bf1eac548577c4bbad7ce3f7f301a2f52ef17)
Signed-off-by: Andrew Or <andrew@databricks.com>
SPARK-2548 [STREAMING] JavaRecoverableWordCount is missing

Here's my attempt to re-port `RecoverableNetworkWordCount` to Java, following the example of its Scala and Java siblings. I fixed a few minor doc/formatting issues along the way I believe.

Author: Sean Owen <sowen@cloudera.com>

Closes #2564 from srowen/SPARK-2548 and squashes the following commits:

0d0bf29 [Sean Owen] Update checkpoint call as in https://github.com/apache/spark/pull/2735
35f23e3 [Sean Owen] Remove old comment about running in standalone mode
179b3c2 [Sean Owen] Re-port RecoverableNetworkWordCount to Java example, and touch up doc / formatting in related examples

(cherry picked from commit 3a02d416cd82a7a942fd6ff4a0e05ff070eb218a)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
Update RecoverableNetworkWordCount.scala

Trying this example, I missed the moment when the checkpoint was iniciated

Author: comcmipi <pitonak@fns.uniba.sk>

Closes #2735 from comcmipi/patch-1 and squashes the following commits:

b6d8001 [comcmipi] Update RecoverableNetworkWordCount.scala
96fe274 [comcmipi] Update RecoverableNetworkWordCount.scala

(cherry picked from commit 0340c56a921d4eb4bc9058e25e926721f8df594c)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
[SPARK-2548][HOTFIX][Streaming] Removed use of o.a.s.streaming.Durations in branch 1.1

Author: Tathagata Das <tathagata.das1565@gmail.com>

Closes #3188 from tdas/branch-1.1 and squashes the following commits:

f1996d3 [Tathagata Das] [SPARK-2548][HOTFIX] Removed use of o.a.s.streaming.Durations
[SPARK-4308][SQL] Follow up of #3175 for branch 1.1

PR #3175 is for master branch only and can't be backported to branch 1.1 directly because Hive 0.13.1 support.

Author: Cheng Lian <lian@databricks.com>

Closes #3176 from liancheng/fix-op-state-for-1.1 and squashes the following commits:

8791d87 [Cheng Lian] This is a follow up of #3175 for branch 1.1
[SPARK-3971][SQL] Backport #2843 to branch-1.1

This PR backports #2843 to branch-1.1. The key difference is that this one doesn't support Hive 0.13.1 and thus always returns `0.12.0` when `spark.sql.hive.version` is queried.

6 other commits on which #2843 depends were also backported, they are:

- #2887 for `SessionState` lifecycle control
- #2675, #2823 & #3060 for major test suite refactoring and bug fixes
- #2164, for Parquet test suites updates
- #2493, for reading `spark.sql.*` configurations

Author: Cheng Lian <lian@databricks.com>
Author: Cheng Lian <lian.cs.zju@gmail.com>
Author: Michael Armbrust <michael@databricks.com>

Closes #3113 from liancheng/get-info-for-1.1 and squashes the following commits:

d354161 [Cheng Lian] Provides Spark and Hive version in HiveThriftServer2 for branch-1.1
0c2a244 [Michael Armbrust] [SPARK-3646][SQL] Copy SQL configuration from SparkConf when a SQLContext is created.
3202a36 [Michael Armbrust] [SQL] Decrease partitions when testing
7f395b7 [Cheng Lian] [SQL] Fixes race condition in CliSuite
0dd28ec [Cheng Lian] [SQL] Fixes the race condition that may cause test failure
5928b39 [Cheng Lian] [SPARK-3809][SQL] Fixes test suites in hive-thriftserver
faeca62 [Cheng Lian] [SPARK-4037][SQL] Removes the SessionState instance created in HiveThriftServer2
[SPARK-3954][Streaming] Optimization to FileInputDStream

about convert files to RDDS there are 3 loops with files sequence in spark source.
loops files sequence:
1.files.map(...)
2.files.zip(fileRDDs)
3.files-size.foreach
It's will very time consuming when lots of files.So I do the following correction:
3 loops with files sequence => only one loop

Author: surq <surq@asiainfo.com>

Closes #2811 from surq/SPARK-3954 and squashes the following commits:

321bbe8 [surq]  updated the code style.The style from [for...yield]to [files.map(file=>{})]
88a2c20 [surq] Merge branch 'master' of https://github.com/apache/spark into SPARK-3954
178066f [surq] modify code's style. [Exceeds 100 columns]
626ef97 [surq] remove redundant import(ArrayBuffer)
739341f [surq] promote the speed of convert files to RDDS

(cherry picked from commit ce6ed2abd14de26b9ceaa415e9a42fbb1338f5fa)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
[SPARK-3495][SPARK-3496] Backporting block replication fixes made in master to branch 1.1

The original PR was #2366

This backport was non-trivial because Spark 1.1 uses ConnectionManager instead of NioBlockTransferService, which required slight modification to unit tests. Other than that the code is exactly same as in the original PR. Please refer to discussion in the original PR if you have any thoughts.

Author: Tathagata Das <tathagata.das1565@gmail.com>

Closes #3191 from tdas/replication-fix-branch-1.1-backport and squashes the following commits:

593214a [Tathagata Das] Merge remote-tracking branch 'apache-github/branch-1.1' into branch-1.1
2ed927f [Tathagata Das] Fixed error in unit test.
de4ff73 [Tathagata Das] [SPARK-3495] Block replication fails continuously when the replication target node is dead AND [SPARK-3496] Block replication by mistake chooses driver as target
Update versions for 1.1.1 release
[SQL] Backport backtick and smallint JDBC fixes to 1.1

Author: Michael Armbrust <michael@databricks.com>
Author: ravipesala <ravindra.pesala@huawei.com>
Author: scwf <wangfei1@huawei.com>

Closes #3199 from marmbrus/backport1.1 and squashes the following commits:

019a0dd [Michael Armbrust] Drop incorrectly ported test cases
4c9f3e6 [ravipesala] [SPARK-3708][SQL] Backticks aren't handled correctly is aliases
064750d [scwf] [SPARK-3704][SQL] Fix ColumnValue type for Short values in thrift server
f4e17cd [ravipesala] [SPARK-3834][SQL] Backticks not correctly handled in subquery aliases
[SPARK-4330][Doc] Link to proper URL for YARN overview

In running-on-yarn.md, a link to YARN overview is here.
But the URL is to YARN alpha's.
It should be stable's.

Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>

Closes #3196 from sarutak/SPARK-4330 and squashes the following commits:

30baa21 [Kousuke Saruta] Fixed running-on-yarn.md to point proper URL for YARN

(cherry picked from commit 3c07b8f08240bafcdff5d174989fb433f4bc80b6)
Signed-off-by: Matei Zaharia <matei@databricks.com>
[BRANCH-1.1][SPARK-2652] change the default spark.serializer in pyspark back to Kryo

This reverts #2916 . We shouldn't change the default settings in a minor release. JoshRosen davies

Author: Xiangrui Meng <meng@databricks.com>

Closes #3187 from mengxr/SPARK-2652-1.1 and squashes the following commits:

372166b [Xiangrui Meng] change the default spark.serializer in pyspark back to Kryo
[branch-1.1][SPARK-3990] add a note on ALS usage

Because we switched back to Kryo in #3187 , we need to leave a note about the workaround.

Author: Xiangrui Meng <meng@databricks.com>

Closes #3190 from mengxr/SPARK-3990-1.1 and squashes the following commits:

d4818f3 [Xiangrui Meng] fix python style
53725b0 [Xiangrui Meng] add a note about SPARK-3990
56ad70e [Xiangrui Meng] add a note about SPARK-3990
[SPARK-4295][External]Fix exception in SparkSinkSuite

Handle exception in SparkSinkSuite, please refer to [SPARK-4295]

Author: maji2014 <maji3@asiainfo.com>

Closes #3177 from maji2014/spark-4295 and squashes the following commits:

312620a [maji2014] change a new statement for spark-4295
24c3d21 [maji2014] add log4j.properties for SparkSinkSuite and spark-4295
c807bf6 [maji2014] Fix exception in SparkSinkSuite

(cherry picked from commit f8811a5695af2dfe156f07431288db7b8cd97159)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
Merge branch 'branch-1.1' of github.com:apache/spark into csd-1.1

Conflicts:
	docs/_config.yml
	yarn/alpha/pom.xml
Version to 1.1.1-candidate-csd-1-SNAPSHOT