Workaround in Spark for ConcurrentModification issue (JIRA Hadoop-10456, Spark-1097)

This fix has gone into Hadoop 2.4.1. For developers using <  2.4.1, it would be good to have a workaround in Spark as well.

Fix has been tested for performance as well, no regressions found.

Author: nravi <nravi@c1704.halxg.cloudera.com>

Closes #1000 from nishkamravi2/master and squashes the following commits:

eb663ca [nravi] Merge branch 'master' of https://github.com/apache/spark
df2aeb1 [nravi] Improved fix for ConcurrentModificationIssue (Spark-1097, Hadoop-10456)
6b840f0 [nravi] Undo the fix for SPARK-1758 (the problem is fixed)
5108700 [nravi] Fix in Spark for the Concurrent thread modification issue (SPARK-1097, HADOOP-10456)
681b36f [nravi] Fix for SPARK-1758: failing test org.apache.spark.JavaAPISuite.wholeTextFiles

(cherry picked from commit 70c8116c0aecba293234edc44a7f8e58e5008649)
Signed-off-by: Reynold Xin <rxin@apache.org>
[SPARK-1964][SQL] Add timestamp to HiveMetastoreTypes.toMetastoreType

Author: Michael Armbrust <michael@databricks.com>

Closes #1061 from marmbrus/timestamp and squashes the following commits:

79c3903 [Michael Armbrust] Add timestamp to HiveMetastoreTypes.toMetastoreType()

(cherry picked from commit 1c2fd015b05b65abc83c4874ada825deac578af8)
Signed-off-by: Michael Armbrust <michael@databricks.com>
Small correction in Streaming Programming Guide doc

Corrected description of `repartition` function under 'Level of Parallelism in Data Receiving'.

Author: akkomar <ak.komar@gmail.com>

Closes #1079 from akkomar/streaming-guide-doc and squashes the following commits:

32dfc62 [akkomar] Corrected description of `repartition` function under 'Level of Parallelism in Data Receiving'.

(cherry picked from commit edb1f0e3164b99b483b6878efe0b6ea5ae9f97ed)
Signed-off-by: Reynold Xin <rxin@apache.org>
[SQL] Update SparkSQL and ScalaTest in branch-1.0 to match master.

#511 and #863 got left out of branch-1.0 since we were really close to the release.  Now that they have been tested a little I see no reason to leave them out.

Author: Michael Armbrust <michael@databricks.com>
Author: witgo <witgo@qq.com>

Closes #1078 from marmbrus/branch-1.0 and squashes the following commits:

22be674 [witgo]  [SPARK-1841]: update scalatest to version 2.1.5
fc8fc79 [Michael Armbrust] Include #1071 as well.
c5d0adf [Michael Armbrust] Update SparkSQL in branch-1.0 to match master.
[Spark-2137][SQL] Timestamp UDFs broken

https://issues.apache.org/jira/browse/SPARK-2137

Author: Yin Huai <huai@cse.ohio-state.edu>

Closes #1081 from yhuai/SPARK-2137 and squashes the following commits:

c04f910 [Yin Huai] Merge remote-tracking branch 'upstream/master' into SPARK-2137
205f17b [Yin Huai] Make Hive UDF wrapper support Timestamp.

(cherry picked from commit 891968509105d8d8cf5a608ad9473aeeed747089)
Signed-off-by: Reynold Xin <rxin@apache.org>