HOTFIX: Updating Python doc version
Updating versions for branch-1.0
[SPARK-2455] Mark (Shippable)VertexPartition serializable

VertexPartition and ShippableVertexPartition are contained in RDDs but are not marked Serializable, leading to NotSerializableExceptions when using Java serialization.

The fix is simply to mark them as Serializable. This PR does that and adds a test for serializing them using Java and Kryo serialization.

Author: Ankur Dave <ankurdave@gmail.com>

Closes #1376 from ankurdave/SPARK-2455 and squashes the following commits:

ed4a51b [Ankur Dave] Make (Shippable)VertexPartition serializable
1fd42c5 [Ankur Dave] Add failing tests for Java serialization

(cherry picked from commit 7a0135293192aaefc6ae20b57e15a90945bd8a4e)
Signed-off-by: Reynold Xin <rxin@apache.org>
[SPARK-2441][SQL] Add more efficient distinct operator.

Author: Michael Armbrust <michael@databricks.com>

Closes #1366 from marmbrus/partialDistinct and squashes the following commits:

12a31ab [Michael Armbrust] Add more efficient distinct operator.

(cherry picked from commit 7e26b57615f6c1d3f9058f9c19c05ec91f017f4c)
Signed-off-by: Reynold Xin <rxin@apache.org>
[SPARK-2405][SQL] Reusue same byte buffers when creating new instance of InMemoryRelation

Reuse byte buffers when creating unique attributes for multiple instances of an InMemoryRelation in a single query plan.

Author: Michael Armbrust <michael@databricks.com>

Closes #1332 from marmbrus/doubleCache and squashes the following commits:

4a19609 [Michael Armbrust] Clean up concurrency story by calculating buffersn the constructor.
b39c931 [Michael Armbrust] Allocations are kind of a side effect.
f67eff7 [Michael Armbrust] Reusue same byte buffers when creating new instance of InMemoryRelation

(cherry picked from commit 1a7d7cc85fb24de21f1cde67d04467171b82e845)
Signed-off-by: Reynold Xin <rxin@apache.org>
[maven-release-plugin] prepare release v1.0.1-rc3
[maven-release-plugin] prepare for next development iteration
[SPARK-2443][SQL] Fix slow read from partitioned tables

This fix obtains a comparable performance boost as [PR #1390](https://github.com/apache/spark/pull/1390) by moving an array update and deserializer initialization out of a potentially very long loop. Suggested by yhuai. The below results are updated for this fix.

## Benchmarks
Generated a local text file with 10M rows of simple key-value pairs. The data is loaded as a table through Hive. Results are obtained on my local machine using hive/console.

Without the fix:

Type | Non-partitioned | Partitioned (1 part)
------------ | ------------ | -------------
First run | 9.52s end-to-end (1.64s Spark job) | 36.6s (28.3s)
Stablized runs | 1.21s (1.18s) | 27.6s (27.5s)

With this fix:

Type | Non-partitioned | Partitioned (1 part)
------------ | ------------ | -------------
First run | 9.57s (1.46s) | 11.0s (1.69s)
Stablized runs | 1.13s (1.10s) | 1.23s (1.19s)

Author: Zongheng Yang <zongheng.y@gmail.com>

Closes #1408 from concretevitamin/slow-read-2 and squashes the following commits:

d86e437 [Zongheng Yang] Move update & initialization out of potentially long loop.

(cherry picked from commit d60b09bb60cff106fa0acddebf35714503b20f03)
Signed-off-by: Michael Armbrust <michael@databricks.com>
Add/increase severity of warning in documentation of groupBy()

groupBy()/groupByKey() is notorious for being a very convenient API that can lead to poor performance when used incorrectly.

This PR just makes it clear that users should be cautious not to rely on this API when they really want a different (more performant) one, such as reduceByKey().

(Note that one source of confusion is the name; this groupBy() is not the same as a SQL GROUP-BY, which is used for aggregation and is more similar in nature to Spark's reduceByKey().)

Author: Aaron Davidson <aaron@databricks.com>

Closes #1380 from aarondav/warning and squashes the following commits:

f60da39 [Aaron Davidson] Give better advice
d0afb68 [Aaron Davidson] Add/increase severity of warning in documentation of groupBy()
(cherry picked from commit a2aa7bebae31e1e7ec23d31aaa436283743b283b)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
[SPARK-2485][SQL] Lock usage of hive client.

Author: Michael Armbrust <michael@databricks.com>

Closes #1412 from marmbrus/lockHiveClient and squashes the following commits:

4bc9d5a [Michael Armbrust] protected[hive]
22e9177 [Michael Armbrust] Add comments.
7aa8554 [Michael Armbrust] Don't lock on hive's object.
a6edc5f [Michael Armbrust] Lock usage of hive client.

(cherry picked from commit c7c7ac83392b10abb011e6aead1bf92e7c73695e)
Signed-off-by: Aaron Davidson <aaron@databricks.com>
[SPARK-2483][SQL] Fix parsing of repeated, nested data access.

Author: Michael Armbrust <michael@databricks.com>

Closes #1411 from marmbrus/nestedRepeated and squashes the following commits:

044fa09 [Michael Armbrust] Fix parsing of repeated, nested data access.

(cherry picked from commit 0f98ef1a2c9ecf328f6c5918808fa5ca486e8afd)
Signed-off-by: Michael Armbrust <michael@databricks.com>
[SQL] Whitelist more Hive tests.

Author: Michael Armbrust <michael@databricks.com>

Closes #1396 from marmbrus/moreTests and squashes the following commits:

6660b60 [Michael Armbrust] Blacklist a test that requires DFS command.
8b6001c [Michael Armbrust] Add golden files.
ccd8f97 [Michael Armbrust] Whitelist more tests.

(cherry picked from commit bcd0c30c7eea4c50301cb732c733fdf4d4142060)
Signed-off-by: Michael Armbrust <michael@databricks.com>
SPARK-2407: Added internal implementation of SQL SUBSTR()

This replaces the Hive UDF for SUBSTR(ING) with an implementation in Catalyst
and adds tests to verify correct operation.

Author: William Benton <willb@redhat.com>

Closes #1359 from willb/internalSqlSubstring and squashes the following commits:

ccedc47 [William Benton] Fixed too-long line.
a30a037 [William Benton] replace view bounds with implicit parameters
ec35c80 [William Benton] Adds fixes from review:
4f3bfdb [William Benton] Added internal implementation of SQL SUBSTR()

(cherry picked from commit 61de65bc69f9a5fc396b76713193c6415436d452)
Signed-off-by: Michael Armbrust <michael@databricks.com>
[SQL] Attribute equality comparisons should be done by exprId.

Author: Michael Armbrust <michael@databricks.com>

Closes #1414 from marmbrus/exprIdResolution and squashes the following commits:

97b47bc [Michael Armbrust] Attribute equality comparisons should be done by exprId.

(cherry picked from commit 502f90782ad474e2630ed5be4d3c4be7dab09c34)
Signed-off-by: Michael Armbrust <michael@databricks.com>
[SPARK-2498] [SQL] Synchronize on a lock when using scala reflection inside data type objects.

JIRA ticket: https://issues.apache.org/jira/browse/SPARK-2498

Author: Zongheng Yang <zongheng.y@gmail.com>

Closes #1423 from concretevitamin/scala-ref-catalyst and squashes the following commits:

325a149 [Zongheng Yang] Synchronize on a lock when initializing data type objects in Catalyst.

(cherry picked from commit c2048a5165b270f5baf2003fdfef7bc6c5875715)
Signed-off-by: Michael Armbrust <michael@databricks.com>
[SPARK-2314][SQL] Override collect and take in JavaSchemaRDD, forwarding to SchemaRDD implementations.

Author: Aaron Staple <aaron.staple@gmail.com>

Closes #1421 from staple/SPARK-2314 and squashes the following commits:

73e04dc [Aaron Staple] [SPARK-2314] Override collect and take in JavaSchemaRDD, forwarding to SchemaRDD implementations.

(cherry picked from commit 90ca532a0fd95dc85cff8c5722d371e8368b2687)
Signed-off-by: Reynold Xin <rxin@apache.org>
[SPARK-2509][SQL] Add optimization for Substring.

`Substring` including `null` literal cases could be added to `NullPropagation`.

Author: Takuya UESHIN <ueshin@happy-camper.st>

Closes #1428 from ueshin/issues/SPARK-2509 and squashes the following commits:

d9eb85f [Takuya UESHIN] Add Substring cases to NullPropagation.

(cherry picked from commit 9b38b7c71352bb5e6d359515111ad9ca33299127)
Signed-off-by: Reynold Xin <rxin@apache.org>
[SPARK-2504][SQL] Fix nullability of Substring expression.

This is a follow-up of #1359 with nullability narrowing.

Author: Takuya UESHIN <ueshin@happy-camper.st>

Closes #1426 from ueshin/issues/SPARK-2504 and squashes the following commits:

5157832 [Takuya UESHIN] Remove unnecessary white spaces.
80958ac [Takuya UESHIN] Fix nullability of Substring expression.

(cherry picked from commit 632fb3d9a9ebb3d2218385403145d5b89c41c025)
Signed-off-by: Reynold Xin <rxin@apache.org>
[SPARK-2525][SQL] Remove as many compilation warning messages as possible in Spark SQL

JIRA: https://issues.apache.org/jira/browse/SPARK-2525.

Author: Yin Huai <huai@cse.ohio-state.edu>

Closes #1444 from yhuai/SPARK-2517 and squashes the following commits:

edbac3f [Yin Huai] Removed some compiler type erasure warnings.

(cherry picked from commit df95d82da7c76c074fd4064f7c870d55d99e0d8e)
Signed-off-by: Reynold Xin <rxin@apache.org>
[SPARK-2518][SQL] Fix foldability of Substring expression.

This is a follow-up of #1428.

Author: Takuya UESHIN <ueshin@happy-camper.st>

Closes #1432 from ueshin/issues/SPARK-2518 and squashes the following commits:

37d1ace [Takuya UESHIN] Fix foldability of Substring expression.

(cherry picked from commit cc965eea510397642830acb21f61127b68c098d6)
Signed-off-by: Reynold Xin <rxin@apache.org>