[SQL][DOC][Minor] Spark SQL Hive now support dynamic partitioning

Author: wangfei <wangfei1@huawei.com>

Closes #3127 from scwf/patch-9 and squashes the following commits:

e39a560 [wangfei] now support dynamic partitioning

(cherry picked from commit 636d7bcc96b912f5b5caa91110cd55b55fa38ad8)
Signed-off-by: Michael Armbrust <michael@databricks.com>
[SPARK-4225][SQL] Resorts to SparkContext.version to inspect Spark version

This PR resorts to `SparkContext.version` rather than META-INF/MANIFEST.MF in the assembly jar to inspect Spark version. Currently, when built with Maven, the MANIFEST.MF file in the assembly jar is incorrectly replaced by Guava 15.0 MANIFEST.MF, probably because of the assembly/shading tricks.

Another related PR is #3103, which tries to fix the MANIFEST issue.

Author: Cheng Lian <lian@databricks.com>

Closes #3105 from liancheng/spark-4225 and squashes the following commits:

d9585e1 [Cheng Lian] Resorts to SparkContext.version to inspect Spark version

(cherry picked from commit 86e9eaa3f0ec23cb38bce67585adb2d5f484f4ee)
Signed-off-by: Michael Armbrust <michael@databricks.com>
[SQL] Support ScalaReflection of schema in different universes

Author: Michael Armbrust <michael@databricks.com>

Closes #3096 from marmbrus/reflectionContext and squashes the following commits:

adc221f [Michael Armbrust] Support ScalaReflection of schema in different universes

(cherry picked from commit 8154ed7df6c5407e638f465d3bd86b43f36216ef)
Signed-off-by: Michael Armbrust <michael@databricks.com>
[SQL] Modify keyword val location according to ordering

'DOUBLE' should be moved before 'ELSE' according to the ordering convension

Author: Jacky Li <jacky.likun@gmail.com>

Closes #3080 from jackylk/patch-5 and squashes the following commits:

3c11df7 [Jacky Li] [SQL] Modify keyword val location according to ordering

(cherry picked from commit 68609c51ad1ab2def302df3c4a1c0bc1ec6e1075)
Signed-off-by: Michael Armbrust <michael@databricks.com>
[SPARK-4213][SQL] ParquetFilters - No support for LT, LTE, GT, GTE operators

Following description is quoted from JIRA:

When I issue a hql query against a HiveContext where my predicate uses a column of string type with one of LT, LTE, GT, or GTE operator, I get the following error:
scala.MatchError: StringType (of class org.apache.spark.sql.catalyst.types.StringType$)
Looking at the code in org.apache.spark.sql.parquet.ParquetFilters, StringType is absent from the corresponding functions for creating these filters.
To reproduce, in a Hive 0.13.1 shell, I created the following table (at a specified DB):

    create table sparkbug (
    id int,
    event string
    ) stored as parquet;

Insert some sample data:

    insert into table sparkbug select 1, '2011-06-18' from <some table> limit 1;
    insert into table sparkbug select 2, '2012-01-01' from <some table> limit 1;

Launch a spark shell and create a HiveContext to the metastore where the table above is located.

    import org.apache.spark.sql._
    import org.apache.spark.sql.SQLContext
    import org.apache.spark.sql.hive.HiveContext
    val hc = new HiveContext(sc)
    hc.setConf("spark.sql.shuffle.partitions", "10")
    hc.setConf("spark.sql.hive.convertMetastoreParquet", "true")
    hc.setConf("spark.sql.parquet.compression.codec", "snappy")
    import hc._
    hc.hql("select * from <db>.sparkbug where event >= '2011-12-01'")

A scala.MatchError will appear in the output.

Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>

Closes #3083 from sarutak/SPARK-4213 and squashes the following commits:

4ab6e56 [Kousuke Saruta] WIP
b6890c6 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-4213
9a1fae7 [Kousuke Saruta] Fixed ParquetFilters so that compare Strings

(cherry picked from commit 14c54f1876fcf91b5c10e80be2df5421c7328557)
Signed-off-by: Michael Armbrust <michael@databricks.com>
[SPARK-4272] [SQL] Add more unwrapper functions for primitive type in TableReader

Currently, the data "unwrap" only support couple of primitive types, not all, it will not cause exception, but may get some performance in table scanning for the type like binary, date, timestamp, decimal etc.

Author: Cheng Hao <hao.cheng@intel.com>

Closes #3136 from chenghao-intel/table_reader and squashes the following commits:

fffb729 [Cheng Hao] fix bug for retrieving the timestamp object
e9c97a4 [Cheng Hao] Add more unwrapper functions for primitive type in TableReader

(cherry picked from commit 60ab80f501b8384ddf48a9ac0ba0c2b9eb548b28)
Signed-off-by: Michael Armbrust <michael@databricks.com>