SPARK-2180:  support HAVING clauses in Hive queries

This PR extends Spark's HiveQL support to handle HAVING clauses in aggregations.  The HAVING test from the Hive compatibility suite doesn't appear to be runnable from within Spark, so I added a simple comparable test to `HiveQuerySuite`.

Author: William Benton <willb@redhat.com>

Closes #1136 from willb/SPARK-2180 and squashes the following commits:

3bbaf26 [William Benton] Added casts to HAVING expressions
83f1340 [William Benton] scalastyle fixes
18387f1 [William Benton] Add test for HAVING without GROUP BY
b880bef [William Benton] Added semantic error for HAVING without GROUP BY
942428e [William Benton] Added test coverage for SPARK-2180.
56084cc [William Benton] Add support for HAVING clauses in Hive queries.

(cherry picked from commit 171ebb3a824a577d69443ec68a3543b27914cf6d)
Signed-off-by: Reynold Xin <rxin@apache.org>
[SPARK-2225] Turn HAVING without GROUP BY into WHERE.

@willb

Author: Reynold Xin <rxin@apache.org>

Closes #1161 from rxin/having-filter and squashes the following commits:

fa8359a [Reynold Xin] [SPARK-2225] Turn HAVING without GROUP BY into WHERE.

(cherry picked from commit 0ac71d1284cd4f011d5763181cba9ecb49337b66)
Signed-off-by: Reynold Xin <rxin@apache.org>
Move ScriptTransformation into the appropriate place.

Author: Reynold Xin <rxin@apache.org>

Closes #1162 from rxin/script and squashes the following commits:

2c836b9 [Reynold Xin] Move ScriptTransformation into the appropriate place.

(cherry picked from commit d4c7572dba1be49e55ceb38713652e5bcf485be8)
Signed-off-by: Reynold Xin <rxin@apache.org>
[SQL] Use hive.SessionState, not the thread local SessionState

Note that this is simply mimicing lookupRelation(). I do not have a concrete notion of why this solution is necessarily right-er than SessionState.get, but SessionState.get is returning null, which is bad.

Author: Aaron Davidson <aaron@databricks.com>

Closes #1148 from aarondav/createtable and squashes the following commits:

37c3e7c [Aaron Davidson] [SQL] Use hive.SessionState, not the thread local SessionState

(cherry picked from commit 2044784915554a890ca6f8450d8403495b2ee4f3)
Signed-off-by: Reynold Xin <rxin@apache.org>
[SQL] Pass SQLContext instead of SparkContext into physical operators.

This makes it easier to use config options in operators.

Author: Reynold Xin <rxin@apache.org>

Closes #1164 from rxin/sqlcontext and squashes the following commits:

797b2fd [Reynold Xin] Pass SQLContext instead of SparkContext into physical operators.

(cherry picked from commit ca5d8b5904dc6dd5b691af506d3a842e508b3673)
Signed-off-by: Reynold Xin <rxin@apache.org>
[SQL] Break hiveOperators.scala into multiple files.

The single file was getting very long (500+ loc).

Author: Reynold Xin <rxin@apache.org>

Closes #1166 from rxin/hiveOperators and squashes the following commits:

5b43068 [Reynold Xin] [SQL] Break hiveOperators.scala into multiple files.

(cherry picked from commit ec935abce13b60f353236566da149c0c87bb1002)
Signed-off-by: Michael Armbrust <michael@databricks.com>
SPARK-2034. KafkaInputDStream doesn't close resources and may prevent JVM shutdown

Tobias noted today on the mailing list:

========

I am trying to use Spark Streaming with Kafka, which works like a
charm – except for shutdown. When I run my program with "sbt
run-main", sbt will never exit, because there are two non-daemon
threads left that don't die.
I created a minimal example at
<https://gist.github.com/tgpfeiffer/b1e765064e983449c6b6#file-kafkadoesntshutdown-scala>.
It starts a StreamingContext and does nothing more than connecting to
a Kafka server and printing what it receives. Using the `future
Unknown macro: { ... }
` construct, I shut down the StreamingContext after some seconds and
then print the difference between the threads at start time and at end
time. The output can be found at
<https://gist.github.com/tgpfeiffer/b1e765064e983449c6b6#file-output1>.
There are a number of threads remaining that will prevent sbt from
exiting.
When I replace `KafkaUtils.createStream(...)` with a call that does
exactly the same, except that it calls `consumerConnector.shutdown()`
in `KafkaReceiver.onStop()` (which it should, IMO), the output is as
shown at <https://gist.github.com/tgpfeiffer/b1e765064e983449c6b6#file-output2>.
Does anyone have any idea what is going on here and why the program
doesn't shut down properly? The behavior is the same with both kafka
0.8.0 and 0.8.1.1, by the way.

========

Something similar was noted last year:

http://mail-archives.apache.org/mod_mbox/spark-dev/201309.mbox/%3C1380220041.2428.YahooMailNeo@web160804.mail.bf1.yahoo.com%3E

KafkaInputDStream doesn't close `ConsumerConnector` in `onStop()`, and does not close the `Executor` it creates. The latter leaves non-daemon threads and can prevent the JVM from shutting down even if streaming is closed properly.

Author: Sean Owen <sowen@cloudera.com>

Closes #980 from srowen/SPARK-2034 and squashes the following commits:

9f31a8d [Sean Owen] Restore ClassTag to private class because MIMA flags it; is the shadowing intended?
2d579a8 [Sean Owen] Close ConsumerConnector in onStop; shutdown() the local Executor that is created so that its threads stop when done; close the Zookeeper client even on exception; fix a few typos; log exceptions that otherwise vanish
(cherry picked from commit 476581e8c8ca03a5940c404fee8a06361ff94cb5)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
[SPARK-1112, 2156] (1.0 edition) Use correct akka frame size and overhead amounts.

SPARK-1112: This is a more conservative version of #1132 that doesn't change
around the actor system initialization on the executor. Instead we just
directly read the current frame size limit from the ActorSystem.

SPARK-2156: This uses the same fixe as in #1132.

Author: Patrick Wendell <pwendell@gmail.com>

Closes #1172 from pwendell/akka-10-fix and squashes the following commits:

d56297e [Patrick Wendell] Set limit in LocalBackend to preserve test expectations
9f5ed19 [Patrick Wendell] [SPARK-1112, 2156] (1.0 edition) Use correct akka frame size and overhead amounts.
SPARK-2241: quote command line args in ec2 script

To preserve quoted command line args (in case options have space in them).

Author: Ori Kremer <ori.kremer@gmail.com>

Closes #1169 from orikremer/quote_cmd_line_args and squashes the following commits:

67e2aa1 [Ori Kremer] quote command line args
(cherry picked from commit 9fc373e3a9a8ba7bea9df0950775f48918f63a8a)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Fixed small running on YARN docs typo

The backslash is needed for multiline command

Author: Vlad <frolvlad@gmail.com>

Closes #1158 from frol/patch-1 and squashes the following commits:

e258044 [Vlad] Fixed small running on YARN docs typo

(cherry picked from commit b88238faeed8ba723986cf78d64f84965facb236)
Signed-off-by: Thomas Graves <tgraves@apache.org>
Fix mvn detection

When mvn is not detected (not in executor's path), 'set -e' causes the
detection to terminate the script before the helpful error message can
be displayed.

Author: Matthew Farrellee <matt@redhat.com>

Closes #1181 from mattf/master-0 and squashes the following commits:

506549f [Matthew Farrellee] Fix mvn detection
(cherry picked from commit 853a2b951d4c7f6c6c37f53b465b3c7b77691b7c)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
[SPARK-1669][SQL]  Made cacheTable idempotent

JIRA issue: [SPARK-1669](https://issues.apache.org/jira/browse/SPARK-1669)

Caching the same table multiple times should end up with only 1 in-memory columnar representation of this table.

Before:

```
scala> loadTestTable("src")
...
scala> cacheTable("src")
...
scala> cacheTable("src")
...
scala> table("src")
...
== Query Plan ==
InMemoryColumnarTableScan [key#2,value#3], (InMemoryRelation [key#2,value#3], false, (InMemoryColumnarTableScan [key#2,value#3], (InMemoryRelation [key#2,value#3], false, (HiveTableScan [key#2,value#3], (MetastoreRelation default, src, None), None))))
```

After:

```
scala> loadTestTable("src")
...
scala> cacheTable("src")
...
scala> cacheTable("src")
...
scala> table("src")
...
== Query Plan ==
InMemoryColumnarTableScan [key#2,value#3], (InMemoryRelation [key#2,value#3], false, (HiveTableScan [key#2,value#3], (MetastoreRelation default, src, None), None))
```

Author: Cheng Lian <lian.cs.zju@gmail.com>

Closes #1183 from liancheng/spark-1669 and squashes the following commits:

68f8a20 [Cheng Lian] Removed an unused import
51bae90 [Cheng Lian] Made cacheTable idempotent

(cherry picked from commit a4bc442ca2c35444de8a33376b6f27c6c2a9003d)
Signed-off-by: Michael Armbrust <michael@databricks.com>
[SPARK-2227] Support dfs command in SQL.

Note that nothing gets printed to the console because we don't properly maintain session state right now.

I will have a followup PR that fixes it.

Author: Reynold Xin <rxin@apache.org>

Closes #1167 from rxin/commands and squashes the following commits:

56f04f8 [Reynold Xin] [SPARK-2227] Support dfs command in SQL.

(cherry picked from commit 51c8168377a89d20d0b2d7b9a28af58593a0fe0c)
Signed-off-by: Reynold Xin <rxin@apache.org>
[SPARK-2252] Fix MathJax for HTTPs.

Found out about this from the Hacker News link to GraphX which was using HTTPs.

@mengxr

Author: Reynold Xin <rxin@apache.org>

Closes #1189 from rxin/mllib-doc and squashes the following commits:

5328be0 [Reynold Xin] [SPARK-2252] Fix MathJax for HTTPs.

(cherry picked from commit 420c1c3e1beea03453e0eb9dc06f226c80496d68)
Signed-off-by: Reynold Xin <rxin@apache.org>
[SQL]Add base row updating methods for JoinedRow

This will be helpful in join operators.

Author: Cheng Hao <hao.cheng@intel.com>

Closes #1187 from chenghao-intel/joinedRow and squashes the following commits:

87c19e3 [Cheng Hao] Add base row set methods for JoinedRow

(cherry picked from commit 133495d82672c3f34d40a6298cc80c31f91faf5c)
Signed-off-by: Michael Armbrust <michael@databricks.com>
[SPARK-2264][SQL] Fix failing CachedTableSuite

Author: Michael Armbrust <michael@databricks.com>

Closes #1201 from marmbrus/fixCacheTests and squashes the following commits:

9d87ed1 [Michael Armbrust] Use analyzer (which runs to fixed point) instead of manually removing analysis operators.

Conflicts:

	sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala
Fix possible null pointer in acumulator toString

Author: Michael Armbrust <michael@databricks.com>

Closes #1204 from marmbrus/nullPointerToString and squashes the following commits:

35b5fce [Michael Armbrust] Fix possible null pointer in acumulator toString

(cherry picked from commit 2714968e1b40221739c5dfba7ca4c0c06953dbe2)
Signed-off-by: Reynold Xin <rxin@apache.org>
[SPARK-2263][SQL] Support inserting MAP<K, V> to Hive tables

JIRA issue: [SPARK-2263](https://issues.apache.org/jira/browse/SPARK-2263)

Map objects were not converted to Hive types before inserting into Hive tables.

Author: Cheng Lian <lian.cs.zju@gmail.com>

Closes #1205 from liancheng/spark-2263 and squashes the following commits:

c7a4373 [Cheng Lian] Addressed @concretevitamin's comment
784940b [Cheng Lian] SARPK-2263: support inserting MAP<K, V> to Hive tables

(cherry picked from commit 8fade8973e5fc97f781de5344beb66b90bd6e524)
Signed-off-by: Reynold Xin <rxin@apache.org>
[BUGFIX][SQL] Should match java.math.BigDecimal when wnrapping Hive output

The `BigDecimal` branch in `unwrap` matches to `scala.math.BigDecimal` rather than `java.math.BigDecimal`.

Author: Cheng Lian <lian.cs.zju@gmail.com>

Closes #1199 from liancheng/javaBigDecimal and squashes the following commits:

e9bb481 [Cheng Lian] Should match java.math.BigDecimal when wnrapping Hive output

(cherry picked from commit 22036aeb1b2cac7f48cd60afea925b42a5318631)
Signed-off-by: Reynold Xin <rxin@apache.org>
[SPARK-2267] Log exception when TaskResultGetter fails to fetch/deserialze task result

Note that this is only for branch-1.0 because master's been fixed.

Author: Reynold Xin <rxin@apache.org>

Closes #1202 from rxin/SPARK-2267 and squashes the following commits:

ce1b19b [Reynold Xin] [SPARK-2267] Log exception when TaskResultGetter fails to fetch/deserialize task result