HIVE-9019 - Avoid using SPARK_JAVA_OPTS [Spark Branch] (Rui Li via Brock)

git-svn-id: https://svn.apache.org/repos/asf/hive/branches/spark@1643896 13f79535-47bb-0310-9956-ffa450edef68
HIVE-9040 - Spark Memory can be formatted string [Spark Branch] (Jimmy Xiang via Brock)

git-svn-id: https://svn.apache.org/repos/asf/hive/branches/spark@1643962 13f79535-47bb-0310-9956-ffa450edef68
HIVE-8638: Implement bucket map join optimization [Spark Branch] (Jimmy via Xuefu)

git-svn-id: https://svn.apache.org/repos/asf/hive/branches/spark@1643976 13f79535-47bb-0310-9956-ffa450edef68
[spark-client] Netty-based RPC implementation.

This is the RPC layer for the spark-client; it just provides the RPC mechanism
for replacing akka in the spark-client communication. See README.md for
more discussion about the approach.

The client impl is still using akka - that will be changed in a separate commit.
[spark-client] Goodbye akka, hello netty.

Use the netty-based RPC implementation for spark-client. API is still
exactly the same, just the internals have changed. API semantics should
still be the same too, although with the new RPC system there is still
room for improvement.

Main thing to remember right now is that all types sent over the wire
need an empty constructor. Currently, failure to do that will result
in errors to be printed to the logs and RPC channels to close, but
for easier debugging in the future I'm planning to change the RPC
internals a bit so that failure to deserialize the payload causes an
RPC failure, instead of the current behavior.
[spark-client] Use a Promise in JobHandleImpl.

Makes it integrate better with the RPC layer, and avoids some ugly
code in the process.
[spark-client] Set up RPC thread factory, add missing timeout.
[spark-client] Better handle serialization errors.

It's hard to report back serialization errors that happen during an
RPC if the RPC header and payload are serialized as a single object.
So break those down into two different objects; we're sure the internal
RPC header is serializable, so this allows us to better handle the case
when the user payload is not serializable and provide nicer errors,
making it easier to debug things.
Fix buglets.
Fix some serialization issues.
More fixes.

- More types needed serialization tweaks (empty constructors or
  transient fields)
- Fix a race when registering outgoing RPCs.
- Add a TODO to fix some very suspicious code in SparkCounter.
Fix SparkCounters serialization.

This code is a little bit weird; the "Accumulator" instance cannot
be serialized by Kryo's default serializer, so we cannot include
it when sending SparkCounters back to the RSC. But we can't make the
field transient, because apparently it need to be serialized when
sent to actual tasks.

Long term it would be better to separate internal counter usage
from RSC counter views (in the spirit of keeping RSC completely
independent from the rest of Hive). But right now make things work
by copying things.
Configurable channel traffic log level.
Avoid an ugly exception when remote end dies.
Make rpc server instance private in factory.
Add max message size checks.
Fix bug when looking for a suitable address.
Check negative message sizes.
Demote log level for exceptions in channel pipeline.

These can be noisy and misleading, especially on the server side when
things like port scanners are running. Instead, print a simplified message
in the non-debug case.

For the sending side, code should be checking the result of the futures
returned by the RPC layer to check for exceptions in the outbound
channel.
Fix race when sending job.

Need to register that the job was sent before issuing the RPC; that
way we're sure that we know the job was sent before any reply has the
chance to arrive.