Split HadoopRDD into one for general Hadoop datasets and one tailored to Hadoop files, which is a common case.

This is the first step to avoiding unnecessary Configuration broadcasts per HadoopRDD instantiation.
Add a cache for HadoopRDD metadata needed during computation.

Currently, the cache is in SparkHadoopUtils, since it's conveniently a member of the SparkEnv.
Move Configuration broadcasts to SparkContext.
Merge remote-tracking branch 'oldsparkme/hadoopRDD-broadcast-change' into hadoop-config-cache
Merge HadoopDatasetRDD into HadoopRDD.
SPARK-920 - JSON endpoint URI scheme part (spark://) duplicated
SPARK-921 - Add Application UI URL to ApplicationInfo Json output
Fixing SPARK-602: PythonPartitioner

Currently PythonPartitioner determines partition ID by hashing a
byte-array representation of PySpark's key. This PR lets
PythonPartitioner use the actual partition ID, which is required e.g.
for sorting via PySpark.
Merge pull request #33 from AndreSchumacher/pyspark_partition_key_change

Fixing SPARK-602: PythonPartitioner

Currently PythonPartitioner determines partition ID by hashing a
byte-array representation of PySpark's key. This PR lets
PythonPartitioner use the actual partition ID, which is required e.g.
for sorting via PySpark.
Merge pull request #27 from davidmccauley/master

SPARK-920/921 - JSON endpoint updates

920 - Removal of duplicate scheme part of Spark URI, it was appearing as spark://spark//host:port in the JSON field.

JSON now delivered as:
url:spark://127.0.0.1:7077

921 - Adding the URL of the Main Application UI will allow custom interfaces (that use the JSON output) to redirect from the standalone UI.
Fix API changes; lines > 100 chars.
Bumping EC2 default version in master to `0.8.0`.
Make HadoopRDD object Spark private.
Merge pull request #36 from pwendell/versions

Bumping EC2 default version in master to .

This change was already made on . This PR ports the change up to master.
Some comments regarding JobConf and InputFormat caching for HadoopRDDs.
Merge pull request #20 from harveyfeng/hadoop-config-cache

Allow users to pass broadcasted Configurations and cache InputFormats across Hadoop file reads.

Note: originally from https://github.com/mesos/spark/pull/942

Currently motivated by Shark queries on Hive-partitioned tables, where there's a JobConf broadcast for every Hive-partition (i.e., every subdirectory read). The only thing different about those JobConfs is the input path - the Hadoop Configuration that the JobConfs are constructed from remain the same.
This PR only modifies the old Hadoop API RDDs, but similar additions to the new API might reduce computation latencies a little bit for high-frequency FileInputDStreams (which only uses the new API right now).

As a small bonus, added InputFormats caching, to avoid reflection calls for every RDD#compute().

Few other notes:

Added a general soft-reference hashmap in SparkHadoopUtil because I wanted to avoid adding another class to SparkEnv.
SparkContext default hadoopConfiguration isn't cached. There's no equals() method for Configuration, so there isn't a good way to determine when configuration properties have changed.
Merging build changes in from 0.8
Merge pull request #37 from pwendell/merge-0.8

merge in remaining changes from `branch-0.8`

This merges in the following changes from `branch-0.8`:

- The scala version is included in the published maven artifact names
- A unit tests which had non-deterministic failures is ignored (see SPARK-908)
- A minor documentation change shows the short version instead of the full version
- Moving the kafka jar to be "provided"
- Changing the default spark ec2 version.
- Some spacing changes caused by Maven's release plugin

Note that I've squashed this into a single commit rather than pull in the branch-0.8 history. There are a bunch of release/revert commits there that make the history super ugly.
Merge pull request #31 from sundeepn/branch-0.8

Resolving package conflicts with hadoop 0.23.9

Hadoop 0.23.9 is having a package conflict with easymock's dependencies.

(cherry picked from commit 023e3fdf008b3194a36985a07923df9aaf64e520)
Signed-off-by: Reynold Xin <rxin@apache.org>
Adding Shark 0.7.1 to EC2 scripts