apache/spark — pull request #1764 by aarondav (2014-08-04 07:35:19+00:00)
Title: [SPARK-2824/2825][SQL] Work towards separating data location from format

Currently, there is a fundamental assumption in SparkSQL that a Parquet
table is stored at a certain Hadoop path and that a Metastore table is
stored within the Hive warehouse. However, the fact that a table is
Parquet or serialized as an object file is independent of where the data
is actually located.

This patch attempts to work towards creating a cleaner separation between
where the data is located and the format the data is in by introducing
two concepts: a TableFormat and a TableLocation. This abstraction
enables code like the following:

```scala
val myTable = // ...
myTable.saveAsTable("myTable", classOf[ParquetFormat])
hql("SELECT * FROM myTable").collect // reads from Parquet!

// Also allows expansion of file-writing later:
myTable.saveAsFile("/my/file", classOf[ParquetFormat])
```

Additionally, this allows us to trivially support external tables with arbitrary formats.

However, this PR doesn't attempt to make any radical changes. Parquet files still only
support being written to a single Hadoop directory, but this can be part of a Hive table or
a normal directory. The MetastoreRelation still requires living within the Metastore because
it relies heavily on the metadata there. The hope of this patch is that it enables the two linked
features ([SPARK-2824](https://issues.apache.org/jira/browse/SPARK-2824) and [SPARK-2825](https://issues.apache.org/jira/browse/SPARK-2825)) while adding a useful abstraction for the future.