apache/spark — pull request #2475 by marmbrus (2014-09-21 01:38:08+00:00)
Title: [WIP][SPARK-3247][SQL] An API for adding foreign data sources to Spark SQL

**Work in progress - APIs may change**

This PR introduces a new set of APIs to Spark SQL that allow other developers to add support for reading data from new sources. As an example, a library is included for reading data encoded using Avro.

New sources must implement the interface `BaseRelation`, which is responsible for describing the schema of the data. This base relation must also implement at least one `Scan` interface, which is responsible for producing an RDD containing row objects. The various Scan interfaces allow for optimizations such as column pruning and filter push down, when the underlying data source can handle these operations.

External data sources can be accessed using either the programatic API or using pure SQL. For example, the included Avro library could be called from Scala query DSL as follows:

```scala
import org.apache.spark.sql.avro._

val results = TestSQLContext
.avroFile("../hive/src/test/resources/data/files/episodes.avro")
.select('title)
.collect()
```

The same can be done in pure SQL, for example from the SQL command line or JDBC interface.

```sql
CREATE FOREIGN TEMPORARY TABLE avroTable
USING org.apache.spark.sql.avro
OPTIONS (path "../hive/src/test/resources/data/files/episodes.avro");

SELECT * FROM avroTable;
```

TODO:
- [ ] Move command refactoring into separate PR
- [ ] Transition parquet and json support to new API
- [ ] Figure out how to package data sources and their dependencies
- [ ] Examples / implementation of more advanced scan types
- [ ] Support for foreign catalogs
- [ ] Introspection like `describe` for foreign tables.
- [ ] More tests