amplab/shark — pull request #219 by harveyfeng (2013-11-22 04:46:21+00:00)
Title: New semantics for caching tables, including persistence across Shark sessions.

Caching tables relies on a single `shark.cache` table property, which can have to following values:
- `memory`: table will be stored in memory and on disk (cache is write-through). Persistent across Shark sessions. By default, a Shark session will reload all previously cached tables into memory.
- `memory-only`: table will only be stored in memory, and dropped at the end of a session.
tachyon
- `none`: normal Hive table
This replaces the shark.cache.storageLevel and shark.cache.unifyView properties available to the user ATM.

From https://github.com/amplab/shark/pull/213,
Enables Shark to maintain a "unified view" of cached tables, in that its in-memory, columnar-backed store is synchronized with the backing disk store (the table's data directory in the Hive warehouse). Both the cached and on-disk copies are modified for any table updates (e.g., specified by `LOAD`, `INSERT`, or `CREATE TABLE AS SELECT`) - the behavior is similar to a write-through cache.

In terms of usability,

- No longer required to create new tables in order to cache existing ones.

Instead of ```CREATE TABLE src_cached AS SELECT * FROM src```, tables can be cached using ``` CACHE src```. Uncaching is just an ```UNCACHE src```

* Since unified views are backed by a disk copy, updates on cached tables persist across Shark sessions.

Previously, the `src_cached` table and any updates to it were only accessible from the Shark session that it was created in. Accessing it from a different session would cause a thrown runtime exception (something like a `QueryExecutionException: Table src not found in block manager`).
By default, all cached tables are unified. This can be switched off by setting `"shark.cache" = "false"` in the table properties, or calling `set shark.cache.unify.default = false` in the Hive CLI.

In terms of implementation, the main additions for handling commands targeting unified views are:

- Maintenance of two SerDes

All unified views have `LazySimpleSerDe` as its SerDe property. `TableScanOperator` will use a `ColumnarSerDe` if it notices that the table scan is targeted at an in memory table.

- Different query plans generated for `INSERT` and `CTAS`.

Previously, the `MemoryStoreSinkOperator` was used for operations on cached tables. Now, it's only used for operations on cached tables that are not unified (i.e., just stored in memory).
For operations on unified views, query plans will use a `FileSinkOperator` that's modified by the `OperatorFactory` to use a disk SerDe instead of `ColumnarSerDe`, since the operator's input rows may be lazy objects.

- New SparkLoadTask

For `CREATE TABLE`, `CTAS`, or `LOAD ... OVERWRITE INTO`, the execution process starts with a `HadoopTableReader` that scans tables or partitions data directories from the Hive warehouse into a Hadoop RDD. That RDD is then transformed and materialized it into an `RDD[TablePartition]` backed by columnar store.
For `INSERT INTO`, or `LOAD ... INTO` append updates, the `HadoopTableReader` uses a path filter that will only accept new files loaded into the target table/partition data directory by Hive's `FileSinkOperator` and `MoveTask`. That path filter is created using a snapshot of the target directory during semantic analysis.