baishuo/spark — pull request #1 by liancheng (2014-09-17 08:54:38+00:00)
Title: Refactors dynamic partitioning support

Major refactoring work include:

- Renamed `SparkHiveHadoopWriter` to `SparkHiveWriterContainer` and moved the file to the proper source folder

Also simplified its implementation with lazy `val`s, and refactored several interface methods to make the following point happen.

- Fixed the job commitment issue mentioned in [this comment](https://github.com/apache/spark/pull/2226#discussion_r17625214)

This PR tries to mimic Hive `DynamicPartitionFileRecordWriterContainer` with `SparkHiveDynamicPartitionWriterContainer`, which inherits from `SparkHiveWriterContainer`. Instead of creating multiple `SparkHiveHadoopWriter` on the fly at the executor side, now we create one writer container at the driver side and ask the container to create actual Hive writers on the fly. This brings several advantages:

1. Job can be properly committed on driver side.
2. Dynamic partitioning specific logics (calculating partition directory paths and creating writers, etc.) is moved to `SparkHiveDynamicPartitionWriterContainer`, leaving `InsertIntoHiveTable` much cleaner.

- Fixed the partition spec validation bug (see [here](https://github.com/apache/spark/pull/2226#discussion_r17625525))
- Inherits compression settings when writing dynamic partitioned table (see [here](https://github.com/apache/spark/pull/2226#discussion-diff-17625337))
- Took the chance to make `InsertInoHiveTable` a subclass of `org.apache.spark.sql.executionCommand`.