Netflix/PigPen — pull request #44 by mbossenbroek (2014-07-30 23:41:12+00:00)
Title: Add Parquet support

Add parquet support to PigPen. See https://github.com/apache/incubator-parquet-mr

Thanks @mping for the idea and a lot of the initial work on this one.

Usage:

```clj
(require '[pigpen.core :as pig]
'[pigpen.parquet.core :as pig-pq])

(->>
(pig/return [{:x 1 :y "a"}
{:x 2 :y "b"}
{:x 3 :y "c"}])
(pig-pq/store-parquet "test.pq" {:x :int, :y :chararray})
(pig/dump))

(->>
(pig-pq/load-parquet "test.pq/part-m-00001.parquet" {:x :int, :y :chararray})
(pig/dump))

=> [{:y "a", :x 1} {:y "b", :x 2} {:y "c", :x 3}]
```

This also changes how local-mode is done for load and store commands. The new version uses `PigPenLocalLoader` and `PigPenLocalStorage` protocols in `pigpen-core/src/main/clojure/pigpen/local.clj` to implement local versions of load/store commands.

I added a bunch of code to assist in running hadoop and pig classes directly. Hopefully these should be useful when adding new formats.

cc @daveray