apache/spark — pull request #3488 by maropu (2014-11-27 02:02:07+00:00)
Title: [SPARK-4633] Support GZIPOutputStream in spark.io.compression.codec

gzip is widely used in other frameworks such as hadoop mapreduce and tez, and also
I think that gizip is more stable than other codecs in terms of both performance
and space overheads.

I have one open question; current spark configurations have a block size option
for each codec (spark.io.compression.[gzip|lz4|snappy].block.size).
As # of codecs increases, the configurations have more options and
I think that it is sort of complicated for non-expert users.

To mitigate it, my thought follows;
the three configurations are replaced with a single option for block size
(spark.io.compression.block.size). Then, 'Meaning' in configurations
will describe "This option makes an effect on gzip, lz4, and snappy.
Block size (in bytes) used in compression, in the case when these compression
codecs are used. Lowering...".

Thought?