Data Platform/Systems/Hive/Compression

== Uncompressed vs. Snappy compressed Sequence Files ==
I just ran some rough comparisons of data sizes and Hive queries of webrequest data stored in HDFS uncompressed vs. as Snappy compressed Sequence Files.
=== Uncompressed ===
==== Size ====
In the first 8 hours of January 7th, 2014, uncompressed JSON formatted mobile webrequest logs imported into HDFS via Kafka totaled 91.1 GB. Each hourly import was between 8 and 12 GB each.
hdfs dfs -du -s -h /wmf/data/external/webrequest_mobile/hourly/2014/01/07/{00..08}
10.6 G /wmf/data/external/webrequest_mobile/hourly/2014/01/07/00
10.9 G /wmf/data/external/webrequest_mobile/hourly/2014/01/07/01
11.3 G /wmf/data/external/webrequest_mobile/hourly/2014/01/07/02
11.7 G /wmf/data/external/webrequest_mobile/hourly/2014/01/07/03
10.6 G /wmf/data/external/webrequest_mobile/hourly/2014/01/07/04
9.9 G /wmf/data/external/webrequest_mobile/hourly/2014/01/07/05
9.1 G /wmf/data/external/webrequest_mobile/hourly/2014/01/07/06
8.6 G /wmf/data/external/webrequest_mobile/hourly/2014/01/07/07
8.4 G /wmf/data/external/webrequest_mobile/hourly/2014/01/07/08
91.1 G
==== Query Time ====
Running a select count(*) query on a single hour took 44.276 seconds, and launched 42 mappers. Running the same query on 8 hours of data took 158.627 seconds and launched 343 mappers.
-- 
select count(*) from webrequest_mobile where year=2014 and month=01 and day=07 and hour=00;
...
MapReduce Total cumulative CPU time: 16 minutes 11 seconds 670 msec
Ended Job = job_1387838787660_0365
MapReduce Jobs Launched:
Job 0: Map: 42 Reduce: 1 Cumulative CPU: 971.67 sec HDFS Read: 11422909543 HDFS Write: 9 SUCCESS
Total MapReduce CPU Time Spent: 16 minutes 11 seconds 670 msec
OK
_c0
16641115
Time taken: 44.276 seconds
select count(*) from webrequest_mobile where year=2014 and month=01 and day=07 and hour between 00 and 08;
...
MapReduce Total cumulative CPU time: 0 days 4 hours 16 minutes 12 seconds 420 msec
Ended Job = job_1387838787660_0363
MapReduce Jobs Launched:
Job 0: Map: 343 Reduce: 1 Cumulative CPU: 15372.42 sec HDFS Read: 98055786272 HDFS Write: 10 SUCCESS
Total MapReduce CPU Time Spent: 0 days 4 hours 16 minutes 12 seconds 420 msec
OK
_c0
143199253
Time taken: 158.627 seconds
=== Snappy compressed Sequence Files ===
I recently got SequenceFileRecordWriterProvider.java merged upstream in LinkedIn's Camus. Using this rather than StringRecordWriterProvider.java writes out the same data as Snappy compressed Hadoop Sequence Files.
==== Size ====
JSON data imported for the same 8 hour period and Snappy compressed was 21.9 GB, 24% of the original size.
hdfs dfs -du -s -h /user/otto/data/compressed/webrequest_mobile/hourly/2014/01/07/{00..08}
2.5 G data/compressed/webrequest_mobile/hourly/2014/01/07/00
2.6 G data/compressed/webrequest_mobile/hourly/2014/01/07/01
2.7 G data/compressed/webrequest_mobile/hourly/2014/01/07/02
2.8 G data/compressed/webrequest_mobile/hourly/2014/01/07/03
2.5 G data/compressed/webrequest_mobile/hourly/2014/01/07/04
2.4 G data/compressed/webrequest_mobile/hourly/2014/01/07/05
2.2 G data/compressed/webrequest_mobile/hourly/2014/01/07/06
2.1 G data/compressed/webrequest_mobile/hourly/2014/01/07/07
2.1 G data/compressed/webrequest_mobile/hourly/2014/01/07/08
21.9 G
==== Query Time ====
The same select count(*) query on a single hour of compressed data took 86.232 seconds, about twice as long as on uncompressed data. Running the query on 8 hours worth of compressed data took 158.627 seconds, which is only 8% longer than when run on 8 hours of uncompressed data. The number of mappers launched was the same as in the uncompressed case.
select count(*) from webrequest_mobile where year=2014 and month=01 and day=07 and hour=00;
...
MapReduce Total cumulative CPU time: 16 minutes 11 seconds 670 msec
Ended Job = job_1387838787660_0365
MapReduce Jobs Launched:
Job 0: Map: 42 Reduce: 1 Cumulative CPU: 971.67 sec HDFS Read: 11422909543 HDFS Write: 9 SUCCESS
Total MapReduce CPU Time Spent: 16 minutes 11 seconds 670 msec
OK
_c0
16641115
Time taken: 44.276 seconds
select count(*) from webrequest_mobile where year=2014 and month=01 and day=07 and hour between 00 and 08;
...
MapReduce Total cumulative CPU time: 0 days 4 hours 16 minutes 12 seconds 420 msec
Ended Job = job_1387838787660_0363
MapReduce Jobs Launched:
Job 0: Map: 343 Reduce: 1 Cumulative CPU: 15372.42 sec HDFS Read: 98055786272 HDFS Write: 10 SUCCESS
Total MapReduce CPU Time Spent: 0 days 4 hours 16 minutes 12 seconds 420 msec
OK
_c0
143199253
Time taken: 158.627 seconds
=== Summary ===
Using Snappy to compress the JSON webrequest logs results in significant space savings, and only a slight reduction in performance for large queries. Query performance is affected for smaller data sets. I will run another test once I have more data to compare (a month), but if results are approximately the same I will not update this page. 
Recommendation: use snappy compression for all webrequest imports.
Category:Hive
Category:Data platform
Category:Data platform systems