emk/rust-cld2 — pull request #1 by lifthrasiir (2014-11-21 04:29:36+00:00)
Title: Add cld2-sys/prepackage.sh which cuts cld2 to fit within 10MB.

This commit should cut `cld2-sys-*.crate` down to about 6MB. You have to call `./prepackage.sh` every time `cld2-sys` gets updated, but well, at least we can work around rust-lang/crates.io#40.

Observations: First, apparently we are not using all available tables (e.g. `cld2_generated_quad0720.cc`, one of the largest unused file). Second, most tables have informational comments interspersed with actual data, which is very bad at its own *and* for the compression (brings lots of excess information entropy to code, missed opportunities due to the effectively narrowed window size, etc.). `prepackage.sh` should eliminate both.

Future direction: I've also tested a simple packer and unpacker that will turn the textual tables into the binary form (while removing comments). This does *not* fit in 10MB but still is promising, the best I can get was about 13MB. This does not remove any tables. If you are interested in this (and if crates.io abandons the strict size limit :) I can write the robust code to do this.