amanjain91/bismark-data-server — pull request #1 by sburnett (2013-02-23 20:37:55+00:00)
Title: Robustness for tar and upload scripts

I've improved the robustness of the tar and upload scripts so they work even when there are many files. I also implemented a few of the suggestions I made by email a couple of days ago.

1. Call `tar` in append mode (rather than create mode), because `find` might call `tar` multiple times if there are many files (thousands).
2. Fetch results from `find` using a `while` loop, since `for` can only accept a limited number of parameters.
3. Create per-day directories for tarballs and traces. This will improve performance, particularly for extracted versions of the tarballs.
4. Compress the tarballs.
5. Canonicalize the directory structure in the tarballs.

Please let me know if you have questions. I haven't tested this on AWS, so sorry if there are dumb bugs / typos.