pulp/pulp — pull request #660 by jortel (2013-10-11 22:48:44+00:00)
Title: 1017924 - unzip the units.json instead of reading/seeking using gzip.

The main problem:

The unit.json.gz is a zipped file containing json objects. For RHEL-6 repositories w/ 13,000 packages, this file compressed is ~80 MB and uncompressed it is ~ 820 MB. To keep the memory footprint from being too large, the file is processed as follows:

1. opened, read completely (one json object at a time) - just grabbing all of the unit keys.
2. opened, read byte-range to fetch details for a unit that needs to be added/updated.

Prior to this PR, the file was downloaded and remained zipped. So, the operations listed above were performed on a zipped file. Which means, each subsequent time we needed to fetch an entire unit (to be added), we were incurring the decompression overhead. The true cost of this was undetectable in smaller repositories (~5000). On really large files 80 MB compressed, it was taking ~2-3 seconds to seek into the file beyond 30%. We were eating this for each unit to be added. 2-3 seconds x = long-time.

This fix decompresses the file prior to operations 1 & 2. Seeks reduced from 2-3 seconds to ~1ms.

ADDITIONAL:

To support renaming the downloaded file, the manifest structure needed to be updated. The manifest has a version attribute to detect changes to the structure. The long term intent was to provide migration so users did not have to re-publish the manifest if the version needed to change. Hence, the migration stuff in this PR.