pulp/pulp — pull request #484 by jortel (2013-05-21 19:06:43+00:00)
Title: 960159 - nodes memory consumption when publishing large repositories

https://bugzilla.redhat.com/show_bug.cgi?id=960159

Primary changes are:
- Custom query used to fetch units to publish instead of using the get_units() in the conduit. This query is also used by the importer to fetch the content units already on the child node.
- Manifest redesigned to provide access to downloaded content units via iterator instead of reading them into memory. The iterator yields a lightweight version of the content units and a unit reference. The lightweight unit is used to determine if the unit needs to be installed on the child and includes information to download the bits (associated rpm etc). After the file associated with the unit has been downloaded, the unit ref is used to fetch the complete unit (including metadata) so it can be added to the child node.
- The importer strategy flow needed to be updated to leverage changes in the unit inventory.
- The unit inventory needed to be updated to leverage the new manifest.
- The download listener (renamed to a manager) needed to be updated to round-trip the unit reference and for slight api changes.
- The file compression factored out to separate module with minor improvements. A "while I was there, thing".
- Misc conversion of try: finally: to "with" statement just because I was there and "with" statements are much simpler (and they rock).

Some of the diffs are pretty confusing even to me. For those files, I'd suggest just looking at the file directly. They're all pretty small.