pulp/pulp — pull request #551 by jortel (2013-08-02 17:27:49+00:00)
Title: Nodes manifest caching.

The *nodes* distributor publishes a manifest which is a small json file that describes the data that is published. A little like a yum repomd file. Also published is an units.json.gz file containing json describing all of the content units associated with the repository.

During the node sync, the manifest and associated units file is downloaded and used to synchronize the units in the repository on the child node. During the sync of an entire node, this data is downloaded regardless of whether it changed or not. This story (PR) is to download the manifest (very small) and compare to the previously downloaded manifest (if it exits) and decided whether downloading the unit data is needed. Basically if the ID in the manifest matches the manifest already downloaded, then the repository has not been re-published and there is no need to download the units. The one we have is sufficient.

While in the process of adding this functionality, I needed to make some small modifications to the classes in the manifest module. After taking a 2nd look, I decided there were other improvements I wanted to make. Such as minor refactoring of the Manifest classes (The diff makes it look more extensive than it really is); Getting rid of the compression module and the practice of compressing entire files in favor or compressing and decompressing as the files are written/read.