trickortweak/flickr-uploader — pull request #38 by davidjb (2014-12-07 23:35:08+00:00)
Title: Improve efficiency of upload when not checking changes

This pull request greatly improves efficiency of the script when MANAGE_CHANGES is False. For large sets of media, this can represent the difference between running in seconds versus minutes, hours or longer.

The script currently enumerates all files (``self.grabNewFiles()`` returns all files) and then runs ``self.uploadFile`` on each path. In ``self.uploadFile``, the database is loaded & queried, and ``MANAGE_CHANGES`` is checked. With ``MANAGE_CHANGES`` set to False, this is a noop for existing images -- lots of processing happened to do ignore said file. For a set of 2000 images with 5 new files, this means this (expensive) work is called 2000 times, once for each path.

This PR runs the code for only new files - eg, the ones not yet in the database. It computes the set difference between the complete list of files from ``self.grabNewFiles`` and the list of paths from the database. Then, it runs the upload on only these paths. The result, using the example above, is the code only has the essential 5 calls required to upload the new files and ignores the rest.

If ``MANAGE_CHANGES`` is ``True``, the code carries on as it originally was, as it needs to check each and every image for hash changes.