saltstack/salt — pull request #16325 by thusoy (2014-10-02 14:21:33+00:00)
Title: Use salt.utils.get_hash for hashing

Prevents memory errors for large files since get_hash uses chunked
hashing.

This is similar to https://github.com/saltstack/salt/issues/14784, but this is all the other places I found where custom hashing is used, didn't check for others at the time of the old issue.

One thing: We use comparing digests several places for detecting changes to files, which is quite inefficient for large files, since it requires reading the entirety of both files, hashing them both, and then compare, instead of simply iterating over both in chunks and stop on the first differing byte. This is the approach used by the [filecmp.cmp](https://docs.python.org/2/library/filecmp.html#filecmp.cmp) f. ex. Any thoughts on this?