Always restrict changes to files we care about.

The main reason I wasn't doing this was because of not wanting to code
the extra step to have it work with new files and/or deleted files.
That wasn't as bad to solve as I thought it might be, so here we are :)