Only loop through the matching objects

This doesn't increase performance at all because we are now looping
through the set twice, but this is setting the stage for a further
change where we track progress by count of bytes instead of count of
files.