Make the logic of which files to skip simpler

... and make it clear why they're being skipped.