Only change permissions on files that need it in.