Only keep the files we changed