Only clean up manifest for files that were actually removed, closes #1440