Remove the files/folders that are supposed to be ignored