Ensure that we remove the old file when we rename.