Correctly remove file for old dists