remove duplicates before determining the selection function
rmdups at the beginning of the file and create the duplicate-free file
rmdups before doing anything else and cache the result