using list(candidates) really won't scale.  the length of candidates can be in the many millions. 

the splitting function, f, will need to be a generator. look at scoreDuplicates for how we do this now.