wrote new cleaned datafile. This is significantly smaller than the last, around 29k lines. Duplicates removed, may be able to be visually cleaned up and added back in at a later date, but to automate would make many assumptions.