move duplicate handling stuff to lib.