t.co urls get regenerated at the drop of a hat. Ignore them when de-duping
add multithreading support