replaced custom multiset with stl multiset
added checking for duplicate links introduced through redirects