handle duplicates of same urls