change deduping logic to be first come first
server, but site rank trumps. fixed bug from
fix before.