check and ignore duplicated links using hash function