Added the removal of duplicates from the links