sort and uniq urls after generating them