added code to the crawler to get rid of the duplicates in all of the lists.