Resolved relative URLs to Absolute URLs from crawled links and identified
unique URLs by removing the duplicates