Fixed an issue where not all pages were added as out-links.

Because of a recent commit implementing robots.txt functionality, when parsing
an HTML page to add its out links, out links that have already been encountered
were not added to the out links list. Correct behavior is to add the page to
the list, but not add it to the crawl queue (should now be working).