Improved crawling logic when searching for crawling a page for links.

README updates