When only a document name is given, it can be either a fully qualified path or a relative path. Once we have a fully-specified URL for a hyperlink, we can add it to our queue of URLs to be scanned. One concern that crops up is how to limit our search to a given subset of the Internet. We could also extend the program to specify a set of legal hosts, allowing a small group of servers to be searched for content. Another issue that arises when handling the links we've found is how to prevent the spider from going in circles.
Circular hyperlinks are very common on the Web.
If we point our spider at page A, it finds the link to B and checks it out. On B it finds a link to A and checks it out. This loop continues indefinitely. The easiest way to avoid getting trapped in a loop is to keep track of where the spider has been and ensure that it doesn't return. The spider program doesn't remove the URL from the queue.