Patent ID: 8024774

Claim:
A method for configuring a policy management protocol for a web crawler, the method comprising: determining a web space that is to be crawled by a web crawler, wherein the web space is comprised of at least one IP address and at least one range of IP addresses; storing the at least one IP address and the at least one range of IP addresses within a URL frontier; configuring the web crawler to crawl the at least one IP address or at least one range of IP addresses, and determine additional hostnames that are associated with the at least one IP address or at least one range of IP addresses; and performing a web crawling function upon the determined the at least one IP address, the at least one range of IP addresses, and the determined additional hostnames by the web crawler; wherein performing web crawling includes evaluating a link in a crawled website to determine if the link belongs to the web space, the evaluating including: determining that the host identification segment of the link is an IP address and adding the link to the URL frontier if the link IP address matches the at least one IP address or at least one range of IP addresses of the web space; determining that the host identification segment of the link is not an IP address and adding the link to the URL frontier if link fulfills host/prefix rules for the web space; determining that the host identification segment of the link is not an IP address; determining that the link does not fulfill host/prefix rules for the web space; performing a DNS lookup operation to obtain an IP address for the link; and adding the link to the URL frontier if the link IP address matches the at least one IP address or at least one range of IP addresses of the web space.