Patent ID: 7676553

Claim:
A system that facilitates incremental web crawls comprising the following components stored in computer memory and executable by a processor: a selecting component that selects a first chunk from a group that includes at least one index chunk that stores information associated with an index of items, at least one rank chunk that stores at least one static rank associated with the at least one index chunk, at least one content chunk that stores cached copies of contents of pages crawled, at least one re-crawl chunk that stores a list of Uniform Resource Locators to be re-crawled, and at least one webmap chunk that stores at least a portion of a web map used for calculating the at least one static rank; a chunk map that stores properties associated with the first chunk and that is employed to determine the first chunk based on the stored properties, and wherein the stored properties include average time between change and average importance and are shared by all items in the first chunk; a parsing component that parses the first chunk for Uniform Resource Locators; a crawling component that re-crawls the Uniform Resource Locators; a receiving component that (1) receives one or more documents as a result of re-crawling the uniform resource locators, (2) stores a document from the one or more documents in an appropriate chunk when it is determined that the document belongs to the appropriate chunk, and (3) forms a second chunk separate from the first chunk based, at least in part, upon the re-crawled uniform resource locators when it is determined that the document does not belong to any chunk from the group, and storing the document in the second chunk.