Patent ID: 8117536

Claim:
A system for controlling downloading web pages, the system comprising: a setting module configured for setting parameters for controlling downloading of the web pages, the parameters comprising a current web page, selected links of the current web page to examine, and a date posted range of the web pages pointed to by the selected links; a parsing module configured for parsing the current web page to generate information about the web pages pointed to by the selected links; an obtaining module configured for obtaining information about each of the web pages pointed to by the selected links, wherein the information of each of the web pages pointed to by the selected links comprises a posted date; a determining module configured for determining if the web pages pointed to by the selected links have been downloaded previously, and if the posted date of web pages pointed to by the selected links is within the date posted range; a generating module configured for generating script commands for downloading the web pages pointed to by the selected links upon the condition that the web pages pointed to by the selected links have not been downloaded and the posted date of the web page pointed to by the selected link is within the date posted range; a counting module configured for counting the number of the selected links by the setting module and the number of script commands generated by the generating module; a downloading module configured for downloading the web pages pointed to by the selected links in a database using the script commands along with the information of each of the web pages pointed to by the selected links; a next page module configured for switching to a next page upon the condition that the current web page comprises two or more pages and the number of the script commands equals the number of the selected links; and at least one processor executing the setting module, the parsing module, the counting module, the determining module, the obtaining module, the generating module, the downloading module, and the next page module.