Abstract:
A data collection and retrieval system and associated method allow the capture and replication of data presented at various Web pages into a database application through text parsing of the HTML source code of that document. The system allows the user to select one or more Web pages containing data of interest; to specify exactly which data within any page is to be captured; to specify how frequently data is to be collected; and to specify the conditions for collection and retrieval. The advantage of the system is realized through efficient, automated data collection that would otherwise be impractical. The system includes an initialization stage and an automatic execution stage. The initialization stage provides the user interface which allows the user to select the source file that contains data the user wishes to copy, target database that will receive the data, and timing criteria for automatic transfer of data. The automatic execution stage automatically transfers the data from the source file to the target database as instructed by the user in the initialization stage.

Description:
FIELD OF THE INVENTION  
         [0001]    The present invention generally relates to the field of data processing, and particularly to a software system and associated method for use with Web pages to transfer data from a Web page source file to a target database. More specifically, this invention pertains to a method for parsing a Web page source file, collecting the specified data on the Web page, automatically retrieving the specified data on a predetermined frequency or scheduled time, and then storing the collected data in a database structure.  
         BACKGROUND OF THE INVENTION  
         [0002]    The World Wide Web (WWW) is comprised of an expansive network of interconnected computers upon which businesses, governments, groups, and individuals throughout the world maintain inter-linked computer files known as Web pages. Users navigate these pages by means of computer software programs commonly known as Internet browsers. Web sites and Web pages found on the WWW present a wide range of data to users in varying formats. This data, such as weather information or stock quotes, may be useful in a variety of ways to the Internet user beyond simply viewing that data in the form and location as it is displayed on the Internet.  
           [0003]    Transferring data found on websites to a database format would allow significant data management and manipulation capabilities afforded through standard database software. However, there are currently no effective methods for capturing and replicating that data within a database application. The user can always manually transfer the data viewed on the Internet into a database. However, when transfer is done on a regular basis, such as might be done for a regular recording of stock prices, the transfer of data from the website to the user&#39;s database becomes inefficient, labor intensive, and prone to transcription errors. This is particularly true for large amounts of data on a large variety of Web pages that are retrieved periodically, such as every 10 minutes, 45 minutes, daily, etc.  
           [0004]    Current systems exist for transferring Internet data to databases for archiving Web content in the form of a source file and visual representation, allowing subsequent viewing of that data at a later date. While allowing the user to capture data and archive it, these systems do not provide a mechanism for parsing individual data elements, replicating that data in a relational database format, or automatic data collection in a manner specified by the user.  
           [0005]    Another system described in U.S. Pat. No. 6,078,924 to Ainsbury et. al. retrieves data from a wide range of document formats and converts that data into a common format and location. This system includes the capability to collect data automatically in a manner specified by the user. It also allows the retrieval of data from a wide range of data formats, including the capability to retrieve data from HTML documents. However, while the data is collected and replicated in a central format and location such as a spreadsheet, it does not effectively account for issues of relational database replication.  
           [0006]    Thus, there is need for a data collection and retrieval system and method to efficiently manipulate and analyze data displayed on websites. The system and method should be able to retrieve Internet data automatically and to assign that data to appropriate places in a database structure defined by the user. For data that changes regularly such as stock quotes or weather data, a system is desired which will automatically retrieve data based on a user defined schedule. The need for such a system and associated method has heretofore remained unsatisfied.  
         SUMMARY OF THE INVENTION  
         [0007]    The system and method for data collection and retrieval satisfy this need, and allow the capture and replication of data presented at various Web pages into a database application through text parsing of, for example, the HTML source code of the retrieved documents. Consequently, users may efficiently collect a wide range of data from a wide variety of sources, and have that data available in user-specified applications, for further data management and processing.  
           [0008]    The data collection and retrieval system of the current invention allows the user to select one or more Web pages containing data of interest; to specify exactly which data within any page is to be captured; to specify how frequently data is to be collected; and to specify the conditions for collection and retrieval. The conditions for replication include data formatting and database parameters. The advantage of the present system and method is realized through efficient, automated data collection that would otherwise be impractical.  
           [0009]    The present system and method generally comprise an initialization stage and an automatic execution stage. The initialization stage provides the user interface which allows the user to select:  
           [0010]    the source file that contains data the user wishes to copy;  
           [0011]    the target database that will receive the data; and  
           [0012]    the timing criteria for automatic transfer of data.  
           [0013]    The automatic execution stage automatically transfers the data from the HTML source file to the target database as instructed by the user in the initialization stage.  
       
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0014]    The various features of the present invention and the manner of attaining them will be described in greater detail with reference to the following description, claims, and drawings, wherein reference numerals are reused, where appropriate, to indicate a correspondence between the referenced items, and wherein:  
         [0015]    [0015]FIG. 1 is a schematic illustration of an exemplary operating environment in which a data collection and retrieval system of the present invention can be used;  
         [0016]    [0016]FIG. 2 is a schematic illustration further showing the manner in which the data collection and retrieval system of the present invention accesses data residing on various Web pages;  
         [0017]    [0017]FIG. 3 is a block diagram that illustrates a high level architecture of the system of FIG. 1 shown used in the context of the World Wide Web;  
         [0018]    [0018]FIG. 4 is a flow chart that depicts the operation of the initialization stage of the system of FIGS. 1 and 3,  
         [0019]    [0019]FIG. 5 is comprised of FIGS. 5A and 5B, and represents a flow chart that depicts the operation of the automatic execution stage of the system of FIGS. 1 and 3; and  
         [0020]    [0020]FIG. 6 is a screen shot of a Web page with data that can be collected and retrieved by the system of FIG. 1.  
     
    
     DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS  
       [0021]    The following definitions and explanations provide background information pertaining to the technical field of the present invention, and are intended to facilitate the understanding of the present invention without limiting its scope:  
         [0022]    Database: The term “database” refers, for example, to a database, a data storage device, or a memory.  
         [0023]    GUI: Graphical User Interface, a program interface that takes advantage of the computer&#39;s graphics capabilities to make the program easier to use.  
         [0024]    HTML (Hypertext Markup Language): A standard language for attaching presentation and linking attributes to informational content within documents. During a document authoring stage, HTML “tags” are embedded within the informational content of the document. When the Web document (or “HTML document”) is subsequently transmitted by a Web server to a Web browser, the tags are interpreted by the browser and used to parse and display the document. In addition to specifying how the Web browser is to display the document, HTML tags can be used to create hyperlinks to other Web documents.  
         [0025]    Internet: A collection of interconnected public and private computer networks that are linked together with routers by a set of stand protocols to form a global distributed network.  
         [0026]    Schema: A database design, comprised of tables with columns, indexes, constraints, and relationships to other tables. The column specification includes a data type and related parameters such as the precision of a decimal or floating-point number.  
         [0027]    Server: A software program or a computer that responds to requests from a Web browser by returning (“serving”) Web documents.  
         [0028]    URL (Uniform Resource Locator): A unique address that fully specifies the location of a content object on the Internet. The general format of a URL is protocol://server-address/path/filename.  
         [0029]    Web browser: A software program that allows users to request and read hypertext documents. The browser gives some means of viewing the contents of Web documents and of navigating from one document to another.  
         [0030]    Web Site: A database or other collection of inter-linked hypertext documents (“Web documents” or “Web pages”) and associated data entities, which is accessible via a computer network, and which forms part of a larger, distributed informational system such as the WWW. In general, a Web site corresponds to a particular Internet domain name, and includes the content of a particular organization. Other types of Web sites may include, for example, a hypertext database of a corporate “intranet” (i.e., an internal network which uses standard Internet protocols), or a site of a hypertext system that uses document retrieval protocols other than those of the WWW.  
         [0031]    World Wide Web (WWW): An Internet user—server hypertext distributed information retrieval system.  
         [0032]    [0032]FIG. 1 portrays the overall environment in which a data collection and retrieval system  10  according to the present invention may be used. The system  10  includes a software or computer program product that is typically embedded within, or installed on a host server (or a client server or machine)  15 . Alternatively, the system  10  can be saved on a suitable storage medium such as a diskette, a CD, a hard drive, or like devices. While the system  10  will be described in connection with the WWW, the system  10  can be used with a stand-alone database of terms that may have been derived from the WWW or other sources.  
         [0033]    The cloud-like communication network  20  is comprised of communication lines and switches connecting servers such as servers  25 ,  27 , to gateways such as gateway  30 . The servers  25 ,  27  and the gateway  30  provide the communication access to the Internet. Users, such as remote Internet users, are represented by a variety of computers such as computers  35 ,  37 ,  39 , and can query the host server  15  for desired information through the communication network  20 .  
         [0034]    The server  15  is connected to the network  20  via a communications link  42  such as a telephone, cable, or satellite link. The servers  25 ,  27  can be connected via high-speed Internet network lines  44 ,  46  to other computers and gateways. The servers  25 ,  27  provide access to stored information such as hypertext or Web documents indicated generally at  50 ,  55 , and  60 . The hypertext documents  50 ,  55 ,  60  most likely include embedded hypertext link to other locally stored pages, and hypertext links  70 ,  72 ,  74 ,  76  to other webs sites or documents  55 ,  60  that are stored by various Web servers such as the server  27 .  
         [0035]    Typically, the system  10  accesses Web pages that have been selected by the user. These Web pages are the source of the data to be transferred to a user defined database. As an example, in FIG. 2 the data collection and retrieval system  10  connects through the communication network  20  to an Internet portal  205  such as Yahoo!®. The user selects a subsite  210  on Yahoo!®, and continues selecting subsites as shown by sub-subsite  215 , until a webpage such as webpage  220  is found that contains the data in which the user is interested to transcribe (or copy) to a database or storage  370 .  
         [0036]    An exemplary high-level architecture of the data collection and retrieval system  10  is shown in FIG. 3. System  10  includes an initialization stage  305  and an automatic execution stage  310 . The initialization stage  305  provides the user interface that allows the user to select a source Web page, i.e.,  220 , the target database, i.e.,  370 , that will receive and store the data, and the timing criteria for the automatic transfer of data.  
         [0037]    The automatic execution stage  310  automatically transfers the data from the HTML source file  320  for the Web page  220 , to the target database  370  as instructed by the user in the initialization stage  305 . The initialization stage  305  includes a user input GUI  315 , a source file  320 , the parser  325 , an array file  330 , a refined user GUI  340 , and an array locations file  345 . The automatic execution stage  310  includes a timer  350 , a defined source file  355 , an array file  360 , a data retrieval module  365 , and the target database  370 .  
         [0038]    In operation, and with further reference to the operational flowcharts of FIGS. 4 and 5, the user opens system  10  in step  405 , and the system  10  displays the user input GUI  315 . In step  410 , the user specifies the URL for an HTML page  220  that contains data to be transcribed to the target database  370 . In so doing, the user selects the Web page  220  as shown by FIG. 2, and saves the Web page URL in system  10 . The user may select the URL of the source file ( 320 ) by manually selecting the Web page URL and copying it to a specific field. Alternatively, the user may open the URL for a source file  320  and automatically save the address to a “favorites” list in a designated list, field, drop-down menu, or list box.  
         [0039]    System  10  then downloads the HTML source file  320  for the Web page  220  in step  415 . In step  420  the parser  415  processes the source file  320 , tokenizes the HTML format of the Web page  220  and places it in a data array  330 . Ideally, the parser  325  divides the data in the source into identifiable units of the Web page  220 , exactly replicating the Web page  220  and displaying the replicated Web page to the user in step  425  as the refined user input GUI  340 . The purpose of the parsing is to isolate potentially usable data and to describe how to subsequently locate this data in the Web page source. The system  10  presents a possible schema for storing the selected data in a relational database. This may be based on existing tables in the database, or it may be a default schema constructed to match the selected data. If needed, the system  10  creates the database schema and automatically transfers the data to the database, performing data type conversions as necessary.  
         [0040]    Preferably, but not necessarily, the user has access only to data that is embedded in the HTML source file  320  of the replicated Web page. For example, the user might select a Web page  220  containing stock quotes for a variety of stocks. From Web page  220 , the user can only select quotes that are displayed and part of the HTML source file  320 .  
         [0041]    The user then selects data and or categories of interest in step  430 . In step  435 , system  10  saves the sequence of data or categories selected by the user to an array locations file  345 . System  10  remembers the location of the data in the array  330  and retrieves data from the same locations on each successive data transfer.  
         [0042]    In step  440 , the user sets the timer  350  by selecting the frequency of data retrieval; i.e., every 5 minutes, 15 minutes, at a specific time such as 12:00 p.m., etc. The selection frequencies can be stored along with other control information, in database tables. Examples of such other control information include URLs, data arrays, user-selected data element locations and formats, and locations for storing the data in the database. The user then specifies in step  445  where to store the retrieved data, such as database  370 . The retrieved data may be stored in a specific database, the user may specify specific fields and records in the database, and the user may also specify the format the data should assume after it has been transferred.  
         [0043]    The initialization stage  305  is complete once the user has specified the source file  320 , the desired data or categories for retrieval, the destination for the retrieved information, database  370 , and the frequency of operation. System  10  can now perform the automatic retrieval of data as illustrated by the process flow charts of FIGS. 5A and 5B.  
         [0044]    At a time specified by the user, the timer  350  automatically begins the data retrieval process in step  505  and accesses the Web page  220  in step  510 . Each time system  10  accesses the Web page  220 , a new source file  355  is created to reflect the changing data on the Web page  220 . The parser  325  parses the source file  355  in step  515  and creates an array file  360  in which the data is accessible to the data retrieval module  365 .  
         [0045]    The data retrieval module  365  extracts the data from the array  360  in step  520  and places it in step  525  in the database  370  as previously specified by the user. This process is repeated by the timer  350  at the frequency or times previously specified by the user in step  440  of FIG. 4.  
         [0046]    Optionally, after system  10  retrieves selected data from the source file  355  in step  520 , system  10  may map the data previously stored in database  370  onto the new array  355  retrieved from the source file  320  in step  530 . If in step  535  the stored array in database  370  matches array  360 , system  10  continues to step  540  and transfers the data in array  320  to the appropriate locations in database  370 . Operational control then transfers back to the timer  350  in step  505  until the next specified time occurs.  
         [0047]    If, however, the two arrays do not match in step  535 , system  10  automatically attempts to adjust the new array  360  to match the array stored in database  370  in step  545 . If the arrays match in step  550 , system  10  proceeds, as before, to step  540 , stores the data in database  370 , and returns control to the timer in step  505 . If the new array  360  does not match the array stored in database  370 , system  10  notifies the user in step  555 , and stops the data retrieval process in step  560  until the user approves the adjustment or reconfigures the retrieval process.  
         [0048]    An example of an exemplary webpage  220  from which a user wishes to transcribe data is shown in FIG. 6. The data the user wants to collect and retrieve is shown, for instance, at locations  605  and  610 . The user may specify that system  10  retrieve just the data, or the data and categories describing the data (i.e., metadata). At location  605 , the user selects both the category and data as shown in Table 1 below.  
                             TABLE 1                           Categories and data location 605                Category   Data                       Stock Symbol   IBM           Last Trade   9:43 am               84.90           Change   +0.90               +1.07           Volume   451,200                      
 
         [0049]    The value for price and time at location  605  does not have a complete descriptive category, so the user provides to system  10  the desired category “Last Trade Time” and “Last Trade Value” for the values listed under “Last Trade”. Similarly, the user assigns new categories “change Value” and “Change Percent” to differentiate the two values found under the category “Change” at location  605 . The user also wishes to select the date and time this transaction took place. This data is found at location  610 , but has no associated categories. The user selects the value “May 23, 2002” and assigns the category “Date” and selects “10:03 am ET” and assigns the category “Transfer Time”.  
         [0050]    [0050] 
         [0051]    The data as stored in the target database  370  is shown in Table 2 below.  
                             TABLE 2                           Categories and data stored in the target database 370.                Category   Data                       Stock Symbol   IBM           Last Trade Time   9:43 am           Last Trade Value   84.90           Change Value   +0.90           Change Percent   +1.07%           Volume   451,200           Date   May 23 2002           Transfer Time   10:03 am ET                      
 
         [0052]    The user then specifies the times data should be transferred from the Web page to the database: 12 p.m., 3 p.m., and 5 p.m. eastern standard time. System  10  then sets the timer  350  to initiate data collection and retrieval at those times, transferring the data from Web page  220  to database  370  at those times, in the configuration specified by the user.  
         [0053]    It is to be understood that the specific embodiments of the invention that have been described are merely illustrative of certain application of the principle of the present invention. Numerous modifications may be made to the data collection and retrieval system invention described herein without departing from the spirit and scope of the present invention. Moreover, while the present invention is described for illustration purpose only in relation to the WWW, it should be clear that the invention is applicable as well to any collection of data or databases accessible either through an Internet or Intranet connection.