Abstract:
An integrated search and information discovery system is disclosed. The simultaneous and integrated access to a dynamic plurality of arbitrary search services and data stores is enabled, relieving users of such services and stores from the time-consuming task of accessing them individually or in otherwise inefficient manners. Further, a user-oriented derivative of the common webcrawling process is introduced and utilized to discover information not held in or indexed by the accessed search services and data stores using content and links delivered by those search services and data stores in response to an integrated user query. Finally, a modular information analysis framework is utilized to allow for the use of a plurality of information analysis methods depending on the needs of a user.

Description:
CROSS REFERENCE TO RELATED APPLICATION  
       [0001]    This application claims the priority benefits of copending U.S. Provisional Application No. 60/307,261, filed on Jul. 23, 2001. 
     
    
     
       STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT  
         [0002]    Not Applicable  
         FIELD OF THE INVENTION  
         [0003]    This invention relates to the field of search and information retrieval. Specifically, the present invention relates to a process and system that enables a user: (a) to dynamically integrate arbitrary types of search services and data stores into a search for simultaneous access and querying, and (b) to apply webcrawling techniques and processes to a restricted set of qualified content so as to avoid common pitfalls when working with current search technologies.  
         BACKGROUND OF THE INVENTION  
         [0004]    Information may be stored and distributed in many diverse ways given the current proliferation of electronic computing devices and ways in which to network and connect those devices so that information may be transmitted between them. These connected collections of electronic devices often contain vast stores of information, usually organized into discrete documents. (The World Wide Web is one example of a connected collection of electronic devices; your personal computer system is another, with its connected set of storage and processing subsystems.) Understandably, the users of these electronic devices often wish to find a particular document or a set of documents that match a given set of criteria. Searching for specific information in this way may be accomplished using any of the hundreds of search utilities, search engines, indexing and database tools, or browsing utilities available today. All of these approaches (which are hereafter collectively referred to as search services and data stores) share a common set of technical and usage characteristics that must be understood prior to considering the current invention&#39;s approach.  
           [0005]    The methods and processes currently available for performing non-trivial information searches are largely identical in scope, construction, and strategy. First, a search service or other data store must gather a collection of information; this may occur in one step (especially with smaller, well-defined collections), or continuously over time (if the collection is particularly large or difficult to analyze—the World Wide Web is a good example of this). This is the most critical step in the entire process, in that it defines the scope within which any searches over the gathered collection must operate. To illustrate this, consider a collection of information that contains nothing about India; any queries about India made to a service based on that collection will immediately fail. share a common set of technical and usage characteristics that must be understood prior to considering the current invention&#39;s approach.  
           [0006]    The methods and processes currently available for performing non-trivial information searches are largely identical in scope, construction, and strategy. First, a search service or other data store must gather a collection of information; this may occur in one step (especially with smaller, well-defined collections), or continuously over time (if the collection is particularly large or difficult to analyze—the World Wide Web is a good example of this). This is the most critical step in the entire process, in that it defines the scope within which any searches over the gathered collection must operate. To illustrate this, consider a collection of information that contains nothing about India; any queries about India made to a service based on that collection will immediately fail.  
           [0007]    In the case of the World Wide Web, which is possibly the largest collection of information, building a collection of information is almost always done by employing some sort of webcrawling process. This process begins with a small sample of documents from the web that contain links, bits of meta-data that describe the location of other documents that are usually related or associated with the document containing the links. The webcrawling process attempts to follow every link that exists within those documents to find new documents, repeating the same process for the set of new documents. This variety of webcrawling, by far the most widespread, is monolithic in its operation: in general, it does not attempt to determine whether a particular document is “worth” adding to the collection being built, because the process cannot have any parameters describing what is “worthwhile”. After all, the process does not know what users will be searching the gathered collection for.  
           [0008]    (It is possible to qualify or disqualify documents when building a collection of information, but doing so must restrict the collection to those documents clearly related to a specific topic of interest thereby minimizing the scope of the collection dramatically.)  
           [0009]    This collection of information is then analyzed and indexed. The indexing process involves taking a “snapshot” of the structure of each item in the collection, and saving the plurality of snapshots into a database where they may be accessed rapidly. Once an index is built, the search service or other data store must simply provide an interface to it so users may query the index.  
           [0010]    A variation on prototypical search services is manifested in the concept of a metasearch engine. A typical metasearch engine does not build or maintain an index; rather, it acts as a front-end to a plurality of search services or data stores, allowing a user query to be distributed to that plurality, with applicable results from that group returned to the user. Metasearch engines are nearly uniform in that the set of search services and data stores that they operate over is static, at least from the user&#39;s perspective.  
           [0011]    Regardless of what sort of search service or data store is used to locate information, the process from a user&#39;s perspective is essentially identical from one instance to another. First, he or she submits a query to a search service or data store, which, after consulting its index(es), returns a set of results containing links to information that is supposed to be qualified with regard to the user&#39;s query. Then the user must manually activate or open each link and determine the true level of qualification of each link&#39;s associated document(s). Finally, the user must manually (and for an indefinite period of time) follow additional links held in the found documents in order to either discover additional qualified documents that were not returned by the search service or data store, or to discover documents that are qualified to a greater degree.  
         STATEMENT OF SHORTCOMINGS OF PRIOR ART  
         [0012]    Two fundamental shortcomings affect all methods and processes designed to allow a user to find qualified information efficiently. The first is that virtually all of those methods and processes rely on querying essentially static databases that index content that is located (either spatially, logically, or topically) where the indexing algorithm believes that qualified information may exist. This is ideal for sets of static content, but as the influence of the Internet and other networking technologies grows, so does the tendency for content to be dynamic, fluid, everchanging in both form and substance. Put simply, indexing algorithms and current (and foreseeable) database technology cannot keep pace with the rate of flux that occurs in certain information collections; the World Wide Web and the Internet as a whole is the best example of this phenomenon currently, but it is reasonable to expect that as the pervasiveness of networking technologies expands and accelerates, other information collections that aren&#39;t necessarily associated with the Internet will become similarly difficult to track and catalogue using current and foreseeable indexing and database technology. Based on current growth and flux trends observed in both Internet content and in other information collections, this has been the nearly unanimous judgment of essentially every analyst that has examined the problem.  
           [0013]    To exemplify this shortcoming more concretely, one needs only to consider current web search engines. Marvels of database and indexing technology, they are nonetheless far behind in cataloging the entirety of the World Wide Web, and they are falling further behind every day: with the size of the World Wide Web estimated to be growing at a rate upwards of 500% per year and advances in database and indexing technology sure to be unable to match such velocity, search engines are forced to concentrate their activities on content that is most likely to be needed by their particular set of users in the near future. In addition, the rapid pace of change of that content means that web search engines are constantly using out of date indexes of that content: the frequency of irrelevant search results and “dead links” pointing at content that no longer exists is testimony to that fact.  
           [0014]    A secondary consequence of this first shortcoming is that because current search systems do not (and often cannot) deliver to a user links to all content that may satisfy a user query because of their inability to keep pace with the rapid flux of that content, a user is often required to engage in very time-consuming and tedious manual searching. This manual searching usually involves querying a search service, examining the content delivered by the search service in response to a user query (either directly or via indirect links), and following additional links in that content to find more qualified content that was not delivered by the search service due to indexing limitations. This process often is iterative, with the user following links to (hopefully) additional qualified content through many “levels” of such links. This is widely considered to be a productivity-draining and ineffective searching method, but one that is very necessary given the limitations of indexing and database technology in relationship to the rate of flux of content sought by users.  
           [0015]    The second fundamental shortcoming affecting current methods and processes designed to allow a user to find qualified information efficiently is best described as index Balkanization. Search services, which include web search engines, specialized subscription-based services, and other databases of all sorts, are very fragmented, preventing a user from efficiently utilizing a set of search services instead of just one or a couple. This is significant in that each search service is very unique in the content that it catalogues and provides access to; even in the realm of the World Wide Web, where every search engine potentially has access to the same set of information, there is surprisingly very little overlap in what content is examined and catalogued by those search engines. Therefore, in order to effectively search multiple stores of information, a user must manually (and at great expense in terms of time, effort, and possibly cost) access each search service in turn.  
           [0016]    Metasearch engines and services have attempted to address this problem to some extent, but their general approach is also insufficient for two reasons: (a) metasearch engines and services (in practice) query a very select and limited subset of the possible search services that the metasearch engine might have access to (which are almost always internet-based, ignoring other possible search services), and (b) no metasearch engine currently allows a user to customize and extend the engine so that it accesses a set of search services entirely of the user&#39;s choosing. This becomes a very difficult barrier when a user wishes to utilize metasearch techniques and methods to make searching some personally-chosen set of search services more efficient. An example of this might be a doctor that wishes to access with a single query a set of web search engines, the medical database PubMed, and a local database containing research data. No solution is currently available for such a need.  
           [0017]    It is clear that a new information search method must be put forward that can address these shortcomings such that users may conduct non-trivial searches over a plurality of search services and data stores and further refine and prosecute those searches in an automated way, negating the need for time-consuming manual searching.  
         SUMMARY OF THE INVENTION  
         [0018]    The current invention seeks to remedy the above shortcomings of current search methods and processes by advancing three new variations and improvements upon existing search methods.  
           [0019]    The first advance is the specification of a metasearch process that (a) is not limited to internet search services and data stores, enabling users to include diverse information collections in their searches, such as subscription information services (i.e. Lexis-Nexis, Ovid, library catalogs), private databases, or local storage devices, and (b) is customizable and extendable, enabling users to specify how to access and query the aforementioned diverse information collections.  
           [0020]    The second advance is the specification of a new webcrawling process that (a) is not limited to functioning within the confines of the World Wide Web, but rather can access documents and extract and follow links over a diverse set of communications methods connecting a diverse set of information storage mediums, and (b) is user-centric, in that it operates in real-time upon the submission of a user query, and only crawls documents that can be qualified with regard to the parameters specified in the user query.  
           [0021]    The third advance is the merging of the aforementioned metasearch and webcrawling processes into a single search and information discovery system that enables users to utilize the results and output of the metasearch process as the starting point(s) for the webcrawling process.  
           [0022]    Other features of the present invention will be apparent from the accompanying figures and from the detailed description that follows.  
           [0023]    An embodiment of the current invention has been commercialized in the form of a product called the Gemini Unified Datamining System, developed and distributed by Snowtide Informatics Systems, Inc. of South Hadley, Mass. 
       
    
    
     BRIEF DESCRIPTIONS OF THE FIGURES  
       [0024]    [0024]FIG. 1 illustrates the top-level architecture of the described embodiment of this invention.  
         [0025]    [0025]FIG. 2 illustrates the functional interaction between a user and the described embodiment of this invention, as well as the components of the functional interface between the user and and said embodiment.  
         [0026]    [0026]FIG. 3 illustrates the functionality of the Query Manager, which coordinates all processes of the described embodiment of this invention.  
         [0027]    [0027]FIG. 4 illustrates the top-level functionality of the Outside Index Query Module, a component of the described embodiment of this invention.  
         [0028]    [0028]FIG. 5 illustrates the operation of the Communications Interface and the Evaluation Module, two components of the described embodiment of this invention.  
         [0029]    [0029]FIG. 6 illustrates the operation of the Network and Crawling Module, a component of the described embodiment of this invention.  
         [0030]    [0030]FIG. 7 illustrates the operation of a critical sub-component of the Outside Index Query Module, a component of the described embodiment of this invention. 
     
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT  
       [0031]    The embodiment of an integrated search and information discovery system according to the present invention are hereinafter described in detail with reference to the accompanying figures.  
         [0032]    The terms used in the description of the preferred embodiment as well as the remainder of this disclosure are defined as follows:  
         [0033]    Link: Any reference to a body of content. Links are often found within content, thereby enabling bodies of content to cross reference other bodies of content. Common embodiments of links include (but are not limited to) World Wide Web hyperlinks and database references.  
         [0034]    Content: Any human-readable or—viewable data stored in a digital medium that often serves to communicate information in a structured form. Content includes (but is not limited to) written material as well as visual and audible material.  
         [0035]    Meta-Data: Any data that is associated with a body of content in order to describe the state, disposition, source, destination, or other structured properties of said content. Meta-data can include (but is not limited to) properties such as when a body of content was created, when it was modified, when it was transmitted, who or what authored it, how much storage space it occupies, and links to related content.  
         [0036]    User Query: Information that is manually inputted by a user that consists of parameters defining or indicating what type or form of content said user wishes to find. Additional parameters may be related to the system that processes the user query.  
         [0037]    Data Store: A static collection of data that either contains or refers to content using links. Data stores are usually inert, requiring an independent agent to process the data store&#39;s contents. Data stores that are not inert are usually referred to as search services (see below). Examples of data stores include (but are not limited to) standalone databases, indexes of content unaccompanied by systems to process said indexes, and electronic storage devices such as hard disks, tape drives, and memory systems.  
         [0038]    Search Service: Any data store that, when presented or sent a user query, responds with content holding links to other content that is deemed to be consistent with the parameters of said user query. Search services are inherently dynamic, able to respond to interaction and requests from external agents without said agents participating in the creation of said response. Search services almost always are grounded in one or many indexes, which are the source of the raw data forming said response. Search engines are the most common embodiment of search services (although other embodiments are possible).  
         [0039]    Webcrawling: The process of iteratively and cyclically following links embedded in bodies of content in order to discover (and usually process in some way) other bodies of content. Webcrawling can operate over any set of content held in any electronic medium that supports the semantics of links; while webcrawling is traditionally and originally associated with the processing of content on the World Wide Web, within the scope of this disclosure no assumptions should be made as to what electronic medium holds the content that is to be processed, nor as to the protocols or communications methods used to transmit said content.  
         [0040]    Crawling: See ‘Webcrawling’.  
         [0041]    Index: A representation of a set of bodies of content that may be rapidly searched. Indexes are almost always built using some variation of webcrawling.  
         [0042]    User Interface: Any method or apparatus that enables a user of the current invention to interact with said invention&#39;s parameters, controls, and outputs.  
         [0043]    Evaluation: Any analysis method with a goal of qualifying content in accordance with the parameters of a user query.  
         [0044]    Qualified: A possible state of a body of content as determined by evaluation of said content whereby said content satisfies the minimum requirements of a user query&#39;s parameters.  
         [0045]    Database: Any organized collection of data.  
         [0046]    Seed Address: A representation of a particular link; groups of seed addresses are used to initialize the webcrawling process.  
         [0047]    Thread: An independently-operating process of execution within a computer system.  
         [0048]    Sub-Query: Data and/or instructions derived from a user query and information about the syntax and protocol of search services and data stores that, without any additional external procedural information, enable a system to interact with said search services and data stores in an abstracted way.  
         [0049]    Template Query: An intermediary structure used to create a sub-query. A template query is a framework that describes how to access a given search service or data store. A sub-query is created when a template queries “blanks” are filled with properties from a user query. An example of a template query for a web search engine might be: http://www.search.com/r=5&amp;keywords=**, where ‘**’ is the blank that must be filled with user query-specific parameters in order to effectively access the search engine in accordance with said user query. Template queries may be built using any language or protocol compatible with the target set of search services or data stores, including (but are not limited to) SQL statements, Remote Procedure Calls, and Simple Object Access Protocol requests.  
         [0050]    Module: A component of a software process that is complete in and of itself that can accomplish a certain task or process without depending on external processes or components. A module is replaceable, given additional modules that can accomplish said task or process.  
         [0051]    Iteration: A single cycle of operation of a webcrawling process, consisting of the steps of locating bodies of content referred to by seed addresses, retrieving said bodies of content, extracting links to additional content from some or all said bodies of content. The newly-extracted links are then used as seed addresses for another iteration.  
         [0052]    Network: A collection of computing devices that can communicate between themselves.  
         [0053]    Block: A single functional sub-process.  
         [0054]    Next, the preferred embodiment of the present invention is described in detail.  
         [0055]    [0055]FIG. 1 illustrates the top-level architecture of the preferred embodiment. All action is initiated by a user  1 , who creates a user query  2 . Creating a user query may be accomplished using any method or apparatus that allows user  1  to specify all of the possible parameters of the user query  2 , which may include (but not be limited to) a specification of what content should be considered qualified, which search services and data stores should be accessed, whether and to what extent the webcrawling process should proceed, as well as specification of various other parameters affecting the operation of the preferred embodiment.  
         [0056]    Once the user query  2  is created, it is directed to the Search System Interface (SSI)  3 , the operation of which is illustrated in FIG. 2. Block  11  in the SSI  3  accepts the user query  2 , and performs any formatting or pre-processing that is necessitated by the preferred embodiment&#39;s implementation prior to proceeding with the full processing of the user query  2 .  
         [0057]    Once all such pre-processing is completed, the user query  2  is forwarded to the Query Manager (QM)  4 , the operation of which is illustrated in FIG. 3. Block  13  in the QM  4  accepts the formatted user query  2 , and stores it along with new status information in a new search record inside running search database  21 . The status information encompasses all data related to the processing of a user query, which includes (but is not limited to) its original parameters, how many webcrawling iterations have been completed, and all data on qualified content as such becomes available.  
         [0058]    Once block  13  has created the new search record in database  21 , the user query  2  is passed to block  14 , which determines whether or not the said user query&#39;s parameters require that search services and/or data stores be accessed. If a user manually entered seed addresses into the user query  2  instead of requiring that search services and/or data stores be accessed to retrieve seed addresses, processing will advance to block  16 , the functionality of which is detailed below. If the user query  2  specifies that search services and/or data stores are to be accessed to retrieve seed addresses (perhaps in addition to more seed addresses entered into the user query  2  by a user), processing will advance to block  15 .  
         [0059]    Block  15  updates the running search record created by block  13  in the database  21  to indicate that the user query  2  is being forwarded for search service and data store processing. Block  15  then forwards the user query  2  to the Outside Index Query Manager (OIQM)  5 . Block  22  in the OIQM  5 , the operation of which is illustrated in FIG. 7, accepts the user query  2 . The user query  2  is forwarded to block  46 , which extracts all parameters from said user query that relate to search service and data store operations. These parameters are forwarded to block  47 , which determines specifically which search services and data stores need to be accessed in order to satisfy said parameters. Information about which search services and data stores to access is forwarded to block  48 .  
         [0060]    Database  49  contains any and all knowledge required to interface with a set of search services and data stores, of which the search services and data stores to be accessed must be a subset. This knowledge mainly (but not exclusively) consists of instructions for how to establish a communication with search services and data stores, and what content or syntax must be transmitted over said connection in order to effectively access the search services and data stores. This knowledge may be modified, created, or updated in order to allow a user query to be translated into a form appropriate for any search service or data source.  
         [0061]    Block  48  retrieves all knowledge held in database  49  related to the search services and data stores that are to be accessed, and forwards this knowledge to block  50 . Block  50  uses said knowledge to create one template query for each search service and data store that is to be accessed. All created template queries are then forwarded to block  51 , which populates the template queries with user query-specific parameters to form full sub-queries. Said sub-queries are forwarded through block  52  to block  23 .  
         [0062]    Block  23  sends each sub-query (either in turn or concurrently using threads) to block  53  in the Communications Interface  6 , which is illustrated in FIG. 5. Block  53  establishes all necessary connections and operates all necessary protocols to communicate with each search service and data store over a plurality of networks and storage mediums, represented by entity  7 . The sub-query created for each search service and data store is then transmitted via said connection(s) and protocol(s) to each said search service and data store. As each search service and data store respond to their respective sub-queries, block  53  receives said response, and forwards it to block  54 . Block  54  extracts any and all meta-data from each response, and forwards both the meta-data and the content of each response to block  23  in the OIQM.  
         [0063]    The content and meta-data of each search service&#39;s and data store&#39;s response is then forwarded to block  24 , which extracts any and all links from said content and meta-data, and creates seed addresses with said links. When all possible seed addresses have been created using the responses of all accessed search services and data stores, said seed addresses are forwarded to block  25 .  
         [0064]    Block  25  sends a status update containing results of accessing the search services and data stores to the QM  4 , which is received by block  17 . Block  17  updates the running search record with said status update to reflect progress in the search, and then returns control to block  25  in the OIQM  5 . Block  25  then forwards all seed addresses created from search service and data store responses to block  62 .  
         [0065]    Block  62  combines received seed addresses with any and all seed addresses held by the user query  2  that were entered by the user  1  manually. This combined set of seed addresses is forwarded to the Network and Crawling Manager (NCM)  8 , the operation of which is illustrated in FIG. 6, and is received by block  26 .  
         [0066]    Block  26  creates a new thread of execution for each seed address; the processing of each seed address after this point occurs concurrently along with all other seed addresses within the context of its own thread. Each seed address&#39; thread then progresses to block  27 . Database  31  acts as a caching mechanism: if the content and meta-data associated with a seed address is already stored in the cache, then said content and meta-data can be retrieved from the cache without taxing external network and other I/O channels. The oldest contents in database  31  should be purged occasionally in order to ensure that the most recent content and meta-data associated with each seed address is being utilized.  
         [0067]    Block  27  accesses database  31  to determine if the seed address&#39; content and meta-data are stored there. If so, the seed address&#39; thread proceeds to block  30 , where the content and meta-data associated with said seed address is retrieved from database  31 , and said content and meta-data is forwarded to block  29 . If the seed address&#39; content and meta-data are not available from database  31 , then the seed address&#39; thread proceeds to block  28 .  
         [0068]    Block  28  sends the seed address to the Communications Interface  6 , where its associated content and meta-data are retrieved in much the same way as search services and data stores are accessed, described earlier. When all available associated content and meta-data have been retrieved, the Communications Interface  6  returns control to block  28 , which stores the newly-retrieved content and meta-data in database  31  for future use. The seed address&#39; thread then progresses to block  29 .  
         [0069]    Block  29  forwards the seed address&#39; content and meta-data to the Evaluation Module  9 , the operation of which is illustrated in FIG. 5, and is received by block  55 . The interface  63  between the Evaluation Module  9  and the NCM  8  is specifically designed to allow different modules to take the role of the Evaluation Module  9 , allowing for the logistically simple customization of the evaluation process. Alternative embodiments of the current invention may therefore substitute, at a user&#39;s discretion, very different implementations of the general functions of the Evaluation Module  9 .  
         [0070]    Block  55  analyzes the received content and meta-data to determine their associated seed address&#39; qualification with regard to the parameters stored in user query  2 . The preferred embodiment&#39;s criteria for qualification is relevancy of the seed address&#39; content and meta-data to keywords provided by the user  1 , stored in the user query  2 . Other implementations of the Evaluation Module  9  utilized through interface  63  may have very different criteria. Once all analysis in block  55  is concluded, control is forwarded to block  56 .  
         [0071]    Block  56  assigns a rating, which is usually but not necessarily numerical, to the seed address based on its level of qualification in accordance with user query  2 . This rating is forwarded to block  57 .  
         [0072]    If the assigned qualification rating is above some threshold specified in user query  2 , then block  57  will forward the seed address&#39; content and meta-data to block  58 ; otherwise, control is transferred to block  60 .  
         [0073]    Block  58  in the preferred embodiment of the Evaluation Module scans the seed address&#39; content for any embedded or linked content in accordance with the specifications in the user query  2 , and makes note of the presence of any such content. For example, the user  1  may specify in the user query  2  that the presence of or links to certain types of video files should be noted and reported. The seed address&#39; content and meta-data is then forwarded to block  59 .  
         [0074]    Block  59  generates a summary or report based on the seed address&#39; content and meta-data, and transfers control to block  60 .  
         [0075]    Block  60  forwards all results of the Evaluation Module&#39;s analysis to block  29  in the NCM  8 , which includes the qualification rating, notations of the presence of or links to any special content types specified in user query  2 , and the summary or report based on the seed address&#39; content and meta-data.  
         [0076]    Block  32  in the NCM  8  determines if the seed address&#39; content and meta-data are qualified with regard to the parameters stored in user query  2  based on the qualification rating returned to block  29  by block  60  in the Evaluation Module  9 . If the seed address&#39; content is not qualified, then block  34  in NCM  8  disposes of the thread processing said seed address and any system resources associated with said processing. If the seed address&#39; content is qualified, then it and the analysis results associated with it are forwarded to block  33 .  
         [0077]    Database  35  contains records holding qualified addresses and information associated with them: their content and meta-data and the results of the analysis performed on said content and meta-data by the Evaluation Module  9 . Block  33  stores the qualified seed address, its content and meta-data, and its associated analysis results in database  35 , and then passes control to block  38 .  
         [0078]    Block  38  creates a status report that details the state of the NCM  8  and its processing of seed addresses associated with user query  2 , including how many threads are still active. Block  38  sends this status report to block  19  in the Query Manager  4 . Block  19  updates the running search record in database  21  to reflect the contents of said status report. If said status report indicates that all threads within the NCM  8  have finished processing and if user query  2  requires that localized webcrawling be utilized, then block  19  sends a request for localized webcrawling back to block  38  in the NCM  8 .  
         [0079]    When block  38  in the NCM  8  receives a response to the status report it sent to block  19  in the Query Manager  4 , said response is sent to block  37 .  
         [0080]    If block  37  finds that the Query Manager&#39;s response does not include a request to conduct localized webcrawling, then control is passed to block  36 . Block  36  fetches all qualified addresses, their associated content and meta-data, and the results of the analysis of said content and meta-data from database  35 , and sends the entirety of those data to block  20  in the Query Manager  4 .  
         [0081]    Block  20  closes the running search record associated with user query  2  in database  21 , and then forwards to block  12  in the SSI  3  the search results provided by block  36 . Block  12  then formats said results, leading to the creation of a set of user-viewable—usable results (document  10 ). Document  10  is then sent to the user  1  via communications channel  61 , which may constitute any method or pathway that can adequately relate the contents of document  10  to user  1 .  
         [0082]    If block  37  does determine that the Query Manager&#39;s response forwarded by block  38  contains a request to perform localized webcrawling, then control is passed to block  39 .  
         [0083]    Block  39  retrieves from database  35  a set of highly-qualified seed addresses whose associated content and meta-data have not yet been crawled in connection with user query  2 . The content and meta-data associated with said highly-qualified seed addresses is then passed to block  40 .  
         [0084]    Block  40  extracts all available links held in the content and meta-data that are provided to it; said links are used to create a new set of seed addresses that are sent to block  26 .  
         [0085]    While a preferred embodiment of the invention has been shown in detail above, it will be understood by those skilled in the art that various changes in form and details may be effected therein without departing from the spirit and scope of the invention as specified by the appended claims.