Patent Publication Number: US-10324984-B2

Title: System and method for content selection for web page indexing

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application is a continuation of and claims priority from U.S. patent application Ser. No. 12/435,777 filed on May 5, 2009, entitled SYSTEM AND METHOD FOR CONTENT SELECTION FOR WEB PAGE INDEXING; this application is incorporated herein by reference in its&#39; entirety. 
    
    
     FIELD OF THE INVENTION 
     This disclosure relates to a system and method for indexing of web pages, and in particular for indexing web pages based on sections of a document. 
     BACKGROUND OF THE INVENTION 
     When a search engine performs a search of web pages on the internet, the search engine uses an index of web pages to determine those web pages that match the search terms entered by a user. A problem with current indexing methods, is that the indexing is typically performed across an entire document (i.e. web page) so that a web page will be determined to be a match if the entered search terms appear anywhere in the document. Often, when a user enters multiple search terms, the search results will include documents in which the multiple terms are found but in unrelated parts of the document. This problem is exacerbated because the raw markup content of a document observed by the search engine may locate words of unrelated areas, such as side menus and the like, in proximity to content of the primary material of the document, thereby reducing the effectiveness of search term proximity searches. This can lead to content being erroneously related in the index and to non-meaningful content, such as link lists etc being included in the index. 
     What is required is a system, method and computer readable medium that is able to provide improved document indexing. 
     SUMMARY OF THE INVENTION 
     In one aspect of the disclosure, there is provided a method for indexing a document comprising determining indexable content of the document and indexing the indexable content. Determining the indexable content comprises dividing the document into a plurality of document elements, determining an attention history for a plurality of the document elements, and determining one or more document elements that meet an attention history requirement. 
     In one aspect of the disclosure, there is provided a system for indexing web pages comprising a content selection module and an indexing module. The content selection module processes interaction data for a web page to select indexable content of the web page and the indexing module indexes the indexable content of the web page. 
     In one aspect of the disclosure, there is provided a computer-readable medium comprising computer-executable instructions for execution by a processor, that, when executed, cause the processor to determine interaction data for a web page, determine a document object model element of the web page associated with the interaction data, and add the document object model element to a set of indexable content for the web page. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       Reference will now be made, by way of example only, to specific embodiments and to the accompanying drawings in which: 
         FIG. 1  illustrates a system for indexing documents; 
         FIG. 2  illustrates a method for indexing documents; 
         FIG. 3  illustrates a content selection process; 
         FIG. 4  illustrates a method for indexing document sections; 
         FIG. 5  illustrates a search engine example; 
         FIG. 6  illustrates a processor and memory of a content selection module; 
         FIG. 7  illustrates an instruction set that may be executed on the processor and memory of  FIG. 6 ; and 
         FIG. 8  illustrates the processor and memory of  FIG. 6  in association with a processor and memory of a indexing module. 
     
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     In  FIG. 1 , there is shown a system  10  for providing web page indexing. The system  10  includes an indexer module  12 , a web page data store  15 , an interaction data store  19  and a document index  17 . The indexer module  12  performs indexing of documents, e.g. web pages of the web page data store  15 . In the embodiment shown, the web page data store is primarily a web page repository but may be any arbitrary document store. 
     The indexer module  12  includes a content selection module  13  and an index processor module  14 . 
     In a web page indexing embodiment, the document index  17  is a searchable database that stores correlations between URLs and various content elements, search terms etc for the URL. In alternative embodiments, the document index may similarly index many different document types. 
     The interaction data store stores interaction data for various web pages. The interaction data represents an attention history for web pages and may include user interface events and attention data from past interactions with documents and web pages, such as described in the Applicant&#39;s co-pending patent application Ser. No. 12/435,767, filed May 5, 2009, the entire contents of which are herein incorporated by reference. As described in referenced patent application, in typical webpage protocols, a document object model (DOM) of a webpage structure may include a document, a body and several elements, including elements embedded within each other. Events, in particular user triggered events such as mouse movements, cursor movements, mouse clicks or keyboard events may be generated at various locations on the page. Application code, e.g. javascript applications, running in the webpage environment may register event handlers on various web page objects. The event handlers may handle events in a variety of ways. A first event handling method is referred to as event capturing. Event capturing provides an event first to the top most object in the web page hierarchical structure, i.e. DOM, and onward in a descending fashion as follows:
         DOCUMENT→BODY→ELEMENT A→ELEMENT B       

     Event bubbling provides an event first to the bottom most object and onward in an ascending fashion as follows:
         ELEMENT B→ELEMENT A→BODY→DOCUMENT P As described in the Applicant&#39;s co-pending application Ser. No. 12/435,767, filed May 5, 2009, referenced above, event streams may be recorded during an interaction between a user and a particular web page. These event streams may be analyzed, in conjunction with the DOM structure, to determine if the event streams represent user attention events for the respective document elements during the user interaction. A statistical analysis may be performed across multiple interactions for a document or URL to form the interaction data of the interaction data store  19 . That is, the interaction data comprises statistical data that associates the elements of a document or web page (e.g. DOM elements) with the amount of attention, in particular human-dependent attention, that the respective document element receives during user interactions. Attention events may include not only blatant events such as complete mouse clicks, but also inferred attention events such as partial mouse clicks, mouse moves, hovers, lingers and hints, as described in the Applicant&#39;s co-pending application Ser. No. 12/435,767, filed May 5, 2009, referenced above.       

     A method for indexing documents is illustrated in the flowchart  100  of  FIG. 2 . At step  101 , a document is divided into document elements. Attention history for the document elements is then determined  102  and document elements that meet an attention history requirement are selected  103  for adding to indexable content of the document  104 . The indexable content is then indexed  105 . 
     The content selection process of the indexer module  12  will now be described with reference to  FIG. 1  and the flowchart  200  of  FIG. 3 . At step  201 , the content selection module  13  sends a fetch document request to the web page data store  15 . For example, the content selection module  13  may request a document by providing the URL for the document to the web page data store  15 , which returns the document  21  matching the URL to the content selection module  13 . The URL may be selected from an indexing schedule, controlled by the indexing module or other components of a search engine or web crawler. For example, new URLs may be determined by a web crawler, the content of the URLs may be downloaded to the web page data store  15  and the URL may be scheduled for indexing. Similarly, known and existing URLs may be periodically scheduled for indexing in order to ensure that document index is maintained up to date. 
     At step  202 , the content selection module  13  requests interaction data for the document  21 , e.g. by URL, from the interaction data store  19 . If no interaction data exists for the URL, as determined at step  203 , then the content selection module passes all elements of the document  21  to the index processor module  14  (step  210 ) where the document is indexed based on the entire contents of the document. 
     If the URL has associated interaction data  22 , then the document is divided into its document object elements and the content selection module  13  retrieves the DOM element IDs for all interactions from the interaction data store (step  204 ). The content selection module  13  then analyzes each of the DOM elements of the web page. After selecting an initial DOM element of the web page (step  205 ), the content selection module determines if the selected DOM element is found in the interaction data set of DOM element IDs (step  206 ). If the DOM element has no interaction data, then the content selection module returns to step  204  where a next DOM element is selected for the web page. If interaction data does exist for the DOM element, then a threshold requirement may optionally be applied (step  207 ). That is, the interaction data for the DOM element may be analyzed to determine if the DOM element receives attention events above a threshold level. In one embodiment, the threshold may be limited to only human classified attention events. If the DOM element does not meet the optional threshold requirements, then the DOM element may be ignored and a next DOM element may be selected. 
     As described in the Applicant&#39;s co-pending patent application Ser. No. 12/435,767, filed May 5, 2009, referenced above, over multiple retrievals a web page or document may exist in multiple instances having different content elements, such as updatable sections or contextually placed elements. Therefore, the DOM elements for a document, and the content (e.g. text) within them, may not be consistent across all interactions. In one embodiment the DOM element identifiers include a locality sensitive hash (LSH) code such as Nilsimsa digest as described in the aforementioned reference. The result is that DOM elements with nearly equivalent content have nearly equivalent LSH codes in their DOM element identifiers. Two DOM elements, from different interactions, may be analyzed as being the same if they have sufficiently similar LSH codes. 
     If the DOM element satisfies the threshold requirements, or if no threshold requirements are applied, then the DOM element is added to the indexable content for the web page (step  208 ). If further DOM elements are to be analyzed, then the process returns to step  204  and a next DOM element is selected. Once all DOM elements of the web page have been analyzed, the content selection module  13  outputs the total indexable content to the index processor module  14  (step  209 ). The index processor module  14  then performs a routine indexing for the web page, typically as an inverted index, but using only those elements of the web page that are provided in the indexable content. The output of the index processor module is used to add to or update the document index  17 . The index built is typically an inverted index that maps words to the documents that they occur in. An inverted index may also include information about (i) the frequency of each word within each document and (ii) the positions of each word within each document. Indexing methods are known in the art and the particular indexing method is not considered relevant to the present embodiments. Thus no further description of particular indexing methods is provided here. 
     In one alternative embodiment, document elements with no interaction data may be added to the indexable content, thus allowing these elements to be indexed and potentially driving traffic towards those elements. However, these elements may be flagged and if no interaction data exists at the next indexing of the web page, then content for these elements may be dropped from the index. Indexing of a document may include updating the index to remove index entries that are no longer relevant as determined by the attention data. 
     In one embodiment, outclick events may be excluded from consideration as these types of events indicate the attention of a user shifting away from the web page. 
     In the embodiments described with reference to  FIGS. 2 and 3 , the document is processed as if it were one entire section. In an alternative embodiment, described with reference to the flowchart  300  of  FIG. 4 , the document may be first divided into sections  301 . Each section may be divided into its respective one or more document elements  302 . Identified sections are mutually disjoint (i.e. non-overlapping with respect to content and elements). Each identified section may be processed as a complete document, similar to the method described above with reference to  FIG. 2 , with the content in each section being segregated from other section content when it is indexed. That is, at step  303 , the attention history for the elements of a section is determined and the elements that meet an attention history requirement are selected at step  304 . The selected elements are added to the indexable content for the respective document section ( 305 ). Each section that has indexable content can then be indexed at step  306 . 
     The process  200  of  FIG. 3  performed by the content selection module  13  may be modified by including a section analysis step, e.g. prior to step  204 , and then performing the steps  204  to  209  on a section by section basis. The section analysis step may be performed “on the fly” or may be performed by retrieving historical section analysis data, e.g. from the interaction data store  19  or another suitable data store. Further description of the section analysis is provided in greater detail below. 
     In the embodiments of  FIGS. 2 and 3 , when indexing a URL, the index processor module  14  requests a URL from the content selection module  13  and gets back a single document, albeit only threshold relevant document elements. However, for embodiments the URL document has been divided into sections, different sets of document elements are identified as being co-relevant and these document element sets (sections) are processed independently. For the index processor module  13  this means requesting a single URL and receiving back multiple “documents” to be indexed. These “documents” are subsets (sections) of the document that was retrieved from the web page data store. 
     The embodiments above demonstrate a system in which only the elements of a web page that have sufficient human attention data, as measured via events, are used to index the web page. Furthermore, different sections of a document, as identified by analysis of the interaction data, are indexed separately. By separating the web page into sections for indexing purposes, several benefits may be achieved. One advantage is that subareas of web pages that do not receive at least a certain level of attention are not indexed, thereby reducing search engine spam. 
     A further advantage is that the relevance of search results may be improved. For example, a simple three column web page  50  is shown in  FIG. 5 . In this web page, the left most column  51  contains titles and links to other articles while the rightmost two columns  52  contain the article/content for the web page  50 . It is understood that sections  51 ,  52  are each comprised of several DOM elements. In this example, the set of DOM elements  51  contain the term “Term 1 ” and the set of DOM elements  52  contain the terms “Term 2 ” and “Term 3 ”. 
     The aggregate event and attention data for this web page may reveal at least two basic patterns of interaction:
         1) visitors who spend their interaction time on the right most columns  52 , presumably having found the content of interest; and   2) visitors who only briefly focus on the rightmost columns and then interact with the left column  51  and exit, presumably in search of content of more relevant interest than the content found in the rightmost columns  52 .       

     In the above described embodiments, these two clearly distinguished sections would be treated as completely separate “documents” for indexing purposes. That is, during indexing, the content of the leftmost column would be indexed (containing “Term 1 ” but not “Term 2 ” or “Term 3 ”). Then separately the rightmost two columns would be indexed (containing “Term 2 ” and “Term 3 ” but not “Term 1 ”). For both of these indexing operations we presume that the individual DOM elements containing the terms did receive enough attention to meet the threshold requirements. 
     A user may enter search terms such as Term 1  and Term 2  into a search engine. Under prior art indexing arrangements, the search engine would indicate this URL as having a match, and thus as relevant, to the search because the URL was indexed for both Term 1  and Term 2  (the entire document). However, it is clear in this example that the context of these terms shows the document is not relevant to a search for both of Term 1  and Term 2  because there is no actual content for Term 1  in the main content section  52 . In the indexing arrangement of the present disclosure, the URL representing the document would not be indicated as relevant for a search using Term 1  and Term 2  as search terms. This is because Term 1  would be indexed to the URL with the rest of the content of section  51 . Separately, Term 2  and Term 3  would be indexed to the URL with the content of section  52 . A search for terms Term 2  and Term 3  would however indicate this URL as relevant, as these terms both appear in the same section of the document considered to be indexable by the content selection module  13 . Thus the present embodiments can be used to enhance search engine performance by removing irrelevant URLs from the list of search results. 
     Most indexing methods use the URL as a unique identifier for the terms indexed (e.g. in an inverted index). To maintain separation of each section&#39;s content the Content Selection Processor  13  may optionally associate a distinct URL with each identified section of a web page. These per-section URLs may be a modification of the document URL via adding a prefix, a suffix, or any other suitable alteration so that (i) the Index Processor Module will process each section as a separate distinct “document” and (ii) the modification can be reversed as needed so the user can see the document URL in search results lists. 
     It is the job of section analysis to determine how many sections a document has. As described in the Applicant&#39;s co-pending patent application Ser. No. 12/435,767, filed May 5, 2009, low level interaction event data and the higher level attention analysis data (hints, lingers, hovers, etc.) are available in the Interaction Data Store  19 . Both data is DOM element centric, though the attention data may have a source and destination element. Section analysis is the process of performing statistical analysis on aggregate interaction data in order to subdivide the document based on interaction types. Section analysis may consider all the DOM elements receiving attention (and optionally interaction events) for individual interactions. For example, there may be 100 interactions of the sample page of  FIG. 5 . Section analysis observes that 20/100 interactions spend 98% of the attention in the set of DOM elements comprising the left column  51 . Further, 72/100 interactions spend at least 84% of the attention within the set of DOM elements comprising the rightmost two columns  52 . The remaining 8/100 interactions in our example do not show a preference for a sub-area of the document during the interaction. In this example section analysis determines that the document has two sections based on the aggregate interaction patterns. The sections are the DOM elements (and content) comprising  51  and the DOM elements (and content) comprising  52 . Though the section analysis is described herein as occurring during content selection, the section analysis can also be performed by various means (e.g. offline) and in one embodiment, the results of section analysis may already be stored within the interaction data store along with the attention data and the interaction event data. 
     A further advantage of the present embodiments is that by removing from indexing the content found in DOM elements that do not meet threshold levels of interest (individually or by sections), less text is indexed per document which leads to smaller indexes and faster search engines. Identifying and not indexing content can also prevent several types of “spamdexing” where content is created to be seen by web crawlers but not by humans. Because the indexing arrangement of the present embodiments relies on human dependent interactions with a web page, any content that is not seen by humans will eventually be dropped from the indexing such that the spamdexing strategy becomes ineffective. One example of spamdexing is where the same color is used for the document background and for some “spam text” that is meant only for web-crawlers and search engines. The lack of human attention on such an element will result in it not meeting the threshold criteria and it will not be indexed. 
     In one embodiment, the indexer module  12  may be embodied within a search engine (not shown). Alternatively or in addition, the indexer module may be provided as a third party service to a search engine or web server. 
     In the Applicant&#39;s co-pending patent application Ser. No. 12/435,774, filed May 5, 2009, the entire contents of which are herein incorporated by reference, there is described a web crawler that selects URLs for document retrieval and storage in a web page data repository dependent on whether the URL or the URL&#39;s document contains human-dependent interaction data. In the present embodiments, if documents are located using a web crawler such as described in the Applicant&#39;s co-pending application referenced above, some pre-filtering of stored document elements may have already been applied. 
     The components of the system  10  such as the indexer module  12 , web page data store  15 , interaction data store  19  etc. may be provided within a single computer system or may be provided in a distributed manner with suitable communications links between modules and components. In various embodiments, data services such as the interaction data store  19  and web page data store  15  may be provided as third party data services to the indexer module  12 . 
     In one embodiment, the content selection module  13  may include a processor  61  operatively associated with a memory  62  as shown in  FIG. 6 . The memory  62  may store instructions that are executable on the processor  61 . In addition, the memory  62  may provide elements of the interaction data store  19  or web page data store  15 . An instruction set  400  that may be executed on the content selection processor  61  is depicted in the flowchart of  FIG. 7 . Specifically, when executed, the instruction set  400  allows the processor to determine interaction data for a web page ( 401 ), determine a document object element of the web page associated with the interaction data ( 402 ), and add the document object element to a set of indexable content for the web page. By executing the instruction set  400 , the processor is able to transform the contents of a document into a smaller subset of the document for indexing. The indexable content of the document may be passed to a second processor, such as a processor  71  of an indexing module  14  shown in  FIG. 8 , through a suitable communications link  65 . The indexing module processor  71  may be operatively associated with a memory  72  which may store an executable instruction set, which, when executed, may cause the indexing module processor  71  to receive the indexable content from the content selection processor  61  through the communications link  65  and to index the indexable content. The memory  72  may provide elements of the document index  17  which receives updated index data from the processor  71 . Alternatively, the document index  17  may be provided by a database (not shown) in operative association with the indexing module processor  71  which may be transformed by the indexing module processor  71  when new or amended indexing data is generated. 
     While specific reference has been made herein to indexing of web pages, a person skilled in the art will readily understand that the systems and methods described herein may be used to index and search on many different document types and all such document types are intended to be embraced herein. For example, documents may have a variety of divisions (such as chapters, pages, etc) that can be indexed concomitant with sections and elements identified for indexing based on user attention attracted to the sections and elements. 
     Although embodiments of the present invention have been illustrated in the accompanied drawings and described in the foregoing description, it will be understood that the invention is not limited to the embodiments disclosed, but is capable of numerous rearrangements, modifications, and substitutions without departing from the spirit of the invention as set forth and defined by the following claims. For example, the capabilities of the invention can be performed fully and/or partially by one or more of the blocks, modules, processors or memories. Also, these capabilities may be performed in the current manner or in a distributed manner and on, or via, any device able to provide and/or receive information. Further, although depicted in a particular manner, various modules or blocks may be repositioned without departing from the scope of the current invention. Still further, although depicted in a particular manner, a greater or lesser number of modules and connections can be utilized with the present invention in order to accomplish the present invention, to provide additional known features to the present invention, and/or to make the present invention more efficient. Also, the information sent between various modules can be sent between the modules via at least one of a data network, the Internet, an Internet Protocol network, a wireless source, and a wired source and via plurality of protocols.