Abstract:
A method or algorithm for classifying downloaded links or URL&#39;s based on the reason behind the download. Downloads are classified into categories, for example, a “visited” URL or an “embedded” URL. Categorizing these downloads allows other applications to collect information for storage, upload, or other action. This algorithm uses information from the browser history and packet streams to obtain and categorize the links or URL&#39;s for classification.

Description:
FIELD OF THE INVENTION 
       [0001]    The present invention relates to a method or algorithm for differentiating between browser links (or URL&#39;s) visited on a page versus those embedded which are simply embedded on a given Web site. 
       BACKGROUND OF THE INVENTION 
       [0002]    Web Browsing has become a part of every-day life. At work one may use a Web Browser to access e-mail, interact with customers, or look up information on the Internet. Children use the Web and thus Web Browsers to review assignments from class, turn in homework, or simply socialize with their friends. In the home, people use Web Browsers to read news, manager bills, or plan a vacation, among other uses. 
         [0003]    The effectiveness of Web based advertising is an important question with significant economic implications. Businesses such as Google have been extremely successful based on Web based advertising models. In the Prior Art, it was relatively straightforward to count the number of times a specific web page had been downloaded to a device. Counting the number of times a specific web page had been downloaded may be accomplished using techniques that prevent web pages from being cached, effectively allowing the server to count every time the page is downloaded (referred to as “hits”). But, if there are references to a web site embedded into other web sites, the question remains, how many of these “hits” are counted because a user requested the URL (Universal Resource Locator) to be downloaded or whether the URL was merely present in another web page. Prior Art techniques for counting “hits” may thus be inaccurate, and advertisers may be charged improperly for advertising services. For businesses to understand the value of using embedded links for advertising, it would be valuable to know how frequently URLs presented to users are visited. 
         [0004]    Another problem with Prior Art web browsing relates to parental monitoring of Web usage. Many web sites, while they themselves may be harmless, may include embedded links that may not be appropriate for children. In the Prior Art, parents may be able to block specific web sites using parental blocking software or services. However such blocking software may block entire websites only, and thus preventing access to web pages with acceptable content for children, as well as more objectionable material. For example, research and encyclopedia sites may contain web pages with information that a child may wish to access to complete a homework assignment or paper. However, links within such pages may lead to other pages with objectionable images or adult content. It would be useful to allow a child to selectively visit a page with non-objectionable material, even if the page contains links to objectionable material, while at the same time blocking links to the objectionable material pages. It would also be useful to parents to know if a particular web page was actually selected by the user, or if it was downloaded only because that particular page was referenced by an embedded URL. 
       SUMMARY OF THE INVENTION 
       [0005]    For businesses to understand the value of using embedded links for advertising, it would be valuable to know how frequently URLs presented to users are visited. The present invention provides a method and algorithm for determining if a URL was simply presented to the user or if it was actually visited by the user. The power of this method is, given a few pieces of data, a determination can be made whether the user actually clicked the link rather than just had it show up because they visited a site. 
         [0006]    With regard to parental monitoring of Web usage, the algorithm and method of the present invention may be used in an application to provide information to parents indicating whether a particular web page was actually selected by the user or if it was downloaded only because it was an embedded URL. This information may also be used within parental blocking software to allow access to web pages that may contain content appropriate for children, while blocking links on such pages which may lead to inappropriate material. 
         [0007]    The present invention includes a method and apparatus for differentiating between browser links (or URL&#39;s) actually visited on a page versus those links where are simply embedded on a given Web site. Embedded URL&#39;s are downloaded simply because they exist on an accessed page, not because they have been specifically requested by the browser user (examples of embedded URL&#39;s include but are not limited to images, ads, style-sheets, and the like). In particular, the present invention is directed at classifying browser links for data mining, security, and other purposes. 
         [0008]    The method of the invention uses existing browser histories and packet processing to determine the reason the web browser is accessing the requested URL. The result of this classification may be used for different purposes, such as saving URL history and classification for later upload to a server, or for blocking of URL loading and/or display on a user device. 
         [0009]    The method or algorithm for classifying downloaded links or URLs is based on the reason behind the download. Downloads are classified into categories, for example, a “visited” URL or an “embedded” URL. Categorizing these downloads allows other applications to collect information for storage, upload, or other action. The algorithm of the present invention uses information from the browser history and packet streams to obtain and categorize the links or URL&#39;s for classification. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0010]      FIG. 1  is a diagram illustrating the set of URL types and their relationship. 
           [0011]      FIG. 2  is an illustration of an actual HTTP request (in packet dump mode) with key fields highlighted. 
           [0012]      FIG. 3  is a system-level processing diagram. 
           [0013]      FIG. 4  is a detailed flow diagram of the URL classification algorithm. 
           [0014]      FIG. 5  illustrates three examples of HTTP requests with key fields highlighted and the associated example Browser History. 
           [0015]      FIG. 6  is a highlighted version of the flow diagram of  FIG. 4 , illustrating the flow of HTTP example request  610   
           [0016]      FIG. 7  is a highlighted version of the flow diagram of  FIG. 4 , illustrating the flow of HTTP example request  620 . 
           [0017]      FIG. 8  is a highlighted version of the flow diagram of  FIG. 4 , illustrating the flow of HTTP example request  630 . 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       [0018]    For the purposes of this description, a “requested” URL is defined as any URL being accessed through an HTTP (Hyper-Text Transfer Protocol) request from the web browser. A “visited” URL is the actual URL being visited by the user. An “embedded” URL is any URL that is requested while loading a visited URL, for example, images, ads, or style-sheets.  FIG. 1  illustrates the relationship between these three types of URL&#39;s. “Visited” and “embedded” URL&#39;s are a subset of “requested” URL&#39;s. 
         [0019]    HTTP requests contain two descriptive fields used in the classification algorithm. The first of these fields is the “Host” field. This field is required in an HTTP request and gives the address that is hosting the current requested URL. The second of these fields is the “Referer” field, which is the address that referred the browser or user to the current requested URL. The “Referer” field is optional in HTTP requests.  FIG. 2  contains an actual HTTP request with these two descriptive fields highlighted. 
         [0020]    The algorithm of the present invention classifies the request into either a “visited” URL or “embedded” URL using these fields and allows for storage into one or more databases. These databases can be remotely or locally located and can take many different forms. The database for “visited” URL&#39;s is represented by component  350  of  FIG. 3 . The database for “embedded” URL&#39;s is represented by component  340  of  FIG. 3 . 
         [0021]    Packets received on a device implementing this algorithm are intercepted in a device specific manner. Packets may be analyzed directly or duplicated and provided to the algorithm (component  330  of  FIG. 3 ).  FIG. 3  illustrates an approach where the packet is intercepted and duplicated for processing by this algorithm. Component  300  represents a stream of data packets. Each packet may or may NOT be an HTTP request. Component  310  represents the device specific manner in which packets are duplicated and provided to the URL Classification Algorithm (Component  330 ). Component  320  represents a duplicated packet being passed to URL Classification Algorithm. Component  330  processes the incoming packet and classifies the packet with additional information obtain from Browser History (Component  390 ), providing the URL names to the appropriate databases (Components  340  and  350 ). Remaining components ( 360 ,  370 ) represent normal system processing that is unaffected by the URL Classification Algorithm. 
         [0022]      FIG. 4  represents a flow chart of the URL Classification Algorithm (Component  330 ). Referring to  FIG. 4 , each HTTP request contains the requested URL, the domain (defined by the “Host” field), and optionally the “Referer”. In step  410 , the first HTTP request is assumed to be a “visited” URL. Every time a URL is classified as a “visited” URL, the “stored domain” is updated to the domain represented in the “Host” field in step  430 . This “stored domain” is then used for comparisons with other URL&#39;s. 
         [0023]    If the requested URL is not first, as determined by step  410 , then the domain is compared against the “stored domain” in step  420 . If the domains are the same, and the requested URL is not in the browser history as determined in step  440 , then it is determined that the requested URL is an “embedded” URL and database  340  may be updated. If the requested URL is in the browser history, as determined in step  440 , then the requested URL is classified as a “visited” URL in database  350 . 
         [0024]    If the domain of the requested URL is different from the “stored domain”, as determined in step  420 , then the optional “Referer” field may be examined in step  450 . If the “Referer” field does not exist in the HTTP request, and the requested URL appears in the browser history, as determined in step  460 , then this is classified as a “visited” URL and database  350  is updated. If the “Referer” field doesn&#39;t exist in the HTTP request, as determined by step  450 , and the requested URL is not in the browser history, as determined in step  460 , then this URL is classified as an “embedded” URL and database  340  is updated. 
         [0025]    If the “Referer” field exists in the HTTP request, as determined in step  450 , then the domain of the referer (the “referer domain”) is compared against the “stored domain” in step  470 . If they are the same, and the requested URL is in the browser history, then this is classified as a “visited” URL and database  350  is updated. If the “stored domain” and the “referer domain” are the same, as determined in step  450 , but the requested URL is not in the browser history, as determined in step  470 , then the URL is classified as an “embedded” URL and database  340  is updated. 
         [0026]      FIG. 5  illustrates three examples of HTTP requests with key fields highlighted and the associated example Browser History. The purpose of these examples is to walk through the invention flow chart illustrated in  FIG. 4  using the sample HTTP requests  610 ,  620 ,  630  and the sample Browser History  640  of  FIG. 5 . To support these examples, the three flow charts of  FIGS. 6-8  will show the highlighted path taken for the three HTTP requests being analyzed, using the flow chart of  FIG. 4  described above. 
         [0027]    Referring to  FIG. 5 , HTTP request  610 , is the first URL received in this example list of HTTP requests. Referring to  FIG. 6 , Step  410  analyzes the URL provided by the Host field (http://www.walkinghotspot.com/), and makes Decision  501  that this is the First URL in the sequence of HTTP Requests. The next step is to Update Stored Domain in Step  430 , which in turn, classifies the URL of HTTP request  610  as a “Visited” URL, stores domain www.walkinghotspot.com as a Stored Domain in step  430 , and updates “Visited” URLs database  350 . 
         [0028]    Referring back to  FIG. 5 , the next HTTP request in the example, HTTP request  620 , contains the URL www.walkinghotspot.com/library/styles/whs.css, and this is not the First URL in this example list of HTTP requests, which was discovered during the processing as described with regard to  FIG. 6 . Referring to  FIG. 7 , Step  410  analyzes whether the HTTP  620  request contains the First URL, and Decision  502  is reached. Next, in Step  420 , the “Host” field, or Domain, www.walkinghotspot.com is compared to the Stored Domain www.walkinghotspot.com obtained during the processing described with regard to  FIG. 6 . The example shows they are equal, producing Decision  503 . After performing Step  440  and checking the Browser History  640 , the exact URL is not found; therefore, decision  506  is made, which classifies the URL www.walkinghotspot.com/library/styles/whs.css of HTTP request  620  as an “Embedded” URL in database  340 . 
         [0029]    Referring back to  FIG. 5 , the final HTTP request in the example is HTTP request  630 , which has URL and Domain given in the ‘Host’ field (www.taprootsystems.com), and this is different from the Stored Domain (www.walkinghotspot.com). Referring to  FIG. 8 , Step  410  analyzes whether the HTTP request contains the First URL in the sequence of HTTP Requests, and Decision  502  is reached. Next, in Step  420 , the Domain www.taprootsystems.com is analyzed, and Decision  504  is reached, because the domain is not the same as the Stored Domain www.walkinghotspot.com. Next the Referer Exists analysis in Step  450  is performed. The HTTP request  630  shows that the Referer field exists, and Decision  507  is made, which then requires a Browser History check in Step  470 . In this example, referring back to  FIG. 5 , Browser History  640  contains a URL, which matches the requested URL (http://www.taprootsystems.com) provided in the HTTP Request, so Decision  511  is made. This leads to Update Stored Domain in Step  430 . Finally, the URL www.taprootsystems.com in HTTP request  630  is now classified as a “Visited” URL. 
         [0030]    The examples illustrated in  FIGS. 6-8  show how a URL can be determined to be a “Visited” or “Embedded” URL. As the algorithm of the present invention can determine the difference between an actual visit and an embedded URL, the present invention may provide a means by which an advertiser can more accurately determine whether a website has actually been visited, or whether just the embedded URL has been displayed. Advertising rates may be determined based on total number of hits (visited and embedded) and also on how many hits actually lead to a visit to the website of interest. Such data may be output as a ratio of hits to visits, or as raw data indicating the number of visited URLs (database  350 ) and embedded URLs (database  340 ). 
         [0031]    For parental control or other type of access restriction software, the algorithm may be used to allow a user to access a page with an embedded URL, which may be on a blacklist, but prevent the user from visiting the page on the blacklist. As the user browses the web, the URLs are classified according to the algorithm  330 . If a URL is determined to be an embedded URL  340 , the user&#39;s access to a page with that embedded URL may be allowed. However, if the URL is a visited (or attempted visit) to a blacklisted URL (determined by comparing the visited URL database  350  with a predetermined blacklisted database  350 ) then access to such a database may be denied or logged. In addition, the present invention may be used by web crawlers or the like to determine whether a blacklisted URL is embedded in another web page, in order to determine whether additional web pages should be black-listed. 
         [0032]    While the preferred embodiment and various alternative embodiments of the invention have been disclosed and described in detail herein, it may be apparent to those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope thereof.