Abstract:
A method and system are described for collecting addresses for remotely accessible information sources. Messages, such as emails, carried by a messaging network (N1) are intercepted before reaching a destined terminal. Addresses for remotely accessible information sources (i.e. URLs) are identified from the intercepted email messages. The messages are analysed to be classified as either a first type of message (e.g. spam or virus messages) or a second, different, type of message. If the intercepted message is classified as the first spam/virus type then data indicative of the identified address (URL) is transmitted to a filtering system ( 100 ) which controls access to the remotely accessible information sources. As a result, addresses (URLs) are gleaned from transmitted messages such as spam e-mail and supplied to a filtering system ( 100 ) which controls access to the resources accessible at those addresses.

Description:
FIELD OF THE INVENTION 
       [0001]    The present invention relates in general to a method of, and system for, collecting addresses for remotely accessible information sources. Embodiments of the invention are particularly suited for use in updating a filtering system which controls access to links to web pages. The invention is particularly well suited for collecting such addresses from electronic messages and short messages such as Wireless Access Protocol (WAP) and Multimedia Messaging Systems (MMS) messages. 
       BACKGROUND OF THE INVENTION 
       [0002]    The Internet is a global system of computers that are linked together to facilitate communication between computers. These computers can be accessed by users so as to download and display informational pages therefrom. The easy access and inexpensive cost of retrieving Internet pages has led to several problems for controlling access to inappropriate information, such as pornography. Several solutions to this problem have been proposed, including rating systems similar to that used for rating movies so that a parent or employer can control access to Internet servers, or pages, that have a particular rating. In addition to a rating scheme, others have developed databases that contain the uniform resource locator (URL) address of sites to be blocked. These databases are integrated into network computer systems and Internet firewalls so that a person wishing access to the Internet first has their URL request matched against the database of blocked sites and is denied access to access any URL found in the database. One such system is described in U.S. Pat. No. 5,678,041. 
         [0003]    Such systems rely on the completeness of the database of accessed sites, and, since new servers and URLs are being added to the Internet on a daily basis, these databases do not provide a complete list of sites that should be blocked. An improvement to the system described in U.S. Pat. No. 5,678,041 is presented in EP1318468, which describes a system distributed between a central “database factory” arranged to perform URL categorization and to store the results of the categorization in a central database, and many local “access systems”, each of which is associated with a given LAN and can connect to the database factory. Any given access system is loaded with a copy of, and updates to, the categorization data from the database factory, and additionally includes a so-called filter module, which can perform a certain amount of processing in relation to uncategorized URLs. These uncategorized URLs are any URLs that are requested by a client on the LAN and that are not listed in the categorised data downloaded from the database factory. 
         [0004]    When a local access system receives a URL request from a client machine on its LAN, it can often identify a category for that URL on the basis of the categorisation data received from the database factory. However, if a category cannot be obtained from the database factory, the local filter module will perform some local processing such as text processing in order to identify whether or not it is safe for the client to access the URL. The output of the local processing will be stored in association with the uncategorized URL. For any given accessed URL the filter module also stores a counter, which is incremented whenever a given URL is requested. At certain times—e.g. at a set time, at a random time, every n units of time, or when a given URL has been requested a specified number of times—the database factory requests the uncategorized URL and any associated processed data from an access system. Since the database factory collects data from different access systems and downloads its output to each of the access systems, any given access system can benefit from knowledge obtained via URL requests received from access systems connected to unrelated LANs. 
         [0005]    One feature that is common to all known internet access control systems is the trigger for the URL analysis, which is the receipt of a request for access to a given URL; thus analysis of URLs is triggered by a direct request for access to a computer on the Internet corresponding to the URL. 
       SUMMARY OF THE INVENTION 
       [0006]    According to the present invention there is provided an apparatus and method as set forth in the appended claims. Other features of the invention will be apparent from the dependent claims, and the description which follows. 
         [0007]    In one aspect the present invention there is provided a method of collecting addresses for remotely accessible information sources for use in updating a filtering system which controls access to such information via a public access network, the method comprising: interrupting transmission of a message destined for a terminal other than the filtering system; identifying an address for a remotely accessible information source from the message; analyzing the message on the basis of an attribute of the received message so as to classify the received message as either a first type of message or a second, different, type of message; and if the received message is classified as the first type of message, transmitting a message comprising data indicative of the identified address to the filtering system. 
         [0008]    Thus with embodiments of the invention, addresses for remotely accessible information sources such as URLs are captured from analysis of messages, such as electronic messages, which are transmitted through the network. This provides a new and request-independent source of potentially problematic URLs to a URL filtering system, and one that operates asynchronously with respect to the filtering system. 
         [0009]    In one arrangement the received message comprises a message body which contains data specifying a network location corresponding to a remotely accessible information source, and the method includes analysing the transmitted electronic message on the basis of a criterion relating to the network location specified within the message so as to classify the message as a first or a second type of message. 
         [0010]    In another arrangement the transmitted message can be analysed on the basis of a criterion relating to an attribute of the message such as sender (name, IP address), subject line and subject matter contained with the message, so as to classify the message as a first or a second type of message. Typical examples of first types of messages include spam, virus and phishing classifications, and the analysis can include deriving a message pattern for the message and comparing the derived message pattern with message patterns associated with previously transmitted messages. 
         [0011]    Embodiments of the invention also provide a system for classifying messages and transmitting data indicative of potentially problematic URLs to a URL filtering system. Preferably the functionality is provided as part of an email analysis system, so that the identification of potentially problematic URLs is a by-product of in-transmit email analysis. Further features and advantages of the invention will become apparent from the following description of preferred embodiments of the invention, given by way of example only, which is made with reference to the accompanying drawings. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0012]      FIG. 1   a  is a schematic block diagram showing components of an email filtering system and categorisation system within which embodiments of the invention operate; 
           [0013]      FIG. 1   b  is a schematic timing diagram showing an overview of data transfer between the various components shown in  FIG. 1   a;    
           [0014]      FIG. 2  is a schematic block diagram showing components of an email filtering system according to embodiments of the invention; 
           [0015]      FIG. 3  is a schematic flow diagram showing steps performed by the email filtering system of  FIG. 2 ; and 
           [0016]      FIG. 4  is a schematic block diagram showing components of a categorisation system according to embodiments of the invention. 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       [0017]    As described above, embodiments of the invention provide a system for capturing and categorizing URLs. In comparison to known systems, in which URLs are captured as part of a user request to access the URL, with embodiments of the invention URLs are captured during a process involving steps other than receiving a direct request from a user. In at least some embodiments of the invention, a given URL is captured as a by-product of another process, and, broadly speaking, the trigger for, and process for categorisation of, the given URL is decoupled from user requests to access the given URL. 
         [0018]    Turning to  FIG. 1   a,  an overview of a categorisation system  100  with which embodiments of the invention operate will first be described. The system  100  enables a terminal T 1  to request data from a web server  101  via a browser  102  or a bespoke client application loaded on the terminal T 1 . The categorisation system  100  is arranged to filter (control and/or analyze) the terminal&#39;s T 1  access to resources and the functionality to achieve this monitoring of the terminal T 1  can be split between a local filter module F 1  and the categorisation system  100 , which together can provide for the real-time or off-line filtering of computer network access so as to analyse the data associated with the attempted access of the resources and, where appropriate, permit data, etc. to be downloaded from the address associated with the access request to the web server  101 . The categorisation system  100  cooperates with a database DB 1 , which is arranged to store details of URLs and a category assigned thereto, and, in arrangements including a local filter module Fl in addition to the categorisation system  100 , a copy of the categorised data held in the database DB 1  is stored in a mirror database shown as DB 1 ′. 
         [0019]      FIG. 1  a also shows network elements associated with an email filtering system  103 , which, in one embodiment, is used to capture URLs and send them onto the categorisation system  100 . The email filtering system  103  is implemented within the normal delivery path of an email and utilised to identify spam emails, phishing emails or viruses; referring to  FIGS. 1   a  and  1   b,  such a delivery path is shown in the context of sender A creating and sending an email to recipient B: the email is sent from terminal T 3 , at which the sender A has composed the email, and is delivered to a terminal T 2  corresponding to the email address of the email. As is known in the art, the email is sent from terminal T 3  to a local mail server S 1  cooperating with terminal T 3 , which may reside on a local area network or at an ISP; when the local mail server S 1  receives the email, the mail server S 1  queries a Domain Name Server (DNS)  101  to obtain the Internet Protocol (IP) address for the recipient&#39;s email address. According to a preferred arrangement, the DNS record corresponding to the recipient&#39;s address is preconfigured to comprise data identifying a email filtering system  103  (e.g. by modifying the associated mail exchange (MX) record), which means that all emails destined for recipient B are automatically directed to the email filtering system  103  instead of to the mail server corresponding to recipient B&#39;s email address. 
         [0020]    As shown in  FIG. 2  the email filtering system  103  preferably comprises an application server, a web server, various conventional and bespoke processing components, and can comprise one or a plurality of machines, which collectively analyse emails in accordance with various criteria, as described in more detail below. The email filtering system  103  is connected to various servers and gateways, typically via a public network such as the Internet N 1 , including the destination email server S 2  corresponding to the recipient&#39;s email address. 
         [0021]    Under certain conditions, such as when an email contains a URL satisfying various conditions or when the email originates from a source identified to be a certain type of Internet service, the email filtering system  103  sends data indicative of the URL and/or source address of the email to the categorisation system  100 . This action can be performed and at the same time as, or before, taking action in relation to the email itself. Turning to  FIG. 3 , when an email is received by the email filtering system  103 , it is passed to the application server  205  for processing by the URL software component  207  (step  301 ). Once received by the URL software component  207 , the email is analysed for the presence of otherwise of one or more web links (step  303 ); in the event that no such link is identified, the software component  207  passes the email to the spam detection software component  209  for signature code and other similar email analysis and, if appropriate, subsequently to the application server  205  for routing to server S 2  in the normal manner (step  305 ). 
         [0022]    In one embodiment, the presence of a URL string may be detected in an email formatted in accordance with a mark-up language (e. g., HTML) based on formatting data surrounding the URL string in the message. For example, in the HTML message, the URL string may have the following format: &lt;a href=“http://www.sesame.com”&gt;, in which case the software component  207  is configured to search through the HTML message for a string comprising “http:” or “href=”, extracting the data immediately following this string. 
         [0023]    In the case where the email is determined to contain a URL, the software component  207  can proceed to analyse the email (step  307 ), either on the basis of the URL alone or in conjunction with an analysis of other features of the email (such as subject, format of sender and recipient address, or text within the email); methods are known in the art for performing such analysis of an email and are commonly referred to as “Spam detection methods”. 
         [0024]    In relation to the URL, the software component  207  can review the URL on the basis of one or more of the following parameters, which are stored in storage  209 : 
         [0025]    Domains: URLs may include sub-domains and redirections to target URLs, and the which points to the same target URL. For example, URL “http://spamtastic.test.co.uk” can be specified from URLs that include different sub-domains. Accordingly the URL string can be examined for the presence of different sub-domains (here there are two: spamtastic.test.co.uk; test.co.uk); 
         [0026]    Redirect URLs: a URL can comprise another URL that points to a particular target URL; for example, the URL “http://random.com/date/ 03 /*http://www.spamtasic.co.uk” actually points to the target URL “http://www.spamtasic.co.uk”. Accordingly the URL string can be examined for redirection to a different URL; 
         [0027]    Extraneous information within the URL: a user name, password, “@” signs, numeric character references (which specify the code position of a character in the document character set), character entity references (which use symbolic names), buffer overflows, null-padding, and embedded null characters. Accordingly the URL string can be examined for the presence of such information; 
         [0028]    Certain words within the URL: pornographic or offensive words, stored for example in the database DB 2 . Accordingly the URL string can be examined for the presence of such words. 
         [0029]    In addition, the email filtering system  103  can maintain, or have access to, a URL blacklist that includes URLs that are known to be indicative of spam and/or relating to an email virus and/or phishing, and the software component  207  can be arranged to compare the or each URL within the email with those listed in the black-list; when the blacklisted URLs are stored as hash values, it is convenient for the software component  207  to generate a hash value of the or each URL within the email, enabling the comparison to be performed on the basis of the generated and stored hash values. 
         [0030]    In one arrangement, the email is scored on the basis of one, some or all of the above parameters, and the individual scores are combined dependent on prespecified weightings. Once an overall score has been evaluated the software component  207  compares the score with a predetermined threshold (step  309 ); in the event that the score exceeds the threshold, data indicative of the URL is transmitted to the categorisation system  100 , as a categorisation request message M 1 . 
         [0031]    As described above, the email filtering system  103  is capable of applying various spam detection techniques to incoming emails via spam detection software component  209 ; in relation to emails that do not contain a URL link, these are subject to analysis on the basis of subject, format of sender and recipient address, and/or text within the email so as to derive a signature code for the email (step  305 ). As a result of such analysis, data corresponding to the various email attributes can be stored in the database DB 2 , and for those emails determined to be one of a batch of bulk emails—in other words an email with a text portion (signature code) identical to that sent to at least one other recipient—the email filtering system  103  can send a categorisation request message M 1  comprising data indicative of the originating address of the emails to the categorisation system  100 . 
         [0032]    Preferably the categorisation request message M 1  includes data indicative of the type of classification derived by the email filtering system  103  as a result of the above-mentioned analysis; for example the categorisation request message M 1  includes fields associated with spam, virus and phishing classifications, and if a respective field of a given categorisation request message M 1  includes a “1” therein, this indicates an associated classification to the categorisation system  100 . This can aid machine-recognition by the categorisation system  100 . 
         [0033]    Embodiments of the invention thus advantageously provide a new mechanism for notifying URLs to the categorisation system  100 , and since the categorisation request messages M 1  are received independently of a request to access the URL, processing of the request and categorisation of the URL is handled independently of the processing of client-originating resource access requests. As a result, a greater number and variety of URLs will be analysed (and categorised) than is possible with conventional systems, importantly without incurring any additional overheads in relation to the processing of client-originating resource access requests. In addition and in view of this new source of URLs, the repository of categorised URLs is significantly increased, increasing the probability that a previously unseen client-originating resource access request can be served with a response indicative of a (previously assigned) category. 
         [0034]    The components and operation of the categorisation system  100  will now be described; whilst the nature of the categorisation performed by the system  100  is conventional, the categorisation system  100  is uniquely configured to respond to requests from nodes other than an authenticated client device. In addition, the categorisation system  100  is selectively arranged to respond to the categorisation request messages M 1  to transmit a (prestored) category and/or to store the results of the categorisation and/or to transmit the results of the categorisation in a response message M 2  in dependence on the source of the categorisation request message M 1 . 
         [0035]    Turning now to  FIG. 4 , in one arrangement the categorisation system  100  comprises various conventional input/output, storage, memory and operating system processing components together with various bespoke processing components including a request handling component  401  for receiving categorisation requests, and an automated categorization or classification component  403  to determine the appropriate category or categories of the URL. The request handling component  401  is configured to identify the source of the categorisation request message M 1 , to retrieve the URL within the categorisation request message M 1  and to determine, on the basis of the data stored in database DB 1 , whether the URL in the request message M 1  requires categorizing. If the URL has not been previously analysed, the request handling component  401  passes the URL and any additional data associated with the URL to the categorization component  403 . If the URL has already been categorised (i.e. if it is stored in the database DB 1 ), the request handling component  401  is arranged to retrieve the category from the database DB 1  and to create a message M 2  having a message body containing the category for transmission to the source of the categorisation request message M 1 , namely the email filtering system  103 . In the event that the categorisation request message M 1  identified a URL that had been retrieved from within an email (as opposed to a sender of bulk emails), the email filtering system  103  stores the category data in conjunction with the URL in the database DB 2  upon receipt thereof. Thus this category information advantageously provides a second level check in relation to the analysis performed at steps  305  and  307  by the email filtering system  103 , and can be utilised in any decisions taken in relation to the handling of the emails in which the URL originated. 
         [0036]    Assuming the message M 1  to contain an URL corresponding to (a) page(s) held by the web server S 1 , the categorisation component  403  retrieves the page(s) from the server S 1  and parses the page using a categorization engine to determine the appropriate category or categories of the URL, determining statistical probabilities and multidimensional vectors during the categorization process. Categorization can be based upon word analysis, adaptive learning systems, and/or image analysis, in which case the categorization component  403  can include a language analyser arranged to determine the language of the website for a given URL; examples of suitable techniques are described in International Patent Application having publication number WO0163835. In addition the categorization component  403  can interface with a human checker to determine the appropriate category or categories of the URL. 
         [0037]    Once the web page corresponding to the URL has been categorized, the categorization component  403  can be arranged to store the URL along with its associated one or more categories in the database DB 1  of categorized sites; the database DB 1  includes functionality for scheduling updates to local filter modules such as F 1  so as to ensure that the URL and its associated category or categories are copied to the mirror database DB 1 ′. 
         [0038]    The above embodiments are to be understood as illustrative examples of the invention. Further embodiments of the invention are envisaged. For example, the request handling component  401  can be configured to respond to categorisation request messages from analysers associated with processing of Wireless Access Protocol (WAP) and Multimedia Messaging Systems (MMS) messages 
         [0039]    It is to be understood that any feature described in relation to any one embodiment may be used alone, or in combination with other features described, and may also be used in combination with one or more features of any other of the embodiments, or any combination of any other of the embodiments. Furthermore, equivalents and modifications not described above may also be employed without departing from the scope of the invention, which is defined in the accompanying claims.