Patent Publication Number: US-8973678-B1

Title: Misspelled word analysis for undesirable message classification

Description:
TECHNICAL FIELD 
     This invention pertains generally to computer security and more specifically to identifying undesirable electronic messages. 
     BACKGROUND 
     Current statistical spam detection techniques rely heavily on their ability to find words that are known to be associated with spam email during classification of electronic messages. The authors of spam emails have become aware of this, and have started to purposefully misspell words in their messages. Most legitimate email contains many more correctly spelled words than misspelled words. On the other hand, a lot of spam email (especially short spam messages) often contains so many misspelled words that it is difficult to read at normal speed. Additionally, phishing emails are well known for containing spelling errors. 
     What is needed are methods, computer readable media and computer systems for allowing detection of undesirable emails, even where misspelled words have been inserted. 
     SUMMARY 
     Misspelled words are identified in incoming email messages. The presence of misspelled words in emails is used to help determine which of the emails are spam. Various statistical information concerning the number, prevalence, distribution, etc. of misspelled words in email messages is analyzed to detect spam or other forms of undesirable email, such as phishing emails. In some embodiments, the language in which an email is written is identified in order to aid in the identification of misspelled words. In some embodiments, the analysis of the misspelling information is combined with other techniques used to identify undesirable email. 
     The features and advantages described in this summary and in the following detailed description are not all-inclusive, and particularly, many additional features and advantages will be apparent to one of ordinary skill in the relevant art in view of the drawings, specification, and claims hereof. Moreover, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter, resort to the claims being necessary to determine such inventive subject matter. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a block diagram illustrating the operation of an anti-spam manager, according to some embodiments of the present invention. 
     
    
    
     The FIGURES depict embodiments of the present invention for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the invention described herein. 
     DETAILED DESCRIPTION 
       FIG. 1  illustrates an anti-spam manager  101 , according to some embodiments of the present invention. It is to be understood that although the anti-spam manager  101  is illustrated as a single entity, as the term is used herein an anti-spam manager  101  refers to a collection of functionalities which can be implemented as software, hardware, firmware or any combination of these. Where an anti-spam manager  101  is implemented as software, it can be implemented as a standalone program, but can also be implemented in other ways, for example as part of a larger program, as a plurality of separate programs, as a kernel loadable module, as one or more device drivers or as one or more statically or dynamically linked libraries. 
     As illustrated in  FIG. 1 , a filtering component  102  of the anti-spam manager  101  filters incoming email messages  103  so that the emails  103  can be further analyzed. As the term is used herein, filtering incoming email messages  103  simply refers to identifying incoming email messages  103  and processing them in some way before the messages arrive at their target destination. Various mechanisms are known by those of ordinary skill in the art for filtering an email  103  stream, all of which can be used within the context of the present invention. For example, anything from a plug-in to a simple mail transfer protocol server to a proxy server or a gateway can be used for message  103  filtering. So as not to distract from the explanation of the functionality of interest, these specific instantiations of the filtering component  102  are not separately illustrated. 
     In some embodiments of the present invention, a language identification component  104  of the anti-spam manager  101  performs language identification of the filtered email messages  103  (i.e., identification of the primary language in which the message  103  is written). Various techniques for language identification are known by those of ordinary skill in the relevant art. Most such techniques involve performing an n-gram based statistical analysis of the content of the message  103 . In other words, combinations of symbols of various lengths (n-grams) are identified in the message  103 , and a statistical analysis is performed, which takes into account in which language the detected n-grams are most likely to appear. As such, even if the majority of words are misspelled, they typically contain enough very similar bi-grams or tri-grams for the language identification to succeed. In some embodiments, no language identification is performed, and instead a default language (e.g., English) is assumed. In some other embodiments, language identification is performed, but a default language is assumed in the unlikely event that the language identification fails. 
     A spell check component  105  of the anti-spam manager  101  runs a spell check on incoming messages  103  (e.g., by using an appropriate language specific dictionary  107 ). The spell check component  105  performs a spell check much like any word processor, except that instead of correcting misspelled words, the spell check component  105  collects relevant statistics (“misspelling metrics”)  109 . The statistics  109  can include information such as the number of misspelled words in a message  103 , the total number of words in the message  103 , the number of sentences that contain exactly one misspelled word, the total number of sentences that contain exactly two misspelled words, the total number of sentences that contain exactly three misspelled words, the total number of sentences that contain four or more misspelled words (and so on), the total number of sentences, the average number of misspelled words per sentence, the average number of words per sentence, etc. The exact misspelling metrics  109  to collect is a variable design choice. 
     A detection component  108  subjects the incoming messages  103  to spam detection. The detection component  108  receives the misspelling metrics  109 , and uses them as an aid in the spam detection process. In one embodiment, the detection component  108  comprises a threshold analyzer, which simply checks whether any of the misspelling metrics  109  exceed a set threshold  111  (such as average misspelled words per sentence greater than 50% of average number of words per sentence, ratio of misspelled words to total visible words in the message exceeding 40%, etc.). In one such embodiment, if a misspell metric  109  exceeds a corresponding threshold value  111 , the message  103  is classified as spam. In other embodiments, the detection component  108  only adjudicates the email  103  to be spam in response to a minimum number of metrics  109  or specific combinations of metrics  109  exceeding corresponding threshold values  111 . In yet other embodiments, one or more metrics  109  exceeding the threshold value(s)  111  is taken as one piece of evidence against the legitimacy of the message  103 . It is to be understood that the specific threshold values  111  to use are variable design parameters, which can be adjusted up or down as desired. 
     In many embodiments, the misspelling metric  109  analysis is not used in isolation. For example, in some embodiments the misspelling metrics  109  can be used as one of multiple heuristics in a heuristic analysis of messages  103  to identify spam. In some embodiments, existing techniques of mitigating false positive adjudications of emails by pre-filtering legitimate email  103  can be used in conjunction with an analysis of misspelling metrics  109  (and, optionally, in combination with any of the many other heuristics at play in an antispam implementation). In some embodiments the detection component  108  comprises a statistical engine which performs a statistical analysis of electronic message  103  content in order to identify statistical patterns associated with undesirable electronic messages  103 , using misspelling metrics as one factor in the statistical analysis. For example, the statistical engine can utilize the misspelling metrics  109  as one additional domain specific input feature to a classification component  110 . In this context, the misspelling metrics  109  can be combined with other input features to help the classification component  110  determine the proper category for the associated message  103 . As will be understood by those of ordinary skill in the relevant art in light of this specification, the misspelling metric  109  analysis can be used as an additional feature in any system used to detect spam and/or other forms of undesirable messages, such as phishing attempts. It is to be further understood that various techniques for undesirable email detection are known to those of ordinary skill in the relevant art, and the use of any of these in combination with misspelling metric  109  analysis is within the scope of the present invention. 
     As will be understood by those familiar with the art, the invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Likewise, the particular naming and division of the portions, modules, agents, managers, components, functions, procedures, actions, layers, features, attributes, methodologies and other aspects are not mandatory or significant, and the mechanisms that implement the invention or its features may have different names, divisions and/or formats. Furthermore, as will be apparent to one of ordinary skill in the relevant art, the portions, modules, agents, managers, components, functions, procedures, actions, layers, features, attributes, methodologies and other aspects of the invention can be implemented as software, hardware, firmware or any combination of the three. Of course, wherever a component of the present invention is implemented as software, the component can be implemented as a script, as a standalone program, as part of a larger program, as a plurality of separate scripts and/or programs, as a statically or dynamically linked library, as a kernel loadable module, as a device driver, and/or in every and any other way known now or in the future to those of skill in the art of computer programming. Additionally, the present invention is in no way limited to implementation in any specific programming language, or for any specific operating system or environment. Furthermore, it will be readily apparent to those of ordinary skill in the relevant art that where the present invention is implemented in whole or in part in software, the software components thereof can be stored on computer readable media as computer program products. Any form of computer readable medium can be used in this context, such as magnetic or optical storage media. Additionally, software portions of the present invention can be instantiated (for example as object code or executable images) within the memory of any programmable computing device. Accordingly, the disclosure of the present invention is intended to be illustrative, but not limiting, of the scope of the invention, which is set forth in the following claims.