Assessing behavior patterns and reputation scores related to email messages

A computer-implemented method includes generating behavior patterns based on historical behavior of a plurality of emails. The method further includes receiving an email message from a sender, wherein the email message is withheld from delivery to a recipient. The method further includes extracting a plurality of features from the email message. The method further includes determining whether content of the email message matches at least one criterion for suspicious content. The method further includes determining a reputation score associated with the sender based on a comparison of the extracted features with the behavior patterns, wherein the extracted features include an identity of the sender. The method further includes responsive to the content of the email message not matching the at least one criterion for suspicious content and the reputation score meeting a reputation threshold, delivering the email message to the recipient.

FIELD

Embodiments relate generally to assessing behavior patterns and reputation scores related to email messages. More particularly, embodiments relate to methods, systems, and computer readable media that determine behavior patterns of email messages and assign reputation scores to email messages to determine whether to deliver the email messages to recipients.

BACKGROUND

Attempts at using email messages to perform malicious activities are becoming increasingly sophisticated. Previous generation scams involved attacks on the message body; headers; attachments; and Simple Mail Transfer Protocol (SMTP) parameters. Mitigation techniques, such as Domain-Based Message Authentication, Reporting and Conformance (DMARC), Sender Policy Framework (SPF), and Domain Keys Identified Mail (DKIM) have been used to mitigate the effects of such attacks. Because such attacks are well-defined high-volume threats, email providers and applications can combat the threats by determining a signature, deploying the signature, and detecting and/or blocking the malicious emails based on the signature. One disadvantage of signatures is that it takes time to prepare and deploy signatures.

Next generation threats include scams targeting companies that conduct wire transfers and have suppliers abroad (e.g., Business Email Compromise (BEC)), targeted phishing attacks, and account compromise. Next generation threats are more difficult to detect because there is an insufficient volume of the attacks to generate a signature. In addition, even if there is a sufficient volume, next generation threats may take place over a short duration such as a few minutes, which means the delay inherent in preparing signatures makes them infeasible as a sufficient mitigation measure.

SUMMARY

Embodiments of this application relate to determining whether to deliver the email messages to recipients. A computer-implemented method includes generating behavior patterns based on historical behavior of a plurality of emails. The method further includes receiving an email message from a sender, wherein the email message is withheld from delivery to a recipient. The method further includes extracting a plurality of features from the email message. The method further includes determining whether content of the email message matches at least one criterion for suspicious content. The method further includes determining a reputation score associated with the sender based on a comparison of the extracted features with the behavior patterns, wherein the extracted features include an identity of the sender. The method further includes responsive to the content of the email message not matching the at least one criterion for suspicious content and the reputation score meeting a reputation threshold, delivering the email message to the recipient.

In some embodiments, determining the reputation score is further based on an association of the sender to another sender with a low reputation score. In some embodiments, the method further comprises providing a user interface that includes the email message and an option to report the email message as suspicious. In some embodiments, the method further comprises responsive to the content of the email message not matching any criterion for suspicious content and the reputation score being less than the reputation threshold, alerting the recipient that the email message is suspicious. In some embodiments, alerting the recipient that the email message is suspicious includes providing a user interface that includes an alert and an option to ignore the alert and provide the email message to the recipient. In some embodiments, the method further comprises generating, during offline analysis, cached analytics and comparing the extracted features to the cached analytics, where providing the email message to the recipient is further responsive to the comparing the extracted features to the cached analytics not identifying suspicious content. In some embodiments, generating the behavior patterns based on the historical behavior includes: training a machine-learning model based on training data that includes suspicious content and safe content and providing the extracted features to the machine-learning model, and determining the reputation score is further based on the extracted features provided to the machine-learning model. In some embodiments, the extracted features include one or more of: information from a Simple Mail Transfer Protocol (SMTP) process that generates the email message, results from one or more scanners that perform the scanning, or data added by an enriched scanner context. In some embodiments, the information from the SMTP process includes one or more of: an SMTP client Internet Protocol (IP) address associated with the email message, a number of email messages that were delivered in an SMTP session, wherein the email message is included in the SMTP session, results of a reputation lookup on the SMTP client IP address, whether the sender passed one or more of a Domain-based Message Authentication, Reporting and Conformance (DMARC) test, a Domain Keys Identified Mail (DKIM) test, or a Sender Policy Framework (SPF) test; a HELO parameter or an Extended HELO (EHLO) parameter; SMTP verbs (e.g., “MAIL FROM” versus “Mail From”); whether the email message exited the SMTP early; and/or whether the email message sent an SMTP command before a response from a previous command was received. In some embodiments, the results from the one or more scanners include one or more of: one or more results of reputation lookup in a third-party Domain Name System-based Remote Block List (NDS RBL), one or more results of reputation lookup in local systems, or one or more results of reputation lookup in local systems that are based on a client IP address, a sender domain, or a domain included in an Extended HELO (EHLO) command. In some embodiments, the data added by the enriched scanner context includes one or more of: details of a history of correspondence between the sender and the recipient, whether the sender has an affinity with other senders or clusters of senders and a corresponding reputation score, whether the email message is part of a suspicious pattern of correspondence, or machine-learning scores for machine-learning models that were exposed to the email message and a corresponding graph of correspondence that includes the email message.

A system comprises one or more processors and one or more computer-readable media, having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: generating behavior patterns based on historical behavior of a plurality of emails, receiving an email message from a sender, wherein the email message is withheld from delivery to a recipient, extracting a plurality of features from the email message, determining whether content of the email message matches at least one criterion for suspicious content, determining a reputation score associated with the sender based on a comparison of the extracted features with the behavior patterns, wherein the extracted features include an identity of the sender, and responsive to the content of the email message not matching the at least one criterion for suspicious content and the reputation score meeting a reputation threshold, delivering the email message to the recipient.

In some embodiments, the operations further include providing a user interface that includes the email message and an option to report the email message as suspicious. In some embodiments, the operations further include responsive to the content of the email message not matching any criterion for suspicious content and the reputation score being less than the reputation threshold, alerting the recipient that the email message is suspicious. In some embodiments, the extracted features include one or more of: information from a SMTP process that generates the email message, results from one or more scanners that perform the scanning, or data added by an enriched scanner context.

A computer-program product that includes one or more non-transitory computer-readable media with instructions stored thereon that, when executed by one or more computers, cause the one or more computers to perform operations comprising: generating behavior patterns based on historical behavior of a plurality of emails, receiving an email message from a sender, wherein the email message is withheld from delivery to a recipient, extracting a plurality of features from the email message, determining whether content of the email message matches at least one criterion for suspicious content, determining a reputation score associated with the sender based on a comparison of the extracted features with the behavior patterns, wherein the extracted features include an identity of the sender, and responsive to the content of the email message not matching the at least one criterion for suspicious content and the reputation score meeting a reputation threshold, delivering the email message to the recipient.

In some embodiments, the operations further include providing a user interface that includes the email message and an option to report the email message as suspicious. In some embodiments, the operations further include responsive to the content of the email message not matching any criterion for suspicious content and the reputation score being less than the reputation threshold, alerting the recipient that the email message is suspicious. In some embodiments, alerting the recipient that the email message is suspicious includes providing a user interface that includes an alert and an option to ignore the alert and provide the email message to the recipient. In some embodiments, the extracted features include one or more of: information from a SMTP process that generates the email message, results from one or more scanners that perform the scanning, or data added by an enriched scanner context.

The specification advantageously describes a solution that uses a history of correspondence between the sender, recipient, and intermediaries to identify behavior patterns. The solution also works in near real-time (such as a few seconds to a few minutes) to identify a threat and deploy protective actions. Lastly, the solution also includes scalability and cost, where high volumes of raw data can be processed at a reasonable cost.

DETAILED DESCRIPTION

FIG.1illustrates a block diagram of an example environment100. In some embodiments, the environment100includes an email server101, user devices115a,115n, and a network105. Users125a,115nmay be associated with the user devices115a,115n. In some embodiments, the environment100may include other servers or devices not shown inFIG.1. InFIG.1and the remaining figures, a letter after a reference number, e.g., “115a,” represents a reference to the element having that particular reference number. A reference number in the text without a following letter, e.g., “115,” represents a general reference to embodiments of the element bearing that reference number.

The email server101includes a processor, a memory, and network communication hardware. In some embodiments, the email server101is a hardware server. WhileFIG.1illustrates one email server101, the disclosure applies to a system architecture having one or more email servers101. The email server101is communicatively coupled to the network105. In some embodiments, the email server101sends and receives data to and from the user devices115a,115nvia the network105. The email server101may include an email application103aand a database199.

In some embodiments, the email application103areceives email messages from senders and delivers the emails to recipients if the emails are detected to not have suspicious content. More specifically, the email application103includes code and routines operable to generate behavior patterns based on historical behavior of emails, receive an email message from a sender, determine whether the email message matches at least one criterion for suspicious content, determine a reputation score, and responsive to the content of the email message not matching the at least one criterion for suspicious content and the reputation score meeting a reputation threshold, deliver the email message to a recipient.

In some embodiments, the email application103ais implemented using hardware including a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), any other type of processor, or a combination thereof. In some embodiments, the email application103ais implemented using a combination of hardware and software. In some embodiments, a user accesses the email application103avia a browser.

The database199may be a non-transitory computer readable memory (e.g., random access memory), a cache, a database system, or another type of component or device capable of storing data. The database199may store data associated with the email application103, such as user profile data, historical behavior of senders and recipients, historical behavior of email messages and corresponding content, scan results, etc.

The user device115may be a computing device that includes a memory and a hardware processor. For example, the user device115may include a desktop computer, a laptop computer, a tablet computer, a mobile device, a smartphone, a wearable device, a reader device, or another electronic device capable of accessing a network105.

In some embodiments, the user device115aincludes an email application103bwith code and routines operable to send email messages and receive email messages subject to the protections discussed below regarding suspicious content in emails. In some embodiments, the email application103breceives an email message from the email application103astored on the email server and displays the email message. In some embodiments, the email application103bperforms steps for determining whether the email message includes suspicious content and determining a reputation score before delivering the email to the user125.

In the illustrated embodiment, the entities of the environment100are communicatively coupled via a network105. The network105may include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or wide area network (WAN)), a wired network (e.g., Ethernet network), a wireless network (e.g., an 802.11 network, a Wi-Fi® network, or wireless LAN (WLAN)), a cellular network (e.g., a Long Term Evolution (LTE) network), routers, hubs, switches, server computers, or a combination thereof. AlthoughFIG.1illustrates one network105coupled to the user devices115and the email server101, in practice one or more networks105may be coupled to these entities.

FIG.2is a block diagram of an example computing device200that may be used to implement one or more features described herein. Computing device200can be any suitable computer system, server, or other electronic or hardware device. In some embodiments, computing device200is the email server101. In some embodiments, computing device200is the user device115.

In some embodiments, computing device200includes a processor235, a memory237, a I/O interface239, a display241, and a storage device245.

The processor235includes an arithmetic logic unit, a microprocessor, a general-purpose controller, or some other processor array to perform computations and provide instructions to a display device. Processor235processes data and may include various computing architectures including a complex instruction set computer (CISC) architecture, a reduced instruction set computer (RISC) architecture, or an architecture implementing a combination of instruction sets. AlthoughFIG.2illustrates a single processor235, multiple processors235may be included. In different embodiments, processor235may be a single-core processor or a multicore processor. Other processors (e.g., graphics processing units), operating systems, sensors, displays, and/or physical configurations may be part of the computing device200. The processor235is coupled to the bus218for communication with the other components via signal line222.

The memory237may be a computer-readable media that stores instructions that may be executed by the processor235and/or data. The instructions may include code and/or routines for performing the techniques described herein. The memory237may be a dynamic random access memory (DRAM) device, a static RAM, or some other memory device. In some embodiments, the memory237also includes a non-volatile memory, such as a static random access memory (SRAM) device or flash memory, or similar permanent storage device and media including a hard disk drive, a compact disc read only memory (CD-ROM) device, a DVD-ROM device, a DVD-RAM device, a DVD-RW device, a flash memory device, or some other mass storage device for storing information on a more permanent basis. The memory237includes code and routines operable to execute the email application103, which is described in greater detail below. The memory237is coupled to the bus218for communication with the other components via signal line224.

I/O interface239can provide functions to enable interfacing the computing device200with other systems and devices. Interfaced devices can be included as part of the computing device200or can be separate and communicate with the computing device200. For example, network communication devices, storage devices (e.g., memory237and/or storage device245), and input/output devices can communicate via I/O interface239. In another example, the I/O interface239can receive data, such as email messages, from a user device115and deliver the data to the email application103and components of the email application103, such as the pattern module204. In some embodiments, the I/O interface239can connect to interface devices such as input devices (keyboard, pointing device, touchscreen, microphone, camera, scanner, sensors, etc.) and/or output devices (display devices, speaker devices, printers, monitors, etc.). The I/O interface239is coupled to the bus218for communication with the other components via signal line226.

Some examples of interfaced devices that can connect to I/O interface239can include a display241that can be used to display content, e.g., an email message received from the sender. The display241can include any suitable display device such as a liquid crystal display (LCD), light emitting diode (LED), or plasma display screen, cathode ray tube (CRT), television, monitor, touchscreen, three-dimensional display screen, or other visual display device. The display241may be coupled to the bus218via signal line228.

The storage device245stores data related to the email application103. For example, the storage device245may store user profile data, historical behavior of senders and recipients, historical behavior of email messages and corresponding content, scan results, etc. In embodiments where the email application103is part of the email server101, the storage device245is the same as (or stores) the database199inFIG.1. The storage device245may be coupled to the bus218via signal line230.

In some embodiments, components of the computing device200may not be present depending on the type of computing device200. For example, if the computing device200is an email server101, the computing device200may not include the display241.

Example Email Application103

Various embodiments described herein perform automated computer-based analysis of email messages, including message content and metadata. Such automated analysis is performed with explicit user permission, in compliance with applicable laws and regulations. No content is shared with a third-party or reviewed by a human, other than those authorized by users. For example, the described techniques may be implemented in a security platform that performs automated scanning and threat mitigation. The security platform is configurable and may include various privacy settings. The security platform may be implemented by an email recipient organization, such as an organization (company, university, non-profit, government, etc.) and/or an email service provider. Email messages and/or features extracted from email messages may be stored and utilized in accordance with user-permitted settings.

FIG.2illustrates a computing device200that executes an example email application103stored on the memory237that includes a scanner202, a pattern module204, an optional machine-learning module206, an analytics module208, and a user interface module210. Although the modules are illustrated as being part of the same email application103, persons of ordinary skill in the art will recognize that the modules may be implemented by different entities in the operating environment100. For example, the email application103aon the email server101may implement the scanner202, the machine-learning module206, and the analytics module208while the email application103bon the user device115may implement the user interface module210.

The scanner202may include one or more scanners202that scan email messages for content and extract features from the email messages. Feature extraction is an automated process using one or more techniques such as text analysis, image analysis, video analysis, or other techniques to extract features from email content and/or metadata. Feature extraction is performed with user permission. Feature extraction can be performed using any suitable techniques such as machine learning, heuristics, pattern matching, hashing, etc. In some embodiments, the scanner202includes a set of instructions executable by the processor235to scan email messages. In some embodiments, the scanner202is stored in the memory237of the computing device200and can be accessible and executable by the processor235.

In some embodiments, the scanners202scan email messages that are used by the pattern module204to determine whether the email message or prior messages are associated with malicious activity, such as a phishing attack, a malware attack, etc. The scanners202record the determination of historical behavior of the email messages and determine behavior patterns based on the historical behavior. The scanners provide202the behavior patterns to the machine-learning module206as training data for one or more machine-learning models. The machine-learning model receives an email message from a user125as input and compares it to the behavior patterns to determine whether content of the email message matches at least one criterion for suspicious content and a reputation score for the sender. In some embodiments, the scanner202performs one or more of these operations during a Simple Mail Transfer Protocol (SMTP) session.

In some embodiments, the scanner202is divided into multiple scanners202that each perform a different scanning function. For example, a first scanner202may perform feature extraction, such as extracting metadata including identifying a sender, a recipient, identifying an envelope, identifying a header, etc. In some embodiments, the first scanner202extracts raw per-email data that includes identity vectors for the sender and all intermediate relays (public and private), Autonomous System Numbers (ASN), Domain Name System (DNS) hosting, and sender and intermediary authentication results. The first scanner202may determine whether an email message is an initial email message or a reply email message. Further, the first scanner202may receive scanning results from other scanners202, such as the malware scanner and resulting actions and forward the scanning results to the pattern module204and/or the machine-learning module206.

A second scanner202may identify if malware is present in an email message, and optionally, identify the malware. In some embodiments, the second scanner202may automatically block any email message that is identified as containing malware. In some embodiments, the second scanner202may transmit email messages that contain malware to the pattern module204or the machine-learning module206to provide examples of malware for analysis, for example, in the form of training data.

In some embodiments, the scanner202actions may be classified as extracting: (1) information from a Simple Mail Transfer Protocol (SMTP) process that generates the email messages, (2) results from the one or more scanners202that perform the scanning, and/or (3) data added by an enriched scanner202context.

The information from the SMTP process may include: senders and recipients of an email message; an SMTP client Internet Protocol (IP) address associated with an email message; a number of email messages that were delivered in an SMTP session; results of a reputation lookup on the SMTP client IP address; whether a sender passed one or more of a Domain-based Message Authentication, Reporting and Conformance (DMARC) test, Domain Keys Identified Mail (DKIM) test, or a Sender Policy Framework (SFP) test; a HELO parameter or an Extended HELO (EHLO) parameter; SMTP verbs (e.g., “MAIL FROM” versus “Mail From”); whether the email message exited the SMTP early; and/or whether the email message sent an SMTP command before a response from a previous command was received.

The results from the one or more scanners202may include results of a reputation lookup in a third-party Domain Name System-based Remote Block List (DNS RBL) (e.g., Spamhaus); results of reputation lookup in local systems; and/or results of reputation lookup in local systems, e.g., Sophos eXtensible List (SXL)) that are based on a client IP address, a sender domain, or a domain included in an Extended HELO (EHLO) command.

In some embodiments, the enriched scanner202context may include information from the pattern module204, the machine-learning module206, and/or the analytics module208.

The data added by the enriched scanner202context may include details of a history of correspondence between the sender and the recipient, such as a number of email messages received during a predetermined amount of time (e.g., day, week, month, etc.), how many of the email messages were initiated by a customer and not by an external person, and/or what was an average length of each exchange. The data added by the enriched scanner202content may include whether the sender has an affinity with other senders or clusters of senders and, if so, a corresponding reputation score for the sender. The data added by the enriched scanner202content may include whether the email message is part of a suspicious pattern of correspondence, such as when an unknown external sender emails certain groups of customer recipients unprompted. The data added by the enriched scanner202content may include or machine-learning scores for machine-learning models that were exposed to the email message and a corresponding graph of correspondence that includes the email message.

The pattern module204generates behavior patterns based on historical behavior of email messages, determines whether an email message includes suspicious content, and assigns a reputation score to senders. In some embodiments, the pattern module204includes a set of instructions executable by the processor235to generate behavior patterns and identify suspicious content. In some embodiments, the pattern module204is stored in the memory237of the computing device200and can be accessible and executable by the processor235.

In some embodiments, the pattern module204receives extracted features from email messages that were extracted by the scanners202and determines historical behavior based on the extracted features. In some embodiments, the pattern module204updates the historical behavior in real-time as new email messages are received.

The pattern module204generates behavior patterns based on the historical behavior. For example, the pattern module204tracks a history of emails between senders and recipients because a recipient that often communicates with a sender via email messages is less likely to send malicious content. In another example, the pattern module204aggregates counts of senders by recipient and recipients by sender and groups the senders by the similarity of recipients. In some embodiments, the pattern module204may generate a graph or nodes that track relationships between senders and recipients.

Turning toFIG.3A, an example diagram300is illustrated that shows historical behavior of senders of email messages based on similarity of recipients. In this example, sender 1 emails recipient a; sender 2 emails recipients a and b; sender 3 emails recipient a; sender 4 emails recipients b, c, and d; sender 5 emails recipients e and f; and sender 6 emails recipient e.

The pattern module204groups senders based on historical behavior of the senders emailing the same recipients. In this example, senders 2 and 3 both email only recipients a and b, so senders 2 and 3 are very similar. Sender 1 only emails recipient a, so sender 1 is somewhat similar to senders 2 and 3 because senders 2 and 3 also email recipient a. Sender 1 is not as similar to senders 2 and 3 and senders 2 and 3 are to each other because sender 1 only emails recipient a and senders 2 and 3 also email recipient b. Sender 5 emails neither recipients a nor b, so sender 5 is different from sender 1 and 2, but sender 5 has some similarity to sender 6 because both sender 5 and sender 6 email recipient e. As a result, senders 2 and 3 are the most similar to each other, and senders 1, 2, 3, and 4 as well as senders 5 and 6 are somewhat similar.

FIG.3Billustrates a sender affinity graph310and a sender taint graph320and how the same advantage can become the same disadvantage. In some embodiments, the sender affinity graph310and the sender taint graph320are generated by the pattern module204and stored in the storage device245. The thickness of the lines in the sender affinity graph310and the sender taint graph320indicate similarity between the senders based on emailing the same groups of recipients. The sender taint graph320additionally includes nodes that are progressively darker as more taint is associated with a node. The pattern module204determines that the behavior pattern of certain senders are associated. For example, if sender 2 attempted a business email compromise (BEC), the pattern module204would identify sender 3 as also likely to attempt a BEC. Conversely, because sender 4 is more distantly associated with and tainted with the actions of sender 3, sender 4 is less likely to attempt the same BEC. Lastly, because senders 5 and 6 are unassociated with sender 3, they are unaffected by the BEC. As a result, the pattern module204uses the thickness of lines in the graphs to propagate good reputation scores and bad reputation scores around the graphs.

In some embodiments, the pattern module204generates behavior patterns based on analyzing the body of the email for patterns, such as a language of the text, character encoding, an order of phrases or emojis in the text, etc. In some embodiments, the pattern module204generates behavior patterns based on characterization of email messages as a function of time, such as how much spam a sender has sent during a predetermined amount of time (e.g., the last three weeks).

In some embodiments, the pattern module204generates behavior patterns for the following variables: a label associated with an email message (e.g., clean, virus, malicious Uniform Resource Locator (URL), etc.), a number of emails from a sender, a number of recipients and domains that a sender emails, a number of times that a sender sent an email message to an n recipient (e.g., first recipient, second recipient, etc.) in the to field, a number of times that the n recipient replied. In some embodiments, the volume of traffic is highly correlated with the reputation score of a sender.

In some embodiments, the pattern module204generates criteria for determining suspicious content in an email message. In some embodiments, the suspicious content includes historical features associated with the email address and any aspect of the email message including metadata (e.g., the header of the email address), information from the SMTP protocol, time of day, body of the email message, etc. For example, the historical features may include that the email address arrives from a different route than is previously seen from the sender, the email message fails the DMARC test when email messages from the sender previously passed the DMARC test, etc.

For example, the criteria may include an association of one sender with another sender that is known to send suspicious content, domain addresses that are known to be suspicious, email messages with no text in the body of the email message but that include an attachment, a filename for an attachment that is known to be associated with malicious content, etc.

In some embodiments, the pattern module204determines a reputation score associated with a sender of an email message based on a comparison of extracted features with behavior patterns of the sender. For example, the pattern module204may use the extracted features to determine the identity of the sender and then retrieve behavior patterns generated for the sender. In some embodiments, the pattern module204may generate a profile for a sender that includes different types of extracted features, such as multiple email addresses that are associated with the same sender profile. The reputation score may indicate that a sender is associated with a high risk based on at least one of the domain names for the email address being associated with known email scams, one of the email addresses being new, etc.

In some embodiments, the pattern module204determines the reputation score based on extracted features that include an association of one sender with another and behavior patterns. For example, the pattern module204may generate a reputation score for a first user that indicates that the first user is associated with a high risk because the first user is in frequent communication with a second user that has a reputation score that indicates that the second user is extremely risky (e.g., is known to send malicious content to other users).

The real-time aspect of the pattern module204advantageously enables the pattern module204to react to new events rapidly, such as when a sender that was previously determined to be safe suddenly starts deviating from previously determined behavior patterns. The real-time aspect of the pattern module204tracks email messages as they evolve in real-time. For example, in one scenario the pattern module204identifies when unsolicited email messages are received from an unknown sender that arrive at an organization. The pattern module204tracks how some recipients respond and establish conversations and how other recipients forward the email messages to their internal Internet Technology (IT) department. Based on the recipients behavior, the pattern module204generates a behavior pattern for the unknown sender. For example, if 99% of the recipients engage with emails from the unknown sender, the pattern module204assigns a reputation score indicating that the unknown sender is likely safe.

The pattern module204receives extracted features from an email message that is addressed to a recipient. The pattern module204determines whether content of the email message matches at least one criterion for suspicious content. For example, the pattern module204may determine that the Internet Protocol (IP) address associated with the sender is associated with suspicious content. The pattern module204identifies the sender of the email message based on the extracted features.

In some embodiments, the pattern module204determines a reputation score associated with the sender based on a comparison of the extracted features with behavior patterns of the sender. For example, the pattern module204compares an extracted feature of a recipient of the email to the behavior pattern of recipients that the sender emails to determine whether the sender has been frequently communicating with this particular recipient or if this is a new recipient.

In some embodiments where the pattern module204does not generate a reputation score, the machine-learning module206trains a machine-learning model (or multiple models) to output a reputation score and updates parameters of the machine-learning model based on feedback. In some embodiments, the machine-learning module206includes a set of instructions executable by the processor235to train a machine-learning model to output the reputation score. In some embodiments, the machine-learning module206is stored in the memory237of the computing device200and can be accessible and executable by the processor235.

In some embodiments, the machine-learning module206receives training data that includes extracted features that are associated with suspicious content or safe content. For example, the training data may include an email message in which a virus was detected and the parameters (i.e., extracted features) related to the email message. In some embodiments, the determination of suspicious content or safe content are each associated with a confidence level. For example, an email may be identified as 85% likely to include suspicious content based on a reputation score associated with the sender. In some embodiments, the historical behavior and behavior patterns determined by the pattern module204are also part of the training data. In some embodiments, the training data includes information from the analytics module208, such as offline training and analytics as discussed in greater detail below with reference toFIG.6.

The machine-learning module206may train the machine-learning model, using the training data set, to classify content of an email message as suspicious content or not suspicious content. In some embodiments, the machine-learning module206generates clusters based on similarity of extracted features. For example, one cluster may relate to similarity of senders, another cluster may relate to a number of emails that were delivered during an SMTP session, another cluster may relate to results of reputation lookup, etc. Different combinations of extracted features that correspond to different clusters are possible.

The machine-learning module206may also train the machine-learning model to output a reputation score for the sender associated with an email message. In some embodiments, the machine-learning model generates a cluster of correspondence that includes an email (e.g., a graph), compares the email to the cluster of correspondence, and outputs a reputation score.

In some embodiments, the machine-learning module206receives extracted features that correspond to the email message associated with the recipient as input. The machine-learning module206outputs a reputation score for the sender based on the extracted features. In some embodiments, the machine-learning module206transmits the reputation score to the analytics module208.

In some embodiments, the machine-learning module206updates parameters for the machine-learning model based on feedback. For example, in some embodiments, a user may provide feedback indicating that an email message classified as suspicious is not suspicious. The machine-learning module206may modify a parameter related to how one or more extracted features are utilized, based on the false positive. For example, if the machine-learning module206output a reputation score for a sender based on a parameter that associated the sender with another sender that was classified as being suspicious, the machine-learning module206may modify a parameter such that the association has less of an effect on the similarity of the cluster.

In another example, the feedback may include a situation where a user identifies an email message as suspicious because it includes a phishing attempt when the machine-learning module206output a determination that the content of the email message did not match at least one criterion for suspicious content. This is a false negative. In this example, the machine-learning module206may modify the parameters of the machine-learning model to include a new criterion as being associated with suspicious content and generates a cluster for the new criterion.

The analytics module208determines whether to withhold an email message based on content of the email message matching a criterion for suspicious content and/or based on whether the reputation score meets a reputation threshold. In some embodiments, the analytics module208includes a set of instructions executable by the processor235to determine whether to withhold the email message. In some embodiments, the analytics module208is stored in the memory237of the computing device200and can be accessible and executable by the processor235.

The analytics module208receives a determination of whether content of an email message matches at least one criterion for suspicious content from the pattern module204. The analytics module208receives a reputation score for the sender of the email message from the pattern module204if the reputation score is determined using a rules-based analysis or from the machine-learning module206if the reputation score is determined using a machine-learning model.

In some embodiments, if the content of the email message matches at least one criterion for suspicious content and/or the reputation score is less than the reputation threshold, the analytics module208instructs the user interface module210to generate an alert for the recipient. In some embodiments, the analytics module208delivers the email message with the alert, but other options are possible such as delivering a notification that an email message is available but suspicious, delivering an email message with the attachments removed, etc. In some embodiments, the analytics module208instructs the user interface module210to display some information about the email message without the content of the email message and with an explanation about why the email message was not delivered.

In some embodiments, if the content of the email does not match at least one criterion for suspicious content and the reputation score meets the reputation threshold, the analytics module208delivers the email message.

In some embodiments, the analytics module208performs analysis of offline information. The offline information may be a larger data set than that used for the behavior pattern analysis because the analytics module208analyzes the offline information to determine patterns and outliers during batch processing of the data. In some embodiments, the analytics module208aggregates all data received by elements of the email application103and performs batch processing to discover patterns and outliers in the data. In some embodiments, the analytics module208transmits the patterns and outliers to the machine-learning module206to serve as training data for the machine-learning model.

In some embodiments, the analytics module208generates cached analytics from the offline analysis as well as the behavior patterns. For example, the analytics module208may generate a cache that is part of the storage device245from data that is not older than a predetermined time period. The analytics module208may compare extracted features of an email to the cached analytics and additionally determine whether to deliver an email to the recipient based on suspicious content that is identified after comparing the extracted features to the cached analytics.

The following is an example use case that includes cached analytics. In this example, the pattern module204determines that an email from “Jane Doe”<badhat45551@gmail.com> is suspicious when the recipient has previously seen emails from “Jane Doe” <Jane.Doe@Sophos.com>. In some embodiments, the scanner202stores email addresses keyed by domain name to record senders for the domain name and makes this information available as part of the cached analytics via cached analytics, such as a local data cache. The scanner202performs a lookup for each domain name encountered in live traffic, and the pattern module204compares the email address with previously seen email addresses. If the email addresses are different, the pattern module204determines that the content of the email address, namely the email address for the sender, matches a criterion for suspicious content, namely that the email addresses are different. In some embodiments, the pattern module204normalizes the domain name by performing case flattening (e.g., such that the terms “Sophos,” “SOPHOS,” and “sophos” are treated the same), whitespace normalization (e.g., such that the name “Jane Doe” and “Janedoe” are treated the same or “Jane Doe” and “Jane Doe” are treated the same), homograph flattening (e.g., such that pairs of words look the same or very similar to the eye, but actually have different letters such as replacing the Latin letter o with the Greek letter omicron or capital O with the digit 0 are treated the same), nickname normalization (e.g., “Johnnie Doe” and “Jack Doe” are mapped to the same form, such as “John Doe”), and initials normalization (e.g., “John Stuart Doe,” “John S Doe,” and “Doe, John” are all mapped to the same form, such as “John Doe”). If the recipient matches the same domain name after normalization, the analytics module208may deliver the message to the recipient because a person knows if an email is really from themselves.

The user interface module210generates a user interface. In some embodiments, the user interface module210includes a set of instructions executable by the processor235to generate the user interface. In some embodiments, the user interface module210is stored in the memory237of the computing device200and can be accessible and executable by the processor235.

The user interface module210generates graphical data that is displayed by the display241. The user interface may be displayed on a computing device as part of the email application103, as a website, or as another graphical interface, and may generally provide an interface for user interaction with the analytics module208, e.g., for email message management, warnings of suspicious content, network administration, audit, configuration and so forth. The user interface may generally facilitate action on potential threats, e.g., by presenting threats along with other supplemental information, and providing controls for a user to dispose of such threats as desired, e.g., by permitting execution or access, by denying execution or access, or by engaging in remedial measures such as sandboxing, quarantining, vaccinating, and so forth.

In some embodiments, where the analytics module208delivers the email message to a recipient, the user interface module210generates a user interface that includes the email message. The user interface may include an option to report the email message as suspicious. For example, the user interface may include a button for reporting, a link for reporting, etc.

In some embodiments, where the analytics module208determines that content of the email message matches a criterion for suspicious content and/or the reputation score of a sender is less than a reputation threshold, the user interface module210may generate a user interface with an alert and an option to ignore the alert and provide the email message to the recipient.

In some embodiments, the user interface module210generates a user interface that an administrator can use to modify settings of the email application103. For example, the user interface may include an option for configuring how email messages are treated when the email message matches a criterion for suspicious content and/or the reputation score of a sender is less than a reputation threshold.

Turning toFIG.4, an example email message400is illustrated with an alert that the email message is suspicious. In this example, the pattern module204maintains a count of emails from external senders to users that are recipients and flags email messages using the techniques discussed above. The analytics module208determines that the reputation score for the sender is less than a reputation threshold. The analytics module208delivers the email message with an explanation about the suspicious content. The user interface module210generated a graphical interface that displays the email along with the explanation that “Some people who received this message don't often get email from “Jane.Doe@Sophos.com.”

Turning toFIG.5, an example email message500is illustrated with an option to report a suspicious email. In this example, the analytics module208determined that the content of the email message does not match at least one criterion for suspicious content and that the reputation score for the sender meets a reputation threshold. As a result, the analytics module208delivers the email message to the recipient. The user interface module210generates a user interface that includes the email message and also a button505for reporting the email in the event that a user decides that the email message is suspicious.

FIG.6is an example block diagram that illustrates the interactions between different elements of the network environment600. In general, the network environment may include a scanner context605, real-time analytics610, offline training and analytics625, and cached analytics630. Each of the entities depicted inFIG.6may, for example, be implemented on one or more computing devices such as the computing device200described with reference toFIG.2. A number of systems may be distributed across these various elements to support the identification of suspicious email messages, such as the scanner context605, the real-time analytics610, the offline training and analytics625, and the cached analytics630, each of which may include software components executing on any of the foregoing system components, and each of which may support improved detection of suspicious email messages.

The scanner context605performs orchestration and scanning of incoming email messages in order to provide a scanning context during an SMTP session. The scanner context605includes feature extraction, scanning dispositions606, enrich scanner context607, and scanners608.

The orchestration aspect of the scanner context605may include what types of elements in the network environment600to use based on policy. For example, the policy may dictate that certain scanners608are not applicable if an email message originates from an administrator. In some embodiments, administrators may define and enforce policies that control access to and use of the scanner context605. Administrators may update policies such as by designating authorized users and conditions for use and access. The scanner context605may update and enforce those policies at various levels of control that are available.

The feature extraction, scanning dispositions606extract raw per-email data from the email messages, such as information from the SMTP protocol as the email messages were received, such as senders and recipients of an email message; an SMTP client Internet Protocol (IP) address associated with an email message; a number of email messages that were delivered in an SMTP session; results of a reputation lookup on the SMTP client IP address; and/or whether a sender passed one or more of a Domain-based Message Authentication, Reporting and Conformance (DMARC) test, Domain Keys Identified Mail (DKIM) test, or a Sender Policy Framework (SFP) test.

The enrich scanner context607may include details of a history of correspondence between the sender and the recipient, how many of the email messages were initiated by a customer and not by an external person, and/or what was an average length of each exchange. The data added by the enrich scanner context607content may include whether the sender has an affinity with other senders or clusters of senders and, if so, a corresponding reputation score. The data added by the enrich scanner context607may include whether the email message is part of a suspicious pattern of correspondence, such as when an unknown external sender emails certain groups of customer recipients unprompted. The data added by the enrich scanner context607may include machine-learning scores for machine-learning models that were exposed to the email message and a corresponding graph of correspondence that includes the email message.

The scanners608may each perform specific functions. For example, one scanner608amay lookup reputations of a sender in third-party DNS RBLs, another scanner608bmay lookup reputations of a sender in internal systems, and another scanner608cmay lookup reputations of a sender in internal systems with regard to specific attributes, such as a client IP address, a sender domain, and/or an EHLO domain.

The scanner context607may use different models609to perform different scanning functions. The models609may be rule-based, machine-learning models, or a combination of both (e.g., one rule-based model and a plurality of machine-learning models).

The results of the scanner context605are pushed to the real-time analytics610for behavior pattern generation and to the offline training and analytics625for batch processing.

The real-time analytics610tracks email conversation states as they evolve in real-time. In some embodiments, the real-time analytics610generates behavior patterns based on the historical behavior of emails. For example, the real-time analytics610identifies that a sender has started acting suspiciously.

In some embodiments, the real-time analytics610determines whether to deliver an email message to a recipient. For example, the real-time analytics610receives features extracted from the feature extraction, scanning dispositions606associated with the email message. In some embodiments, the real-time analytics610determines whether content of the email message matches at least one criterion for suspicious content and determines reputation scores for each of the senders based on a comparison of the extracted features to the behavior patterns. If the content of the email message matches the at least one criterion for suspicious content and/or the reputation score is less than a reputation threshold, the real-time analytics610may perform protective actions. The protective actions may include withholding the email message from the recipient, providing the email message to the recipient with certain features (e.g., an ability to download attachments) disabled, providing the email message to the recipient with an option to report the email message as suspicious, etc.

In some embodiments, the real-time analytics610also uses end user feedback events615and events from other products620.

End user feedback events615includes a recipient's reactions to the protective actions. For example, if the recipient receives the email and chooses to download the attachment, the real-time analytics610may update parameters to account for the email message being less suspicious than was previously determined. In another example, if the recipient receives the email message and reports the email message as suspicious, the real-time analytics610may reinforce parameters to emphasize that the protective actions were properly performed.

The events from other products620may include other sources of email message analysis. For example, the other products620may include any networked computer-based infrastructure. For example, the other products620may be corporate, commercial, organizational, educational, governmental, or the like. Other products620may also or instead include a personal network such as a home or a group of homes. The other products'620computer network may be distributed amongst a plurality of physical premises such as buildings on a campus, and located in one or in a plurality of geographical locations.

The other products620may include a firewall, a wireless access point, an endpoint, a server, a mobile device, an appliance or Internet-of-Things (IoT) device, and/or a cloud computing instance. The other products620may be implemented in hardware (e.g., a hardware firewall, a hardware wireless access point, a hardware mobile device, a hardware IoT device, etc.) or in software (e.g., a virtual machine configured as a server or firewall or mobile device).

The other products620may provide particular analysis of different systems in the network environment600, such as an analysis of threats to a firewall, a wireless access point, an endpoint, a server, a mobile device, an IoT, and/or a cloud computing instance. For example, the particular analysis may include Data Loss Prevention (DLP) that monitors and restricts the transfer of files containing sensitive data. The other products620may provide information about remedial actions taken when threats were present, such as sandboxing, quarantining, removing, or otherwise remediating or managing malicious code or malicious activity.

In some embodiments, the offline training and analytics625aggregates all data from the elements of the network environment600for processing. The offline training and analytics625may perform batch processing to discover patterns and outliers in the data. The offline training and analytics625includes so much data that it is a separate analysis from the real-time analytics610. In some embodiments, the offline training and analytics625provide the patterns and outliers to a machine-learning model associated with the cached analytics630.

In some embodiments, the cached analytics630receives information from both the offline training and analytics625and the real-time analytics610. The cached analytics630may discard data that is older than a predetermined time period so that the cached analytics630is fresh and a smaller amount of data that is more manageable for quick decisions. For example, the cached analytics630may define data as stale if it is older than an hour, a day, etc.

In some embodiments, the cached analytics630includes a machine-learning module that uses the patterns and outliers from the offline training and analytics625as training data to help train the machine-learning module to discriminate between email messages with suspicious content and email messages with safe content. In some embodiments, the machine-learning module receives data from the real-time analytics610to keep the machine-learning module current.

In some embodiments, the cached analytics630transmits data to the enrich scanner context607to enrich the input to various elements of the network environment600including any downstream processing elements.

In some embodiments, the cached analytics630is not part of the network environment600. Instead of storing the analytics in a cache, in some embodiments the enrich scanner context607sends a query event to the real-time analytics610and receives a response directly from the real-time analytics610containing the analytics.

FIG.7is an example flow diagram700to determine whether to deliver an email message to a recipient. The method illustrated in flowchart700is performed by a computing device200where the computing device200may be user device115, an email server101, or in part a user device115and in part an email server101.

The method700may begin at block702. At block702, behavior patterns are generated based on historical behavior of a plurality of emails. Block702may be followed by block704.

At block704, an email message is received from a sender, where the email message is withheld from delivery to a recipient. Block704may be followed by block706.

At block706, a plurality of features are extracted from the email message, Block706may be followed by block708, For example, the extracted features may be any information that was extracted by the scanner202inFIG.2.

At block708, it is determined whether the content of the email message matches at least one criterion for suspicious content. If the content of the email message matches at least one criterion for suspicious content, block708may be followed by block710. At block710, a remedial action is taken. For example, the recipient may receive an alert that the email is suspicious, an email with attachments removed, an email with a warning explaining why the email is suspicious, etc. In another example, the email may be quarantined. In some embodiments, the recipient may indicate that the email is not suspicious and, in that case, the feedback may be used to improve the process.

If the content of the email message does not match at least one criterion for suspicious content, block708may be followed by block712. At block712, a reputation score associated with the sender is determined based on a comparison of the extracted features with the behavior patterns, where the extracted features include an identity of the sender. Block712may be followed by block714. AlthoughFIG.7illustrates712as following708, other embodiments are possible, such as performing708and712independently of each other.

At block714, it is determined whether the reputation score is less than a reputation threshold. If the reputation score is less than a reputation threshold, block714may be followed by block710. At block710, the recipient is alerted that the email message is suspicious.

If the reputation score meets the reputation threshold, block714may be followed by block716. At block716, the email message is delivered to the recipient. In some examples, the email message may include an option for a user to report that the email is suspicious. If the user reports that email as suspicious, the feedback may be used to improve the process.

Reference in the specification to “some embodiments” or “some instances” means that a particular feature, structure, or characteristic described in connection with the embodiments or instances can be included in at least one implementation of the description. The appearances of the phrase “in some embodiments” in various places in the specification are not necessarily all referring to the same embodiments.

The embodiments of the specification can also relate to a processor for performing one or more steps of the methods described above. The processor may be a special-purpose processor selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory computer-readable storage medium, including, but not limited to, any type of disk including optical disks, ROMs, CD-ROMs, magnetic disks, RAMs, EPROMs, EEPROMs, magnetic or optical cards, flash memories including USB keys with non-volatile memory, or any type of media suitable for storing electronic instructions, each coupled to a computer system bus.

The specification can take the form of some entirely hardware embodiments, some entirely software embodiments or some embodiments containing both hardware and software elements. In some embodiments, the specification is implemented in software, which includes, but is not limited to, firmware, resident software, microcode, etc.