Abstract:
An automated arrangement for detecting adversaries is provided in which assessments of detected adversaries are reported to a reputation service from security devices, such as unified threat management systems in deployed customer networks. By using actual deployed networks, the number of available sensors can be very large to increase the scope of the adversary detection, while still observing real attacks and threats including those that are targeted to small sets of customers. The reputation service performs a number of correlations and validations on the received assessments to then return a reputation back to the security device in the enterprise network that can be used for blocking adversaries, but only when multiple, distinct sources report the same adversary in their assessments to thus ensure that the reputation is accurate and reliable.

Description:
STATEMENT OF RELATED APPLICATION 
       [0001]    This application claims the benefit of U.S. Provisional Patent Application Ser. No. 60/923,663, filed Apr. 16, 2007, entitled “Detection of Adversaries through Collection and Correlation of Assessments,” which is incorporated herein by reference in its entirety. 
     
    
     BACKGROUND 
       [0002]    The term computer “hacker” is increasingly used to refer to persons who engage in malicious or illegal activities to gain access to, or attack computer systems without authorization. Such activities by hackers have evolved far beyond those that were simply pranks or caused some minor inconveniences into a key component of highly organized criminal enterprises in which billions of dollars can be made each year. 
         [0003]    Hackers often seek to launch attacks on computer systems in an automated manner by using large networks called “botnets” of compromised computers called “bots” (i.e., software robots) or “drones.” While bots can be supported by hosts that bypass most local Internet content regulation (so called “bullet-proof hosts”), bots are primarily found in computers used by innocent home users who are completely unaware that their systems have been taken over and are being used for illegitimate purposes. Botnets are thought to be organized in herds as large as one hundred thousand to a half million or more bots that can be geographically spread over many countries. 
         [0004]    Botnets can employ both active and passive attacks. In an active attack, a botnet originates attacking traffic such as spam, adware, or denial of service (“DoS”) traffic which is sent over a network such as the Internet to its victims. In a passive attack, bots function as servers which, when accessed by a user, serve malware such as viruses, rootkits, trojan horses etc., typically using HTTP (Hypertext Transfer Protocol). 
         [0005]    Reputation services have been established to address the problem of automated attacks and other hacker activities by compiling black lists of URLs (Uniform Resource Locators) and IP (Internet Protocol) addresses of known adversaries. A variety of technologies such as mail relay servers, firewalls, and unified threat management gateways can query the reputation service through an online connection to decide whether to accept traffic from, or send traffic to, a given computer on the Internet. 
         [0006]    Current reputation services often run their own laboratories that are equipped with a variety of tools which are used to scan the Internet to locate adversaries and establish the reputation. These tools include web crawlers, honeypots (passive, dummy data or network sites that appear to contain information of value to attract attackers), honey monkeys (virtual computers that visit websites and seek code designed to attack a computer), virtual machines, and other global sensors. 
         [0007]    Reputation services face several significant challenges that can affect their use and success in combating hackers. For example, reputation services must reliably detect and confirm adversaries that are deployed in vast numbers all over the world. Hackers can also change URLs and IP addresses of bots frequently, so reputation services must be able to dynamically respond with equal speed to detect them and not block legitimate users who might reuse the same URL or IP address a few hours later. This problem of false positives in which URLs and IP addresses of innocent (i.e., non-malicious) computers are wrongly identified as adversaries can cause significant disruptions to users and result in high costs to service providers to resolve disputes and restore services. 
         [0008]    In addition, reputation services need to successfully deal with hackers who are increasingly targeting attacks on small sets of customer or enterprise networks that frequently go unobserved by the technologies of existing reputation services. 
         [0009]    This Background is provided to introduce a brief context for the Summary and Detailed Description that follow. This Background is not intended to be an aid in determining the scope of the claimed subject matter nor be viewed as limiting the claimed subject matter to implementations that solve any or all of the disadvantages or problems presented above. 
       SUMMARY 
       [0010]    An automated arrangement for detecting adversaries is provided in which assessments of detected adversaries are reported to a reputation service from security devices, such as unified threat management (“UTM”) systems, in deployed enterprise networks. By using actual deployed networks, the number of available sensors can be very large to increase the scope of the adversary detection, while still observing real attacks and threats including those that are targeted to small sets of customers. The reputation service performs a number of correlations and validations on the received assessments to then return a reputation back to the security device that can be used for blocking adversaries. 
         [0011]    In various illustrative examples, the assessment includes a URL or IP address of the adversary plus a severity level (e.g., low, medium, high, critical) of the incident associated with the attack or malware. The reputation service verifies that the assessment comes from a authenticated known sources, to make it expensive for a hacker to misuse assessments to damage the service through DoS attacks or the filing of false reports. Authentication may be performed using a certificate to sign assessments or other cryptographic methods. 
         [0012]    Each reported assessment is assigned a time-to-live (“TTL”) value that sets the length of the time the assessment is valid to deal with an adversary changing IP addresses and URLs of bots under its control. If, after a TTL expires, the same adversary is detected again, another assessment is sent where the TTL is increased, for example, using an algorithm that increases the TTL value with each recurring detection (e.g., from an initial TTL value of 4 hours to 8 hours, then 16 hours, and so forth upon each detection recurrence). 
         [0013]    The reputation service establishes fidelity (i.e., confidence level) of a reputation according to the number of enterprises or customers reporting the same adversary. Only when multiple, distinct enterprises report the same adversary in valid assessments (i.e., those with an unexpired TTL) will the reputation have sufficient fidelity to be sent back to the reporting enterprises to be used to actually block adversaries. 
         [0014]    Certain types of hosts such as large proxies and share web hosting sites are included in a list of known exclusions and are not subjected to blocking. In addition, privacy concerns may be addressed by hashing the adversary&#39;s URL or IP address in the assessments reported by the security devices in the network to thereby mask the identity of particular adversaries that affect particular enterprises or customers. 
         [0015]    This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. 
     
    
     
       DESCRIPTION OF THE DRAWINGS 
         [0016]      FIG. 1  shows an illustrative computer network environment in which the present arrangement for detection of adversaries through collection and correlation of assessments may be utilized; 
           [0017]      FIG. 2  shows functional components of an illustrative botnet; 
           [0018]      FIG. 3  shows the message flow between network security products and a reputation service; and 
           [0019]      FIG. 4  shows an illustrative environment in which large proxies and shared web hosting are present. 
       
    
    
       [0020]    Similar reference numerals indicate similar elements in the drawings. 
       DETAILED DESCRIPTION 
       [0021]      FIG. 1  shows an illustrative computer network environment  100  in which the present arrangement for detection of adversaries through collection and correlation of assessments may be utilized. A number of enterprise or customer networks  105 - 1  . . . N are included in the environment  100 . Customer network  105 S includes a number of hosts  108  and customer network  105   N  includes hosts  112 . Networks  105  are protected by UTMs  121 - 1  . . . N that perform a variety of functions to protect the hosts inside the networks  105  from external threats and adversaries, particularly those that are accessed via the Internet  125 . UTMs are security products that are commonly built on edge or network firewall products that incorporate many features in a single product including, for example, e-mail filtering, anti-virus (“AV”) and anti-malware (“AM”) protection, and intrusion detection, along with traditional proxy-based (i.e., application-layer) firewall functions. 
         [0022]    An illustrative botnet  132  comprises a herd  137  of compromised hosts, such as home computers with broadband connectivity. Botnet  132  typically uses the Internet  125  in its attempts to attack hosts inside the customer networks  105 . Botnet herd  137  can include any number of compromised hosts and could number in the several hundred thousands or even millions. Botnet  132  is typically spread over the globe and can thus be very difficult to deactivate. 
         [0023]    Details of botnet  132  and herd  137  are shown in  FIG. 2 . Each comprised host is typically infected with malware which includes a number of functional modules, as shown. It is emphasized that the functional modules shown in  FIG. 2  are illustrative, and a bot deployed in any given botnet might utilize only some of the modules, or additional functionality may be implemented in some cases. A spread engine  205  looks to expand the botnet further by attacking operating systems and applications on other hosts to exploit possible vulnerabilities. A concealment module  211  operates to conceal and protect the bot malware on the infected host by being configured as a rootkit that hides the bot&#39;s utilization of files, network connections, memory addresses, or registry entries to evade detection. Concealment module  211  may disable the host&#39;s anti-malware or anti-virus software. In addition, concealment module  211  may even install pirated anti-virus functionality in order to protect the host&#39;s takeover by another botnet. An anonymizer module  216  protects identifying information about the infected host from being revealed by using a SOCKS proxy server in order to transfer and retrieve data anonymously. 
         [0024]    A payloads runner  221  includes a number of functional modules to carry out the malicious purposes of the botnet  132 . These functional modules include a keylogger  227  that may be used for identity theft, a spam proxy  230  for sending spam, a back-end phishing web server  235  for hosting spoofed web sites, a DoS flooder  241  for supporting DoS for extortion or other attacks, and a click fraud module  246  that provides automated click throughs on web advertising in order to fraudulently gain per-click advertising revenue or reduce advertising costs by manipulating auctions. 
         [0025]    An update engine  250  is arranged to keep the compromised host updated in order to dynamically reconfigure the host as may be required to conduct a variety of nefarious activities (it is believed that botnet herders lease botnets to various “customers” on a time-share basis). A control agent  253  pulls commands from an IRC (Internet Relay Chat) server  265  on a bullet-proof host or hosted by another drone in the botnet  132 . A command and control center  261  provides a centralized command post that is either supported by a bullet-proof host or another drone in the botnet  132 . 
         [0026]    Returning back to  FIG. 1 , the UTMs  121  are arranged to detect adversaries, such as the bots in botnet  132 , and collect their URLs and IP addresses. Such detection is typically performed reliably by observing or blocking attack attempts against the hosts in the customer networks  105 . Such techniques for observing or blocking include, for example, observing spam sources, observing websites that host viruses, observing IP addresses that send distributed DoS attacks, and detecting command and control channel destinations to which the compromised hosts in the customer networks  105  connect. 
         [0027]    In accordance with the principles of the present adversary detection arrangement, UTMs  121  are arranged to communicate with a reputation service  140 . UTMs  121  report assessments of detected adversaries via telemetry that it uploads to the reputation service  140 . As UTMs  121 , or other security products having similar adversary detection functionalities, are commonly utilized by enterprises, businesses, and corporations, they can thus number in the hundreds of thousands to provide a very broad array of distributed adversary detection sensors. In addition, consumer products such as network routers and set top televisions terminals etc., may also be arranged to incorporate detection functionalities and thus be used to report assessments. It is emphasized that benefits of the present arrangement can be substantially realized even with a relatively small number of UTMs (e.g., a few dozen). In addition, assessments may be optionally collected from security products deployed in other networks, including home networks and other known resources, as collectively identified as third parties feeds  146  in  FIG. 1 . 
         [0028]    As shown in  FIG. 3 , each of the plurality of UTMs  121   1-N  reports the detected attacks as part of an assessment  306  that is reported to the reputation service  140 . The reputation service  140  collects all the assessments from the multiple UTMs  121 , and optionally from assessment  312  from the third party feed  146  in order to generate a reputation  318 , as discussed in detail below, which is then downloaded by the UTMs  121 . Assessments are typically stored in database  315  that is operatively coupled to the reputation service  140 , as shown. 
         [0029]    There is a possibility that a hacker might create false assessments in an attempt to damage the credibility of the reputation service  140 , or launch a DoS attack, for example on a legitimate website and falsely accuse it of participating in malicious attacks. To prevent reporting of such false assessments, the reputation service  140  authenticates the UTMs  121  making the reports through use of a unique customer identification or security certificate to prove that the reporting UTM is legitimate and not a hacker. In addition, if false information is discovered by the reputation service, it can be traced back to a source UTM, and all further assessments sent from that UTM will be disregarded. 
         [0030]    As indicated by reference numeral  325 , an assessment, in this illustrative example, includes data fields which contain the URL or IP address of the detected adversary, a severity of the incident associated with the attack by the adversary, and a time-to-live (“TTL”) value. The severity (e.g., low, medium, high, critical) describes the seriousness of an incident that is associated with the attack, which as noted above, can be both active and passive. For example, a host in customer network  105  ( FIG. 1 ) might receive an e-mail containing a virus that is detected by the UTM or local anti-virus product and removed. The severity in the assessment  306  reported to the reputation service  140  could be “low,” for example, given that the virus was detected and removed before it could cause any damage to the enterprise. 
         [0031]    Each reported assessment has an assigned TTL value that defines the time interval over which the assessment is valid. Once the TTL expires, the assessment is no longer valid. The TTL is utilized in recognition that hackers can often quickly change the URL or IP addresses of the bots in their botnets. By using the TTL, the possibility is lessened for blocking a legitimate user who subsequently uses a URL or IP address after it has been abandoned by the hacker. 
         [0032]    In this illustrative example, the default initial TTL value is four hours. However, if a UTM  121  detects the same adversary on a recurring basis, the TTL value in its subsequent reported assessments will be extended in time. Various types of algorithms may be used to extend the TTL value according to the needs of a specific application of the present adversary detection arrangement. For example, an exponentially increasing or geometrically increasing algorithm can be applied to double each TTL value with each recurring detection (e.g., 4, 8, 16 hours . . . ). 
         [0033]    As noted above, the reputation service  140  uses the collected assessments from the UTMs  121  to generate the reputation  318 . To do this, the reputation service  140  correlates the collected assessments to derive a fidelity (i.e., level of confidence) that will be associated with the reputation  318 . In some implementations, such reputation fidelity can be reported back to the UTMs  121  with various levels, for example, low, medium, or high fidelity. Alternatively, fidelity can be arranged in a binary fashion (i.e., a reputation has sufficient fidelity to be relied upon, or has no fidelity). In either case, in accordance with the principles herein, a set or predetermined amount of fidelity must be present before a reputation may be used by a UTM to block traffic. 
         [0034]    Various techniques or algorithms may be used to establish fidelity and the local rules governing assessment use may vary, but the general principle applied by all is that multiple, distinct UTMs (i.e., UTMs operating on different customer networks) must corroborate an adversary so that no single assessment is used to generate a reputation at the reputation service  140 . For example, one technique would be to require a minimum number, such as 10, valid (i.e., having unexpired TTLs) assessments identifying the same adversary received from distinct UTMs, in order for the reputation service  140  to generate a reputation  318  having high fidelity. In this example, only high fidelity reputations are allowed to be used by a UTM  121  to block an adversary. In another example, a low fidelity reputation is generated when between one and five assessments identifying the same adversary are received from distinct UTMs. But in this example, a given UTM  121  might apply a different local rule to block the adversary associated with the low fidelity reputation, but only if the UTM also detects the adversary. Thus, corroboration from an outside source, even if it is low fidelity, is sufficient evidence when combined with the UTM&#39;s own observations to warrant taking a blocking action. It may also be desirable to use a fidelity algorithm that adjusts according to the severity of the reported incidents. For example, high or critical severity incidents that have the potential to cause greater harm might require fewer assessments from multiple, distinct sources to generate a high-fidelity reputation than when the severity is lower. Another illustrative technique is to increase the fidelity of a reputation when multiple different types of attacks are launched from the same adversary. For example, a hacker (or an entity leasing a botnet) might use a botnet for spam at one given time, and then for a DoS attack, followed by a identity theft attack. In this case, the reputation service  140  can assign greater fidelity to the reputation for the source initiating these multiple types of attack, even if such multiple attacks are directed to a smaller number of nodes (e.g., UTMs  121 ) which would result in a fewer number of reported assessments to the reputation service  140 . 
         [0035]    By requiring correlation of assessments from multiple distinct sources before issuing a reputation, the present arrangement prevents a hacker from simply installing a pirated UTM in order to influence a reputation. This provides a measure of security that may be used in addition to the use of authentication of assessment sources. Should a hacker attempt to provide assessments including fraudulent or misleading data, the fact that such assessments are not corroborated by other UTMs  121  can be used as a justification for revoking the certificate for the pirated UTM. 
         [0036]    In some applications of the present arrangement, a UTM  121  sends a request  330  to the reputation service when it encounters an unknown URL or IP address to check the reputation of the URL or IP address before allowing access to the URL or accepting traffic from the IP address. In other applications, reputations are generated and sent to the UTMs  121  whenever a sufficient number of assessments are collected and correlated by the reputation service  140  irrespective of an explicit request. 
         [0037]    As indicated by reference numeral  331  in  FIG. 3 , the reputation  318  that is downloaded by the UTM  121  includes, in this illustrative example, data fields containing reputation data, fidelity, and an optionally utilized TTL value. The reputation data can provide any of a variety of information that describes the reputation of the URL or IP address. This can be a simple binary indication (i.e., the URL or IP address has a good or bad reputation), or it can include more detailed reputation information, for example an IP address is known for transmitting spam, or phishing attacks, or a URL is known to include potentially threatening downloadable content, etc. In some cases, it may be desirable to include the URL or IP address of the adversary in the reputation as a part of a blacklist that is provided to a UTM. However, the particular reputation data that is provided to a UTM will typically be based on the requirements of a specific application of the present arrangement. 
         [0038]    The fidelity field contains a fidelity indicator such as low, medium, high etc. The fidelity field can be optionally eliminated in some implementations. It is typically not used in cases where reputations are binary in nature and are thus only generated and sent when the derived fidelity reaches a predetermined threshold (and which would make reporting the actual fidelity value somewhat meaningless). The optionally-utilized TTL value in a reputation may similarly vary according to the requirements of a specific application. For example, a TTL value for the reputation  318  might be selected to be equal to the largest TTL value contained in a received assessment and a UTM  121  should block the URL or IP address only so long as the TTL remains valid. In other illustrative examples, no TTL is used and the reputation stays valid until it is explicitly revoked. 
         [0039]      FIG. 4  shows an illustrative environment  400  in which the presence of large proxies and NAT (network address translation) devices, identified by reference numeral  410 , present a special case for the present arrangement for adversary detection, as do shared web hosting sites  416 . In the first case, large proxies/NAT arrangements such as America Online (“AOL®”) can have many thousand computers, some of which may be infected hosts that function as bots in a botnet. Thus, simply blocking the proxy will block not only adversaries, but legitimate users as well and create undesirable false positives. To address this special case, a list of known exclusions  422  is maintained by the reputation service  140 . The exclusion list includes known large proxies which will be used by the reputation service  140  so that assessments identifying adversaries with IP addresses of the known large proxies will be ignored. In some implementations of the present arrangement, known large proxies can register with the reputation service  140  and be notified when attacks from hosts using their proxy are detected. The known large proxy can then be given a period of time to fix the problem before the proxy is blocked. In addition, an HTTP extension that is derived from an IP address can be used by the known large proxy as an ID for the compromised host to thereby allow a reputation to be selectively targeted to that host. 
         [0040]    For the second special case of shared web hosting sites like MySpace® and MSN Hotmail™, the known exclusion list  422  includes a white list of URLs associated with shared web hosting sites that will not be blocked since those sites are shared by many users, including legitimate users. In some implementations, it may be possible to block certain sites by path, but not by domain of the shared web hosting site. 
         [0041]    It is possible that some customers operating networks that engage in the present arrangement with a reputation service may have privacy concerns and not want to disclose information on incidents and attacks. Accordingly, as an optional feature, instead of reporting actual URLs and IP addresses in an assessment  306  ( FIG. 3 ), information derived from the URL and/or IP address is reported. For example, the URLs and IP addresses can be subjected to a conventional hash algorithm such as CRC32, SHA-1 or MD-5 before being uploaded to the reputation service  140 . Hashing the URLs and IP addresses will prevent observation, by the reputation service  140  or any middleman between the UTM and the reputation service  140 , of the particular URLs or IP addresses that were responsible for attacking a particular customer network. In order to prevent recovery of the information input into the hash function through use of a rainbow table, conventional cryptographic measures may be utilized such as increasing a hash bit number, or using a dynamic salt value (i.e., random bits) which is added to the hash, etc. 
         [0042]    Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.