Distributed system for adaptive protection against web-service- targeted vulnerability scanners

A method includes obtaining a dictionary, data for a set of web requests, and definitions of a first set of clusters associated with vulnerability scanners. The method includes identifying a set of clients that transmitted the second set of web requests. The method includes generating a second set of feature vectors, which each corresponds to one of the clients. Each element in each feature vector corresponds respectively to an entry in the dictionary. The method includes clustering the second set of feature vectors into a second set of clusters. The method includes, in response to a first distance between a selected cluster of the second set of clusters and one of the first set of clusters being less than a first predetermined distance, (i) identifying one of the set of web services that received web requests corresponding to feature vectors in the selected cluster and (ii) generating a scanning alert.

FIELD

The present disclosure relates to information security and more particularly to protecting publicly-facing web services against vulnerability scanning.

BACKGROUND

Publicly-facing web services, including web applications, generally accept connections from anywhere on the Internet. While this provides high availability for customers or clients, it also means that malicious actors are able to connect to the web service. Because of the complexity of a web service stack—including an operating system, a web server, custom code or plugins, and perhaps a hypervisor—the web service may have one or more known or unknown vulnerabilities.

A malicious actor may employ a vulnerability scanner that transmits a series of requests to the web service to probe for vulnerabilities. In the event that the web service has not updated every component of its software stack, known vulnerabilities may be found. Further, vulnerabilities may exist for which fixes are not yet available or may never be available. Once a malicious actor identifies a vulnerability, the malicious actor may exploit the vulnerability to cause any number of harms to the interests of the administrator of the web service and its legitimate clients. These harms include, for example, denial of service, corruption of data, defacing of the web service, exfiltration of data, lateral compromise of other systems, and injection of malicious code.

There are a number of known vulnerability scanners, such as the open-source Wfuzz vulnerability scanner, the open-source Nikto vulnerability scanner, the open-source Firebug browser extension, the Metasploit penetration testing framework, the SHODAN search engine, etc. A system administrator may therefore attempt to configure their web service to identify signatures of these known vulnerability scanners and prevent the web service from responding to the known vulnerability scanners. However, this approach requires substantial manual effort and is brittle, failing to account for new variations or generations of vulnerability scanners.

SUMMARY

A system includes at least one processor and a computer-readable medium configured to store instructions for execution by the at least one processor. The instructions include obtaining a dictionary and definitions of a first set of clusters associated with a plurality of vulnerability scanners. The instructions include receiving data for a second set of web requests transmitted to a set of web services. The instructions include identifying a set of clients that transmitted the second set of web requests. The instructions include generating a second set of feature vectors. For each feature vector in the second set of feature vectors, the feature vector corresponds to one of the set of web services and to one of the set of clients and each element in the feature vector corresponds respectively to an entry in the dictionary. The instructions include clustering the second set of feature vectors into a second set of clusters. The instructions include, in response to a first distance between a selected cluster of the second set of clusters and one of the first set of clusters being less than a first predetermined distance, (i) identifying one of the set of web services that received web requests corresponding to feature vectors in the selected cluster and (ii) generating an alert for an administrator of the one of the set of web services.

In other features, the instructions include obtaining a first set of web requests associated with the plurality of vulnerability scanners. The instructions include identifying patterns from the first set of web requests. Each of the patterns includes information from at least one of the first set of web requests. The instructions include creating the dictionary based on the identified patterns. The instructions include generating a first set of feature vectors. For each feature vector in the first set of feature vectors, the feature vector corresponds to one of the plurality of vulnerability scanners and each element in the feature vector corresponds respectively to an entry in the dictionary. The instructions include clustering the first set of feature vectors into the first set of clusters.

In other features, the instructions include dividing the identified patterns into non-overlapping first and second subsets. Patterns in the first subset are more frequent in the first set of web requests than are patterns of the second subset. The dictionary is created from the first subset exclusive of the second subset. In other features, the first set of web requests are hypertext transfer protocol (HTTP) requests. Each of the HTTP requests is associated with an Internet Protocol (IP) address, a user agent string, and a path. Each of the identified patterns includes a uniform resource identifier (URI) from each of at least one of the HTTP requests.

In other features, for each pattern of the identified patterns, the pattern includes a uniform resource identifier (URI) from a respective plurality of the first set of web requests. A time interval between an earliest one of the respective plurality of the first set of web requests and a latest one of the respective plurality of the first set of web requests is less than a predetermined time interval. In other features, the instructions include, in response to the first distance being greater than the first predetermined distance but less than a second predetermined distance, (i) adding the selected cluster to the second set of clusters and (ii) generating an alert for the administrator.

In other features, the instructions include, in response to the first distance being greater than the second predetermined distance but less than a third predetermined distance, (i) proposing addition of the selected cluster to the second set of clusters and (ii) selectively generating an alert for the administrator. The instructions include, in response to analyst input that the selected cluster represents vulnerability scanning activity, adding the selected cluster to the second set of clusters.

In other features, the instructions include, in response to the first distance being less than the first predetermined distance, selectively updating the one of the first set of clusters to encompass the selected cluster. In other features, the first distance is determined based on cosine similarity between a centroid of the selected cluster and a centroid of the one of the first set of clusters. In other features, the one of the first set of clusters is chosen such that the first distance is less than or equal to respective distances between the selected cluster and remaining ones of the first set of clusters.

In other features, the second set of web requests are hypertext transfer protocol (HTTP) requests. Each of the HTTP requests is associated with an Internet Protocol (IP) address, a user agent string, and a path. Each of the set of clients is associated with a unique combination of IP address and user agent string. In other features, the instructions include, in response to a request from the administrator, sending a command to a packet filter to transmit an alert when a packet stream targeted at the one of the set of web services matches a signature that is based on the first set of clusters. In other features, the instructions include, in response to a request from the administrator, sending a command to the packet filter to temporarily blacklist a sender when the packet stream matches the signature.

A method includes obtaining a dictionary and definitions of a first set of clusters associated with a plurality of vulnerability scanners. The method includes receiving data for a second set of web requests transmitted to a set of web services. The method includes identifying a set of clients that transmitted the second set of web requests. The method includes generating a second set of feature vectors. For each feature vector in the second set of feature vectors, the feature vector corresponds to one of the set of web services and to one of the set of clients and each element in the feature vector corresponds respectively to an entry in the dictionary. The method includes clustering the second set of feature vectors into a second set of clusters. The method includes, in response to a first distance between a selected cluster of the second set of clusters and one of the first set of clusters being less than a first predetermined distance, (i) identifying one of the set of web services that received web requests corresponding to feature vectors in the selected cluster and (ii) generating a scanning alert for an administrator of the one of the set of web services.

In other features, the method includes obtaining a first set of web requests associated with the plurality of vulnerability scanners. The method includes identifying patterns from the first set of web requests. Each of the patterns includes information from at least one of the first set of web requests. The method includes creating the dictionary based on the identified patterns. The method includes generating a first set of feature vectors. For each feature vector in the first set of feature vectors, the feature vector corresponds to one of the plurality of vulnerability scanners and each element in the feature vector corresponds respectively to an entry in the dictionary. The method includes clustering the first set of feature vectors into the first set of clusters.

In other features, the method includes dividing the identified patterns into non-overlapping first and second subsets. Patterns in the first subset are more frequent in the first set of web requests than are patterns of the second subset. The dictionary is created from the first subset exclusive of the second subset. In other features, the first set of web requests are hypertext transfer protocol (HTTP) requests. The second set of web requests are HTTP requests. Each of the HTTP requests is associated with an Internet Protocol (IP) address, a user agent string, and a path. Each of the identified patterns includes a uniform resource identifier (URI) from each of at least one of the HTTP requests. Each of the set of clients is associated with a unique combination of IP address and user agent string.

In other features, for each pattern of the identified patterns, the pattern includes a uniform resource identifier (URI) from a respective plurality of the first set of web requests and a time interval between an earliest one of the respective plurality of the first set of web requests and a latest one of the respective plurality of the first set of web requests is less than a predetermined time interval.

In other features, the first distance is determined based on cosine similarity between a centroid of the selected cluster and a centroid of the one of the first set of clusters. The method includes, in response to the first distance being less than the first predetermined distance, selectively updating the one of the first set of clusters to encompass the selected cluster. The method includes, in response to the first distance being greater than the first predetermined distance but less than a second predetermined distance, (i) adding the selected cluster to the second set of clusters and (ii) generating a scanning alert for the administrator. The method includes, in response to the first distance being greater than the second predetermined distance but less than a third predetermined distance, (i) proposing addition of the selected cluster to the second set of clusters and (ii) selectively generating a scanning alert for the administrator, and (iii) in response to analyst input that the selected cluster represents vulnerability scanning activity, adding the selected cluster to the second set of clusters.

A computer-readable medium stores processor-executable instructions including obtaining a dictionary and definitions of a first set of clusters associated with a plurality of vulnerability scanners. The instructions include receiving data for a second set of web requests transmitted to a set of web services. The instructions include identifying a set of clients that transmitted the second set of web requests. The instructions include generating a second set of feature vectors. For each feature vector in the second set of feature vectors, the feature vector corresponds to one of the set of web services and to one of the set of clients and each element in the feature vector corresponds respectively to an entry in the dictionary. The instructions include clustering the second set of feature vectors into a second set of clusters. The instructions include, in response to a first distance between a selected cluster of the second set of clusters and one of the first set of clusters being less than a first predetermined distance, (i) identifying one of the set of web services that received web requests corresponding to feature vectors in the selected cluster and (ii) generating a vulnerability scanning alert for an administrator of the one of the set of web services.

DETAILED DESCRIPTION

Introduction

Once activity from a vulnerability scanner has been identified, that activity can be used to create a signature for the vulnerability scanner. That signature can then be used to detect future operation of instances of that vulnerability scanner. However, as the vulnerability scanner is updated, forked, or intentionally obfuscated, detecting operation of the vulnerability scanner becomes much more difficult. While skilled system administrators may be able to review logs to identify vulnerability scanning activity, less experienced system administrators or businesses without dedicated system administrators may be unable to detect modified vulnerability scanning behavior.

The present disclosure therefore describes a technological approach to automating what previously required human expertise: characterizing vulnerability scanner activity and creating signatures for the vulnerability scanning activity. Further, the present disclosure describes adaptively updating those signatures as the vulnerability scanning behavior changes. The present disclosure may allow for real-time (that is, within seconds) notification of web service administrators regarding vulnerability scanning. In other implementations, notification follows a capture interval (such as one hour, two hours, or 24 hours). In addition, based on the vulnerability scanner signatures, vulnerability scanning traffic may be proactively blocked by a packet filter.

Vulnerability scanning behavior may be characterized by sets of hypertext transfer protocol (HTTP) requests. For example, certain pairs, triplets, etc. of HTTP requests may be indicative of a certain vulnerability scanner. When these groups of requests are seen, the activity can be attributed to the vulnerability scanner. To evaluate traffic, then, groups of N requests are formed from the traffic.

N-groups of HTTP requests may be formed by taking all of the HTTP requests from a certain sender within a specified time interval and looking at each combination of N requests from the collected requests. For example, that time interval could be 10 seconds, 30 seconds, or one minute. For example, in a case where N=3 (the N-group is a triplet) and 20 requests have been received within a defined time interval, the N-groups used for evaluation are each of the 1140 combinations of 3 requests chosen from the 20 requests. Each of the N-groups of requests is called a pattern.

To establish an initial dictionary, the activity of multiple known vulnerability scanners is analyzed. Specifically, the HTTP requests transmitted by the vulnerability scanners are analyzed. The logs for known vulnerability scanners may be limited to only vulnerability scanner traffic or may more broadly contain everything from certain known bad IP addresses.

Every group of N HTTP requests transmitted within a certain time interval is identified as a feature. In addition, features may include groups of P requests (P is different than N) and Q requests (Q is different than P and N). For example, 2-groups, 3-groups, and 4-groups could all be identified as features. For simplicity of explanation, the remainder of the disclosure describes features as being groups of two requests.

Once the features have been determined for the known vulnerability scanners, the frequency of the features is analyzed. Those features that are more common are added to a dictionary, while the rarer features are excluded from the dictionary. The frequency of features is determined across the set of known vulnerability scanners and across the set of web services being monitored so that malicious activity, which should have lower variation than legitimate user activity, will stand out. The malicious activity is not only repeated by multiple instances of a scanning platform but is also repeated against multiple targets, leading to higher frequencies of related features.

Once the feature dictionary is produced, a vector is determined for each vulnerability scanner. The vector has an element for every entry in the dictionary. The element may be binary, indicating whether the dictionary pattern occurred in the vulnerability scanner traffic, or may be numeric indicating how many times the dictionary pattern occurred in the vulnerability scanner traffic. The feature vectors for all of the vulnerability scanners can then be analyzed by a clustering mechanism, such as k-means clustering. These clusters of feature vectors then characterize vulnerability scanner activity.

Once clusters of known vulnerability scanner activity are determined, logs of HTTP requests can be analyzed for similarity to the known vulnerability scanner activity. For example, HTTP requests received by multiple web services can be analyzed and, for each client (defined by IP address and user agent) exchange with a web service, a feature vector is determined. The client-service feature vector includes an element for each entry in the dictionary, indicating whether or how many times that feature was present in the traffic sent by the client to the web service.

All of these feature vectors are then clustered and the resulting clusters are compared to the known vulnerability scanner clusters. In the case that a detected cluster overlaps with a vulnerability scanner cluster, this is an indication that the vulnerability scanner was active and the system administrator for the corresponding web service can be notified. If a cluster of observed traffic is close to a cluster of a known vulnerability scanner, this new cluster may be considered an evolution or variation of the vulnerability scanner and added to the set of vulnerability scanner clusters. The system administrator is again notified that vulnerability scanning activity has occurred. Meanwhile, clusters that are further away may require review by a security analyst (referred to as an operator below) to determine whether they are in fact variations of a vulnerability scanner or benign traffic. Finally, clusters of observed traffic at a greater distance from clusters of known vulnerability scanners are ignored as regular traffic.

Environment

Below are simplistic examples of a distributed computing environment in which the systems and methods of the present disclosure can be implemented. Throughout the description, references to terms such as servers, client devices, applications and so on are for illustrative purposes only. The terms server and client device are to be understood broadly as representing computing devices with one or more processors and memory configured to execute machine readable instructions. The terms application and computer program are to be understood broadly as representing machine readable instructions executable by the computing devices.

FIG. 1shows a simplified example of a distributed computing system100. The distributed computing system100includes a distributed communications system110, one or more client devices120-1,120-2, . . . , and120-M (collectively, client devices120), and one or more servers130-1,130-2, . . . , and130-N (collectively, servers130). M and N are integers greater than or equal to one. The distributed communications system110may include a local area network (LAN), a wide area network (WAN) such as the Internet, or other type of network. The client devices120and the servers130may be located at different geographical locations and communicate with each other via the distributed communications system110. The client devices120and the servers130connect to the distributed communications system110using wireless and/or wired connections.

The client devices120may include smartphones, personal digital assistants (PDAs), tablets, laptop computers, personal computers (PCs), etc. The servers130may provide multiple services to the client devices120. For example, the servers130may execute software applications developed by one or more vendors. The servers130may host multiple databases that are relied on by the software applications in providing services to users of the client devices120.

The client devices120are able to connect to a web service; however, there is no guarantee that all of the client devices120are legitimate users. Instead, one or more of the client devices120may be executing a vulnerability scanner application. The servers130may individually or collectively implement systems according to the present disclosure. For example, the server130-1may operate a web service, while the server130-2may operate a firewall protecting the web service executing on the server130-1. The server130-N may implement an analysis engine to identify vulnerability scanning activity directed at the server130-1.

FIG. 2shows a simplified example of the client device120-1. The client device120-1may typically include a central processing unit (CPU) or processor150, one or more input devices152(e.g., a keypad, touchpad, mouse, touchscreen, etc.), a display subsystem154including a display156, a network interface158, memory160, and bulk storage162.

The network interface158connects the client device120-1to the distributed computing system100via the distributed communications system110. For example, the network interface158may include a wired interface (for example, an Ethernet interface) and/or a wireless interface (for example, a Wi-Fi, Bluetooth, near field communication (NFC), or other wireless interface). The memory160may include volatile or nonvolatile memory, cache, or other type of memory. The bulk storage162may include flash memory, a magnetic hard disk drive (HDD), and other bulk storage devices.

The processor150of the client device120-1executes an operating system (OS)164and one or more client applications166. The client applications166include an application that accesses the servers130via the distributed communications system110.

FIG. 3shows a simplified example of the server130-1. The server130-1typically includes one or more CPUs or processors170, a network interface178, memory180, and bulk storage182. In some implementations, the server130-1may be a general-purpose server and include one or more input devices172(e.g., a keypad, touchpad, mouse, and so on) and a display subsystem174including a display176.

The network interface178connects the server130-1to the distributed communications system110. For example, the network interface178may include a wired interface (e.g., an Ethernet interface) and/or a wireless interface (e.g., a Wi-Fi, Bluetooth, near field communication (NFC), or other wireless interface). The memory180may include volatile or nonvolatile memory, cache, or other type of memory. The bulk storage182may include flash memory, one or more magnetic hard disk drives (HDDs), or other bulk storage devices.

The processor170of the server130-1executes an operating system (OS)184and one or more server applications186, which may be housed in a virtual machine hypervisor or containerized architecture. The bulk storage182may store one or more databases188that store data structures used by the server applications186to perform respective functions.

Block Diagrams

InFIG. 4, a platform as a service204implements a first web application (app)208for a first tenant and a second web app212for a second tenant. The platform as a service204may be a distributed computing system including numerous compute, networking, and storage resources. As an example only, the platform as a service204may be the Azure web hosting platform from Microsoft Corp. While described in this figure as a platform as a service, the principles of the present disclosure apply to any web services or apps, whether hosted on premises, in rented facilities, or in a hosted environment. Further, the principles of the present disclosure apply to web services running directly on native operating systems, hypervisors, containers, etc.

As the web apps208and212operate, log files are generated and stored into log storage216. While shown as a unitary block, the log storage216may be physically or logically separate for the web apps208and212and may be distributed across multiple storage resources and even across multiple geographic regions. As two examples, the logs stored by the log storage216may take the form of text files or database entries.

An analysis engine240periodically retrieves logs from the log storage216and analyzes the logs for vulnerability scanner activity. Identified vulnerability scanner activity is alerted to a security center244. The security center244may transmit an alert to a corresponding system administrator. The alert may take the form of an email, a text message, an app notification, or any other communication framework, such as the Slack messaging system from Slack Technologies. Additionally or alternatively, the security center244may retain the alert for display to a system administrator upon accessing the security center244, such as through a web portal.

As an example, an administrator of the first tenant is represented schematically at248. The administrator248manages the first web app208and communicates with the security center244to establish security policies for the first web app208as well as to monitor alerts, such as warnings of vulnerability scanning activity directed at the first web app208. The security center244may also provide administrator248with suggestions for remediation. For example, the security center244may indicate to the administrator248that certain modules of the first web app208need to be upgraded. In addition, the security center244may suggest blocking certain ports or certain uniform resource identifiers (URIs) to decrease the exposure of the first web app208.

An operator (also called a security analyst)252associated with the analysis engine240provides vulnerability scanner activity logs to the analysis engine240. These logs are used by the analysis engine240to determine clusters of vulnerability scanner activity. The operator252may refer to a person, a role, or a team related to security analysis.

The analysis engine240may identify a cluster of activity that is close to existing vulnerability scanner activity but not close enough to be confident of such a determination. The analysis engine240may therefore propose the cluster to a review framework260. The operator252can then, via the review framework260, determine whether the proposed cluster is associated with vulnerability scanning behavior or is benign activity.

Determinations made by the operator252are provided to the analysis engine240by the review framework260. If the proposed cluster does not represent vulnerability scanner activity, the analysis engine240may discard the proposed cluster. The review framework260may also allow the operator252to identify vulnerability scanner fingerprints264that form part of a ruleset268in a distributed firewall272. The distributed firewall272intercepts traffic between the distributed communications system110and the web apps208and212.

A packet filter276of the distributed firewall272may make decisions about where to route packets and whether to drop certain packets based on the ruleset268. For example, when certain packets match the vulnerability scanner fingerprints264of the ruleset268, the packet filter276may drop those packets or may transmit an alert to the security center244. When the web apps208and212communicate using HTTP Secure (HTTPS), the packet filter may securely store private keys from the web apps208and212in order to access encrypted HTTP headers. In a more secure implementation, the packet filter276may act as an HTTPS proxy, where the web apps208and212are configured to trust the certificate of the packet filter276. The packet filter276can then present its certificate to clients, analyzing traffic and then re-encrypting the data for transmission to the web apps208and212.

InFIG. 5, an example implementation of the analysis engine240includes an ingest module304that receives logs from the log storage216ofFIG. 4. The ingest module304may apply processing to the logs and store the preprocessed logs into storage308. For example, the ingest module304may take logs from different sources in different formats, and potentially having different schemas, and perform extract, transform, load (ETL) operations to harmonize the logs into a single schema.

In addition, the ingest module304may perform some amount of filtering or normalization. For example, user agent strings may be normalized to reduce the ability of a vulnerability scanner to avoid detection by simply fuzzing the user agent string. In addition, the ingest module304may discard portions of the path in the HTTP request to allow for more uniform comparisons of HTTP requests. In other words, the uniform resource identifier (URI) associated with an HTTP request may be a subset of the path contained in the HTTP request. The subset may not always be a proper subset.

As an example, consider multiple web apps hosted at the same domain, with the web apps being distinguished by the first portion of the path. Without removing that first portion from the path, two identical requests going to two different web apps at the same domain will have different paths. In such cases, the ingest module304may excise the portion of the path indicating the web app so that the resulting URIs will match. In other implementations, the ingest module304may excise everything prior to the final forward slash of the path.

A feature extraction module312identifies features of the logged traffic for each client of each web service. The client may be defined by the combination of source Internet Protocol (IP) address and user agent string. Dictionary storage316defines features of interest, which form elements of the feature vector for the client-service traffic.

A machine learning module320may use supervised or unsupervised learning to identify traffic indicative of vulnerability scanning. For example, the machine learning module320may be trained with known vulnerability scanning traffic as well as known benign traffic. In the disclosure below, the machine learning module320is described as being implemented by unsupervised classification. First, as described in more detail below, the machine learning module320is provided with known vulnerability scanner activity. The machine learning module320defines a dictionary relevant to features of the known vulnerability scanner activity and stores the dictionary entries into dictionary storage316.

The machine learning module320uses the dictionary entries to determine feature vectors of vulnerability scanners and clusters the feature vectors to store definitions of vulnerability scanner clusters into cluster definition storage324. Further, the machine learning module320may store the feature vectors into feature vector storage328for future clustering analyses. While the cluster definition storage324maintains information about the clusters of vulnerability scanner activity, the actual vulnerability scanner activity of individual vulnerability scanners may be maintained by the feature vector storage328.

The feature vector storage328may include its own dictionary entries, as the dictionary entries in the dictionary storage316may vary over time. For example, while the dictionary storage316may include the most frequent features of vulnerability scanners, the feature vector storage328may store more or all of the feature vectors of vulnerability scanners. In this way, over time, certain features may cumulatively become more prominent and may then merit inclusion in the dictionary storage316.

Once the machine learning module320has stored the cluster definitions of vulnerability scanners into the cluster definition storage324, the machine learning module320can evaluate new feature vectors to determine whether they cluster with the known vulnerability scanner cluster definitions from the cluster definition storage324.

The machine learning module320activates an alert module340in response to identifying clusters of activity that match clusters of vulnerability scanner activity. When the machine learning module320identifies a cluster of activity that is similar to known vulnerability scanner activity, this cluster may be provided as a proposed cluster by a cluster proposal module344for evaluation by the operator252ofFIG. 4. Operator feedback is received by the machine learning module320. If the operator identifies a proposed cluster as representing vulnerability scanner activity, the machine learning module320stores definitional information about that cluster (such as a centroid of the cluster) into cluster definition storage324.

InFIGS. 6A-6C, cluster definitions for known vulnerability scanner activity are determined. The analysis below is described for illustration only using explicit loops and nested loops. Actual implementations may, for efficiency, execute much of the analysis in parallel, such as by using a distributed computation system. For example only, a map-reduce framework may be used to parse HTTP logs and create feature vectors. In another example, U-SQL queries may be used to perform the analysis.

InFIG. 6A, control begins by command of an operator at404, where control obtains vulnerability scanner logs from the operator. At408, control selects the first vulnerability scanner from the logs for processing. At412, control prepares a chronological list of HTTP requests made by the selected vulnerability scanner. At416, control initializes an empty pattern store. At420, control selects the first HTTP request from the chronological list. At424, control determines whether there are any HTTP requests proximate to the selected HTTP request. If so, control transfers to428; otherwise, control transfers to432. A proximate request is an HTTP request that is received within a predetermined window of time after the selected request.

At428, for each of the HTTP requests qualifying as a proximate request, control adds the pair of the selected request and the proximate request as a pattern to the pattern store. The pattern may be formed from a uniform resource identifier (URI) of the selected request and a URI of the proximate request. For example only, the URI of the selected request may be some or all of a path contained in the selected request while the URI of the proximate request may be some or all of a path contained in the proximate request. Control then continues at432.

At432, control determines whether the selected request is the last in the chronological list of HTTP requests. If so, control transfers to436; otherwise, control transfers to440. At440, control selects the next HTTP request and returns to424. At436, control saves the pattern store as the signature for the selected vulnerability scanner. At444, control determines whether logs were obtained for additional vulnerability scanners. If so, control transfers to448; otherwise, control transfers to460inFIG. 6B. At448, control selects the next vulnerability scanner and returns to412.

At460inFIG. 6B, control determines whether a pattern dictionary has been established. If so, control continues at464; otherwise, control transfers to468. At468, control initializes an empty dictionary and continues at464. At464, control initializes a candidate list for candidate patterns for potential addition to the dictionary. At466, control selects the first vulnerability scanner signature. At472, control selects the first pattern from the selected signature. At476, control determines whether the pattern is in the candidate list. If so, control transfers to480; otherwise, control transfers to484.

At476, the pattern may be compared to the candidate list preserving order (as a permutation) or irrespective of order (as a combination). For example, a group of two requests may be considered the same pattern regardless of which order the requests were made. In other implementations, the order may be considered. When the pattern is added to the candidate list in484, if order is irrelevant the pattern may be structured so that the order of the requests in the pattern is sorted according to a predetermined rubric. For example, the requests may simply be sorted alphabetically. When order matters, the pattern may be sorted chronologically or reverse chronologically.

At484, control adds the pattern to the candidate list and sets a metadata value, count, associated with the candidate list entry to 0. Control then continues at480. At480, control increments the count value for the pattern in the candidate list. At488, control determines whether there are additional patterns in the selected signature. If so, control transfers to492; otherwise, control transfers to496. At492, control selects the next pattern in the selected signature and returns to476. At496, if there are additional signatures for further vulnerability scanners, control transfers to500; otherwise, control transfers to504. At500, control selects the next signature for further vulnerability scanners and returns to472.

At504, control determines a count threshold. This establishes which patterns in the dictionary will be added to the dictionary and which are too infrequent to retain. For example, the count threshold may be a fixed number or may be determined as a percentage of the highest count, in either the dictionary or the candidate list. In other implementations, the count threshold may be a count that is one or more standard deviations below the mean of the counts in the dictionary. In other implementations, the count threshold may be determined such that a fixed number of patterns are stored in the dictionary. In other words, the count threshold may be set to limit the dictionary to a certain number (such as 10,000) of the most common patterns. In yet another implementation, the count threshold may be set so that a certain percentage (such as 50%) of the patterns in the candidate list are added to the dictionary. In other implementations, the count threshold may be set at a percentage, such as 50%, of the median value of counts in the dictionary.

At508, control selects the set of patterns in the candidate list whose count is greater than the count threshold. The selected set is a subset of the set of patterns, which may not necessarily be a proper subset and which may be the empty set. At512, the selected set of patterns is added to the dictionary. In various implementations, patterns in the candidate list that are already present in the dictionary may be removed from the candidate list; the counts for those patterns may be added to the counts in the dictionary. In various implementations, patterns in the candidate list that were already present in the dictionary may be removed before determining the count threshold at504.

At516, control adjusts stored vectors based on updates to the dictionary. For example, when new entries are added to the dictionary, zero-valued elements are added to vectors that did not previously have elements corresponding to the new dictionary entries. In some implementations, patterns may be deleted from the dictionary, such as when the patterns have not been observed for a predetermined period of time, or when the rate of observation has been below a threshold for the predetermined period of time. When a vector includes entries for a dictionary entry that has been deleted, those elements of the vector are also deleted. In various implementations, the control at516may be omitted, such as when vectors are not individually retained in storage.

Control continues at540inFIG. 6C, where control selects the first vulnerability scanner signature. At544, control creates a vector for the selected signature. The vector has one value (binary or numeric) for each pattern in the dictionary. This vector may be determined by parsing the pattern store for the selected vulnerability scanner. Various techniques for increasing efficiency may be implemented, such as creating the vector for the selected signature while the patterns are being identified from the HTTP requests of the vulnerability scanner. Then, the vector can be adjusted based on changes to the dictionary and parsing through the pattern store again may be avoided.

At548, control determines whether there are additional vulnerability scanner signatures. If so, control transfers to552; otherwise, control transfers to556. At552, control selects the next vulnerability scanner signature and returns to544. At556, control determines whether there are prior vulnerability scanner vectors available. If so, control transfers to560; otherwise, control transfers to564. At560, control combines the prior vulnerability scanner vectors with the currently determined vulnerability scanner vectors from544. Control then continues at568. At564, control determines whether vulnerability scanner cluster definitions have been previously stored. If so, control transfers to572; otherwise, control transfers to568.

At568, control performs a clustering analysis on the vectors to determine vulnerability scanner clusters. For example, the clustering analysis may use k-means clustering. Control then continues at576. At572, control loads the prior vulnerability scanner cluster definitions. At580, control determines whether all of the vulnerability scanner vectors cluster with prior vulnerability scanner clusters. If so, control transfers to576; otherwise, control transfers to584. At584, control determines new vulnerability scanner clusters for vulnerability scanner vectors that do not cluster with the prior vulnerability scanner clusters. Control then continues at576.

At576, control assigns each of the vulnerability scanner vectors to the respective closest vulnerability scanner cluster. The closeness of a vector to a cluster may be determined using cosine similarity between the vector and a centroid of the cluster. At588, control determines definitions of the vulnerability scanner clusters based on the assigned vulnerability scanner vectors. For example, this may include identifying a centroid of each cluster. At592, control saves the definition data for the vulnerability scanner clusters. At596, control optionally stores the vulnerability scanner vectors for future re-clustering analysis. Control then ends.

InFIGS. 7A-7C, an example control for evaluating unknown web service logs is presented. Control begins on a periodic schedule at604. At604, control obtains HTTP logs of web applications (apps). For example, these logs may be obtained from a single repository or from multiple distributed locations. At608, control preprocesses the HTTP logs, such as described above with respect to the ingest module304.

At612, control selects the first web app whose logs were obtained. At616, control parses the logs for the selected web app to identify a list of remote IP addresses that accessed the selected web app. At620, control selects a first IP address from the IP list. The sorting of the IP list may be numerical or may simply be the order in which the IP addresses were encountered in the logs. At624, control selects the first user agent string associated with the selected web app in the selected IP address. In combination, the selected user agent, the selected IP address, and the selected web app establish a server-client triplet. This pairing is then evaluated inFIG. 7Bat640.

At640, control initializes a vector whose length is equal to the number of patterns of the dictionary and whose elements are all set to either binary zero or integer zero. At644, control creates a chronological list of HTTP requests for the selected client-server triplet. At648, control selects the first HTTP request from the chronological list.

At652, control determines whether there are any proximate requests within a predetermined window of time following the selected request. If so, control transfers to656. Otherwise, control transfers to660. At656, control selects the first pair of selected request and proximate request from among the proximate requests. At664, control determines whether the selected pair is in the dictionary. If so, control transfers to668; otherwise, control transfers to672. Determining whether the selected pair is in the dictionary may be performed based on order or irrespective of order. If irrespective of order, the pair may be ordered according to a predetermined rule so that the order will match the order of any existing pair in the dictionary.

At668, control increments the corresponding element in the vector. Because the elements begin at zero, the element in the vector indicates the number of times the pair (feature) appears in this client-server interaction. In various other implementations, the vector elements may be binary, in which case the corresponding element is simply set, such as to a binary1. Each additional time that feature is encountered, the binary value is simply set once again. Control then continues at672.

At672, control determines whether there are additional proximate requests. If so, control transfers to676; otherwise, control transfers to660. At676, control selects the next pair of selected request and proximate request and returns to664. At660, control determines whether the selected request is the last in the chronological list. If so, control transfers to680; otherwise, control transfers to684. At684, control selects the next HTTP request and returns to652. At680, control stores the created vector for the selected client-server server triplet and returns to700inFIG. 7A.

InFIG. 7Aat700, control determines whether there are additional user agent strings associated with the selected IP address for the selected web app. If so, control transfers to704; otherwise, control transfers to708. At704, control selects the next user agent string and transfers to640ofFIG. 7Bto process this next client-server triplet. At708, control determines whether there are additional IP addresses in the IP list for the selected web app. If so, control transfers to712; otherwise, control transfers to716. At712, control selects the next IP address from the IP list and continues at624. At716, control determines whether logs were ingested for additional web apps. If so, control transfers to720; otherwise, control transfers to740inFIG. 7C. At720, control selects the next web app and continues at616.

At740inFIG. 7C, control loads vulnerability scanner cluster definitions. At744, control groups vectors determined inFIG. 7Binto a set of clusters, such as by using k-means clustering. At748, control selects the first cluster of the set of clusters. At752, control identifies the vulnerability scanner cluster having a minimum distance to the selected cluster. For example, the distance may be determined using cosine similarity between the centroids of the clusters.

At756, control determines whether the distance between the identified cluster and the selected cluster is below a low threshold. If so, control transfers to760; otherwise, control transfers to764. At764, control determines whether the distance between the identified cluster and the selected cluster is less than a medium threshold. If so, control transfers to768; otherwise, control transfers to772. At772, control determines whether the distance between the identified cluster and the selected cluster is less than a high threshold. If so, control transfers to776; otherwise, control transfers to780.

The thresholds are described as low, medium, and high only relative to each other: the low threshold is simply defined as being less than the medium threshold, while the medium threshold is defined as being less than the high threshold. In other words, there is no subjective standard for whether a threshold is low, medium, or high.

At760, control updates the definition of the identified vulnerability scanner cluster to encompass the selected cluster. Control continues at784, where a vulnerability scan is reported to the administrator(s) of the web apps experiencing the identified vulnerability scanning activity. For example, the reporting may be made to the security center244ofFIG. 4. Control then continues at780.

At768, control stores the definition of the selected cluster as a new vulnerability scanner cluster given its closeness to the existing vulnerability scanner cluster. The vulnerability scanning activity is reported to administrators of the affected web services at788. Control continues at792, where the addition of the new vulnerability scanner cluster is reported to the operator. In addition to awareness of new vulnerability scanner activity, this allows the operator to vet the new vulnerability scanner cluster and identify false positives. Control continues at780.

At776, control stores the definition of the selected cluster as a potential vulnerability scanner cluster. At796, control selectively reports the potential vulnerability scanning activity to the administrator. Whether this potential vulnerability scanning is reported may be under control of the operator and also may be controlled by the administrator. For example, the administrator may want to turn off reports of potential vulnerability scanning to avoid false positives. Control continues at800, where the potential vulnerability scanner cluster is reported to the operator for review and possible inclusion as a vulnerability scanner cluster for future analyses. Control continues at780.

At780, control determines whether there are additional clusters generated by744. If so, control transfers to804; otherwise, analysis has concluded and control ends. At804, control selects the next cluster and returns to752.

InFIG. 8, example operation of portions of a security center, such as the security center244ofFIG. 4, are presented. Control begins at904, where an authentication interface is exposed to administrators. For example, the authentication interface may take the form of a web-based login interface. Authentication may rely on a username and password, optionally supplemented by multifactor authentication, and may integrate with single sign-on solutions. At908, if authentication has been successful, control transfers to912; otherwise, control returns to904.

At912, control presents a graphical user interface to the administrator. At916, control determines whether the administrator is associated with multiple web applications. If so, control transfers to920; otherwise, control transfers to924. At920, control solicits administrator selection of the web app of interest. Control then continues at928. At924, control selects the only associated web app and continues at928.

At928, control determines whether an administrator request to view alerts has been received. If so, control transfers to932; otherwise, control transfers to936. At932, control displays identified vulnerability scanning activity for the selected web application. Control continues at940, where control displays links to resources and actions that can mitigate the risk of vulnerability scanning. For example, these links may point to download pages for updated modules or to administrator panels that allow for updating or changing of security settings. Control continues at936.

At936, control determines whether an administrator request to adjust real-time vulnerability scanning detection has been received. If so, control transfers to944; otherwise, control transfers to948. At948, control determines whether the login is still valid for the administrator. If so, control transfers to912; otherwise, control returns to904. The login may remain valid for a predetermined period of time or until the administrator logs out or closes a browser window.

At944, control displays real-time options and solicits administrator input. At952, control determines whether the administrator has requested to disable real-time vulnerability scanning detection. If so, control transfers to956; otherwise, control transfers to960. At956, control commands the packet filter associated with the selected web app to ignore vulnerability scanner signatures for the selected web app. Control then continues at948.

At960, control determines which option the administrator has selected. If the administrator selected an alert only option, control transfers to964; if the administrator selected a block option control transfers to968; and if the administrator selected a blacklist option, control transfers to972. At964, control commands the packet filter to send an alert to the administrator (such as by the security center244) when HTTP requests that match vulnerability scanner signatures are observed. Control then continues at948. At968, control commands the packet filter to drop packets containing HTTP requests that match vulnerability scanner signatures. Control then continues at948.

At972, control commands the packet filter to drop packets from an IP address once HTTP requests that match vulnerability scanner signatures are observed. For example, this may cause the packet filter to drop packets from that IP address for a predetermined period of time, such as 10 minutes, one hour, or 24 hours. Control then continues at948.

CONCLUSION