Abstract:
A method for monitoring online security threats comprising of a machine-learning service that receives data related to a plurality of features related to internet traffic metrics, the service then processes said data by performing operations selected from among: an operation of ranking at least one feature, an operation of classifying at least one feature, an operation of predicting at least one feature, and an operation of clustering at least one feature, and as a result the machine learning service outputs metrics that aid in the detection, identification, and prediction of an attack.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
       [0001]    This present disclosure claims the benefit of U.S. Provisional Application Ser. No. 62/028,197, filed on Jul. 23, 2014. 
     
    
     TECHNICAL FIELD 
       [0002]    The present invention relates generally to online security and the prevention of malicious attacks. Specifically, the present invention relates to an automated computer system implemented for the identification, detection, prediction, and correlation of online security threats. 
       BRIEF BACKGROUND 
       [0003]    The term “big data” is used ubiquitously these days as a synonym for the management of data sets that are well beyond the capability of an individual person. In the arena of internet security, for example, security experts are tasked with handling increasing larger amounts of threat feeds and logs (“big data”) that need to be analyzed and cross referenced in order to find patterns to detect potential online threats to companies, institutions, agencies, and internet users worldwide. Currently the industry is so overwhelmed by the vast amounts information that there is a shortage of experts in the field of big data and machine learning who can tackle these challenges. 
         [0004]    In order to make effective use of all this security data, there is also a rising demand for “Security Data Scientists”. These scientists are not only highly trained data scientists, who can apply machine learning and data mining approaches to handle big data and detect patterns in them, but they are also security researchers who understand the online threat landscape and are experts in identifying and detecting Internet threats. However finding such talent nowadays is proving extremely difficult due to the dual set of expertise that is required. Indeed it would take an individual an entire career to become an expert in just one of these fields. Additionally, due to the exponential growth and complexity of the Internet it is proving increasingly difficult for organizations to find and retrain talented security data scientists who can help track and monitor all of the detectable and potential threats online. 
         [0005]    Thus what is needed is a scalable method for monitoring online threats that can scale with the growth of the Internet. 
       SUMMARY OF THE INVENTION 
       [0006]    The present invention overcomes these human limitations through a plug and play platform that enables security researchers and analysts to apply big data and machine learning approaches to security problems at the click of a mouse. The present invention further utilizes machine learning techniques in order to harness the information provided by the platform&#39;s users and partners in order to implement a scalable computer platform for dealing with online threats. The platform and its machine learning capacities culminate to create a machine learning service that may be trained to automatically recognize suspicious patterns in internet traffic and internet registry data and to alert the appropriate users and client systems. 
         [0007]    The machine learning service of the present invention comprises of at least four novel components: 1) a threat plug and play platform, 2) a threat identification and detection engine, 3) a threat prediction engine, and 4) a threat correlation engine. Each one of these components is described in detail in the accompanying illustrations and respective descriptions. 
     
    
     
       BRIEF DESCRIPTION OF THE FIGURES 
         [0008]      FIG. 1A  is an overview of a threat plug and play platform operating in a training mode in which threat models are generated based on features aggregated from incoming internet threat data. 
           [0009]      FIG. 1B  is an overview of a threat plug and play platform operating in a testing mode in which incoming internet threat data is compared against existing threat models. 
           [0010]      FIG. 2  illustrates an exemplary internet threat model generation scheme that is used to create a threat detection model. 
           [0011]      FIG. 3A  illustrates an exemplary threat detection engine and threat score generator. 
           [0012]      FIG. 3B  illustrates a features aggregator with various feature categories and sub-categories. 
           [0013]      FIG. 4  illustrates an exemplary IP registry classification scheme that is used to create a threat prediction model. 
           [0014]      FIG. 5A  illustrates an exemplary threat prediction engine and threat list generator. 
           [0015]      FIG. 5B  illustrates a features aggregator with various feature categories and sub-categories. 
           [0016]      FIG. 6  illustrates an exemplary multiple threat feed classifier that is used to extract and correlate suspicious features from multiple sources in order to improve threat detection and prediction models. 
           [0017]      FIG. 7  illustrates the interaction between multiple clients and a machine learning service in which data from the multiple clients is correlated in order to better identify, detect, and predict online attacks. 
           [0018]      FIG. 8  is a decision tree that illustrates how the infection rate and infection duration in a network may be used to determine the likelihood of the maliciousness of the network. 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       [0019]    For the purpose of promoting an understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the drawings and specific language will be used to describe the same. It will, nevertheless, be understood that no limitation of the scope of the disclosure is thereby intended; any alterations and further modifications of the described or illustrated embodiments, and any further applications of the principles of the disclosure as illustrated therein are contemplated as would normally occur to one skilled in the art to which the disclosure relates. All limitations of scope should be determined in accordance with and as expressed in the claims. 
       Platform Overview 
       [0020]    Applying data mining and machine learning algorithms usually requires scripting and coding as well as basic knowledge of the theory behind these algorithms. The present invention however maybe used by internet security researchers and analysts that do not need any prior theoretical knowledge of these algorithms or even any scripting or coding skills. As depicted in  FIGS. 1A-B , the platform  120  of the present invention, operating on a server, works as a security data scientist&#39;s “black box”. 
       Machine Learning 
       [0021]    Machine learning is a scientific discipline that explores the construction and study of algorithms that can learn from data. Such algorithms operate by building a model based on inputs and using those models to make predictions or decisions, rather than only following explicitly programmed instructions. 
         [0022]    The present invention makes use of these types of algorithms by way of building models of “healthy” and “unhealthy” internet networks and traffic—based on past internet data—and then comparing those models against contemporaneous internet data to estimate the level of risk that is present. 
         [0023]      FIG. 1A  is an overview of a threat plug and play platform  120  operating in a “training mode” in which threat models  160   170  are generated based on features aggregated from existing internet threat data  110 . With respect to the present invention, in order to implement a machine learning service in platform  120 , the system must first undergo training sessions in which the threat data  110  is fed into, processed, and sorted by a data categorizer  130 . Categorizer  130  is a module in which relevant features are extracted from incoming data  110  and utilized by either the network features trainer  200  or registry features trainer  400 . Network related features  140 A are then used to build trained models for the detection  160  of suspicious internet activity. Registry related features  150 A are used to build trained models for prediction  170  of suspicious internet activity. 
         [0024]      FIG. 1B  is an overview of a threat plug and play platform  120  operating in a “testing mode” in which new incoming internet threat data and IPs  110  are compared against existing threat models  160   170  in order to identify and predict suspicious activity. In testing mode internet data  110  is fed into the platform  120 , clustering and classification algorithms are run on the fly in a data categorizer  130 , and are accumulated by either a network features aggregator  310  or a registry features aggregator  510 . Relevant network features  140 B are sent to an attack identification and detection engine  300  while relevant registry features  150 B are sent to an attack prediction engine  500 . Engine  300  then compares a set of features  140 B of the current data  110  against the trained model  160  in order to determine scores and insights with respect to each IP  180 . Similarly, engine  500  then compares a set of features  150 B of the current data  110  against the trained model  170  in order to generate a predictive threat list  190 . This list  190  comprises networks and IP addresses that are expected to be used in future attacks. 
       Data Input 
       [0025]    The present invention is enabled by its capability to receive various sources of data inputs of different formats. In one embodiment the automated threat detection service  120  receives a package of data  110  from a user that comprises of any of a multiple sources of internet traffic data  100   a - i.    
         [0026]    Threat Feeds  100   a , are malicious IPs, network blocks, domains, or URLs that have been reported by CERTs, sinkhole operators, mail server operators, etc. They may include a list of IP addresses, network blocks (also known as CIDRs), domain names, or URLs that either participated in online attacks or were compromised by attackers. The platform  120  can read multiple formats of blacklists, including: plain text files (separated by line feeds), JSON objects, XML files, and CSV files. Zeus Tracker is an example of such an IP blacklist. 
         [0027]    The threat detection service  120  may also accept an entire host of internet activity logs including, but not limited to, DNS logs  100   b , HTTP logs  100   c , Webserver logs  100   d , and Syslogs  100   g.    
         [0028]    In addition, the system  120 , may also accept malware binary files  100   e . These include raw executable files that are malicious. Such files are run in a sandbox, on the system  120 , in order to extract the characteristics of the malware, i.e. identifying the HTTP and DNS connections, and identifying the processes that were initiated, spawned or killed. 
         [0029]    Similarly, pre-generated malware sandbox output files  100   f  may be accepted by the system  120  as well. Examples of commercial malware sandboxes include: Bluecoat and GFI. 
         [0030]    In addition, the system  120 , may also accept packet capture files (PCAPs)  100   h . These files are network packets that are recorded for a short period of time. 
         [0031]    Finally system  120  may also accept regional Internet registry (RIR) data  100   i . An RIR is an organization that manages the allocation and registration of Internet number resources within a particular region of the world. Internet number resources include IP addresses and autonomous system (AS) numbers. 
       Data Output 
       [0032]    In one embodiment, when prompted, the automated service  120  may produce a detailed output listing Malice Scores and Malicious Components  180  as well as Network Risk Reports  190 . Malice Scores may be numbers ranging from 0, indicating benign traffic, and 1, indicating malicious traffic. Malicious Components may include IP addresses, domain names, network blocks, and URLs. The service may also include a reason why such traffic was classified as malicious. Network Risk Reports may include an updated list of IPs, domains, and CIDRs that have high threat scores. 
         [0033]    To offer an example, consider the following code which makes an API call to a fictitious threat detection service hosted at “XYZsecurity.com.” If a user wanted to investigate the IP address 91.220.62.190 the user may issue the following command:
       https://api.XYZsecurity.com/intel_score/91.220.62.190
 
Once entered the IP addresses 91.220.62.190 is sent to the server  120 . Subsequently the automated threat detection service  120  will respond  180  with the following information is displayed to the user:
   {“CIDR_SCORE”:“0.96”,   “ASN_SCORE”:“0.96”,   “IP_SCORE”:“0.96”,   “RIR”:“RI”,   “ASN”:“61322”,   “ASN_DESC”:“SOTAL-AS ZAO Sotal-Interactive”,   “CIDR”:“91.220.62.0/24”,   “CC”:“RU”,   “IP”:“91.220.62.190”}
 
The following is a brief description of each displayed parameter:
 
“CIDR_SCORE” represents the malicious score of the CIDR between 0 and 1 with 0 being benign and 1 being malicious.
 
“ASN_SCORE” represents the malicious score of the ASN between 0 and 1 with 0 being benign and 1 being malicious.
 
“IP_SCORE” represents the malicious score of the IP address and can range between 0 and 1 with 0 being benign and 1 being malicious.
 
“RIR” identifies the Regional Internet Registry the IP belongs to. In this example “RI” is the abbreviation for RIPE which is the European registry.
 
“ASN” is the Autonomous System Number that the IP belongs to.
 
“ASN_DESC” is a textual description of the owner of the ASN.
 
“CIDR” is the network block that the IP belongs to. In this example the network block is “/24.”
 
“CC” is the Country Code of where the IP resides. In this example “RU” stands for Russia.
 
“IP” is the IP address in question. In this example the address is of course “91.220.62.190.”
       
 
       Threat Identification and Detection Models 
       [0044]      FIG. 2  illustrates an exemplary internet threat data classification scheme that is used to create a threat detection model. In order to create a threat detection model  160  threat data  110  is fed into the data categorizer  130 , of platform  120 , where the network features trainer  200  processes the data in four stages: Meta Information Extraction  210 , Feature Extraction  220 , Network Size Clustering  230 , and Model Building  240 . 
       Meta Information Extraction 
       [0045]    The Meta Information Extraction  210  stage comprises of constructing structured meta information from the un-structured threat data. The categorizer  130  may extract six core pieces of evidence from threat data  110 : IP addresses, Timestamps of Attacks, URLs, Domain Names, Attack Category, and the Threat Feed that reported the attack. 
         [0046]    Once the IP address is extracted it is fed it to a Boarder Gateway Protocol (BGP) extraction engine to find the network prefix (CIDR) and the Autonomous System (AS) that the IP maps to. In addition, this extraction provides the geo-location of the IP and the RIR that the IP belongs to. 
         [0047]    Next, for every IP the categorizer  130  constructs a time series comprising all timestamps that an attack was reported on that IP. This log of timestamps is beneficial in extracting the queueing-based features in the Feature Extraction  220  stage. The attack categories may be grouped into five main attack categories:
       1. Botnet Communication: This category may include IPs, domains, and URLs that are botnet C&amp;Cs, config droppers, and malware droppers.   2. Phishing Threats: This category may include IPs, domains, and URLs that host a phishing attack.   3. Defacement Threats: This category may include IPs, domains, and URLs (if any) for defaced web servers.   4. Network Scanners: This category may include IPs that perform network scanning activities that are trying to find vulnerable servers on the Internet.   5. Malware Threats: This category may include IPs, domains, and URLs that host or drop malware.       
 
         [0053]    The end product of the Meta Information Extraction  210  stage is a set of Meta Data  211  which is then used for Feature Extraction  220 . 
       Feature Construction 
       [0054]    In the Feature Extraction  220  stage the categorizer  130  extracts four categories of features: Queueing-Based Features, URL-Based Features, Domain-Based Features, and BGP-Based Features. 
       Queueing-Based Features 
       [0055]    Queueing-Based Features are modeled on five components of a network: i) IP address, ii) Network block, also known as CIDR, iii) Autonomous System (AS), which is a group of CIDRs that have the same routing policy, iv) Country, which is the geolocation of the IP, and v) Regional Internet Registry (RIR), which is the region the IP resides in. Each of these components may be considered a “queue.” 
         [0056]    The rate at which attacks arrive at the network are considered the “infection rate.” The rate in which they get taken down is considered the “departure rate.” The duration of how long an attack stays on a network is the “service rate.” The difference between arrival rate and the departure rate is the “network utilization.” 
         [0057]    It is assumed that attacks (infections) arrive to the queue, stay in the queue during the infection period, and finally get taken down, which is simply when the infection is cleaned. Thus there are five important properties of the queue:
       1. Infection arrival rate (λ).   2. Infection service rate (μ).   3. Network utilization, i.e. how busy is the queue (ρ=λ/μ).   4. Average number of infections in the queue.   5. Average wait time of infection in queue.       
 
       URL-Based Features 
       [0063]    URL-Based Features are extracted statistical features that capture the following patterns in URLs:
       1. Entropy-based features: These cover the entropy of the URL, file name, and file extension in the URL.   2. Length-based entropies: Lengths of URL, directories, file name, and file extension.   3. Statistical-based features: These represent sum, max, min, median, mean, and standard divination of lengths, entropies, etc.   4. Categorial features: File extension type, domain TLD.       
 
       Domain-Based Features 
       [0068]    Domain-based Features are extractions of the following attributes and aspects of domains:
       1. Passive DNS features: Passive DNS is a historical replication of DNS data for IP addresses and domain names on the Internet. It can be acquired by monitoring DNS servers around the globe and recording all DNS requests that these servers issue. By doing this and with time, one builds huge historical information database that can map all domains and IPs addresses on the Internet. For example, one can find all the IPs that google.com pointed to at certain points in time. In addition, one can find all the domains that pointed to 74.125.239.101 at certain points in time.   2. Top-level-domain features: these includes counts, diversities, entropy features, etc.   3. Domain name entropy features: these can be a count of the number of characters and/or other statistical features, etc.       
 
       BGP-Based Features 
       [0072]    BGP-based Features are features that are related to CIDRs and ASNs. The following features may be extracted per IP:
       1. CIDR size, represented by the number of bits.   2. ASN size, represent by the accumulative number of CIDRs that have been announced by a particular AS.   3. BGP route length.   4. Flags indicating if the IP belongs to a hijacked route or prefix.
 
The following features are extracted per domain name:
   1. Graph of all relative CIDRs.   2. Graph of relative ASNs.   3. Number of disconnected components in both graphs.       
 
         [0080]    The end product of the Feature Extraction  220  stage is a set of Features  221 , which are then clustered according to network size 230. 
       Clustering 
       [0081]    In the Network Size Clustering  220  stage the categorizer  130  may use the k-means clustering algorithm to cluster the data (namely IPs, domains, and URLs) and features into four clusters depending on their CIDR size. This clustering step is necessary because larger networks cannot be modeled the same as small networks and thus the models need to be trained and classified independently. The clusters are determined as follows:
       1. Cluster 1: contains all data that belongs to CIDRs that are between [/32, /24).   2. Cluster 2: contains all data that belongs to CIDRs that are between [/24, /16).   3. Cluster 3: contains all data that belongs to CIDRs that are between [/16, /8).   4. Cluster 4: contains all data that belongs to CIDRs that are between [/8, /0).       
 
         [0086]    In practice it is often the case that the characteristics of Cluster 3 and Cluster 4 are similar enough that they may be combined into a single cluster [/16, /0) to save processing time, storage, and other computing resources. 
       Model Building and Training 
       [0087]    In the Model Building  240  stage the categorizer  130  trains a Random Forest classifier for each of the clusters that were created in the previous section. To train the classifier we construct a training set that comprises a positive set and a negative set. The positive set contains malicious samples that the classifier needs to learn the patterns for. The negative set contains benign samples that the classifier needs to discriminate against. The data in the dataset corresponds to the features that were discussed in the previous sections. 
         [0088]    Some features are represented based on whether or not they exist. For example, one feature can be if an IP belongs to cluster 5. This feature is represented as 1 or 0. The feature will be 1 if the IP belongs to cluster 5 and 0 if the IP does not belong to cluster 5. Other features are presented as numerical values. For example, one feature can be the total number of IPs in a network. Eventually the dataset can be thought of as a table, in which the rows comprise the sample points, which is in our case; IP addresses. And the columns are the features that we extracted for these IP addresses. Since we are dealing with a classification problem (i.e. classifying traffic into benign and malicious) the dataset must contain a column that shows the label of the data, which is simply if this IP is benign (0) or malicious (1). 
         [0089]    To evaluate the classifier that was built in the previous step, the training set may be divided into two sets: a training set and a test set. The test set is used to evaluate the performance of the classifier (the performance in terms of detection not speed). Based on the detection of the sample data in the test set, one can evaluate the accuracy of the classifier, the error rate, the false positive rate, and the false negative rate. 
         [0090]      FIG. 8  is a decision tree that illustrates how the infection rate and infection duration in a network may be used to determine the likelihood of the maliciousness of the network. As an example, in one embodiment the system, if the infection arrival rate  801  is greater than 0.3/day and the network utilization ratio  803  is greater than 0.05 then the probability of that this node is infected is 80%, otherwise the probability is 20%. Similarly if the infection arrival rate  801  is less than 0.3/day and the infection departure rate  802  is greater than 0.1/day and also the average infection period  805  is greater than 4.2/day the probability that the node is infected is 90%, otherwise it is 10%. Finally if the infection arrival rate  801  is less than 0.3/day and the infection departure rate  802  is less than 0.1/day and also the service rate  804  is greater than 0.6 days the probability that the node is infected is 70%, otherwise it is 30%. It should be noted that the numerical values for the duration and probabilities illustrated in  FIG. 8  are merely examples and are not meant to be taken literally. In practice the actual values and weights representing the significance of these features will be determined and updated every time a model is trained and built. 
         [0091]    The end product of the Model Building  240  stage is a set of trained Threat Classifiers  241  for each Cluster Grouping  231 . These classifiers exist as trained models  160  that may later be compared against future internet data  110  in order to identify and detect potential threats. 
       Threat Identification and Detection Engine 
       [0092]    Once models  160  are trained by the network feature trainer  200  they may be used by an attack ID and detection engine  300  to analyze potential threats.  FIG. 3A  illustrates an exemplary threat detection engine and threat score generator. A network features aggregator  310  collects all the important features—the relevance of which is determined by the trained model  160 —related to IP  311  CIDR  312  ASN  313  CC  314  and RIR  315 . These features then undergo a clustering step  320  in which the features are grouped by the network size clusters they pertain. The clustered features are then run through their respective classifiers in a classification engine  330 . The comparison of the aggregated features  310  with the trained models  160  in the classification engine  330  results in an IP score  341  which represents the maliciousness of that particular IP. This IP score  341  may then be used to derive a CIDR score  342  which in turn may be used to derive an ASN score  343 . These scores along with any meta information  344  about the IP under investigation may be displayed to the user via a results module  340 . 
         [0093]      FIG. 3B  illustrates a network features aggregator  310  with various feature categories  311 - 315  and sub-categories. Each feature category  311 - 315  may comprise of sub-categories such as Queuing Features, Attack Category Diversity Features, Feed Diversity Features, Attack Category Stats, and Feed Stats. 
         [0094]    The IP Features  311  category may further comprise of IP Stats which include the number of threat feeds that list the IP and the number of attack categories the IP falls under. The CIDR Features  312  category may further comprise of CIDR Stats which include the CIDR size, the number of infected IPs within the CIDR, and the cluster ID. The ASN Features  313  category may comprise of ASN Stats including the number of CIDRs within the ASN, the number of infected IPs within each CIDR, and thus the number of infected CIDRs. The CC Features  314  category may further comprise of CC Stats including the number of infected IP&#39;s, thus the number of infected CIDR&#39;s, and thus the number of infected ASN&#39;s. The RIR Features  315  category may comprise of RIR Stats including the number of infected IP&#39;s, thus the number of infected CIDR&#39;s, thus the number of infected ASN&#39;s, and thus the number of infected CC&#39;s. 
         [0095]    Since the listed feature categories  311 - 315  follow a hierarchy (i.e. IPs reside on CIDRs, which reside within ASNs, which further reside within CCs, which finally exist within RIRs) the aggregated averages 316 of some of features in one feature category may be used to estimate the stats in another feature category. For example  FIG. 3B  illustrates how, within the RIR Features  315  category, the statistics related to the number of infected IPs, CIDRs, ASNs, CCs may be determined by taking the averages of these said statistics from other feature categories  311 - 314 . 
       Threat Prediction Models 
       [0096]      FIG. 4  illustrates an exemplary IP registry classification scheme that is used for create a threat prediction model. Similar to how threat detection models  160  were created, in order to create a threat prediction model  170  threat data  110  is fed into the data categorizer  130 , of platform  120 , where the RIR features trainer  400  processes the data in four stages: Information Extraction  410 , Feature Extraction  420 , Network Size Clustering  430 , and Model Building  440 . 
       Information Extraction 
       [0097]    The Information Extraction  410  stage comprises of first receiving daily information about IP  401  and network assignments, reassignments, allocations, reallocations  402  and newly registered domain names from top level domain (TLD) zone files  403 . This information is acquired through the five RIRs, i.e. ARIN, RIPE, APnic, AFRInic, and LACnic. At this stage the categorizer  130  identifies the individuals or groups that the IPs or network blocks were assigned to  411 . 
       Feature Construction 
       [0098]    In the Feature Extraction  420  stage the categorizer  130  extracts  2  categories of features: Contact Entropy Based Features and pDNS Based Features. 
       Contact Entropy Based Features 
       [0099]    Contact Entropy Based Features are features used to detect network blocks that will be used by, threat actors. The threat actors use anonymous or private registration information when they register for reassigned network blocks. Thus in order to identify these malicious actors the entropy of the registration information for newly assigned network blocks are features that need to be aggregated and correlated. Suspicious networks will likely have higher entropy. 
       pDNS Based Features 
       [0100]    The system further finds passive DNS (pDNS) evidence on the IPs that were identified in the registration information from the previous feature. The system further calculates pDNS features on the IPs and domains that are retrieved from the previous step. Then the system correlates the domains and IPs with a malware DB to find which IPs and domains were associated with malware in the past. Finally the system calculates maliciousness scores for all IPs and domains that it gets from the pDNS evidence. 
       Clustering and Model Building 
       [0101]    In a manner analogous to how threat detection models  160  were generated in the earlier example these datasets are also grouped into clusters  430  depending on the CIDR size and analyzed by their respective cluster classifiers  440 . 
         [0102]    The end product of the Model Building  440  stage is a set of trained Threat Classifiers  441  for each Cluster Grouping  431 . These classifiers exist as trained models  170  that may later be compared against future internet data  110  in order to predict potential threats. 
       Threat Prediction Engine 
       [0103]    Once models  170  are trained by the network feature trainer  400  they may be used by an attack prediction engine  500  to analyze potential threats.  FIG. 5A  illustrates an exemplary threat prediction engine and threat list generator. A registry features aggregator  510  collects all the important features—the relevance of which is determined by the trained model  170 —related to contact entropy  511  pDNS  512 . These registrations then undergo a clustering step  520  in which the features are grouped by the network size clusters they pertain. The clustered registrations are then run through their respective classifiers in a classification engine  530 . The comparison of the aggregated features  510  with the trained models  170  in the classification engine  530  results in a list of high scoring IPs, domains, and CIDRs  541  which represent the maliciousness of those respective networks. These scores along with any meta information  344  about the networks of interest may be displayed to the user via a results module  340 . 
         [0104]      FIG. 5B  illustrates a registry features aggregator  510  with various feature categories  511 - 512  and sub-categories. Each feature category  511 - 512  may comprise of sub-categories such as Contact-Based Features and pDNS-Based Features. 
         [0105]    The Contact-Based Features  511  category may further comprise of sub-categories including Shannon Diversity Index of Registration Information, Shannon Entropy of Registry Information, Shannon Diversity Index of Registrants Addresses, and Shannon Entropy Index of Registrants Addresses. 
         [0106]    The pDNS-Based Features  512  category may further comprise of sub-categories including Average Shannon Entropy for Domain Names, Statistical Features for Domain Name Entropy (e.g. min, max, standard deviation of entropy), Shannon Diversity Index of Top Level Domains, and Statistical Features for Top Level Domains Entropy (e.g. min, max, mean, standard deviation of entropy). 
       Threat Correlation Classifier 
       [0107]      FIG. 6  illustrates an exemplary multiple threat feed classifier that is used to extract and correlate suspicious features from multiple sources in order to detect and predict threats. The automated threat detection system  120  can be used as a threat correlation classifier  600  that will aggregate several threat feeds. If these feeds contain noisy data, false positives, or duplicates, the system  120  normalizes and cleanses these threat feeds. 
         [0108]    In order to build a model for correlation, the system must be fed with the same six entities that are extracted in the network features trainer  200 . The correlation classifier  600  then periodizes the timestamps of the various attacks in the different feeds. This is done by grouping and aggregating  610  IP address, CIDR, ASN, CC, and RIR information in all of these feeds. Then on every IP, CIDR, ASN, CC, and RIR the classifier  600  groups the attacks by their categories. The classifier  600  then periodizes attacks on each of these entities by their attack category. 
         [0109]    Next the classifier  600  extracts features  620  on all six entities similar to the steps we followed in the network features trainer  200 . The classifier  600  then performs the familiar a clustering step  630  and then builds models  640  for the four clusters. The models that were generated in this process are then used to score all the threat data from all threat feeds  100   a - n . Finally, the classifier  600  defines a threshold, as a cutoff point, to select which data will be used in the normalized (cleansed) threat feed and which data will be discarded. 
         [0110]      FIG. 7  illustrates the interaction between multiple clients  700   a - c  and a machine learning service  710  in which data from the multiple clients is correlated in order to better identify, detect, and predict online attacks. 
         [0111]    For example, in one embodiment, clients  700   a - c  may send input data  110   x - z , respectively, to machine learning service  710 . The platform interface  720 , of machine learning service  710 , may then aggregate the input data  110   n  from multiple sources and feed it into the data categorizer  130  of the platform backend  730 . Data categorizer  130  may then extract features from the aggregated input data  110   n  and feed them into attack ID and detection engine  160  and attack prediction engine  170 . Engines  160  and  170  may then output aggregated threat scores  180   n  and aggregated threat lists  190   n , respectively, to the platform interface  720 . The platform interface  720  may then forward the relevant scores  180   x - z  and lists  190   x - z  to clients  700   a - c , respectively. Platform interface  720  may determine which information to forward based on each respective clients&#39;  700   a - c  preferences, requirements, network characteristics and input data  110   x - z.    
         [0112]    This method of aggregating input data  110   x - z  from multiple clients  700   a - c  is particularly useful because it allows for a machine learning service  710  to determine the commonalities and differences between the unique data sets  110   x - z  and thus train more extensive models that better represent the variability of malicious activity on the internet. 
       Hardware Limitations 
       [0113]    Naturally, in order to process the requisite amount of data, the present invention requires significant hardware resources to operate. To deploy an instance of the analytics engine that supports up to 100 active users, the automated threat detection service  120  requires, at a minimum, the following hardware specifications: A CPU with 8 cores, each operating at 2.0 GHz; 32 GB of RAM; and 50 GB of free HDD space. While the system  120  at times may be functional with less resources the requirements specified above should be scaled according to the number of users for effective performance. 
       Updates 
       [0114]    As with any automated services the platform  120  will sometimes misclassify legitimate traffic as malicious (also known as false positives) or classify malicious traffic as legitimate (also known as false negatives). While these incidents should be rare, the present invention may require a mechanism to feed these misclassifications to the platform  120  so that its classifiers can be retrained. This may be done simply by pushing a configuration file update to the platform  120 . This configuration file may contain amendments to a classifier model or alternatively include an entirely new retrained model. 
       GLOSSARY 
       [0000]    
       
         1. Features: Unique attributes that are extracted from raw data. This can either be a statistical, categorical, or numerical representation. 
         2. Feature Construction: The act of generating features from raw data. 
         3. Feature Set: Group of features that are associated with observation of the actual data. Such a set can be labeled or unlabeled. 
         4. Training: Learning patterns in a feature set. 
         5. Supervised Learning: Training a system when the data is labeled. 
         6. Unsupervised learning: Training a system data is unlabeled. 
         7. Clustering: Unsupervised learning system, via the application of an algorithm, that can learn patterns in data. The data set is then divided into related clusters. 
         8. Classifier: Supervised learning system, via the application of an algorithm, than can learn patterns in data or a feature set. The data is assigned the closest label in the feature set. 
         9. Model: The generated object (or output) of a classifier. A “trained model” is a model that has gone through a classifier. 
         10. Engine: System accepts user input and applies it to a trained model to provide an output. 
         11. Training Set: Part of the data corpus that is used to build a model. 
         12. Test Set: Part of the data corpus is used to verify the accuracy of the model. Test set data is already labeled and verified. 
         13. Testing/Validation: Testing the accuracy of the model by comparing known labels of a test set with the output of the model. 
         14. Platform: A software service that comprises servers, clients, API, dashboards, or web portals.