Patent ID: 12231461

Like reference numbers and designations in the various drawings indicate like elements.

DETAILED DESCRIPTION

Implementations of the present disclosure are directed to mitigating cyber security risk in enterprise networks. More particularly, implementations of the present disclosure are directed to systems and methods to automatically prioritize cyber-security security controls using intelligent digital twins. A Cyber Digital Twin (CDT) platform executes simulations on a digital twin of an enterprise network to determine and prioritize security controls requirements to mitigate cyber security risk in the enterprise network.

The proposed techniques integrate a knowledge graph (e.g., D3FEND) into the cyber digital twin in order to identifying and prioritize to most effective counter measures to be employed in order to reduce system's risk. An open source knowledge graph holds information related to digital artifacts and their related attack tactics and security controls.

As described in further detail herein, implementations of the present disclosure provide a CDT platform that executes simulations using a digital twin of an enterprise network based on attack graph analytics. In some implementations, the cyber digital twin, also referred to herein as digital twin, is used to automatically gather and prioritize security requirements at scale over a respective (active) enterprise network. The digital twin represents information about the computer network, and is used to associate the information with attack tactics, measure the efficiency of implemented security controls requirements, and automatically detect missing security controls. The digital twin is used to evaluate cyber risk, measured as a risk value, over the attack graphs and proposes prioritization of detected requirements towards rapid reduction of risk under active system constraints. In some implementations, a CDT simulator offers several risk reduction methods for automatically selecting security controls requirements. Data used for constructing a contextual digital twin is defined including relations between security controls and attack tactics. Calculations used for ranking security controls risk impact, the algorithm for security controls requirements prioritization, and a demonstration of successful results using a field experiment conducted on an active enterprise network are each described in further detail herein.

FIG.1depicts an example architecture100in accordance with implementations of the present disclosure. In the depicted example, the example architecture100includes a client device102, a network106, and a server system108. The server system108includes one or more server devices and databases (e.g., processors, memory). In the depicted example, a user112interacts with the client device102.

In some examples, the client device102can communicate with the server system108over the network106. In some examples, the client device102includes any appropriate type of computing device such as a desktop computer, a laptop computer, a handheld computer, a tablet computer, a personal digital assistant (PDA), a cellular telephone, a network appliance, a camera, a smart phone, an enhanced general packet radio service (EGPRS) mobile phone, a media player, a navigation device, an email device, a game console, or an appropriate combination of any two or more of these devices or other data processing devices. In some implementations, the network106can include a large computer network, such as a local area network (LAN), a wide area network (WAN), the Internet, a cellular network, a telephone network (e.g., PSTN) or an appropriate combination thereof connecting any number of communication devices, mobile computing devices, fixed computing devices and server systems.

In some implementations, the server system108includes at least one server and at least one data store. In the example ofFIG.1, the server system108is intended to represent various forms of servers including, but not limited to a web server, an application server, a proxy server, a network server, and/or a server pool. In general, server systems accept requests for application services and provides such services to any number of client devices (e.g., the client device102over the network106). In accordance with implementations of the present disclosure, and as noted above, the server system108can host a CDT platform.

In the example ofFIG.1, an enterprise network120is depicted. The enterprise network120represents a network implemented by an enterprise to perform its operations. In some examples, the enterprise network120represents on-premise systems (e.g., local and/or distributed), cloud-based systems, and/or combinations thereof. In some examples, the enterprise network120includes IT systems and OT systems. In general, IT systems include hardware (e.g., computing devices, servers, computers, mobile devices) and software used to store, retrieve, transmit, and/or manipulate data within the enterprise network120. In general, OT systems include hardware and software used to monitor and detect or cause changes in processes within the enterprise network120as well as store, retrieve, transmit, and/or manipulate data. In some examples, the enterprise network120includes multiple assets. Example assets include, without limitation, users122, computing devices124, electronic documents126, and servers128.

In some implementations, the CDT platform is hosted within the server system108, and monitors and acts on the enterprise network120, as described herein. More particularly, and as described in further detail herein, the CDT platform executes simulations on a digital twin of an enterprise network (e.g., the enterprise network120) to determine and prioritize security controls requirements to mitigate cyber security risk in the enterprise network. In some examples, the CDT platform is provided as part of a security platform, such as an agile security platform discussed herein. In some examples, the CDT platform is separate from and interacts with a security platform, such as the agile security platform discussed herein. As described in further detail herein, one or more security controls can be implemented in the enterprise network120based on security controls requirements identified through simulation in order to reduce risk of cyber-attack in the enterprise network120.

To provide further context for implementations of the present disclosure, in the area of cyber security resiliency (e.g., how resilient an enterprise network is to cyber-attack), enterprise practitioners may rely on the implementation of procedures, processes, and automation tools, which can be collectively referred to as Security Controls (SCs). In general, a SC (also referred to as a remediation action) can be associated with a fact and/or a rule, as a known remediation action for mitigating the fact and/or rule (e.g., preventing the fact from occurring by preventing execution of a rule). An example SC can include, without limitation, installing a security patch to resolve a vulnerability of a particular version of software. Executing efficient SCs is aimed at preventing and handling security issues and problems prior to the materialization of the consequent cyber risk. Such security issues are introduced to the enterprise cyber space in an overwhelming rate, driving the need for constant optimization and prioritization of requirements that are more critical to the business first, followed by automatic implementation to remediate the imminent cyber risks. In addition, being able to trace the enterprise cyber posture to employed automation and practices of SC requirements (SCRs) is not a direct indication of success in preventing attacks by hackers. As such, goals of employing SCs are to reduce the attack surface over time as suggested by compliance drives, and focus less on prevention of coordinated attacks and imminent risk reduction toward targeted crown jewels.

Investment in SCs materializes in tools and processes aimed at solving optimized and automated known security needs, while other investments in SCs are projected and planned according to new types of threats, or adjustments to new and more efficient security tools. The existing implementation of SCs may be evaluated by constantly tracing and correlating security requirements with the active attack surface of the network, such as discovering and analyzing pathways to specific targets. As the attack surface mutates due to new threat intelligence introduced and changes in the infrastructure settings and configuration, implemented security tools can become invalid, while new tools may be inadequate.

One option to evaluate the efficiency of implemented security tools or detecting a gap that highlights missing SCs is by understanding how hackers may move within the network. Exploitation of vulnerabilities in the network is done by performing lateral movements between acquired IT assets. Such actions are possible due to missing implementation of some SCs that could have prevented these actions in the first place. As such, the pure existence of modeled attack pathways is an indication of lack of performance or missing implementation of SCRs. Compliance needs and standards are key drivers for defining security requirements that are manifested in SCs. Such needs are adjusted to the type of the organization domain, the type of network, connected systems, etc. As discussed herein, such definitions of requirements do not consider potential movements of hackers in the network, exploiting existing vulnerabilities within the context of targeted attack pathways. The context of the network hackability state is missing from the considerations and definitions of the requirements.

A connection between the ability to hack an active system and automatically discover SCRs in a context of potential cyber-attacks is described herein. A method and technology for constructing a contextual cyber digital twin that maps the relations between SCs and attack tactics that, in turn, represent SCRs are also described in detail herein. Also described are definitions of SCs and types of attack graphs, and a framework and method for using a cyber digital twin that captures attack pathways as a means for discovering SCRs. The methodology details how to measure an impact of an attack and the relation to relevant SCs as well as ways for prioritization of SCRs to reduce the risk impact as quick as possible. A simulator is provided for performing the SCRs analysis, including incremental and iterative manner for rapid decay of risk. In some examples, SCs refer to a set of SCs (e.g., one or more SCs), each SC being a tool (e.g., software) and/or action that can be executed in the enterprise network to mitigate risk. In some examples, SCRs refer to a set of SCRs (e.g., one or more SCRs), each SCR representing an absence of a SC in the enterprise network that should be addressed by implementing a SC to mitigate risk.

In some examples, SCs may be defined as a combination of policies, methods, and tools that are aimed at protecting an enterprise network from cyber-attack. In some examples, SCs are classified into fourteen groups according to ISO/IEC 27001 standards, while the U.S. government's National Institute of Standards and Technology (NIST) provides seventeen groups of SCs. To illustrate the implications of SCs in the present disclosure, some of the classifications listed in Table 1 are used.

TABLE 1Example SC Groups by Industry Standarization Authorities.ISO/IEC 27001NIST, 800-53, Revision 4A.5: Information security policiesAC: Access ControlA.6: How information security isAT: Awareness and TrainingorganizedAU: Audit and AccountabilityA.7: Human resources securityCA: Security Assessment andA.8: Asset managementAuthorizationA.9: Access controls and managing userCM: Configuration ManagementaccessCP: Contingency PlanningA.10: Cryptographic technologyIA: Identification and AuthenticationA.11: Physical security and equipmentIR: Incident ResponseA.12: Operational securityMA: MaintenanceA.13: Secure communications and dataMP: Media ProtectiontransferPE: Physical and EnvironmentalA.14: Secure acquisition, development,Protectionand support of information systemsPL: PlanningA.15: Security for suppliers and thirdPS: Personnel SecuritypartiesRA: Risk AssessmentA.16: Incident managementSA: System and Services AcquisitionA.17: Business continuity and disaster recoverySC: System and Communications ProtectionA.18: Security ComplianceSI: System and Information Integrity

Each SC group depicted within Table 1 contains specific policies. Examples are the ISO/IEC 27001 A.8 asset management group, that contains policy #8.1.1 for asset inventory and policy #8.1.3 for assets acceptable usage, for example. Described in further detail herein is a mechanism and technology for detecting and tracking missing implementations of specific sub-set of security requirements that are part of the generic SCs, yet specifically relevant to the context of an enterprise network that is under analysis. Consequently, the technology supports the evaluation of SCs effectiveness and allocated capital relevant to the enterprise network under investigation and tuned to defined business targets.

SCRs can be gathered in various manners. In some implementations, a detailed analysis of assets of an enterprise network is performed and an implementation plan based on best practices is provided. In some implementations, SCRs can be generated from an examined system using industry standards. Some implementations for cyber security investment assessment rely on methods of game theory, multi-objective optimization, stochastic calculus, and other ideas. For example, the investment strategy may be modeled as a game between a defender and an attacker. The defender's goal may be to assemble a defense toolkit that would minimize potential attack damage given a limited budget constraint. The task may be reduced to an optimization problem and solved by algorithms such as the Knapsack problem. The underlying data used is the output of a network scanner as well as information about vulnerabilities published at a public catalog such as the National Vulnerability Database (NVD) provided by the U.S. government. However, despite the validity and efficiency of this mathematical apparatus, the approach may not address how the vulnerabilities are exploited in practice during a real-life, targeted attack. If an attack is undertaken against a specific target or a set of targets within a computer network, a different location of identical vulnerabilities in the network has a different impact. Possible attacker actions and different pathways to the targets that generate different impacts may be included in evaluation of security control effectiveness.

Attack pathways can be modeled as attack trees, petri nets, and lately mostly used, as attack graphs. Modeling approaches that capture nodes as physical assets (e.g., a workstation machine) and edges as potentials lateral movements of a hacker between two assets are referred to as Physical Attack Graphs (PAG). Modeling logical rules that define how an attacker advances within the network are represented in Analytical Attack Graphs (AAGs). These logical rules are in essence a representation of security requirements, as enablers to adversarial lateral movement, which a defender is required to eliminate and nullify. Use of AAGs in mitigating attacks on computer networks is described in further detail in commonly assigned U.S. application Ser. No. 16/554,846, entitled Generating Attack Graphs in Agile Security Platforms, and filed on Aug. 29, 2019, the disclosure of which is expressly incorporated herein by reference in the entirety for all purposes. Further, generation of AAGs is described in further detail in commonly assigned U.S. application Ser. No. 16/924,483, entitled Resource-efficient Generation of Analytical Attack Graphs, and filed on Jul. 9, 2020, the disclosure of which is expressly incorporated herein by reference in the entirety for all purposes.

In some implementations, cyber security investment analysis relies on attack graphs. In some implementations, the problem of SCRs is modeled as a multi-objective optimization problem with the goals of: (1) minimizing the cyber security risk on targets, (2) reducing direct costs of a security control deployment, and (3) reducing indirect costs of security control implementation. An example of indirect cost is employee's loss of time due to more sophisticated compliance procedures.

In some implementations, attack graphs may be generated based on network scanners output, known published vulnerabilities, and network related data sources (e.g., firewall rule analyzers) as a good approximation to some types of attacks. Probabilistic graphical models, such as Bayesian networks, can be used to reflect difficulty differences of exploiting vulnerability. However, modern attacks are more sophisticated. To model a hacker's movement adequately during a complex attack, not only endpoint vulnerabilities and network firewall rules should be considered, but also amongst other, user access permissions and open sessions status. In some implementations, a complete approach to address the complexity of a modern attack may be used. AAGs may be created based on a wide range of facts about the network assets and their interaction. The data can be received from multiple sources and can include features such as, for example, user account hierarchies, complete software installation compared to those detected by port scanner, clear access authorization credentials, and others. In some examples, rules may be provided from the MITRE ATT&CK™ knowledgebase of attack tactics.

In accordance with implementations of the present disclosure, the CDT platform generates detailed AAGs to multiple targets with multiple origins. A digital twin can be described as a software-implemented replica of a physical entity that captures aspects of the physical entity. In the instant case, the physical entity is an enterprise network and an aspect is a cyber security posture. A digital twin is unique, because it captures and models the aspect of the enterprise's cyber security posture from a hacker's perspective and models hacker's movements, instead of the pure system monitored state. In short, the digital twin can be described as an inferencing model of the enterprise network.

As described in further detail herein, the CDT platform of the present disclosure enables creation and use of digital twins for automatically gathering and prioritizing SCRs within large industrial settings, where scale and complexity hampers the ability of manual analysis. More particularly, the CDT platform includes a simulator that tracks lack of SCRs implementation within the context of targets that are exposed to cyber security risks. The analysis of the attack graph towards which set of SCRs should be implemented first, is aimed at rapidly reducing the attack surface size. As such, the simulation examines how the implementation of a set of SCs can affect the organization's overall cyber risk exposure. Described in further detail herein are details of the methodology and the data types the simulator uses, the attack graph in the form of AAG, a SCs Traceability Matrix (TM) and a set of configuration parameters. For illustrative purposes, the SCs are selected to be policies and rules extracted from the ISO/IEC 27001 standard. In some cases, a goal of an enterprise is to comply with these policies. In other cases, the goal may be to prioritize budget proposals or to assess effectiveness of previous year budget allocations for cyber security tools. In real world scenarios, the SCs could be mapped to projects that represent cyber security budget proposals.

An attack graph is based on associating rules and impacts according to facts (each of rules, impacts, and facts being represented as respective nodes) that are based on evidence collected from Configuration Items (CI). CIs are network assets such as computers, user accounts, and the like. Some of the CIs can be target CIs, commonly referred to as crown jewels, which are highly valuable machines, applications, or processes. Rules are attack tactics that are derived from the collected facts, forming associations and links in the graph. Rules can be provided from a data source, such as the rules provided from MITRE ATT&CK™. Facts can be information on the CI such as, for example, identification, configuration, installed software and its version, open sessions, memory map, vulnerability, user group membership, or a network share access permission, and the like. Rules represent needed SCRs to be implemented. SCRs can be, for example, to isolate an application in a sandbox, to disable a program, segment a network, to change user privileges, and the like. Consequently, a rule may be to implement one of the former requirements in order to prevent the ability to and attack tactic of executing a code on a remote machine, for example. Impacts are the outcomes of not implementing a rule inference (e.g., ability to elevate user privileges on a given machine under a specific account). The discovered information is fed into a proprietary rule engine that uses the discovered facts, applies the rules, and generates the impacts, in an aggregated manner across all CIs, towards all target CIs (crown jewels). The output of the process is the attack graph (e.g., AAG).

In order to evaluate the success of risk reduction and to decide which SCRs to prioritize, a Graph Risk Value (GRV) is defined, as in Eq. 1, as a measure of the cyber risk exposure of the enterprise network. GRV is a single-valued scalar metric that is related to an exponential cost model and can be provided as:
GRV=Σi∈MRi(1)
where Riis a risk measure of an individual target impact and M is a set of target impacts. Risk can be determined as follows:

Ri=e-Hiα·Ci(2)
where Hiis hardness of all paths to target i, α is damping constant (e.g., α=8), and Ciis a graph theory eigenvector centrality measure of target impact node in the modeled AAG. Hardness is a value defined by a cyber security expert for each rule indicating how difficult the rule is to perform based on, for example, available tools, script, previous experience and the like. In some examples, Hiis calculated as an average over all rules on all paths that lead to a target. The rules that are used to generate the AAG may be defined by cyber security experts. For each rule, the experts may also define a set of SCs that can mitigate the effect of the rule.

To illustrate this principle, an example attack tactic is referenced. The example attack tactic includes T1175 (provided from MITRE ATT&CK™) that defines a lateral movement of a hacker from one machine to another, by utilizing MS Windows DCOM infrastructure. In order to use this tactic, an adversary must acquire a user account with certain privileges. Such an account should be a member of the Distributed COM group on a host machine. Consequently, the hacker can perform a remote procedure call (RPC) over the network to a target machine. In addition, the target machine must be listening on a predefined set of ports supported by DCOM infrastructure. To mitigate the exposure to this type of attack, MITRE offers several tactics that are mapped to SCs defined by ISO/IEC 27001 standard as depicted in Table 2.

TABLE 2Example of Requirements for T11175.Proposed Mitigation for T1175Required Security Controls(MITRE ATT&K)(ISO/IEC 27001)1. Application Isolation and SandboxingA.9.1.1 Access Control2. Disable or Remove Feature orA.9.1.2 Access to NetworksProgramand Network Services3. Network SegmentationA.13.1.3 Segregation in4. Privileged Account ManagementNetworks

Namely, MITRE T1175 requirements is to implement three mitigations, in which a security expert may need to implement several SCs. In the case of T1175, the security expert may opt to implement an Access Control Policy (A911), an Access to Networks and Network Services policy (A912), and a Segregation in Networks policy (A1313). By implementing even one of these three SCRs, a defender can eliminate the potential lateral movement. Accordingly, the conditional logic is an AND relation between the policies.

This Boolean condition can be defined as a Prolog rule used to generate the attack graph in the form of:

Listing 1: Example Rule in Prolog.execCode(Host, User) :=userInLocalGroup(Host, ‘DISTRIBUTED COM USERS>’, User),canNetComm(SrcHost, Host),execCode(SrcHost, User),@ports Host:or(135, U135, 1029)

Listing 1 is an example of a Boolean AND condition of MITRE T1175 rules (e.g., AND multiplication between existing user group, network access, and ability to remotely execute code, applied on an open port). This example Prolog syntax indicates that four conditions must be met for an adversary to be able to execute code on Host under privileges of the User account. First, userInLocalGroup indicates that a User must be a member of specified local group including indirect membership, or effective membership as a result of other attack tactics defined as rules. The second condition, canNetComm, requests a network communication between the source and destination hosts. Thirdly, the User must already be able to execute code, execCode, on the source host machine. Finally, the host must have a service listening on either one of TCP ports135,1029or on UDP port135, using the request @ports. It should be noted that the term @ports is an extension to standard Prolog syntax.

Another cardinal data set processed by the simulator is a mapping between SCs and lateral movement rules. This mapping is provided in the form of a sparse matrix that is referred to as a Traceability Matrix (TM). For example, TM=(tmij), where tmij≠0, if security control i can be used to mitigate attack tactic j representing a rule node in AAG, otherwise tmij=0. Possible values for tmijcan include values in a set of 3 symbols: {0, *, +}.

If tmij≠0, then the relation between the SC and the attack tactic is defined by a Boolean logic operator. It may either be a logical AND (*) or a logical OR (+). A Boolean OR operator instructs implementing one SC to eliminate the risk of the attack tactic. In the example above, it is enough to either redefine user permissions or to isolate DCOM server by updating firewall rules. Each SC implementation will result in mitigation (e.g., elimination) of a particular attack-related risk. However, there could be a situation when only implementation of all SCs related to the attack tactic will lead to its elimination. In such a case, the relation is defined by a Boolean AND operator. Table 3 depicts an example TM with logical operators.

TABLE 2SC Traceability Matrix with Logic OperatorsSC1SC2SC3SC4SC5SCnLR1+00+0+LR20++000LR3000++0LR4+0+0++LRm00+0+0

In Table 3, SC1, SC2, . . . SCNare security controls LR1, LR2, . . . , LRMare Lateral movement rules that are mapped to SCRs, n is index of the evaluated SC. The TM values indicate the relations between the requirements and SCs, and the exposure to a rule. The “+” symbol means a Boolean OR relations, and the “*” symbol indicates a Boolean AND.

In some implementations, the TM is used to calculate impact influence scores on each SC as provided in Eq. 3:
SCi=Σj∈Atmij·rcj(3)
where SCiis influence score of security control i, rcjis a count of rules that correspond to tactic j in the AAG, where function ƒ(x) is defined in Eq. 4:

f⁡(x)={1,if⁢x=*or⁢x=+0,otherwise(4)
The value of SCirepresents how many times an attacker can use the tactic within the enterprise network toward the defined target. The set A contains all implemented rules (attack tactics).

To visualize the contribution SCs make to mitigating the overall cyber risk exposure of an enterprise, an influence score histogram can be used.FIG.2depicts an example impact influence histogram200. In the example ofFIG.2, the SCs are ordered by their influence score.

In some examples, different types of SCs are provided. Example types include immutable and mutable. Immutable SCs, also referred to as mandatory SCs, which cannot be eliminated due to restrictions that are not controlled by the security team, or that are not implemented at all, namely, a gap in the organization cyber resiliency. A list of immutable SCs, if there are any, is also defined as a configuration setting of the simulator.

Implementation of SCRs is costly and time consuming. The simulator conducts sensitivity analysis based on different risk reduction techniques aimed at proposing which SCs to handle first. This analysis is conducted in an iterative and incremental manner. Each technique simulates implementation of a set of SCs to consequently reduce the number of rule types in the corresponding AAG.

FIGS.3,4, and5illustrate a comparison of techniques.FIG.3depicts an influence score SC plotted as a histogram300for a small number of SCs. However, a large organization may have thousands of different SCs.FIG.4is a plot histogram400that depicts a curve that contains 3000 SCs. The histogram400is ordered as such that the top influencing SCs are on the left, as such an obvious strategy is to start by implementing SCs that have the highest SC influence score, and by selecting a certain quantity of those SCs. The gradient reduction technique referenced inFIG.4refers to an algorithm that continuously calculates the slope gradient between decaying SCs. The algorithm stops when the slope reaches a threshold value selected by the user of the simulator. This enables the user to select a point on the curve where the gradient, or the rate of change, slows down. This algorithm is useful for exponential drops. In practice this means some SCRs are disproportionate to the overall cyber security exposure of the enterprise network under investigation. An example can be to select SCs that are above 0.6 slope angle.

FIG.4depicts a graph illustrating the area under curve (AUC) technique. AUC is suitable for a curve with monotonic influence reduction. This is the case when there is no clear winner, and the contribution of different SCs to the overall cyber security risk is relatively uniform. The algorithm receives the percentage of AUC to reduce as a configuration parameter and enumerates the SCs which contribute to such a segment. An example can be to select the area that contains 10% of the overall number of rules in the existing AAG. The simulator provides the ability to feed a custom selection and de-selection of SCs for implementation due to business restrictions and constraints that are known to the users.

In some implementations, the CDT platform of the present disclosure can be executed as part of an agile security platform. In some examples, the agile security platform determines asset vulnerability of enterprise-wide assets including cyber-intelligence and discovery aspects of enterprise information technology (IT) systems and operational technology (OT) systems, asset value, potential for asset breach and criticality of attack paths towards target(s) including hacking analytics of enterprise IT/OT systems.

FIG.6depicts an example conceptual architecture600of an agile security (AgiSec) platform. In the example ofFIG.6, an agile security system602and an agile SC (AgiSC) module604are depicted. In some examples, the agile SC module604executes the simulator of the CDT platform of the present disclosure.

In further detail, the agile SC module604processes data provided by an agile discovery (AgiDis) module and an agile security hacker lateral movement (AgiHack) module. In some examples, the AgiDis module discovers assets and vulnerabilities of the enterprise network using third-party tools. The extracted data is stored in a data lake for further analysis by other AgiSec modules. In some examples, the AgiHack module generates an AAG of the enterpriser network by extracting AgiDis data and employing attack rules created by cyber security experts. The AgiHack module explores attack paths an attacker can traverse to advance towards targets in order to identify possible impacts on these targets. The agile SC module604receives the AAG from the AgiHack module and maps attack tactics of the AAG to SCs. Further detail of the agile security system602and the respective modules is provided in each of U.S. application Ser. No. 16/554,846 and U.S. application Ser. No. 16/924,483 introduced above. While the agile SC module604is depicted as being separate from the agile security system602inFIG.6, it is contemplated that the agile SC module604can be included as part of the agile security system602.

In some implementations, the agile SC module604includes a configuration defining the TM and a list of constraints over SCs, indicating which SC cannot be handled due to real-systems restrictions (e.g., mandatory SCs). In some examples, the agile SC module604provides output to a graphical user interface (GUI) for manual selection of a reduction technique and a decay configuration parameter per iteration. An example GUI700is depicted inFIG.7.

For each iteration, for a given input AAG, the agile SC module604selects SCs that are to be prioritized first, according to a selected reduction algorithm and corresponding parameters. Subsequently, the agile SC module604eliminates all nodes in the AAG that represent rules and requirements that are associated with the selected SCs. Pruning of the nodes results in a reduced version of the AAG, which is referred to as a residual AAG. The removed rule nodes define the current iteration list of requirements to implement, in order of importance, according to the number of appearances of the rules in the most critical SC. The agile SC module604creates an influence histogram that reflects the residual rules that where not handled yet. If all rules related to a certain SC were handled previously, the SC is eliminated from the remaining backlog of requirements and will no longer be presented in subsequent histograms. As the simulation progresses, each former AAG becomes an input AAG for the next iteration, and different methods of reductions can be employed. Over the iterations, the residual AAGs become sparser until the attack surface is eliminated. Subsequently, the produced list of security requirements is appended to the former batch of requirements.

FIG.8is a flowchart of an example process800that can be executed in accordance with implementations of the present disclosure. In some implementations, the example process800may be performed using one or more computer-executable programs executed using one or more computing devices. The example process800can be performed for cyber-attack risk evaluation and mitigation in accordance with implementations of the present disclosure.

Data representative of an enterprise network is received (802). For example, a discovery service (e.g., executed by the AgiDis module ofFIG.6) discovers assets and vulnerabilities of the enterprise network using third-party tools. In some examples, the discovery services scans an enterprise network to identify assets (e.g., CIs) within the enterprise network and, for each asset, information representative of a configuration of the asset (e.g., identification, configuration, installed software and its version, open sessions, memory map, user group membership, network share access permission). In some examples, vulnerabilities of one or more assets can be determined. For example, each asset can be cross-referenced with a data source (e.g., MITRE ATT&K™) to identify one or more attack tactics that a respective asset is vulnerable to. An AAG is generated (804). For example, an AAG generation service (e.g., executed by the AgiHack module ofFIG.6) processes the data representative of the enterprise network to generate an AAG with respect to one or more target assets (crown jewels), the AAG representing one or more paths to each target asset within the enterprise network. In some examples, the AAG is generated as described in U.S. application Ser. No. 16/924,483, introduced above.

A security rules distribution is determined (806). For example, each rule (attack tactic) depicted in the AAG is mapped to one or more SCs (i.e., an SC that mitigates the rule). In some examples, and as described herein, a traceability matrix (TM) is provided, which defines a mapping between SCs and attack tactics (rules). In some examples, a SC can mitigate one or more rules. Accordingly, a rules distribution for each SC can be provided, each rules distribution indicating one or more rules that a respective SC mitigates. A SC influence histogram and a rules distribution are displayed (808). For example, and as described in detail herein, for each SC, an influence score is calculated (e.g., as described herein with reference to Eq. 3), which represents a degree of influence the respective SC has on cyber risk in the enterprise network (e.g., if the SC were to be implemented, how much influence the SC would have in mitigating overall cyber risk). An influence histogram with rules distribution is provided, such as that depicted inFIG.10A.

Optionally (as indicated in dashed line), a decay method and value are adjusted (810). In some examples, a decay method and value can be preset (e.g., default settings of a simulator). In some examples, the decay method and value can be automatically selected. For example, the influence histogram can be analyzed to determine a type of curve that is represented (e.g., exponential drop, monotonic reduction). In some examples, if the influence histogram is of a first type (e.g., exponential drop), then a first decay method (reduction) (e.g., gradient reduction) is selected. In some examples, if the influence histogram is of a second type (e.g., monotonic reduction), then a second decay method (reduction) (e.g., AUC) is selected. In some examples, the decay method and value are displayed by the simulator, and a user can adjust, if desired.

A reduction is applied (812). For example, and as described herein, the simulator applies the decay method to the AAG based on the value to reduce rules in the AAG, as represented in a resulting residual AAG. In some examples, the decay method is applied until the value is achieved. For example, the value represents a threshold that indicates ending of the decay method for an iteration of simulation (e.g., threshold slope angle, threshold AUC reduction). Prioritized rules and requirements are appended (814). For example, and as described herein, a residual AAG is provided and includes remaining rules as respective SCs. It is determined whether all remaining SCs are mandatory (816). If all remaining SCs are mandatory, a GRV summary is provided (818) and the simulation ends. If not all remaining SCs are mandatory, the example process800loops back to execute a next iteration of the simulation.

Implementations of the present disclosure were evaluated through execution of an experiment. The experiment was conducted on virtual network of four active servers. One of the servers had Internet access as a starting point for an attacker. Each server had a Microsoft Windows workstation connected to a Microsoft Active Directory. In the experiments, the environment (enterprise network) was contaminated with a set of vulnerabilities that can be exploited by MITRE ATT&K™ tactics for Active Directory environments. The attack target was defined as the domain controller (DC) server denoted as target X herein.FIGS.9A-9Cdepict an evolution of an example AAG over multiple iterations of the experiment.FIGS.10A-10Cdepicts an evolution of an influence histogram corresponding to the example AAGs ofFIGS.9A-9C. The experiment included a simulation employing two iterations. The first iteration employed the AUC reduction method, and the second iteration employed the Gradient reduction method. The sequence of reduction techniques was chosen according to the shape of the influence histograms.

The overall reduction results achieved in the experiment are depicted inFIGS.9A-9C and10A-10C. In the experiment, the notations of SC1, SC2, . . . , SC15are used to identify different SCs. A mapping is provided that maps attack tactics to corresponding SCs. The purpose of the depicted reduction steps is to minimize the number of possible pathways a hacker can traverse through a computer network, with the goal of ideally eliminating all paths to the DC server (i.e., target, crown jewel).

With particular reference toFIGS.9A and10A, an original state is provided and an AAG900is generated. The AAG900ofFIG.9Arepresents the original estimation of the SC gap. In the depicted example, the AAG900is a three-dimensional representation of the enterprise network in terms of nodes and edges between nodes. Shapes are used for discerning types of a nodes (e.g., spherical nodes represent facts, pyramid nodes represent rules, cubical nodes represent impacts). The AAG900ofFIG.9Aincludes 268 attack nodes, of which 159 are rule nodes representing SCRs, 324 edges, and an overall GRV of 6.108. The influence histogram1000ofFIG.10Arepresents the starting condition, which includes fifteen SCs that need improvement or are missing. The order of SCs from left to right indicates the prioritization of which SC impacts the cyber posture more and should be handled first.

With particular reference toFIGS.9B and10B, a first state is represented after execution of a first iteration. An influence histogram1002ofFIG.10Brepresents the result of applying the first reduction method of AUC with a configuration parameter of 0.4 (indicating 40% area selection). Accordingly, based on the measured data, three SCs with the highest distribution, enumerated as SC15, SC6, SC10, are removed along with two additional SCs, namely SC1, SC14. This results in the removal of respective rule nodes from the AAG900ofFIG.9Ato provide an AAG902(residual AAG) ofFIG.9B. The AAG902in hand with the influence histogram1002indicate that ten SCs remain after the first iteration. The AAG902is much smaller than the AAG900, now including 146 nodes, of which sixty are rule nodes representing SCRs, 168 edges, and a GRV of 1.624. As a result, a decay ratio of 73% in GRV and 62% decay ratio of SCRs were achieved.

With particular reference toFIGS.9C and10C, in the second iteration, the Gradient reduction method was applied with gradient value of 0.61, which eliminated SC2and SC7. Since SCs SC9, SC3, SC4, SC8include the same rules as SC2and SC7, they were also removed from the inherited AAG containing four residual SCs. This results in the removal of respective rule nodes from the AAG902ofFIG.9Bto provide an AAG904(residual AAG) ofFIG.9C. The AAG904in hand with the influence histogram1004indicate that four SCs remain after the second iteration. The AAG904is smaller than the AAG902and includes ninety nodes, of which thirty-four are rule nodes representing SCRs, ninety-five edges, and an overall GRV of 1.02. As a result, an overall a decay ratio of 37% in GRV, and 43% decay ratio of SCRs, were achieved.

In this example, the remaining four SCs represented in the histogram1004include mandatory SCs, namely SCs that cannot be addresses either due to limitations of real systems, or lack of existing SCs at all, indicating a need for future investment. As such, the simulator highlights the order of SCs to be optimized due to lack of tuned implementation, and ones that are needed to be implemented in the future.

In the simulation of the experiment, automatic reduction methods were employed. It can be noted that other selectin criteria can be used, such as Top K and customized selection. The Gradient, AUC, and Top K methods are targeted to remove the most impacting SCs and the custom selection method is aimed at performing manual adjustments.

FIG.11depicts a graph1100representing the decrease of GRV over the simulation for the example experiment. The graph1100illustrates how AAG residual risk is decayed over time, and the level of hackability is reduced once the proposed SCs are implemented. As such, several different decaying method combinations can be explored, in order to find the best decaying curve amongst various decaying methods.

In accordance with implementations of the present disclosure, sub-sets of security controls can be implemented in the enterprise network based on the results of a simulation. For example, and as described herein, each simulation provides a series of sub-sets of security controls and a resulting profile for decrease in GRV. In some examples, at least partially in response to a profile for decrease in GRV, a series of sub-sets of security controls can be implemented in the enterprise network that is represented by the AAG to mitigate cyber risk in the enterprise network. For example, and with reference to the experiment detailed above, a first sub-set of security controls (e.g., SC15, SC6, SC10) can be initially implemented in the enterprise network, and a second sub-set of security controls (e.g., SC2, SC7) can be subsequently implemented in the enterprise network.

As described herein, the CDT platform of the present disclosure provides an approach to automatically gather SCRs based on current security exposure of an enterprise network by analyzing a unique digital twin at least partially provided as an AAG. Implementations of the present disclosure also provide for simulating the implementation of SCs in view of identified SCRs in order to assess their impact on the overall cyber risk reduction of the attack surface. The digital twin of the present disclosure combines detailed information about network assets such as computers, user accounts, firewall rules, and such, with associated known attack tactics.

Further, implementations of the present disclosure provide a simulator that evaluates a proportion of each SC's contribution to the cyber-attack pathways and provides multiple methods to simulate attack surface reduction through potential implementation of SCs. Accordingly, the simulator enables automatic gathering of SCRs that represent where SCs can be implemented, and enables fast reduction of cyber impact and an ordered prioritization of SCs to optimize, followed by constrained or missing security controls for future implementation. The simulator also provides transparency for decision makers regarding the impact of separate SCs and potential risk decay by selecting the order of SCRs and enabling a “what-if” simulation for evaluating the speed of risk reduction. The simulations may be used as a valuable tool in cyber security existing spending and future budget analysis by proposing what needs to be fixed now with employed SCs, and which SCRs, for which SCs are absent.

This section describes how attack graphs could be integrated with a public knowledge graph (KG) such as D3FEND KG, and how advanced analytics over the attack graph enables automated recommendation of relevant security controls. In some examples, the KG can be a defensive model of a D3FEND framework. A D3FEND KG is a catalog of defensive cybersecurity techniques and their relationships to offensive/adversary techniques.

FIG.12provides conceptualization diagram1200of how information can be integrated. The AAG1202includes node types of fact nodes1204, impact nodes1206, and rule nodes1208. In some examples, a fact node indicates facts that are provided as input within a configuration. In some examples, impact nodes indicate a derived fact that results from applying one or more input facts and/or one or more derived facts to a rule. A fact node can represent a system or network configuration that is a condition that provides possibilities for actions by an attacker,

Every rule type1208is mapped to one or more MITRE attack tactic1210, and every tactic1210is mapped to one or more rule type1208. Furthermore, every fact1204and impact1206type have mapping to one or more digital artifact1220, or asset. Each attack tactic1210has relation with at least one digital artifact1220and each digital artifact1220has relation with relevant security controls1230. Therefore, given an algorithm that returns the most impactful nodes over the AAG1202, this integration allows automatic detection of relevant security controls1230from the KG.

FIG.13depicts a diagram of an example architecture1300of the disclosed techniques. The architecture1300includes a cyber digital twin creation system1301and an analytical service1303. The cyber digital twin creation system1301obtains specifications1302. The specifications1302include inference rules, fact types, and impact types. The cyber digital twin creation system1301obtains evidences1304. The cyber digital twin creation system1301generates1310an AAG from the specifications1302and evidences1304.

The cyber digital twin creation system1301obtains a KG1306. The cyber digital twin creation system1301performs a KG enrichment1312of the AAG by integrating1314the KG with the AAG. The cyber digital twin creation system1301integrates1314the KG with the AAG using a mapping file1308. The mapping file1308includes data identifying connections between AAG node types and concepts from the KG. In some examples, the connections between the AAG node types and the KG concepts can be mined automatically, e.g. using natural language processing techniques. In some examples, the connections between the AAG node types and the KG concepts can be manually modelled by a user.

The integration1314includes adding connections between the AAG and the KG based on the mapping file1308. A graph database1315loads the AAG, the KG, and the mapped connections between the AAG and the KG. The integrated AAG is stored in a graph database1315.

The analytical service1303reads graph data from the graph database1315. The analytical service1303applies a where to cut algorithm1316. The analytical service1303prioritizes a list of AAG nodes using the where to cut algorithm1316. The analytical service1303detects countermeasures1318using the prioritized list of AAG nodes. The analytical service1303prioritizes the countermeasures, e.g., security controls, based on the prioritized list of AAG nodes. The analytical service1303provides the prioritized list of countermeasures to a consumer1320.

FIG.14depicts an example process1400for integration of a public knowledge graph with an AAG. The process1400can be performed by the cyber digital twin creation system1301. The process1400includes obtaining specifications that can include inference rules1404, fact types1402, and impact types1406. The process1400also includes obtaining evidences collected from the network, such as fact instances.

The process1400includes generating1310an AAG1405. The AAG1405can be extended to the form of a process-aware attack graph, or integrated AAG1415. The example shown inFIG.14shows that five evidences, or facts (F)1402collected from the system trigger five instantiations of rules (R)1404that create five potential impacts (I)1406.

The process1400includes obtaining a publicly available OWL/RDF file of the KG1410, and importing the KG1410to a graph database, e.g. graph database1315. The example shown inFIG.14shows that five attack tactics (T)1408are related to three digital artifacts (A)1412, that could be protected by four security controls, or countermeasures (C)1414.

The process1400includes obtaining a mapping specification, e.g., mapping file1308, between elements of the KG1410and elements of the AAG1405, and creating relationships within the graphs. The mapping includes connections1416between rule types1404and attack tactics1408, between fact types1402and digital artifacts1412, and between impact types1406and digital artifacts1412.

The example integrated AAG1415shown inFIG.14includes mapped connections between rules instances within the AAG1405and attack tactics within the KG1410. This can be done by creating a new edge between each rule instance node to an attack tactic node according to a predefined mapping between rule types and attack tactics. A process for detailed mapping between nodes of the graphs can include mapping each rule1404to a MITRE attack tactic1408during design time, since rules can express attack tactics.

Digital assets, or artifacts1412can represent configuration items such as application software, machines, files, etc. Facts1402represent pieces of information collected from individual configuration items. Facts1402can be defined within the context of digital assets ontology. For instance, ‘Service Application’ and ‘Host’ are entities within the digital asset ontology. An example for a fact that enables spotting an attack is ‘Service Access.’ This fact expresses whether its executable path may be modified by an adversary. This fact can be connected within the attack graph to ‘Service Application’ and ‘Host’ entities within the KG1410. The same is applicable for impacts that represent how implementation of an attack tactic could affect these digital assets.

The Digital Artifact Ontology and Defensive model can be extended by adding constraints that apply to specific asset/remediation pair. An example constraint is a maintenance schedule of a specific machine, which prevents restart for security update installation. At a pipeline plant, a maintenance window may constitute, for example, a few hours a year.

FIG.15depicts an example process1500for prioritizing security controls using a public knowledge graph, e.g., KG1410. In general, the process1500includes obtaining an AAG and generating a prioritized list of nodes that should be eliminated to reduce a systems' risk. The example shown inFIG.15shows that addressing issues related to nodes R1, R2, R3will eliminate the graph and reduce the cyber risk.

After the construction of the AAG and the alignment of each AAG node to its counterpart in the KG, as in the process1400, the analytical service1303can run a where-to-cut algorithm1316over the AAG1405. The where-to-cut algorithm can include using a mitigation simulator to simulate facts removal. In some examples, nodes of the AAG1405can be prioritized by calculating the importance of fact nodes of the AAG1405. The importance can be calculated, for example, using a reverse eigenvector centrality to calculate importance scores. Fact nodes of the AAG1405with higher importance scores can be prioritized over fact nodes with lower importance scores. Prioritization of AAG nodes is further described in U.S. Ser. No. 17/675,330, the disclosure of which is expressly incorporated herein by reference in the entirety.

Results of the where-to-cut algorithm1316include a prioritized list of AAG nodes, or issues that should be addressed in order to mitigate the risk to the system. Then, for each issue, the integration of the AAG1405with KG1410is leveraged to detect relevant security controls.

Attack tactics observed in the integrated attack graph1415can be mitigated using various security controls. Thus, the process1500includes identifying a subset of the most effective security controls through countermeasure detection1318. This can be done in various ways. For example, security controls, or countermeasures1414, can be prioritized by an amount of issues, or nodes, in the AAG1405that the security controls can resolve. This could be done by running a reachability analysis from each security control to the AAG rule nodes returned by the where-to-cut algorithm1316. Then, countermeasures with higher reachability will be prioritized.

Reachability analysis can be performed by finding shortest paths between each KG node and the AAG rule nodes. Reachability analysis can be performed, for example, using Dijkstra's algorithm for finding the shortest paths between nodes in a graph. Countermeasures, or security controls, with higher numbers of shortest paths to AAG nodes can be assigned a higher reachability, and can be prioritized over countermeasures with lower numbers of shortest paths to AAG nodes. The results of countermeasure detection1318includes a prioritized list of countermeasures.

FIG.16depicts results of example integrated AAG1600for performing prioritization of security controls. In some examples, security controls represented by countermeasure nodes1606, can be prioritized by their residual risk. In some examples, risk prediction methodology can be applied to an integrated AAG to predict the system's risk. In an iterative process, at every step a greedy search can be performed to detect the security control with the maximal risk reduction. For example, the security control with the maximal risk reduction can be the security control that removes the set of AAG rule nodes1602that reduces the risk the most. Then, the process includes removing the security control's related issues, or nodes, from the AAG, until risk is reduced to an acceptable level.

The example integrated AAG1600includes an AAG1605integrated with a KG1610. In the example integrated AAG1600, issue 1 includes a fact node of Email11612, a rule node of CollectbyAttacker1614, and an impact node of Email1 Compromised1616. The integrated AAG1600includes a connection between the rule node of Collectbyattacker1614an attack technique, or tactic node, of Email Collection1618. The tactic node of Email Collection1618is connected to a digital artifact node of Email1622. The digital artifact node of Email1622is connected to countermeasure nodes1606of Homoglyph Detection1624, Reputation Analysis1626, and Reputation Message Transfer Agent (MTA) Analysis1628.

In the example integrated AAG1600, issue 5 includes a fact node of File11632, a rule node of FileFound1634, and an impact node of FileCompromised1636. The integrated AAG1600includes a connection between the rule node of FileFound1634and an attack technique, or tactic node of File Discovery1638. The tactic node of File Discovery1638is connected to a digital artifact node of File1642. The digital artifact node of File1642is connected to the countermeasure node of Reputation MTA Analysis1628.

The analytical service1303prioritizes the list of AAG nodes, or issues. In the example ofFIG.16, issue 1 has a highest priority and issue 5 has a lowest priority of the list of issues1648. The analytical service1303performs countermeasure detection to detect countermeasures for the list of issues1648. The analytical service1303determines a prioritized list of countermeasures1650. The prioritize list of countermeasures1650includes Reputation MTA Analysis1628having a highest priority, Reputation Analysis1626having a second highest priority, and Homoglyph Detection1624having a third, lowest priority.

As described with reference toFIG.15, countermeasure prioritization can be performed using various criteria. In some examples, countermeasures can be prioritized based on the number of fact nodes of the AAG that are resolved by the countermeasures. For example, a first countermeasure that resolves more AAG nodes can be prioritized over a second countermeasure that resolves fewer AAG nodes. In some examples, countermeasures can be prioritized based on achieving a specified amount of risk reduction while reducing or minimizing the number of actions required. In some examples, countermeasures can be prioritized based on achieving a specified amount of risk reduction while reducing or minimizing the cost required.

In some examples, the subset of prioritized security controls can be consumed by a system that automates the repair. In some examples, a list of the subset of prioritized security controls can be provided to a user. Furthermore, given a set of constraints, a remediation plan can be formulated as an optimization problem in order to offer a solution that takes into account business constraints of performing security remediation.

Implementations and all of the functional operations described in this specification may be realized in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Implementations may be realized as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them. The term “computing system” encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus may include, in addition to hardware, code that creates an execution environment for the computer program in question (e.g., code) that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagated signal is an artificially generated signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode information for transmission to suitable receiver apparatus.

A computer program (also known as a program, software, software application, script, or code) may be written in any appropriate form of programming language, including compiled or interpreted languages, and it may be deployed in any appropriate form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program may be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

The processes and logic flows described in this specification may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and apparatus may also be implemented as, special purpose logic circuitry (e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit)).

Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any appropriate kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. Elements of a computer can include a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data (e.g., magnetic, magneto optical disks, or optical disks). However, a computer need not have such devices. Moreover, a computer may be embedded in another device (e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio player, a Global Positioning System (GPS) receiver). Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, implementations may be realized on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display), LED (light-emitting diode) monitor, for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball), by which the user may provide input to the computer. Other kinds of devices may be used to provide for interaction with a user as well; for example, feedback provided to the user may be any appropriate form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any appropriate form, including acoustic, speech, or tactile input.

Implementations may be realized in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which a user may interact with an implementation), or any appropriate combination of one or more such back end, middleware, or front end components. The components of the system may be interconnected by any appropriate form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”) (e.g., the Internet).

The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

While this specification contains many specifics, these should not be construed as limitations on the scope of the disclosure or of what may be claimed, but rather as descriptions of features specific to particular implementations. Certain features that are described in this specification in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation may also be implemented in multiple implementations separately or in any suitable sub-combination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination may in some cases be excised from the combination, and the claimed combination may be directed to a sub-combination or variation of a sub-combination.

Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products.

A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. For example, various forms of the flows shown above may be used, with steps re-ordered, added, or removed. Accordingly, other implementations are within the scope of the following claims.