Patent Description:
Faults often occur in cloud networks and systems and contribute to a significant portion of cloud operational costs. For instance, an infrastructure fault (e.g. a central processing unit (CPU), memory or hard disk drive (HDD) fault) detected at one node of an edge cloud, if not properly handled, may propagate and may spread up towards the application level. It may also propagate across several cloud nodes in the same cluster of the edge cloud. Once spread, it can be hard to identify and trace the root cause of the fault, which can delay the identification of appropriate fault recovery and prevention procedures. Recent developments in fifth generation (<NUM>) networks have involved various technologies related to the edge and distributed cloud computing; the challenge of fault management (e.g. recovery procedures, prevention procedures) in such networks is threefold.

First, due to the heterogeneous devices and cloud deployments, it is very difficult to select the appropriate procedure to recover or prevent such faults in all the network and system domains. When such faults are detected, it requires human feedback or input to identify appropriate action to solve or handle the identified faults. This approach requires a lot of manual configuration and prior knowledge about the metrics and actions having a direct impact on the faults in a specific device or cloud. In addition, manual troubleshooting cannot guarantee to cover all fault scenarios and to propose the appropriate recovery or prevention procedures given the complexity and the size of current networks and systems. Hence, troubleshooting expertise can only be mastered after years of experience, especially in large cloud systems.

Second, the selection of appropriate fault management procedures for each domain and scenario becomes a challenge to avoid fault propagation. For example, procedures used in a data center may be very different from those used in content delivery networks. In addition, several procedures can be used to recover a detected or predicted fault and these procedures may perform differently under different circumstances. Furthermore, cloud systems may change in unforeseen ways because the workload and the cloud infrastructure can change over time, which may lead to the occurrence of new faults that require new recovery procedures. Consequently, there is no explicit, global, troubleshooting methodology that can be automated and utilized in such network and system domains due to the dynamic and complex nature of cloud systems.

Third, the cloud infrastructure and the available resources may change frequently, especially in edge domains. This introduces a challenge for a selected recovery and prevention procedure as it needs to adapt to changes due to the dynamic and unpredictable cloud contexts. For example, a method used to recover a CPU fault may become unusable or inappropriate (e.g. it may take a longer time than expected or it cannot be applied on the processor) according to available resources from one domain to another.

<CIT> describes a system for managing network capacities. The system comprises a processor implementing a data collection module configured to collect a plurality of capacity utilization data. The processor also implements a capacity analysis module configured to produce an actual capacity utilization map using the plurality of capacity utilization data. The capacity analysis module is also configured to produce a projected capacity utilization map using a plurality of designed node and link capacities, and a plurality of capacity engineering constraints. The processor also implements a simulation module configured to simulate a network management action and to produce simulated capacity utilization maps. The processor also implements a network management action module configured to suggest network management actions based on the actual capacity utilization map and the projected capacity utilization map and to rank the one or more network management actions based on the one or more simulated capacity utilization maps.

<CIT> describes a technique for managing actions implemented on a network element within a telecommunications network. A method includes generating a list of actions from an action catalogue, each of said actions being a candidate for implementation on the network element, ranking the list of actions in accordance with one or more ranking parameters indicative of the suitability or perceived of said actions, and providing at least a portion of the ranked list of actions to a management agent arranged to manage said network element to enable the management agent to select and implement one, none or some of the actions.

<NPL>, proposes a framework to detect and predict several faults at infrastructure-level of edge clouds using supervised machine learning and statistical techniques. The proposed framework is composed of three main components responsible for: (<NUM>) data pre-processing, (<NUM>) fault detection, and (<NUM>) fault prediction.

<NPL>, describe PAT, a toolkit for flexible and efficient system analysis under fairness. A unified algorithm is proposed to model check systems with a variety of fairness effectively in two different settings. PAT complements existing model checkers in terms of fairness.

There is therefore a need to design an automated method to determine the appropriate recovery or prevention procedures for detected or predicted faults in cloud systems.

There is provided a method for automatically managing an event in a cloud system. The method comprises determining a candidate action to be applied to the cloud system for managing the event. The method also comprises applying the candidate action to a model of the cloud system. The method also comprises, upon determining, when the model of the cloud system meets at least one performance indicator, that the candidate action is a proved action, applying the proved action to the cloud system. The event is predicted.

There is provided a system for automatically managing an event in a cloud system. The system comprises processing circuits and a memory, the memory containing instructions executable by the processing circuits. The system, upon executing the instructions, is operative to determine a candidate action to be applied to the cloud system for managing the event. The system is also operative to apply the candidate action to a model of the cloud system. The system is also operative to, upon determining, when the model of the cloud system meets at least one performance indicator, that the candidate action is a proved action, apply the proved action to the cloud system. The event is predicted.

There is also provided a non-transitory computer readable media having stored thereon instructions for managing an event in a cloud system. The instructions can comprise any of the steps described herein.

The methods, systems and computer readable medias provided herein present improvements to the way fault management in cloud platforms operate.

Various aspects, features and embodiments will now be described with reference to the drawings to fully convey the scope of the disclosure to those skilled in the art.

Sequences of actions or functions may be used within this disclosure. It should be recognized that some functions or actions, in some contexts, could be performed by specialized circuits, by program instructions being executed by one or more processors, or by a combination of both.

Further, computer readable carrier or carrier wave may contain an appropriate set of computer instructions that would cause a processor to carry out the techniques described herein.

The functions/actions described herein may occur out of the order noted in the sequence of actions or simultaneously. Furthermore, in some illustrations, some blocks, functions or actions may be optional and may or may not be executed.

Referring to <FIG>, which is a schematic representation of an action, the action <NUM> includes information about one or more task or function to be performed to repair a specific fault. Herein, the action is defined as a combination of the following information: an identifier (ID) <NUM>, one or more inference rules <NUM>, the target component <NUM> experiencing the fault, one or more key performance indicators (KPIs) <NUM> to be improved, and a success rate <NUM> the action achieves. A person skilled in the art would realize that the action could be defined differently, with more or less information and could still be used to achieve a similar purpose.

And inference rule <NUM> is a logical transformation or function that can be used to infer a conclusion, it comprises at least one premise <NUM> that is used to create an argument. It takes the premises, analyzes their syntax, and returns one or more conclusion <NUM>. According to this proposed definition, an inference rule <NUM> can be composed of a set of premises <NUM> and a set of conclusions <NUM>.

A premise <NUM> is a sequence of propositions that a given statement will justify and return a specific conclusion. Herein, a premise includes a fault type <NUM> and one or more axiom <NUM> or a set of axioms.

A fault type <NUM> refers to the nature of the experienced fault (memory, HDD, CPU, network, etc.).

An axiom <NUM> is a proposition or statement that serves as a premise leading to a specific conclusion or reasoning when its value is true.

A conclusion <NUM> is the logical consequence or deduction that is obtained given that some statements or propositions are true. Herein, it is the results of the combination of (<NUM>) the fault type <NUM> and (<NUM>) the axiom <NUM> values. This means that whenever the fault type <NUM> and axiom <NUM> are true, then the conclusion <NUM> is true.

A component <NUM> is an element of the monitored system in relation to which a fault may occur and to which an action <NUM> will be applied to repair a given fault.

A mapped KPI <NUM> is a type of performance measurement for an affected KPI associated with a specific fault.

A success rate <NUM> is a calculated percentage of successes among a number of attempts when applying a specific action with respect to a specific fault.

Different approaches exist concerning reasoning for fault management. These existing approaches can be grouped in four different classes: rule-based reasoning, case-based reasoning, model-based reasoning and machine learning or artificial intelligence (AI).

Rule-based reasoning combines a collection of rules, where each rule has the form of Boolean expression and action. When a Boolean expression is matched with a problem or fault, a corresponding action is executed. However, it this has several shortcomings: it requires exact matching, it fails if the fault is not anticipated by the rules, and it requires explicit updates for new faults.

Case-based reasoning can reason based on past problems and solutions. It is case-specific, and it is not easy to generalize to all cases. Case-based reasoning can incorporate the learning of new cases. A large case base improves the technique but can slow things down.

Model-based reasoning refers to developing a model of the physical world. Then, at run time, the model build from previous knowledge is combined with observed data to derive conclusions such as diagnosis or prediction. Key limitations of model-based reasoning include model validation, model re-use for another system, and the degree of model accuracy. In addition, model-based reasoning can handle problems explained by the model only.

Machine learning or AI based approaches require the compilation of libraries of healthy and fault patterns for the performance of a device. These libraries do not provide knowledge-rich structures or justifications for device behavior or failures.

Automatic and adaptive fault reasoning-management is a challenging task. It is difficult to design efficient and appropriate models to deal with fault management in any cloud platform or software system. Hence, it is of interest to design an automated solution to overcome the above-mentioned limitations.

Described herein is a method and system to automatically reason and select an appropriate action to be applied to recover from and/or prevent faults in cloud systems. The method comprises:.

Some of the advantages of such a method/system, includes:.

Although the detailed description presented herein highlights an automated reasoning solution for fault management, the underlying method can be generalized to cover other applications, such as performance or security management.

<FIG> illustrates a system <NUM> which comprises three main components. The first component is an "Action Selector" <NUM> which identifies the main characteristics of a detected or predicted fault provided by a fault detector-Predictor <NUM> and which combines a set of inference rules to find a good candidate action (selected within an action pool <NUM>) to be applied on the cloud system to recover from the fault or prevent its occurrence. The input of this component is mainly the detected or predicted fault and the output is mainly a candidate action to be applied on the cloud system.

The second component is a "Cloud Model Analyzer" <NUM>, which takes as input a description of a given cloud system <NUM> and which provides as output a formal description of the cloud system. The main goal of the cloud model analyzer is to provide a high-level representation, or description of the architecture, of the cloud system as well as the different logical connections between its blocks using a formal language.

The third component is an "Action Reasoner" <NUM>, which uses a candidate action output from the "Action Selector" component, applies it to the formal cloud model output from the "Cloud Model Analyzer" component and verifies its corresponding changes on the "formal cloud model". Next, the action reasoner checks whether the cloud model meets certain input specifications. When the specifications are met, the action reasoner approves the candidate action and notifies the "Action Selector" to apply it on the real cloud system. Next, the "Action Selector" gets feedback about the system's new state with respect to the detected fault and its recovery. The applied action and its feedback are stored in an "Action Pool" for future uses (e.g., future detected similar faults having the same characteristics). Otherwise, it automatically reasons and analyzes the system performance and/or traces to propose other appropriate candidate actions to be applied or to identify appropriate adjustments to the "candidate action". The identified adjustments are sent to the "Action Selector" to modify the proposed candidate action accordingly.

The Performance Detector-Predictor <NUM> and the Security Detector <NUM> are elements that could be switched with the Fault Detector-Predictor <NUM> to provide other functionalities to the cloud system, based on the same technique.

<FIG> illustrates the Fault Detector-Predictor <NUM> of <FIG> in more details. The Fault Detector-Predictor <NUM> uses offline historical data samples to train a model using machine learning algorithms (i.e., Neural Network, Random Forest, Support Vector Machine, etc.). Once the model is trained, online data samples that are collected at different times (which may be at regular time intervals) can be fed to the model to determine or predict faults which will trigger the automated reasoner. The Performance Detector-Predictor <NUM> and the Security Detector <NUM> could be implemented using similar techniques.

<FIG> presents a detailed flowchart of the different steps involved in the proposed process <NUM>. Given the cloud system description, step <NUM>, the system builds and generates a formal model for the corresponding cloud system, step <NUM>, that describes the logical connections between the blocks of the cloud system.

Given a new online data sample, the Fault Detector and Predictor is able to identify the occurrence of a fault, step 403a or 403b, when or before it happens. When no fault is detected or predicted, the Fault Detector and Predictor waits for the next online data sample, step <NUM>. Then, the Fault Detector and Predictor waits until getting a new data sample from the monitored cloud system.

A data sample may include metrics concerning the status of CPU, storage, memory, Input/Output, temperature, node capacity, etc. at a given time, such as go _memstats_heap_inuse_bytes, node_load1, go_memstats_alloc_bytes, go_memstats_heap_alloc_bytes, node_vmstat_nr_unevictable, node_memory_Dirty, node_memory_Unevictable, node_vmstat_nr_mlock, node_memory_Mlocked, go_memstats_stack_inuse_bytes, etc..

In the presence of a fault, the Fault Detector and Predictor determines the main characteristics of the detected or predicted fault (e.g., comparing the metrics between the no-faulty state and the faulty state and getting the deviation from the normal cloud system state). The Fault Detector and Predictor checks, step <NUM>, whether there exist similar faults stored in the "Action Pool" that were previously analyzed by the system (e.g. it checks metrics describing the fault and their deviations from the normal range, the metrics may be ordered according to their importance).

When similar faults are found, the Fault Detector and Predictor applies the selected action on the real cloud system, step <NUM>, and updates the Action Pool and the cloud formal model accordingly, step 412b.

When no similar faults are found, the Fault Detector and Predictor selects a combination of the input inference rules that would compose the new candidate action given the characteristics of the identified fault, step <NUM>.

The proposed candidate action is applied on the formal cloud model to verify its usefulness and efficiency, step <NUM>.

The Fault Detector and Predictor checks, step <NUM>, whether the proposed candidate action meets an input specification to determine its efficiency to recover or prevent the fault. The input specifications are the KPIs that reflect the characteristics of the system when it recovers or prevents a fault. Those KPIs can be monitored through variables to track change on the system.

When the KPI specification is met, the "Action Selector" applies the approved action on the real cloud system, step <NUM>, and get a feedback about the new cloud system status after recovering the fault (e.g. fault recovered or not, metrics related to the corresponding fault). The received feedback is later used to store the applied action and its system feedback within an Action Pool, step 412b. The Action Pool is used to save the cost associated while looking for the appropriate action for future similar faults. The received feedback is used as well to update the formal cloud model to reflect the new changes accordingly, step 412a.

When the KPI specification is not met, the "Action Selector" analyzes the obtained results from the formal cloud model analyzer and identifies the possible reasons that lead to the failure of "candidate action". The "Action Selector analyzes and reasons over the obtained "possible causes" and determines adjustments (e.g. new inferences rules) to the "Action Selector" to modify the candidate action, step <NUM>. When no new rules/candidates are discovered, the system generates an alarm to notify the user to give his input or feedback with respect to the triggered scenario.

<FIG> and <FIG> present the architecture and the flowchart of the "Action Selector", respectively.

<FIG> illustrates the architecture of the Action Selector. Illustrated there is a "Features Deviation Analyzer" <NUM> that is responsible to identify a list of deviated features, step <NUM>, in the presence of a detected-predicted fault, given an online data sample from the monitored system. Precisely, the Features Deviation Analyzer checks whether the online data sample (value) is within the range of the statistical properties (e.g. mean, standard deviation, etc.) of the training data used to train the detection and prediction model.

Given the list of the deviated features from the previous step, the "Fault-Features Similarities Analyzer" first checks whether there were similar faults previously discovered by the "Action Selector", step <NUM>. The "Fault-Features Similarities Analyzer" <NUM> checks if there exists a previous fault with similar deviated features in the "Action Pool" <NUM>. Next, it sorts the identifies similar faults by the amount of the deviation for each feature compared to the training data. Finally, the "Fault-Features Similarities Analyzer" select the most relevant "N" similar faults. "N" can be tuned with a random or specific variable based on the feedback from the monitored system.

Given the list of top "N" similar faults, step <NUM>, the "Fault-Features Similarities Analyzer" compares the amount of the deviation with an input threshold (e.g., <NUM>%, <NUM>%, <NUM>%) that is initialized by the user and tuned later when receiving system feedback. Particularly, the "Fault-Features Similarities Analyzer" selects the most similar faults, step <NUM>, and ranks these faults based on their deviation similarities results and it chooses the one with the highest score. If the system shows the same deviation in the presence of faults, that means that the same action can be applied to get it back to the normal state. Therefore, the same action applied previously can be used on the most similar fault.

When the specified threshold is not met or no similar faults were found, the "Fault-Features Similarities Analyzer" will initiate the process of finding new candidate action, step <NUM>, based on the characteristics of the detected-predicted fault.

A "Rules Conflict Solver" <NUM> analyzes the list of identified similar faults and their composing inferences rules. A combination of the rules can be found from the actions applied to similar faults to go back to the normal state. The goal of this solver is to go through the inference rules of the identified similar faults and get the list of the ones that are non-conflicting, step <NUM>. The solver can be modeled as a constraint satisfaction problem (as described below) which can be solved using one of the known algorithms: backtracking, forward checking, maintaining arch consistency.

The Constraints Satisfaction Problem consists of assigning values to variables while satisfying certain constraints. Constraints Satisfaction Problem consists of three components:.

Finite set of variables which are the conclusions in the inference rules of the selected actions.

Finite set of values for each variable <MAT>.

where each variable can take one of the values in V. <MAT> <MAT> <MAT> <MAT>.

Finite set of constraints between the variables. There are two types of constraints: combination of conclusions in inference rules should reside in the same action, and combination of conclusions in inference rules should not reside in the same action.

The Constraints Satisfaction Problem can be represented by a graph where nodes correspond to variables and edges correspond to constraints, as shown in <FIG>. The problem is to assign a value for each variable such that the constraints are met. The problem can be solved by backtracking, forward checking, maintaining arch consistency.

When no conflicting rules or no similar faults are found, the solver select some rules from the "Rule Pool" <NUM> to compose one or more new candidate action(s), step <NUM>.

A "Rules Optimization Analyzer" <NUM> uses the resulting non-conflicting rules from the previous step, or the rules selected by the solver (when no similar faults were found) to find a good (aiming towards optimal) combination of inference rules. The main objective of the analyzer is to determine a new candidate action to be applied on the cloud system to recover or prevent a fault. Particle Swarm Optimization (PSO) can be used to find a candidate action, step <NUM>. PSO is selected because it can optimize a problem by iteratively trying to improve a candidate solution with regard to a given measure. The PSO tries to move the candidate solutions called particles around the search-space according to a mathematical formula as explained below.

Quantum Particle Swarm Optimization (QPSO) is a discrete version of PSO to solve problems with binary-valued solution elements.

Simulated Annealing (SA), Genetic Algorithm (GA), and Column-Generation (CG) are other examples of algorithms that could alternatively be used. PSO is chosen because of its effectiveness in solving a wide range of applications. It has the ability to find optimal or near-optimal solutions for large-space problems in a short time compared to other heuristics.

The QPSO steps are now described. First, the particles are initialized. A particle is defined based on the quantum bit. Two vectors are initialized.

A quantum particle vector V(t)i, which is the velocity for particle i and is initialized to random values between [<NUM>,<NUM>]: <MAT> and a discrete particle vector p(t)i, which is initialized by initializing a random number for each <MAT>.

Then, according to the condition in (<NUM>) and (<NUM>), the discrete particle vector p(t)i is initialized: <MAT> where n is the size of the problem, i.e., the total number of non-conflicting rules. <MAT> <MAT>.

The initial population is evaluated by calculating the objective function for each particle, e.g., optimize the mapped KPI.

The particles that represent non-dominated solutions are stored in a repository REP. Each particle keeps track of its best local position, which is the best solution obtained by this particle so far (PilocalBest).

At each iteration, the algorithm selects PglobalBest that denotes the best position achieved so far by any particle in the population. There are several ways to select PglobalBest. One way is to rank the solutions in the repository and choose the one with the highest rank.

The velocity equation is updated according to equation (<NUM>) and the particle vector is updated in the same way as in equations (<NUM>) to (<NUM>) <MAT> <MAT> <MAT> where α + β = <NUM>, β < <NUM>, <NUM> < α, α and β are control parameters, w represents the degree of belief on oneself, c<NUM> is the local maximum, and c<NUM> is the global maximum.

Pilocalbest is updated by applying Pareto dominance. If the current position is dominated by the one in the memory, the one in the memory is kept; otherwise, the one in the memory is replaced by the current position. Algorithm <NUM> shows how the algorithm can be implemented.

An "Action Evaluator" <NUM> is responsible for applying the proposed action on the cloud system, step <NUM>, with respect to the detected-predicted fault.

Next, the Action Evaluator gets the system feedback, step <NUM>, on whether the applied action was able to properly handle the fault or not.

According to the result of the previous steps, the Action Evaluator also updates the parameters "N" and "the similarity threshold", step <NUM>. For instance, one possible update could be to enlarge the search space for the most similar faults by increasing the value of "N". Another possible update could be to increase the similarity threshold to make sure that the system will recover from the fault when applying the same action.

Based on the received feedback, the Action Evaluator stores those information (e.g., fault, deviated features, applied action, system feedback) on the "Action Pool", step <NUM>.

<FIG> and <FIG> present the architecture and the flowchart of the "Cloud Model Analyzer" <NUM> in the system, respectively.

The goal of the "Cloud Model Analyzer" is the construction of a formal model for the cloud system and its properties. To this aim, the Communication Sequential Processes language is used to formally model the cloud system because it enables the modeling of synchronous and concurrent systems. Specifically, it enables to model the behavior and communication of multiple processes and parallel components for different distributed systems.

Linear Temporal Logic (LTL) is also used to provide a description of the properties that are verified. In the context of fault management, the properties to verify are the KPI of the monitored cloud system when applying a specific action to recover or prevent a fault.

Concretely, a model is constructed to formally describe the components of the cloud system including the dependencies between the components, updates of the system and the KPI updates when an action is applied.

A "Components Analyzer" <NUM> is responsible for getting the list of the different blocks composing the cloud system, steps <NUM> and <NUM>, with respect to the input given from a user and a system description.

A "Dependencies Analyzer" <NUM> is responsible for representing a high-level description of the connections between the identified components, step <NUM>.

An "Actions Analyzer" <NUM> is responsible for tracking the changes on the formal model of the cloud after applying an action, step <NUM>.

A "KPI Analyzer" <NUM> is responsible for tracking the change on the KPI of the cloud system and the formal model accordingly, with respect to the changes on the applied actions, step <NUM>.

A "Formal Model Builder" <NUM> is responsible to combine the output of the previous analyzer (actions, components, KPI,. ) to build a formal model of the cloud, step <NUM>. Next, the Formal Model Builder checks the validity of the proposed model, step <NUM>, and tunes its parameters accordingly, step <NUM>.

<FIG> and <FIG> present the architecture and the flowchart of the "Action Reasoner" <NUM> in the system, respectively.

The input of the "Action Reasoner" <NUM> are: (<NUM>) a "Formal Cloud Model" and "KPI Specification" from the "Cloud Model Analyzer" and (<NUM>) a new "Candidate Action" from the "Action Selector". The output of the "Action Reasoner" are: (<NUM>) a "proved action" when the new candidate action meets the KPI specification otherwise it generates (<NUM>) "new rules" to be used to compose new candidate actions. The generated output will be sent to the "Action Selector" either to confirm applying a candidate action or to refine the received actions accordingly.

The "Action Executor" <NUM> is responsible for analyzing the input "new candidate action" and to retrieve the rules list composing it, steps <NUM>, <NUM>. Following the order in the obtained list, it formally applies each rule, step <NUM>, on the formal cloud model. Given the abstract description of the dependencies and the updates described in each process in the Communication Sequential Processes model.

The "Model Updater" <NUM> is responsible for tracking the changes on the abstract formal model when applying the rules composing the received candidate action following the flow of execution for those rules. The execution of the rules is reflected on the variables of the model that track the changes on the formal model (e.g., CPU utilization, size of queue, IO latency. ), step <NUM>.

The "Prover" <NUM> is responsible for checking whether the updated cloud model meets the input KPI specification when applying the new candidate action with respect to recovering-preventing the detected-predicted fault in the monitored system.

The prover that is used can be an existing prover available in the state of the art, including model checkers. For instance, the Process Analysis Toolkit (PAT) model checker can be used to verify the properties of the formal model to perform the formal quantitative analysis of the KPI in the cloud system. Other model checkers that could be used include NuSMV (http://nusmv. eu/), PRISM (https://www. prismmodelchecker. org/), UPPAAL (http://www. org/), TAPAAL (http://www. net/), SPIN (http://spinroot. com/spin/whatispin. html) or ROMEO (http://romeo. rts-software.

The choice of PAT is motivated by the fact that PAT is based on Communication Sequential Processes and that it showed good results to simulate and verify concurrent, real-time systems, etc..

<FIG> presents how the prover <NUM> works, in general, when receiving a property to be verified. Based on a high-level/abstract description of a given system and its environment, the prover first composes/builds a "model" that mostly reflect the functionalities of the input system and its behavior. Given the composed model, the prover can then check whether a given property reflects the system status at a given time by comparing the given value on the input property and the status of elements composing the model. Finally, the prover can generate either a proof that the property is verified or traces as counterexamples to explain why the system did not meet the input property.

Returning to <FIG> and <FIG>, the prover checks the input KPI specification, step <NUM>, according to the updated version of the formal model and the applied actions.

When the prover <NUM> generates a "YES" and a proved action, the "Action Reasoner" <NUM> notifies the "Action Selector" <NUM> to apply the proved candidate action, step <NUM>.

When the prover generates a "NO" and counterexamples traces, it sends the generated traces to the "Rules Reasoner" <NUM> to analyze them, step <NUM>.

The "Rules Reasoner" <NUM> is responsible for providing adjustments or refinements to the input candidate action that fails the verification step, meaning that the prover finds out that it is not the appropriate candidate action to overcome the fault, step <NUM> and <NUM>.

Given the traces generated by the PAT model checker, the Rules Reasoner parses the traces and extracts data about possible relations between the actions-rules and the resulting KPI. To this aim, the states are checked where the formal cloud model does (not) meet a given KPI-property (e.g. target KPI could be that the CPU utilization rate becomes less than <NUM>% to recover a CPU fault) and correlate these states to the obtained KPI performance.

As a result, the correlation between the cloud changes, the applied action, and the KPI rate can be investigated. Based on the obtained correlation results, the proposed "Rules Reasoner" can provide or suggest possible inference rules to overcome the detected or predicted fault using inductive reasoning.

Inductive learning enables a system to recognize patterns and regularities in previous knowledge or training data and to extract general rules from these patterns. The identified and extracted generalized rules can be used in reasoning and problem solving.

There exist several inductive algorithms in the literature including, for example, RULES (RULES: A Simple Rule Extraction System). RULES is a simple inductive learning algorithm for extracting IF-THEN rules from a set of training examples. Algorithms under RULES family are usually available in data mining tools, such as Knowledge Extraction based on Evolutionary Learning (KEEL) and Waikato Environment for Knowledge Analysis (WEKA), known for knowledge extraction and decision making.

For this step, inductive reasoning algorithms that generate generalizations from specific observations can be used. Precisely, it uses the generated data from a given system to make conclusions. New inference rules are built by going from the specific to the general meaning obtained from many observations to produce generalizations or a pattern according to an explanation or a theory. Here is an example of an inductive reasoning: every windstorm in this area comes from the north. I can see a big cloud of dust in the distance. A new windstorm is coming from the north.

Through the process of finding new inferences rules, step <NUM>, a timing condition is used as a stop criteria in order to notify the administrator, step <NUM>, when (<NUM>) the process is taking longer than expected or (<NUM>) the reasoner did not find new rules from the generated traces, or (<NUM>) the detected fault is critical like HDD fault that the system should find an appropriate action in a reasonable time because its severity of the read-write operations.

Some illustrative examples will now be described in relation with <FIG> and <FIG>, to demonstrate the functionalities of the system.

For instance, the system can be deployed in different cloud infrastructure environments. This is because it does not depend on the type of the cloud where it is deployed but, it is more related to the type of the experienced events or faults and the system feedback with respect to the applied corrective actions. Moreover, a self-learning solution was proposed that adapts according to the changes captured from the monitored cloud system. Therefore, as an example, Ericsson Network Functions Virtualization Infrastructure (NFVI) is selected as a potential product where the system can be deployed and tested.

NFVI is a cloud platform where telecom, operations support system (OSS), business support system (BSS), media, etc. applications are running. Those applications are sensitive and dependent on the quality of the infrastructure on which they are deployed. A fault can negatively impact the quality of service delivered to the users (i.e., media applications) as well as the performance of operations (i.e., OSS and BSS applications). The solution presented herein can be used to avoid those issues and can help in making better decisions and improving the business value for the NFVI. For example, for the OSS and BSS, the system can select the appropriate action to balance, scale up or down the running load when there is a CPU fault to improve the performance of the operations systems. As a result, it can improve the business value associated with those operations or it may align the operations with given business inputs and targets specified by the user.

Table <NUM>, <FIG>, presents some examples of actions that may be stored in the "Action Pool" in the proposed framework. Different infrastructure faults are chosen including CPU, HDD, Network at both host and virtual machine (VM) levels in a cloud environment.

In the following, <NUM> examples of faults that are (<NUM>) network overloaded, and (<NUM>) disk fault, are provided.

<FIG> shows an example of a cloud environment to better illustrate the selected examples.

In example <NUM>, the network is overloaded. Let's assume that the link bandwidths between the master node and <NUM> slave nodes are allocated as follows: <NUM>% for `Slave <NUM>', <NUM>% for `Slave <NUM>', and <NUM>% for `Slave <NUM>'. At some point, the "Fault Detector-Predictor" detects a 'Network-Overloaded' fault on link <NUM>. First, the "Action Selector" analyzes the deviation of the detected fault-data compared to the training data to find previous similar faults. In this example it is assumed that it will find a similar fault, will then check the Action Pool, and select Action #<NUM> as the Candidate Action because it was applied on an identified similar fault. By the similarity examination, the "Action Reasoner" confirms that applying Action #<NUM> is enough to resolve the detected fault. Accordingly, the bandwidth of Link #<NUM> is increased to <NUM>%, and the one for Link #<NUM> is decreased by <NUM>% (referring to Action #<NUM> in Table <NUM>).

In example <NUM>, the hard disk (HD) is full. At some point in time, a fault 'Disk Fault: full disk (VM)' on VM1 is predicted by the "The fault Detector-Predictor". To proactively handle this fault, the "Action Selector" selects Action #<NUM> as the Candidate Action, based on the similarity between the predicted fault and the fault to which Action#<NUM> was previously applied. The "Action Selector" passes Action #<NUM> to the "Action Reasoner" for further evaluation. However, given the fact that there is only <NUM> residual capacity in HD <NUM> on host Slave1, applying Action #<NUM> may cause on 'Disk Fault: full disk (Host)' on host Slave1 (assuming that disk utilization on any host should be lower than <NUM>%, predefined by the admin). The "Action Reasoner" produces a composite action that combines Action #<NUM> and Action #<NUM> as the Action Adjustments after applying the composite actions on the formal cloud model generated by and output from the "Cloud Model Analyzer". Specifically, the size of HD1 is to be expanded by applying Action #<NUM> (new hard disk attached), then the size of VM Disk (VMD)<NUM> is expanded by <NUM>% (40Gto <NUM>) by applying Action #<NUM>. The composite action is then stored in the Action Pool as a new action.

The system presented herein can be implemented and deployed within any distributed or centralized infrastructure cloud system. In addition, it can be implemented in one module or it can be distributed in different modules that are connected.

<FIG> illustrates a method <NUM> for automatically managing an event in a cloud system. The method comprises determining, step <NUM>, a candidate action to be applied to the cloud system for managing the event. The method also comprises applying, step <NUM>, the candidate action to a model of the cloud system. The method also comprises, upon determining that the model of the cloud system meets at least one performance indicator and that the candidate action is a proved action, applying, step <NUM>, the proved action to the cloud system.

The event is predicted. The event may be predicted by feeding online data collected from the cloud system to a model trained by machine learning using data samples of previous events and getting an output from the model predicting the event.

Determining the candidate action may comprise identifying, step <NUM>, at least one deviation caused to the cloud system by the event by comparing online data collected for the event from the cloud system with data samples of previous events. Determining the candidate action may comprise searching, step <NUM>, previously defined actions executed in response to at least one similar deviation, in an action pool, wherein similar deviations are deviations that obtain a same result when compared with a given threshold. Determining the candidate action may comprise sorting, step <NUM>, the previously defined actions according to an amount of each of the at least one deviation. Determining the candidate action may comprise selecting, step <NUM>, one of the previously defined action as the candidate action according to the sorting. The determining (or identification) of candidate actions can be done by comparing with thresholds such as <NUM>%, <NUM>%, or <NUM>%, for example. In this example, deviations above <NUM>%, <NUM>%, or <NUM>% would be deemed similar, respectively.

Selecting one of the previously defined action as the candidate action may further comprise comparing, step <NUM>, the at least one deviation of the sorted previously defined actions with at least one corresponding predetermined threshold and selecting, step <NUM>, the previously defined action with a highest ranking determined based on the comparing with the at least one corresponding predetermined threshold.

Determining the candidate action may comprise identifying, step <NUM>, at least one deviation caused to the cloud system by the event by comparing online data collected for the event from the cloud system with data samples of previous events. Determining the candidate action may comprise upon determining that no previously defined actions have been executed in response to at least one similar deviation, wherein similar deviations are deviations that obtain a same result when compared with a given threshold, by searching an action pool containing stored candidate actions, creating, step <NUM>, a new candidate action to be used as the candidate action to apply to the cloud system.

Creating a new candidate action may comprise retrieving, step <NUM>, from the action pool, a plurality of reference candidate actions, each having at least one similar deviation, wherein similar deviations are deviations that obtain a same result when compared with a given threshold. Creating a new candidate action may comprise identifying, step <NUM>, a plurality of inference rules composing the plurality reference candidate actions. Creating a new candidate action may comprise identifying, step <NUM>, a list of inference rules from the plurality of reference rules that are non-conflicting and using, step <NUM>, a constraints satisfaction problem solver for selecting a combination of inference rules, from the list of inference rules, to compose the new candidate action. The reference candidate actions can be selected e.g. according to a ranking that is done based on the comparison with thresholds such as <NUM>%, <NUM>%, or <NUM>%. In this example, deviations below <NUM>%, <NUM>%, or <NUM>% would be deemed similar, respectively.

The model of the cloud system may be a formal model of the cloud system and applying the candidate action to the model of the cloud system may comprise applying the candidate action to the formal model of the cloud system, the formal model describing logical connections between blocks of the cloud system.

The at least one performance indicator may comprise key performance indicators (KPIs) that reflect characteristics of the cloud system when functioning in a normal state. The KPIs may be monitored through metrics that are used track deviations in the cloud system. The metrics may comprise at least one of central processing unit (CPU) load, storage usage, memory usage, Input/Output usage, temperature, node used capacity.

Applying the proved action to the cloud system may comprise getting feedback, step <NUM>, from the cloud system to determine if the event was properly handled.

The method may further comprise, upon determining that the event was properly handled, updating, step <NUM> an action pool of candidate actions with the proved action and updating a formal model of the cloud system which models the cloud system, to reflect a result of applying, the proved action to the cloud system. The event may be a fault, a change in a performance indicator or a security alarm.

It should be understood that the term "event" as used in relation with <FIG> may refer to a fault related event, a performance change related event, or a security related event, although the previous figures only exemplified fault management.

Referring to <FIG>, there is provided a virtualization environment in which functions and steps described herein can be implemented.

A virtualization environment (which may go beyond what is illustrated in <FIG>), may comprise systems, networks, servers, nodes, devices, etc., that are in communication with each other either through wire or wirelessly. Some or all of the functions and steps described herein may be implemented as one or more virtual components (e.g., via one or more applications, components, functions, virtual machines or containers, etc.) executing on one or more physical apparatus in one or more networks, systems, environment, etc..

A virtualization environment provides hardware comprising processing circuitry <NUM> and memory <NUM>. The memory can contain instructions executable by the processing circuitry whereby functions and steps described herein may be executed to provide any of the relevant features and benefits disclosed herein.

Implementation of the techniques described herein can be made in a system such as the one illustrated in <FIG>. The system for automatically managing an event in a cloud system comprises processing circuits <NUM> and a memory <NUM>. The memory contains instructions executable by the processing circuits whereby the system is operative to determine a candidate action to be applied to the cloud system for managing the event. The system is also operative to apply the candidate action to a model of the cloud system. The system is also operative to, upon determining that the model of the cloud system meets at least one performance indicator and that the candidate action is a proved action, apply the proved action to the cloud system.

The event may be detected or predicted. The event may be predicted by feeding online data collected from the cloud system to a model trained by machine learning using data samples of previous events and getting an output from the model predicting the event.

The system may be further operative to identify at least one deviation caused to the cloud system by the event by comparing online data collected for the event from the cloud system with data samples of previous events. The system may be further operative to search previously defined actions executed in response to at least one similar deviation, in an action pool, wherein similar deviations are deviations that obtain a same result when compared with a given threshold. The system may be further operative to sort the previously defined actions according to an amount of each of the at least one deviation. The system may be further operative to select one of the previously defined action as the candidate action according to the sorting.

The system may be further operative to compare the at least one deviation of the sorted previously defined actions with at least one corresponding predetermined threshold and select the previously defined action with a highest ranking determined based on the comparing with the at least one corresponding predetermined threshold.

The system may be further operative to identify at least one deviation caused to the cloud system by the event by comparing online data collected for the event from the cloud system with data samples of previous events. The system may be further operative to, upon determining that no previously defined actions have been executed in response to at least one similar deviation, wherein similar deviations are deviations that obtain a same result when compared with a given threshold, by searching an action pool containing stored candidate actions, create a new candidate action to be used as the candidate action to apply to the cloud system.

The system may be further operative to retrieve, from the action pool, a plurality of reference candidate actions, each having at least one similar deviation, wherein similar deviations are deviations that obtain a same result when compared with a given threshold. The system may be further operative to identify a plurality of inference rules composing the plurality reference candidate actions. The system may be further operative to identify a list of inference rules from the plurality of reference rules that are non-conflicting. The system may be further operative to use a constraints satisfaction problem solver for selecting a combination of inference rules, from the list of inference rules, to compose the new candidate action.

Applying the proved action to the cloud system may comprise getting feedback from the cloud system to determine if the event was properly handled.

The system may be further operative to update an action pool of candidate actions with the proved action and update a formal model of the cloud system which models the cloud system, to reflect a result of applying the proved action to the cloud system.

The event may be a fault, a change in a performance indicator or a security alarm.

The hardware may also include non-transitory, persistent, machine readable storage media <NUM> having stored therein software and/or instruction <NUM> executable by processing circuitry to execute functions and steps described herein.

Claim 1:
A method for automatically managing an event in a cloud system, comprising:
- determining (<NUM>) a candidate action to be applied to the cloud system for managing the event;
- applying (<NUM>) the candidate action to a model of the cloud system; and
- upon determining, when the model of the cloud system meets at least one performance indicator, that the candidate action is a proved action, applying (<NUM>) the proved action to the cloud system,
wherein the event is predicted.