# A Two-Layer Architecture for Failure Prediction Based on High-Dimension Monitoring Sequences

Authors: Xue Wang; Fan Liu; Yixin Feng; Jiabao Zhao
Journal: Complexity
 (2021)

---

## Abstract

In recent years, the distributed architecture has been widely adopted by security companies with the rapid expansion of their business. A distributed system is comprised of many computing nodes of different components which are connected by high-speed communication networks. With the increasing functionality and complexity of the systems, failures of nodes are inevitable which may result in considerable loss. In order to identify anomalies of the possible failures and enable DevOps engineers to operate in advance, this paper proposes a two-layer prediction architecture based on the monitoring sequences of nodes status. Generally speaking, in the first layer, we make use of EXPoSE anomaly detection technique to derive anomaly scores in constant time which are then used as input data for ensemble learning in the second layer. Experiments are conducted on the data provided by one of the largest security companies, and the results demonstrate the predictability of the proposed approach.


---

## Body

## 1. Introduction

Due to the complexity of financial services, the financial industry demands applicable, reliable, and robust business systems provided by security companies. For this purpose, the distributed architecture, with its strength in dealing with large-scale data and high elasticity, has been widely applied [1]. Generally, a distributed system is comprised of computing nodes of different components. Each node is arranged to perform a specific computation task and is connected to others through high-speed communication network.Along with the rapid growth of financial market, the functionality and complexity of the distributed business systems are increasing. As a result, the failures of nodes are inevitable which may lead to system crash. Loss from system crash could be very large. For example, Amazon lost nearly 2 million dollars in about 30 minutes of service downtime caused by network problem. As is reported by Silva and Alonso [2], the cost during downtime can be as high as 6 million dollars per hour for online brokerage services. Therefore, the task of predicting potential failures of nodes is of great importance to guarantee the sustainability of systems.Motivated by this, some companies construct elaborate monitoring infrastructures to collect real-time attributes of each node, which reflect systems’ health in time. The attributes of interest include the load of CPUs, the usage of memory, and the storage, forming high-dimensional data. At the same time, cloud computing techniques along with machine learning methods can help analyse the monitoring data [3]. Specifically, detecting anomalies assists companies to predict failures and prepare for system recovery. Anomaly detection refers to the problem of identifying anomalous patterns in data, and the deployment of detectors to KPIs (key performance indicators) metrics of a distributed system has been widely adopted by many Internet companies [4–6]. For interesting researchers, we recommend review papers by Chandola et al., [7], Agrawal [8], and Xu et al. [9].Our paper proposes a two-layer potential failure prediction framework. Specifically, the first layer generates anomaly scores based on high-dimensional monitoring data of a given component, by using a real-time unsupervised anomaly detector. A higher score indicates a higher anomalous probability of the current state of the sequence. Based on anomaly scores, the second layer employs random forest, one of the successful ensemble classification methods, to predict whether nodes will fail within a given time interval.The main contributions of this study are as follows. First, it describes a real-time, efficient, and applicable method to identify whether the components of distributed systems would fail or not in the future based on the self-monitoring data. The online method is essential for industrial big data scenario, such as the Internet of Things (IoT) in the future [10]. We apply the methodology on the data provided by one of the largest security companies in China. According to a test on 6-day data, the precision is over 0.88, and the recall could be as high as 0.61 with proper forecasting interval time. The results demonstrate the predictability of our method. In that, this paper enriches the study on online failure prediction for distributed systems. Second, we take advantage of ensemble learning to leverage the performance of anomaly detector and give some IT operation suggestions from our explainable model for DevOps engineers. Compared with results using only one detector in the first layer, ensemble learning provides higher prediction accuracy and more knowledge discovery.The reminder of paper is organized as follows. Section2 gives a brief review on anomaly detection and fault prediction. A detailed description of the proposed prediction framework is presented in Section 3, followed by a case study with discussion in section 4. Finally, Section 5 includes several general conclusions and directions for future work.


## 2. Literature Review

Many previous works focus on failure prediction and diagnose based on text data such as logs, reported issues, and warnings [11–14]. However, due to the irregularities of text data, these methods build complex models for better performance.Nowadays, more researchers directly analyse temporal monitoring data. Nosayba El-Sayed et al. identified relative factors of unsuccessful jobs from log and then designed a random forests framework learning form configuration message and monitoring metrics to predict the unsuccessful termination of a job [15]. He et al. proposed Log3C [11], a novel cascading clustering algorithm for log sequences and then identified the impactful problems by correlating the clusters of log sequences with system KPIs. Lin et al. introduced a failure prediction technique MING [16], which combines a LSTM model incorporating the temporal data with a random forest model consuming spatial data. Haowen Xu et al. proposed Donut [17], an unsupervised anomaly detection algorithm based on VAE for Seasonal KPIs. Zhao et al. proposed eWarn [18], which extracts a set of features (including textual features and statistical features) to train classification models (XGBoost) with alert. However, there are tens of TBs of logs and thousands of metrics per day for a large-scale distributed system in reality, meaning it is a great challenge to select useful features, train, and update a prediction model [19]. Besides, some black-box ML and DL models, like LSTM and MING-based models, is hard to explain the results. The lack of decision interpretability is not convenient for maintenance work of DevOps engineers [20]. Therefore this paper calculate the monitoring time series anomaly score in real-time, which is automatic feature engineering to build a random forest-based prediction model in the second layer.There is extensive work on anomaly detection methods that can be mainly categorized into classification-based, statistics-based like distanced-based [21–24], density-based [25], nonlinear model based [26], and so on. But, most of these approaches are required the full dataset and not unsuitable for real-time streaming sequences. For sequence data, the length increases as time goes. Some methods are to make one-step ahead forecast and defines the observations with relative large prediction errors as anomalies. The prediction methods that have been adopted include auto-regressive integrated moving average (ARIMA) [27], exponential smoothing [28], long short-term memory (LSTM)[29], and so on. The s-based methods like extreme studentized deviate, k-sigma, median absolute deviation [30], and Bayesian online detection [31] are also applied for sequence data with proper selection of time window. Our anomaly detection method is also this genre. But, what distinguishes is that we chose a kernel-based method, which make no assumption about the data distribution and score so fast only in constant time.


## 3. Prediction Framework

LetXit=xi1,xi2,…,xit,i=1,2,…,p,t=1,2,… denote the monitoring data for ith attribute of one certain node until time t. Then, the monitoring data for p attributes of one node forms multidimensional sequences, denoted by Xt and(1)Xt=x11x12⋯x1tx21x22⋯x2t⋮⋮⋱⋮xp1xp2⋯xpt.In this section, we construct a two-layer prediction framework which transformsXt into the classification of whether the node will fail within time interval t,t+H, where H is the prespecified length of time, as is shown in Figure 1. The tow-layer architecture consists of one anomaly detection layer and one classification layer.Figure 1
Two-layer prediction framework.The anomaly detection layer contains a real-time anomaly detector EXPoSE [32]. For each monitoring sequence, the detector generates anomaly score vector Sit=si1,si2,…,sit,i=1,2,…,p, where sit quantifies the anomalous level of the ith sequence at time t. The higher the anomaly scores the higher the probability that the sequences are anomalous. These scores are then used as input for the second layer, where a classification model is built. The details of both layers are presented below.


### 3.1. Anomaly Detection Layer

In this paper, we choose a real-time anomaly detector for sequence, EXPected Similarity Estimation (EXPoSE). They are chosen because (1) the real-time scoring is fast only in constant time and in a fixed amount of memory; (2) EXPoSE, the kernel-based detector, makes no prior assumption about the data distribution, which could score different types sequences. (3) The anomaly score quantifying the anomalous level is the base of interpretability in the second layer.The EXPected Similarity Estimation (EXPoSE) computes the similarity between new data and the distribution of historical normal data [32]. Anomaly scores are represented by the opposite side of similarity. Specifically, EXPoSE uses the inner product of feature map ϕxt and kernel mean map μℙt to represent the normal likelihood, ηxt, of the current observation (the likelihood of xt is normal). We have(2)ηxt=ϕxt⋅μℙt,where xt is a random variable whose values is in a measurable space X. Let ℙ be a Borel probability measure on X, which represents the distribution of normal data. μℙt is the kernel embedding of X, defined as(3)μℙt=∫Xkxt,⋅dℙxt.The kernel mean mapμℙt is approximated by empirical embedding of samples from ℙ:(4)μ^ℙt=1t∑i=1tϕxt.As for the feature mapϕxt, it is approximated based on random kitchen sinks (RBS) [33] with Gaussian Radial Basis Function (RBF) kernel. We first draw N samples G=G1,G2,…,GNT from univariate Gaussian distribution, then approximate ϕxt by(5)ϕ^xt=1NexpiGxt.At last, the anomaly score, denoted bys4t, could be derived by(6)s4t=1−ϕ^xt⋅μ^ℙt.Furthermore, sliding windowing and decay mechanism help to adapt the concept drift [34]. The sliding windowing is requiring keep the past part of observations in memory (window size l) and updating incrementally:(7)wt=1l∑i=t−l+1tϕxi.Due to time series dependence, the decay mechanism pays more attention on recent data:(8)wt=γϕxt+1−γwt−1,fort>1.With the above anomaly detector, for one node withp attributes, there are correspondingly p scores at time t. Therefore, we adopt ensemble learning approach in the second layer.


### 3.2. Classification Layer

Random forest is a strong ensemble learning method to uncover the law between features and labels. It consists of a series of decision trees which greedily and recursively chooses an available feature to split the data into purer subsets measured by the Gini index.


#### 3.2.1. Decision Tree

If the dataD contain K classes (in our case, K=2), and for each class, there is a proportion of pk in the data, then the Gini index for data D is(9)EntD=−∑k=1Kpklog2pk.IfD is split into D1,D2,…,Da subsets according to feature a, and each subset Da is with size Da, then the information gain of D after this split is(10)GainD,a=EntD−∑v=1VDvDEntDv.The bigger theGainD,a, the best it splits the data. Suppose A is the set of features, then the best feature a∗ is chosen by(11)a∗=argmaxa∈AGainD,a.As the features are all continuous, we build binary trees. That is, for each subset of the original data, we split it into two parts as long as there is any available feature and the increase of information gain is significant.


#### 3.2.2. Random Forest

Decision tree is easy-to-implement but may not perform well in classification, especially when the dimension of features is high. Random forest classifies observations based on the majority votes across all the trees, which is proved to be more precise and robust.As is illustrated in Figure1, each decision tree is built on a subset of features which are randomly selected from the p features and a dataset sampled from the original data by bootstrap methods. Then, the final failure prediction is made by voting by all decision trees.


## 3.1. Anomaly Detection Layer

In this paper, we choose a real-time anomaly detector for sequence, EXPected Similarity Estimation (EXPoSE). They are chosen because (1) the real-time scoring is fast only in constant time and in a fixed amount of memory; (2) EXPoSE, the kernel-based detector, makes no prior assumption about the data distribution, which could score different types sequences. (3) The anomaly score quantifying the anomalous level is the base of interpretability in the second layer.The EXPected Similarity Estimation (EXPoSE) computes the similarity between new data and the distribution of historical normal data [32]. Anomaly scores are represented by the opposite side of similarity. Specifically, EXPoSE uses the inner product of feature map ϕxt and kernel mean map μℙt to represent the normal likelihood, ηxt, of the current observation (the likelihood of xt is normal). We have(2)ηxt=ϕxt⋅μℙt,where xt is a random variable whose values is in a measurable space X. Let ℙ be a Borel probability measure on X, which represents the distribution of normal data. μℙt is the kernel embedding of X, defined as(3)μℙt=∫Xkxt,⋅dℙxt.The kernel mean mapμℙt is approximated by empirical embedding of samples from ℙ:(4)μ^ℙt=1t∑i=1tϕxt.As for the feature mapϕxt, it is approximated based on random kitchen sinks (RBS) [33] with Gaussian Radial Basis Function (RBF) kernel. We first draw N samples G=G1,G2,…,GNT from univariate Gaussian distribution, then approximate ϕxt by(5)ϕ^xt=1NexpiGxt.At last, the anomaly score, denoted bys4t, could be derived by(6)s4t=1−ϕ^xt⋅μ^ℙt.Furthermore, sliding windowing and decay mechanism help to adapt the concept drift [34]. The sliding windowing is requiring keep the past part of observations in memory (window size l) and updating incrementally:(7)wt=1l∑i=t−l+1tϕxi.Due to time series dependence, the decay mechanism pays more attention on recent data:(8)wt=γϕxt+1−γwt−1,fort>1.With the above anomaly detector, for one node withp attributes, there are correspondingly p scores at time t. Therefore, we adopt ensemble learning approach in the second layer.


## 3.2. Classification Layer

Random forest is a strong ensemble learning method to uncover the law between features and labels. It consists of a series of decision trees which greedily and recursively chooses an available feature to split the data into purer subsets measured by the Gini index.


### 3.2.1. Decision Tree

If the dataD contain K classes (in our case, K=2), and for each class, there is a proportion of pk in the data, then the Gini index for data D is(9)EntD=−∑k=1Kpklog2pk.IfD is split into D1,D2,…,Da subsets according to feature a, and each subset Da is with size Da, then the information gain of D after this split is(10)GainD,a=EntD−∑v=1VDvDEntDv.The bigger theGainD,a, the best it splits the data. Suppose A is the set of features, then the best feature a∗ is chosen by(11)a∗=argmaxa∈AGainD,a.As the features are all continuous, we build binary trees. That is, for each subset of the original data, we split it into two parts as long as there is any available feature and the increase of information gain is significant.


### 3.2.2. Random Forest

Decision tree is easy-to-implement but may not perform well in classification, especially when the dimension of features is high. Random forest classifies observations based on the majority votes across all the trees, which is proved to be more precise and robust.As is illustrated in Figure1, each decision tree is built on a subset of features which are randomly selected from the p features and a dataset sampled from the original data by bootstrap methods. Then, the final failure prediction is made by voting by all decision trees.


## 3.2.1. Decision Tree

If the dataD contain K classes (in our case, K=2), and for each class, there is a proportion of pk in the data, then the Gini index for data D is(9)EntD=−∑k=1Kpklog2pk.IfD is split into D1,D2,…,Da subsets according to feature a, and each subset Da is with size Da, then the information gain of D after this split is(10)GainD,a=EntD−∑v=1VDvDEntDv.The bigger theGainD,a, the best it splits the data. Suppose A is the set of features, then the best feature a∗ is chosen by(11)a∗=argmaxa∈AGainD,a.As the features are all continuous, we build binary trees. That is, for each subset of the original data, we split it into two parts as long as there is any available feature and the increase of information gain is significant.


## 3.2.2. Random Forest

Decision tree is easy-to-implement but may not perform well in classification, especially when the dimension of features is high. Random forest classifies observations based on the majority votes across all the trees, which is proved to be more precise and robust.As is illustrated in Figure1, each decision tree is built on a subset of features which are randomly selected from the p features and a dataset sampled from the original data by bootstrap methods. Then, the final failure prediction is made by voting by all decision trees.


## 4. Case Study

### 4.1. Data Description

We collect data from one of the largest security companies in China using its self monitoring system. The system consists of Logstach which parses all kinds of logs, Kafka which transmits high volumes of data to Jstorm with low latency, Jstorm which offers on-the-fly data aggregation and calculation, and Elasticsearch which stores, searches, and analyzes large amounts of data. This monitoring system generates alerts when the system breaks some rules set by the DevOps engineers or softwares. These alerts contain information about potential failures [35, 36]. Specifically, we list all kinds of alerts and relative potential failures in self monitoring system (Table 1). For example, when thread fails, the monitoring system will label this time with one failure event and record it in the logs. With such a system, one can track the status of the nodes as they evolve with time.Table 1
Total types of alerts.
Alerts typeNumbers of alertsComponentPotential failureISR has increased in last 15 m37KafkaIO parallelism is highISR has decreased in last 15 m17KafkaData replica is limited, which could cause data inconsistencyLeader election has occurred in last 15 m8KafkaNew kind of data or maybe kafka leader failsKafka health check failed17KafkaKafka healthy problemIP port detection abnormal8AllNewwork problemEs cluster health unassigned shards¿018ElasticsearchSlave shards that are not assigned, indicating that the health of the cluster has deterioratedEs cluster health number of pending tasks¿1009ElasticsearchTask stacking problemDisk IO response time is long26ElasticsearchSystem latency is largeFinally, the data used contain 30 nodes from Elasticsearch component which save the company’s centralized trading system data, each is observed on five attributes, namely, available storage of the node (ASN), usage of Java Virtual Machine (JVM) memory of heap (UJM), recent historical CPU usage for the whole system (HLC), one-minute current load average of CPU (CLU), and disk IO time (DIO). The attributes are recorded every minute from November 8, 2018, to November 22, 2018. Since some sequences basically do not change, there are finally 139 time series for modeling. Then, there are more than 20,000 observations for each sequence, of which less than 0.5% are labeled with alerts by the self monitoring system (Table2).Table 2
Total number of failures.
NodeNumber of failuresFailure ratio (%)1170.0852170.0853130.0654130.065580.04660.03740.02840.02940.021020.011120.01Total900.45


### 4.2. Data Preprocessing

We group and average the monitoring sequences every half hour since the data granularity of per minute is too small with more noisy. Note that at any timet, our aim is to predict the potential failures of each node with t,t+H, where H is the time interval defined in section 3. Therefore, we label the observation as 1, if there is any failure within t,t+H, or 0, otherwise. After labeling, there are 13% of the data with label 1, which does not require other complicated oversampling techniques. [16].We use the first 10-day data as training sample based on which random forest is trained. The remaining 4-day data are used for testing.The procedures for classifyingxt for one node are as follows. For the above 5th (i=1,2,3,4,5) attribute, we generate 5 anomaly scores si1t,…,si5t according to xi1,…,xit. Then, the 5 scores are used as input features in the trained random forest and generate the label of xt.


### 4.3. Results and Discussion

#### 4.3.1. First Layer

In the first layer, we apply EXPoSE with window size 2 days and decay rate 0.01 on each of the five attributes of each node. Because there are over 750 scores for each series, we only illustrate the anomaly scores for training, from November 10, 2018, to November 17, 2018. Since our sliding window size is two days, the first two days are not shown in the next figures.In the two figures, theX-axis denotes the calendar time and the Y-axis denotes the scores. The shape of plus denotes that there is a failure alert, and one can find the actual time via the vertical dashed line. The failure alert in Figure 2 is from the same node of five attributes. Additionally, Figure 3 shows all alerts of Elasticsearch component with five attributes. Through figures, we find some interesting knowledge.(1)
The anomaly score of UJM sequence is high before the failure alert, which indicates the anomaly detector is efficient(2)
Synchronization effect of sequence anomaly scores: attributes like CLU, HLC, and DIO definitely change synchronously, indicating a certain correlation between these metrics(3)
Failure alerts of other nodes will also be reflected in the metrics of the current node reference to Figure3(4)
Also some sequences are not related to alert, like ASN in Figures2 and 3. These sequences could distract DevOps engineers in realityFigure 2
Results from EXPoSE and alerts of one node.Figure 3
Results from EXPoSE and all alerts of Elasticsearch.Then, we calculate correlation matrix between the five series and their anomaly scores in Figures2 and 3 and illustrate them in Tables 3 to 4.Table 3
Correlations between the five attributes of one node.
ASNUJMCLUHLCDIOASN1.000−0.174−0.007−0.01−0.01UJM—1.0000.0650.0760.076CLU——1.0000.9870.98HLC———1.0000.980DIO————1.000Table 4
Correlations between the anomaly scores generated by EXPoSE.
ASNUJMCLUHLCDIOASN1.0000.0090.160.200.307UJM—1.000−0.11−0.074−0.18CLU——1.0000.920.903HLC———1.0000.853DIO————1.000It is found that the patterns of score series related to attribute ASN are almost independent from others, but the series related to HLC highly correlate with DIO. Such findings may indicate a potential topological relationship between the HLC and DIO.


#### 4.3.2. Second Layer

Predicting the failures is actually a classification problem (failure or nonfailure), thus we employ precision and recall to evaluate the predictability. Without loss of generality, we treat the observations labeled with failures as positive class and the remaining as negative class. Then, the random forest produces four types of outcome. Two of them refer to correct (or true) classification, including true positives (TP) and true negatives (TN), meaning the number of the failure which is correctly classified as positive or negative. True negative (TN) and false negatives (FP) mean the number of the normal observation is correctly classified as negative or positive in the next formulas. Then, the precision and recall are calculated by(12)precision=TPTP+FP,recall=TPTP+FN,F1_score=2×precision×recallprecision+recall,respectively.The final random forest contains 400 decision trees; max depth of the tree is 6. We tested 5 cases as the time intervalH increases from 30 minutes to 4 hours, and the results are shown in Table 5.Table 5
Failure prediction results with interval H.
MeasuresPrespecifiedHh122.533.54.0Recall0.400.480.540.610.630.64Precision0.900.870.890.880.800.67f1_score0.600.620.670.730.710.65We find the following:(1)
The precisions are all higher than 85% whenH is less than 3.5 hour, meaning that our model rarely makes wrong prediction. Precision perform best (88%) when H equal 3, meaning our method give accurately three hours’ early warning. It is significant for DevOps engineers to focus on the true failure in advance.(2)
The recalls are fairly acceptable. Since our labeling contains more probable failure observations as long as time is withint,t+H.(3)
AS the time interval increases, both measures become higher which means the model performs better. However, when the time interval is too large, it is quite difficult to make preparation for failure recovery since the actual failure time is hard to be located.(4)
Since the random forest-based model is interpretable, we could derive the feature importance for DevOps engineers to pay more attention on crucial sequences and better improve the monitoring system (Table6). The feature importance is Gini importance; the average of impurity decreases over all trees in the forest, which reflect how each feature plays an important role in the split.Table 6
Top five important features.
NodesFeature nameFeature importance (%)1CLU2.272ASN2.243ASN2.184CLU2.185UJM1.91


#### 4.3.3. Comparisons

Although the proposed method performs well, two questions still remain:(1)
Whether or not should we apply anomaly detection before the classification method is applied?(2)
Whether or not other anomaly method performs better?To answer the first question, we remove the anomaly detection layer and directly use the attributes as input for developing a random forest. Besides, we compare with other classical machine learning classification methods (support vector classification (SVM)). The results are shown in the first three rows of Table7.Table 7
Anomaly detector comparation.
ModelPrecisionRecallf1_scoreExpose RF0.880.610.73RF0.200.010.11SVM0.010.040.02LSTM0.600.910.65Numenta0.640.390.48KNNCAD0.950.040.08Obviously, the results in both precision and recall are much worse than the two-layer prediction framework. There may be two reasons. First, the original data may contain too much noise which harms the prediction accuracy. Second, the anomaly detection technique uses not only the current value of the sequences but also the previous values, so anomaly scores to some extent summarize the overall performance of the attributes and mine key point of change. This experiment demonstrates that ensemble learning is necessary to improve the forecasting accuracy.On the other hand, we compare with the LSTM anomaly detector [37]. We train LSTM model from normal data of one week and then predict xt′ based on the previous three hours to derive anomaly score pt:(13)pt=xt′−xtxt.Besides, we compare with another real-time detector, Numenta [38] and KNNCAD [39], with the same window parameters. Finally, random forest learns all 138 anomaly score and alerts label (H equals 3 hour). The results are showed in the last two rows of Table 7.We find that LSTM performs fairly good from Table7, but the precision is still too low. Though recall is high, it is not important in reality for DevOps engineers due to H labeling. However, for KNNCAD, the recall is too low to apply. Furthermore, computation efficiency of expose is 70 times faster than LSTM. Expose does not even need to train the model offline like LSTM and is better for parallel computation. From another angle, our architecture is also universal. Different anomaly detectors mean more flexibility to detect diverse anomalies in reality.


## 4.1. Data Description

We collect data from one of the largest security companies in China using its self monitoring system. The system consists of Logstach which parses all kinds of logs, Kafka which transmits high volumes of data to Jstorm with low latency, Jstorm which offers on-the-fly data aggregation and calculation, and Elasticsearch which stores, searches, and analyzes large amounts of data. This monitoring system generates alerts when the system breaks some rules set by the DevOps engineers or softwares. These alerts contain information about potential failures [35, 36]. Specifically, we list all kinds of alerts and relative potential failures in self monitoring system (Table 1). For example, when thread fails, the monitoring system will label this time with one failure event and record it in the logs. With such a system, one can track the status of the nodes as they evolve with time.Table 1
Total types of alerts.
Alerts typeNumbers of alertsComponentPotential failureISR has increased in last 15 m37KafkaIO parallelism is highISR has decreased in last 15 m17KafkaData replica is limited, which could cause data inconsistencyLeader election has occurred in last 15 m8KafkaNew kind of data or maybe kafka leader failsKafka health check failed17KafkaKafka healthy problemIP port detection abnormal8AllNewwork problemEs cluster health unassigned shards¿018ElasticsearchSlave shards that are not assigned, indicating that the health of the cluster has deterioratedEs cluster health number of pending tasks¿1009ElasticsearchTask stacking problemDisk IO response time is long26ElasticsearchSystem latency is largeFinally, the data used contain 30 nodes from Elasticsearch component which save the company’s centralized trading system data, each is observed on five attributes, namely, available storage of the node (ASN), usage of Java Virtual Machine (JVM) memory of heap (UJM), recent historical CPU usage for the whole system (HLC), one-minute current load average of CPU (CLU), and disk IO time (DIO). The attributes are recorded every minute from November 8, 2018, to November 22, 2018. Since some sequences basically do not change, there are finally 139 time series for modeling. Then, there are more than 20,000 observations for each sequence, of which less than 0.5% are labeled with alerts by the self monitoring system (Table2).Table 2
Total number of failures.
NodeNumber of failuresFailure ratio (%)1170.0852170.0853130.0654130.065580.04660.03740.02840.02940.021020.011120.01Total900.45


## 4.2. Data Preprocessing

We group and average the monitoring sequences every half hour since the data granularity of per minute is too small with more noisy. Note that at any timet, our aim is to predict the potential failures of each node with t,t+H, where H is the time interval defined in section 3. Therefore, we label the observation as 1, if there is any failure within t,t+H, or 0, otherwise. After labeling, there are 13% of the data with label 1, which does not require other complicated oversampling techniques. [16].We use the first 10-day data as training sample based on which random forest is trained. The remaining 4-day data are used for testing.The procedures for classifyingxt for one node are as follows. For the above 5th (i=1,2,3,4,5) attribute, we generate 5 anomaly scores si1t,…,si5t according to xi1,…,xit. Then, the 5 scores are used as input features in the trained random forest and generate the label of xt.


## 4.3. Results and Discussion

### 4.3.1. First Layer

In the first layer, we apply EXPoSE with window size 2 days and decay rate 0.01 on each of the five attributes of each node. Because there are over 750 scores for each series, we only illustrate the anomaly scores for training, from November 10, 2018, to November 17, 2018. Since our sliding window size is two days, the first two days are not shown in the next figures.In the two figures, theX-axis denotes the calendar time and the Y-axis denotes the scores. The shape of plus denotes that there is a failure alert, and one can find the actual time via the vertical dashed line. The failure alert in Figure 2 is from the same node of five attributes. Additionally, Figure 3 shows all alerts of Elasticsearch component with five attributes. Through figures, we find some interesting knowledge.(1)
The anomaly score of UJM sequence is high before the failure alert, which indicates the anomaly detector is efficient(2)
Synchronization effect of sequence anomaly scores: attributes like CLU, HLC, and DIO definitely change synchronously, indicating a certain correlation between these metrics(3)
Failure alerts of other nodes will also be reflected in the metrics of the current node reference to Figure3(4)
Also some sequences are not related to alert, like ASN in Figures2 and 3. These sequences could distract DevOps engineers in realityFigure 2
Results from EXPoSE and alerts of one node.Figure 3
Results from EXPoSE and all alerts of Elasticsearch.Then, we calculate correlation matrix between the five series and their anomaly scores in Figures2 and 3 and illustrate them in Tables 3 to 4.Table 3
Correlations between the five attributes of one node.
ASNUJMCLUHLCDIOASN1.000−0.174−0.007−0.01−0.01UJM—1.0000.0650.0760.076CLU——1.0000.9870.98HLC———1.0000.980DIO————1.000Table 4
Correlations between the anomaly scores generated by EXPoSE.
ASNUJMCLUHLCDIOASN1.0000.0090.160.200.307UJM—1.000−0.11−0.074−0.18CLU——1.0000.920.903HLC———1.0000.853DIO————1.000It is found that the patterns of score series related to attribute ASN are almost independent from others, but the series related to HLC highly correlate with DIO. Such findings may indicate a potential topological relationship between the HLC and DIO.


### 4.3.2. Second Layer

Predicting the failures is actually a classification problem (failure or nonfailure), thus we employ precision and recall to evaluate the predictability. Without loss of generality, we treat the observations labeled with failures as positive class and the remaining as negative class. Then, the random forest produces four types of outcome. Two of them refer to correct (or true) classification, including true positives (TP) and true negatives (TN), meaning the number of the failure which is correctly classified as positive or negative. True negative (TN) and false negatives (FP) mean the number of the normal observation is correctly classified as negative or positive in the next formulas. Then, the precision and recall are calculated by(12)precision=TPTP+FP,recall=TPTP+FN,F1_score=2×precision×recallprecision+recall,respectively.The final random forest contains 400 decision trees; max depth of the tree is 6. We tested 5 cases as the time intervalH increases from 30 minutes to 4 hours, and the results are shown in Table 5.Table 5
Failure prediction results with interval H.
MeasuresPrespecifiedHh122.533.54.0Recall0.400.480.540.610.630.64Precision0.900.870.890.880.800.67f1_score0.600.620.670.730.710.65We find the following:(1)
The precisions are all higher than 85% whenH is less than 3.5 hour, meaning that our model rarely makes wrong prediction. Precision perform best (88%) when H equal 3, meaning our method give accurately three hours’ early warning. It is significant for DevOps engineers to focus on the true failure in advance.(2)
The recalls are fairly acceptable. Since our labeling contains more probable failure observations as long as time is withint,t+H.(3)
AS the time interval increases, both measures become higher which means the model performs better. However, when the time interval is too large, it is quite difficult to make preparation for failure recovery since the actual failure time is hard to be located.(4)
Since the random forest-based model is interpretable, we could derive the feature importance for DevOps engineers to pay more attention on crucial sequences and better improve the monitoring system (Table6). The feature importance is Gini importance; the average of impurity decreases over all trees in the forest, which reflect how each feature plays an important role in the split.Table 6
Top five important features.
NodesFeature nameFeature importance (%)1CLU2.272ASN2.243ASN2.184CLU2.185UJM1.91


### 4.3.3. Comparisons

Although the proposed method performs well, two questions still remain:(1)
Whether or not should we apply anomaly detection before the classification method is applied?(2)
Whether or not other anomaly method performs better?To answer the first question, we remove the anomaly detection layer and directly use the attributes as input for developing a random forest. Besides, we compare with other classical machine learning classification methods (support vector classification (SVM)). The results are shown in the first three rows of Table7.Table 7
Anomaly detector comparation.
ModelPrecisionRecallf1_scoreExpose RF0.880.610.73RF0.200.010.11SVM0.010.040.02LSTM0.600.910.65Numenta0.640.390.48KNNCAD0.950.040.08Obviously, the results in both precision and recall are much worse than the two-layer prediction framework. There may be two reasons. First, the original data may contain too much noise which harms the prediction accuracy. Second, the anomaly detection technique uses not only the current value of the sequences but also the previous values, so anomaly scores to some extent summarize the overall performance of the attributes and mine key point of change. This experiment demonstrates that ensemble learning is necessary to improve the forecasting accuracy.On the other hand, we compare with the LSTM anomaly detector [37]. We train LSTM model from normal data of one week and then predict xt′ based on the previous three hours to derive anomaly score pt:(13)pt=xt′−xtxt.Besides, we compare with another real-time detector, Numenta [38] and KNNCAD [39], with the same window parameters. Finally, random forest learns all 138 anomaly score and alerts label (H equals 3 hour). The results are showed in the last two rows of Table 7.We find that LSTM performs fairly good from Table7, but the precision is still too low. Though recall is high, it is not important in reality for DevOps engineers due to H labeling. However, for KNNCAD, the recall is too low to apply. Furthermore, computation efficiency of expose is 70 times faster than LSTM. Expose does not even need to train the model offline like LSTM and is better for parallel computation. From another angle, our architecture is also universal. Different anomaly detectors mean more flexibility to detect diverse anomalies in reality.


## 4.3.1. First Layer

In the first layer, we apply EXPoSE with window size 2 days and decay rate 0.01 on each of the five attributes of each node. Because there are over 750 scores for each series, we only illustrate the anomaly scores for training, from November 10, 2018, to November 17, 2018. Since our sliding window size is two days, the first two days are not shown in the next figures.In the two figures, theX-axis denotes the calendar time and the Y-axis denotes the scores. The shape of plus denotes that there is a failure alert, and one can find the actual time via the vertical dashed line. The failure alert in Figure 2 is from the same node of five attributes. Additionally, Figure 3 shows all alerts of Elasticsearch component with five attributes. Through figures, we find some interesting knowledge.(1)
The anomaly score of UJM sequence is high before the failure alert, which indicates the anomaly detector is efficient(2)
Synchronization effect of sequence anomaly scores: attributes like CLU, HLC, and DIO definitely change synchronously, indicating a certain correlation between these metrics(3)
Failure alerts of other nodes will also be reflected in the metrics of the current node reference to Figure3(4)
Also some sequences are not related to alert, like ASN in Figures2 and 3. These sequences could distract DevOps engineers in realityFigure 2
Results from EXPoSE and alerts of one node.Figure 3
Results from EXPoSE and all alerts of Elasticsearch.Then, we calculate correlation matrix between the five series and their anomaly scores in Figures2 and 3 and illustrate them in Tables 3 to 4.Table 3
Correlations between the five attributes of one node.
ASNUJMCLUHLCDIOASN1.000−0.174−0.007−0.01−0.01UJM—1.0000.0650.0760.076CLU——1.0000.9870.98HLC———1.0000.980DIO————1.000Table 4
Correlations between the anomaly scores generated by EXPoSE.
ASNUJMCLUHLCDIOASN1.0000.0090.160.200.307UJM—1.000−0.11−0.074−0.18CLU——1.0000.920.903HLC———1.0000.853DIO————1.000It is found that the patterns of score series related to attribute ASN are almost independent from others, but the series related to HLC highly correlate with DIO. Such findings may indicate a potential topological relationship between the HLC and DIO.


## 4.3.2. Second Layer

Predicting the failures is actually a classification problem (failure or nonfailure), thus we employ precision and recall to evaluate the predictability. Without loss of generality, we treat the observations labeled with failures as positive class and the remaining as negative class. Then, the random forest produces four types of outcome. Two of them refer to correct (or true) classification, including true positives (TP) and true negatives (TN), meaning the number of the failure which is correctly classified as positive or negative. True negative (TN) and false negatives (FP) mean the number of the normal observation is correctly classified as negative or positive in the next formulas. Then, the precision and recall are calculated by(12)precision=TPTP+FP,recall=TPTP+FN,F1_score=2×precision×recallprecision+recall,respectively.The final random forest contains 400 decision trees; max depth of the tree is 6. We tested 5 cases as the time intervalH increases from 30 minutes to 4 hours, and the results are shown in Table 5.Table 5
Failure prediction results with interval H.
MeasuresPrespecifiedHh122.533.54.0Recall0.400.480.540.610.630.64Precision0.900.870.890.880.800.67f1_score0.600.620.670.730.710.65We find the following:(1)
The precisions are all higher than 85% whenH is less than 3.5 hour, meaning that our model rarely makes wrong prediction. Precision perform best (88%) when H equal 3, meaning our method give accurately three hours’ early warning. It is significant for DevOps engineers to focus on the true failure in advance.(2)
The recalls are fairly acceptable. Since our labeling contains more probable failure observations as long as time is withint,t+H.(3)
AS the time interval increases, both measures become higher which means the model performs better. However, when the time interval is too large, it is quite difficult to make preparation for failure recovery since the actual failure time is hard to be located.(4)
Since the random forest-based model is interpretable, we could derive the feature importance for DevOps engineers to pay more attention on crucial sequences and better improve the monitoring system (Table6). The feature importance is Gini importance; the average of impurity decreases over all trees in the forest, which reflect how each feature plays an important role in the split.Table 6
Top five important features.
NodesFeature nameFeature importance (%)1CLU2.272ASN2.243ASN2.184CLU2.185UJM1.91


## 4.3.3. Comparisons

Although the proposed method performs well, two questions still remain:(1)
Whether or not should we apply anomaly detection before the classification method is applied?(2)
Whether or not other anomaly method performs better?To answer the first question, we remove the anomaly detection layer and directly use the attributes as input for developing a random forest. Besides, we compare with other classical machine learning classification methods (support vector classification (SVM)). The results are shown in the first three rows of Table7.Table 7
Anomaly detector comparation.
ModelPrecisionRecallf1_scoreExpose RF0.880.610.73RF0.200.010.11SVM0.010.040.02LSTM0.600.910.65Numenta0.640.390.48KNNCAD0.950.040.08Obviously, the results in both precision and recall are much worse than the two-layer prediction framework. There may be two reasons. First, the original data may contain too much noise which harms the prediction accuracy. Second, the anomaly detection technique uses not only the current value of the sequences but also the previous values, so anomaly scores to some extent summarize the overall performance of the attributes and mine key point of change. This experiment demonstrates that ensemble learning is necessary to improve the forecasting accuracy.On the other hand, we compare with the LSTM anomaly detector [37]. We train LSTM model from normal data of one week and then predict xt′ based on the previous three hours to derive anomaly score pt:(13)pt=xt′−xtxt.Besides, we compare with another real-time detector, Numenta [38] and KNNCAD [39], with the same window parameters. Finally, random forest learns all 138 anomaly score and alerts label (H equals 3 hour). The results are showed in the last two rows of Table 7.We find that LSTM performs fairly good from Table7, but the precision is still too low. Though recall is high, it is not important in reality for DevOps engineers due to H labeling. However, for KNNCAD, the recall is too low to apply. Furthermore, computation efficiency of expose is 70 times faster than LSTM. Expose does not even need to train the model offline like LSTM and is better for parallel computation. From another angle, our architecture is also universal. Different anomaly detectors mean more flexibility to detect diverse anomalies in reality.


## 5. Conclusion

Predicting the failures of components is of great importance for sustaining business systems of security companies and the development of financial market. In this paper, we present a new architecture for transforming system monitoring data into the predictions of potential component failures. Our method consists of two layers: an anomaly detection layer for generating anomaly score series and a classification layer for identifying whether the node is suspicious to fail. Since there are a variety of effective anomaly detectors, random forest, an ensemble learning approach, is employed to utilize and gives play to the comparative advantages of different detectors. The case study shows that the method performs very well in terms of precision and recall relative to alternative methods.There are a number of promising directions for future research. First, our method focuses on predicting whether the component fail within a time interval but ignores the importance of lead time of prediction. For example, correctly identifying failures occurring over (6: 00, 6: 10] at time 5: 00 is better than the time 5: 30, because the former one provides more time to prepare for potential problems. Incorporating the lead time in loss function of the classification problem is a valuable next step. In a different direction, one may consider how to extend our method by considering the topological relationship between different nodes of the same component to find the root cause of failure. There may be a pass-through effect that one failure may cause another. Besides, we also expect researchers to use more data, even from other industries, to validate our method.



---

*Source: 6623666-2021-03-29.xml*