# Classification and Prediction of Software Incidents Using Machine Learning Techniques

Authors: Sikandar Ali; Muhammad Adeel; Sumaira Johar; Muhammad Zeeshan; Samad Baseer; Azeem Irshad
Journal: Security and Communication Networks
 (2021)

---

## Abstract

An incident, in the perception of information technology, is an event that is not part of a normal process and disrupts operational procedure. This research work particularly focuses on software failure incidents. In any operational environment, software failure can put the quality and performance of services at risk. Many efforts are made to overcome this incident of software failure and to restore normal service as soon as possible. The main contribution of this study is software failure incidents classification and prediction using machine learning. In this study, an active learning approach is used to selectively label those data which is considered to be more informative to build models. Firstly, the sample with the highest randomness (entropy) is selected for labeling. Secondly, to classify the labeled observation into either failure or no failure classes, a binary classifier is used that predicts the target class label as failure or not. For classification, Support Vector Machine is used as a main classifier to classify the data. We derived our prediction models from the failure log files collected from the ECLIPSE software repository.


---

## Body

## 1. Introduction

In any particular system, failure befalls when the provided service no longer obeys the specified specifications [1]. Specifications are the agreed description of the system’s functional behavior to provide expected service [1]. This definition applies to both software and hardware failures. According to Dalal and Chhillar [2], the most common software failure incidents on the web are pages not downloading properly due to sluggish response from the application server or application lack of compatibility with the browser, or it may be other performance issues such as slow load time, run time, or access time. Failures are of different types; i.e., not all the failures are fatal and some of them are even harmless and do not affect the functionality of the system. However, other failures are so fatal that they crash the whole system and make the system unavailable for specified services. But the types and levels of severity vary from software to software [2]. Faults, errors, and bugs in the software artifact are the ultimate cause of the software failure, which are the inappropriate process or step in the software artifact. Failures are the incapability of the software to perform the required action or in other words the deviation from required performance [2].A failure or even a fractional failure of one service can cause other services that depend on it to break down. This incident can create a chain of service failures that propagates until it reaches critical components and causes the software to fail. According to Gray [3] in 1986, environment issues (e.g., cooling and power) and hardware issues (e.g., memory, network, and disk) caused 32% of the incidents, which in 1999 decreased to 20% [4]. On the other hand software incidents increased from 26% to 40%. Some authors like Gray [4] even stated that 58% of the total incidents are software related. Incidents can be of different types, i.e., software incidents, hardware incidents, and technical incidents. This study focuses particularly on software incidents, which refers to the questionable behavior of the software. Software sometimes does not perform as it is expected due to many causes such as errors, bugs, and defects in it. These errors, bugs, and defects most of the time lead to software failure. In this study, we have extensively explored the software failure incidents, their causes, impacts, and the techniques proposed for their prediction. Keeping in view all these facts, this study builds a model for the prediction of software failure incidents. IT service providers are constantly seeking more efficient methods and implementations to increase the effectiveness and superiority of the process. IT Infrastructure Library (ITIL) is the widely used framework for IT services due to its best management guidelines. It provides the best guidelines on how to manage, develop, and maintain IT infrastructure. Above all, it also gives guidelines on improving the quality of the IT infrastructure. Organizations are investing heavily in operational environmental management applications. Software incidents in the operational environment are defined as unscheduled interruptions, which affect employees’ productivity and also have impacts on the cost. To decrease the unscheduled interruptions and increase the performance, many incident management techniques are introduced. Software failure incidents on the web proposed that most of the failures occur during the system upgradation or the system maintenance and may sometimes be due to the system integration. There are many causes discussed in the relevant literature of software failures; such failures in software during operation are unavoidable. This causes the unavailability of the system which results in cost and dissatisfied customers and clients. These failures need to be reduced and removed for cost-effectiveness and the satisfaction of the customers. Most of the shared and agreed causes are inadequate testing or poor testing, flaws in documentation or the poor understanding of the system complexity, resource exhaustion, complex fault recovery routines, and system overload.


### 1.1. Contribution of the Study

The main contribution of this study is software failure incidents classification and prediction using machine learning. In this study, an active learning approach is used to selectively label those data that are considered to be more informative to build models. Firstly, the sample with the highest randomness (entropy) is selected for labeling. Secondly, to classify the labeled observation into either failure or no failure classes, a binary classifier is used that predicts the target class label as failure or not. For classification, Support Vector Machine is used as the main classifier of the data. We derived our prediction models from the failure log files collected from the ECLIPSE software repository.


### 1.2. Organization of the Paper

The remaining of the paper is organized into the following sections. Section2 is based on related literature. Section 3 presents the classification and prediction method. Section 4 is results and analysis. Section 5 consists of results descriptions. Section 6 makes a discussion on the obtained results while Section 7 concludes the results and gives future directions.


## 1.1. Contribution of the Study

The main contribution of this study is software failure incidents classification and prediction using machine learning. In this study, an active learning approach is used to selectively label those data that are considered to be more informative to build models. Firstly, the sample with the highest randomness (entropy) is selected for labeling. Secondly, to classify the labeled observation into either failure or no failure classes, a binary classifier is used that predicts the target class label as failure or not. For classification, Support Vector Machine is used as the main classifier of the data. We derived our prediction models from the failure log files collected from the ECLIPSE software repository.


## 1.2. Organization of the Paper

The remaining of the paper is organized into the following sections. Section2 is based on related literature. Section 3 presents the classification and prediction method. Section 4 is results and analysis. Section 5 consists of results descriptions. Section 6 makes a discussion on the obtained results while Section 7 concludes the results and gives future directions.


## 2. Related Work

Efforts to foresee failures have been notable in recent decades. Failures, or prediction of failures, is a broad notion in software engineering that is not restricted to software failure. In both hardware and software, failure prediction techniques are widely used. These techniques are widely explored in the literature in hardware (e.g., satellite [5], distributed mission-critical systems [6], cluster computing systems [6], and telecommunication systems [7]). However, as software systems have become more complicated and there has been a greater requirement for reliability, the problems have migrated to the software [8]. Taherdoost et al. [9] surveyed to investigate the reasons for the failure and success of various information technology projects. They performed the survey, which included both technical and nontechnical aspects that are directly or indirectly related to the causes of failures, such as people and procedures.Liang et al. [10] proposed an approach for predicting the failures in IBM’s Blue Gene/L from the event logs generated by the systems. Event logs containing the records of the events generated by the system at different points of time are used for prediction. Sequential density is used to cover all the events at a single location. A lot of papers have been proposed in recent years analyzing high-performance computing (HPC) for prediction purposes. But many of these predictors are unable to use the required data for a long time; instead they use it only for short time. Furthermore, they required the new training phase after some time. This is the limitation of these predicting techniques. But many of the researchers tried to overcome these limitations such as Gu et al. [11]. In [11], they proposed two techniques; one is a meta-learning predictor to boost the accuracy and the second is the dynamic approach to collect and deal with the changing training set. The meta-learning predictor was proposed to provide a comparison between the rules-based and the statistical methods and further choose which of them is best for prediction purposes.Nakka et al. [12] employ a hybrid technique to forecast failures in HPC systems, based on their usage as well as information from failure log files. This hybrid approach combines data mining classifications and signal analysis techniques. Another approach for failure prediction is proposed by Zheng and Yu [13] based on the reliability, availability, and serviceability (RAS) and job log files of the high computing system, i.e., Blue Gene/P. In comparison to other approaches, this approach does not predict the failures but filter those that do not affect the applications running on the system. A quite different approach for mining the interdependencies among the components of the HPC systems was proposed by Lou et al. [14]. They also used the log messages from the HPC system applications log messages to extract the information for mining the component’s dependencies.Gainaru et al. [15] suggested a new hybrid approach for predicting high-performance HPC failures using Blue Gene/L log files combining signal analysis and data mining. They also discussed the problems and limitations attached to the failure prediction approaches. Xue et al. [16] talked about the failures in the cluster system and found the methods of collecting and processing data for failure prediction. They suggested a method for preprocessing the data in the log files. The researchers looked at rule-based classification, time series analysis, semi-Markov process models, and Bayesian network models as basic prediction methods. Gainaru et al. [17] presented a novel methodology for online failure prediction and showed that using this model, prediction is possible and easy for small systems. They showed the analysis of the feasibility of the online failure prediction methods on the Blue Waters system on pet scale machines.Shalan and Zulkernine [18] proposed an approach for forecasting the failures in the software system during the system runtime. With the prediction of the failure, this approach also forecasts the occurrence of the modes in the software at the runtime. Pitakrat [19] also presented an online failure prediction approach called Hora. Hora is an online failure prediction approach based on the components of large-scale systems. This approach generates submodels for each component and then combines them using the interdependencies of the components. They used the Kieker framework and other tools such as WEKA and OPAD. Salfner et al. [20] discussed different online failure prediction approaches and developed a taxonomy that shows different approaches, their applications, and the results on implementation. Zhang et al. [9] proposed the new approach CASSANDRA for predicting runtime failures. The two current methodologies, design time and run time analysis techniques, were combined to create this new proposed approach. By developing an on-the-fly model of the future k-step global state space, they were able to forecast runtime problems.Gupta et al. [21] surveyed the statistical method, time series analysis used for the prediction purposes. They studied the time series analysis and elaborated its working and the past work done using it for the software anomalies prediction. Liu et al. [22] proposed a hybrid version for short- and long-term software program failure time forecasting. This version consists of the SSA (singular spectrum analysis) and ARIMA for forecasting the time series of the software failure time. Fan et al. [23] used the time series modeling methods to analyze and forecast the failures in the construction equipment. They used time series approach to detect rules and patterns from huge amounts of data on equipment failures obtained through failure analysis and predictions for construction tools.Among the many predictions strategies, time series analysis is common, but it carries some disadvantages too. A single message which is the source of the information in this approach is thought not to be enough for the failure prediction (Pinheiro et al., [24]). Li et al. [25] proposed the approach based on the time series analysis for detecting and estimating resource exhaustion time due to software aging. Time series ARMA model was developed to identify aging and predict resource exhaustion timeframes.


## 3. Methodology

We suggested a model for predicting software failure incidents using active learning and the Support Vector Machine (SVM) in this study. The dataset was subjected to active learning, which reduced the size of the dataset and picked a sample from it to serve as the training set for the SVM classifier. The sample was chosen because it had occurrences that were both unique and relevant in terms of training the classifier. The clustering technique is used to do active learning in this study. In our approach of clustering, we use k-mean clustering to feed the active learning process. The data was initially clustered using a k-mean clustering approach, and then the cluster representatives were utilized to label the data. These occurrences at the cluster’s center were gathered and labeled by hand. These labeled data were utilized as the SVM classifier’s training set, and classification was performed on it. After clustering, the training set appeared to be devoid of any repeated data and instances that had no useful information. Clustering was kept constant in this research, and no label propagation was used.Clusters were analyzed in different ways to get the well-organized and the most “informative” sample from the dataset. The entropy of every cluster is measured and the clusters with higher entropy were considered to be the most informative. Clusters with diverse classes were also taken into consideration for having the best informative instances. To get the most informative set, different techniques were performed on the clusters. The final sample of the instances obtained was then labeled manually. The labeled training set was then used as the input to the SVM classifier. Sequential minimal optimization (SMO) algorithm of the SVM was selected to perform the classification. Data were split through “percentage splitter” and target class “level” from the attributes set was selected and started the classification procedure. This generated results which are shared.


## 4. Results and Analysis

### 4.1. WEKA 3.8.0

#### 4.1.1. Objectives

Several conventional machine learning algorithms have been included in the program “Workbench” truncated WEKA by the Waikato team (Waikato Environment for Knowledge Analysis). With WEKA, the researcher can better utilize the Ml and extract knowledge from it that would otherwise be impossible to obtain from a vast quantity of data.


#### 4.1.2. Documented Features

The WEKA contains a library of algorithms for perdition and data mining challenges. The software is written in Java 2 and contains a standardized interface to machine learning algorithms. WEKA makes use of the following data mining techniques.(1)
Selection of Attribute.(2)
Clustering.(3)
Classifiers (nonnumeric and both numeric).(4)
Rules for Association.(5)
Filters.(6)
Estimators.


### 4.2. Preprocessing of the Data

In this research log files of the eclipse, software is used as the dataset for the training and testing purposes of the predicting classifier. Log files generated during the last 3 months are collected from the repository of the software. As we know, WEKA uses mostly the ARFF format files and the CSV files; therefore, we transferred the data of the log files into the CSV file format. The dataset consists of 4 attributes, “Date and Time,” “Source,” “Event ID,” and “Task Category”.(1)
Date and Time attribute contains the time of the event occurrence.(2)
Source attribute mentions the node on which the event has been created such as the “software protection service failed,” “Microsoft-Windows-DNS-Client,” “TIMEOUT,” “need updating,” “Rtop service failed,” “application error,” and “ending window installer transaction'.(3)
Event ID contains the IDs for each type of event, but the same sources hold the same IDs even with different levels of severity.(4)
Task Category contains the category each task belongs to, such as “Event System,” “none”,” “−7,” and “−212”.Figure 1
Clustered data.Figure 2
Clustered data.


### 4.3. K-Mean Clustering of the Data

(1)
There are 100 instances and four attributes in our dataset.(2)
After loading the file in the WEKA, the data were subjected to clustering.(3)
Data were then clustered using the simple K-means clustering as shown in Figure1.(4)
Three clusters of the hundred instances were created.(5)
Cluster 0, Cluster 1, and Cluster 2 are as shown in Figure2.


### 4.4. Data Cluster Visualization

The data are clustered and visualized as shown in Figures3 and 4 and in Table 1.Figure 3
Clustered and visualized data.Figure 4
Clustered and visualized data.Table 1
Clustered and visualized data.
NumberInstance_SourceEvent IDTask CategoryClusters10ReasonBy10NoneCluster121App-configuration7000NoneCluster032App-configuration7009NoneCluster043Rtop service1002AddressCluster254Rtop service219−212Cluster265Rtop service4001NoneCluster176ReasonBy10NoneCluster187Network4NoneCluster198Ending37−7Cluster2109Ending37−7Cluster21110Rtop service37−7Cluster21211Rtop service37−7Cluster21312App-configuration7000NoneCluster01413App-configuration7009NoneCluster01514Ending219−212Cluster21615Rtop service4001NoneCluster11716Ending4621Event Sys1Cluster01817App-configuration7000NoneCluster01918App-configurationNoneCluster02019Rtop service1002AddressCluster22120Rtop service219−212Cluster22221Ending4001NoneCluster12322Rtop service4001NoneCluster12423ReasonBy10NoneCluster12524App-configurationNoneCluster02625Rtop service1002AddressCluster22726Ending219−212Cluster22827ReasonBy10NoneCluster12926App-configuration7000NoneCluster03031App-configuration7009NoneCluster03130Rtop service1002AddressCluster23231Software219−212Cluster23332Rtop service4001NoneCluster13433Rtop service4621Event Sys1Cluster23534Rtop service4001NoneCluster13635Network4NoneCluster13736Rtop service1002AddressCluster23837App-configuration7000NoneCluster03938App-configuration7009NoneCluster04039Rtop service4001NoneCluster14140ReasonBy10NoneCluster14241Network4NoneCluster14342Software219NoneCluster14443Software219NoneCluster24544App-configuration7000NoneCluster04645Rtop service4001NoneCluster14746Ending4621Event Sys1Cluster04847Software219-212Cluster21615Rtop service4001NoneCluster15049Need upd1002AddressCluster25150Ending4621Event Sys1Cluster05251App-configuration7000NoneCluster05352Microsoft1002AddressCluster25453Initialization219−212Cluster25554App-configuration7000NoneCluster05655App-configuration7009NoneCluster05756Rtop service4001NoneCluster15857ReasonBy10NoneCluster15958Network4NoneCluster16059Software219−212Cluster26160Rtop service4001NoneCluster16261ReasonBy10NoneCluster16362Application1000−100Cluster26463Application1000−100Cluster26564Rtop service4001NoneCluster16665Need upd4NoneCluster26766Software1101Event Sys1Cluster26867ReasonBy10NoneCluster16968Application1000−100Cluster27069Application1000−100Cluster27170Ending1014NoneCluster27271Software1014NoneCluster07372Software1002AddressCluster27473ReasonBy10NoneCluster17574Ending1014NoneCluster27675Microsoft1014NoneCluster17776Microsoft1014NoneCluster17877Need upd4NoneCluster17978Network4NoneCluster18079Software4621Event Sys1Cluster08180Rtop service4001NoneCluster18281Microsoft52NoneCluster18382Software4NoneCluster08483Software5NoneCluster08584Duplicate4001NoneCluster18685Software219−212Cluster28786Rtop service4001NoneCluster18887Rtop service4621Event Sys1Cluster28988Rtop service4001NoneCluster19089Network4NoneCluster19190Rtop service1002AddressCluster29291App-configuration7000NoneCluster09392Network4NoneCluster19493Rtop service4001NoneCluster19594Microsoft4001NoneCluster09695ReasonBy10NoneCluster19796Ending1014NoneCluster29897Microsoft1014NoneCluster19998Network4NoneCluster110099Need upd4NoneCluster1101100Rtop service37−7Cluster0102101Ending219−212Cluster2103102Network4NoneCluster1104103Software4621Event Sys1Cluster0


### 4.5. Entropy Calculation of the Clusters

(1)
In step one, we created the clusters of the data using the k-mean clustering technique.(2)
Three clusters were created for the “100” instances.(3)
The entropy of each cluster is then measured using the “entropy triangle” package installed in the WEKA.(4)
Cluster “2” is found to have the highest entropy as shown in Figure5.


### 4.6. Cluster with Higher Entropy

Cluster 2 with 38 instances (Table2) was found to have higher entropy due to the diversity of the data it contains. As discussed earlier, the higher the entropy is, the more the randomness of data is. The data in the cluster are assumed to be the most diverse and best for training and testing the classifier due to its uncertainty for the labels.Figure 5
Entropy triangle.Table 2
Clusters with higher entropy.
NumberInstance_SourceEvent IDTask categoryClusters43Rtop service1002AddressCluster254Rtop service219−212Cluster298Ending37−7Cluster2109Ending37−7Cluster21110Rtop service37−7Cluster21211Rtop service37−7Cluster21514Ending219−212Cluster22019Rtop service1002AddressCluster22120Rtop service219−212Cluster22625Rtop service1002AddressCluster22726Ending219−212Cluster23130Rtop service1002AddressCluster23231Software219−212Cluster23433Rtop service4621Event Sys1Cluster23736Rtop service1002AddressCluster24847Software219−212Cluster25049Need upd1002AddressCluster25352Microsoft1002AddressCluster25453Initialization219−212Cluster26059Software219−212Cluster26362Application1000−100Cluster26463Application1000−100Cluster2Table 3
Cluster 1 labels.
Number.InstanceSourceEvent IDTask categoryClusters43Rtop service1002AddressFailed54Rtop service219−212Failed98Ending37−7Status ok109Ending37−7Status ok1110Rtop service37−7Failed1211Rtop service37−7Failed1514Ending219−212Status ok2019Rtop service1002AddressError2120Rtop service219−212Warning2625Rtop service1002AddressWarning2726Ending219−212Status ok3130Rtop service1002AddressError3231Software219−212Warning3433Rtop service4621Event Sys1Error3736Rtop service1002AddressError4847Software219−212Failed5049Need upd1002AddressError5352Microsoft1002AddressError5453Initialization219−212Warning6059Software219−212Warning6362Application1000−100Error6463Application1000−100ErrorTable 4
Cluster 2 labels.
NumberInstance_SourceEvent IDTask categoryClusters43Rtop service1002AddressFailed54Rtop service219−212Failed98Ending37−7Status ok109Ending37−7Status ok1110Rtop service37−7Failed1211Rtop service37−7Failed1514Ending219−212Status ok2019Rtop service1002AddressError2120Rtop service219−212Warning2625Rtop service1002AddressWarning2726Ending219−212Status ok3130Rtop service1002AddressError3231Software219−212Warning3433Rtop service4621Event Sys1Error3736Rtop service1002AddressError3938Microsoft1014NoneWarning4140Microsoft1014NoneWarning4241Need upd4NoneWarning4544Network4NoneWarning4645Software4621Event Sys1Failed4746Microsoft52NoneWarning4847Software219−212Failed5049Need upd1002AddressError5352Microsoft1002AddressError5453Initialization219−212Warning6059Software219−212Warning6362Application1000−100Error6463Application1000−100Error


### 4.7. Evaluation of the Entropy Calculation (Manual Labeling)

(1)
The entropy of Cluster 2 is the highest among all the clusters.(2)
The highest entropy means it contains diverse data.(3)
Cluster 1 has more instances than Cluster 2, but it does not have a variety of classes.(4)
Table1 shows that Cluster 1 has the “warning” class instances more than any other class.(5)
Figure5 shows that Cluster 2 with higher entropy has a variety of classes.(6)
A comparison of both is shown in Tables3 and 4.(7)
Both were manually labeled to evaluate the entropy calculation.(8)
A new attribute with the name “LEVEL” is added to the dataset. The level attribute contains the data of the level of the severity of the generated log file against any event that occurred in the software. The levels can be of 4 types in our dataset “WARNING,” “ERROR,” “FAILED,” and “STATUS-OK”.


## 4.1. WEKA 3.8.0

### 4.1.1. Objectives

Several conventional machine learning algorithms have been included in the program “Workbench” truncated WEKA by the Waikato team (Waikato Environment for Knowledge Analysis). With WEKA, the researcher can better utilize the Ml and extract knowledge from it that would otherwise be impossible to obtain from a vast quantity of data.


### 4.1.2. Documented Features

The WEKA contains a library of algorithms for perdition and data mining challenges. The software is written in Java 2 and contains a standardized interface to machine learning algorithms. WEKA makes use of the following data mining techniques.(1)
Selection of Attribute.(2)
Clustering.(3)
Classifiers (nonnumeric and both numeric).(4)
Rules for Association.(5)
Filters.(6)
Estimators.


## 4.1.1. Objectives

Several conventional machine learning algorithms have been included in the program “Workbench” truncated WEKA by the Waikato team (Waikato Environment for Knowledge Analysis). With WEKA, the researcher can better utilize the Ml and extract knowledge from it that would otherwise be impossible to obtain from a vast quantity of data.


## 4.1.2. Documented Features

The WEKA contains a library of algorithms for perdition and data mining challenges. The software is written in Java 2 and contains a standardized interface to machine learning algorithms. WEKA makes use of the following data mining techniques.(1)
Selection of Attribute.(2)
Clustering.(3)
Classifiers (nonnumeric and both numeric).(4)
Rules for Association.(5)
Filters.(6)
Estimators.


## 4.2. Preprocessing of the Data

In this research log files of the eclipse, software is used as the dataset for the training and testing purposes of the predicting classifier. Log files generated during the last 3 months are collected from the repository of the software. As we know, WEKA uses mostly the ARFF format files and the CSV files; therefore, we transferred the data of the log files into the CSV file format. The dataset consists of 4 attributes, “Date and Time,” “Source,” “Event ID,” and “Task Category”.(1)
Date and Time attribute contains the time of the event occurrence.(2)
Source attribute mentions the node on which the event has been created such as the “software protection service failed,” “Microsoft-Windows-DNS-Client,” “TIMEOUT,” “need updating,” “Rtop service failed,” “application error,” and “ending window installer transaction'.(3)
Event ID contains the IDs for each type of event, but the same sources hold the same IDs even with different levels of severity.(4)
Task Category contains the category each task belongs to, such as “Event System,” “none”,” “−7,” and “−212”.Figure 1
Clustered data.Figure 2
Clustered data.


## 4.3. K-Mean Clustering of the Data

(1)
There are 100 instances and four attributes in our dataset.(2)
After loading the file in the WEKA, the data were subjected to clustering.(3)
Data were then clustered using the simple K-means clustering as shown in Figure1.(4)
Three clusters of the hundred instances were created.(5)
Cluster 0, Cluster 1, and Cluster 2 are as shown in Figure2.


## 4.4. Data Cluster Visualization

The data are clustered and visualized as shown in Figures3 and 4 and in Table 1.Figure 3
Clustered and visualized data.Figure 4
Clustered and visualized data.Table 1
Clustered and visualized data.
NumberInstance_SourceEvent IDTask CategoryClusters10ReasonBy10NoneCluster121App-configuration7000NoneCluster032App-configuration7009NoneCluster043Rtop service1002AddressCluster254Rtop service219−212Cluster265Rtop service4001NoneCluster176ReasonBy10NoneCluster187Network4NoneCluster198Ending37−7Cluster2109Ending37−7Cluster21110Rtop service37−7Cluster21211Rtop service37−7Cluster21312App-configuration7000NoneCluster01413App-configuration7009NoneCluster01514Ending219−212Cluster21615Rtop service4001NoneCluster11716Ending4621Event Sys1Cluster01817App-configuration7000NoneCluster01918App-configurationNoneCluster02019Rtop service1002AddressCluster22120Rtop service219−212Cluster22221Ending4001NoneCluster12322Rtop service4001NoneCluster12423ReasonBy10NoneCluster12524App-configurationNoneCluster02625Rtop service1002AddressCluster22726Ending219−212Cluster22827ReasonBy10NoneCluster12926App-configuration7000NoneCluster03031App-configuration7009NoneCluster03130Rtop service1002AddressCluster23231Software219−212Cluster23332Rtop service4001NoneCluster13433Rtop service4621Event Sys1Cluster23534Rtop service4001NoneCluster13635Network4NoneCluster13736Rtop service1002AddressCluster23837App-configuration7000NoneCluster03938App-configuration7009NoneCluster04039Rtop service4001NoneCluster14140ReasonBy10NoneCluster14241Network4NoneCluster14342Software219NoneCluster14443Software219NoneCluster24544App-configuration7000NoneCluster04645Rtop service4001NoneCluster14746Ending4621Event Sys1Cluster04847Software219-212Cluster21615Rtop service4001NoneCluster15049Need upd1002AddressCluster25150Ending4621Event Sys1Cluster05251App-configuration7000NoneCluster05352Microsoft1002AddressCluster25453Initialization219−212Cluster25554App-configuration7000NoneCluster05655App-configuration7009NoneCluster05756Rtop service4001NoneCluster15857ReasonBy10NoneCluster15958Network4NoneCluster16059Software219−212Cluster26160Rtop service4001NoneCluster16261ReasonBy10NoneCluster16362Application1000−100Cluster26463Application1000−100Cluster26564Rtop service4001NoneCluster16665Need upd4NoneCluster26766Software1101Event Sys1Cluster26867ReasonBy10NoneCluster16968Application1000−100Cluster27069Application1000−100Cluster27170Ending1014NoneCluster27271Software1014NoneCluster07372Software1002AddressCluster27473ReasonBy10NoneCluster17574Ending1014NoneCluster27675Microsoft1014NoneCluster17776Microsoft1014NoneCluster17877Need upd4NoneCluster17978Network4NoneCluster18079Software4621Event Sys1Cluster08180Rtop service4001NoneCluster18281Microsoft52NoneCluster18382Software4NoneCluster08483Software5NoneCluster08584Duplicate4001NoneCluster18685Software219−212Cluster28786Rtop service4001NoneCluster18887Rtop service4621Event Sys1Cluster28988Rtop service4001NoneCluster19089Network4NoneCluster19190Rtop service1002AddressCluster29291App-configuration7000NoneCluster09392Network4NoneCluster19493Rtop service4001NoneCluster19594Microsoft4001NoneCluster09695ReasonBy10NoneCluster19796Ending1014NoneCluster29897Microsoft1014NoneCluster19998Network4NoneCluster110099Need upd4NoneCluster1101100Rtop service37−7Cluster0102101Ending219−212Cluster2103102Network4NoneCluster1104103Software4621Event Sys1Cluster0


## 4.5. Entropy Calculation of the Clusters

(1)
In step one, we created the clusters of the data using the k-mean clustering technique.(2)
Three clusters were created for the “100” instances.(3)
The entropy of each cluster is then measured using the “entropy triangle” package installed in the WEKA.(4)
Cluster “2” is found to have the highest entropy as shown in Figure5.


## 4.6. Cluster with Higher Entropy

Cluster 2 with 38 instances (Table2) was found to have higher entropy due to the diversity of the data it contains. As discussed earlier, the higher the entropy is, the more the randomness of data is. The data in the cluster are assumed to be the most diverse and best for training and testing the classifier due to its uncertainty for the labels.Figure 5
Entropy triangle.Table 2
Clusters with higher entropy.
NumberInstance_SourceEvent IDTask categoryClusters43Rtop service1002AddressCluster254Rtop service219−212Cluster298Ending37−7Cluster2109Ending37−7Cluster21110Rtop service37−7Cluster21211Rtop service37−7Cluster21514Ending219−212Cluster22019Rtop service1002AddressCluster22120Rtop service219−212Cluster22625Rtop service1002AddressCluster22726Ending219−212Cluster23130Rtop service1002AddressCluster23231Software219−212Cluster23433Rtop service4621Event Sys1Cluster23736Rtop service1002AddressCluster24847Software219−212Cluster25049Need upd1002AddressCluster25352Microsoft1002AddressCluster25453Initialization219−212Cluster26059Software219−212Cluster26362Application1000−100Cluster26463Application1000−100Cluster2Table 3
Cluster 1 labels.
Number.InstanceSourceEvent IDTask categoryClusters43Rtop service1002AddressFailed54Rtop service219−212Failed98Ending37−7Status ok109Ending37−7Status ok1110Rtop service37−7Failed1211Rtop service37−7Failed1514Ending219−212Status ok2019Rtop service1002AddressError2120Rtop service219−212Warning2625Rtop service1002AddressWarning2726Ending219−212Status ok3130Rtop service1002AddressError3231Software219−212Warning3433Rtop service4621Event Sys1Error3736Rtop service1002AddressError4847Software219−212Failed5049Need upd1002AddressError5352Microsoft1002AddressError5453Initialization219−212Warning6059Software219−212Warning6362Application1000−100Error6463Application1000−100ErrorTable 4
Cluster 2 labels.
NumberInstance_SourceEvent IDTask categoryClusters43Rtop service1002AddressFailed54Rtop service219−212Failed98Ending37−7Status ok109Ending37−7Status ok1110Rtop service37−7Failed1211Rtop service37−7Failed1514Ending219−212Status ok2019Rtop service1002AddressError2120Rtop service219−212Warning2625Rtop service1002AddressWarning2726Ending219−212Status ok3130Rtop service1002AddressError3231Software219−212Warning3433Rtop service4621Event Sys1Error3736Rtop service1002AddressError3938Microsoft1014NoneWarning4140Microsoft1014NoneWarning4241Need upd4NoneWarning4544Network4NoneWarning4645Software4621Event Sys1Failed4746Microsoft52NoneWarning4847Software219−212Failed5049Need upd1002AddressError5352Microsoft1002AddressError5453Initialization219−212Warning6059Software219−212Warning6362Application1000−100Error6463Application1000−100Error


## 4.7. Evaluation of the Entropy Calculation (Manual Labeling)

(1)
The entropy of Cluster 2 is the highest among all the clusters.(2)
The highest entropy means it contains diverse data.(3)
Cluster 1 has more instances than Cluster 2, but it does not have a variety of classes.(4)
Table1 shows that Cluster 1 has the “warning” class instances more than any other class.(5)
Figure5 shows that Cluster 2 with higher entropy has a variety of classes.(6)
A comparison of both is shown in Tables3 and 4.(7)
Both were manually labeled to evaluate the entropy calculation.(8)
A new attribute with the name “LEVEL” is added to the dataset. The level attribute contains the data of the level of the severity of the generated log file against any event that occurred in the software. The levels can be of 4 types in our dataset “WARNING,” “ERROR,” “FAILED,” and “STATUS-OK”.


## 5. Results Descriptions

We have started our process of model building with the “Explorer” window. Explorer window toolbar items “preprocessing,” “Classify,” and “Cluster” are used to build the model. Figure1 depicts our dataset file with the load in WEKA, WEKA calculated the attributes, instances, weighted averages, uniqueness, and the classes. The right corner of the window gave the graphical visualization of our instances. Figure 2 is the type of table containing the summary of the clustering. Clustering of the whole dataset is performed, and 3 clusters were created.Tables5 and 6 are the summaries of the k means clustering performed in the WEKA. Table 7 shows the model and evaluation training on set. Figures 3 and 4 are the graphical visualizations of the k-mean clustering. The clusters of the 100 instances saved for labeling are shown in Table 1. Table 2 is the Cluster 2 data chosen for labeling. Figure 5 shows that Cluster 2 with higher entropy has a variety of classes. Table 3 is Cluster 1, which shows the repetition of the same class. Table 4 is Cluster 2 for labeling. Table 8 is the Cluster 2 data chosen for labeling. Figure 6 is the extracted set of instances from the whole dataset chosen for labeling and is the most informative dataset for training the classifier. The dataset was labeled manually and then was expected to be the most useful subset for training the classifier. Table 9 manual labeling of Cluster 2. Three clusters can be seen in the window with different numbers of the data points in Table 10. Table 11 shows the detailed accuracy by class and Table 12 is the confusion matrix. Figure 5 is the entropy calculation of the clusters. The entropy of each cluster is calculated and the clusters with high entropy and the large size are chosen for labeling and considered the most informative cluster for training the classifier.Table 5
Summary of K-mean clustering.
k-Means======Number of iterations: 4Within cluster sum of squared errors: 186.97500542403395Initial starting points (random):Cluster 0: “7/20/2017 23:10”“App-configuration failed”7009NoneCluster 1: “7/18/2017 23:09”“Need updation”6008NoneCluster 2: “7/20/2017 23:34”“Ending window installer transaction”37−7Missing values globally replaced with mean/modeFinal cluster centroids:Clusters #AttributeFull dataCluster 0Cluster 1Cluster 2(100)(21.0)(41.0)(38.0)Date and time7/17/2017 12:077/21/2017 11:157/17/2017 12:077/20/2017 23:34SourceSoftware protection service failedApp configuration failedSoftware protection service failedRtop service failedEvent ID2204.016617.42861477.0244549.3947Task categoryNoneNoneNone−212Time taken to build the model (full training data): 0.02 secondsTable 6
Summary of k-mean clustering.
AttributeFull data (100.0)0 (21.0)1 (41.0)2 (38.0)Instance number49.534.666754.073252.7632Date and time7/17/2017 12:077/21/2017 11:557/17/2017 12:077/20/2017 23:34SourceSoftware protection service failedApp-configuration failedSoftware protection service failedRtop service failedEvent ID2204.016617.42861477.0244549.3947Task categoryNoneNoneNone-212ClusterCluster1Cluster0Cluster1Cluster2Table 7
Model and evaluation on training set.
Clustered instances021 (21%)141 (41%)238 (38%)Table 8
Selection of clusters.
Number.InstanceSourceEvent IDTask categoryClusters43Rtop service1002AddressCluster254Rtop service219−212Cluster298Ending37−7Cluster2109Ending37−7Cluster21110Rtop service37−7Cluster21211Rtop service37−7Cluster21514Ending219−212Cluster22019Rtop service1002AddressCluster22120Rtop service219−212Cluster22625Rtop service1002AddressCluster22726Ending219−212Cluster23130Rtop service1002AddressCluster23231Software219−212Cluster23433Rtop service4621Event Sys1Cluster23736Rtop service1002AddressCluster24847Software219−212Cluster25049Need upd1002AddressCluster25352Microsoft1002AddressCluster25453Initialization219−212Cluster26059Software219−212Cluster26362Application1000−100Cluster26463Application1000−100Cluster2Figure 6
Selection of clusters.Table 9
Selection of clusters.
Number.Instance_SourceEvent IDTask categoryClusters43Rtop service1002AddressFailed54Rtop service219−212Failed98Ending37−7Status ok109Ending37−7Status ok1110Rtop service37−7Failed1211Rtop service37−7Failed1514Ending219−212Status ok2019Rtop service1002AddressError2120Rtop service219−212Warning2625Rtop service1002AddressWarning2726Ending219−212Status ok3130Rtop service1002AddressError3231Software219−212Warning3433Rtop service4621Event Sys1Error3736Rtop service1002AddressError3938Microsoft1014NoneWarning4140Microsoft1014NoneWarning4241Need upd4NoneWarning4544Network4NoneWarning4645Software4621Event Sys1Failed4746Microsoft52NoneWarning4847Software219−212Failed5049Need upd1002AddressError5352Microsoft1002AddressError5453Initialization219−212Warning6059Software219−212Warning6362Application1000−100Error6463Application1000−100ErrorTable 10
Clustered instances.
Clustered instances021 (21%)141 (41%)238 (38%)Table 11
Detailed accuracy.
TP rateFP ratePrecisionRecallF-measureMCCROC areaPRC areaClass1.0000.0830.5001.0000.6670.6670.9580.500Failed0.5000.0001.0000.5000.6670.6670.9770.833Status ok1.0000.1110.8001.0000.8890.8430.9440.800Error0.8330.0001.0000.8330.9090.8540.9170.910Warning0.0000.0000.0000.0000.0000.000??CriticalWeighted AVG0.8460.0410.9000.8460.8470.8100.9380.833Table 12
Confusion matrix.
abcdE<--Classified as10000|a=Failed01100|b=Status ok00400|c=Error10050|d=WarningFigures7 and 8 are the summary of the classification performed on the selected cluster. Cluster 2 is chosen because it has high entropy and the number of instances it has is higher than the other clusters. With percentage, the False Positive (FP) Rate, True Positive (TP) Rate, Precision Recall, F-measure, ROC, MCC, and PRC Area for each characteristic are displayed in the test summary as well as the correctly classified and nonclassified occurrences.Figure 7
Kernel selection.Figure 8
Model for test data.


## 6. Discussion

Our model had an accuracy of 84 percent, properly categorizing the vast majority of the occurrences with only two exceptions. The test model, as shown in Figure8, displayed the model’s detailed accuracy measure using terminology such as F-measure and Precision Recall. Some terminology, such as FP Rate, TP Rate, F-measure, Precision Recall, MCC, ROC, and PRC Area must be understood before addressing the measure.


### 6.1. True Positive (TP)

Positive values are both observed and forecasted to be positive. In our model, the TPR for the “failed” level is 1.000; for status ok it is 0.5; for error it is 1.00; and for warning it is 0.833.


### 6.2. False Positive (FP)

When a negative value is observed, a positive forecast result is obtained. In our situation, FP stands for failure level which is 0.08 and for error level it is 0.111.


### 6.3. Precision

Precision is calculated as the number of accurately identified positive events divided by the total number of occurrences predicted. As indicated in Figure9, the precision rate for failure is 0.5, status ok is 1.000, error is 0.8, and warning is 1.00.(1)Precision=True PositiveTrue Positive+False Positive.Figure 9
Failure precision (plot under ROC).


### 6.4. Recall

The number of accurately projected positive values divided by the total number of observations is called Recall. As previously stated, recall is the proportion of true positive observations to total observations. In our situation, the Recall is “1.000” for failure, “0.5” for status ok, “1.000” for error, and “0.833” for warning.(2)Recall=True positiveTrue positive+False Negative.


### 6.5. F-Measure

As indicated in Figure10, the F-measure for failure is “0.667,” status ok is “0.677,” error is “0.84,” and warning is “0.85” in our model.(3)Fmeasure=2Recall×PrecisionRecall×Precision.Figure 10
FailureF-measure (plot under ROC).


### 6.6. Confusion Matrix

The confusion matrix, also known as the error matrix (Table12), is a visual representation of the technique’s or algorithm’s performance. The projected cases are in the rows, whereas the actual instances are in the columns.


## 6.1. True Positive (TP)

Positive values are both observed and forecasted to be positive. In our model, the TPR for the “failed” level is 1.000; for status ok it is 0.5; for error it is 1.00; and for warning it is 0.833.


## 6.2. False Positive (FP)

When a negative value is observed, a positive forecast result is obtained. In our situation, FP stands for failure level which is 0.08 and for error level it is 0.111.


## 6.3. Precision

Precision is calculated as the number of accurately identified positive events divided by the total number of occurrences predicted. As indicated in Figure9, the precision rate for failure is 0.5, status ok is 1.000, error is 0.8, and warning is 1.00.(1)Precision=True PositiveTrue Positive+False Positive.Figure 9
Failure precision (plot under ROC).


## 6.4. Recall

The number of accurately projected positive values divided by the total number of observations is called Recall. As previously stated, recall is the proportion of true positive observations to total observations. In our situation, the Recall is “1.000” for failure, “0.5” for status ok, “1.000” for error, and “0.833” for warning.(2)Recall=True positiveTrue positive+False Negative.


## 6.5. F-Measure

As indicated in Figure10, the F-measure for failure is “0.667,” status ok is “0.677,” error is “0.84,” and warning is “0.85” in our model.(3)Fmeasure=2Recall×PrecisionRecall×Precision.Figure 10
FailureF-measure (plot under ROC).


## 6.6. Confusion Matrix

The confusion matrix, also known as the error matrix (Table12), is a visual representation of the technique’s or algorithm’s performance. The projected cases are in the rows, whereas the actual instances are in the columns.


## 7. Conclusion and Future Work

Every failure, in general, is critical in terms of both security and cost. Forecasting techniques can be used to recognize enhanced and better maintenance schedules. Failure forecasts aid in the prediction of maintenance times, reducing both costs and security richness. This research provided a model for forecasting failures based on machine learning methodologies and techniques, active learning via clustering, and SVM classification of selected examples. Although SVM calculations are known to be capable of predicting, it is unclear how to choose the parameter values that will provide a satisfactory result. However, modeling a function for transformative computations to be used in determining requirements for the combination of a large number of possible outcomes is difficult. The goal of this study is to predict software faults in order to optimize maintenance schedules and demonstrate and predict sophisticated software system failures. We used two machine learning algorithms to do this. We gathered log papers with the four qualities and 100 examples. We used a dynamic learning method to reduce the number of variables.Grouping and SVM were used to display event-driven error log records. Review, exactness,F-measure, and precision were used to describe the models’ quality. Our findings show that active learning and SVM are the most commonly used techniques. Expecting all failures to keep a strategic distance from our showing techniques may result in a request for a significant modification in framework accessibility. The goal is to achieve the best execution with the most useful information.



---

*Source: 9609823-2021-12-30.xml*