Anomaly detection for condition monitoring data using auxiliary feature vector and density‐based clustering

IET Generation, Transmission & DistributionVolume 14, Issue 1 p. 108-118 Research ArticleFree Access Anomaly detection for condition monitoring data using auxiliary feature vector and density-based clustering Hang Liu, Corresponding Author Hang Liu liuhangsheep@163.com State Key Laboratory of Power Transmission Equipment and System Security and New Technology, Chongqing University, Chongqing, 400044 People's Republic of ChinaSearch for more papers by this authorYouyuan Wang, Youyuan Wang State Key Laboratory of Power Transmission Equipment and System Security and New Technology, Chongqing University, Chongqing, 400044 People's Republic of ChinaSearch for more papers by this authorWeiGen Chen, WeiGen Chen State Key Laboratory of Power Transmission Equipment and System Security and New Technology, Chongqing University, Chongqing, 400044 People's Republic of ChinaSearch for more papers by this author Hang Liu, Corresponding Author Hang Liu liuhangsheep@163.com State Key Laboratory of Power Transmission Equipment and System Security and New Technology, Chongqing University, Chongqing, 400044 People's Republic of ChinaSearch for more papers by this authorYouyuan Wang, Youyuan Wang State Key Laboratory of Power Transmission Equipment and System Security and New Technology, Chongqing University, Chongqing, 400044 People's Republic of ChinaSearch for more papers by this authorWeiGen Chen, WeiGen Chen State Key Laboratory of Power Transmission Equipment and System Security and New Technology, Chongqing University, Chongqing, 400044 People's Republic of ChinaSearch for more papers by this author First published: 02 December 2019 https://doi.org/10.1049/iet-gtd.2019.0682Citations: 4AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract High-quality condition monitoring data can provide vital information on power equipment condition assessment and fault diagnosis. However, data quality is difficult to guarantee because valid and invalid anomalies, and different normal data patterns inevitably occur in datasets. This study presents a method for anomaly detection based on auxiliary feature vector and density-based spatial clustering of applications with noise (DBSCAN). The auxiliary feature vectors of each condition variable are constructed for clustering to recognise normal data patterns and different types of anomalies. Furthermore, a heuristic method based on the 'number of clusters–Eps' curve is proposed for the parameter selection of DBSCAN in an unsupervised setting. Different application examples are implemented on data of dissolved gas content in transformer oil. Compared with state-of-the-art anomaly detection techniques, the proposed method shows an ability to identify and distinguish normal data patterns and valid and invalid anomalies accurately, provided that the condition monitoring data satisfy the assumption of stationarity. 1 Introduction 1.1 Motivation Condition monitoring data are essential to condition assessment and fault diagnosis of power equipment [1, 2]. However, the original data are always corrupted by different anomalies during data acquisition. Poor quality data may lead to incorrect fault diagnosis results, which lead to equipment failure and power supply disruption. Therefore, anomaly detection for original condition monitoring data is crucial before they are applied for further data analysis [3]. However, anomalies are difficult to detect due to complicated data patterns. (i) Changes in normal data patterns: Basic data distribution can evolve with time, resulting in multiple normal data patterns known as 'concept drift' [4, 5]. Some anomaly detection methods may incorrectly treat the drift as outliers. (ii) Valid anomalies: The observations from condition monitoring data considerably deviate from normal regions and become valid anomalies when the equipment suffers from different types of faults or defects. Valid anomalies provide important information for fault diagnosis and should not be cleaned by any data filtering process. (iii) Invalid anomalies: Data that deviate irregularly from normal patterns are considered invalid anomalies or outliers. They are generated by various random factors, including sensor faults or noise. These data should be detected and cleaned before further data analysis due to the irrelevance of equipment conditions or faults. According to the characteristics of condition monitoring data, anomaly detection should accurately identify and distinguish different normal data patterns and valid and invalid anomalies. It should not be addressed as a filtering process for invalid data because it will delete vital valid anomalies. 1.2 Literature review Anomaly detection has been developed and applied to different domains [6]. On the basis of data label availability, the techniques can be divided into three broad categories. Supervised anomaly detection is performed by using labelled training data and a classifier. In general, normal observations are set to a positive class (+1), whereas anomalous observations are labelled as a negative class (−1). Artificial neural networks and support vector machines (SVMs) are commonly used classifiers. For semisupervised anomaly detection, training data label observations only for the normal class. One-class classifiers, such as one-class SVM (OCSVM) and one-class kernel fisher discriminants, are trained to learn a discriminative boundary around normal instances [6]. In recent years, new and precise semisupervised methods have been proposed. Gautam et al. [7] proposed a localised multiple kernel learning approach for anomaly detection (LMKAD), which outperformed conventional multiple kernel anomaly detection and OCSVM according to the experiment results from over 25 benchmark datasets. Miao et al. [8] proposed a distributed OCSVM for wireless sensor network (WSN) anomaly detection, which requires relatively shorter running time and lower CPU memory consumption compared with OCSVM. Zhang et al. [9] introduced a positive and unlabelled learning framework for anomaly detection by using limited labelled normal samples. Nevertheless, these two technologies are not applicable to condition monitoring data due to the following factors: (i) Most of the observations are unlabelled in the datasets. Accurate and representative labels are costly and challenging to obtain for all the observations. (ii) Condition monitoring data are presented as time series. Therefore, a training dataset that covers all the possible data patterns is impossible to obtain because new data patterns or new types of anomalies may occur in the future. (iii) The anomalies are significantly fewer than normal data, resulting in an imbalanced dataset that affects the performance of machine learning algorithms. (iv) Most existing methods cannot distinguish different valid and invalid anomalies because they have the same label (−1). In this study, we are concerned about unsupervised anomaly detection technologies, which can be grouped into statistic-, nearest neighbour-, and model-based technologies. Statistic-based anomaly detection establishes the lower and upper confidence intervals to describe the boundary of normal data. The observations outside the intervals are identified as anomalies. This technology has been widely applied to the power industry. For example, the boxplot rule, the 2σ rule of Gaussian and gamma distributions, and non-parametric regression methods have been applied to load curve data cleaning [10, 11]. Catterson and Mcarthur [12] used Gaussian mixture modelling to obtain the distribution and confidence intervals of the normal condition monitoring data of power transformers. Beghi et al. [13] exploited principal component analysis (PCA) to distinguish anomalies from normal operation variability. Numerous new PCA-based extensions, such as kernel PCA (KPCA) [14, 15] and ensemble KPCA (EKPCA) [16], have also been proposed. However, the confidence intervals are generally determined using a typical significant level, which is not optimal in practical applications. Nearest neighbour-based technologies assume that normal observations occur in dense neighbourhoods, whereas anomalies occur far from their neighbours [6]. In general, an anomalous observation is identified if it is distant from its neighbours or in a low-density region. Campos et al. [17] summarised 12 types of methods, including but not limited to k nearest neighbour (k NN), k NN-weight (k NNW), local outlier factor (LOF) [18], and connectivity-based outlier factor [19]. In recent years, some new variants, such as a relative density-based outlier score (RDOS) [20] and a new local outlier detection approach named N2DLOF [21], have been proposed. According to the numerical experiment results and conclusions summarised in [17], seminal methods k NN, k NNW, and LOF are standard, and none of the new methods seem to offer a comprehensive improvement of the seminal methods. In recent years, many excellent model-based methods have been presented. Liu et al. [22] proposed an isolation-based anomaly detection method named i Forest, which assumed that anomalies are 'few and different'. i Forest requires a low-linear time complexity and small memory requirement, which is suitable for large and high-dimensional datasets. Zhou et al. [23] adopted an unsupervised ensemble learning approach to develop bad data detection for PMU data, which invoked a set of base detectors to generate anomaly scores and made decisions by aggregating the scores from each detector. Ahmed et al. [24] proposed a minimum spanning tree-based anomaly detection method, which provided a new distance measure to capture the relative connectivity of data points or clusters and identify anomalies in unsupervised learning settings. Ester et al. [25] presented a clustering-based anomaly detection method named density-based spatial clustering of applications with noise (DBSCAN). This method relies on the assumption that normal observations belong to different clusters, whereas anomalies do not belong to any cluster. DBSCAN is widely used for anomaly detection in many domains, such as WSNs [26], screw chiller sensors [27], and phasor measurement units [28]. The anomaly detection ability is a by-product because the main aim of DBSCAN is to find clusters. DBSCAN can identify different normal data patterns and anomalies, thereby avoiding any incorrect deletion of valid anomalous data. 1.3 Model and contribution To distinguish different data patterns of the condition monitoring data, especially valid and invalid anomalies, we propose an anomaly detection method based on auxiliary feature vector and DBSCAN, which can recognise patterns instead of simply detecting anomalies. Initially, the original time series of that condition monitoring data is divided into different segments in accordance with the positions of data interruptions. For each data segment, the auxiliary feature vectors of each condition variable are constructed for clustering via DBSCAN to obtain an accurate recognition of normal patterns and valid and invalid anomalies. To select the optimal parameters of DBSCAN in unsupervised settings, we propose a heuristic method based on the 'number of clusters–Eps' curve. In summary, our contribution is as follows: (i) Identification and discrimination of normal data patterns and valid and invalid anomalies are obtained using auxiliary feature vectors and DBSCAN. (ii) The optimal parameters of DBSCAN are selected by the 'number of clusters–Eps' curve. (iii) The proposed method is appropriate for detecting unlabelled and imbalanced conditions in monitoring datasets, which rarely have anomalies. 1.4 Outline of the paper The rest of the paper is organised as follows: The characteristics of condition monitoring data are illustrated in Section 2. Data preprocessing and auxiliary feature vector construction are presented in Section 3. Section 4 presents the parameter selection method based on the 'number of clusters–Eps' curve and anomaly detection procedure. Different application examples are presented in Section 5, and conclusions are drawn in Section 6. 2 Characteristics of condition monitoring data 2.1 Data measurement and acquisition Various types of normal condition monitoring data have satisfied the assumption of stationarity [29]. An example is the dissolved gases in transformer oil, along with the moisture content, density, and pressure of SF6 circuit breakers. The stationarity assumption is not satisfied due to the influence of changes in normal data patterns and valid and invalid anomalies. In this section, the characteristics of condition monitoring data are analysed by studying dissolved gases, including hydrogen (H2), methane (CH4), ethane (C2 H6), ethylene (C2 H4), acetylene (C2 H2), carbon monoxide (CO), and carbon dioxide (CO2), in oil [30, 31]. The gas content can be measured using online- and offline-based methods. The basic procedure is shown in Fig. 1. Fig. 1Open in figure viewerPowerPoint Measurement of dissolved gases in transformer oil In this study, historical failure cases provided by several electric power companies in China were measured using the offline-based method before 2010 because the transformers were not equipped with online monitoring devices. Based on [32], offline measurement consists of three major steps. (1) The sample oil is extracted from an oil valve using a glass syringe. (2) The gas mixtures are extracted from the sample oil. (3) The gas mixtures are separated and measured using a gas chromatograph. Each observation is measured at an unfixed time interval from several days or several weeks because the offline measurement is complicated and time-consuming. After 2010, transformers beyond 110 kV of an electric power company in China had been equipped with online monitoring devices (The models include MAG2000-7H, Trom600, and ZF3000.). Each observation is measured at a fixed interval of one day. The measurement of the dissolved gas content using the online-based method is quite different from the offline-based method. First, the gas mixtures from the oil are extracted through a permeable membrane. Then, the gas mixtures are separated using chromatographic columns. The gas content is converted into electrical signal sensors, and the signals are transmitted to the industry computer through a communication cable. Finally, the original signal is processed, and the measured data are stored in the datasets. In this study, online monitoring data are obtained by consulting the datasets. However, faults rarely occur, and valid anomalies in online monitoring data lack due to the high reliability of the transformer and the wide application of condition-based maintenance. 2.2 Changes in normal data patterns In dynamically changing and nonstationary operation environments, the data distribution of a time series commonly evolves as time progresses; this phenomenon is known as concept drift [4, 5]. An example of this is in the H2 content data of a transformer (220/121/10.5 kV, 150 MVA), as shown in Fig. 2. The gas content abruptly increases and decreases in the two blue dotted boxes. These phenomena are caused by various factors. For instance, the decline of gas content is commonly caused by the replacement or calibration of sensors, replenishment or filtration of transformer oil, an outage of the transformer. Concept drift causes multiple normal data patterns and increases the difficulty of anomaly detection. For example, it may lead to the failure of statistic-based methods because the observations of a new normal pattern may be outside the confidence intervals determined by the old normal patterns. Fig. 2Open in figure viewerPowerPoint Online monitoring data of H2 content 2.3 Valid anomalies The faults or defects of a transformer cause anomalous and consecutive increasing trends in various gas components. These anomalous observations are defined as valid anomalies. Here, we take 29 historical overheating fault cases and 22 historical discharging fault cases as examples. Each fault case comes from different transformers with different nameplate ratings. The fault observations in each case are continuously measured and presented in Figs. 3a and b, respectively. The horizontal axis in Fig. 3 represents the measurement time of the fault observations (Measurement time is translated such that the first observation of each case coincides with the same reference date.). Two trend lines are fitted using the exponential function, and the R2 coefficient results are 0.5112 and 0.2226, respectively. In Fig. 3, the dispersity of the historical fault cases is large because of the differences, such as nameplate ratings, operation environment, and severity of the fault, between each transformer. Though the R2 coefficient is small, the increasing trend of gas content as time progresses is clear. Gases generated in oil can be used for the qualitative determination of fault types based on different 'key gases'. The key gases of overheating and discharging faults are summarised in Table 1 on the basis of the IEEE standard [31]. According to the analysis above, valid anomalies present the characteristics of 'concurrency and consecution'. Fig. 3Open in figure viewerPowerPoint Increasing trend of H2 content in different fault types (a) Overheating fault, (b) Discharging fault Table 1. Key gases of overheating and discharging faults based on the IEEE standard [31] Fault type Principal gases Secondary gases overheating CH4, C2 H4 H2, C2 H6 discharging H2, C2 H2 CH4, C2 H4, C2 H6 2.4 Invalid anomalies Invalid anomalies refer to the observations that deviate from normal data but are not caused by equipment faults or defects. On the basis of positions and durations, anomalies can be divided into point, contextual, and consecutive anomalies [6]. Fig. 4 shows an example of H2 content in 90 days, and three types of invalid anomalies are defined as follows: (i) Point anomalies refer to the observations that deviate significantly from the rest of the data, such as the point at the 68th day in Fig. 4. Fig. 4Open in figure viewerPowerPoint Different types of invalid anomalies of H2 content (ii) If an observation is anomalous in a specific period but normal in other periods, it is termed as a contextual anomaly [6]. A point at the 85th day is a typical contextual anomaly because a point is considered normal when it appears before the 80th day. In Fig. 4, contextual anomalies are more difficult to detect compared with point anomalies. (iii) Consecutive anomalies indicate that multiple observations are consecutively anomalous in a specific time interval. The points from the 27th to the 59th day are obviously continuous anomalies (Fig. 4). 2.5 Key data assumptions The establishment of anomaly detection methods relies on the key assumptions of normal and anomalous data. According to the analyses, we summarise the key assumptions of condition monitoring data. (i) Normal data satisfy the assumption of stationarity, thereby enabling normal observations to be concentrated in a dense space or cluster. However, concept drift, which leads to multiple normal data patterns, represents a violation of the stationarity hypothesis. (ii) The point anomalies are sparsely isolated and do not belong to any clusters of normal data. A contextual anomalous observation belongs to a certain cluster, but the clustering category is different from adjacent observations. (iii) The valid anomalies satisfy the assumptions of 'concurrency and consecution'. 3 Data preprocessing The time series of condition monitoring data is represented by the design matrix as follows: (1)where X m×n is an m × n matrix. Each row of the matrix represents an observation (also called a point), and each column represents a condition variable (attribute). The vector T = [t1, t2, …, ti, …, tm]T is the measurement of time in each observation and the contextual attribute of the time series [6]. In general, observations are equally spaced in time. X m×n becomes a larger matrix with time, and most anomaly detection technologies often do not work well due to the following reasons: (i) More drifts may occur with the length of the time series, resulting in more data patterns and contextual anomalies that increase the difficulty of anomaly detection. (ii) Most existing technologies are designed for detecting anomalous observations. By contrast, fault diagnosis technologies are concerned about the anomalous condition variables rather than the observations. To solve these problems, we present a data preprocessing method in the following subsections. 3.1 Data segmentation The original time series is always interrupted and divided into different data segments due to equipment outages, sensor replacements, or malfunctions. Therefore, the data pattern may change during the interruption time. In accordance with the positions of data interruptions, a design matrix X can be reasonably split into K-segments and expressed by (2)where k is the sequence number of each segment. In each segment, the number of data patterns is reduced, and some contextual anomalies are transformed into point anomalies. 3.2 Data normalisation The variables with larger magnitudes compared with others are likely to dominate the computations. Thus, the z-score method is applied to normalise the data of each variable. The normalisation of the j th variable in X (k) is shown in the following equation: (3)where is the i th normalised point of the j th variable; Mean[x·,j (k)] and Std[x·, j (k)] are the mean and standard deviation values of the j th variable, respectively. m (k) is the number of samples in the k th data segment. 3.3 Construction of auxiliary feature vector To identify valid and invalid anomalies accurately and establish the pattern recognition rules of different data patterns, we construct the auxiliary feature vectors of each variable based on two new features, which are defined as the backward differential change rate γ and the warning value sign w. With the j th variable as an example, the normalised auxiliary feature vector is shown in the following equation: (4)Two features are introduced. (i) Backward differential change rate γ: In normal conditions, the fluctuations between adjacent points are small. However, they increase dramatically between anomalies and normal neighbour points. The normalised γ between the i th and i + 1th observations of the j th variable can be quantified using the following equation: (5)where mk is the number of observations in X (k). (ii) Warning value sign w: The alarm is triggered when the measured value exceeds the warning value, which is a concern for equipment managers. The warning value sign w is constructed using the following equation: (6)where xw,j is the warning value of the j th variable. 4 Anomaly detection based on DBSCAN The key idea of DBSCAN is that for each observation of a cluster, the neighbourhood of a given radius, defined as Eps, must contain at least MinPts observations. Although Eps and MinPts have a significant influence on the clustering results, the optimal values for Eps and MinPts are difficult to select in an unsupervised setting because of the cross-validation method that worked for supervised learning does not apply here. In this section, a heuristic method based on the 'number of clusters–Eps' curve is designed and proposed for Ep s selection. 4.1 Parameter selection by the sorted k-dist graph Eps and MinPts are commonly selected by the sorted k-dist graph [25]. Let k be equal to MinPts, and Euclidean distance is used to quantify the Eps. Parameter selection is a two-step procedure. 4.1.1 k (MinPts) As a rule of thumb, k can be selected in a wide range, and a minimum k can be derived from the number of variables n as (k ≥ 3)∩(k ≥ n + 1) [33, 34]. In this study, k is set to an integer multiple of five and satisfies n + 1 < k ≤ n + 6. For example, we set k to 10 for a dissolved gas dataset X 2251×6. Fig. 5 shows the sorted 10-dist graph. Fig. 5Open in figure viewerPowerPoint Sorted 10-dist graph of dataset X 2251×6 4.1.2 Eps Eps is determined by the 'valley' of the sorted k-dist graph. A 'valley' refers to a threshold at which a sharp change appears beside the k-dist graph, and the ordinate value of the 'valley' is Eps. However, the appropriate 'valley' is difficult to identify based on visual inspection. In Fig. 5, the curve increased rapidly when the horizontal axis value is greater than 2100, and more than one possible 'valley' appeared (as indicated by the red circles). Therefore, a reliable method for Eps selection must be established. 4.2 Eps selection using the 'number of clusters–Eps' curve The 'number of clusters–Eps' curve is designed and proposed for Eps selection. The key steps are as follows: (i) Eps (s0) is defined as the lower threshold of Eps. When Eps = 0, most of the observations are treated as anomalies because the radius threshold is too small to contain enough neighbour points. With the increase in Eps, numerous split clusters are generated, resulting in a dramatic increase in the number of clusters. Let N0 be the number of clusters when Eps = 0. Eps (s0) is determined when the number of clusters starts to increase to a value N1 larger than N0. (ii) Epsmax is defined as the upper threshold of Eps and selected by a possible 'valley' with a large value in the sorted k-dist graph, such as the right-most point in the red circle in Fig. 5. (iii) A vector {Eps (s0)|i=1, …, Eps (i), …, Epsmax |i=10} of ten logarithmically spaced points between Eps (s0) and Epsmax is generated. The data in X m×n are inputted into ten DBSCAN models with respect to (Eps (i) and MinPts). The number of clusters N (i) between different Eps (i) is obtained to plot the 'number of clusters–Eps' curve. After testing multiple manuals and real datasets, the common shape of the curve is shown in Fig. 6, which consists of four segments. (i) In Segment 1, the generation of numerous split clusters results in a dramatic rise in the curve. The global maximum Nmax is reached when Eps = Eps (s1). Fig. 6Open in figure viewerPowerPoint 'Number of clusters–Eps' curve (ii) In Segment 2, the split clusters begin to merge when Eps (i) is further increased, and the curve shows a rapid decline. (iii) In Segment 3, the point (Eps (s2), Nb) is the first local minimum point on the right side of the point (Eps (s1), Nmax) in the curve. Sometimes, it is the left-most point of the first horizontal line on the right side of (Eps (s1), Nmax). (iv) In Segment 4, the number of clusters is reduced consistently, indicating that most of the observations are in a big cluster, and only a few anomalies remain separated. When Eps = Eps (s2), the normal observations of different data patterns are divided into different clusters, and most anomalous observations are determined by category 0. When Eps < Eps (s2), more observations are wrongly identified as anomalies, and data patterns are split. Some anomalies and different data patterns are merged excessively when Eps > Eps (s2). Ideally, Eps (s2) is the optimal 'valley' in the sorted k-dist graph. According to the 'number of clusters–Eps' curve, it can effectively prevent an overly large or overly small selection of Eps results compared with the visual inspection in the sorted k-dist graph. The detailed procedures of data clustering and Eps selection are described in Algorithm 1 (see Fig. 7). Fig. 7Open in figure viewerPowerPoint Algorithm 1: Data clustering and Eps selection 4.3 Procedure of anomaly detection The anomaly detection framework and the pseudo-codes of the proposed method are presented in Algorithm 2 (see Fig. 8). (i) To obtain the pattern recognition rules of different data patterns, we present a self-learning process. First, in a relatively clean data segment X, some original observations are randomly corrupted and labelled by different normal patterns and valid and invalid anomalies to generate a manual anomaly segment X *. Second, the auxiliary feature vectors of each variable in X * are constructed after data normalisation. Then, they are clustered using Algorithm 1 (Fig. 7), respectively. Finally, the pattern recognition rules of normal data and valid and invalid anomalies are summarised in accordance with the observations and the corresponding clustering results. It should be noted that if pattern recognition rules are obtained, this process should be omitted. Fig. 8Open in figure viewerPowerPoint Algorithm 2: Procedure of anomaly detection (ii) The anomaly detection of each new and unlabelled data segment of X is performed. The data patterns of each observation are determined according to pattern recognition rules. 5 Application example In this section, the proposed method is tested using dissolved gases in the oil of a transformer (220/121/10.5 kV, 150 MVA). All experiments are conducted on a 64-bit computer with an Intel Core i7 CPU of 2.8-GHz. The proposed methods are