Adaptive boosting in ensembles for outlier detection: Base learner selection and fusion via local domain competence

ETRI JournalVolume 42, Issue 6 p. 886-898 ORIGINAL ARTICLEFree Access Adaptive boosting in ensembles for outlier detection: Base learner selection and fusion via local domain competence Joash Kiprotich Bii, Corresponding Author kbjoash@gmail.com orcid.org/0000-0003-1969-1019 Department of Computing, School of Computing and Information Technology, Jomo Kenyatta University of Agriculture and Technology, Nairobi, Kenya Correspondence Joash Kiprotich Bii, Department of Computing, School of Computing and Information Technology, Jomo Kenyatta University of Agriculture and Technology, Nairobi, Kenya. Email: kbjoash@gmail.comSearch for more papers by this authorRichard Rimiru, Department of Computing, School of Computing and Information Technology, Jomo Kenyatta University of Agriculture and Technology, Nairobi, KenyaSearch for more papers by this authorRonald Waweru Mwangi, Department of Computing, School of Computing and Information Technology, Jomo Kenyatta University of Agriculture and Technology, Nairobi, KenyaSearch for more papers by this author Joash Kiprotich Bii, Corresponding Author kbjoash@gmail.com orcid.org/0000-0003-1969-1019 Department of Computing, School of Computing and Information Technology, Jomo Kenyatta University of Agriculture and Technology, Nairobi, Kenya Correspondence Joash Kiprotich Bii, Department of Computing, School of Computing and Information Technology, Jomo Kenyatta University of Agriculture and Technology, Nairobi, Kenya. Email: kbjoash@gmail.comSearch for more papers by this authorRichard Rimiru, Department of Computing, School of Computing and Information Technology, Jomo Kenyatta University of Agriculture and Technology, Nairobi, KenyaSearch for more papers by this authorRonald Waweru Mwangi, Department of Computing, School of Computing and Information Technology, Jomo Kenyatta University of Agriculture and Technology, Nairobi, KenyaSearch for more papers by this author First published: 30 March 2020 https://doi.org/10.4218/etrij.2019-0205Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract Unusual data patterns or outliers can be generated because of human errors, incorrect measurements, or malicious activities. Detecting outliers is a difficult task that requires complex ensembles. An ideal outlier detection ensemble should consider the strengths of individual base detectors while carefully combining their outputs to create a strong overall ensemble and achieve unbiased accuracy with minimal variance. Selecting and combining the outputs of dissimilar base learners is a challenging task. This paper proposes a model that utilizes heterogeneous base learners. It adaptively boosts the outcomes of preceding learners in the first phase by assigning weights and identifying high-performing learners based on their local domains, and then carefully fuses their outcomes in the second phase to improve overall accuracy. Experimental results from 10 benchmark datasets are used to train and test the proposed model. To investigate its accuracy in terms of separating outliers from inliers, the proposed model is tested and evaluated using accuracy metrics. The analyzed data are presented as crosstabs and percentages, followed by a descriptive method for synthesis and interpretation. 1 INTRODUCTION The task of differentiating outlier data points from normal points given a particular definition of anomalous behavior is referred to as anomaly or outlier detection [1. Outlier detection methods are useful in numerous applications, including the detection of defective mechanical parts [2, intrusions in wireless sensor networks [3, credit card fraud [4, and gene separation [5. Outlier detection techniques have evolved over time to form both supervised and unsupervised techniques. For most outlier datasets, ground truth data are lacking; hence, most outlier detection methods are unsupervised [6. However, in the unsupervised category, there have been reports of methods producing too many false positives and negatives [7. To improve the detection accuracy of unsupervised methods, the use of ensembles or groups of learners and detectors to solve the outlier mining problem has become a norm [8-10. Therefore, many novel outlier ensemble methods have been developed [10-13. An ensemble is a collection of trained classifier models whose predictions are combined to reach a final decision [14. Ensemble learning is a machine learning concept in which multiple learners are trained to solve the same problem. In contrast to ordinary machine learning approaches, which attempt to learn a single hypothesis from training data, ensemble methods attempt to construct a set of hypotheses and combine them for detection [15. An ensemble contains a number of learners, which are referred to as base learners. The generalization ability of an ensemble is typically much stronger than that of its individual base learners. In essence, ensemble learning is advantageous because it is able to boost weak learners, which are slightly better than random guesses, to create strong learners that can make more accurate predictions [16. Problem statement: An ideal outlier detection ensemble should consider the strengths of individual base learners while carefully combining their outputs to create a strong learner to achieve unbiased overall detection accuracy with minimal variance. Existing outlier detection ensembles utilize either parallel and/or sequential combination structures to fuse multiple detectors (weak base learners) in an attempt to improve overall detection accuracy by deriving a combined result (majority vote) from the detectors. Parallel combination structures are designed with the goal of reducing variance while serial combination structures are designed with the goal of reducing bias [17. However, trusting the results from all weak learners may deteriorate the overall performance of an ensemble because some learners can provide inaccurate results depending on data types and the underlying rules of a learner, particularly in the context of outliers with a lack of ground truth data. Outlier detection ensembles that mark safe instances as anomalous or that mark anomalous instances as safe can make systems unsafe, untrustworthy, or redundant. In certain applications, such as medical diagnosis, misclassifications can have catastrophic and irreparable consequences [18. Therefore, it is necessary to study which detectors should be used as base learners, how they should be combined, and which types of data they are suitable for. To address these problems, this paper presents an adaptive boosting framework for heterogeneous ensembles for outlier detection (ADAHO). The advantage of ADAHO lies in its ability to integrate various high-performance base detectors by utilizing a hybrid structure to minimize bias and variance while improving overall detection accuracy. The main contributions of this work can be summarized as follows: ADAHO selects high-performance or optimal heterogeneous base learners by assessing their capabilities in their local domains or areas of expertise. This is because every outlier detection technique performs best within a specific domain within the entire problem space [19. ADAHO adaptively boosts the outputs of preceding base learners during the first phase of training and carefully combines high-performance base learners in the second phase to generate more accurate predictions compared to simply averaging the outcomes of all base learners. ADAHO is able to fuse heterogeneous detectors that produce different types of outlier outputs at different scales into a unified function for identifying outliers. The remainder of the paper is organized as follows. A literature review and background information on existing detection ensembles are presented in Sections 2 and 3, respectively. Our design methodology is presented in Section 4. Experiments and empirical results are discussed in Sections 5 and 6, respectively. Finally, in Section 7, conclusions are drawn and directions for future research are discussed. 2 LITERATURE REVIEW 2.1 Outlier detection Outlier detection refers to the task of identifying anomalous patterns in a given dataset according to a particular definition of anomalous behavior [1. Outlier detection methods are specific solutions to outlier detection problems. The output of an outlier detection method could be a labeled pattern. For example, outlier labels or normal labels, or scores assigned based on the degree to which a pattern is considered an outlier. Outliers are detected by analyzing events, where each event is designated by a data instance. Each data instance has features (ie, attributes) that describe it [3. A data instance with a single feature is a univariate instance and a data instance with multiple features is a multivariate instance [1. Features are crucial for distinguishing normal behavior from anomalous behavior. An outlier was defined by Hawkins as an observation that deviates far enough from other observations to arouse suspicion that it was generated by a different mechanism [20. In most literature, the authors have described outliers as observations that appear to be inconsistent with the remainder of dataset, which is the main problem associated with handling outliers. Outlier detection methods attempt to solve this problem using various approaches, such as statistical and probabilistic knowledge, distance and similarity-dissimilarity functions, metrics for accuracy when dealing with labeled data, association rules, properties of patterns, and other domain features. 2.2 Key aspects of the outlier detection problem In outlier detection, the fundamental problem lies in discerning the unknown data space. A straightforward approach to detecting outliers is to find unusual points by computing a measure of normality or similarity to their neighboring points. However, several factors make this task very challenging. For example, Chandola's survey described seven factors [1. (a) Defining a normal region (ie, one that covers every possible normal behavior) is very difficult. (b) Normal regions continue shifting in many cases and an existing understanding of a normal region may not hold true in the future. (c) The margin between a normal region and outlier region is almost always uncertain; this means that an outlying point that lies near the margin could be a normal point or vice versa. (d) The exact definition of an outlier differs for different application areas. (e) Data classes or labels for training or testing are not always available. (f) In cases where outliers are generated by malicious acts, malicious agents can adjust themselves to appear normal, thereby making the task of identifying normal behavior much more difficult. (g) Noise is present in most data, and noise can mimic outliers, making it difficult to separate and eliminate real outliers. In response to these challenges, outlier detection techniques have attempted to simplify the problem by adding various types of information, including the definition or nature of data and the types of outliers to be detected. Therefore, the application area dictates the methods used to solve the outlier detection problem; this means that there are a number of different methods used in various areas, such as data mining, statistics, and machine learning. For example, by narrowing down Hawkins's concept [20, two major outlier detection techniques can be derived: distance-based techniques and density-based techniques. Distance-based techniques find data points that lie far from their nearest neighbors and density-based techniques find data points that reside in lower-density regions compared to their nearest neighbors. Some researchers have also studied other types of outlier detection algorithms, such as supervised and unsupervised learning algorithms. Supervised learning algorithms detect outliers using labeled data; this means that records are classified as either "normal" or "outlier." Unsupervised learning algorithms use unlabeled data; thus, the outliers (and inliers) are unknown [21. 2.3 Domain significance and data associations Knowing the associations among data instances is very significant for the outlier detection problem. Therefore, outlier detection methods can be classified based on their use of global data associations, meaning they generate decisions using all data points [10, or local domains, meaning decisions are made based on a few locally selected points [22. In both cases, applications are domain specific. For example, in high-dimensional data spaces, where outliers lie far from the rest of the data, global associations among all data points are very useful [11. However, such global associations may underestimate the probability of other outliers lying in local domains [23. Furthermore, drawing the conclusion that a given data point picked from a random set of samples should lie in a similar region to all other data points could be an inaccurate assumption. Based on this analysis, methods such as the local outlier probability (LoOP) [24, local outlier factor (LOF) [25, and Gloss [23 have been developed. Until the development of recent solutions presented in [13 and [26, the significance of local data regions for classifier selection and fusion in outlier detection problems had rarely been considered. However, it is noteworthy that both of these works utilized homogeneous ensembles. In [13, methods for determining data locality and the regions of competence of base learners were mentioned based on either the k-nearest neighbors (k-NN) algorithm or data clustering. 2.4 Value of fusion In many cases, the nature of data is very unpredictable and base learners can make mistakes for different training instances. It is also the case that some base learners will excel at discerning certain subspaces while others may perform poorly in those subspaces [27, but the overall accuracy of the former learners may be worse than that of the latter. In other words, a base learner can have a domain of expertise in a local domain while performing poorly when considering the entire feature space. To take advantage of each learner's domain of expertise, it is crucial to fuse them to reduce the total error [28. This strategy is preferable because it helps to overcome the deficiencies of individual detectors while improving overall detection accuracy. The authors of [16 proposed the concept of data locality, which was later improved in [29 to perform dynamic classifier selection in local spaces of training points. Techniques that dynamically select and fuse base classifiers have yielded better results compared to static techniques, such as those that simply take a vote based on all base classifier outputs. Voting with weights and voting without weights are the two most common methods for combining not only homogeneous, but also heterogeneous models [14. Using an optimal Bayes concept to filter high-performance learners from low-performance learners informed dynamic classifier selection in [30. This concept was extended into dynamic ensemble selection in [31, which introduced a second-layer fusion strategy using multiple base classifiers. However, in [31, heterogeneous learner fusion was not considered. In [26, the authors asserted that by using multiple base learners, stability can be improved compared to using only the best learner. However, in [13, a single learner or combination of learners was adopted depending on which configuration yielded better results. The concept of a "crowd of experts" is more helpful for diffusing biases compared to a single learner [26. Experimental results demonstrated that using dynamic ensemble selection or groups of learners is more stable as compared to the case when dynamic classifier selection is used [31. In [26, additional issues regarding ensemble formation were raised. First, existing fusion strategies are static or lack any learner selection procedures, which reduces the value of fusion because individual base learners may not fully identify all outliers [32. Second, the performance of a base learner has always been assessed based on an entire training dataset instead of focusing on the data domains related to a particular testing instance. When all data dimensions are taken into consideration, few outliers can be identified [26. In most outlier datasets, the majority of the data are normal, meaning most outliers can be identified by looking at small subsets of features [33. The concept of detector evaluation based on the locality of data instead of global evaluation, which was proposed in [26, is very important. A fusion strategy such as weighted averaging, which utilizes the Pearson correlations between learner outcomes and actual or simulated labels on all training data points [34, is desirable. Additional techniques that support the concept of outlier mining from local data domains include those proposed in [13, 23, 24. Outlier detection using heterogeneous algorithms yields heterogeneous scores, which cannot be directly combined because different methods produce different outlier scales [33. For example, k-NN yields scales that are smaller than those of the cumulative neighborhood technique. Different methods also yield different types of vectors, such as binary-valued or real-valued scores, implying that score unifying processes are required for score fusion. 2.5 Bias-variance tradeoff Generalization errors in classification can be examined using the bias-variance tradeoff framework [12. When labels are provided with data samples (ie, supervised learning), this tradeoff is elaborately defined as the quantification of bias and variance. When labels are not provided, unsupervised methods are used; hence, data labels are treated as unknown entities. Outlier detection techniques can be viewed as binary problems with inliers (majority class) and outliers (minority class) by converting detector outlier scores into class labels [10. Additionally, within the same reference, the error of a base learner can be divided into two parts, namely the reducible error and the irreducible error caused by noise in the data. It is possible to minimize the reducible error to increase detector accuracy by considering two types of errors, namely errors from squared bias and errors from variance. In an effort to reduce both types of errors, compromises (ie, tradeoffs) must be made between the two types. The difference between an expected output and actual unseen value over the training samples defines bias, whereas the difference between the outcome of a detector over a single training set and the expected outcome over the entire data space defines variance. A base learner with low bias is flexible for fitting data properly, but will fit every training set in a different way, resulting in high variance. In contrast, an inflexible base learner with less variance will yield significant bias. It is the objective of an outlier detection ensemble to reduce the errors caused by both bias and variance to reduce the overall effects of generalization errors. The goal of this study was to improve accuracy as much as possible by reducing both bias and variance and improving the selective outlier ensemble approach by introducing the concept of heterogeneous base detectors into an ensemble. Our approach first identifies high-performance base learners based on their strengths in the domains in close proximity to their training cases (ie, local domains), then utilizes adaptive boosting, where incorrectly classified samples become inputs for subsequent base learners, leading to a final output based on a fusion function. 3 ENSEMBLE FORMATION Ensemble formation involves three key steps [10. The first is the creation step, in which a number of base learners are created. The second is the selection step, in which high-performance learners are picked from the pool of created learners. The third is the fusion step, in which a fusion strategy is used to fuse base model outcomes to create a strong learner. Implementing a good fusion strategy during the third step is very critical because there is a potential threat of weak learners deteriorating the performance of other models in the ensemble, rather than improving their performance [11. Some outlier ensemble formations are designed in parallel order, such as feature bagging [35 or dynamic combination of detector scores for outlier ensembles (DCSO) [26. Some are designed with sequential ordering, such as SELECT [11. Finally, some formations use hybrid ordering, such as BORE [36 and XGBOD [12. For classification, ensemble methods include bagging [37, boosting [38, and stacking [39. These ensemble methods can be biased toward some specific characteristics of a dataset because their training procedures utilize only a single base learner. To introduce diversity and eliminate biases, different types of base learners can be introduced into an ensemble to form a heterogeneous ensemble. This allows different data characteristics to be learned by a diverse set of base learners. However, when all base learner outcomes are fused to generate a final result without considering how well each base learner performs, it can lead to poor overall results or generally weak algorithms [10, 11 because poorly performing learners drag down scores of high-performance learners (eg, when averaging is used as a fusion method based on the mean of all scores [40). Furthermore, when maximization is used as a fusion strategy, it can yield unstable results [9. However, it is possible to use either the average of maximization results or the maximum of average results because these two-stage fusion methods can potentially improve an algorithm's overall performance and stability [26. In an attempt to generate base learners sequentially, the SELECT [11 and CARE [10 algorithms were developed. These methods select high-performance base learners while disregarding low-performance base learners during fusion. SELECT produces a simulated label or ground truth by averaging base learner outlier probabilities, each of which may have biases in different directions, then disregarding low-performance base learners or selecting base learners to retain based on the weighted Pearson correlations between simulated labels and base learner outlier probabilities [11. This technique yields promising results for temporal graphs and multi-dimensional data. Other approaches, such as the Full ensemble [33, Div-E [41, and ULARA [42, were not designed around the concept of local domain competency or bias variance reduction, unlike SELECT. In the Full ensemble, all base detector outcomes are used, even those with strong biases, which deteriorates the final detector. Although ULARA computes the relative weights of base detectors, it still includes those with strong biases. Div-E hand picks diverse base detectors, including those with strong biases, based on the concept of maximizing diversity, which generally reduces accuracy. Additional fusion techniques, such as majority voting, weighted majority voting, summation, product, maximum, minimum, fuzzy integral, Dempster-Shafer, and decision templates [28, have also been proposed. For voting, the value with the most votes, highest mean ranking, or highest mean probability is selected [14. For weighted voting, each base detector is assigned a coefficient or weight according to its performance. It is important to note that most fusion approaches utilize the entire data space for decision making by calculating an overall average of learner scores [26. We refer to such approaches as (G) fusion techniques. Some of these techniques can be summarized as follows. (i) In global threshold summation (G_TS), all learner outputs below a certain threshold are discarded and the sum of the remaining learner outputs is calculated. (ii) In global maximization (G_Max), the maximum outlier score over all base learners is calculated. (iii) In global averaging (G_AVG), the average of the scores of all base learners is taken as the final outlier score. (iv) In the global average of maximums method (G_AoM), base learners are divided into groups and the maximum score for each subgroup is taken as that subgroup's score. The final score is obtained by calculating the average of all subgroup scores. (v) In the global maximum of averages method (G_MoV), base learners are divided into groups and the average score for each subgroup is taken as the subgroup score. The final score is obtained by choosing the maximum value among all subgroup scores. (vi) In global weighted averaging (G_WA), a simulated training label or ground truth label is generated by calculating the average of all base learner outputs. Each base learner is then assigned a weight by calculating the Pearson correlation between its training output and the simulated label. After the weights are obtained, the final score is calculated as the average of the weights of all base learner scores. (1) The Pearson correlation in (1) measures the similarity between two vectors s and t (n denotes vector length), where and are the averages of the vectors. 4 METHODOLOGY The proposed method utilizes the principles of prediction based on historical outcomes. Our research design uses experiments, derivations, and reasoning as tools for understanding a problem or behavior. In other words, we implemented an experimental research design. This design involves the manipulation of variables and predictions based on past observations and data analysis to identify outliers. This paper presents an adaptive boosting technique for outlier detection called ADAHO. This method combines a set of heterogeneous weak learners to obtain an optimized composite model that provides more accurate and reliable predictions compared to using a single model. It uses weighted versions of a single training dataset instead of random subsampling or bootstrapping. Therefore, the training dataset does not need to be as large as that required by many other methods. Successive weak classifiers are trained using reweighted versions of the training data, where the weights depend on the accuracy of the previous classifiers. Error rates are computed in the context of an instance's local neighbors, rather than a global training set. Testing instances focus on local domains or regions within a training dataset. At every iteration, the training instances are weighted according to the misclassifications (errors) of previous classifiers. This allows weak learners to focus on patterns that were poorly classified by previous classifiers. The final result is a fusion of carefully selected results from individual learners. The main steps in the ADAHO algorithm are presented in Algorithm 1. Step 1: Different detection algorithms (base detectors) are selected with unique measures to score individual vectors. Step 2: A subset of the detector results is selected using a selection strategy. Step 3: The selected results are fused using different fusion methods to create intermediate aggregate outcomes. Step 4: Subsets of the outcomes from Step 3 are selected. Step 5: The selected subset of outcomes is fused. 4.1 Base learners Heterogeneous base classifiers are selected because ensemble methods are very effective when base classifiers of dissimilar types are used [10, 43. Based on the differences between classifiers, the unique properties in data can be discovered or learned. When base learners of the same type (homogeneous) are used, the advantages of learner fusion are lost, unless different data subsamples, parameters, or features are used for training each classifier [27, 43. The proposed algorithm uses different base learners (heterogeneous) to construct a group of models to improve efficiency. Distance- and density-based methods are well-known unsupervised methods for outlier detection. Distance-based methods [44, 45 attempt to identify global outliers that lie far from the rest of the data. Such methods typically use the k-nearest neighbors (kNN) algorithm to detect outliers. Density-based methods [25 attempt to find local outliers that lie in less dense regions compared to their kNN. This paper focuses on unsupervised outlier detection. Such techniques assign scores to individual data points, facilitating the ranking of such points based on their probability of being outliers. Motivated by the critical importance of data locality and dynamic learner fusion in DCSO [13, 26 and the concept of heterogeneous detector formations in SELECT [11, we adopted two distance-based algorithms to detect global outliers and two density-based algorithms to detect local outliers. (a) The local distance-based outlier factor (LDOF) [45 calculates the distance from a point to its kNN and compares that value to the average distances between nearest neighbors. (b) The traditional kNN algorithm uses the kNN distances of individual instances is used as outlier scores. (c) The LOF [25 calculates the local deviation of a given data point relative to its neighbors. (d) Finally, LoOP [46 is a local-density-based outlier detection method that provides outlier scores in the range of [0,1]. Formally, given a dataset of size d features, we divide the dataset into training and testing sets as to signify training data with n data points and to signify test data with m points. Next, a pool of heterogen