Scalable feature selection using ReliefF aided by locality‐sensitive hashing

International Journal of Intelligent SystemsVolume 36, Issue 11 p. 6161-6179 RESEARCH ARTICLEOpen Access Scalable feature selection using ReliefF aided by locality-sensitive hashing Carlos Eiras-Franco, Corresponding Author carlos.eiras.franco@udc.es orcid.org/0000-0001-6322-7593 Research Center on Information and Communication Technologies (CITIC), Universidade da Coruña, A Coruña, Spain Correspondence Carlos Eiras-Franco, Research Center on Information and Communication Technologies (CITIC), Universidade da Coruña, Edificio Área Científica D1.02, Campus de Elviña, 15008 A Coruña, Spain. Email: carlos.eiras.franco@udc.esSearch for more papers by this authorBertha Guijarro-Berdiñas, orcid.org/0000-0001-8901-5441 Research Center on Information and Communication Technologies (CITIC), Universidade da Coruña, A Coruña, SpainSearch for more papers by this authorAmparo Alonso-Betanzos, orcid.org/0000-0003-0950-0012 Research Center on Information and Communication Technologies (CITIC), Universidade da Coruña, A Coruña, SpainSearch for more papers by this authorAntonio Bahamonde, orcid.org/0000-0002-2188-9035 Department of Computer Science, Universidad de Oviedo, Asturias, SpainSearch for more papers by this author Carlos Eiras-Franco, Corresponding Author carlos.eiras.franco@udc.es orcid.org/0000-0001-6322-7593 Research Center on Information and Communication Technologies (CITIC), Universidade da Coruña, A Coruña, Spain Correspondence Carlos Eiras-Franco, Research Center on Information and Communication Technologies (CITIC), Universidade da Coruña, Edificio Área Científica D1.02, Campus de Elviña, 15008 A Coruña, Spain. Email: carlos.eiras.franco@udc.esSearch for more papers by this authorBertha Guijarro-Berdiñas, orcid.org/0000-0001-8901-5441 Research Center on Information and Communication Technologies (CITIC), Universidade da Coruña, A Coruña, SpainSearch for more papers by this authorAmparo Alonso-Betanzos, orcid.org/0000-0003-0950-0012 Research Center on Information and Communication Technologies (CITIC), Universidade da Coruña, A Coruña, SpainSearch for more papers by this authorAntonio Bahamonde, orcid.org/0000-0002-2188-9035 Department of Computer Science, Universidad de Oviedo, Asturias, SpainSearch for more papers by this author First published: 18 June 2021 https://doi.org/10.1002/int.22546AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract Feature selection algorithms, such as ReliefF, are very important for processing high-dimensionality data sets. However, widespread use of popular and effective such algorithms is limited by their computational cost. We describe an adaptation of the ReliefF algorithm that simplifies the costliest of its step by approximating the nearest neighbor graph using locality-sensitive hashing (LSH). The resulting ReliefF-LSH algorithm can process data sets that are too large for the original ReliefF, a capability further enhanced by distributed implementation in Apache Spark. Furthermore, ReliefF-LSH obtains better results and is more generally applicable than currently available alternatives to the original ReliefF, as it can handle regression and multiclass data sets. The fact that it does not require any additional hyperparameters with respect to ReliefF also avoids costly tuning. A set of experiments demonstrates the validity of this new approach and confirms its good scalability. 1 INTRODUCTION The greatly increased popularity of data science in recent years is partly due to the accumulation of enormous volumes of data and that continue to be produced daily. So-called big data1 is, nonetheless, a double-edged sword, since the promise of greater insight is counteracted by the inherent difficulty of processing vast amounts of data, rendering many machine learning algorithms unusable. Data sets may contain many samples, many variables per sample, or both. For data sets with many variables, dimensionality reduction is advisable to improve the performance of learning methods.2 Dimensionality reduction techniques can be classified as feature extraction or feature selection techniques. In feature extraction, the number of features in a data set is reduced by creating new features from original features that are then discarded, while in feature selection, irrelevant or redundant variables are discarded to obtain a reduced subset of input variables that still accurately describes the problem.2 Reducing the number of variables enhances the comprehensibility of the data set and the fit of data by learning methods. Another important way to tackle big data is distributed computing. Developed as a software solution are distributed processing platforms based on using clusters of several desktop computers to process enormous quantities of data. This approach started gaining popularity with Google's unveiling of the MapReduce programming paradigm in 2008.3 Since then several open-source implementations have been described that follow this paradigm, the most popular being Apache Hadoop4 and, launched a few years later, Apache Spark.5 Their success has given rise to machine learning libraries, such as Mahout6 for Hadoop and MLLib7 for Spark, not to mention the re-implementation of popular algorithms—including some feature selection algorithms—to leverage distributed computing. Nonetheless, sometimes the computational complexity of an algorithm cannot be reduced or the volume of data is such that even distributed implementation requires lengthy execution times. A possible solution is for data scientists to use techniques that obtain an approximation of the exact (but computationally more costly) model, which, despite being approximate, may be comparable to the exact model in terms of accuracy. We describe an approximation of the ReliefF feature selection algorithm that addresses its main limitation—a high computational cost. While the effectiveness of ReliefF has been amply demonstrated, as will be shown in Section 2, its use for large-scale data sets has been hindered by the great computational demand. Accounting for most of ReliefF's computational load is its use of the nearest neighbor graph. To address this crucial limitation, we used a locality-sensitive hashing (LSH) algorithm to build an approximate, rather than an exact, graph. This approach, which has yielded encouraging results in preliminary tests,8 is now fully developed and compared with the state of the art. The contributions of this article are as follows: 1. Our algorithm can accurately approximate the work done by ReliefF for a fraction of the computational cost. 2. Our implementation in Apache Spark leverages distributed computing to process data sets beyond the reach of ReliefF. Distributed computation can also be used to speed up the processing of data sets of any size. 3. Like the original ReliefF, and unlike current alternatives for approximating the nearest neighbor graph, our algorithm can handle binary, multiclass, and regression data sets. 4. Since no new hyperparameters are introduced, our method does not require manual hyperparameter tuning. 2 RELATED WORK The original Relief algorithm led to the development of a family of Relief-based feature selection algorithms, including ReliefF. Relief is a supervised feature ranking algorithm that estimates the relevance of features and ranks them for classification purposes9 by establishing a relevance threshold above which only important features are retained. Weights are assigned to each attribute that help discerns elements that are very close together. In the case of classification data sets, for each example, Relief searches for the nearest elements within the same class (nearest hit) and within a different class (nearest miss) and updates the weight of each attribute A in proportion to the difference between the example value and the nearest hit and nearest miss values. The resulting weight W of attribute A can be interpreted in terms of probability. W approximates the following difference of conditional probabilities10: W [ A ] = P ( different value of A ∣ different class ) − P ( different value of A ∣ same class ) . (1) The good results obtained using Relief encouraged further research that yielded numerous extensions capable of dealing with multiclass problems and incomplete or noisy examples.10 ReliefF is one such extension that has become even more popular than the original algorithm. Implementations of ReliefF are present in many machine learning software libraries. Further specializations of the algorithm were later introduced11 that enabled it to tackle regression problems12 and multilabel data sets,13, 14 and that factored in the cost of obtaining attributes.15 While ReliefF has been thoroughly tested16, 17 and has been shown to be useful in a great variety of problems,18-23 its great computational complexity makes its use on large-scale data sets prohibitively costly.24 While modifications to increase ReliefF's ability to handle large data sets have been developed that use distributed implementations,25, 26 sampling,27 or random kd-trees to approximate the nearest neighbor graph,28 their effectiveness in handling very large-dimensional data sets is limited. Of the ReliefF alternatives, the most efficient is DiReliefF,26 which approximates ReliefF's attribute ranking by using a sampling scheme that greatly reduces the number of calculated neighbors. DiReliefF is also parallelizable, with its distributed implementation in Apache Spark simplifying the handling of large data sets. Like many other feature selection, information retrieval, data mining, and machine learning methods, ReliefF relies on the analysis of similarity graphs. The k-nearest neighbor graph (kNN-graph) is the most popular such graph used in machine learning. A kNN-graph is a directed graph constructed over a set with n elements D = { x 1 , x 2 , … , x n } , such that edges ( x i , x j ) indicate that x j is among the k more similar elements to x i according to some similarity measure σ ( x i , x j ) . The resulting graph is very useful, but is computationally very costly to build because n ( n − 1 ) ∕ 2 pairwise comparisons are required; the result is a computational complexity value of O ( n 2 ) . Some variations of the algorithm speed up calculations in specific conditions, for example, small dimensionality in the input space29 or use of certain similarity measures.30 However, only approximate solutions to dealing with high-dimensionality data in a reasonable time have been developed. One approach is to use generic similarity measures to replicate the exact graph as closely as possible while maintaining a low computational cost. Algorithms that obtain approximate graphs—as opposed to exact graphs—have been based on divide-and-conquer approaches31 and local search.32 However, LSH33, 34 is the approach that has shown most success to date. Although the optimal way to exploit this technique remains an open problem, LSH essentially builds data structures that enable an element in a data set to be searched for in sublinear time. LSH maps data of any size into a smaller space using a hashing function that maximizes the possibility of assigning the same value to elements that are similar. It has been used, for instance, selection35 and also has been successfully employed for kNN-graph construction33, 34 and for the related problem of kNN search.36 Solutions are based on isolating small clusters of similar elements, used to calculate small isolated subgraphs that are then merged to obtain the complete approximate graph. Each element is only compared to other elements that LSH deems to be similar, that is, comparisons are avoided with elements that are very distant in the input space and so not among the nearest neighbors. This reduction in the number of unnecessary pairwise comparisons is what brings the computational cost down. 2.1 Variable resolution LSH Variable resolution LSH (VRLSH)34, 37 is the most recent algorithm that uses the LSH approach. As shown in Algorithm 1 and as described in detail below, it takes iterative LSH steps to obtain an approximate kNN-graph. To compute the approximate kNN-graph, VRLSH first assigns one or several hash keys to each element x i in the data set D, using the selected LSH function (line 4). This function gives the same hash key to elements that are similar according to a specific similarity measure. A given, initially large, resolution level for similarity is used to assign the same hash key to very similar elements that are then grouped (line 5). Thanks to the properties of the hash function, these groups—or buckets—will contain elements that are very similar according to the similarity measure selected. An exact subgraph is constructed for each of the groups using the brute-force procedure, that is, computing each possible pairwise similarity. Since each element x i is assigned several keys, it can appear in various groups and, therefore, in various subgraphs; those elements can be used to merge the initially isolated subgraphs, yielding larger subgraphs (loop in line 6). The data set D is then simplified by removing elements that have been involved in a prespecified C M A X > k number of pairwise comparisons. Those steps are repeated using the simplified D, generating new subgraphs that are merged with the existing subgraphs until the result is a single graph containing all points and links to the nearest neighbors. After each iteration the search resolution is reduced, forcing LSH to create groups of elements that are slightly more dissimilar than in the previous iteration. As a final step (line 10 on), if the simplified data set has fewer than k elements, the process is stopped and, for every x i ∈ D, the nearest neighbors will be sought among the neighbors of the neighbors, or, if there are no outgoing links in the graph, among the neighbors of elements selected at random. An additional advantage of VRLSH is that the estimation procedure finds suitable values for the hyperparameters of the method. It does this by efficiently searching for resolution values and hash function hyperparameters that yield adequately sized buckets to start computing the graph (a process described in detail in Reference [34]). With this automated procedure, the user only needs to provide the data set and indicate the desired number of neighbors to be obtained in the graph. In avoiding costly test runs of the complete algorithm with different hyperparameter values, the total computational effort of using this method is effectively reduced. Moreover, although VRLSH automatically finds a suitable value for C M A X that offers a reasonable trade-off between execution time and accuracy, this value can be overridden by users to obtain, depending on their needs, either a faster or a more accurate approximation. 2.2 Limitations of kNN-graph computing algorithms for ReliefF The above-described algorithms are designed to obtain a single kNN-graph for the whole data set, while ReliefF requires a graph that links each element to its k nearest neighbors in each class in the data. Needed to approximate the graph required by ReliefF are nontrivial modifications, which depend on the structure of the graph-building algorithm. Those modifications can have a great impact on the performance and scalability of the resulting method. In the worst-case scenario, the building process needs to be repeated for the whole data set as many times as there are classes in the data, which, for multiclass data sets, results in a great computational overhead. To enhance VRLSH suitability for tackling this type of problem, we propose a profound modification that obtains the desired graph in a single pass over the data. 3 PROPOSED ALGORITHM Our aim was to obtain an algorithm that is capable of handling all the data set types that can be processed with the most popular implementations of ReliefF—including multiclass and regression data sets—and that yields an approximation of the ReliefF ranking output while requiring less computational effort.12 The computational cost of this algorithm predominantly comes from the process that obtains the nearest hits and misses for each element, which entails obtaining the kNN-graph. Our proposal to reduce the computational cost is, therefore, to substitute the exact kNN-graph required for ReliefF with an approximate kNN-graph computed with a new VRLSH-based method. For regression data sets, VRLSH does not need to be changed for ReliefF to use the resulting approximate kNN-graph, as no distinction needs to be made between hits and misses for regression data. For classification data sets, however, VRLSH has to be vastly changed for ReliefF to be able to rank attributes by weight. While the approximate graph calculated by VRLSH has a single list of neighbors for each element in the data set, for classification data sets ReliefF needs to distinguish between hits and misses. VRLSH therefore has to be extended to maintain, for each element in the data set, a list of neighbors for each class. VRLSH also simplifies the initial data set D after each iteration (see line 8 in Algorithm 1), a process that needs to be modified to accommodate classification problems. In particular, given element x i , VRLSH keeps c o u n t _ x i , which is a count of pairwise comparisons, to determine if x i needs to be removed from the data set (when c o u n t _ x i > C M A X ). The c o u n t _ x i scalar count has to be transformed to a vector of counts c o u n t _ x i j of pairwise comparisons with elements of each class c j . In this new scenario, x i is only removed from the data set when it has been involved in at least C M A X comparisons for every possible class, that is, c o u n t _ x i j > C M A X ∀ c j ∈ C where C is the set of possible classes. Moreover, depending on the distribution of the classes in the input space, some points may be distant from a given class c k and, for these to be involved in C M A X , pairwise comparison requires maintaining them in the data set for a large number of iterations. Those points will therefore accumulate a much larger number of pairwise comparisons than C M A X for some classes, that is, c o u n t _ x i j ≫ C M A X for some j ≠ k. This challenges the ability of the method to perform a low total number of pairwise calculations. We propose tackling this problem by modifying the bucketing step so that we avoid comparing points x i with points of class c j if c o u n t _ x i j > C M A X . To simplify the notation, we will say that point x i requests class c j if c o u n t _ x i j < C M A X , that is, it still needs to be compared to elements of class c j before it is removed. This new bucketing step requires adding an additional component h r for each hash. Considering the requested classes, h r effectively splits the buckets of similar elements originated by the LSH function. Hence, point x i of class c j given hash h by the LSH function will have an updated set of hashes H ′ computed as described in Algorithm 2. The first element of H ′ is always the hash ( h , c j ) , as described in line 1, intended to keep x i available for other elements that request class c j . H ′ is completed by one of two alternatives: if x i requests all possible classes, line 3 forces x i to be compared to any element in the same situation; otherwise line 7 adds a hash for every requested class to H ′. This process originates two types of buckets that have to be treated differently according to their characteristics. Since this has the effect of invalidating the bucketing step in VRLSH (corresponding to the loop in line 6 of Algorithm 1), an updated version (detailed in Algorithm 3) is required. Figure 1 summarizes the modified bucketing procedure, in which buckets containing elements given the same hash are handled according to h r , the modifier component added to their hash. The first option, shown in line 3 of Algorithm 2, processes buckets of elements requesting all classes by computing all possible pairwise comparisons between elements in the bucket, as in the original Algorithm 1. This guarantees that those elements are compared to all points deemed similar by the LSH function, quickly retrieving neighbors from every nearby class. The second option deals with buckets containing elements that either request a class or belong to a requested class. Lines 6 and 7 divide each bucket with hash ( h , h r ) into two sets: requesters, that is, points requesting h r , and targets, that is, points with class h r . The pairKNN function then measures the similarity of each target to each requester and builds the corresponding subgraph, as described in line 8. In doing this, no requester–requester or target–target comparisons are performed, which saves on a great number of operations, as does division of each bucket into smaller and more specific buckets by appending h r to hashes. Figure 1Open in figure viewerPowerPoint Proposed hashing and bucket-forming procedure. x i are, in this example, similar enough to be assigned the same hash value h and the hasher outputs only one hash per element. This hash, augmented with the h r value, informs the bucket-forming procedure. A different number of pairwise comparisons are performed for each type of bucket, as described in Algorithm 3 We refer to the resulting algorithm that integrates all these parts as ReliefF-LSH, for which a distributed implementation in the Apache Spark framework is available for download from https://github.com/eirasf/ReliefF-LSH. Although the ReliefF-LSH algorithm is designed to work with any family of LSH functions, in this study we implement and test it for a commonly used family, sensitive to Euclidean distance, which performs hashing through random projections onto one-dimensional lines.38 As shown in Section 5, satisfactory results are obtained with this family of functions. 3.1 Spatial and computational complexity Given a data set with n elements, each described with d variables, and a hasher that generates hashes of length l for each element t, the greatest memory demand on the algorithm is from storing the generated hashes. The proposed algorithm therefore requires memory in proportion to the product of n, l, and t, that is, O ( n l t ) . The memory requirement is managed by distributing calculations across various computing nodes. Furthermore, given that the number of variables d in the data set does not directly impact on memory requirements, for high-dimensionality data the space needed to store the hashes and the space occupied by the data set are comparable. The memory overhead is therefore smaller for high-dimensionality data sets, which is the focus of our approach. The computational complexity of the method cannot be analytically established, as the probabilistic nature of the hashing function (and, therefore, of the bucketing process) makes it difficult to establish an upper bound for the complexity of both the parameter tuning and bucketing procedures. Computational complexity also depends on the data set, so even an average-case study is not possible. Nonetheless, the experimental results reported in Section 5 highlight the efficiency of this algorithm in terms of computational complexity. 4 EXPERIMENTAL SETTINGS We report two sets of experiments performed to validate the effectiveness of ReliefF-LSH, whose performance is compared with those of ReliefF and of DiReliefF—the best alternative to ReliefF,26 as discussed in Section 2. We also compared results to those for ReliefF with an approximate kNN-graph precomputed using the most popular method for this task, namely, the Fast Library for Approximate Nearest Neighbors (FLANN),39 which relies on randomized kd-trees41 to build the graph. In this composite method, the algorithm, which we call ReliefF-FLANN, was run over the whole data set for each class and the resulting graphs were merged to obtain the graph required for ReliefF. In a first set of experiments, for real data sets we recorded the execution time required to compute exact weights using ReliefF and then compared that time with the times taken by ReliefF-LSH, DiReliefF, and ReliefF-FLANN. Since ReliefF-LSH, DiReliefF, and ReliefF-FLANN are all approximate methods, the accuracy of their rankings was determined by measuring the level of agreement of the approximate rankings with the ground-truth rankings calculated by ReliefF. In a second set of experiments, we determined the level of scalability of ReliefF-LSH, reporting the ReliefF-LSH runtimes when applied to the same problem with varying amounts of computational resources assigned to the task. 4.1 Equipment and data sets Each experiment was executed in a computer cluster consisting of machines with 12 computing cores. The cluster nodes are described in Table 1. We ran ReliefF-LSH on Apache Spark version 2.4.0 on Hadoop 3.0.0-6.1.0, and DiReliefF on Apache Spark version 1.6.1 on Hadoop 2.7.1.2.4.2.0-258, because the available implementation required it. CentOS Linux release 7.4.1708 is the operating system used by those machines. ReliefF-FLANN was run on a single machine using the Python bindings contained in PyFLANN 1.6.14. Table 1. Cluster description 32 Nodes. Individual specifications Processor 2 × Intel Xeon E5-2620 v3 and 2.40 GHz Cores 6 per processor (12 per node) Threads 2 per core (24 total per node) Storage 12 × 2TB NL SATA 6 Gbps 3.5" G2HS RAM 64 GB Network 1 x 10 Gbps + 2 x 1 Gbps For the experiments we selected nine real-world high-dimensional data sets, as described in Table 2, selected to represent all problems that ReliefF can handle, namely, regression ( Y e a r s m a l l ), binary classification (Higgs,40 H i g g s s m a l l , E p s i l o n s m a l l ), and multiclass classification ( K D D 9 9 s m a l l , C T s m a l l , Cifar10,42SVHN,43Sensorless44). The data sets reflect various problems, including computer vision and intrusion detection.1 Table 2. Data set description Data set Attributes Elements Classes Y e a r s m a l l 90 46,371 – Higgs 28 11,000,000 2 H i g g s s m a l l 28 55,000 2 E p s i l o n s m a l l 2000 50,000 2 K D D 9 9 s m a l l 41 48,984 23 C T s m a l l 54 58,101 7 Cifar10 3072 50,000 10 SVHN 3072 73,257 10 Sensorless 84 58,509 11 Some of the data sets contain a very large number of elements, making their processing with the original ReliefF unfeasible (several weeks would be required, even using 12 computing cores). In the first set of experiments, to compare runtimes for ReliefF, DiReliefF, and ReliefF-LSH, therefore, we used trimmed versions of the largest data sets, that is, only the top N elements were used: Y e a r s m a l l is the top 10% of the YearPredictionMSD44 data set, H i g g s s m a l l is the top 0.5% of the Higgs data set (55,000 samples), E p s i l o n s m a l l is the top 10% (50,000 elements) of Epsilon,45 K D D 9 9 s m a l l is the top 1% (48,984 elements) of KDD99,46 and C T s m a l l is the top 10% of CoverType.44 In our second set of experiments we used the full Higgs data set to test the capability of our method to handle large data sets. 4.2 Methodology We used three different measures to compare the results achieved with ReliefF and with DiReliefF, ReliefF-LSH, and ReliefF-FLANN: runtimes, recall, and weight error. First, we evaluated time efficiency by measuring algorithm runtimes for each data set. Second, we measured the accuracy of the results, using the recall measure at various selection levels to determine the correctness of the retrieved rankings of attributes. Since the main purpose of the ReliefF ranking is to determine a subset of relevant attributes, comparing the retrieved subsets shows the effectiveness of the approximate algorithms, that is, DiReliefF, ReliefF-LSH, and ReliefF-FLANN. For a given selection level t, attribute ranking ℰ obtained with ReliefF and attribute ranking A computed by an approximate algorithm, we define recall as r e c a l l ( t ) = ∣ E . f i r s t ( t ) ∩ A . f i r s t ( t ) ∣ t ,(2) where X . f i r s t ( t ) symbolizes the top t elements of list X. Third, to give a more fine-grained account of the accuracy of the rankings, a distinction had to be drawn between sets of attributes with the same number of wrong selections. Since ReliefF is a ranking method, attribute selection is based on ranking attributes by weight and keeping the top N. For some data sets, many attributes may have similar weights; hence, keeping one of these similarly weighted attributes does not have as adverse an impact as keeping an attribute with a much smaller weight. We quantified the magnitude of errors for a given selection level by measuring the difference between the weights in the exact list and the weights in the approximate list, that is, W D ( t ) = − ( ∑ a ∈ E . f i r s t ( t ) E ( a ) − ∑ a ∈ A . f i r s t ( t ) E ( a ) ) ,(3) where ℰ is the