X-Forest: Approximate Random Projection Trees for Similarity Measurement

1 Introduction . 1.1 Background and motivation . Similarity measurement is to measure the similarity between every pair of items in a given dataset . Generally , an item can be represented by a data point in the space ( e.g. , Euclidean space ) . The target of similarity measurement is to generate a similarity matrix M whose element Mij represents the similarity value between two data points : i and j. Similarity measurement plays a central role in data mining and machine learning , and also has practical applications in other fields such as biochemistry , biology , botany , etc . In data mining , similarity is a vital criterion in unsupervised clustering which is to classify objects into groups and eliminate inappropriate data Santos et al . ( 2013 ) ; Kushawah & Yadav ( 2016 ) ; Jarvis & Patrick ( 1973 ) . The result of clustering can be applied in various specific fields , e.g. , the accurate segmentation of liver lesions Jha et al . ( 2010 ) , the characterization of chemical structures and biological activity spectra Fliri et al . ( 2005 ) , or for ligand identification Koch et al . ( 2004 ) . In machine learning , similarity can be used in social filtering algorithms to make predictions for recommendation systems Billsus & Pazzani ( 1998 ) . Therefore , these extensive applications require the similarity measurement solution to maintain high accuracy in different datasets Ma & Manjunath ( 1996 ) , and this is the first design goal of this paper . The second design goal is to achieve efficient similarity measurement . 1.2 Prior art and their limitations . For similarity measurement , existing works can be divided into two kinds : mathematical distance-based similarity and multi-partition based similarity . Currently the prevailing approaches belong to the first kind of solutions , such as Minkowski distance family , Fidelity or Squared-chord family , Shannon ’ s entropy Cha ( 2007 ) , Cosine similarity Irani et al . ( 2016 ) , etc . These similarity measurement solutions only depend on the pairwise information ( i.e. , partial information ) , but neglect the overall information , such as the dimensions , features , distribution of the dataset . Consequently , they have low versatility , i.e. , lacking the flexibility to adapt to different datasets . Furthermore , they are not accurate enough , because they may not preserve the perceptual similarity ( intuitive similarity ) of the dataset , especially when encountered with high dimension datasets Ma & Manjunath ( 1996 ) ; Dasgupta & Freund ( 2008 ) . The second kind of solutions such as Multiple RP+EM Fern & Brodley ( 2003 ) and RF similarity Gray et al . ( 2013 ) overcome the shortcomings of the first kind by projecting and partitioning the data . Unfortunately , this kind of solutions often depends on priori knowledge about data distribution or data labels , which can hardly be acquired in common circumstances . Consequently , The third design goal of this paper is to eliminate the dependence on priori knowledge . No existing works can achieve all the design goals at the same time . 1.3 Our contributions . This paper aims to achieve the above three design goals at the same time . Towards the first and third goal , we introduce the Random Projection Tree ( RP Tree ) to similarity measurement . RP Tree is used to randomly partition a set of data points in a space into several disjoint subsets . It is well known that in an RP Tree , data points that are closely distributed , indicating their high level of similarity in space , are always partitioned into the same subset Dasgupta & Freund ( 2008 ) . This means that RP Tree can achieve the first goal – high accuracy . As RP Tree uses random partitions , thus achieves the third goal – eliminating priori knowledge dependence . Unfortunately , directly using RP Tree can not achieve the second goal – high efficiency , because during each partition , we need to project all data points into a random vector which is time consuming ( see details in Section 2 ) . To address this problem , we propose the X-Forest . The key idea is to allow nodes at i , i+X , i+ 2X ... ( i=0,1,2 ... ) layers of the tree to share the same projection vector for partitioning . For example , let us assume that we are given a complete binary RP Tree with 4 layers . The root node denotes the entire dataset and each node denotes a subset of the entire dataset . For each inner node , the standard RP Tree needs to project all data points into one random vector . In this way , it projects each data point four times . Our solution is to let the nodes at the first and third layers share the same projection vector , and do so for the second and fourth layers . For the nodes at the third layer , we reuse the projection results of the first layer , and do so for the four layer . In this way , we project each data point twice instead of four times . Obviously , sharing vectors sacrifices the randomness of projection vectors , further degrading the accuracy . Fortunately , the accuracy loss can be almost eliminated by using a great many Trees and carefully choosing the sharing parameter – X ( see details in Section 3.2 ) . We propose β-Similarity to record the results in X-Forest into a similarity matrix . It features better representing the similarity relationship between data points , which is proved by the higher accuracy in our clustering experiments . Our key contributions are as follows : • We introduce RP Trees into similarity measurement and proposed a new similarity matrix β-Similarity , which better reveals the similarity relationships between data points than traditional distance based similarity measurement . • We propose X-Forest which significantly reduces the time of building procedure of RP Trees by sharing projection vectors . • We conduct extensive experiments on three real datasets , and our experimental results show that RP Trees achieves two design goals , while X-Forest achieves all three desigan goals at the same time . We have released codes in github anonymously so as to meet the demand of reproducibility sou . The mathematical proofs of our algorithm is detailed in supplementary materials . 2 Background and Related Work . 2.1 Similarity Measurement . Existing solutions for similarity measurement can be classified into two categories : 1 ) mathematical distance based similarity and 2 ) multi-partition based similarity . The first kind is Mathematical distance based similarity which is widely used . It includes Minkowski distance family , Fidelity or Squared-chord and Shannon ’ s entropy Cha ( 2007 ) , Cosine similarity Irani et al . ( 2016 ) , the correlation coefficient Billsus & Pazzani ( 1998 ) , travel time , and edit distance Chen et al . ( 2009 ) , etc . According to the applying method of distance , it can be divided into two cases : 1 ) distance is directly applied as similarity ; 2 ) distance is first computed as a criterion for further similarity evaluation . An example of the second case is shared neighbours based clustering Jarvis & Patrick ( 1973 ) . The k nearest neighbours of each data point are found using Euclidean distance . The similarity between data point i and j is defined as the number of common neighbours . Existing works of the second kind , multi-partition based similarity , is rare , and an example is Multiple RP+EM Fern & Brodley ( 2003 ) . In each operation of RP+EM , the dimension of the original dataset is degraded through a linear transformation . Then , it applies EM clustering to generate a probabilistic model θ of a mixture of k Gaussians . The similarity between data points i and j is defined as the average value of P θij = ∑k l=1 P ( l | i , θ ) ×P ( l | j , θ ) of each RP+EM . The first kind of solutions falls short in terms of leading to unsatisfied results , especially in high-dimensional spaces . And the second kind depends on prior knowledge about data labels or data distribution . 2.2 Random Projection Tree . Sanjoy Dasgupta and Yoav Freund first propose the idea of RP Tree Dasgupta & Freund ( 2008 ) . An RP Tree is a variant of k − d tree Bentley ( 1975 ) . The most popular application of RP Tree is in nearest neighbours finding , where it compensates k − d tree ’ s diminishing efficacy in high-dimensional spaces Dasgupta & Sinha ( 2015 ) . Other applications of RP Tree cover clustering Yan et al . ( 2009 ) , pattern discovering Minnen et al . ( 2007 ) and nearest neighbours finding , vector quantization Dasgupta & Freund ( 2009 ) , local symmetry detection in natural images Shen et al . ( 2016 ) , etc . The details of building an RP Tree are as follows . In an RP Tree , the root node includes all items in a given set S. Through RP operation , RP Tree partitions S into two disjoint sets , each of which is a child node of the root node . The child node will be recursively partitioned until its size is smaller than a predefined threshold τ . In the end , each leaf node of an RP Tree forms a set of size less than τ . Figure 2 shows the structure of an RP Tree . The main operation of RP Tree , the RP operation , generates a random unit direction vector e for each partition . After all data points have been projected into the random direction , we uniformly choose a partition point at random within the projection range , where the projection range refers to the interval between the smallest and the largest projection value . The formal description of an RP operation is as follows . RP Tree computes the projection value of each point in the set S into the unit directional vector e. Let P be the set of projection values , i.e. , P = { x · e | x ∈ S } . Let Pmax be the maximum value in P and Pmin be the minimum value in P . The partition point Pτ is uniformly and randomly selected from ( Pmin , Pmax ) . Pτ partitions the set S into two disjoint subsets SL = { x · e 6 Pτ | x ∈ S } and SR = { x · e > Pτ | x ∈ S } . This is equivalent to partitioning the space into two parts with a ( d− 1 ) -dimensional hyperplane xT · e = Pτ . When d = 2 , as shown in Figure 1 , the ( d − 1 ) -dimensional hyperplane degrades to a line . Building an RP Tree suffers from high computation complexity since , a large number of RP operations are involved whenever an RP Tree is built , worsened by the fact that , it requires another large number of inner products to complete a single RP operation . 3 The X-Forest Algorithm . As detailed in Section 2.2 , RP Tree cluster similarly data points into the same node . In this section , we first show how to use a number of RP Trees to generate the similarity matrix . Then we introduce X-Forest , which significantly accelerate the building procedure of these trees .

Summary: This paper proposes the similarity measure called 'beta-similarity' generated by an ensemble of Random projection trees (RP trees) by Dasgupta & Freund (2008). To reduce the computational costs for building many RP trees, the paper develops an efficient approximate version called X-Projection trees by first generating X independent random projection directions, and then by sharing them at layers in turns. X-Forest is a set of X-Projection trees with X different random projections, and the proposed 'beta similarity' between x and y is defined by distances between a leaf region having x and one having y in PR trees. Experimental evaluations demonstrated that the use of beta similarity improves the clustering accuracy using it within three types of methods (kernel k-means, DBSCAN, Spectral clustering).