X-Forest: Approximate Random Projection Trees for Similarity Measurement

1 Introduction . 1.1 Background and motivation . Similarity measurement is to measure the similarity between every pair of items in a given dataset . Generally , an item can be represented by a data point in the space ( e.g. , Euclidean space ) . The target of similarity measurement is to generate a similarity matrix M whose element Mij represents the similarity value between two data points : i and j. Similarity measurement plays a central role in data mining and machine learning , and also has practical applications in other fields such as biochemistry , biology , botany , etc . In data mining , similarity is a vital criterion in unsupervised clustering which is to classify objects into groups and eliminate inappropriate data Santos et al . ( 2013 ) ; Kushawah & Yadav ( 2016 ) ; Jarvis & Patrick ( 1973 ) . The result of clustering can be applied in various specific fields , e.g. , the accurate segmentation of liver lesions Jha et al . ( 2010 ) , the characterization of chemical structures and biological activity spectra Fliri et al . ( 2005 ) , or for ligand identification Koch et al . ( 2004 ) . In machine learning , similarity can be used in social filtering algorithms to make predictions for recommendation systems Billsus & Pazzani ( 1998 ) . Therefore , these extensive applications require the similarity measurement solution to maintain high accuracy in different datasets Ma & Manjunath ( 1996 ) , and this is the first design goal of this paper . The second design goal is to achieve efficient similarity measurement . 1.2 Prior art and their limitations . For similarity measurement , existing works can be divided into two kinds : mathematical distance-based similarity and multi-partition based similarity . Currently the prevailing approaches belong to the first kind of solutions , such as Minkowski distance family , Fidelity or Squared-chord family , Shannon ’ s entropy Cha ( 2007 ) , Cosine similarity Irani et al . ( 2016 ) , etc . These similarity measurement solutions only depend on the pairwise information ( i.e. , partial information ) , but neglect the overall information , such as the dimensions , features , distribution of the dataset . Consequently , they have low versatility , i.e. , lacking the flexibility to adapt to different datasets . Furthermore , they are not accurate enough , because they may not preserve the perceptual similarity ( intuitive similarity ) of the dataset , especially when encountered with high dimension datasets Ma & Manjunath ( 1996 ) ; Dasgupta & Freund ( 2008 ) . The second kind of solutions such as Multiple RP+EM Fern & Brodley ( 2003 ) and RF similarity Gray et al . ( 2013 ) overcome the shortcomings of the first kind by projecting and partitioning the data . Unfortunately , this kind of solutions often depends on priori knowledge about data distribution or data labels , which can hardly be acquired in common circumstances . Consequently , The third design goal of this paper is to eliminate the dependence on priori knowledge . No existing works can achieve all the design goals at the same time . 1.3 Our contributions . This paper aims to achieve the above three design goals at the same time . Towards the first and third goal , we introduce the Random Projection Tree ( RP Tree ) to similarity measurement . RP Tree is used to randomly partition a set of data points in a space into several disjoint subsets . It is well known that in an RP Tree , data points that are closely distributed , indicating their high level of similarity in space , are always partitioned into the same subset Dasgupta & Freund ( 2008 ) . This means that RP Tree can achieve the first goal – high accuracy . As RP Tree uses random partitions , thus achieves the third goal – eliminating priori knowledge dependence . Unfortunately , directly using RP Tree can not achieve the second goal – high efficiency , because during each partition , we need to project all data points into a random vector which is time consuming ( see details in Section 2 ) . To address this problem , we propose the X-Forest . The key idea is to allow nodes at i , i+X , i+ 2X ... ( i=0,1,2 ... ) layers of the tree to share the same projection vector for partitioning . For example , let us assume that we are given a complete binary RP Tree with 4 layers . The root node denotes the entire dataset and each node denotes a subset of the entire dataset . For each inner node , the standard RP Tree needs to project all data points into one random vector . In this way , it projects each data point four times . Our solution is to let the nodes at the first and third layers share the same projection vector , and do so for the second and fourth layers . For the nodes at the third layer , we reuse the projection results of the first layer , and do so for the four layer . In this way , we project each data point twice instead of four times . Obviously , sharing vectors sacrifices the randomness of projection vectors , further degrading the accuracy . Fortunately , the accuracy loss can be almost eliminated by using a great many Trees and carefully choosing the sharing parameter – X ( see details in Section 3.2 ) . We propose β-Similarity to record the results in X-Forest into a similarity matrix . It features better representing the similarity relationship between data points , which is proved by the higher accuracy in our clustering experiments . Our key contributions are as follows : • We introduce RP Trees into similarity measurement and proposed a new similarity matrix β-Similarity , which better reveals the similarity relationships between data points than traditional distance based similarity measurement . • We propose X-Forest which significantly reduces the time of building procedure of RP Trees by sharing projection vectors . • We conduct extensive experiments on three real datasets , and our experimental results show that RP Trees achieves two design goals , while X-Forest achieves all three desigan goals at the same time . We have released codes in github anonymously so as to meet the demand of reproducibility sou . The mathematical proofs of our algorithm is detailed in supplementary materials . 2 Background and Related Work . 2.1 Similarity Measurement . Existing solutions for similarity measurement can be classified into two categories : 1 ) mathematical distance based similarity and 2 ) multi-partition based similarity . The first kind is Mathematical distance based similarity which is widely used . It includes Minkowski distance family , Fidelity or Squared-chord and Shannon ’ s entropy Cha ( 2007 ) , Cosine similarity Irani et al . ( 2016 ) , the correlation coefficient Billsus & Pazzani ( 1998 ) , travel time , and edit distance Chen et al . ( 2009 ) , etc . According to the applying method of distance , it can be divided into two cases : 1 ) distance is directly applied as similarity ; 2 ) distance is first computed as a criterion for further similarity evaluation . An example of the second case is shared neighbours based clustering Jarvis & Patrick ( 1973 ) . The k nearest neighbours of each data point are found using Euclidean distance . The similarity between data point i and j is defined as the number of common neighbours . Existing works of the second kind , multi-partition based similarity , is rare , and an example is Multiple RP+EM Fern & Brodley ( 2003 ) . In each operation of RP+EM , the dimension of the original dataset is degraded through a linear transformation . Then , it applies EM clustering to generate a probabilistic model θ of a mixture of k Gaussians . The similarity between data points i and j is defined as the average value of P θij = ∑k l=1 P ( l | i , θ ) ×P ( l | j , θ ) of each RP+EM . The first kind of solutions falls short in terms of leading to unsatisfied results , especially in high-dimensional spaces . And the second kind depends on prior knowledge about data labels or data distribution . 2.2 Random Projection Tree . Sanjoy Dasgupta and Yoav Freund first propose the idea of RP Tree Dasgupta & Freund ( 2008 ) . An RP Tree is a variant of k − d tree Bentley ( 1975 ) . The most popular application of RP Tree is in nearest neighbours finding , where it compensates k − d tree ’ s diminishing efficacy in high-dimensional spaces Dasgupta & Sinha ( 2015 ) . Other applications of RP Tree cover clustering Yan et al . ( 2009 ) , pattern discovering Minnen et al . ( 2007 ) and nearest neighbours finding , vector quantization Dasgupta & Freund ( 2009 ) , local symmetry detection in natural images Shen et al . ( 2016 ) , etc . The details of building an RP Tree are as follows . In an RP Tree , the root node includes all items in a given set S. Through RP operation , RP Tree partitions S into two disjoint sets , each of which is a child node of the root node . The child node will be recursively partitioned until its size is smaller than a predefined threshold τ . In the end , each leaf node of an RP Tree forms a set of size less than τ . Figure 2 shows the structure of an RP Tree . The main operation of RP Tree , the RP operation , generates a random unit direction vector e for each partition . After all data points have been projected into the random direction , we uniformly choose a partition point at random within the projection range , where the projection range refers to the interval between the smallest and the largest projection value . The formal description of an RP operation is as follows . RP Tree computes the projection value of each point in the set S into the unit directional vector e. Let P be the set of projection values , i.e. , P = { x · e | x ∈ S } . Let Pmax be the maximum value in P and Pmin be the minimum value in P . The partition point Pτ is uniformly and randomly selected from ( Pmin , Pmax ) . Pτ partitions the set S into two disjoint subsets SL = { x · e 6 Pτ | x ∈ S } and SR = { x · e > Pτ | x ∈ S } . This is equivalent to partitioning the space into two parts with a ( d− 1 ) -dimensional hyperplane xT · e = Pτ . When d = 2 , as shown in Figure 1 , the ( d − 1 ) -dimensional hyperplane degrades to a line . Building an RP Tree suffers from high computation complexity since , a large number of RP operations are involved whenever an RP Tree is built , worsened by the fact that , it requires another large number of inner products to complete a single RP operation . 3 The X-Forest Algorithm . As detailed in Section 2.2 , RP Tree cluster similarly data points into the same node . In this section , we first show how to use a number of RP Trees to generate the similarity matrix . Then we introduce X-Forest , which significantly accelerate the building procedure of these trees .

Summary: This paper proposes a new method for measuring pairwise similarity between data points. The idea is to define the similarity between two data points to be the probability (over the randomness in constructing the trees) that they are close in an RP tree. More concretely, the proposed method constructs a collection of RP trees (albeit with some modifications), and takes the similarity to be the average over different RP trees of a strictly decreasing function of the distance between the leaf nodes containing the data points in each RP tree. The key modification to the RP tree is to limit the number of projection vectors used in an RP tree and re-use previous projection vectors.