Effective Subspace Indexing via Interpolation on Stiefel and Grassmann manifolds

1 INTRODUCTION . Subspace selection algorithms have been successful in many application problems related to dimension reduction ( Zhou et al . ( 2010 ) , Bian & Tao ( 2011 ) , Si et al . ( 2010 ) , Zhang et al . ( 2009 ) ) , with applications including , e.g. , human face recognition ( Fu & Huang ( 2008 ) ) , speech and gait recognition ( Tao et al . ( 2007 ) ) , etc .. The classical approaches of subspace selection in dimension reduction include algorithms like Principle Component Analysis ( PCA , see Jolliffe ( 2002 ) ) and Linear Discriminant Analysis ( LDA , see Belhumeur et al . ( 1997 ) , Tao et al . ( 2009 ) ) . They are looking for globally linear subspace models . Therefore , they fail to estimate the nonlinearity of the intrinsic data manifold , and ignore the local variation of the data ( Saul & Roweis ( 2003 ) , Strassen ( 1969 ) ) . Consequently , these globally linear models are often ineffective for search problems on large scale image data sets . To resolve this difficulty , nonlinear algorithms such as kernel algorithms ( Ham et al . ( 2004 ) ) and manifold learning algorithms ( Belkin et al . ( 2006 ) , Guan et al . ( 2011 ) ) are proposed . However , even though these nonlinear methods significantly improve the recognition performance , they face a serious computational challenge dealing with large-scale data sets due to the complexity of matrix decomposition at the size of the number of training samples . Here we propose a simple method , Subspace Indexing Model with Interpolation ( SIM-I ) , that produces from a given data set a piece-wise linear , locality-aware and globally nonlinear model of low-dimensional embedding . SIM-I is constructed via two steps : in the first step we build a piecewise linear affinity-aware subspace model under a given partition of the data set ; in the second step we interpolate between several adjacent linear subspace models constructed previously using the “ center of mass ” calculation on Stiefel and Grassmann manifolds ( Edleman et al . ( 1999 ) , Kaneko et al . ( 2013 ) , Marrinan et al . ( 2014 ) ) . The interpolation step outputs a “ smoothed ” version ( Figure 1 ) of the original piece-wise linear model , and can be regarded as a regularization process . Compared to previously mentioned subspace methods , SIM-I enjoys the following advantages : ( 1 ) it captures the global nonlinearity and thus the local fluctuations of the data set ; ( 2 ) it is computationally feasible to large-scale data sets since it avoids the complexity in matrix decomposition at the size of the number of training samples ; ( 3 ) it includes a regularization step via interpolating between several adjacent pieces of subspace models . Numerical experiments on PCA recovery task for SIFT data set and classification tasks via nearest-neighbor method for MNIST and CIFAR-10 data sets further validate the effectiveness of SIM-I . 2 PIECE-WISE LINEAR LOCALITY PRESERVING PROJECTION ( LPP ) MODEL . If an image data point x ∈ RD is represented as a vector in a very high-dimensional space , then we want to find a low-dimensional embedding y = f ( x ) ∈ Rd , d < < D such that the embedding function f retains some meaningful properties of the original image data set , ideally close to its intrinsic dimension . If we restrict ourselves to linear maps of the form y = WTx ∈ Rd , where the D × d projection matrix W = ( wij ) 1≤i≤D,1≤j≤d ( assuming full rank ) , then such a procedure is called a locally linear low-dimensional embedding ( see Roweis & Saulm ( 2000 ) ; Van Der Maaten et al . ( 2009 ) ) . The target is to search for a “ good ” projection matrix W , such that the projection x 7→ y = WTx must preserve certain locality in the data set ( this is called a Locality Preserving Projection , or LPP projection , see He & Niyogi ( 2003 ) ) . The locality is interpreted as a kind of intrinsic relative geometric relations between the data points in the original high-dimensional space , usually represented by the affinity matrix S = ( sij ) 1≤i , j≤n ( which is a symmetric matrix with non-negative terms ) . As an example , given unlabelled data points x1 , ... , xn ∈ RD , we can take sij = exp ( −‖xi−xj‖ 2 2σ2 ) when ‖xi − xj‖ < ε and sij = 0 otherwise . Here σ > 0 and ε > 0 is a small threshold parameter , and ‖xi−xj‖ is the Euclidean norm in RD . Based on the affinity matrix S = ( sij ) , the search for the projection matrix W can be formulated as the following optimization problem min W φ ( W ) = 1 2 n∑ i , j=1 sij‖yi − yj‖2 , ( 1 ) in which yi = WTxi and yj = WTxj and the norm ‖yi − yj‖ is taken in the projected space Rd . Usually when ‖xi − xj‖ is large , the affinity sij will be small , and vice versa . Thus ( 1 ) is seeking for the embedding matrix W such that close pairs of image points xi and xj will be mapped to close pairs of embeddings yi = WTxi and yj = WTxj , and vice versa . This helps to preserve the local geometry of the data set , a.k.a the locality . To solve ( 1 ) , we introduce a weighted fully-connected graph G where the vertex set consists of all data points x1 , ... , xn and the weight on the edge connecting xi and xj is given by sij ≥ 0 . Consider the diagonal matrix D = diag ( D11 , ... , Dnn ) where Dii = n∑ j=1 sij , and we then introduce the graph Laplacian L = D − S. Then the minimization problem ( 1 ) , together with the normalization constraint n∑ i=1 Diiy2i = 1 , reduces to the following generalized eigenvalue problem ( see He & Niyogi ( 2003 ) ) XLXTw = λXDXTw , ( 2 ) where X = [ x1 , ... , xn ] ∈ RD×n . Assume we have obtained an increasing family of eigenvalues 0 = λ0 < λ1 ≤ ... ≤ λn−1 . Let the corresponding eigenvectors be w0 , w1 , ... , wn−1 . Then the low-dimensional embedding matrix can be taken as W = [ w1 , ... , wd ] ( see He et al . ( 2005 ) ) . By choosing different affinity matrices S = ( sij ) , the above LPP framework includes many commonly seen practical examples . For example , if the data x1 , ... , xn are not labelled , then we can take sij = 1 n and ( 2 ) produces the classical Principle Component Analysis ( PCA ) . For labelled data forming subsets X1 , ... , Xm with same labels in each subset , we can take sij = 1 nk when xi , xj ∈ Xk , and sij = 0 other-wise . Here nk is the cardinality of Xk . This will produce Linear Discriminant Analysis ( LDA ) . The detailed justifications of these connections can be found in He et al . ( 2005 ) . Given an input data set X = { x1 , ... , xn } where each xi ∈ RD , either labelled or unlabelled , we can apply a k-d tree ( Bentley ( 1975 ) , Wang et al . ( 2011 ) ) based partition scheme to divide the whole data set X into non-overlapping subsets C1 , ... , C2h where h is the depth of the tree . Conventional subspace selection algorithms could be applied on the whole sample space before the whole space is partitioned and indexed . For example , we can first apply a PCA to X , which selects the first d bases [ a1 , ... , ad ] with largest variance . Based on these bases , the covariance information obtained from global PCA is utilized in the indexing as follows : ( 1 ) we project all sample points x1 , ... , xn onto the maximum variance basis a1 , then we find the median value m1 of the projected samples , and split the whole collection of data along a1 at m1 , i.e. , split the current node into left and right children ; ( 2 ) starting from level i = 2 , for each left and right child , project the whole collection of data along the i-th maximum variance basis ai , find the median value mi , and split all the children at mi ; ( 3 ) increase the level from i to i+ 1 and repeat ( 2 ) until i = h reaches the bottom of the tree . We collect all the 2h children at level i = h and obtain the disjoint subsets C1 , ... , C2h . Each subset Ck , k = 1 , 2 , ... , 2h consists of a family of input data in RD . Based on them , using the above LPP framework , for each Ck , a low-dimensional embedding matrix Wk ∈ RD×d can be constructed . In this way , over the whole data set X , we have constructed a piece-wise linear low-dimensional embedding model f ( x ) : RD → Rd ( see Figure 1 ( a ) ) where x ∈ X . This model is given by the linear embedding matrices W1 , ... , W2h ∈ RD×d . The above model construction can be regarded as a training process from the data set X . For a given test data point x ∈ RD , not included in X , we can find the closest subset Ck ( x ) to it , by selecting the index k = k ( x ) ∈ { 1 , ... , 2h } with the smallest distance ‖x−mk ( x ) ‖ . Here mk is the mean among all data points in the subset Ck . With the subset Ck ( x ) chosen , we map the test point x ∈ RD to its low-dimensional embedding f ( x ) = WTk ( x ) x ∈ R d. Such a procedure extended the piece-wise linear embedding model f ( x ) : RD → Rd to all testing data points in RD . 3 CALCULATING THE “ CENTER OF MASS ” ON STIEFEL AND GRASSMANN MANIFOLDS . Subspace Indexing ( Wang et al . ( 2011 ) ) provides a d-dimensional representation of the data set { x1 , ... , xn } by the subspace span ( w1 , ... , wd ) = { WTx , x ∈ RD } generated from the linear embedding matrix W ∈ RD×d . In this case , we are only interested in the column space of W , so we can assume that w0 , w1 , ... , wn−1 is an orthonormal basis 1 . Such a matrix W belongs to the Stiefel manifold , defined by Definition 1 ( Stiefel manifold ) The compact Stiefel manifold St ( d , D ) is a submanifold of the Eucilidean space RD×d such that St ( d , D ) = { X ∈ RD×d : XTX = Id } . ( 3 ) As an example , if we are interested in signal recovery using low-dimensional PCA embedding , the projections we calculated from PCA analysis will be on Stiefel manifolds . However , for classification tasks , the exact distance information is less important than label information . In this case , two such Stiefel matricesW1 andW2 produce the same embedding ifW1 = W2Od for someOd ∈ O ( d ) , whereO ( d ) is the group of orthogonal matrices in dimension d. In this case , the relevant embedding we obtained is a point on the Grassmann manifold , defined by 1If this is not the case , we can replace the matrix [ w0 w1 ... wn−1 ] by the Q matrix of the QR-decomposition of itself , without changing the corresponding subspace . Definition 2 ( Grassmann manifold ) The Grassmann manifold Gr ( d , D ) is defined to be the quotient manifold Gr ( d , D ) = St ( d , D ) /O ( d ) . A point on Gr ( d , D ) is defined by an equivalence class [ W ] = { WOd , Od ∈ O ( d ) } where W ∈ St ( d , D ) . Given a family of elements on the Stiefel or Grassmann manifold , the center-of-mass is defined as an element on the same manifold that minimizes the functional given by the weighted sum of square distances . To be precise , we have Definition 3 ( Stiefel and Grassmann center-of-masses ) Given a sequence of matrices W1 , ... , Wl ∈ St ( d , D ) and a sequence of weights w1 , ... , wl > 0 , the Stiefel center-of-mass with respect to the distance d ( W1 , W2 ) on St ( d , D ) is defined as a matrix Wc = W Stc ( W1 , ... , Wl ; w1 , ... , wl ) ∈ St ( d , D ) such that Wc = W St c ( W1 , ... , Wl ; w1 , ... , wl ) ≡ arg min W∈St ( d , D ) l∑ j=1 wjd 2 ( W , Wj ) . ( 4 ) Similarly , if the corresponding equivalent classes are [ W1 ] , ... , [ Wl ] ∈ Gr ( d , D ) , then the Grassmann center-of-mass with respect to the distance d ( [ W1 ] , [ W2 ] ) on Gr ( d , D ) is defined as the equivalence class [ Wc ] , where Wc = WGrc ( W1 , ... , Wl ; w1 , ... , wl ) ∈ St ( d , D ) is such that Wc = W Gr c ( W1 , ... , Wl ; w1 , ... , wl ) ≡ arg min W∈St ( d , D ) l∑ j=1 wjd 2 ( [ W ] , [ Wj ] ) . ( 5 ) The distances d ( W1 , W2 ) or d ( [ W1 ] , [ W2 ] ) can be taken in different ways . For example , for W1 , W2 ∈ St ( d , D ) , one way is to consider d ( W1 , W2 ) = dF ( W1 , W2 ) = ‖W1 −W2‖F , the matrix Frobenius norm of W1 −W2 . One can also take a more intrinsic distance , such as the geodesic distance between W1 and W2 on the manifold St ( d , D ) with the metric given by embedded geometry ( see Edleman et al . ( 1999 ) ) . For [ W1 ] , [ W2 ] ∈ Gr ( d , D ) , one way is to consider the projected Frobenius norm d ( [ W1 ] , [ W2 ] ) = dpF ( [ W1 ] , [ W2 ] ) = 2−1/2‖W1WT1 −W2WT2 ‖F . There are also many other choices , such as using the principle angles between the subspaces , chordal norms , or other types of Frobenius norms ( see Edleman et al . ( 1999 , Section 4.3 ) ) . With respect to matrix Frobenius norm and projected Frobenius norm , the Stiefel and Grassmann center-of-masses can be calculated explicitly in the following theorems . Theorem 1 ( Stiefel center-of-mass with respect to Frobenius norm ) We consider the singular value decomposition of the matrix ∑l j=1 wjWj = O1∆O2 , where O1 ∈ O ( D ) and O2 ∈ O ( d ) , ∆ = ( diag ( λ1 , ... , λd ) d×d 0 ( D−d ) ×d ) and λ1 ≥ ... ≥ λd ≥ 0 are the singular values . Then the Stiefel center-of-mass with respect to the distance given by Frobenius norm d ( W1 , W2 ) = ‖W1−W2‖F is given by Wc = O1ΛO2 where Λ = ( diag ( 1 , ... , 1 ) d×d 0 ( D−d ) ×d ) . Theorem 2 ( Grassmann center-of-mass with respect to projected Frobenius norm ) Set Ωj = ( ∑l j=1 wj ) −1 wj . We consider the singular value decomposition of the symmetric matrix l∑ j=1 ΩjWjW T j = Q∆Q T where Q ∈ O ( D ) and ∆ = diag ( σ21 , ... , σ2D ) , σ21 ≥ ... ≥ σ2D ≥ 0 . Then the Grassmann center-of-mass with respect to the distance given by projected Frobenius norm dpF ( [ W1 ] , [ W2 ] ) = 2−1/2‖W1WT1 − W2WT2 ‖F is the equivalence class [ Wc ] determined by Wc = QΛ , where Λ = ( diag ( 1 , ... , 1 ) d×d 0 ( D−d ) ×d ) . 4 SIM-I : INTERPOLATING THE LPP MODEL FAMILY Recall that we have developed a piece-wise linear embedding model f ( x ) : RD → Rd over the data set X = { x1 , ... , xn } . The embedding f ( x ) corresponds to a family of subspace indexing models W1 , ... , W2h ∈ St ( d , D ) ( e.g . for PCA signal recovery tasks ) or [ W1 ] , ... , [ W2h ] ∈ Gr ( d , D ) ( e.g . for classification tasks ) . Each subspace model Wk is built from LPP embedding using the subset Ck ⊂ X developed from k-d tree and h is the depth tree . Given a test point x ∈ RD that does not lie in X , we can map it to the low-dimensional embedding f ( x ) = WTk ( x ) x ∈ R d. The index k ( x ) corresponds to the subset Ck ( x ) that lie closest to x . In practice , we can first compute the means mk over all the data points in the subset Ck for each k = 1 , 2 , ... , 2h and sort the distances ‖x−mk‖ in ascending order ‖x−mk1 ( x ) ‖ ≤ ... ≤ ‖x−mk2h ( x ) ‖ , { k1 ( x ) , ... , k2h ( x ) } = { 1 , ... , 2 h } . We then take k ( x ) = k1 ( x ) to be the index k corresponding to the shortest distance . This is effective when the test point x lies significantly close to one of the subsets Ck ( x ) , see Figure 2 ( a ) . Algorithm 1 SIM-I : Subspace Indexing Model with Interpolation 1 : Input : Data set X = { x1 , ... , xn ∈ RD } and its corresponding affinity matrix S = ( sij ) 1≤i , j≤n ; test point x ∈ RD ; threshold ratio rthr > 1 ; tree depth h ; parameter K > 0 2 : Using an initial PCA and a k-d tree based partition scheme , decompose the data set X into subsets C1 , ... , C2h , where h is the depth of the tree 3 : For each subset Ck , calculate its mean ( center ) mk ∈ RD and its LPP embedding matrix Wk ∈ St ( d , D ) based on the affinity matrix S 4 : Sort the distances ‖x − mk‖ in ascending order ‖x − mk1 ( x ) ‖ ≤ ... ≤ ‖x − mk2h ( x ) ‖ , { k1 ( x ) , ... , k2h ( x ) } = { 1 , ... , 2h } 5 : Determine I , which is the first sub-index i of ki ( x ) such that ‖x−mkI+1 ( x ) ‖ > rthr‖x−mk1 ( x ) ‖ 6 : Set ji ( x ) = ki ( x ) for i = 1 , 2 , ... , I and obtain the embedding matrices Wj1 ( x ) , ... , WjI ( x ) ∈ St ( d , D ) or their corresponding subspaces [ Wj1 ( x ) ] , ... , [ WjI ( x ) ] ∈ Gr ( d , D ) , together with the weights wi = exp ( −K‖x−mji ( x ) ‖2 ) > 0 for i = 1 , ... , I 7 : Find a center-of-mass Wc = W Stc ( Wj1 ( x ) , ... , WjI ( x ) ; w1 , ... , wI ) ( Stiefel case ) or [ Wc ] = [ WGrc ( Wj1 ( x ) , ... , WjI ( x ) ; w1 , ... , wI ) ] ( Grassmann case ) according to Definition 3 and Theorems 1 and 2 . 8 : Output : The low-dimensional embedding f ( x ) = WTc x ∈ Rd However , for a general test point x ∈ RD , it might happen that this point lies at approximately the same distances to the centers of each of the several different subsets adjacent to x ( see Figure 2 ( b ) ) . In this case , we aim to interpolate between several subspace indexing modelsWj1 ( x ) , ... , WjI ( x ) . To do this , we first find the subspace indexes j1 ( x ) , ... , jI ( x ) from the first I subsets Cj1 ( x ) , ... , CjI ( x ) closest to x , i.e. , j1 ( x ) = k1 ( x ) , ... , jI ( x ) = kI ( x ) given the sorted distances ‖x−mk‖ mentioned above . In practice , the number I = I ( x ) is depending on x and can be chosen in the following way : I is the first sub-index i of ki ( x ) such that ‖x−mkI+1 ( x ) ‖ > rthr‖x−mk1 ( x ) ‖ , where rthr > 1 is a threshold ratio that can be tuned . We then pick the weights aswi = exp ( −K‖x−mji ( x ) ‖2 ) for some K > 0 and i = 1 , 2 , ... , I . This is indicating that the closer x is to Cji ( x ) , the heavier weights we assign to Wji ( x ) in the interpolation process . Given the embedding matrices Wj1 ( x ) , ... , WjI ( x ) ∈ St ( d , D ) or their corresponding subspaces [ Wj1 ( x ) ] , ... , [ WjI ( x ) ] ∈ Gr ( d , D ) , together with the weights w1 , ... , wI > 0 , we find a center-of-mass Wc = W Stc ( Wj1 ( x ) , ... , WjI ( x ) ; w1 , ... , wI ) ( Stiefel case ) or [ Wc ] = [ WGrc ( Wj1 ( x ) , ... , WjI ( x ) ; w1 , ... , wI ) ] ( Grassmann case ) according to Definition 3 and Theorems 1 and 2 . Finally , we map the test point x to the low-dimensional embedding f ( x ) = WTc x ∈ Rd . Notice that when I = 1 , the interpolation procedure reduces to projecting x using Wk1 ( x ) calculated from LPP analysis on the closest subset only . In general , the whole interpolation procedure can be regarded as providing a regularized version of the piece-wise linear embedding we discussed in Section 2 ( also see Figure 1 ) . We summarize our interpolation method as the SIM-I Algorithm 1 .

Summary: The authors propose an approach towards computing  globally non-linear, low-dimensional embeddings of high dimensional data. In particular, they consider a sub-space indexing model with interpolation (called SIM-I) which consists of two steps. First, a locally linear model of the data is generated (using kD-trees and PCA) and second, averaging operations on the respective projection matrices are carried out. These are computed based on center of mass computations on Stiefel or Grasammanian manifolds. Anectdotal experimental evidence shows that this approach outperforms simple baseline techniques.