Effective Subspace Indexing via Interpolation on Stiefel and Grassmann manifolds

1 INTRODUCTION . Subspace selection algorithms have been successful in many application problems related to dimension reduction ( Zhou et al . ( 2010 ) , Bian & Tao ( 2011 ) , Si et al . ( 2010 ) , Zhang et al . ( 2009 ) ) , with applications including , e.g. , human face recognition ( Fu & Huang ( 2008 ) ) , speech and gait recognition ( Tao et al . ( 2007 ) ) , etc .. The classical approaches of subspace selection in dimension reduction include algorithms like Principle Component Analysis ( PCA , see Jolliffe ( 2002 ) ) and Linear Discriminant Analysis ( LDA , see Belhumeur et al . ( 1997 ) , Tao et al . ( 2009 ) ) . They are looking for globally linear subspace models . Therefore , they fail to estimate the nonlinearity of the intrinsic data manifold , and ignore the local variation of the data ( Saul & Roweis ( 2003 ) , Strassen ( 1969 ) ) . Consequently , these globally linear models are often ineffective for search problems on large scale image data sets . To resolve this difficulty , nonlinear algorithms such as kernel algorithms ( Ham et al . ( 2004 ) ) and manifold learning algorithms ( Belkin et al . ( 2006 ) , Guan et al . ( 2011 ) ) are proposed . However , even though these nonlinear methods significantly improve the recognition performance , they face a serious computational challenge dealing with large-scale data sets due to the complexity of matrix decomposition at the size of the number of training samples . Here we propose a simple method , Subspace Indexing Model with Interpolation ( SIM-I ) , that produces from a given data set a piece-wise linear , locality-aware and globally nonlinear model of low-dimensional embedding . SIM-I is constructed via two steps : in the first step we build a piecewise linear affinity-aware subspace model under a given partition of the data set ; in the second step we interpolate between several adjacent linear subspace models constructed previously using the “ center of mass ” calculation on Stiefel and Grassmann manifolds ( Edleman et al . ( 1999 ) , Kaneko et al . ( 2013 ) , Marrinan et al . ( 2014 ) ) . The interpolation step outputs a “ smoothed ” version ( Figure 1 ) of the original piece-wise linear model , and can be regarded as a regularization process . Compared to previously mentioned subspace methods , SIM-I enjoys the following advantages : ( 1 ) it captures the global nonlinearity and thus the local fluctuations of the data set ; ( 2 ) it is computationally feasible to large-scale data sets since it avoids the complexity in matrix decomposition at the size of the number of training samples ; ( 3 ) it includes a regularization step via interpolating between several adjacent pieces of subspace models . Numerical experiments on PCA recovery task for SIFT data set and classification tasks via nearest-neighbor method for MNIST and CIFAR-10 data sets further validate the effectiveness of SIM-I . 2 PIECE-WISE LINEAR LOCALITY PRESERVING PROJECTION ( LPP ) MODEL . If an image data point x ∈ RD is represented as a vector in a very high-dimensional space , then we want to find a low-dimensional embedding y = f ( x ) ∈ Rd , d < < D such that the embedding function f retains some meaningful properties of the original image data set , ideally close to its intrinsic dimension . If we restrict ourselves to linear maps of the form y = WTx ∈ Rd , where the D × d projection matrix W = ( wij ) 1≤i≤D,1≤j≤d ( assuming full rank ) , then such a procedure is called a locally linear low-dimensional embedding ( see Roweis & Saulm ( 2000 ) ; Van Der Maaten et al . ( 2009 ) ) . The target is to search for a “ good ” projection matrix W , such that the projection x 7→ y = WTx must preserve certain locality in the data set ( this is called a Locality Preserving Projection , or LPP projection , see He & Niyogi ( 2003 ) ) . The locality is interpreted as a kind of intrinsic relative geometric relations between the data points in the original high-dimensional space , usually represented by the affinity matrix S = ( sij ) 1≤i , j≤n ( which is a symmetric matrix with non-negative terms ) . As an example , given unlabelled data points x1 , ... , xn ∈ RD , we can take sij = exp ( −‖xi−xj‖ 2 2σ2 ) when ‖xi − xj‖ < ε and sij = 0 otherwise . Here σ > 0 and ε > 0 is a small threshold parameter , and ‖xi−xj‖ is the Euclidean norm in RD . Based on the affinity matrix S = ( sij ) , the search for the projection matrix W can be formulated as the following optimization problem min W φ ( W ) = 1 2 n∑ i , j=1 sij‖yi − yj‖2 , ( 1 ) in which yi = WTxi and yj = WTxj and the norm ‖yi − yj‖ is taken in the projected space Rd . Usually when ‖xi − xj‖ is large , the affinity sij will be small , and vice versa . Thus ( 1 ) is seeking for the embedding matrix W such that close pairs of image points xi and xj will be mapped to close pairs of embeddings yi = WTxi and yj = WTxj , and vice versa . This helps to preserve the local geometry of the data set , a.k.a the locality . To solve ( 1 ) , we introduce a weighted fully-connected graph G where the vertex set consists of all data points x1 , ... , xn and the weight on the edge connecting xi and xj is given by sij ≥ 0 . Consider the diagonal matrix D = diag ( D11 , ... , Dnn ) where Dii = n∑ j=1 sij , and we then introduce the graph Laplacian L = D − S. Then the minimization problem ( 1 ) , together with the normalization constraint n∑ i=1 Diiy2i = 1 , reduces to the following generalized eigenvalue problem ( see He & Niyogi ( 2003 ) ) XLXTw = λXDXTw , ( 2 ) where X = [ x1 , ... , xn ] ∈ RD×n . Assume we have obtained an increasing family of eigenvalues 0 = λ0 < λ1 ≤ ... ≤ λn−1 . Let the corresponding eigenvectors be w0 , w1 , ... , wn−1 . Then the low-dimensional embedding matrix can be taken as W = [ w1 , ... , wd ] ( see He et al . ( 2005 ) ) . By choosing different affinity matrices S = ( sij ) , the above LPP framework includes many commonly seen practical examples . For example , if the data x1 , ... , xn are not labelled , then we can take sij = 1 n and ( 2 ) produces the classical Principle Component Analysis ( PCA ) . For labelled data forming subsets X1 , ... , Xm with same labels in each subset , we can take sij = 1 nk when xi , xj ∈ Xk , and sij = 0 other-wise . Here nk is the cardinality of Xk . This will produce Linear Discriminant Analysis ( LDA ) . The detailed justifications of these connections can be found in He et al . ( 2005 ) . Given an input data set X = { x1 , ... , xn } where each xi ∈ RD , either labelled or unlabelled , we can apply a k-d tree ( Bentley ( 1975 ) , Wang et al . ( 2011 ) ) based partition scheme to divide the whole data set X into non-overlapping subsets C1 , ... , C2h where h is the depth of the tree . Conventional subspace selection algorithms could be applied on the whole sample space before the whole space is partitioned and indexed . For example , we can first apply a PCA to X , which selects the first d bases [ a1 , ... , ad ] with largest variance . Based on these bases , the covariance information obtained from global PCA is utilized in the indexing as follows : ( 1 ) we project all sample points x1 , ... , xn onto the maximum variance basis a1 , then we find the median value m1 of the projected samples , and split the whole collection of data along a1 at m1 , i.e. , split the current node into left and right children ; ( 2 ) starting from level i = 2 , for each left and right child , project the whole collection of data along the i-th maximum variance basis ai , find the median value mi , and split all the children at mi ; ( 3 ) increase the level from i to i+ 1 and repeat ( 2 ) until i = h reaches the bottom of the tree . We collect all the 2h children at level i = h and obtain the disjoint subsets C1 , ... , C2h . Each subset Ck , k = 1 , 2 , ... , 2h consists of a family of input data in RD . Based on them , using the above LPP framework , for each Ck , a low-dimensional embedding matrix Wk ∈ RD×d can be constructed . In this way , over the whole data set X , we have constructed a piece-wise linear low-dimensional embedding model f ( x ) : RD → Rd ( see Figure 1 ( a ) ) where x ∈ X . This model is given by the linear embedding matrices W1 , ... , W2h ∈ RD×d . The above model construction can be regarded as a training process from the data set X . For a given test data point x ∈ RD , not included in X , we can find the closest subset Ck ( x ) to it , by selecting the index k = k ( x ) ∈ { 1 , ... , 2h } with the smallest distance ‖x−mk ( x ) ‖ . Here mk is the mean among all data points in the subset Ck . With the subset Ck ( x ) chosen , we map the test point x ∈ RD to its low-dimensional embedding f ( x ) = WTk ( x ) x ∈ R d. Such a procedure extended the piece-wise linear embedding model f ( x ) : RD → Rd to all testing data points in RD . 3 CALCULATING THE “ CENTER OF MASS ” ON STIEFEL AND GRASSMANN MANIFOLDS . Subspace Indexing ( Wang et al . ( 2011 ) ) provides a d-dimensional representation of the data set { x1 , ... , xn } by the subspace span ( w1 , ... , wd ) = { WTx , x ∈ RD } generated from the linear embedding matrix W ∈ RD×d . In this case , we are only interested in the column space of W , so we can assume that w0 , w1 , ... , wn−1 is an orthonormal basis 1 . Such a matrix W belongs to the Stiefel manifold , defined by Definition 1 ( Stiefel manifold ) The compact Stiefel manifold St ( d , D ) is a submanifold of the Eucilidean space RD×d such that St ( d , D ) = { X ∈ RD×d : XTX = Id } . ( 3 ) As an example , if we are interested in signal recovery using low-dimensional PCA embedding , the projections we calculated from PCA analysis will be on Stiefel manifolds . However , for classification tasks , the exact distance information is less important than label information . In this case , two such Stiefel matricesW1 andW2 produce the same embedding ifW1 = W2Od for someOd ∈ O ( d ) , whereO ( d ) is the group of orthogonal matrices in dimension d. In this case , the relevant embedding we obtained is a point on the Grassmann manifold , defined by 1If this is not the case , we can replace the matrix [ w0 w1 ... wn−1 ] by the Q matrix of the QR-decomposition of itself , without changing the corresponding subspace . Definition 2 ( Grassmann manifold ) The Grassmann manifold Gr ( d , D ) is defined to be the quotient manifold Gr ( d , D ) = St ( d , D ) /O ( d ) . A point on Gr ( d , D ) is defined by an equivalence class [ W ] = { WOd , Od ∈ O ( d ) } where W ∈ St ( d , D ) . Given a family of elements on the Stiefel or Grassmann manifold , the center-of-mass is defined as an element on the same manifold that minimizes the functional given by the weighted sum of square distances . To be precise , we have Definition 3 ( Stiefel and Grassmann center-of-masses ) Given a sequence of matrices W1 , ... , Wl ∈ St ( d , D ) and a sequence of weights w1 , ... , wl > 0 , the Stiefel center-of-mass with respect to the distance d ( W1 , W2 ) on St ( d , D ) is defined as a matrix Wc = W Stc ( W1 , ... , Wl ; w1 , ... , wl ) ∈ St ( d , D ) such that Wc = W St c ( W1 , ... , Wl ; w1 , ... , wl ) ≡ arg min W∈St ( d , D ) l∑ j=1 wjd 2 ( W , Wj ) . ( 4 ) Similarly , if the corresponding equivalent classes are [ W1 ] , ... , [ Wl ] ∈ Gr ( d , D ) , then the Grassmann center-of-mass with respect to the distance d ( [ W1 ] , [ W2 ] ) on Gr ( d , D ) is defined as the equivalence class [ Wc ] , where Wc = WGrc ( W1 , ... , Wl ; w1 , ... , wl ) ∈ St ( d , D ) is such that Wc = W Gr c ( W1 , ... , Wl ; w1 , ... , wl ) ≡ arg min W∈St ( d , D ) l∑ j=1 wjd 2 ( [ W ] , [ Wj ] ) . ( 5 ) The distances d ( W1 , W2 ) or d ( [ W1 ] , [ W2 ] ) can be taken in different ways . For example , for W1 , W2 ∈ St ( d , D ) , one way is to consider d ( W1 , W2 ) = dF ( W1 , W2 ) = ‖W1 −W2‖F , the matrix Frobenius norm of W1 −W2 . One can also take a more intrinsic distance , such as the geodesic distance between W1 and W2 on the manifold St ( d , D ) with the metric given by embedded geometry ( see Edleman et al . ( 1999 ) ) . For [ W1 ] , [ W2 ] ∈ Gr ( d , D ) , one way is to consider the projected Frobenius norm d ( [ W1 ] , [ W2 ] ) = dpF ( [ W1 ] , [ W2 ] ) = 2−1/2‖W1WT1 −W2WT2 ‖F . There are also many other choices , such as using the principle angles between the subspaces , chordal norms , or other types of Frobenius norms ( see Edleman et al . ( 1999 , Section 4.3 ) ) . With respect to matrix Frobenius norm and projected Frobenius norm , the Stiefel and Grassmann center-of-masses can be calculated explicitly in the following theorems . Theorem 1 ( Stiefel center-of-mass with respect to Frobenius norm ) We consider the singular value decomposition of the matrix ∑l j=1 wjWj = O1∆O2 , where O1 ∈ O ( D ) and O2 ∈ O ( d ) , ∆ = ( diag ( λ1 , ... , λd ) d×d 0 ( D−d ) ×d ) and λ1 ≥ ... ≥ λd ≥ 0 are the singular values . Then the Stiefel center-of-mass with respect to the distance given by Frobenius norm d ( W1 , W2 ) = ‖W1−W2‖F is given by Wc = O1ΛO2 where Λ = ( diag ( 1 , ... , 1 ) d×d 0 ( D−d ) ×d ) . Theorem 2 ( Grassmann center-of-mass with respect to projected Frobenius norm ) Set Ωj = ( ∑l j=1 wj ) −1 wj . We consider the singular value decomposition of the symmetric matrix l∑ j=1 ΩjWjW T j = Q∆Q T where Q ∈ O ( D ) and ∆ = diag ( σ21 , ... , σ2D ) , σ21 ≥ ... ≥ σ2D ≥ 0 . Then the Grassmann center-of-mass with respect to the distance given by projected Frobenius norm dpF ( [ W1 ] , [ W2 ] ) = 2−1/2‖W1WT1 − W2WT2 ‖F is the equivalence class [ Wc ] determined by Wc = QΛ , where Λ = ( diag ( 1 , ... , 1 ) d×d 0 ( D−d ) ×d ) . 4 SIM-I : INTERPOLATING THE LPP MODEL FAMILY Recall that we have developed a piece-wise linear embedding model f ( x ) : RD → Rd over the data set X = { x1 , ... , xn } . The embedding f ( x ) corresponds to a family of subspace indexing models W1 , ... , W2h ∈ St ( d , D ) ( e.g . for PCA signal recovery tasks ) or [ W1 ] , ... , [ W2h ] ∈ Gr ( d , D ) ( e.g . for classification tasks ) . Each subspace model Wk is built from LPP embedding using the subset Ck ⊂ X developed from k-d tree and h is the depth tree . Given a test point x ∈ RD that does not lie in X , we can map it to the low-dimensional embedding f ( x ) = WTk ( x ) x ∈ R d. The index k ( x ) corresponds to the subset Ck ( x ) that lie closest to x . In practice , we can first compute the means mk over all the data points in the subset Ck for each k = 1 , 2 , ... , 2h and sort the distances ‖x−mk‖ in ascending order ‖x−mk1 ( x ) ‖ ≤ ... ≤ ‖x−mk2h ( x ) ‖ , { k1 ( x ) , ... , k2h ( x ) } = { 1 , ... , 2 h } . We then take k ( x ) = k1 ( x ) to be the index k corresponding to the shortest distance . This is effective when the test point x lies significantly close to one of the subsets Ck ( x ) , see Figure 2 ( a ) . Algorithm 1 SIM-I : Subspace Indexing Model with Interpolation 1 : Input : Data set X = { x1 , ... , xn ∈ RD } and its corresponding affinity matrix S = ( sij ) 1≤i , j≤n ; test point x ∈ RD ; threshold ratio rthr > 1 ; tree depth h ; parameter K > 0 2 : Using an initial PCA and a k-d tree based partition scheme , decompose the data set X into subsets C1 , ... , C2h , where h is the depth of the tree 3 : For each subset Ck , calculate its mean ( center ) mk ∈ RD and its LPP embedding matrix Wk ∈ St ( d , D ) based on the affinity matrix S 4 : Sort the distances ‖x − mk‖ in ascending order ‖x − mk1 ( x ) ‖ ≤ ... ≤ ‖x − mk2h ( x ) ‖ , { k1 ( x ) , ... , k2h ( x ) } = { 1 , ... , 2h } 5 : Determine I , which is the first sub-index i of ki ( x ) such that ‖x−mkI+1 ( x ) ‖ > rthr‖x−mk1 ( x ) ‖ 6 : Set ji ( x ) = ki ( x ) for i = 1 , 2 , ... , I and obtain the embedding matrices Wj1 ( x ) , ... , WjI ( x ) ∈ St ( d , D ) or their corresponding subspaces [ Wj1 ( x ) ] , ... , [ WjI ( x ) ] ∈ Gr ( d , D ) , together with the weights wi = exp ( −K‖x−mji ( x ) ‖2 ) > 0 for i = 1 , ... , I 7 : Find a center-of-mass Wc = W Stc ( Wj1 ( x ) , ... , WjI ( x ) ; w1 , ... , wI ) ( Stiefel case ) or [ Wc ] = [ WGrc ( Wj1 ( x ) , ... , WjI ( x ) ; w1 , ... , wI ) ] ( Grassmann case ) according to Definition 3 and Theorems 1 and 2 . 8 : Output : The low-dimensional embedding f ( x ) = WTc x ∈ Rd However , for a general test point x ∈ RD , it might happen that this point lies at approximately the same distances to the centers of each of the several different subsets adjacent to x ( see Figure 2 ( b ) ) . In this case , we aim to interpolate between several subspace indexing modelsWj1 ( x ) , ... , WjI ( x ) . To do this , we first find the subspace indexes j1 ( x ) , ... , jI ( x ) from the first I subsets Cj1 ( x ) , ... , CjI ( x ) closest to x , i.e. , j1 ( x ) = k1 ( x ) , ... , jI ( x ) = kI ( x ) given the sorted distances ‖x−mk‖ mentioned above . In practice , the number I = I ( x ) is depending on x and can be chosen in the following way : I is the first sub-index i of ki ( x ) such that ‖x−mkI+1 ( x ) ‖ > rthr‖x−mk1 ( x ) ‖ , where rthr > 1 is a threshold ratio that can be tuned . We then pick the weights aswi = exp ( −K‖x−mji ( x ) ‖2 ) for some K > 0 and i = 1 , 2 , ... , I . This is indicating that the closer x is to Cji ( x ) , the heavier weights we assign to Wji ( x ) in the interpolation process . Given the embedding matrices Wj1 ( x ) , ... , WjI ( x ) ∈ St ( d , D ) or their corresponding subspaces [ Wj1 ( x ) ] , ... , [ WjI ( x ) ] ∈ Gr ( d , D ) , together with the weights w1 , ... , wI > 0 , we find a center-of-mass Wc = W Stc ( Wj1 ( x ) , ... , WjI ( x ) ; w1 , ... , wI ) ( Stiefel case ) or [ Wc ] = [ WGrc ( Wj1 ( x ) , ... , WjI ( x ) ; w1 , ... , wI ) ] ( Grassmann case ) according to Definition 3 and Theorems 1 and 2 . Finally , we map the test point x to the low-dimensional embedding f ( x ) = WTc x ∈ Rd . Notice that when I = 1 , the interpolation procedure reduces to projecting x using Wk1 ( x ) calculated from LPP analysis on the closest subset only . In general , the whole interpolation procedure can be regarded as providing a regularized version of the piece-wise linear embedding we discussed in Section 2 ( also see Figure 1 ) . We summarize our interpolation method as the SIM-I Algorithm 1 .

Summary: This paper proposes a subspace indexing model with interpolation (SIM-I) for dimension reduction. To capture the global nonlinearity and the local variation of data, SIM-I split the global space into a collection of local disjoint partitions by a kd-tree style scheme and build a subspace indexing model (e.g., LPP) for each local partition. The main contribution of this paper is to propose an interpolation technique to combine several subspace indexing models to determine a more promising model for low-dimensional embedding. Two experiments on PCA recovery and nearest neighbour classification validate the effectiveness of SIM-I.