Sparse Quadratic Optimisation over the Stiefel Manifold with Application to Permutation Synchronisation

1 Introduction . We are interested in optimisation problems of the form arg max U∈St ( m , d ) f ( U ) with f ( U ) = tr ( UTWU ) and St ( m , d ) : = { X ∈ Rm×d : XTX = Id } , ( 1 ) where Id is the identity matrix of dimension d , W ∈ Rm×m and the set St ( m , d ) denotes the Stiefel manifold ( m ≥ d ) . Throughout the paper , w.l.o.g . we consider a maximisation formulation and we assume that W is a symmetric and positive semidefinite matrix ( see Lemma 1 in Sec . 2 ) . Despite the non-convexity of Problem ( 1 ) , problems of this form can be solved to global optimality based on the eigenvalue decomposition , i.e . by setting U∗ = Vd , where Vd ∈ Rm×d denotes an orthogonal basis of eigenvectors corresponding to the d largest eigenvalues of W , see Lemma 2 . The fact that we can efficiently find global optima of Problem ( 1 ) makes it a popular relaxation formulation of various difficult combinatorial problems . This includes spectral relaxations [ 43 , 26 , 17 ] of the NP-hard quadratic assignment problem ( QAP ) [ 35 ] , spectral clustering [ 33 , 44 ] , or spectral permutation synchronisation [ 34 , 39 , 31 ] . Yet , a major difficulty of such approaches is to discretise the ( relaxed ) continuous solution in order to obtain a feasible solution of the original ( discrete ) problem . For example , in the QAP one optimises over the set of permutation matrices , in clustering one optimises over the set of matrices with rows comprising canonical basis vectors , and in permutation synchronisation one optimises over multiple permutation matrices that are stacked into a large block 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . matrix . Often such combinatorial sets can be characterised by Stiefel matrices that are sparse – for example , the set of ( signed ) d×d permutation matrices can be characterised by matrices in St ( d , d ) that have exactly d non-zero elements , whereas any other element in St ( d , d ) that has more than d non-zero elements is not a signed permutation matrix . The most common approach to obtain sparse solutions in an optimisation problem is to integrate an explicit sparsity-promoting regulariser . However , a major hurdle when considering quadratic optimisation problems over the Stiefel manifold is that the incorporation of sparsity-promoting terms is not compatible with solutions based on eigenvalue decomposition , so that desirable global optimality guarantees are generally no longer maintained . Instead , we depart from the common path of integrating explicit regularisers and instead exploit the orthogonal-invariance present in Problem ( 1 ) to assure sparsity . To be more specific , for any orthogonal matrix Q ∈ O ( d ) : = St ( d , d ) it holds that f ( U ) = tr ( UTWU ) = tr ( UTWUQQT ) = tr ( ( UQ ) TW ( UQ ) ) = f ( UQ ) . Hence , if Ū ∈ St ( m , d ) is a solution to Problem ( 1 ) , so is ŪQ ∈ St ( m , d ) for any Q ∈ O ( d ) . The subspace im ( ŪQ ) that is spanned by ŪQ for a given Ū ∈ St ( m , d ) ( and arbitrary Q ∈ O ( d ) ) is equal to the subspace im ( Ū ) . Motivated by this observation we utilise orthogonal-invariance in order to find a solution U∗ ∈ { U ∈ St ( m , d ) : im ( U ) = im ( Ū ) } such that U∗ is sparse . To this end , we build upon the additional degrees of freedom due to Q ∈ O ( d ) , which allows to rotate a given solution Ū ∈ St ( m , d ) to a sparser representation U∗ = ŪQ ∈ St ( m , d ) – most notably , while remaining a globally optimal solution to Problem ( 1 ) . Main contributions . We summarise our main contributions as follows : ( i ) For the first time we propose an algorithm that exploits the orthogonal-invariance in quadratic optimisation problems over the Stiefel manifold while simultaneously accounting for sparsity in the solution . ( ii ) Despite its simplicity , our algorithm is effective as it builds on a modification of the well-established Orthogonal Iteration algorithm for finding the most dominant eigenspace of a given matrix . ( iii ) Our algorithm is guaranteed to converge to the dominant subspace with the same convergence rate as the original Orthogonal Iteration algorithm , and our solution constitutes a global optimiser of Problem ( 1 ) . ( iv ) We experimentally confirm the efficacy of our approach in the context of the permutation synchronisation problem . 2 Preliminaries & Related Work . In this section we clarify our assumptions , introduce additional preliminaries , and provide reference to related work . Let λ1 , λ2 , . . . , λm be the eigenvalues of W ∈ Rm×m ordered decreasingly . We impose the following assumption on W : Assumption 1 ( Separated eigenspace ) . We assume that λd > λd+1 . Throughout the paper we also assume that W is symmetric and positive semidefinite ( p.s.d . ) , which , however , is not a restriction as the following straightforward result indicates : Lemma 1 ( Symmetry and positive semidefiniteness ) . In Problem ( 1 ) , if W is not symmetric and not p.s.d . there is an equivalent optimisation problem ( i.e . with the same optimisers ) where W has been replaced by a symmetric p.s.d . matrix W̃ . Proof . See Appendix . Next , we define the notion of a dominant subspace and convergence of matrix sequences to such a subspace . Definition 1 ( Dominant invariant subspace ) . The d-dimensional dominant invariant subspace ( or dominant subspace in short ) of the matrix W ∈ Rm×m is defined as the subspace im ( Vd ) ⊆ Rm , where Vd ∈ St ( m , d ) is the matrix whose columns are formed by the d eigenvectors corresponding to the d largest eigenvalues of W . Definition 2 ( Convergence ) . We say that a sequence of matrices { Ut } converges to the dominant subspace of W if limt→∞ ‖VdV Td Ut − Ut‖ = 0 . There exists a close relation between the dominant subspace of W and solutions of Problem ( 1 ) : Lemma 2 ( Solution to Problem ( 1 ) ) . Problem ( 1 ) is solved for any matrix U∗ that forms an orthogonal basis for the d-dimensional dominant subspace of W , i.e . U∗ ∈ { U ∈ St ( m , d ) : im ( U ) = im ( Vd ) } . Proof . See Appendix . Basic algorithms for computing eigenvectors . In order to find the dominant subspace of W we can use algorithms for finding eigenvectors . The Power method [ 21 ] is an efficient way for finding the single most dominant eigenvector of a given matrix , i.e . it considers the case d = 1 . It proceeds by iteratively updating a given initial v0 ∈ St ( m , 1 ) based on the update vt+1 ←Wvt/‖Wvt‖ . In order to find the d most dominant eigenvectors , one can consider the Orthogonal Iteration algorithm [ 21 ] , which generalises the Power method to the case d > 1 . The algorithm proceeds by repeatedly computing Vt+1Rt+1 ←WVt based on the ( thin ) QR-decomposition of WVt , where Vt ∈ St ( m , d ) and Rt ∈ Rd×d is upper triangular . For t→∞ the sequence { Vt } converges ( under mild conditions on the intial V0 ) to the dominant invariant subspace as long as |λd| > |λd+1| , see Thm . 8.2.2 in [ 21 ] . Sparse Stiefel optimisation . There are numerous approaches for addressing general optimisation problems over the Stiefel manifold , including generic manifold optimisation techniques ( e.g . [ 3 , 14 ] ) , or Stiefel-specific approaches ( e.g . [ 30 , 46 ] ) . In the following we will focus on works that consider sparse optimisation over the Stiefel manifold that are most relevant to our approach . In order to promote sparse solutions , sparsity-inducing regularisers can be utilised , for example via the minimisation of the ( non-convex ) ` p- ‘ norm ’ for 0 ≤ p < 1 , or the ( convex ) ` 1-norm [ 36 ] . However , the non-smoothness of such regularisers often constitutes a computational obstacle . The optimisation of non-smooth functions over the Stiefel manifold has been considered in [ 16 ] , where the sum of a non-convex smooth and a convex non-smooth function is optimised via a proximal gradient method . In [ 27 ] , the authors consider the optimisation of a sum of non-smooth weakly convex functions over the Stiefel manifold using Riemannian subgradient-type methods . Yet , in practice often differentiable surrogates of non-smooth sparsity-promoting terms are considered [ 40 , 29 , 38 , 15 ] . Instead of minimising ` p- ‘ norms ’ with 0 ≤ p ≤ 1 , on the Stiefel manifold one may instead choose the maximisation of ` p-norms with p > 2 , such as the ` 3-norm [ 47 ] , or the ` 4-norm [ 49 ] . Further motivation for sparsity promoting higher-order norms in this context can be found in [ 37 , 50 , 28 ] . Optimisation over the Stiefel manifold has a close connection to optimisation over the Grassmannian manifold [ 3 ] . There are numerous approaches for Grassmannian manifold optimisation ( e.g . [ 19 , 18 ] ) , including sparse optimisation via an ` 1-norm regulariser [ 45 ] and the optimisation of non-convex and non-smooth objective functions via a projected Riemannian subgradient method [ 52 ] . In our case , due to the rotation-invariance of the objective f ( U ) = f ( UQ ) for any Q ∈ O ( d ) in Problem ( 1 ) , finding the dominant subspace of W could also be posed as an optimisation problem over the Grassmannian manifold . However , we are not only interested in identifying this subspace ( which can for example be done via the Orthogonal Iterations algorithm [ 21 ] , or via Grassmann-Rayleigh Quotient Iterations [ 4 ] ) , but we want to find a specific choice of coordinates for which the representation of the subspace is sparse . 3 Proposed Method for Sparse Quadratic Optimisation over the Stiefel . Input : W ∈ Rm×m , U0 ∈ Rm×d Output : U∗ Initialise : t← 0 repeat Ut+1Rt+1 ←WUtZ ( Ut ) // unique QR-decomposition until convergence U∗ ← Ut+1 Algorithm 1 : Overview of our proposed algorithm . In terms of convergence properties our algorithm is equivalent to the Orthogonal Iteration algorithm and thus produces a U∗ in the dominant subspace of W . However , our modification introduces the matrix Z ( Ut ) with the purpose of promoting sparsity of the solution ( see Sec . 3.2 how we choose Z ( Ut ) ) . In this section we introduce our Algorithm 1 that can be seen as a modification of the popular Orthogonal Iteration algorithm . The main difference is that we iteratively weigh the matrix Ut ∈ St ( m , d ) by a matrix Z ( Ut ) ∈ Rd×d , where the matrix function Z maps onto the set of full rank matrices . The purpose of the matrix Z ( Ut ) ( see Sec . 3.2 for our specific choice ) is to promote sparsity . Intuitively , we characterise sparsity as having few elements that are large , whereas most elements are close to zero , which is formally expressed in ( 4 ) . First , we focus on the overall interpretation of our approach : We want to ensure that we retrieve a U∗ that is a global maximiser of Problem ( 1 ) , which , in some relaxed sense , is also sparse . However , we can not in general augment the objective function in Problem ( 1 ) with an explicit sparsity regulariser and expect that the respective solution is still a maximiser of the original objective . Instead we steer the solution towards being more sparse by weighing our matrix of interest Ut with the matrix Z ( Ut ) .

Summary: The paper is concerned with pursuing sparse optimal solutions to optimization problems of the form max Trace(U^T W U) [*] where U is an orthonormal matrix, that is U lies in the Stiefel manifold. This problem is meaningful since it typically arises as a relaxation over the Stiefel manifold of combinatorial problems such as spectral clustering and permutation synchronization. It this latter problem that occupies the application aspect of the paper. A global (not necessarily sparse) solution to [*] can be obtained in closed form from the eigendecomposition of W. Algorithmically, this can be computed by the power iteration algorithm. The paper proposes a variation of the power iteration algorithm (Algorithm 1), which includes a sparsity promoting step, forcing the basis matrix U to be close to a power of U. Permutation synchronization experiments with synthetic data and on the CMU "house" sequence for finding correspondences in two views,  show that this approach gives better permutation synchronization than several other existing alternatives.