Sparse Quadratic Optimisation over the Stiefel Manifold with Application to Permutation Synchronisation

1 Introduction . We are interested in optimisation problems of the form arg max U∈St ( m , d ) f ( U ) with f ( U ) = tr ( UTWU ) and St ( m , d ) : = { X ∈ Rm×d : XTX = Id } , ( 1 ) where Id is the identity matrix of dimension d , W ∈ Rm×m and the set St ( m , d ) denotes the Stiefel manifold ( m ≥ d ) . Throughout the paper , w.l.o.g . we consider a maximisation formulation and we assume that W is a symmetric and positive semidefinite matrix ( see Lemma 1 in Sec . 2 ) . Despite the non-convexity of Problem ( 1 ) , problems of this form can be solved to global optimality based on the eigenvalue decomposition , i.e . by setting U∗ = Vd , where Vd ∈ Rm×d denotes an orthogonal basis of eigenvectors corresponding to the d largest eigenvalues of W , see Lemma 2 . The fact that we can efficiently find global optima of Problem ( 1 ) makes it a popular relaxation formulation of various difficult combinatorial problems . This includes spectral relaxations [ 43 , 26 , 17 ] of the NP-hard quadratic assignment problem ( QAP ) [ 35 ] , spectral clustering [ 33 , 44 ] , or spectral permutation synchronisation [ 34 , 39 , 31 ] . Yet , a major difficulty of such approaches is to discretise the ( relaxed ) continuous solution in order to obtain a feasible solution of the original ( discrete ) problem . For example , in the QAP one optimises over the set of permutation matrices , in clustering one optimises over the set of matrices with rows comprising canonical basis vectors , and in permutation synchronisation one optimises over multiple permutation matrices that are stacked into a large block 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . matrix . Often such combinatorial sets can be characterised by Stiefel matrices that are sparse – for example , the set of ( signed ) d×d permutation matrices can be characterised by matrices in St ( d , d ) that have exactly d non-zero elements , whereas any other element in St ( d , d ) that has more than d non-zero elements is not a signed permutation matrix . The most common approach to obtain sparse solutions in an optimisation problem is to integrate an explicit sparsity-promoting regulariser . However , a major hurdle when considering quadratic optimisation problems over the Stiefel manifold is that the incorporation of sparsity-promoting terms is not compatible with solutions based on eigenvalue decomposition , so that desirable global optimality guarantees are generally no longer maintained . Instead , we depart from the common path of integrating explicit regularisers and instead exploit the orthogonal-invariance present in Problem ( 1 ) to assure sparsity . To be more specific , for any orthogonal matrix Q ∈ O ( d ) : = St ( d , d ) it holds that f ( U ) = tr ( UTWU ) = tr ( UTWUQQT ) = tr ( ( UQ ) TW ( UQ ) ) = f ( UQ ) . Hence , if Ū ∈ St ( m , d ) is a solution to Problem ( 1 ) , so is ŪQ ∈ St ( m , d ) for any Q ∈ O ( d ) . The subspace im ( ŪQ ) that is spanned by ŪQ for a given Ū ∈ St ( m , d ) ( and arbitrary Q ∈ O ( d ) ) is equal to the subspace im ( Ū ) . Motivated by this observation we utilise orthogonal-invariance in order to find a solution U∗ ∈ { U ∈ St ( m , d ) : im ( U ) = im ( Ū ) } such that U∗ is sparse . To this end , we build upon the additional degrees of freedom due to Q ∈ O ( d ) , which allows to rotate a given solution Ū ∈ St ( m , d ) to a sparser representation U∗ = ŪQ ∈ St ( m , d ) – most notably , while remaining a globally optimal solution to Problem ( 1 ) . Main contributions . We summarise our main contributions as follows : ( i ) For the first time we propose an algorithm that exploits the orthogonal-invariance in quadratic optimisation problems over the Stiefel manifold while simultaneously accounting for sparsity in the solution . ( ii ) Despite its simplicity , our algorithm is effective as it builds on a modification of the well-established Orthogonal Iteration algorithm for finding the most dominant eigenspace of a given matrix . ( iii ) Our algorithm is guaranteed to converge to the dominant subspace with the same convergence rate as the original Orthogonal Iteration algorithm , and our solution constitutes a global optimiser of Problem ( 1 ) . ( iv ) We experimentally confirm the efficacy of our approach in the context of the permutation synchronisation problem . 2 Preliminaries & Related Work . In this section we clarify our assumptions , introduce additional preliminaries , and provide reference to related work . Let λ1 , λ2 , . . . , λm be the eigenvalues of W ∈ Rm×m ordered decreasingly . We impose the following assumption on W : Assumption 1 ( Separated eigenspace ) . We assume that λd > λd+1 . Throughout the paper we also assume that W is symmetric and positive semidefinite ( p.s.d . ) , which , however , is not a restriction as the following straightforward result indicates : Lemma 1 ( Symmetry and positive semidefiniteness ) . In Problem ( 1 ) , if W is not symmetric and not p.s.d . there is an equivalent optimisation problem ( i.e . with the same optimisers ) where W has been replaced by a symmetric p.s.d . matrix W̃ . Proof . See Appendix . Next , we define the notion of a dominant subspace and convergence of matrix sequences to such a subspace . Definition 1 ( Dominant invariant subspace ) . The d-dimensional dominant invariant subspace ( or dominant subspace in short ) of the matrix W ∈ Rm×m is defined as the subspace im ( Vd ) ⊆ Rm , where Vd ∈ St ( m , d ) is the matrix whose columns are formed by the d eigenvectors corresponding to the d largest eigenvalues of W . Definition 2 ( Convergence ) . We say that a sequence of matrices { Ut } converges to the dominant subspace of W if limt→∞ ‖VdV Td Ut − Ut‖ = 0 . There exists a close relation between the dominant subspace of W and solutions of Problem ( 1 ) : Lemma 2 ( Solution to Problem ( 1 ) ) . Problem ( 1 ) is solved for any matrix U∗ that forms an orthogonal basis for the d-dimensional dominant subspace of W , i.e . U∗ ∈ { U ∈ St ( m , d ) : im ( U ) = im ( Vd ) } . Proof . See Appendix . Basic algorithms for computing eigenvectors . In order to find the dominant subspace of W we can use algorithms for finding eigenvectors . The Power method [ 21 ] is an efficient way for finding the single most dominant eigenvector of a given matrix , i.e . it considers the case d = 1 . It proceeds by iteratively updating a given initial v0 ∈ St ( m , 1 ) based on the update vt+1 ←Wvt/‖Wvt‖ . In order to find the d most dominant eigenvectors , one can consider the Orthogonal Iteration algorithm [ 21 ] , which generalises the Power method to the case d > 1 . The algorithm proceeds by repeatedly computing Vt+1Rt+1 ←WVt based on the ( thin ) QR-decomposition of WVt , where Vt ∈ St ( m , d ) and Rt ∈ Rd×d is upper triangular . For t→∞ the sequence { Vt } converges ( under mild conditions on the intial V0 ) to the dominant invariant subspace as long as |λd| > |λd+1| , see Thm . 8.2.2 in [ 21 ] . Sparse Stiefel optimisation . There are numerous approaches for addressing general optimisation problems over the Stiefel manifold , including generic manifold optimisation techniques ( e.g . [ 3 , 14 ] ) , or Stiefel-specific approaches ( e.g . [ 30 , 46 ] ) . In the following we will focus on works that consider sparse optimisation over the Stiefel manifold that are most relevant to our approach . In order to promote sparse solutions , sparsity-inducing regularisers can be utilised , for example via the minimisation of the ( non-convex ) ` p- ‘ norm ’ for 0 ≤ p < 1 , or the ( convex ) ` 1-norm [ 36 ] . However , the non-smoothness of such regularisers often constitutes a computational obstacle . The optimisation of non-smooth functions over the Stiefel manifold has been considered in [ 16 ] , where the sum of a non-convex smooth and a convex non-smooth function is optimised via a proximal gradient method . In [ 27 ] , the authors consider the optimisation of a sum of non-smooth weakly convex functions over the Stiefel manifold using Riemannian subgradient-type methods . Yet , in practice often differentiable surrogates of non-smooth sparsity-promoting terms are considered [ 40 , 29 , 38 , 15 ] . Instead of minimising ` p- ‘ norms ’ with 0 ≤ p ≤ 1 , on the Stiefel manifold one may instead choose the maximisation of ` p-norms with p > 2 , such as the ` 3-norm [ 47 ] , or the ` 4-norm [ 49 ] . Further motivation for sparsity promoting higher-order norms in this context can be found in [ 37 , 50 , 28 ] . Optimisation over the Stiefel manifold has a close connection to optimisation over the Grassmannian manifold [ 3 ] . There are numerous approaches for Grassmannian manifold optimisation ( e.g . [ 19 , 18 ] ) , including sparse optimisation via an ` 1-norm regulariser [ 45 ] and the optimisation of non-convex and non-smooth objective functions via a projected Riemannian subgradient method [ 52 ] . In our case , due to the rotation-invariance of the objective f ( U ) = f ( UQ ) for any Q ∈ O ( d ) in Problem ( 1 ) , finding the dominant subspace of W could also be posed as an optimisation problem over the Grassmannian manifold . However , we are not only interested in identifying this subspace ( which can for example be done via the Orthogonal Iterations algorithm [ 21 ] , or via Grassmann-Rayleigh Quotient Iterations [ 4 ] ) , but we want to find a specific choice of coordinates for which the representation of the subspace is sparse . 3 Proposed Method for Sparse Quadratic Optimisation over the Stiefel . Input : W ∈ Rm×m , U0 ∈ Rm×d Output : U∗ Initialise : t← 0 repeat Ut+1Rt+1 ←WUtZ ( Ut ) // unique QR-decomposition until convergence U∗ ← Ut+1 Algorithm 1 : Overview of our proposed algorithm . In terms of convergence properties our algorithm is equivalent to the Orthogonal Iteration algorithm and thus produces a U∗ in the dominant subspace of W . However , our modification introduces the matrix Z ( Ut ) with the purpose of promoting sparsity of the solution ( see Sec . 3.2 how we choose Z ( Ut ) ) . In this section we introduce our Algorithm 1 that can be seen as a modification of the popular Orthogonal Iteration algorithm . The main difference is that we iteratively weigh the matrix Ut ∈ St ( m , d ) by a matrix Z ( Ut ) ∈ Rd×d , where the matrix function Z maps onto the set of full rank matrices . The purpose of the matrix Z ( Ut ) ( see Sec . 3.2 for our specific choice ) is to promote sparsity . Intuitively , we characterise sparsity as having few elements that are large , whereas most elements are close to zero , which is formally expressed in ( 4 ) . First , we focus on the overall interpretation of our approach : We want to ensure that we retrieve a U∗ that is a global maximiser of Problem ( 1 ) , which , in some relaxed sense , is also sparse . However , we can not in general augment the objective function in Problem ( 1 ) with an explicit sparsity regulariser and expect that the respective solution is still a maximiser of the original objective . Instead we steer the solution towards being more sparse by weighing our matrix of interest Ut with the matrix Z ( Ut ) .

Summary: In this paper the authors propose a modification of a well-known algorithm, in order to maximize a quadratic function over the Stiefel manifold, promoting sparsity at the same time. The base method is not a gradient-retraction over a manifold algorithm (although some tools are used in the middle), but a QR type of method, specifically the Orthogonal Iteration algorithm. The modification is the addition of a right multiplicative matrix, which is selected in order to promote sparsity. This is done for a maximization problem, and the $l_p$ norm (with $p>2$) is used to that end. The method is tested for the permutation synchronization problem.