Title: OT4P: Unlocking Effective Orthogonal Group Path for Permutation Relaxation
Venue: NeurIPS 2024
Accepted: yes
Keywords: permutation matrix

orthogonal group

differentiable transformation

stochastic optimization
Abstract: Optimization over permutations is typically an NP-hard problem that arises extensively in ranking, matching, tracking, etc. Birkhoff polytope-based relaxation methods have made significant advancements, particularly in penalty-free optimization and probabilistic inference. Relaxation onto the orthogonal group offers unique potential advantages such as a lower representation dimension and preservation of inner products; however, equally effective approaches remain unexplored. To bridge the gap, we present a temperature-controlled differentiable transformation that maps unconstrained vector space to the orthogonal group, where the temperature, in the limit, concentrates orthogonal matrices near permutation matrices. This transformation naturally implements a parameterization for the relaxation of permutation matrices, allowing for gradient-based optimization of problems involving permutations. Additionally, by deriving a re-parameterized gradient estimator, this transformation also provides efficient stochastic optimization over the latent permutations. Extensive experiments involving the optimization over permutation matrices validate the effectiveness of the proposed method.

OT4P: Unlocking Effective Orthogonal Group Path for Permutation Relaxation

Abstract

Optimization over permutations is typically an NP-hard problem that arises extensively in ranking, matching, tracking, etc. Birkhoff polytope-based relaxation methods have made significant advancements, particularly in penalty-free optimization and probabilistic inference. Relaxation onto the orthogonal group offers unique potential advantages such as a lower representation dimension and preservation of inner products; however, equally effective approaches remain unexplored. To bridge the gap, we present a temperature-controlled differentiable transformation that maps unconstrained vector space to the orthogonal group, where the temperature, in the limit, concentrates orthogonal matrices near permutation matrices. This transformation naturally implements a parameterization for the relaxation of permutation matrices, allowing for gradient-based optimization of problems involving permutations. Additionally, by deriving a re-parameterized gradient estimator, this transformation also provides efficient stochastic optimization over the latent permutations. Extensive experiments involving the optimization over permutation matrices validate the effectiveness of the proposed method.

1 Introduction

Permutation refers to the reordering of elements within a finite set, commonly encountered in problems involving bijections between two equally sized sets [17, 42, 15, 12]. A permutation of n elements can be denoted by an n × n permutation matrix, which is a square binary matrix that has exactly one entry of 1 in each row and each column, with all other entries being 0 . We denote the set of all n -order permutation matrices as P n := { P ∈ { 0 , 1 } n × n | ∑ i P i,j = 1 , ∑ j P i,j = 1 ( ∀ i, j ) } . This work considers optimization over permutation matrices:

Due to the combinatorial nature of permutation matrices, the cardinality of the set P n grows factorially with the dimension n , typically rendering the problem NP-hard [26]. From a theoretical perspective, one of the most renowned special cases of Equation (1) is the quadratic assignment problem, which has attracted extensive research [37, 44]. In practical terms, Equation (1) also arises extensively in various machine learning tasks, including ranking [20, 75, 67, 68], matching [2], tracking [43], etc.

Previous studies have proposed relaxing permutation matrices into continuous spaces, including the convex hull of permutation matrices-the Birkhoff polytope [15, 42]-and their embeddings in a differentiable manifold-the orthogonal group [72, 27]. Recently, relaxation methods involving the Birkhoff polytope have made significant advancements, particularly in penalty-free optimization

and probabilistic inference [59, 43, 49]. As a notable example, Mena et al. [49] utilize the Sinkhorn operator [61] to transform matrices into the Birkhoff polytope, bringing them closer to permutation matrices under temperature control. This approach avoids introducing penalty terms and supports variational inference.

However, providing equally good relaxation methods within the orthogonal group remains an unexplored area. Indeed, relaxation onto the orthogonal group offers several unique potential advantages, such as: i) a lower representation dimension ( n ( n -1) 2 ) compared to the Birkhoff polytope ( ( n -1) 2 ), leading to a smaller search space; ii) the orthogonal matrix preserve the inner product of vectors, which is useful for tasks requiring the maintenance of geometric structures. In light of the above advantages, this work aims to develop an effective method for relaxing the permutation matrices onto the orthogonal group, with a particular focus on:

Flexibility: can control the degree of approximation to permutation matrices.

Simplicity: does not rely on additional penalty terms.

Scalability: enables learning the latent variable model with permutations.

In this paper, we present O rthogonal Group-based T ransformation for P ermutation Relaxation ( OT4P ), a temperature-controlled differentiable transformation. OT4P maps unconstrained vector space to the orthogonal group, where the temperature, in the limit, concentrates orthogonal matrices near permutation matrices. As illustrated in Figure 1, OT4P involves two steps: I) map a vector ( ) to an orthogonal matrix ( ) utilizing the Lie exponential; II) move the orthogonal matrix ( ) along the geodesic, controlled by temperature, to another orthogonal matrix ( ), making it nearer to the closest permutation matrix ( or ). OT4P naturally implements a parameterization for the relaxation of permutation matrices, allowing for gradient-based optimization of problems involving permutations. In addition, OT4P , combined with the re-parameterization trick, provides stochastic optimization over the latent permutations.

In summary, our main contributions are as follows:

We present OT4P , a differentiable transformation for relaxing permutation matrices onto the orthogonal group, characterized by its flexibility, simplicity, and scalability (Section 3.1).

We use OT4P to implement a parameterization for the relaxation of permutation matrices, which has the advantages of not altering the original problem, not complicating the original problem, and an efficient optimization process (Section 3.2).

We derive a gradient estimator using OT4P and the re-parameterization trick, providing an efficient tool for stochastic optimization over latent permutations (Section 3.3).

We validate the effectiveness of the proposed method through extensive experiments involving the optimization of permutation matrices, including finding mode connectivity, inferring neuron identities, and solving permutation synchronization (Section 4).

2 Preliminaries

In this section, we give a brief overview of the Riemannian geometry [38] and the Lie group theory [21] involved, with a more comprehensive version available in Appendix B.

An n -dimensional manifold M is a space that can be locally approximated by a Euclidean space R n , where each point x ∈ M possesses a tangent space T x M as a first-order local approximation of M around x . The Riemannian metric is a collection m := { m x | x ∈ M} of inner products m x ( · , · ) : T x M× T x M→ R , which may define distances on the manifold. A geodesic is a smooth curve that the tangent vector is parallel transported along the curve w.r.t. the Levi-Civita connection.

A Lie group G is a differentiable manifold equipped with differentiable group operations, whose tangent space at the identity e is the Lie algebra g . For each g ∈ G , there exist diffeomorphisms (Definition 2) given by the left translation L g ( x ) := gx ( ∀ x ∈ G ) , which lead to a vector space isomorphism (Definition 1) that relates the tangent space T g G to the Lie algebra g , i.e., (d L g ) e : g → T g G . Analogously, one can introduce the right translation : R g ( x ) := xg ( ∀ g, x ∈ G ) . A Riemannian metric m on Lie group G is called left-invariant ( right-invariant ) if it renders each left (right) translation an isometry (Definition 3), allowing us to associate neighborhoods of the identity e with any point g ∈ G using left (right) translation. If a metric on the Lie group is both left and right invariant, it is termed the bi-invariant metric.

The Lie group we are interested in is the orthogonal group O( n ) , which consists of all n × n orthogonal matrices O satisfying O ⊤ O = OO ⊤ = I . We equip O( n ) with the canonical metric, a biinvariant metric defined as ⟨ A,B ⟩ F := trace( A ⊤ B ) , where ⟨· , ·⟩ F is the Frobenius inner product and trace( · ) is the trace of a matrix. The subset of O( n ) with determinant +1 forms a subgroup known as the special orthogonal group , denoted by SO( n ) := { O ∈ R n × n | O ⊤ O = I, det O = +1 } . The Lie algebra so ( n ) of the Lie group SO( n ) comprises n × n skew-symmetric matrices, expressed as so ( n ) := { A ∈ R n × n | A ⊤ = -A } . The Lie exponential expm( · ) , coinciding with the matrix exponential in the context of the matrix Lie group, maps elements in so ( n ) to SO( n ) , defined by

The series in Equation (2) converges for all matrices A . The local inverse function of the matrix exponential is supposed to be the matrix logarithm , which is defined as follows:

The series in Equation (3) converges whenever ∥ A -I ∥ F < 1 , where ∥ A ∥ F := √ ⟨ A,A ⟩ F represents the Frobenius norm induced by the Frobenius inner product.

3 Relaxing permutation on orthogonal group

In Section 3.1, we introduce the two steps of the proposed OT4P and analyze its key properties. Then, in Section 3.2, we demonstrate how to use OT4P to implement a parameterization for the relaxation of permutation matrices, emphasizing the advantages of such a parameterization. Finally, in Section 3.3, we provide efficient stochastic optimization over the latent permutations using OT4P and the re-parameterization trick.

3.1 The proposed OT4P transformation

The proposed OT4P comprises two steps: I) map a point in the vector space to an orthogonal matrix; II) move the orthogonal matrix along the geodesic under temperature control, bringing it nearer to the closest permutation matrix. We summarize the pseudo-code of OT4P in Algorithm 1.

Step I

Consider an unconstrained vector space R n ( n -1) 2 . For a vector a ∈ R n ( n -1) 2 , we can fill it into an upper triangular n × n matrix with zero in the diagonal. For example, in the case of n = 3 :

Algorithm 1 OT4P

In the following, we employ matrices A ∈ R n × n rather than vectors a to represent the elements in R n ( n -1) 2 . A skew-symmetric matrix is uniquely determined by n ( n -1) 2 scalars, i.e., the entries above the main diagonal. Therefore, there exists an isomorphism between the vector space R n ( n -1) 2 and the Lie algebra so ( n ) formed by skew-symmetric matrices, given by

As mentioned in Section 2, we can use the matrix exponential (Lie exponential) expm( · ) to map the Lie algebra so ( n ) to the Lie group, i.e., special orthogonal group SO( n ) :

Combining Equation (4) and Equation (5), we can map the unconstrained vector space R n ( n -1) 2 to the special orthogonal group SO( n ) , denoted as

This mapping belongs to the classical category in Lie group theory and serves as an efficient solution for addressing orthogonal constraints in the field of machine learning [40, 53, 48]. Similarly to Lezcano Casado [39], we present the important properties of the mapping ϕ ( · ) below.

Theorem 1. The mapping ϕ ( · ) is differentiable, surjective, and it is injective on the domain U := { A ∈ R n ( n -1) 2 | Im λ k ( A -A ⊤ ) ∈ ( -π, π ) , ∀ k } with λ k ( · ) the eigenvalues. Additionally, the set SO( n ) \ ϕ ( U ) has a zero Lebesgue measure in SO( n ) .

The theorem indicates that each orthogonal matrix in SO( n ) can be represented by a vector in R n ( n -1) 2 , with each representation being uniquely defined within set U , provided it exists there. However, permutation matrices may include -1 as one of their eigenvalues (see Figure 4), with their corresponding representations precisely lying on the boundary of U . If the optimal solution to Equation (1) is a permutation matrix with an eigenvalue of -1 , it may lead the optimization path to deviate from U . To counter this, we propose shifting the boundary of U to other eigenvalues by left-multiplying the result of Equation (6) with an orthogonal matrix B ∈ SO( n ) . Theoretically, the left translation L B ( O ) := BO ( ∀ O ∈ SO( n )) creates a diffeomorphism (Definition 2) on SO( n ) , where the representation of the permutation matrix P in U is changed from logm( P ) to logm( B ⊤ P ) 2 . We have empirically observed that the left translation L B effectively relocates the majority of permutation matrices' representations into the interior of U . More discussion can be found in Appendix C.

Step II

Given an orthogonal matrix O ∈ SO( n ) , we would like to move it toward the closest permutation matrix along the geodesic. To achieve this, we first need to find the permutation matrix P ∈ P n closest to O , which can be expressed as

where ⟨ A,B ⟩ F = trace( A ⊤ B ) denotes the Frobenius inner product. Equation (7) is a linear assignment problem that can be solved in cubic time using the Hungarian algorithm [36], with further details available in the Appendix E.

Once P is found, we can move O towards P along the geodesic OP . In the Lie group, utilizing the mapping logm( · ) , movement along the geodesic can be transformed into a more manageable movement on the Lie algebra. However, since O or P may be far from the identity matrix I , the convergence speed of the series expansion of logm( · ) may be slow or even fail to converge.

We propose to carry out the above process in the tangent space T P SO( n ) rather than in the Lie algebra so ( n ) . Due to the biinvariant metric ⟨· , ·⟩ F equipped on SO( n ) , the left translation L P ( O ) = PO ( ∀ O ∈ SO( n )) establishes an isometry (Definition 3) between the neighborhoods of I and P , and its derivative (d L P ) e : so → T P SO( n ) provides an isomorphism (Definition 1) between the Lie algebra so ( n ) and the tangent space T P SO( n ) . Hence, we first push O ∈ SO( n ) into the neighbor-

hood of I , then map it to the Lie algebra so ( n ) using logm( · ) , and finally pull the result into the tangent space T P SO( n ) . In this way, we define the logarithm map at P as

Equation (8) maps O ∈ SO( n ) near P to the tangent space T P SO( n ) , and its convergence domain also concentrates near P . Similarly, we can define its local inverse mapping:

Equation (9) maps A ∈ T P SO( n ) onto the special orthogonal group SO( n ) located near P . With the aforementioned tools, we can easily move orthogonal matrix O toward its closest permutation matrix P by interpolation. Specifically, we map P and O to the tangent space T P SO( n ) for linear interpolation, and then map the interpolation result back to SO( n ) , given as

The second equation stems from the fact that logm( I ) = 0 , and the last equation follows from A τ = expm( τ logm( A )) when ∥ A -I ∥ F < 1 . The temperature parameter τ ∈ (0 , 1] is used to control the degree to which the resulting orthogonal matrix ˜ O approaches P . It is clear that lim τ → 0 + ∥ ˜ O -P ∥ F = 0 .

Remark on odd permutations. The attentive reader may notice that Equation (10) presupposes P ∈ SO( n ) , which works only for P corresponds to even permutations. However, we can readily extend it to cases where P corresponds to odd permutations. Firstly, we solve arg min ̂ P ∈ SO( n ) ∥ ̂ P -P ∥ 2 F to identify an agent of P within SO( n ) , which admits an analytical solution ̂ P = PD with D = diag( { 1 , . . . , 1 , -1 } ) . Then, by substituting P with ̂ P in Equation (10), O is moved toward ̂ P to obtain ̂ O . Finally, we right-multiply ̂ O by D ⊤ to map it to the neighborhood of P , resulting in ˜ O = ̂ OD ⊤ . Essentially, the neighborhoods of P and its agent ̂ P are linked through an isometry, specifically the right translation R D ( O ) := OD ( ∀ O ∈ O( n )) . Consequently, when ̂ O is moved close enough to ̂ P , the resulting orthogonal matrix ˜ O will also be sufficiently close to P . Please refer Appendix D for more theoretical details.

Let S P denote the set of orthogonal matrices in SO( n ) whose closest permutation matrix is P , and let S ′ P represent its image obtained through Equation (10). Equation (10) actually transforms the submanifold S P into a new submanifold S ′ P that is closer to the permutation matrix P . We define the manifold consisting of all images as M P := {S ′ P ⊂ O( n ) | P ∈ P n } . Consequently, the special orthogonal group SO( n ) is mapped to a manifold M P that tightly wraps around the permutation matrices, which can be more formally expressed as:

The aforementioned mapping covers all cases, where D = diag( { 1 , . . . , 1 , -1 } ) for odd permutations and D = I for even permutations. In Figure 5, we provide a visualization of the results of ψ τ ( · ) as the temperature parameter τ varies. The mapping ψ τ ( · ) is meaningless at points where ρ ( · ) w.r.t. Equation (7) is discontinuous. It is important to note that ρ ( · ) is a piecewise constant function, changing only at points where multiple permutation matrices are equidistant. The following theorem presents key properties of the mapping ψ τ ( · ) .

Theorem 2. The mapping ψ τ ( · ) is differentiable, surjective, and injective on each submanifold S P . Additionally, the set of meaningless points for ψ τ ( · ) has a zero Lebesgue measure in SO( n ) .

The theorem shows that any point in the relaxation manifold M P of permutation matrices can be uniquely identified by an orthogonal matrix in the special orthogonal group SO( n ) , where the set of meaningless elements (i.e., not mapped any point in M P ) can be disregarded. Using the composite mapping ψ τ ◦ ϕ , we create a one-to-one correspondence between U := { A ∈ R n ( n -1) 2 | Im λ k ( A -A ⊤ ) ∈ ( -π, π ) , ∀ k } and M P , except for points associated with zero measure sets in SO( n ) that are either not representable by ϕ ( · ) or are meaningless for ψ τ ( · ) . In other words, points in the manifold M P , which tightly wraps around the permutation matrices, can almost be represented one-to-one by points in U that lie in the unconstrained vector space.

3.2 Parameterization for gradient-based optimization

This section demonstrates how to use OT4P to implement a parameterization for the relaxation of permutation matrices, thereby allowing gradient-based optimization for Equation (1). More importantly, we present three advantages of this parameterization, making it a reasonable solution.

Recalling the manifold M P obtained from Equation (11), which converges around the permutation matrices controlled by the temperature parameter τ . We first relax Equation (1) into an optimization problem on the manifold M P :

Using the composite mapping ψ τ ◦ ϕ , we transform the constrained optimization problem on the manifold M P into an unconstrained optimization problem in the vector space R n ( n -1) 2 :

For the aforementioned optimization problem, we can employ standard optimization techniques, such as SGD and Adam algorithms [58], to approximate the solution. Below, we thoroughly discuss the three advantages brought about by the parameterization of OT4P .

The surjectivity does not alter the original problem. The surjectivity of the mapping ψ τ ◦ ϕ implies that every point in the manifold M P has at least one corresponding pre-image in the vector space R n ( n -1) 2 . This guarantees that, during the optimization process, any point within the manifold M P can be reached, thereby preventing the overlooking of any potential solutions to Equation (12). In particular, if we find a solution A while dealing with Equation (13), we can solve Equation (12) by mapping O = ψ τ ◦ ϕ ( A ) .

The injectivity does not complicate the original problem. If the optimization stays within U := { A ∈ R n ( n -1) 2 | Im λ k ( A -A ⊤ ) ∈ ( -π, π ) , ∀ k } , the mapping ψ τ ◦ ϕ map different elements in U to different elements on M P . This means that each update in U results in a unique outcome in M P , thereby reducing unnecessary redundant searches. Furthermore, the mapping ψ τ ◦ ϕ does not introduce spurious local minima, as each local minima in M P creates a single local minima in U .

The efficient optimization process. At first glance, the mapping ψ τ ◦ ϕ involves matrix exponential and matrix power, which might demand substantial computational resources during the optimization process. Lezcano-Casado and Martınez-Rubio [40] has proposed a cheap method for computing matrix exponential expm( · ) and its gradient, thanks to the efficient utilization of the scaling-squaring technique and Padé approximation [3]. Therefore, we will focus on how to handle the matrix power function efficiently.

Forward process. The orthogonal matrix O can be factorized, utilizing eigendecomposition, as O = QXQ -1 , where Q ∈ R n × n with each column representing an eigenvector of O , and X = diag( { λ 1 , . . . , λ n } ) is a diagonal matrix whose elements are the eigenvalues of O . In this case, the matrix power O τ can be computed by applying the power function to the eigenvalues while keeping the eigenvectors unchanged [24], yielding O τ = Q diag( { λ τ 1 , . . . , λ τ n } ) Q -1 . It is evident that calculating O τ is not significantly more complex than computing n scalar powers.

Backward process. Given an orthogonal matrix O , we assume that ˜ O = ψ τ ( O ) has been obtained through Equation (11). Then, there exists a unique orthogonal matrix W τ such that ˜ O = W τ O due to the closure property of the Lie group. Therefore, in the forward pass, one can initially acquire ˜ O using Equation (11), followed by computing the equivalent transformation of the mapping ψ τ as W τ = ˜ OO ⊤ . In this way, the forward pass is streamlined into ˜ O = W τ O , thereby rendering the backward pass highly efficient, as it only involves one linear transformation.

3.3 Re-parameterization provides stochastic optimization

In the previous section, we considered deterministic optimization over permutation matrices. However, in many scenarios, we commonly build a probabilistic model to express the uncertainty inherent in the problem [7]. For such a task, it is crucial to have the ability to learn latent variable models associated with the latent nodes corresponding to permutations [20]. This section demonstrates how to perform stochastic optimization over the latent permutations using OT4P and the re-parameterization trick.

We restrict our attention to the scenario where the latent variable is a permutation matrix, z = P , without loss of generality. Therefore, consider the probabilistic form of Equation (1) as follows

The above equation deals with a distribution over permutation matrices rather than a single permutation matrix as in Equation (1). Evaluating and differentiating Equation (14) is challenging due to the expectation involving a sum of n ! terms. To remedy this, we employ the re-parameterization trick [32, 56, 14]. In particular, we simulate q ( P ; θ ) using the mappings ρ ( · ) w.r.t. Equation (7) and ϕ ( · ) w.r.t. Equation (11), expressed as

where ϵ ∼ q ( ϵ ) is a random noise distribution. Equation (15) cleverly decouples the stochastic nature of the distribution q ( P ; θ ) , thereby obviating the dependence of the expectation on the parameters θ . This enables us to draw multiple samples for the evaluation of Equation (14) with lower variance.

However, there exists a not differentiable mapping ρ ( · ) , which hinders gradient-based optimization for θ . Recalling OT4P , we can approximate Equation (15) by relaxing the mapping ρ ( · ) to ψ τ ( · ) . It is evident that samples drawn from distribution ψ τ ( ϕ ( A + Bϵ )) converge almost surely to those from distribution ρ ( ϕ ( A + Bϵ )) owing to lim τ → 0 ψ τ = ρ . To summarize, we can bring the gradient inside the expectation, as depicted below:

which can now be computed using Monte Carlo [50].

4 Experiments

This section conducts experiments to evaluate the performance of OT4P in optimization problems and probabilistic tasks. All experimental details not stated here, along with additional results, can be found in Appendix F. The core code for OT4P is available at https://github.com/YamingGuo98/OT4P .

4.1 Finding mode connectivity

In the first experiment, we consider an optimization problem inspired by the concept of linear mode connectivity. Recent studies have shown that neural networks trained with SGD belong to a set whose weights can be permuted so that we linearly connect those weights with no detriment to the loss [13, 57]. For demonstration purposes, we examine a multi-layer perceptron (MLP) with L layers and denote its weights as θ = { W l | l ∈ [ L ] } . To find the optimal permutation between models θ A and θ B , Ainsworth et al. [2] propose the following data-free optimization problem:

The above problem is challenging because it does not admit a polynomial-time constant-factor approximation scheme [2]. We can use OT4P to relax permutation matrices, as demonstrated in Section 3.2, enabling a gradient-based solution to Equation (17).

We explore a variety of network architectures, including MLP5 (5-layer MLP) [54], VGG11 [60], and ResNet18 [22]. The weights for these networks are derived from official pre-trained models in PyTorch [52], with the exception of the MLP5, which is initialized randomly. For model θ B , we randomly sample permutation matrices from a uniform distribution and apply them to permute the weights, yielding model θ A = π ( θ B ) . The AdamW [45] with an initial learning rate of 0 . 1 is employed to minimize the loss w.r.t. Equation (17), with a maximum of 500 iterations. To evaluate the results, we use the ℓ 1 -Distance, ∥ θ A -π ( θ B ) ∥ 1 , to measure the difference from the target weights. Additionally, we flatten the permutation matrices and evaluate their alignment with the ground truth using Precision, Recall, and Hamming Distance.

We compare: 1) Weight Matching [2], which goes through each layer and greedily selects its best permutation matrix P i ; 2) Sinkhorn [54], relaxing the permutation matrices to the vicinity of the Birkhoff polytope utilizing the Sinkhorn operator [61, 49]; and 3) OT4P , our proposed method, which is evaluated with various temperature parameters. The results are reported in Tables 1 and 4, with each experiment conducted five times. The findings indicate that Weight Matching occasionally fails to reach ground truth due to its sensitivity to random initialization. Additionally, Sinkhorn yields poor results in the VGG11 network architecture, which we attribute to the relaxation on the Birkhoff polytope producing unreliable local minima. In contrast, OT4P finds the optimal permutation matrix in most cases. Indeed, a neural network can be conceptualized as a geometric object whose vertices correspond to the rows of the weight matrices [35]. A reasonable relaxation of the matching task in Equation (17) involves rigid transformations represented by orthogonal matrices. The proposed OT4P relaxes the permutation matrices into the orthogonal group, which is likely the primary reason for its powerful performance.

4.2 Inferring neuron identities

In the second experiment, we tackle a probabilistic task motivated by the study of the neural dynamics in C. elegans [66]. This worm serves as a model organism in neuroscience, with its complete neuronal connectivity known and represented by the adjacency matrix A ∈ { 0 , 1 } n × n . However, matching

traces from the observed neural dynamics Y ∈ R n × 1 to the neurons in the reference connectome A poses a challenging task. Linderman et al. [43] propose simulating neural activity using a linear dynamical system Y t = P ( A ⊙ W ) P ⊤ Y t -1 + ϵ , where ϵ is Gaussian noise, W ∈ R n × n , P ∈ P n are latent variables, and ⊙ is element-wise product. Our goal is to infer the latent permutation P to align the observed Y with the shared dynamics matrix W . We address this task by maximizing the marginal log-likelihood, i.e., max E P ∼ q ( P ; θ ) log p ( Y | P ) , using the techniques outlined in Section 3.3.

Taking the methodology in Linderman et al. [43], we generate parameters A , W , and P with n = 250 and randomly generate 1000 samples, where the noise follows a Gaussian distribution N (0 , 0 . 01) . We formulate tasks of varying difficulty depending on the different proportions of known neurons [43]. Conceiving a constraint matrix C ∈ R n × n where all elements are initialized to 1 , if we ascertain that the reference neuron i corresponds to the observed neuron j , then set all elements to 0 in the i -th row and j -th column except for C i,j (see Equation (22) for an example). This constraint is enforced by zeroing corresponding entries before solving Equation (7). We conduct 500 iterations using the Adam optimizer [31] with an initial learning rate of 0 . 01 . We report the marginal log-likelihood of the best model throughout the training, ranked first by Hamming Distance and then by the marginal log-likelihood (estimated with 5 repeats). As done in Section 4.1, Precision, Recall, and Hamming Distance are utilized to evaluate the permutation matrices obtained from the best model.

For comparison, we include: 1) Naive [43], which does not enforce that P is a permutation matrix and instead normalizes each row using the softmax function; 2) Gumbel-Sinkhorn [49], introducing Gumbel noise for re-parameterization before the Sinkhorn operator; and 3) OT4P , our proposed method using the re-parameterization trick, with different temperature parameters. Each experiment is conducted five times, and the results are presented in Tables 2 and 5. We observe that Naive fails to produce any meaningful solutions, and Gumbel-Sinkhorn performs poorly in the more challenging scenario (Known 5% ). In contrast, OT4P consistently identifies the optimal permutation, except for OT4P ( τ = 0 . 7 ), which achieves suboptimal results in the Known 5% setting. One possible reason why OT4P performs better is that the orthogonal group ( n ( n -1) 2 ) has a lower dimension than the Birkhoff polytope ( ( n -1) 2 ). This lower dimensionality makes the randomness simulated by the noise more effective in exploring latent permutations.

4.3 Solving permutation synchronization

In the third experiment, we aim to explore the effectiveness of our proposed OT4P on large-scale problems. We specifically focus on the permutation synchronization problem [51, 47, 5], which tries to improve matching across multiple objects. Consider k objects with n points each, and let the permutation matrix P ( i, j ) ∈ P n to represent the correspondence between points in objects i and

20

30

40

Instance Size

70.0

80.0

90.0

100.0

F-score (%)

Car

20

30

40

50

Instance Size

70.0

80.0

90.0

100.0

F-score (%)

Duck

Reg

OrthReg

RiemanBirk

Sinkhorn

OT4P(

=0.3

)

OT4P(

=0.5

)

OT4P(

=0.7

)

25

50

75

100

Instance Size

85.0

90.0

95.0

100.0

F-score (%)

Face

20

30

40

Instance Size

40.0

60.0

80.0

100.0

F-score (%)

Motorbike

20

40

60

Instance Size

80.0

85.0

90.0

95.0

100.0

F-score (%)

Winebottle

j . Permutation synchronization seeks to identify the underlying permutations P i ( i ∈ [ k ]) such that P ( i, j ) = P i P ⊤ j for all i, j ∈ [ k ] , which can be expressed as the following optimization problem:

Weselect baselines: 1) Reg optimizes in Euclidean space with a regularization term ∑ j ( ∑ j P i,j -1) 2 that encourages each column to sum to 1 . 2) OrthReg [72] optimizes over the special orthogonal group, using a regularization term 2 3 trace( P T ( P -P ⊙ P )) ( ⊙ is element-wise product) to force orthogonal matrices to converge to permutation matrices. 3) RiemanBirk [6] optimizes on Birkhoff polytope utilizing Riemannian gradient descent. 4) Sinkhorn [49] optimizes in the vicinity of the Birkhoff polytope, using the Sinkhorn operator to adjust positive matrices into approximate doubly stochastic matrices. All algorithms employ the Adam optimizer for 100 iterations, with RiemanBirk utilizing Riemannian Adam [4, 34]. The initial learning rates are tuned within the set { 0 . 1 , 0 . 01 , 0 . 001 , 0 . 0001 } .

We use the WILLOW-ObjectClass dataset [9] to generate problem instances (see Appendix F.4 for more details) and utilize the F-score to evaluate the alignment between the flattened permutation matrices and the ground truth. Each experiment is conducted five times, and the results are shown in Figure 3. RiemanBirk and Sinkhorn demonstrate poorer performance. A primary reason is that both methods are based on Birkhoff polytope to relax permutations, leading to unreliable local minima and preventing optimal solutions. Benefiting from the potential advantages offered by the orthogonal group, OrthReg generally produces competitive results. However, due to the instability of its regularization term, OrthReg sometimes underperforms, which may necessitate careful adjustment of the regularization coefficient for each class. In contrast, our proposed OT4P consistently outperforms other methods and demonstrates robustness to variations in the hyperparameter τ .

5 Conclusion

In this paper, we present a novel differentiable transformation, OT4P , designed for relaxing permutation matrices over the orthogonal group. This method is characterized by its flexibility, simplicity, and scalability. OT4P is utilized to parametrize the relaxation of permutation matrices, with advantages of not altering the original problem, not complicating the original problem, and an efficient optimization process. By deriving a gradient estimator, OT4P further provides an efficient tool for stochastic optimization over latent permutations. Extensive experiments show that OT4P achieves competitive results in optimization problems and probabilistic tasks compared to relaxation methods on the Birkhoff polytope. We believe our elementary work is a significant step toward relaxing permutations onto the orthogonal group. Please see Appendix A for further discussion, including related work, limitations, and broader impacts.

Acknowledgments and Disclosure of Funding

This work is supported by the National Key Research and Development Program of China (No. 2020YFA0714103).

References

Ryan Prescott Adams and Richard S Zemel. Ranking via sinkhorn propagation. arXiv preprint arXiv:1106.1925 , 2011.

Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=CQsmMYmlP5T .

Awad H Al-Mohy and Nicholas J Higham. A new scaling and squaring algorithm for the matrix exponential. SIAM Journal on Matrix Analysis and Applications , 31(3):970-989, 2010.

Gary Becigneul and Octavian-Eugen Ganea. Riemannian adaptive optimization methods. In International Conference on Learning Representations , 2019. URL https://openreview. net/forum?id=r1eiqi09K7 .

Florian Bernard, Daniel Cremers, and Johan Thunberg. Sparse quadratic optimisation over the stiefel manifold with application to permutation synchronisation. Advances in Neural Information Processing Systems , 34:25256-25266, 2021.

Tolga Birdal and Umut Simsekli. Probabilistic permutation synchronization using the riemannian structure of the birkhoff polytope. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11105-11116, 2019.

David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association , 112(518):859-877, 2017.

Rainer E Burkard, Stefan E Karisch, and Franz Rendl. Qaplib-a quadratic assignment problem library. Journal of Global optimization , 10:391-403, 1997.

Minsu Cho, Karteek Alahari, and Jean Ponce. Learning graphs to match. In Proceedings of the IEEE International Conference on Computer Vision , pages 25-32, 2013.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Zvi Drezner. The quadratic assignment problem. Location science , pages 345-363, 2015.

Hannah Dröge, Zorah Lähner, Yuval Bahat, Onofre Martorell Nadal, Felix Heide, and Michael Möller. Kissing to find a match: Efficient low-rank permutation representation. Advances in Neural Information Processing Systems , 36, 2024.

Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=dNigytemkL .

Luca Falorsi, Pim de Haan, Tim R Davidson, and Patrick Forré. Reparameterizing distributions on lie groups. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 3244-3253. PMLR, 2019.

Marcelo Fiori, Pablo Sprechmann, Joshua Vogelstein, Pablo Musé, and Guillermo Sapiro. Robust multimodal graph matching: Sparse coding meets graph matching. Advances in neural information processing systems , 26, 2013.

Matteo Fischetti, Michele Monaci, and Domenico Salvagnin. Three ideas for the quadratic assignment problem. Operations research , 60(4):954-964, 2012.

Fajwel Fogel, Rodolphe Jenatton, Francis Bach, and Alexandre d'Aspremont. Convex relaxations for permutation problems. Advances in neural information processing systems , 26, 2013.

Jean Gallier. Logarithms and square roots of real matrices existence, uniqueness and applications in medical imaging. arXiv preprint arXiv:0805.0245 , 2011.

Zheng Gong and Ying Sun. An energy-centric framework for category-free out-of-distribution node detection in graphs. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 908-919, 2024.

Aditya Grover, Eric Wang, Aaron Zweig, and Stefano Ermon. Stochastic optimization of sorting networks via continuous relaxations. In International Conference on Learning Representations , 2019. URL https://openreview.net/forum?id=H1eSS3CcKX .

Brian C Hall. Lie groups, Lie algebras, and representations . Springer, 2013.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Nicholas J Higham. Functions of matrices: theory and computation . SIAM, 2008.

Nicholas J Higham and Lijing Lin. A schur-padé algorithm for fractional powers of a matrix. SIAM Journal on Matrix Analysis and Applications , 32(3):1056-1078, 2011.

Einar Hille. On roots and logarithms of elements of a complex banach algebra. Mathematische Annalen , 136(1):46-57, 1958.

Bo Jiang, Ya-Feng Liu, and Zaiwen Wen. L_p-norm regularization algorithms for optimization over permutation matrices. SIAM Journal on Optimization , 26(4):2284-2313, 2016.

Bo Jiang, Jin Tang, Chris Ding, and Bin Luo. Nonnegative orthogonal graph matching. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 31, 2017.

Ivan Karpukhin and Andrey Savchenko. Detpp: Leveraging object detection for robust longhorizon event prediction. arXiv preprint arXiv:2408.13131 , 2024.

Ivan Karpukhin, Foma Shipilov, and Andrey Savchenko. Hotpp benchmark: Are we good at the long horizon events forecasting? arXiv preprint arXiv:2406.14341 , 2024.

Sunyoung Kim, Masakazu Kojima, and Kim-Chuan Toh. A lagrangian-dnn relaxation: a fast method for computing tight lower bounds for a class of quadratic optimization problems. Mathematical Programming , 156(1):161-187, 2016.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Virginia Klema and Alan Laub. The singular value decomposition: Its computation and some applications. IEEE Transactions on automatic control , 25(2):164-176, 1980.

Max Kochurov, Rasul Karimov, and Serge Kozlukov. Geoopt: Riemannian optimization in pytorch, 2020.

Vignesh Kothapalli. Neural collapse: A review on modelling principles and generalization. arXiv preprint arXiv:2206.04041 , 2022.

Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly , 2(1-2):83-97, 1955.

Eugene L Lawler. The quadratic assignment problem. Management science , 9(4):586-599, 1963.

John M Lee. Introduction to Riemannian manifolds , volume 2. Springer, 2018.

Mario Lezcano Casado. Trivializations for gradient-based optimization on manifolds. Advances in Neural Information Processing Systems , 32, 2019.

Mario Lezcano-Casado and David Martınez-Rubio. Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group. In International Conference on Machine Learning , pages 3794-3803. PMLR, 2019.

Innar Liiv. Seriation and matrix reordering methods: An historical overview. Statistical Analysis and Data Mining: The ASA Data Science Journal , 3(2):70-91, 2010.

Cong Han Lim and Stephen Wright. Beyond the birkhoff polytope: Convex relaxations for vector permutation problems. Advances in neural information processing systems , 27, 2014.

Scott Linderman, Gonzalo Mena, Hal Cooper, Liam Paninski, and John Cunningham. Reparameterizing the birkhoff polytope for variational permutation inference. In International Conference on Artificial Intelligence and Statistics , pages 1618-1627. PMLR, 2018.

Eliane Maria Loiola, Nair Maria Maia De Abreu, Paulo Oswaldo Boaventura-Netto, Peter Hahn, and Tania Querido. A survey for the quadratic assignment problem. European journal of operational research , 176(2):657-690, 2007.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712 , 2016.

Eleonora Maset, Federica Arrigoni, and Andrea Fusiello. Practical and efficient multi-view matching. In Proceedings of the IEEE International Conference on Computer Vision , pages 4568-4576, 2017.

Estelle Massart and Vinayak Abrol. Coordinate descent on the orthogonal group for recurrent neural network training. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 7744-7751, 2022.

Gonzalo Mena, David Belanger, Scott Linderman, and Jasper Snoek. Learning latent permutations with gumbel-sinkhorn networks. In International Conference on Learning Representations , 2018. URL https://openreview.net/forum?id=Byt3oJ-0W .

Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih. Monte carlo gradient estimation in machine learning. Journal of Machine Learning Research , 21(132):1-62, 2020.

Deepti Pachauri, Risi Kondor, and Vikas Singh. Solving the multi-way matching problem by permutation synchronization. Advances in neural information processing systems , 26, 2013.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Gao Peifeng, Qianqian Xu, Peisong Wen, Zhiyong Yang, Huiyang Shao, and Qingming Huang. Feature directions matter: Long-tailed learning via rotated balanced representation. In International Conference on Machine Learning , pages 27542-27563. PMLR, 2023.

Fidel A Guerrero Peña, Heitor Rapela Medeiros, Thomas Dubail, Masih Aminbeidokhti, Eric Granger, and Marco Pedersoli. Re-basin via implicit sinkhorn differentiation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20237-20246, 2023.

Janez Povh. Semidefinite approximations for quadratic programs over orthogonal matrices. Journal of Global Optimization , 48(3):447-463, 2010.

Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning , pages 1278-1286. PMLR, 2014.

Simone Rossi, Ankit Singh, and Thomas Hannagan. On permutation symmetries in bayesian neural network posteriors: a variational perspective. Advances in Neural Information Processing Systems , 36, 2024.

Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747 , 2016.

Rodrigo Santa Cruz, Basura Fernando, Anoop Cherian, and Stephen Gould. Deeppermnet: Visual permutation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3949-3957, 2017.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Richard Sinkhorn. A relationship between arbitrary positive matrices and doubly stochastic matrices. The annals of mathematical statistics , 35(2):876-879, 1964.

Ying Sun, Hengshu Zhu, Chuan Qin, Fuzhen Zhuang, Qing He, and Hui Xiong. Discerning decision-making process of deep neural networks with hierarchical voting transformation. Advances in Neural Information Processing Systems , 34:17221-17234, 2021.

Ying Sun, Fuzhen Zhuang, Hengshu Zhu, Qing He, and Hui Xiong. Cost-effective and interpretable job skill recommendation with deep reinforcement learning. In Proceedings of the Web Conference 2021 , pages 3827-3838, 2021.

Ying Sun, Hengshu Zhu, Lu Wang, Le Zhang, and Hui Xiong. Large-scale online job search behaviors reveal labor market shifts amid covid-19. Nature Cities , 1(2):150-163, 2024.

David M Tate and Alice E Smith. A genetic approach to the quadratic assignment problem. Computers & Operations Research , 22(1):73-83, 1995.

Lav R Varshney, Beth L Chen, Eric Paniagua, David H Hall, and Dmitri B Chklovskii. Structural properties of the caenorhabditis elegans neuronal network. PLoS computational biology , 7(2): e1001066, 2011.

Chao Wang, Hengshu Zhu, Chen Zhu, Chuan Qin, and Hui Xiong. Setrank: A setwise bayesian approach for collaborative ranking from implicit feedback. In Proceedings of the aaai conference on artificial intelligence , volume 34, pages 6127-6136, 2020.

Chao Wang, Hengshu Zhu, Chen Zhu, Chuan Qin, Enhong Chen, and Hui Xiong. Setrank: A setwise bayesian approach for collaborative ranking in recommender system. ACMTransactions on Information Systems , 42(2):1-32, 2023.

Runzhong Wang, Junchi Yan, and Xiaokang Yang. Neural graph matching network: Learning lawler's quadratic assignment problem with extension to hypergraph and multiple-graph matching. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(9):5261-5279, 2021.

Tianxin Wang, Fuzhen Zhuang, Ying Sun, Xiangliang Zhang, Leyu Lin, Feng Xia, Lei He, and Qing He. Adaptively sharing multi-levels of distributed representations in multi-task learning. Information Sciences , 591:226-234, 2022.

Junchi Yan, Xu-Cheng Yin, Weiyao Lin, Cheng Deng, Hongyuan Zha, and Xiaokang Yang. A short survey of recent advances in graph matching. In Proceedings of the 2016 ACM on international conference on multimedia retrieval , pages 167-174, 2016.

Michael M Zavlanos and George J Pappas. A dynamical systems approach to weighted graph matching. Automatica , 44(11):2817-2824, 2008.

He Zhang, Ying Sun, Weiyu Guo, Yafei Liu, Haonan Lu, Xiaodong Lin, and Hui Xiong. Interactive interior design recommendation via coarse-to-fine multimodal reinforcement learning. In Proceedings of the 31st ACM International Conference on Multimedia , pages 6472-6480, 2023.

Yuting Zhang, Ying Sun, Fuzhen Zhuang, Yongchun Zhu, Zhulin An, and Yongjun Xu. Triple dual learning for opinion-based explainable recommendation. ACMTransactions on Information Systems , 42(3):1-27, 2023.

Hengshu Zhu, Hui Xiong, Yong Ge, and Enhong Chen. Discovery of ranking fraud for mobile apps. IEEE Transactions on knowledge and data engineering , 27(1):74-87, 2014.

A Additional discussions

A.1 Related work

Optimization over permutations is commonly encountered in problems involving bijections between two sets of equal size [17, 42, 15, 12]. From a theoretical perspective, a well-known special case is the quadratic assignment problem, which has been drawing researchers' attention since its first formulation [37, 8, 44]. In practical terms, optimization over permutations is also arises extensively in various machine learning tasks, including ranking [20, 75, 67, 68], matching [2], tracking [43], recommendation [63, 74, 73], etc. Since our interest lies in addressing generic problems defined on permutation matrices, we will not discuss theoretical approaches for specific issues [11], such as exact methods [16], lower bound methods [30], and vertex-based methods [65].

Previous studies have proposed relaxing the permutation matrices to continuous spaces, including the Birkhoff polytope [17, 15, 42] and orthogonal group [72, 55, 27]. The Birkhoff polytope is the convex hull of all permutation matrices, while the orthogonal group serves as a natural embedding of permutation matrices in a differentiable manifold. Such relaxation techniques have proven to be very powerful, with successful applications to various problems such as seriation [1, 17, 41] and graph matching [15, 69, 71]. Most of the methods mentioned above rely on the explicit penalty term and are challenging to extend to probabilistic scenarios.

Recently, relaxation methods involving the Birkhoff polytope have made significant advancements, particularly in penalty-free optimization and probabilistic inference. For instance, Linderman et al. [43] propose a rounding transformation regulated by a temperature parameter, which rounds matrices towards the vertices of the Birkhoff polytope, i.e., permutation matrices. Similarly, Mena et al. [49] utilize the Sinkhorn operator [61] to transform matrices into the Birkhoff polytope, bringing them closer to permutation matrices under temperature control. Additionally, Grover et al. [20] suggest mapping vectors to unimodal row stochastic matrices, a subset of the Birkhoff polytope that removes the requirement of every column sum being equal to 1 . These works not only avoid introducing penalty terms but also provide probabilistic inference. However, providing equally effective relaxation methods within orthogonal groups remains an unexplored area. As mentioned in the main text, relaxation onto the orthogonal group possesses unique potential advantages. Therefore, our elementary work aims to address this gap.

A.2 Limitations

The first concerns the computational efficiency of OT4P when dealing with very large matrices ( n > 1000 ), as the cost of eigendecomposition becomes prohibitive. This could benefit from efficient implementations of eigendecomposition on GPUs. The second limitation involves the noise distribution used for re-parameterization in OT4P , which may not equally capture the latent permutation matrices. This can be improved by more carefully designing the noise distribution under the characteristics of the orthogonal group. The third challenge relates to the boundary issues in representing permutation matrices, hindering OT4P as a building block of deep neural networks. Indeed, such integration assumes all permutation matrices lie within U . We believe that an ideal (analytical) solution can be devised to relocate all permutation matrices back into U , given that they constitute a finite discrete set within the orthogonal group of order n .

A.3 Broader impacts

This work presents a novel differentiable transformation for relaxing permutation matrices onto the orthogonal group, which enables gradient-based (stochastic) optimization of problems involving permutation matrices. Given the theoretical nature of our work, we have not identified any direct ethical concerns or negative societal impacts related to our research. Our study may have the broader impacts for a variety of areas of machine learning, such as deep learning [62], data mining [64]. Similar impacts were observed in multi-task learning [70] and graph learning [19].

B Riemannian Geometry and Lie Group

This section briefly summarizes the key concepts of Riemannian geometry and the Lie group theory involved. For a comprehensive understanding, we recommend the standard textbooks Introduction to

Riemannian manifolds [38] and Lie groups, Lie algebras, and representations [21]. Additionally, the analysis of matrix functions can be found in Functions of matrices: theory and computation [23].

An n -dimensional manifold M is a topological space that can be locally approximated by the Euclidean space R n . At each point x ∈ M , there exists a tangent space T x M , an n -dimensional vector space, serving as a first-order local approximation of M around x . The Riemannian metric is a collection m := { m x | x ∈ M} of inner products m x ( · , · ) : T x M× T x M→ R . It induces a norm ∥·∥ x : T x M→ R defined by ∥ y ∥ x := √ m x ( y, y ) . The length L ( γ ) of a smooth curve γ : [ a, b ] →M is defined as L ( γ ) := ∫ b a ∥ γ ′ ( t ) ∥ γ ( t ) d t . The distance d ( x, y ) is set as the infimum of the lengths of all smooth curves between x and y in M . A geodesic is a smooth curve γ : [ a, b ] →M that the tangent vector γ ′ ( t ) is parallel transported along the curve γ w.r.t. the Levi-Civita connection, i.e., ∇ γ ′ ( t ) γ ′ ( t ) = 0 for all t ∈ [ a, b ] . With the basic concept of manifolds established, we can present the definitions of isomorphism, diffeomorphism, and isometry.

Definition 1 (Vector space isomorphism) . The vector spaces X and Y are called to be isomorphic if there exists a bijection α : X → Y that preserves addition and scalar multiplication.

Definition 2 (Diffeomorphism) . Given two differentiable manifolds M and N , a differentiable map α : M→N is a diffeomorphism if it is a bijection and its inverse α -1 is differentiable as well.

Definition 3 (Isometry) . Given two metric spaces ( X , m X ) and ( Y , m Y ) , a map α : X → Y is called an isometry if m X ( x 1 , x 2 ) = m Y ( α ( x 1 ) , α ( x 2 )) for all points x 1 , x 2 ∈ X .

A Lie group G is a differentiable manifold equipped with differentiable group multiplication and inverse operations. The tangent space at the identity e is known as the Lie algebra of G , denoted as g := T e G . For all g ∈ G , there exist diffeomorphisms given by the left translation L g ( x ) := gx ( ∀ x ∈ G ) and the right translation R g ( x ) := xg ( ∀ x ∈ G ) . The left and right translations lead to a vector space isomorphism that relates the tangent space to the Lie algebra. For left translation L g , the mapping (d L g ) e := g → T g G maps elements of the Lie algebra into the tangent space T g G , with its inverse mapping (d L g -1 ) g := T g G → g . Similarly, for right translation R g , there is mapping (d R g ) e := g → T g G and its inverse (d R g -1 ) g := T g G → g . A Riemannian metric m on the Lie group G is said to be left-invariant ( right-invariant ) if it renders each left (right) translation an isometry. When adopting a left-invariant (right-invariant) metric on the Lie group G , we can associate neighborhoods of the identity e with neighborhoods of any point g ∈ G using left (right) translation, and vice versa. A metric on the Lie group that is both left and right invariant is termed a bi-invariant metric. It is worth noting that compact Lie groups always possess the bi-invariant metric.

The Lie group we are interested in is the orthogonal group O( n ) , consisting of all n × n orthogonal matrices O satisfying O ⊤ O = OO ⊤ = I . We equip O( n ) with the canonical metric, which is a bi-invariant metric inherited from R n × n and defined as ⟨ A,B ⟩ F := trace( A ⊤ B ) , where ⟨· , ·⟩ F is the Frobenius inner product and trace( · ) is the trace of a matrix. The orthogonal group O( n ) is divided into two connected components depending on the value of the determinant, +1 or -1 . The connected component with determinant +1 forms a subgroup known as the special orthogonal group SO( n ) := { O ∈ R n × n | O ⊤ O = I, det O = +1 } . The Lie algebra so ( n ) of the Lie group SO(n) comprises n × n skew-symmetric matrices, expressed as so ( n ) := { A ∈ R n × n | A ⊤ = -A } . It is noteworthy that so ( n ) is a vector space with dimension n ( n -1) 2 . The Lie exponential expm( · ) , coinciding with the matrix exponential in the context of the matrix Lie group, exactly maps elements in so ( n ) to elements in SO( n ) , defined by

The series in the above equation converges for all matrices A .

In general, the matrix logarithm is expected to be the inverse function of the matrix exponential, meaning the logarithm of A is the solution B to the matrix equation expm( B ) = A . Since the complex logarithm is a multi-valued function, there may be an infinite number of matrices B that satisfy expm( B ) = A . If A has no eigenvalues on R -0 := { x ∈ R | x ≤ 0 } , then there exists a unique logarithm called the principal logarithm , denoted as logm( A ) . All eigenvalues λ of logm( A ) satisfy -π < Im λ < π . Assuming convergence of the series, the matrix logarithm logm( · ) can be

defined by Taylor series expansion as follows:

The series in the above equation converges whenever ∥ A -I ∥ F < 1 , where ∥ A ∥ F := √ ⟨ A,A ⟩ F represents the Frobenius norm induced by the Frobenius inner product.

Define the V := { A ∈ R n × n | Im λ k ( A ) ∈ ( -π, π ) , ∀ k } as the set of all matrices A ∈ R n × n for which all eigenvalues λ k satisfies -π < Im λ k < π . Let W := { A ∈ R n × n | λ k ( A ) / ∈ R -0 , ∀ k } denote the set of all matrices A ∈ R n × n without non-positive real eigenvalues. Then, we can verify that (prove in Appendix G.1):

logm(expm( A )) = A, ∀ A ∈ V ;

expm(logm( A )) = A, ∀ A ∈ W .

This indicates that the matrix logarithm is a locally inverse function of the matrix exponential.

C Boundary issues in representation of permutation matrix

In this section, we provide an in-depth analysis of the potential adverse effects on optimization stemming from the direct application of mapping ϕ ( · ) w.r.t. Equation (6) . Subsequently, we show that left-multiplying the result of Equation (6) by an orthogonal matrix, as described in Step I, effectively alleviates this problem.

Since a permutation matrix must be an orthogonal matrix, its eigenvalues λ k lie on the unit circle in the complex plane, i.e., | λ k | = 1 . Additionally, any permutation can be expressed as the product of cycles with disjoint supports. For a cycle of length m , its corresponding m × m submatrix satisfies P m = I , which implies that the eigenvalues of P admit the following form:

In Figure 4, we visualize the eigenvalues corresponding to cycles of lengths 2 , 3 , 4 , and 5 . It is clear that cycles of even length consistently possess an eigenvalue of -1 . Notably, the eigenvalues of a permutation matrix are composed of those from the submatrices corresponding to its contained cycles. Therefore, many permutation matrices have -1 as one of their eigenvalues.

As elucidated Lezcano-Casado and MartınezRubio [40], the representation in U := { A ∈ R n ( n -1) 2 | Im λ k ( A -A ⊤ ) ∈ ( -π, π ) , ∀ k } of permutation matrices with an eigenvalue of -1 precisely lies on the boundary of U . For problems where these permutation matrices serve as the optimal solution, it may cause the optimization path to deviate from U . A straightforward method is to left-multiply the result of Equation (6) by a random orthogonal matrix B ∈ SO( n ) , thereby shifting the boundary of U to other eigenvalues. Theoretically, this left translation L B ( O ) := BO ( ∀ O ∈ SO( n )) creates a diffeomorphism on SO( n ) that transforms the representation of the permutation matrix P in U from logm( P ) to logm( B ⊤ P ) .

1.0

0.5

0.0

0.5

1.0

Real

1.0

0.5

0.0

0.5

1.0

Imaginary

Cycle length 2

Cycle length 3

Cycle length 4

Cycle length 5

To validate the efficacy of the aforementioned approach, we conducted a simple empirical study. Given a random orthogonal matrix B with a determinant of 1 , we randomly generate 1000 permutation

matrices P , where odd permutations are projected to SO( n ) as done in Appendix D. We then estimate the probability of the eigenvalue -1 occurring in P and B ⊤ P , respectively. We examine permutation matrices ranging in dimension n from 3 to 50 . Each experiment is replicated five times and the results are presented in Table 3. The findings show that the left translation L B effectively relocates the majority of permutation matrices' representations into the interior of U .

Remark C.1 . A high-level idea involves dynamically adjusting B to ensure that the permutation matrix near the current iteration point consistently remains within the interior of U . This is a special case of dynamic trivialization proposed in Lezcano Casado [39].

D Theoretical details of the odd permutation

In this section, we provide the details of handling odd permutations and focus on elucidating the underlying theory.

To deal with the case where the permutation matrix P corresponds to an odd permutation, our core idea is to find an agent, ̂ P , of P within SO( n ) . In this way, we may move the orthogonal matrix O ∈ SO( n ) to the vicinity of ̂ P , and then restore the result to the neighborhood of P . This idea is formalized by the following method:

We seek an agent of P within SO( n ) , which is determined by solving the optimization problem:

The analytical solution to Equation (20) is given by ̂ P = PD , where D = diag( { 1 , . . . , 1 , -1 } ) is the identity matrix with the last column multiplied by -1 .

Given that ̂ P ∈ SO( n ) , we can, by substituting P with ̂ P in Equation (10), move the orthogonal matrix O ∈ SO( n ) to the vicinity of ̂ P , yielding ̂ O .

By right-multiplying by D ⊤ , we map ̂ O to the neighborhood of P , resulting in ˜ O = ̂ OD ⊤ .

We now derive the analytical solution to Equation (20). By the definition of the Frobenius norm, we can express:

The last equation uses the fact that both ̂ P and P are orthogonal matrices, i.e., trace( ̂ P ⊤ ̂ P ) = trace( P ⊤ P ) = n . Considering the singular value decomposition (SVD) [33], we have P = U Σ V ⊤ , where U and V are orthogonal matrices, and Σ is a diagonal matrix with non-negative real singular values on the diagonal. Since the orthogonal matrix singular value is 1 , there exists Σ = I . By leveraging the cyclic property of the trace, we have:

where Z := V ⊤ ̂ P ⊤ U is an orthogonal matrix with det Z = -1 . Hence, all elements z i,j of Z satisfy | z i,j | ≤ 1 and Z must have an odd number of -1 eigenvalues. Notice that the trace of a matrix is the sum of its elements on the main diagonal, Equation (21) is maximized when

Z = diag( { 1 , 1 , . . . , 1 , -1 } ) . Thus, the analytical solution to Equation (20) is ̂ P = ( DP ⊤ ) ⊤ = PD , where D = diag( { 1 , . . . , 1 , -1 } ) is the identity matrix with the last column multiplied by -1 .

Since D is an orthogonal matrix, right-multiplying by D becomes a right translation R D ( O ) := OD ( ∀ O ∈ O( n )) . Notably, O( n ) is equipped with a bi-invariant metric ⟨· , ·⟩ F , which implies that R D is an isometry, linking the the neighborhoods of P and its agent ̂ P . Thus, we have ⟨ AD,BD ⟩ F = ⟨ A,B ⟩ F for any A,B ∈ O( n ) . In other words, when ̂ O is moved close enough to ̂ P , ˜ O will also be sufficiently close to P .

E More discussion on Step II

Visualization. In Figure 5, we visualize the results of Equation (11) as the temperature parameter τ varies. The first row corresponds to the even permutation, while the second row corresponds to the odd permutation. The leftmost image ( τ = 1 . 0 ) represents the original orthogonal matrices O from Step I, and the rightmost image ( τ = 0 . 0 ) is the permutation matrix P that is closest to O . The temperature parameter τ controls how closely the resulting orthogonal matrices ˜ O , obtained in Step II, approach P . As τ → 0 , the resulting orthogonal matrices ˜ O increasingly converge to P .

=1.0

=0.7

=0.5

=0.3

=0.0

-0.5

0

0.5

1

even

odd

-1

Rounding to permutation matrix. Generally, the Hungarian algorithm requires a cost matrix without negative values, which is unsuitable for orthogonal matrices that may contain negative elements. To implement the rounding w.r.t. Equation (7) from the orthogonal matrix O , we first eliminate negative elements by subtracting the minimum element found within O , i.e., O -min( O ) . This approach is justified by the fact that arg max P ⟨ P, O ⟩ F = arg max P ⟨ P, O -min( O ) ⟩ F . Subsequent to this adjustment, we employ the Hungarian algorithm, available in existing libraries, to round O -min( O ) to the closest permutation matrix.

Computational costs. The primary computational costs of the proposed OT4P arise from solving the linear assignment problem and performing eigendecomposition, both of which typically scale with O ( n 3 ) . Numerous efforts have been made to accelerate these computations through parallel implementations on GPUs. We have employed existing implementations, specifically torch-linearassignment [28, 29] and torch.linalg.eig [52].

F Experimental details

F.1 Reproducibility and compute resources

To enhance the reproducibility, we provide a comprehensive overview of the experiments in Appendices F.2 to F.4, including but not limited to data generation, hyperparameters, evaluation procedures. For all experiments, we repeat the results using five different random seeds: 2021 , 2022 , 2023 , 2024 , and 2025 . The core code has been released at https://github.com/YamingGuo98/OT4P .

We conducted the experiments (Appendices F.2 and F.3) using a single NVIDIA A800 , where the runtime environment was Python=3.10 , CUDA=11.7 , and PyTorch=2.01 . The experiments in Section 4.1 took approximately 15 hours, while those in Section 4.2 consumed another 15 hours. This estimate does not include the time spent on hyperparameter search or other experiments conducted during the research process.

F.2 Finding mode connectivity

Steup. We investigate various network architectures for Equation (17), including 5-layer MLP [54], VGG11 [60], and ResNet18 [22] For MLP5, both input and output are set to 1 , with each of the four hidden layers having a dimension of 10 , and the hyperbolic tangent serves as the activation function. The model weights of MLP5 are initialized randomly, while VGG11 and ResNet18 are derived from the official pre-trained models in PyTorch [52]. We minimize the loss corresponding to Equation (17) using AdamW [45] with an initial learning rate of 0 . 1 and a maximum of 500 iterations. Following Peña et al. [54], we adopt an early stopping strategy upon finding the optimal permutation matrices.

Baseline. We take Weight Matching [2] and Sinkhorn [54] as baselines. Weight Matching is a method specifically designed for addressing Equation (17), which goes through each layer and greedily selects its best permutation matrix P i . We limit the maximum number of traversal rounds to 10 . Sinkhorn relaxes permutation matrices to the vicinity of the Birkhoff polytope utilizing the Sinkhorn operator [61, 49], solving Equation (17) in a differentiable fashion. The Sinkhorn operator undergoes 20 iterations, with the temperature parameter set to 0 . 5 .

Evaluation. During the evaluation phase, we round the matrices obtained from Sinkhorn and OT4P to permutation matrices using Equation (7). We employ the ℓ 1 -Distance, ∥ θ A -π ( θ B ) ∥ 1 , to measure the difference in weights with the target network. In addition, we flatten the permutation matrices and evaluate their alignment with the ground truth using Precision, Recall, and Hamming Distance.

F.3 Inferring neuron identities

Setup. Following the methodology in Linderman et al. [43], we generate parameters A , W , and P with n = 250 . Specifically, we randomly generate a binary upper triangular matrix A ∈ { 0 , 1 } n × n and its symmetric version A = A + A ⊤ as the adjacency matrix. Parameter W is sampled from a Gaussian distribution N (0 , 1) according to the sparse sparsity defined by A . The permutation matrix P is randomly sampled from a uniform distribution. We generate 1000 samples according to Y t = P ( A ⊙ W ) P ⊤ Y t -1 + ϵ , where the noise ϵ follows a Gaussian distribution N (0 , 0 . 01) .

We formulate tasks of varying difficulty depending on the different proportions of known neurons [43]. Conceiving a constraint matrix C ∈ R n × n where all elements are initialized to 1 . If we ascertain that the reference neuron i corresponds to the observed neuron j , then set all elements to 0 in the i -th row and j -th except for C i,j . Equation (22) provides a simple example, where we know that the observed neuron 1 (or 3 ) corresponds to the reference neuron 2 (or 5 ).

This modeling decision significantly reduces the number of latent permutations that need to be inferred. The constraint matrix C is enforced by zeroing corresponding entries, where for Naive it is before row normalization, for Gumbel-Sinkhorn it is before the Sinkhorn operator, and for OT4P it is before solving Equation (7).

Baseline. We compare Naive [43] and Gumbel-Sinkhorn [49]. Naive does not enforce P to be a permutation matrix but instead normalizes each row using the softmax function. Gumbel-Sinkhorn , an extension of the Gumbel-Softmax [26, 46] method for permutations, introduces Gumbel noise for re-parameterization before the Sinkhorn operator. Both Naive and our OT4P use the stand Gaussian noise for re-parameterization. All methods estimate the gradient of the marginal log-likelihood E P ∼ q ( P ; θ ) log p ( Y | P ) with 5 repeats.

Evaluation. We retain the best model throughout training, ranked first by Hamming Distance and then by the marginal log-likelihood (estimated with 5 repeats). We report the marginal log-likelihood of the best model. As done in Section 4.1, Precision, Recall, and Hamming Distance are utilized to evaluate the permutation matrices obtained from the best model without adding noise.

F.4 Solving permutation synchronization

Setup. We use the WILLOW-ObjectClass dataset [9] to generate problem instances. The WILLOWObjectClass dataset comprises images of five object classes, each containing 10 equal key points of at least 40 images. For each image, we extract interpolated features from the relu4_2 and relu5_1 layers through a pre-trained VGG16 [60] model on ImageNet [10]. The initial pairwise correspondences are established by applying the Hungarian algorithm [36] to the distance matrices of features. We increase the number of objectives, k , from 20 to the its largest value (multiples of 5 ) for each object class.

Permutationness. We also take the ℓ 1 -Distance to assess the 'permutationness' of the final matrix. Specifically, we round the matrix ˜ P , returned by the algorithms, to its closest permutation matrix P , and then calculate the ℓ 1 -Distance between ˜ P and P . Table 6 lists the results for the problem instances corresponding to the largest size (multiples of 5 ) in each object class. We observe that the relaxation extent of Sinkhorn is unstable. Unlike them, OT4P consistently maintains smaller distances in almost all cases and exhibits a positive correlation with changes in the hyperparameter τ .

Impact of Optimizers. We compare the results of different optimizers in Table 7, selecting the largest instances (multiples of 5 ) for each object class. Methods based on the Birkhoff polytope show notable performance improvements on most datasets when using (Riemannian) SGD. For our proposed OT4P , the choice of optimizer appears to be less critical, as it consistently outperforms other methods regardless.

Case study. This case study aims to explore the runtime and memory efficiency of our proposed OT4P on large-scale problems. We use the CMU House [5] image sequence to generate problem instances, comprising 111 frames of a video of a toy house. A total of 111 × 111 pairwise matching, with each image having 30 hand labeled landmark points, are provided in Bernard et al. [5]. We increase the number of objectives, k , from 20 to 110 and utilize the proposed OT4P to approximately solve Equation (18) on the NVIDIA GeForce RTX 3090 . For all instances, we conduct 100 iterations using AdamW [45] with an initial learning rate of 0 . 1 .

20

40

60

80

100

Instance Size

97.0

98.0

99.0

100.0

F-score (%)

F-score

OT4P(

=0.3

)

OT4P(

=0.5

)

OT4P(

=0.7

)

20

40

60

80

100

Instance Size

1.00

2.00

3.00

Time (s)

Runtime

OT4P(

=0.3

)

OT4P(

=0.5

)

OT4P(

=0.7

)

20

40

60

80

100

Instance Size

100

200

300

Memory (MB)

Memory usage

OT4P(

=0.3

)

OT4P(

=0.5

)

OT4P(

=0.7

)

We flatten the permutation matrices and evaluate their alignment with the ground truth using F-score, while also assessing the runtime and the maximum GPU memory consumption during training. Each experiment is conducted five times, and the results are depicted in Figure 6, where the size of problem instances varies along the horizontal axis. The findings indicate that OT4P can find satisfactory solutions (with F-scores exceeding 97% ) in less than 4 seconds, and the memory usage is manageable. Additionally, in Figure 7, we present the matching between the first and last images for k = 110 , where the obtained matchings are connected (correct: , incorrect: ). Overall, the

(c) OT4P ( τ = 0 . 5 )

(d) OT4P ( τ = 0 . 7 )

proposed OT4P demonstrates high efficiency in both runtime and memory usage when dealing with large-scale tasks amenable to parallel processing.

F.5 Licenses

We will publicly release our code under the MIT license, in accordance with community standards. The licenses for the code, data, and models used in this study are provided below. Please refer to individual links for more details.

PyTorch:

BSD-style

Torchvision:

BSD 3-Clause

torch-linear-assignment:

Apache

sinkhorn-rebasin: MIT

SparseStiefelOpt: GNU Affero General Public

geoopt:

Apache

G Proof

In this section, we first elucidate some key properties of the matrix exponential and the matrix logarithm. Based on these, we prove Theorems 1 and 2 in Appendix G.2. Finally, we list some essential lemmas used in the proofs.

G.1 Properties of matrix exponential and logarithm

Theorem G.1. The matrix exponential map expm( · ) is injective over V := { A ∈ R n × n | Im λ k ( A ) ∈ ( -π, π ) , ∀ k } .

Proof. For all A,B ∈ V , our goal is to prove that A = B if expm( A ) = expm( B ) .

According to Lemma G.1, if expm( A ) = expm( B ) , then A and B commute, i.e., AB = BA . In fact, this commutativity implies that the equation expm( A + B ) = expm( A ) expm ( B ) holds [23].

Thus, we can express

The above equation shows that all eigenvalues of expm( A -B ) are one. Since the eigenvalues of the exponential of a matrix are the exponential of its eigenvalues, the eigenvalues of A -B satisfy 2 mπi for integers m ∈ Z . However, given that A,B ∈ V , the range of Im λ k ( A -B ) is restricted to ( -2 π, 2 π ) for all k , yielding m = 0 . Thus, all eigenvalues of A -B are zero, which means A -B is a nilpotent matrix.

Recalling Lemma G.2, we know that under the exponential map, the only nilpotent matrix mapped to the identity matrix I is the null matrix. Therefore, we conclude that A -B = 0 , which implies A = B .

This confirms that the matrix exponential map is injective within the set V , thereby completing the proof.

Theorem G.2. The matrix logarithm map logm( · ) is injective over W := { A ∈ R n × n | λ k ( A ) / ∈ R -0 , ∀ k } , and logm( W ) ⊆ V .

Proof. We first prove that if A ∈ W , then B = logm( A ) ∈ V .

Assume A ∈ W with eigenvalues λ k = ρ k e iθ k , where ρ k > 0 and θ k are real number. As shown in Gallier [18], the complex eigenvalues of B = logm( A ) appear only for real Jordan blocks:

where the eigenvalues are given by log( ρ k ) ± iθ k . Since A has no eigenvalues on R -0 , it follows that θ k ∈ ( -π, π ) for all k . Thus, for any A ∈ W , we have logm( A ) ∈ V , that is, logm( W ) ⊆ V .

Recall that the matrix logarithm is defined as a solution to the equation expm( B ) = A . Coupled with Theorem G.1 and noting that logm( A ) ∈ V , then such a logarithm is unique. This proves that the matrix logarithm map is injective over V and completes the proof.

Corollary G.3. The image expm( V ) of V by the matrix exponential expm( · ) is the set W and the mapping expm : V → W is a diffeomorphism.

Proof. By Theorem G.2, it is clear that W ⊆ expm( V ) . We now prove expm( V ) ⊆ W .

For any matrix A ∈ V , the eigenvalues have the form a + ib , where a and b are real number and -π < b < π . Then, the eigenvalues of expm( A ) take the form e a + ib = e a e ib . Since e ib never lies on R -0 , expm( A ) has no non-positive real eigenvalues, i.e., expm( A ) ∈ W . The arbitrariness of A leads to the conclusion of expm( V ) ⊆ W .

The diffeomorphism results directly from Theorems G.1 and G.2.

G.2 Proof of Theorems 1 and 2

Theorem 1*. The mapping ϕ ( · ) is differentiable, surjective, and it is injective on the domain U := { A ∈ R n ( n -1) 2 | Im λ k ( A -A ⊤ ) ∈ ( -π, π ) , ∀ k } with λ k ( · ) the eigenvalues. Additionally, the set SO( n ) \ ϕ ( U ) has a zero Lebesgue measure in SO( n ) .

Proof. It is trivial that ϕ ( · ) is differentiable.

We start by establishing that ϕ ( · ) is surjective. It is worth mentioning that the conclusion of Lemma G.3 is applicable to SO( n ) due to it being connected and compact. However, we provide a straightforward proof based on Theorem G.1. Indeed, any orthogonal matrix O ∈ SO( n ) can be diagonalized into a block-diagonal matrix with diagonal blocks consisting of 2 × 2 rotation matrices:

where θ ∈ ( -π, π ] represents the rotation angle. For SO(2 n +1) , an extra block containing a single 1 exists. Correspondingly, we have a skew-symmetric matrix B ∈ so ( n ) satisfying expm( B ) = O , which is also a block-diagonal matrix with diagonal blocks of the form:

There is an additional block with a single 0 on so (2 n +1) . As a result, for all orthogonal matrices O ∈ SO( n ) , we can surely find a skew-symmetric matrix B ∈ so ( n ) such that expm( B ) = O . Given that so ( n ) is isomorphism to the vector space R n ( n -1) 2 , this confirms that ϕ ( · ) is surjective. Regarding ϕ ( · ) is injective on the domain U , it can be derived from Theorem G.1 because U ⊆ V .

The above reasoning also indicates that ϕ ( U ) ⊆ SO( n ) ⊆ ϕ ( ¯ U ) holds and that the boundary of U has zero Lebesgue measure. Since the matrix exponential map expm( · ) is injective within the interior of U and the complex exponential function is single-valued, the set SO( n ) \ ϕ ( U ) has a zero Lebesgue measure in SO( n ) .

Theorem 2*. The mapping ψ τ ( · ) is differentiable, surjective, and injective on each submanifold S P . Additionally, the set of meaningless points for ψ τ ( · ) has a zero Lebesgue measure in SO( n ) .

Proof. Since the right translation R D establishes an isometry between the neighborhoods of P and its agent ̂ P , we can, without loss of generality, restrict proof to the case where P = ρ ( O ) is an even permutation. In this case, we rewrite ψ τ as ϕ τ ( O ) = P expm( τ logm( P ⊤ O )) . According to Corollary G.3, if P ⊤ O is within W , then we can assert that ψ τ ( · ) is surjective and injective on each submanifold S P .

Assuming the eigenvalues of P ⊤ O are λ 1 , . . . , λ n , the eigenvalues of P ⊤ O -I are λ 1 -1 , . . . , λ n -1 . Since the inverse mapping of the left translation, L P ⊤ ( O ) := P ⊤ O ( ∀ O ∈ SO( n )) , maps O to a neighborhood around I , we may assume ∥ P ⊤ O -I ∥ F < 1 . A well-known result is r ( A ) ≤ ∥ A ∥ F for any A ∈ R n × n , where r ( A ) := max {| λ k ( A ) | | ∀ k } is the spectral radius. As a result, we have:

Equation (23) shows that the eigenvalues of P ⊤ O are within a unit circle centered at 1 in the complex plane. Therefore, P ⊤ O has no eigenvalues on R -0 , meaning P ⊤ O ∈ W .

Since ρ ( · ) w.r.t. Equation (7) is a piecewise constant function, it follows that the set of points where ψ τ ( · ) is meaningless has a zero Lebesgue measure. This completes the proof.

G.3 Lemmas used

Lemma G.1 (Theorem 4. in Hille [25]) . Let A,B ∈ C n × n . If there are no two eigenvalues in A such that their difference is of the form 2 kπi for k > 0 and, if expm( A ) = expm( B ) , we have that AB = BA .

Let Nil( k ) denote the set of (real or complex) nilpotent matrices, A , of any dimension n ≥ 1 such that A r = 0 and Uni( k ) denote the set of unipotent matrices, B = I + A , where A ∈ Nil( k ) .

Lemma G.2 (Proposition 3.2. in Gallier [18]) . The map expm : Nil( k ) → Uni( k ) is a homeomorphism whose inverse is the matrix logarithm.

Lemma G.3 (Corollary 11.10. in Hall [21]) . If G is a connected, compact matrix Lie group, the exponential map for G is surjective.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims accurately reflect the contribution and scope of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations in Appendix A.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: In the Appendix G, we provide complete proofs for all theoretical results and list some essential lemmas used in the proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In Appendix F.1, we provide all the experimental details, the random seeds used and the runtime environment.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The core code has been released at https://github.com/YamingGuo98/ OT4P .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide a comprehensive overview of the experiments in Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the experimental results using five different random seeds.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Appendix F.1 provides information on compute resources.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the potential social impacts of this work in Appendix A.3 Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Appendix F.5 have included the licenses for the assets used in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This does not apply on our research work.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This work does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This work does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new method for relaxing permutation matrices onto the group of orthogonal matrices. A temperature controlled differentiable transform maps the permutations onto O(n) and this allows for adjusting the strength of regularity vs the problem difficulty. With this relaxation, the paper employs continuous optimization methods to solve various problems including permutation synchronization, a notoriously challenging non-convex problem. The experiments demonstrate a consistent advantage over (Gumbel-) Sinkhorn.\n\nSTRENGTHS:\nFirst of all, optimization over permutations is a very important problem, remaining yet to be solved. To this end, this paper introduces an awesome idea as well as a sensible, well-thought approach to implement it. I especially like the fact that, similar to entropic-OT, the temperature parameter allows for an interpretable control over being a generic orthogonal matrix vs. being a permutation. I also checked the code and it seems reasonable.\n\nWEAKNESSES:\nThere are several clarifications and evidence needed to make the paper fully convincing:\n\n- Permutations are the only strictly positive orthogonal matrices. Therefore, all relaxed matrices in this paper will inherently include some negativity. While acceptable for optimization, this poses a challenge when rounding results back to permutations. This raises two questions:\n    1. How do the authors implement rounding, precisely? In the experimental results, it would be beneficial to see the 'permutationness' of the final matrix, possibly in comparison to accuracy or loss. This insight would inform practitioners about important design choices.\n    2. Why constrain ourselves to the orthogonal group and not introduce temperature directly in $\\mathbb{R}_{+}(n)$? What makes orthogonal matrices special? This point is only partially explained.\n  \n- I'm uncertain about the transformation of odd permutations. $O(n)$ has a disconnected topology, unless it's $SO(n)$ (considering reflections), and transforming to the interior of $\\mathcal{U}$ can distort and change geodesic distances, crucial for optimization. The paper appears to use $O(n)$ and $SO(n)$ interchangeably, which might not be good practice. The consideration of reflections is essential due to potential distortions caused by mapping some matrices into the interior of $\\mathcal{U}$. Are we assuming a connected topology? These points confuse me.\n  \n- Permutation synchronization, a new experiment mentioned only in the supplementary materials, deserves mention in the main text. Also note that, Birdal and Simsekli [5] use the Riemannian geometry of the doubly stochastic matrices to solve permutation synchronization. They also control the strength by analytically deriving the prior induced by the manifold-objective. This work should be discussed, and hopefully compared. \n  \n- The paper lacks another simple baseline: relax permutations onto orthogonal matrices (classical) and use a regularization term to enforce permutation-ness, controlled by coefficient $c$. As $c \\to 0$, optimization on $O(n)$ is naive, while as $c \\to \\infty$, strict permutation-ness makes the problem challenging. This approach could be included in comparisons.\n  \n- In the proposed approach, $T \\to 0$ can cause numerical issues. At what point does this become problematic and unfeasible? Intuitively, why is this approach preferable to entropy-regularized optimal transport, which seems similar in spirit? The paper should discuss or at least cite:\n*Cuturi, Marco. \"Sinkhorn distances: Lightspeed computation of optimal transport.\" Advances in neural information processing systems 26 (2013).*\n\n- Can we see results from different optimizers? Depending on their behavior, advantages might be amplified or diminished. For instance, [5] used Riemannian LBFGS. Note that Adam might not be the most suitable optimizer for such problems. \n  \nMinor issues:\n- Ln. 223: \"is stay\" -> \"stays\"\n- The link for CMU house seems to be broken. Where did authors obtain the dataset?\n\nQUESTIONS:\nPlease see the weaknesses section for most of my questions. In particular, I would be happy if the authors can address:\n- Distinguishing SO(n) from O(n) and stating the impact of the choice, especially in context of the proposed transformation.\n- Some evidence/evaluation/justification that the transformation does not pose a problem for geodesic distance computation.\n- Comparisons and discussions to [5] and the simple regularization baseline (see weaknesses).\n\nLIMITATIONS:\nThe paper discusses the limitations in Appendix A. I would appreciate if some of those would be moved to the main paper. Especially the boundary issues can be discussed within the main text. An appropriate broader impact statement is added to the paper (appendix). I see no concerns with that.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a parameterization of $n\\times n$ permutation matrices by $n(n-1)/2$ unconstrained numbers. This parameterization eases the training of neural networks and is applicable to tasks involving optimization over permutations.\n\nSTRENGTHS:\n- The paper is well-written and easy to follow.\n- The initial idea is very simple: Replace the doubly stochastic relaxation with the orthogonal relaxation. Following this idea, the paper develops good intuition and solid theorems in terms of injective/surjectivity of the parametrization.\n- The experimental results look great, presenting the advantages of the proposed method in some aspects.\n\nWEAKNESSES:\n- Some design choices and technical development do not seem to be very motivated;\n   - What's the role of the temperature parameter $\\tau\\in(0,1]$? What's wrong if we just set it to $0$ and we just want the closest permutations? In fact, the experiments show that smaller $\\tau$ seem to be better (see Table 5). If $\\tau$ is not needed (and if I understand this correctly), then there would not be the problem of odd permutations and there would be no need to introduce $D$ and the remedy.  \n   The reader may be unclear about why special orthogonal groups are used for parameterization. Why not just use orthogonal groups? It would be nice to have this explained in the paper.\n\n- Can authors provide results for the case \"Known 0%\" in Table 2? This is important as it gives the reader a better understanding of the importance of having prior information. \n\n- The running time of the method is unclear and the scalability claim does not seem to be true. \n   - First, the two costly steps of the proposed method are eigenvalue decomposition and solving the linear assignment problem, and, in my experience, the latter is often much slower for various reasons, e.g., eigenvalue decomposition has high-quality implementations and is more amenable to parallel execution. Also, the method of Jonker & Volgenant (1987) [1] is much faster than the Hungarian algorithm. Both steps should be fast for matrices of size 1000x1000. Could the authors comment on why their implementation is slow for $n>1000$ (as mentioned in Appendix A.2)?  Furthermore, matrices of such sizes are not \"very large\"; hence, I think the claim that the proposed method is scalable is an overstatement.\n   - Could the authors compare the proposed method's running time (time complexity) with prior works?\n\n[1] A shortest augmenting path algorithm for dense and sparse linear assignment problems\n\n\n- Some typos or confusion:\n   - In the definition of the domain $U$ of Theorem 1, what is Im? Does it mean the imaginary part of the eigenvalues? It would be nice to define them.\n   - In Eq. 14, the optimization problem is not defined clearly. What are the optimization variables, and is $\\theta$ the parameter of $f$ as well?\n\nQUESTIONS:\nBesides the above, one extra question:\n- Eq. 7 is a linear assignment problem. But since the data matrix is orthogonal, is it possible to have an algorithm faster than cubic time?\n\nLIMITATIONS:\nSee the above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes OT4P, a differentiable transformation that relaxes permutations to the orthogonal group. Based on OT4P, the authors propose novel frameworks for deterministic and stochastic optimization on permutation matrices. Numerical experiments demonstrate its efficiency and scalability in permutation matrix optimization tasks.\n\nSTRENGTHS:\nThe paper is generally well written despite a few confusing sentences. The authors highlight the contributions and make a clear comparison with previously known results, and the main idea is interesting. Mathematical ideas are hard to clarify, but I find the paper easy to follow. The numerical experiment supports the claim that OT4P enjoys flexibility, simplicity and scalability on selected tasks.\n\nWEAKNESSES:\nExperiments are insufficient and do not fully convince me of the superiority of OT4P. It's hard to convince me that OT4P is valuable in real settings, therefore the impact of this work in practice is questionable. \n- The experiments in the main text are both synthetic. Why are they important? \n- Permutation synchronization is only tested on CMU house dataset, and it is only tested to demonstrate the runtime and memory efficiency. You can try more challenging multi-object matching datasets such as Willow[1]. It would be interesting to see if OT4P avoids unreliable local minima and achieves a better matching result, compared to Birkhoff polytope-based methods. \n\n- For permutation synchronization, convex relaxation methods are known to be slow, but you are using parallel computation to accelerate. A detailed explanation on this is necessary. \n\nThere are a few confusing sentences and some room of improvement in writing. \n- Line 223: 'is stay' should be 'stays'. \n- Not sure how to read figure 4. It would be nice to have some explanation. \n- The manifold \\mathcal{M} is an important mathematical object in the paper that relates directly to parameterization in optimization tasks. It is helpful to have more explanation on how it changes with the temperature parameter. \n\n[1] M. Cho, K. Alahari, and J. Ponce. Learning graphs to match. In Proceedings of the IEEE Interational Conference\non Computer Vision, 2013.\n\nQUESTIONS:\n- Do you have any guidance on how to choose the hyperparameter \\tau? It seems that \\tau=0.5 is optimal for all tasks in the experiment section. Should we just fix \\tau=0.5 for practical use?\n\nLIMITATIONS:\nDespite several limitations discussed in the appendix, other limitations of this work include: \n- In real world computer vision problems, people usually have partial permutations instead of permutations, since a keypoint may not be observed in some images. It would be more interesting if this work can be extended to optimization on partial permutations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-29"}