Title: Kissing to Find a Match: Efficient Low-Rank Permutation Representation
Venue: NeurIPS 2023
Accepted: yes
Keywords: low rank

permutation

kissing number

matrix factorization

assigment problem
Abstract: Permutation matrices play a key role in matching and assignment problems across the fields, especially in computer vision and robotics. However, memory for explicitly representing permutation matrices grows quadratically with the size of the problem, prohibiting large problem instances. In this work, we propose to tackle the curse of dimensionality of large  permutation matrices by approximating them using low-rank matrix factorization, followed by a nonlinearity. To this end, we rely on the Kissing number theory to infer the minimal rank required for representing a permutation matrix of a given size, which is significantly smaller than the problem size. This leads to a drastic reduction in computation and memory costs, e.g., up to $3$ orders of magnitude less memory for a problem of size $n=20000$, represented using $8.4\times10^5$ elements in two small matrices instead of using a single huge matrix with $4\times 10^8$ elements. The proposed representation allows for accurate representations of large permutation matrices, which in turn enables handling large problems that would have been infeasible otherwise. We demonstrate the applicability and merits of the proposed approach through a series of experiments on a range of problems that involve predicting permutation matrices, from linear and quadratic assignment to shape matching problems.

Kissing to Find a Match: Efficient Low-Rank Permutation Representation

Abstract

Permutation matrices play a key role in matching and assignment problems across the fields, especially in computer vision and robotics. However, memory for explicitly representing permutation matrices grows quadratically with the size of the problem, prohibiting large problem instances. In this work, we propose to tackle the curse of dimensionality of large permutation matrices by approximating them using low-rank matrix factorization, followed by a nonlinearity. To this end, we rely on the Kissing number theory to infer the minimal rank required for representing a permutation matrix of a given size, which is significantly smaller than the problem size. This leads to a drastic reduction in computation and memory costs, e.g., up to 3 orders of magnitude less memory for a problem of size n = 20000 , represented using 8 . 4 × 10 5 elements in two small matrices instead of using a single huge matrix with 4 × 10 8 elements. The proposed representation allows for accurate representations of large permutation matrices, which in turn enables handling large problems that would have been infeasible otherwise. We demonstrate the applicability and merits of the proposed approach through a series of experiments on a range of problems that involve predicting permutation matrices, from linear and quadratic assignment to shape matching problems.

1 Introduction

Permutation matrices, which encode the reordering of elements, arise naturally in any problem that can be phrased as a bijection between two equally sized sets. As such, they are fundamental to many important computer vision applications, including matching semantically identical key points in images [50, 47, 48, 49], matching 3D shapes or point clouds [18, 46, 27], estimating scene flow on point clouds [35] and solving jigsaw puzzles [28], as well as to various sorting tasks [1, 16].

̸

Apermutation p , corresponding to the bijection from the set { 1 , . . . , n } onto itself, can be represented efficiently without a permutation matrix by merely enumerating the n elements

However, this representation is unsuitable for most computer vision problems that involve estimating p through optimization since this representation (i) is inherently discrete, yielding combinatorial problems for which no natural relaxation exists, and (ii) induces a solution space with a meaningless distance metric, as element i in the set generally is not 'closer' to element i +1 than it is to any other element j .

As a result, almost all methods for predicting permutations, including learning-based methods, favor a permutation matrix representation instead, i.e., formulating a permutation as an element in the set

with p ( i ) = j in representation (1) corresponding to P ij = 1 in the matrix representation form (2), which allows predicting p via optimization methods. Yet, the advantages of the matrix form representation (2) come at the cost of a prohibitive increase in memory, as it requires storing n 2 binary numbers P ij ∈ { 0 , 1 } , or - after commonly used relaxations - even n 2 -many floating point numbers instead of the n integers in (1). This renders matching problems with n > ∼ 10 4 largely infeasible as their corresponding permutation matrix P constitutes over one hundred million entries.

̸

To handle large matrices whose size prohibits explicit processing and storage, existing approaches typically either turn to sparse representations , i.e., storing only a small portion of matrix values in the form of ( i, j, P i,j ) triplets, where P i,j = 0 , or employ low rank representations , i.e., forming a large matrix P as a product of matrices

with V, W ∈ R n × m and m << n . Unfortunately, neither of these approaches is applicable to permutation matrices: sparse representation cannot be used as the sparsity pattern is not only unknown a-priori but actually the sought-after solution to the problem. On top of that, since permutation matrices are by definition full rank, a low-rank representation (3) can yield only a crude approximation at best.

Contributions. In this work, we alleviate the limitation on problem size by harnessing the wellstudied problem of (bounds for) the so-called Kissing number , which, in practice, translates to introducing a simple adaptation to the matrix factorization approach (3). In particular, we exploit the fact that for row-normalized matrices V and W , the entries of V W T correspond to the cosines of the angles between the matrix rows. We then apply a pointwise non-linearity on the product of the matrices in (3), which allows representing any permutation while using m << n . We use the Kissing number theory to provide an estimate for how small an m we can use. We elaborate on these theoretical considerations in Section 3 and provide an illustration of the geometric intuition for our approach in Fig. 1.

We then demonstrate the applicability of the proposed approach through several numerical experiments tackling various problems that involve estimating permutations, including a study on point alignment, linear assignment problems (LAPs), quadratic assignment problems (QAPs), and a realworld shape matching application. We find that the proposed approach trades off only little accuracy to offer significant memory saving, thus enabling handling bijection mapping problems that are larger than was previously possible, when full permutation matrices had to be stored.

2 Related Work

2.1 Permutation Learning and Representation

Permutation learning aims to develop a model capable of predicting the optimal permutation matrix that matches a given input data or label. Previous studies have suggested relaxing permutation matrices to continuous domains, such as the Birkhoff polytope [2], to approximate solutions for these optimization problems but still faced the problem of enforcing the sum-to-one constraints, which are commonly approximated using Sinkhorn layers [34]. In 2011, Adams et al. [34] proposed the Sinkhorn propagation method, with the goal of iteratively learning doubly-stochastic matrices that allow the prediction of permutation matrices. Following the same research line, Cruz et al. [38] proposed Sinkhorn networks, followed by Mena et al. [28] proposing Gumble-Sinkhorn networks, adding Gumbel noise to the Sinkhorn operator to learn latent permutations. Grover et al. [16] made further efforts in the relaxation of a learned permutation matrix and propose to relax the matrix to become unimodal row-stochastic, meaning that the rows must maintain a total sum of one while having a distinct argmax. More recent studies suggest to circumvent the constraint of row and column sums being one by predicting permutations in Lehmer code, whose matrix form results to be row-stochastic [9]. Other works propose alternative representations of permutations in different domains, as to work in the Fourier domain to give a compact representations of distribution over permutations [17, 20], or embed them to the surface of a hypersphere [33].

2.2 Nonlinear Matrix Decomposition

In previous work on nonlinear matrix decomposition, Saul [40] aims to find a low-rank matrix L that approximates a sparse nonnegative matrix P by applying an elementwise nonlinearity σ to L , i.e. P ≈ σ ( L ) , and proposed an alternating minimization method to solve for L directly. In further work, [39], Saul analyzed the geometric relationship between the sparse and low-rank matrices, specifically for σ as a rectified linear unit (ReLU). For ReLU as nonlinearity, Seraghiti et al. [42] subsequently proposed an accelerated method of [40] to solve this problem, and a method, working with a version on L = V W , derived from the product of two matrices. In our work, we exploit this problem of low-rank approximation for permutation matrices and show its relationship to the Kissing number.

2.3 Assignment Problems

The goal of assignment problems is to find a permutation between two ordered sets while minimizing the assignment costs. The two most common versions are the linear and quadratic assignment problem (LAP and QAP) which are based on element-wise and pair-wise costs, respectively. An LAP can be solved in cubic time using the Hungarian algorithm [22]. The QAP was first introduced by Koopmans and Beckmann [21] and can be written as min P ∈P n tr ( APBP ⊤ ) + tr ( C ⊤ P ) , for A,B ∈ R n × n and C ∈ R n × n being a problem where A ∈ R n × n is the cost function between elements of the first object to match, B ∈ R n × n is the distance function between elements in the second object and C ∈ R n × n is a point-wise energy. The QAP has been proved to be NP-hard so no polynomial time solution can be expected for general cases. As a result, many relaxations of the problem exist, for example by relaxing the permutation constraint [14, 36], or by lifting the problem to higher dimensions [18, 51]. A survey on various relaxation approaches can be found in [24]. While the relaxations do ease some aspects of the problems, they normally do not decrease the dimensionality of the problem which remains demanding for large n .

2.4 3D Shape Correspondence

3D shape correspondence is also often posed as an assignment problem between the sets of vertices of a pair of shapes, for example through point descriptors matched by an LAP or in an QAP aiming to

preserve distances between all point pairs. However, 3D shapes are often discretized with thousands of vertices which makes optimization for a permutation computationally challenging. Hence, the permutation constraint is often relaxed [36] and, even though the tightness of relaxation might be known [1, 10], the optimization variables still scale quadratically with the problem size. In [13] and [46] the QAP is deconstructed into smaller problems and then each of them is optimized with a series of LAPs, while [41] solve for permutations as a series of cycles that gradually improve the solution.

Because permutation constraints for large resolution become infeasible, and, hence, the restriction to cases with the same number of vertices, recent methods often do not impose these constraints at all. The functional maps framework [30] converts the output to a mapping between functions on the shapes instead of vertices and can reduce the dimensionality drastically by using the isometry invariance property of the eigenfunctions of the Laplace-Beltrami operator [32]. Other lines of work rely on a given template to constrain the solution [15, 44], impose physical models that regularizes the deformation between inputs [11, 12, 31], or learn a solution space through training [8, 23, 26]. However, with the exception of template-based models, these cannot guarantee permutations.

3 Low-Rank Permutation Matrix Representation

Acommon approach to solve optimization problems with costs E over the set of permutation matrices P n (including those arising from training neural networks for predicting assignments) is to relax the problem by replacing P n by its convex hull conv ( P n ) , i.e., the set of doubly-stochastic matrices:

Since P grows quadratically in n , has an unknown sparsity pattern, and the true solution is always full rank, such problems pose significant challenges for large n . In this work, we make the interesting observation that a non-linearity as simple as a rectified linear unit (ReLU, denoted by σ ) is sufficient not only to restore a full rank, but to represent any permutation matrix exactly. More precisely, we propose to replace the set conv ( P n ) in (4) with the set K m ( P n ) = { σ (2 V W T -1) | V, W ∈ R n × m } and use the so-called Kissing number [4, 29, 52] to show that P n ⊂ K m ( P n ) for a surprisingly small m . Let us first formalize our approach by defining the Kissing number :

Definition 1. For a given m ∈ N , we define the Kissing number Kiss ( m ) as

̸

Note that the Kissing number can be interpreted geometrically as the maximum number of points that can be distributed on an m -dimensional unit sphere such that the angle formed between each pair of different points is at least arccos(0 . 5) . This property quickly establishes P n ⊂ K m ( P n ) :

Proposition 1. Let P ∈ P n be an arbitrary permutation matrix, and let σ , σ ( x ) = max( x, 0) denote a rectified linear unit (ReLU). Then for every m such that n ≤ Kiss ( m ) there exist V, W ∈ R n × m such that

Proof. Let V ∈ R n × m be a matrix that satisfies the equalities and inequalities of (5), and let W = PV . Then it holds that

̸

̸

Consequently which proves the assertion.

To determine the minimal rank m that is required for representing a permutation of n elements, we rely on extensive studies in the past few decades which computed either exact values or lower and upper bounds for different values of m [7].

Using P = σ (2 V W T -1) for relaxing (4) yields a relaxation that requires only 2 mn instead of n 2 parameters, with m << n = Kiss ( m ) . For instance, Kiss (24) = 196560 implies that matrices of rank m = 24 are sufficient for representing any arbitrary permutation matrix of up to n = 196560 elements, thus requiring ∼ 4000 times less storage memory: 2 · 24 · 196560 instead of 196560 2 parameters. Furthermore, P n ⊂ K m ( P n ) ensures that - in stark contrast to direct low-rank factorization any permutation matrix can still be represented exactly . Empirically, the optimization over parametrizations σ (2 V W T -1) turned out to cause significant challenges, likely due to the non-convexity and non-smoothness of the problem. To alleviate this problem, we resort to a smoother version of (6) which can still approximate permutations to an arbitrary desired accuracy:

Proposition 2. Let P ∈ P n and g denote an arbitrary permutation matrix and an arbitrary entrywise strictly monotonically increasing function, respectively, and let s denote the row-wise Softmax function s ( A ) i,j = exp A i,j ∑ k exp A i,k . Then ∀ n ≤ Kiss ( m ) and ∀ ϵ > 0 there exist V, W ∈ R n × m and α > 0 , such that

Proof. Similar to the proof in Proposition 1, we start by choosing V satisfying (5) and setting W = PV to obtain

̸

.

Then ∀ i, j, k s.t. P ij = 1 and k = j (i.e., P ik = 0 ) it holds that g ( V W T ) ij > g ( V W T ) ik . Finally, to yield the assertion we use the Softmax property of converging to the unit vector in the limit s ( αA i, : ) α →∞ → e j (with j = arg max A i, : ), by taking α > 0 to be large enough.

In practice, we use g ( x ) = 2 x , in accordance with the representation in (6). We use this smoother version to validate the proposed low-rank representation for handling large matching problems in the experiments we report next. Please notice that this is only a proof of existence of a decomposition into V, W , not every V, W lead to a permutation and an optimisation is not guaranteed to find a correct decomposition.

4 Experiments

The following experiments validate our efficient permutation estimation method for different applications, and they confirm the ability to scale to very large problem sizes. First, as a proof of concept, we demonstrate our approach on the application of point cloud alignment for the two non-linearities proposed in Section 3 and introduce our sparse training technique. We then validate the effectiveness of our approach in the context of linear assignment problems and show how to handle sparse cost matrices. We perform further experiments in the context of generic NP-hard quadratic assignment problems, and integrate our approach into a state-of-the-art shape matching pipeline, thus providing the same level of accuracy while enabling a higher spatial resolution.

4.1 Implementation Details

We use the PyTorch Adam optimizer [19] with its default hyperparameters in all our experiments. The estimation of V and W is performed in parallel.

Stochastic Optimization. Fully benefiting from our proposed compact representation requires the costs E (or an approximation thereof) to be evaluated without ever forming the full (approximate) permutation matrix , as this step would inherently return to necessitate n 2 many entries. To this end, we introduce the concept of stochastic optimization , which - for our softmax-based representation s (2 αV W T ) arising from Proposition 2 - is not a stochastic training in a classical sense: we propose to fix all but two entries in each row of our approximate permutation. Specifically, in any supervised (learning-based) scenario where it is known that the y i -entry of the i -the row of the final permutation P ought to be equal to one, each step of our optimizers merely computes the y i -th and one randomly chosen ( r i -th entry) of each row, and computes the softmax s on these two entries only while implicitly assuming P i,j = 0 for j / ∈ { y i , r i } , i.e.,

In the above, we used W [ y i ,r i ] , : to denote the 2 × m matrix consisting of the y i -th and the r i -th row of W . Our stochastic approach requires the computation of 2 n entries per gradient descent iteration only and - by randomly choosing the r i - manages to still approximate the desired objective well.

Normalization of V and W . Since propositions 1 and 2 rely on row-normalized matrices, we explicitly enforce this constraint whenever we compute P , by using V i, : ← 1 ∥ V i, : ∥ V i, : , W i, : ← 1 ∥ W i, : ∥ W i, : . We omit this step from the presentations below for the sake of readability.

Softmax Temperature. Since the values of ⟨ V i, : , W j, : ⟩ are bounded by one following the aforementioned normalization, the temperature parameter α determines the trade-off between approximating a hard maximum (as required for accurately representing permutations, see Prop.2) and favorable optimization properties (i.e., meaningful gradients). We specify the schedule (constant or monotonically increasing) in each of the experiments below.

4.2 Point Cloud Alignment

As a proof of concept, we demonstrate that our proposition is correct and the optimization process converges. We explore the different choices of non-linearity, starting with ReLU and continuing with Softmax, using the task of predicting a linear transformation over point clouds. In this task we aim to match a point cloud X 1 ∈ R n × m consisting of n m -dimensional points, uniformly distributed on the unit hyper-sphere, to its linearly transformed and randomly permuted version X 2 ∈ R n × m . To obtain X 2 we multiply X 1 by a randomly drawn matrix Θ GT ∈ R m × m and apply a random permutation. Then, we optimize over the estimated transformation matrix Θ which in this experiment defines our permutation matrix P (Θ) :

Note that our representation in (6) is fully parameterized by Θ , with V = X 1 and W (Θ) = X 2 Θ , and V and W are row-wise normalized in each iteration. Here P (Θ) is equal to the correct permutation if Θ correctly aligns the point clouds, i.e. minimizes the angle between corresponding points in V and W (Θ) while maximizing the angles between non-corresponding points.

We solve for the permutation by performing 20000 minimizing steps with a learning rate set to 0 . 01 over the negative log-likelihood loss

where y i is the index of the point in X 2 which corresponds to the i th point in X 1 . We experiment with different numbers of points n , each time choosing the dimension m to be just big enough to satisfy the Kissing number constraint from Proposition 1, i.e., Kiss ( m ) ≥ n > Kiss ( m -1) . To check that we were able to find the correct transformation matrix Θ - and therefore the correct permutation matrix P - through optimization, we verify that the nearest neighbor (closest point) for each row i in V is located in row j of matrix W that satisfies P i,j = 1 . We find that this is indeed the case in all experiments with different numbers of points n ∈ { 10 , 100 , 1000 , 10000 } , thus establishing that we could reach the correct permutation through optimization. We achieve equally good results when replacing the point-wise non-linearity ReLU with Softmax P (Θ) = s ( 2 αV W (Θ) T ) .

Due to the quadratically growing size of the permutation matrix with an increasing number of points, we further propose to optimize for the permutation matrix stochastically, as described in Section 4.1. We ran experiments with similar settings as above, wherein we gradually increased the value of the temperature parameter α linearly during optimization from α = 5 · 10 -5 to α = 1000 . In these experiments, we again found that each point was paired with its corresponding nearest neighbor. Also, we could reduce the memory consumption, as shown in Fig. 2.

4.3 Point Cloud Alignment on Spectral Point Representation

We conduct an additional experiment on point cloud alignment in the context of the functional maps framework [30]. Here, the goal is to extract a point-to-point correspondence between two shapes X,Y from a m × m -dimensional functional map C [30] where m is much smaller than the number of vertices in X and Y . A possible interpretation of C is that it aligns the spectral point representations Φ X , Φ Y ∈ R n × m in which each point x is represented by the vector of values

of the first m Laplace-Beltrami eigenfunctions at x such that P · Φ X ≈ Φ Y · C where P is the unknown permutation between X and Y . Given C , P can be retrieved by a nearest-neighbor query between Φ X , Φ Y C , as proposed in the original paper (see [30], Section 6.1), or by solving a Linear Assignment Problem (LAP) if a bijection is desired. This is exactly the same setting as in in Section 4.2 with a small amount of noise in the point clouds. We use the FAUST registrations [3] with the original 6890 vertices, a downsampled version to 502 vertices for those experiments and C generated by the ground-truth correspondence. Then, Φ X , Φ Y C can be directly used for V and W in our method.

We compare our method, which calculates correspondences the same way as described in Section 4.2, to a general LAP solver (specifically the Jonker-Volgenant algorithm from sklearn.linear_sum_assignment), nearest neighbor computation, optimal transport (as implemented in the python POT package), and stochastic matrices generated by Sinkhorn iterations. In Table 1 we show that our method outperforms all baselines in terms of geodesic error of the final matching and shows positive trends in terms of runtime and memory consumption.

4.4 Linear Assignment Problems

We next validate our method on balanced Linear Assignment Problems (LAPs), which typically involve assigning a set of agents to an equally sized set of tasks, e.g., when optimizing the allocation of resources. We show results on a collection of regularized LAPs in the form

where P ( V, W ) = s ( 2 αV W T ) is a permutation and A ∈ R n × n is some given similarity matrix. While the Softmax non-linearity ensures all rows sum to one, µ ( P ( V, W )) is a regularization term enforcing columns summing to one as well, to satisfy the permutation constraints:

Due to the row-wise Softmax all rows already sum to one but we incentive the columns to sum to one as well, as is necessary for permutations.

Dense Matrices. We evaluate on a set of LAPs based on descriptor similarities of 3D shapes from the FAUST dataset [3], with n randomly chosen vertices per object [13]. Let D X , D Y ∈ R n × k be two k-dimensional point-wise descriptors of the shapes X,Y corresponding to n points. We use the SHOT [37] ( R n × 352 ) and the heat kernel signature [43] ( R n × 101 ) with their default parameters as descriptors and stack them together to comprise D · ∈ R n × 453 in total, then A = D X · D ⊤ Y . Solving an LAP with this type of similarity matrix A is used e.g. in [46] as the initialization strategy.

Wegenerate 100 problem instances by pairing each of the 100 shapes in FAUST with a random second shape to get the pair X,Y . Then we evaluate the relative error of the energy compared to the energy of the optimal solution (restricted to valid permutation solutions), and the average Hamming distance to the next valid permutation (namely, the number of elements that violate the permutation constraint). We ran experiments with n = 100 , m = 30 , α = 20 and used a greedy heuristic to generate valid

0

0

.

5

1

5

10

-

3

10

-

1

10

1

n

Mem. Cons. in GB

sparse

dense

12

40 64

100

150

0

0

.

2

0

.

4

0

.

6

Instance Size

Relative Error

12

40 64

100

150

0

0

.

5

1

·

10

4

Instance Size

Runtime in Sec.

·

10

permutations from the results violating the permutation constraint (iteratively projecting the maximum value of the permutation to one, and the rest of the corresponding row and column to zero). Out of the 100 instances, 53 lead to valid permutations without the heuristic. The average relative error of immediately valid permutation is 1 . 8% and after pseudo-projection of all instances it is 2 . 0% . Due to the Softmax, every matrix has 100 non-zero entries that are all nearly equal to one. On average, the Hamming distance of invalid permutations to the next valid one is 1 . 38 ( 1 . 4% of the problem size) which means in most cases one would have a valid permutation after adjusting one entry.

̸

Sparse Matrices. Given a matrix A , that is sparsely populated and only contains non-zero entries in a subset S = { ( i, j ) | A i,j = 0 } , we compute and optimize the permutation matrix sparsely in ( i, j ) ∈ S by calculating the matrix factorization only at the required entries, similar to Section 4.1, but without restricting the number of entries per row of P to two. Also we take into account random entries ( q, r ) / ∈ S . We ran experiments for A with a matrix density of | S | = 0 . 01 n 2 for n = 1000 , 5000 and 10000 and m = 20 with increasing α from 1 to 20 iteratively and measure a Hamming distance of at most 0 . 28% of the problem size. To get a valid permutation matrix, we used the same heuristic as in the dense case and measured a relative error below 7 . 8% , compared to the Hungarian algorithm. Also, we could measure a memory reduction by over 65% .

4.5 Quadratic Assignment Problems

Quadratic Assignment Problems (QAPs) is a broadly employed mathematical tool for many real-life problems in operations research such as circuit design and shape matching. We demonstrate the application of our approach to non-convex QAPs of the form

where p ( V, W ) = vec ( s ( 2 αV W T ) ) is the vectorized version of the permutation and A ∈ R n 2 × n 2 is a cost matrix. V and W are normalized. The permutation matrix was optimized in a convex-concave manner, by optimizing the objective function

with β being iteratively increased from -∥ A ∥ 2 to ∥ A ∥ 2 and with µ ( P ( V, W )) being the same permutation constraint regularizer as in (13).

We show results on the QAPLIB [6] library of quadratic assignment problems of real-world applications which range between n = 12 and n = 256 and we choose m = ceil ( n 3 ) . The problems in QAPLIB are meant to be challenging and optimal solutions for some of the larger problems are not provided because they are unknown. Thus, we report the gap to optimality (when known) of our solution and consider a solution good if it falls within 10% of the optimum. We report the relative error and runtime in Fig. 3. In 75 out of 87 instances the result was a valid permutation matrix.

4.6 Shape Matching

Finally, we further assess the effectiveness of our approach for the the application of non-rigid shape matching, a common task in computer graphics and computer vision. To this end, we incorporate

our permutation matrix representation approach into the state-of-the-art shape-matching approach by Marin et al. [26], which learns the point correspondences using two consecutive networks N θ and G θ , predicting shape embeddings and probe functions, respectively. We propose to replace the calculation of the permutation matrix based on the output of the first network N θ by s ( αV W T ) , with α = 40 . The network transforms the vertices of 3D objects X x and X y into embeddings ϕ x = N θ ( X x ) and ϕ y = N θ ( X y ) , which are used to compute V = ϕ x ( ϕ † x P gt ϕ y ) and W = ϕ y . V here replaces a transformed embedding. The network is trained on the modified loss function

for a given ground truth permutation P gt , and V and W being normalized row-wise. Similar to Marin et al. , we train the networks over 1600 epochs on 10000 shapes of the SURREAL dataset [45] and evaluate our experiments on 100 noisy and noise-free objects of different shapes and poses of the FAUST dataset [3], that are provided by [26] in [25].

We follow the evaluation of Marin et al. [26] and calculate the geodesic distance between the ground truth matching and the predicted matching match 1 = N ( ϕ x C T 1 , ϕ y ) for C 1 = (( ϕ † y G θ ( X y )) T ) † ( ϕ † x G θ ( X x )) T whereby N is the nearest neighbor. In the following, we refer to the measured geodesic distance as e prob . A second error ( e emb ) which only concerns the first network's predictions, is measured by the geodesic distance towards match 2 = N ( ϕ x , ϕ y C ) for C = ϕ † y P gt ϕ x , which is, again, calculated following Marin et al. [26]. The results of our experiments are reported in Table 2, showing the average geodesic errors (over 10 runs for each experiment) for the approach presented in [26] and our method. The table reveals improved results compared to [26].

Stochastic Training. Given that the explicit calculation of the permutation matrix in (16) is memory-intensive for a large number of vertices, we employ stochastic training to avoid the need for computing the full permutation matrix. As we describe in Sec. 4.1 we only calculate the loss over a few entries where the final permutation ought to be equal to one and on k (here k can be ≥ 1 ) randomly chosen entries of each row of P in each iteration. This approach reduces the memory requirement and gives us the possibility to train with larger shapes consisting of more vertices. In our experiments, we applied the stochastic training technique on the SURREAL dataset, and then evaluated the performance on FAUST by measuring the error rates for varying values of k , as depicted in Fig. 4b. We observed a small relative increase of less than 17% in e emb , and also a small effect on e prob , but with a less clear tendency as one could see for e emb . For e prob we measured an average standard deviation of 2 . 25 × 10 -3 and for e emb of 3 . 3 × 10 -4 . Two noise-free examples of correspondences, visualized for full training and for stochastic training with k = 1 , are shown in Fig. 4a, with the reference image on the left and the corresponding shapes on the right.

To evaluate the impact on the error and memory consumption when dealing with high-resolution objects (with more vertices), we ran experiments using data from the TOSCA dataset [5]. We train 400 epochs on the objects of the classes victoria and michael ( 32 objects in total) with up to 20000 sampled vertices. These experiments reveal a significant reduction of memory consumption, see Fig. 5a. Additionally, we evaluate the training on the class david and report a relative memory-error trade-off for up to 20 k samples of each object, see Fig. 5b. The graph indicates a correlation between higher memory usage and lower error values for e emb . The trends observed in the memory-error trade-off for e emb are generally applicable to e prob as well, although with some outliers.

ref.

k

= 1

(a) Visualized results on FAUST

1 100

400

full

4

4

.

5

5

·

10

-

2

k

e

prob

2

2

.

5

3

·

10

-

2

e

emb

noise free

noisy

(b) Error values for different

k

0

0

.

5

1

1

.

5

2

4

0

10

20

n

Mem. Cons. in GB

full ([26],ours)

k

= 100

(ours)

k

= 1

(ours)

only netw.

-

0

.

1

0

0

.

1

Rel.

e

prob

n

= 1000

n

= 5000

n

= 10000

n

= 15000

n

= 20000

0.5

1

2

-

0

.

1

0

0

.

1

Relative Mem. Cons.

Rel.

e

emb

·

10

(a) Memory usage during non-stochastic training (red), stochastic training for different k (green), and just calculating the network output (blue)

(b) Relative memory - error trade-off for a varying number of vertices ( n )

5 Conclusion

In this work, we proposed a strategy to represent permutation matrices by a low-rank matrix factorization followed by a nonlinearity and showed that by using the Kissing number theory, we can determine the minimum rank necessary for representing a permutation matrix of a given size, allowing for a more memory-efficient representation. We validated this method with experiments on LAPs and QAPs as well as a real-world shape matching application and showed improvements in the latter. Additionally, we explored the potential of optimizing permutations stochastically to decrease memory usage, which opens the possibility of handling high-resolution data.

Limitations and Broader Impact Our method offers a promising solution to contribute positively to the environment by reducing the computational cost of a variety of problems involving permutation matrices. We do not see any ethical concerns associated with our approach itself. However, it is important to acknowledge a limitation of our method. For certain problem formulations, such as the Koopmans and Beckmann form QAPs, stochastic learning may not be feasible because the double occurrences of the permutation matrix make the stochastic computation not applicable. Moreover, our method requires devising a non-trivial, problem-specific adaptation.

Acknowledgment

Yuval Bahat received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 945422. Zorah Lähner is funded by a KI-Starter grant of the Ministry of Culture and Science of the State of North Rhine-Westphalia. Hannah Dröge acknowledges the support of the German Research Foundation (DFG) grant MO 2962/5-1.

References

Florian Bernard, Christian Theobalt, and Michael Moeller. Ds*: Tighter lifting-free convex relaxations for quadratic matching problems. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4310-4319, 2018.

Garrett Birkhoff. Tres observaciones sobre el algebra lineal [three observations in linear algebra]. Univ. Nac. Tucuman, Ser. A , 5:147-154, 1946.

Federica Bogo, Javier Romero, Matthew Loper, and Michael J Black. Faust: Dataset and evaluation for 3d mesh registration. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3794-3801, 2014.

Peter Boyvalenkov, Stefan Dodunekov, and Oleg R Musin. A survey on the kissing numbers. arXiv preprint arXiv:1507.03631 , 2015.

Alexander M Bronstein, Michael M Bronstein, and Ron Kimmel. Numerical geometry of non-rigid shapes . Springer Science & Business Media, 2008.

R. E. Burkard, S. E. Karisch, and F. Rendl. Qaplib - a quadratic assignment problem libarary. Journal of Global Optimization , 10, 1997.

Fabrício Caluza Machado and Fernando Mário de Oliveira Filho. Improving the semidefinite programming bound for the kissing number by exploiting polynomial symmetry. Experimental Mathematics , 27(3):362-369, 2018.

Dongliang Cao and Florian Bernard. Self-supervised learning for multimodal non-rigid 3d shape matching. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Aïssatou Diallo, Markus Zopf, and Johannes Fürnkranz. Permutation learning via lehmer codes. In ECAI 2020 , pages 1095-1102. IOS Press, 2020.

Nadav Dym, Haggai Maron, and Yaron Lipman. DS++ - A flexible, scalable and provably tight relaxation for matching problems. ACM Transactions on Graphics (TOG) , 36(6), 2017.

M. Eisenberger, D. Novotny, G. Kerchenbaum, P. Labatut, N. Neverova, D. Cremers, and A. Vedaldi. Neuromorph: Unsupervised shape interpolation and correspondence in one go. In IEEE International Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Marvin Eisenberger, Zorah Lähner, and Daniel Cremers. Divergence-free shape correspondence by deformation. In Symposium on Geometry Processing (SGP) , 2019.

Maolin Gao, Zorah Lähner, Johan Thunberg, Daniel Cremers, and Florian Bernard. Isometric multi-shape matching. In Conference on Computer Vision and Pattern Recognition (CVPR) , June 2021.

S. Gold and A. Rangarajan. A graduated assignment algorithm for graph matching. IEEE Trans. on Pattern Analysis and Machine Intelligence , 18, 1996.

Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan Russell, and Mathieu Aubry. 3d-coded : 3d correspondences by deep deformation. In ECCV , 2018.

Aditya Grover, Eric Wang, Aaron Zweig, and Stefano Ermon. Stochastic optimization of sorting networks via continuous relaxations. arXiv preprint arXiv:1903.08850 , 2019.

Jonathan Huang, Carlos Guestrin, and Leonidas Guibas. Fourier theoretic probabilistic inference over permutations. Journal of machine learning research , 10(5), 2009.

Itay Kezurer, Shahar Z Kovalsky, Ronen Basri, and Yaron Lipman. Tight relaxation of quadratic matching. 34(5), 2015.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015.

Risi Kondor, Andrew Howard, and Tony Jebara. Multi-object tracking with representations of the symmetric group. In Marina Meila and Xiaotong Shen, editors, Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics , volume 2 of Proceedings of Machine Learning Research , pages 211-218, San Juan, Puerto Rico, 21-24 Mar 2007. PMLR.

Tjalling C. Koopmans and Martin Beckmann. Assignment problems and the location of economic activities. Econometrica , 25(1), 1957.

Harold W. Kuhn. The hungarian method for the assignment problem. Naval Research Logistics Quaterly , 2, 1955.

Or Litany, Tal Remez, Emanuele Rodolà, Alex Bronstein, and Michael Bronstein. Deep functional maps: Structured prediction for dense shape correspondence. In Proc. of International Conference on Computer Vision (ICCV) , 2017.

E. M. Loiola, N. M. de Abreu, P. O. B. Netto, P. Hahn, and T. M. Querido. A survey for the quadratic assignment problem. European Journal of Operational Research , 176(2), 2007.

Riccardo Marin. Correspondence learning via linearly-invariant embedding (pytorch). https: //github.com/riccardomarin/Diff-FMAPs-PyTorch , 2022.

Riccardo Marin, Marie-Julie Rakotosaona, Simone Melzi, and Maks Ovsjanikov. Correspondence learning via linearly-invariant embedding. Advances in Neural Information Processing Systems , 33:1608-1620, 2020.

Haggai Maron, Nadav Dym, Itay Kezurer, Shahar Kovalsky, and Yaron Lipman. Point registration via efficient convex relaxation. ACM Transactions on Graphics (TOG) , 35(4):73, 2016.

Gonzalo Mena, David Belanger, Scott Linderman, and Jasper Snoek. Learning latent permutations with gumbel-sinkhorn networks. arXiv preprint arXiv:1802.08665 , 2018.

Oleg R Musin. The kissing number in four dimensions. Annals of Mathematics , pages 1-32, 2008.

Maks Ovsjanikov, Mirela Ben-Chen, Justin Solomon, Adrian Butscher, and Leonidas Guibas. Functional maps: a flexible representation of maps between shapes. ACM Transactions on Graphics (ToG) , 31(4):1-11, 2012.

C. Papazov and D. Burschka. Deformable 3d shape registration based on local similarity transforms. In Proc. of Symposium on Geometry Processing (SGP) , 2011.

Ulrich Pinkall and Konrad Polthier. Computing discrete minimal surfaces and their conjugates. Experimental Mathematics , 2(1):15 - 36, 1993.

Sergey M Plis, Stephen McCracken, Terran Lane, and Vince D Calhoun. Directional statistics on permutations. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 600-608. JMLR Workshop and Conference Proceedings, 2011.

Ryan Prescott Adams and Richard S Zemel. Ranking via sinkhorn propagation. arXiv e-prints , pages arXiv-1106, 2011.

Gilles Puy, Alexandre Boulch, and Renaud Marlet. Flot: Scene flow on point clouds guided by optimal transport. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVIII , pages 527-544. Springer, 2020.

Emanuele Rodolá, Alex M. Bronstein, Andrea Albarelli, Filippo Bergamasco, and Andrea Torsello. A game-theoretic approach to deformable shape matching. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2012.

Samuele Salti, Federico Tombari, and Luigi Di Stefano. Shot: Unique signatures of histograms for surface and texture description. Computer Vision and Image Understanding , 125:251-264, 2014.

Rodrigo Santa Cruz, Basura Fernando, Anoop Cherian, and Stephen Gould. Deeppermnet: Visual permutation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3949-3957, 2017.

Lawrence K Saul. A geometrical connection between sparse and low-rank matrices and its application to manifold learning. Transactions on Machine Learning Research , 2022.

Lawrence K Saul. A nonlinear matrix decomposition for mining the zeros of sparse data. SIAM Journal on Mathematics of Data Science , 4(2):431-463, 2022.

Marcel Seelbach Benkner, Zorah Lähner, Vladislav Golyanik, Christof Wunderlich, Christian Theobalt, and Michael Moeller. Q-match: Iterative shape matching via quantum annealing. In International Conference on Computer Vision (ICCV) , 2021.

Giovanni Seraghiti, Atharva Awari, Arnaud Vandaele, Margherita Porcelli, and Nicolas Gillis. Accelerated algorithms for nonlinear matrix decomposition with the relu function. arXiv preprint arXiv:2305.08687 , 2023.

J. Sun, M. Ovsjanikov, and L. Guibas. A concise and provably informative multi-scale signaturebased on heat diffusion. Computer Graphics Forum (CGF) , 28:1383-1392, 2009.

Ramana Sundararaman, Gautam Pai, and Maks Ovsjanikov. Implicit field supervision for robust non-rigid shape matching. In Proc. of European Conference on Computer Vision (ECCV) , 2022.

Gül Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J. Black, Ivan Laptev, and Cordelia Schmid. Learning from synthetic humans. In CVPR , 2017.

Matthias Vestner, Zorah Lähner, Amit Boyarski, Or Litany, Ron Slossberg, Tal Remez, Emanuele Rodolà, Alex M. Bronstein, Michael M. Bronstein, Ron Kimmel, and Daniel Cremers. Efficient deformable shape correspondence via kernel matching. In International Conference on 3D Vision (3DV) , 2017.

Runzhong Wang, Junchi Yan, and Xiaokang Yang. Combinatorial learning of robust deep graph matching: an embedding based approach. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2020.

Runzhong Wang, Junchi Yan, and Xiaokang Yang. Neural graph matching network: Learning lawler's quadratic assignment problem with extension to hypergraph and multiple-graph matching. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(9):5261-5279, 2021.

Tianshu Yu, Runzhong Wang, Junchi Yan, and Baoxin Li. Learning deep graph matching with channel-independent embedding and hungarian attention. In International conference on learning representations , 2020.

Andrei Zanfir and Cristian Sminchisescu. Deep learning of graph matching. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2684-2693, 2018.

Q. Zhao, S. E. Karisch, F. Rendl, and H. Wolkowicz. Semidefinite programming relaxations for the quadratic assignment problem. Journal of Combinatorial Optimization , 2(1), 1998.

C Zong. The kissing numbers of convex bodies-a brief survey. Bulletin of the London Mathematical Society , 30(1):1-10, 1998.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a provably exact and an approximative method for calculating permutation matrices requiring much lower memory than previous approaches. The algorithms are based on Kissing numbers and describe row relationships as their cosine value. This allows for representing the problem with significantly fewer values than the baseline n*n permutation matrix representation. Additionally, the authors propose a relaxation of this exact algorithm, utilizing the SoftMax operator, that alleviates the issues with the optimization for large problems. The authors also show a number of applications where their method is applicable.\n\nSTRENGTHS:\nI believe this is a pretty strong work with a very general and useful contribution. Strengths in more details:\n- A provably exact algorithm that significantly reduces the memory consumption of general permutation matrices.\n- Additionally, a relaxation is proposed that has better convergence properties in optimization.\n- The authors demonstrate on a number of applications that their method is applicable and useful.\n\nWEAKNESSES:\nFirst, I wanted to give a strong accept, but the experiments left me slightly in doubt. \nMy main problem is the following: The experiments focus a little bit too much on the application side (which is amazing), but miss to do a general comparison to baselines and the proposed methods. I would have liked to see comparisons to SOTA algorithms (related works mention many) on finding permutation matrices, showing run-time, memory, and errors. While the exact algorithm will probably have 0 error, some other methods might have a little bit bigger while being orders of magnitude faster. Also, to my understanding, the exact algorithm breaks down with the problem size, when the SoftMax version is used, which is not exact anymore. Also, it would be good to see the breaking point of the exact algorithm where it does not converge anymore. In brief, a thorough comparison to the SOTA is missing.  \n\nI still like the paper, but without such comparison (considering the theoretical value), I do not give a strong accept. If the authors can provide such an analysis to understand the trade-offs in their rebuttal, I consider improving the rating.\n\nQUESTIONS:\nExperiments:\n- L191 is translation not considered on purpose?\n- L204-209 This is a little but unclear. Does the propose method always, without failure, find the correct permutation and transformation? If this is the case, this should be highlighted more. If not, an error value should be shown. \n- L209 Equally good results with Softmax. Does this mean that the approximation in this noise-free case can be compensated?\n- L215 refers to Fig.2, but that seems to show something different (Dense/Sparse) and not the topic the authors mention at this line. \n\nTypos:\nL135 \"either exact\" -> \"either as exact\"\nL156 \"valdidate\" -> \"validate\"\nL179 missing dot from the end of the sentence\nL230 \"similariy\" -> \"similarity\"\n\nLIMITATIONS:\nDiscussed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a formulation for representing high-dimensional permutation matrices. The basic idea is to express the n x n permutation matrix as an elementwise non-linear function of a product of low-rank matrices V,W (n x m). The authors introduce the kissing number theory which gives the minimum number 'm' for which such a decomposition is provably possible and then show intuitive proofs that in scenarios when the decomposition is valid - their construction is appropriate. In addition, they also promote two well-known non-linearities (ReLU and SoftMax) which can be used in the optimization process for the representation.\n\nResults are demonstrated on two synthetic and one recent 3D shape-matching scenario. Broadly the experiments make an impression of a significantly reduced memory consumption and the ability to recover permutations. In the shape-matching example, the proposed formulation also slightly improves on the previous baseline.\n\nSTRENGTHS:\n- The core idea of this paper (low-rank representation of permutations with nonlinearities) is *very* interesting and potentially has a very wide impact in scenarios where matching between pointsets is a crucial problem (linear assignment, quadratic assignment etc)\n- Overall the writing of this paper is very good, and the background and related material on the kissing number theory has been introduced and explained in a way interesting way.  \n- The benefits of the proposed construction are (1.) strong memory reduction in representing permutation matrices (2.) An accompanying optimization scheme that allows for recovering permutations\n\nWEAKNESSES:\n- Broadly, I felt the experimental section is very rudimentary. There are very few comparisons to conceptual baselines: namely using stochastic matrices, optimal transport, Hungarian algorithm and nearest neighbor-like methods. It is not clear from the experiments whether the obtained solutions are: not just some permutations, but the *correct* permutations. \n\n- More specifically, for the point cloud example in section 4.2, what is the accuracy of the recovered transformation $\\Theta$ as a function of n? How do other conceptual baselines compare in this example both in terms of accuracy and memory complexity? \n\n- Despite the experiments on Marin et.al, perhaps a simpler and more convincing demonstration would be to use the proposed permutation representation in either or all of [43], [30], etc. where a linear/quadratic assignment is solved and then compare with the original methods (perhaps yielding an improvement in memory with a comparable or better accuracy)\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nI do not think this paper has any direct negative societal impact. Please see the weaknesses for technical limitations. \n\nOverall, this paper has a very interesting idea and a clever conceptual message on representing permutations. My biggest concern is whether the proposed construction is impactful in terms of ease of optimization and acceptable accuracies for the multitude of shape-matching problems that this can be applied to. Given the lack of comparisons to conceptual baselines (i.e. not in terms of a state-of-the art shape matching paper, but even simply comparing with previous permutation representations like stochastic matrices, or spectral decompositions - in any framework), I am inclined towards a weak reject at this point. I can be convinced of the gains in memory complexity of the proposed representation but am not yet sold on its applicability and accuracy.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

comments: SUMMARY:
The paper proposes a novel approach for representing permutation matrices with low-rank matrix factorisation. The method employs Kissing number theory to find the minimum rank necessary to represent the target matrix. This is often quite a bit lower than the rank of the original matrix, so allows for a more efficient memory representation. The paper also shows how the approach can be used in practice in various relevant problems, such as point cloud alignment or shape matching.

Edit: I have read the rebuttal, and given the scores from other reviewers and I would like to keep my score.

STRENGTHS:
* The work deals with a meaningful problem, and produces an elegant solution with wide ranging applicability.
* The authors demonstrate the performance of their approach in various problems, demonstrating similar / better accuracy with a significant improvement in memory requirement.
* The paper is very well written and easy to read.
* The experiments are relevant.
* The formulation is principled and includes several relevant proofs.

Overall I believe this is a perfect NeurIPS paper. While I have little knowledge of Kissing number theory, the paper solves a very relevant problem in an elegant way, and demonstrates impressive and wide-ranging practical applicability in multiple domains. I therefore strongly believe the work should be accepted.

WEAKNESSES:
I have very little negative notes about the work, though, I could say that the ablation section could be expanded, e.g. by seeing how increasing the stochastic training k affects speed and accuracy.

QUESTIONS:
none

LIMITATIONS:
These are addressed at the end of the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work addresses the problem of estimating large permutation matrices by approximating them with a low-rank factorization follow by a nonlinear mapping, thereby reducing the storage complexity from O(n^2) to O(n).\n\nThe main contribution of the paper is i) a theoretical derivation and proof of the minimal required rank of the factorization matrices, and ii) the use of a nonlinearity, such that the full rank permutation matrix can be restored *exactly*. i) is based on the Kissing number (or bounds on it), while the nonlinearity in ii) can be a ReLu. Importantly, the possibility to exactly represent any permutation matrix is in strong contrast to direct low-rank factorizations which can only approximate the permutation matrix (since unable to recover the full rank).\n\nThe authors propose practical solutions for the optimization: A smooth version of the nonlinearity (softmax), and an optimization scheme (inspired by stochastic optimization; gradients are considering a row of the factorization matrices at a time) that approximates the full objective, but never requires to build the full permutation matrix, thus fully leveraging the compact representation and resulting storage savings.\n\nExperimental results are extensive and demonstrate the applicability of the method on point cloud alignment, linear and quadratic assignment problems, and shape matching.\n\nSTRENGTHS:\n**S1** Proposition 1 and Eq. (7) represent a novel contribution. The insight that a non-linear mapping of a matrix factorization is able to recover the full rank permutation matrix is a strong contribution and is of interest for the wider community.\n\n**S2** The proof of the minimal require factorization matrix rank via the Kissing number is an interesting theoretical contribution. Also practically it provides a clear guidance on the required size of factorization matrices such that the exact permutation matrix can be recovered.\n\n**S3** The proposed stochastic optimization in Sec. 4.1 provides a practical algorithm for optimizing the permutation matrix, while leveraging the compact representation (the full permutation matrix is never built, but only computed element-wise at a time). The non-smoothness of the ReLU is addressed by soft-max function with a controllable temperature parameter that allows to approximate the exact solution with desired accuracy over the course of the optimization procedure.\n\n**S4** Experimental results are performed across different application domains and demonstrate the usefulness of the proposed approach.\n\nWEAKNESSES:\nNone, but I'm also not an expert in the area. I'm especially not knowledgeable about related work and thus can not judge the novelty of the proposed approach.\n\nQUESTIONS:\n**Q1** Is there a relation between the Kissing number and the ReLU? Is it conceivable that there exists another non-linear function that allows an even lower rank factorization while still being able to represent that exact permutation matrix?\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel way to decompose a permutation matrix into two low-rank matrices so that a significant amount of  space can be saved to store a permutation. Then the authors further implement such decomposition to solve practical tasks of point cloud alignment, assignment problem and shape matching.\n\nSTRENGTHS:\nOverall, the paper is well-written and clearly explains the heuristics behind the method. The method of using Kissing numbers and non-linearity to perform low-rank decomposition of a permutation matrix is very interesting and can possibly inspire future research. Based on Fig 5, the method indeed provides memory saving for supervised learning.\n\nWEAKNESSES:\nThe most troubling weakness of this method lies in its value of application. Specifically, I have the following concerns:\n\n1. The method introduces two complex problems to optimization: bi-variable structure and non-linearity. As mentioned by the authors, this requires devising a non-trivial, problem-specific adaptation for each problem. \n\n2. To avoid forming an $n\\times n$ matrix, the authors propose to only optimize over a handful of entries, which basically requires knowing the ground truth permutation (or assuming some sparse structures as in LAP). This leads to a very limited set of application scenarios.\n\nQUESTIONS:\n1. Kissing number is based on a threshold of arccos(0.5), does this mean that if we switch to a smaller angle, we can fit more vectors in a unit sphere, thus save more space to store a permutation? If so, why don't we do so?\n\n2. When both $V$ and $W$ need to be optimized, are they optimized in parallel or alternatingly? Specifically which optimization algorithm is used for your numerical tests?\n\n3. How do you implement Stochastic Optimization for LAP and QAP when $A$ is dense? \n\n4. In the part (b) of Fig 5, what is the scale of y axis? The term \"Relative\" here is very confusing and without explanation.\n\nLIMITATIONS:\nThe limitations of the work are mostly mentioned in the weakness section. The authors indeed addressed them but it is still quite confusing to read for the first time.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}