SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Due to the quadratically growing size of the permutation matrix with an increasing number of points, we further propose to optimize for the permutation matrix stochastically, as described in Section 4.1. We ran experiments with similar settings as above, wherein we gradually increased the value of the temperature parameter  $\alpha$  linearly during optimization from  $\alpha = 5 \cdot 10^{-5}$  to  $\alpha = 1000$ . In these experiments, we again found that each point was paired with its corresponding nearest neighbor. Also, we could reduce the memory consumption, as shown in Fig. 2.
4.3 Point Cloud Alignment on Spectral Point Representation
We conduct an additional experiment on point cloud alignment in the context of the functional maps framework [30]. Here, the goal is to extract a point-to-point correspondence between two shapes  $X,Y$  from a  $m\times m$ -dimensional functional map  $C$  [30] where  $m$  is much smaller than the number of vertices in  $X$  and  $Y$ . A possible interpretation of  $C$  is that it aligns the spectral point representations  $\Phi_X,\Phi_Y\in \mathbb{R}^{n\times m}$  in which each point  $x$  is represented by the vector of values
of the first  $m$  Laplace-Beltrami eigenfunctions at  $x$  such that  $P\cdot \Phi_X\approx \Phi_Y\cdot C$  where  $P$  is the unknown permutation between  $X$  and  $Y$ . Given  $C, P$  can be retrieved by a nearest-neighbor query between  $\Phi_X, \Phi_Y C$ , as proposed in the original paper (see [30], Section 6.1), or by solving a Linear Assignment Problem (LAP) if a bijection is desired. This is exactly the same setting as in Section 4.2 with a small amount of noise in the point clouds. We use the FAUST registrations [3] with the original 6890 vertices, a downsampled version to 502 vertices for those experiments and  $C$  generated by the ground-truth correspondence. Then,  $\Phi_X, \Phi_Y C$  can be directly used for  $V$  and  $W$  in our method.
We compare our method, which calculates correspondences the same way as described in Section 4.2, to a general LAP solver (specifically the Jonker-Volgenant algorithm from sklearn.linear_sum_assignment), nearest neighbor computation, optimal transport (as implemented in the python POT package), and stochastic matrices generated by Sinkhorn iterations. In Table 1 we show that our method outperforms all baselines in terms of geodesic error of the final matching and shows positive trends in terms of runtime and memory consumption.
502 vertices, m = 20	6890 vertices, m = 50
Method	Error	Time	Memory	Error	Time	Memory
LAP	1.3 × 10-1	0.023s	8.02 MB	3.1 × 10-1	79.2s	565.31 MB
Nearest-Neighbors	8.2 × 10-1	0.008s	5.22 MB	4.0 × 10-1	2.6s	34.00 MB
Optimal Transport	3.8 × 10-1	0.524s	12.58 MB	2.0 × 10-1	182.7s	1862.27 MB
Sinkhorn iterations	1.4 × 10-1	0.030s	9.4 MB	3.0 × 10-1	12.0s	750.55 MB
Ours	2.2 × 10-3	0.801s	18.18 MB	2.5 × 10-2	77.6s	42.35 MB
Table 1: Comparisons of point-wise correspondence extraction from ground-truth functional map [30] for FAUST. The error is the mean geodesic matching error of all points. Please note that all code except ours and Sinkhorn iterations are from libraries that are likely more optimized in terms of runtime and memory consumption. The memory consumption is evaluated on CPU only.
4.4 Linear Assignment Problems
We next validate our method on balanced Linear Assignment Problems (LAPs), which typically involve assigning a set of agents to an equally sized set of tasks, e.g., when optimizing the allocation of resources. We show results on a collection of regularized LAPs in the form
$$
\underset {V, W} {\arg \min } \underbrace {\operatorname {t r} (A \cdot P (V , W))} _ {\text {L A P t e r m}} + \underbrace {\mu (P (V , W))} _ {\text {r e g u l a r i z e r}}, \tag {12}
$$
where  $P(V,W) = s\left(2\alpha VW^T\right)$  is a permutation and  $A \in \mathbb{R}^{n \times n}$  is some given similarity matrix. While the Softmax non-linearity ensures all rows sum to one,  $\mu(P(V,W))$  is a regularization term enforcing columns summing to one as well, to satisfy the permutation constraints:
$$
\mu (P) = \sum_ {j} \left(\sum_ {i} P _ {i j} - 1\right) ^ {2}. \tag {13}
$$
Due to the row-wise Softmax all rows already sum to one but we incentive the columns to sum to one as well, as is necessary for permutations.
Dense Matrices. We evaluate on a set of LAPs based on descriptor similarities of 3D shapes from the FAUST dataset [3], with  $n$  randomly chosen vertices per object [13]. Let  $D_X, D_Y \in \mathbb{R}^{n \times k}$  be two  $k$ -dimensional point-wise descriptors of the shapes  $X, Y$  corresponding to  $n$  points. We use the SHOT [37]  $(\mathbb{R}^{n \times 352})$  and the heat kernel signature [43]  $(\mathbb{R}^{n \times 101})$  with their default parameters as descriptors and stack them together to comprise  $D \in \mathbb{R}^{n \times 453}$  in total, then  $A = D_X \cdot D_Y^\top$ . Solving an LAP with this type of similarity matrix  $A$  is used e.g. in [46] as the initialization strategy.
We generate 100 problem instances by pairing each of the 100 shapes in FAUST with a random second shape to get the pair  $X, Y$ . Then we evaluate the relative error of the energy compared to the energy of the optimal solution (restricted to valid permutation solutions), and the average Hamming distance to the next valid permutation (namely, the number of elements that violate the permutation constraint). We ran experiments with  $n = 100$ ,  $m = 30$ ,  $\alpha = 20$  and used a greedy heuristic to generate valid
Figure 2: Memory consumption for point cloud alignment
Figure 3: Relative error and runtime on QAPLIB dataset
permutations from the results violating the permutation constraint (iteratively projecting the maximum value of the permutation to one, and the rest of the corresponding row and column to zero). Out of the 100 instances, 53 lead to valid permutations without the heuristic. The average relative error of immediately valid permutation is  $1.8\%$  and after pseudo-projection of all instances it is  $2.0\%$ . Due to the Softmax, every matrix has 100 non-zero entries that are all nearly equal to one. On average, the Hamming distance of invalid permutations to the next valid one is 1.38 ( $1.4\%$  of the problem size) which means in most cases one would have a valid permutation after adjusting one entry.
Sparse Matrices. Given a matrix  $A$ , that is sparsely populated and only contains non-zero entries in a subset  $S = \{(i,j) | A_{i,j} \neq 0\}$ , we compute and optimize the permutation matrix sparsely in  $(i,j) \in S$  by calculating the matrix factorization only at the required entries, similar to Section 4.1, but without restricting the number of entries per row of  $P$  to two. Also we take into account random entries  $(q,r) \notin S$ . We ran experiments for  $A$  with a matrix density of  $|S| = 0.01n^2$  for  $n = 1000, 5000$  and 10000 and  $m = 20$  with increasing  $\alpha$  from 1 to 20 iteratively and measure a Hamming distance of at most  $0.28\%$  of the problem size. To get a valid permutation matrix, we used the same heuristic as in the dense case and measured a relative error below  $7.8\%$ , compared to the Hungarian algorithm. Also, we could measure a memory reduction by over  $65\%$ .
4.5 Quadratic Assignment Problems
Quadratic Assignment Problems (QAPs) is a broadly employed mathematical tool for many real-life problems in operations research such as circuit design and shape matching. We demonstrate the application of our approach to non-convex QAPs of the form
$$
\underset {V, W} {\arg \min } p (V, W) ^ {T} A p (V, W) \tag {14}
$$
where  $p(V, W) = \operatorname{vec}(s(2\alpha VW^T))$  is the vectorized version of the permutation and  $A \in \mathbb{R}^{n^2 \times n^2}$  is a cost matrix.  $V$  and  $W$  are normalized. The permutation matrix was optimized in a convex-concave manner, by optimizing the objective function
$$
\underset {V, W} {\arg \min } p (V, W) ^ {T} (A - \beta I) p (V, W) + \mu (P (V, W)) \tag {15}
$$
with  $\beta$  being iteratively increased from  $-\| A\| _2$  to  $\| A\| _2$  and with  $\mu (P(V,W))$  being the same permutation constraint regularizer as in (13).
We show results on the QAPLIB [6] library of quadratic assignment problems of real-world applications which range between  $n = 12$  and  $n = 256$  and we choose  $m = \mathrm{ceil}\left(\frac{n}{3}\right)$ . The problems in QAPLIB are meant to be challenging and optimal solutions for some of the larger problems are not provided because they are unknown. Thus, we report the gap to optimality (when known) of our solution and consider a solution good if it falls within  $10\%$  of the optimum. We report the relative error and runtime in Fig. 3. In 75 out of 87 instances the result was a valid permutation matrix.
4.6 Shape Matching
Finally, we further assess the effectiveness of our approach for the application of non-rigid shape matching, a common task in computer graphics and computer vision. To this end, we incorporate
Table 2: Geodesic errors and standard deviation (std) for noise-free and noisy data by Marin et al. [26] and our approach.
eprob	std (eprob)	eemb	std (eemb)
[26]	0.051	17.4 × 10-4	0.029	3.5 × 10-4	noisy
ours	0.047	26.9 × 10-4	0.026	29.2 × 10-4
[26]	0.043	16.3 × 10-4	0.022	3.5 × 10-4	noise-free
ours	0.041	8.1 × 10-4	0.019	3.7 × 10-4
our permutation matrix representation approach into the state-of-the-art shape-matching approach by Marin et al. [26], which learns the point correspondences using two consecutive networks  $N_{\theta}$  and  $G_{\theta}$ , predicting shape embeddings and probe functions, respectively. We propose to replace the calculation of the permutation matrix based on the output of the first network  $N_{\theta}$  by  $s(\alpha VW^T)$ , with  $\alpha = 40$ . The network transforms the vertices of 3D objects  $X_{x}$  and  $X_{y}$  into embeddings  $\phi_x = N_\theta(X_x)$  and  $\phi_y = N_\theta(X_y)$ , which are used to compute  $V = \phi_x(\phi_x^\dagger P_{gt}\phi_y)$  and  $W = \phi_y$ .  $V$  here replaces a transformed embedding. The network is trained on the modified loss function
$$
\min  _ {\theta} \sum_ {l} \| s (2 \alpha V W ^ {T}) ^ {l} X _ {y} ^ {l} - P _ {g t} ^ {l} X _ {y} ^ {l} \| _ {2} ^ {2} \tag {16}
$$
for a given ground truth permutation  $P_{gt}$ , and  $V$  and  $W$  being normalized row-wise. Similar to Marin et al., we train the networks over 1600 epochs on 10000 shapes of the SURREAL dataset [45] and evaluate our experiments on 100 noisy and noise-free objects of different shapes and poses of the FAUST dataset [3], that are provided by [26] in [25].
We follow the evaluation of Marin et al. [26] and calculate the geodesic distance between the ground truth matching and the predicted matching match  $\mathbf{1} = \mathcal{N}(\phi_x C_1^T, \phi_y)$  for  $C_1 = ((\phi_y^\dagger G_\theta(X_y))^T)^\dagger (\phi_x^\dagger G_\theta(X_x))^T$  whereby  $\mathcal{N}$  is the nearest neighbor. In the following, we refer to the measured geodesic distance as  $e_{prob}$ . A second error  $(e_{emb})$  which only concerns the first network's predictions, is measured by the geodesic distance towards match  $\mathbf{2} = \mathcal{N}(\phi_x, \phi_y C)$  for  $C = \phi_y^\dagger P_{gt} \phi_x$ , which is, again, calculated following Marin et al. [26]. The results of our experiments are reported in Table 2, showing the average geodesic errors (over 10 runs for each experiment) for the approach presented in [26] and our method. The table reveals improved results compared to [26].
Stochastic Training. Given that the explicit calculation of the permutation matrix in (16) is memory-intensive for a large number of vertices, we employ stochastic training to avoid the need for computing the full permutation matrix. As we describe in Sec. 4.1 we only calculate the loss over a few entries where the final permutation ought to be equal to one and on  $k$  (here  $k$  can be  $\geq 1$ ) randomly chosen entries of each row of  $P$  in each iteration. This approach reduces the memory requirement and gives us the possibility to train with larger shapes consisting of more vertices. In our experiments, we applied the stochastic training technique on the SURREAL dataset, and then evaluated the performance on FAUST by measuring the error rates for varying values of  $k$ , as depicted in Fig. 4b. We observed a small relative increase of less than  $17\%$  in  $e_{emb}$ , and also a small effect on  $e_{prob}$ , but with a less clear tendency as one could see for  $e_{emb}$ . For  $e_{prob}$  we measured an average standard deviation of  $2.25 \times 10^{-3}$  and for  $e_{emb}$  of  $3.3 \times 10^{-4}$ . Two noise-free examples of correspondences, visualized for full training and for stochastic training with  $k = 1$ , are shown in Fig. 4a, with the reference image on the left and the corresponding shapes on the right.
To evaluate the impact on the error and memory consumption when dealing with high-resolution objects (with more vertices), we ran experiments using data from the TOSCA dataset [5]. We train 400 epochs on the objects of the classes victoria and michael (32 objects in total) with up to 20000 sampled vertices. These experiments reveal a significant reduction of memory consumption, see Fig. 5a. Additionally, we evaluate the training on the class david and report a relative memory-error trade-off for up to  $20k$  samples of each object, see Fig. 5b. The graph indicates a correlation between higher memory usage and lower error values for  $e_{emb}$ . The trends observed in the memory-error trade-off for  $e_{emb}$  are generally applicable to  $e_{prob}$  as well, although with some outliers.
(a) Visualized results on FAUST
(b) Error values for different  $k$
(a) Memory usage during non-stochastic training (red), stochastic training for different  $k$  (green), and just calculating the network output (blue)
Figure 5: (a) shows the memory consumption during training for shape matching for objects with a varying number of vertices  $(n)$  and varying sparseness  $(k)$ . (b) shows the relative memory-error trade-off for varying sparseness (which causes the memory reduction), whereby the memory consumption is relative to  $k = 100$  and the errors are relative to full training by [26] for  $n = 1000$ .
Figure 4: Visualized matching results (a) and error values (b) for the FAUST dataset for different levels of sparseness  $k$  during stochastic training
(b) Relative memory - error trade-off for a varying number of vertices  $(n)$
In this work, we proposed a strategy to represent permutation matrices by a low-rank matrix factorization followed by a nonlinearity and showed that by using the Kissing number theory, we can determine the minimum rank necessary for representing a permutation matrix of a given size, allowing for a more memory-efficient representation. We validated this method with experiments on LAPs and QAPs as well as a real-world shape matching application and showed improvements in the latter. Additionally, we explored the potential of optimizing permutations stochastically to decrease memory usage, which opens the possibility of handling high-resolution data.
**Limitations and Broader Impact** Our method offers a promising solution to contribute positively to the environment by reducing the computational cost of a variety of problems involving permutation matrices. We do not see any ethical concerns associated with our approach itself. However, it is important to acknowledge a limitation of our method. For certain problem formulations, such as the Koopmans and Beckmann form QAPs, stochastic learning may not be feasible because the double occurrences of the permutation matrix make the stochastic computation not applicable. Moreover, our method requires devising a non-trivial, problem-specific adaptation.
Yuval Bahat received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 945422. Zorah Lähner is funded by a KI-Starter grant of the Ministry of Culture and Science of the State of North Rhine-Westphalia. Hannah Dröge acknowledges the support of the German Research Foundation (DFG) grant MO 2962/5-1.
[1] Florian Bernard, Christian Theobalt, and Michael Moeller.  $\mathrm{Ds^{*}}$ : Tighter lifting-free convex relaxations for quadratic matching problems. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4310-4319, 2018.
[2] Garrett Birkhoff. Tres observaciones sobre el algebra lineal [three observations in linear algebra]. Univ. Nac. Tucuman, Ser. A, 5:147-154, 1946.
[3] Federica Bogo, Javier Romero, Matthew Loper, and Michael J Black. Faust: Dataset and evaluation for 3d mesh registration. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3794-3801, 2014.
[4] Peter Boyvalenko, Stefan Dodunekov, and Oleg R Musin. A survey on the kissing numbers. arXiv preprint arXiv:1507.03631, 2015.
[5] Alexander M Bronstein, Michael M Bronstein, and Ron Kimmel. Numerical geometry of non-rigid shapes. Springer Science & Business Media, 2008.
[6] R. E. Burkard, S. E. Karisch, and F. Rendl. Qaplib - a quadratic assignment problem library. Journal of Global Optimization, 10, 1997.
[7] Fabricio Caluza Machado and Fernando Mário de Oliveira Filho. Improving the semidefinite programming bound for the kissing number by exploiting polynomial symmetry. Experimental Mathematics, 27(3):362-369, 2018.
[8] Dongliang Cao and Florian Bernard. Self-supervised learning for multimodal non-rigid 3d shape matching. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[9] Aïssatou Diallo, Markus Zopf, and Johannes Fünkranz. Permutation learning via lehmer codes. In ECAI 2020, pages 1095-1102. IOS Press, 2020.
[10] Nadav Dym, Haggai Maron, and Yaron Lipman. DS++ - A flexible, scalable and provably tight relaxation for matching problems. ACM Transactions on Graphics (TOG), 36(6), 2017.
[11] M. Eisenberger, D. Novotny, G. Kerchenbaum, P. Labatut, N. Neverova, D. Cremers, and A. Vedaldi. Neuromorph: Unsupervised shape interpolation and correspondence in one go. In IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[12] Marvin Eisenberger, Zorah Lähner, and Daniel Cremers. Divergence-free shape correspondence by deformation. In Symposium on Geometry Processing (SGP), 2019.
[13] Maolin Gao, Zorah Lähner, Johan Thunberg, Daniel Cremers, and Florian Bernard. Isometric multi-shape matching. In Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.
[14] S. Gold and A. Rangarajan. A graduated assignment algorithm for graph matching. IEEE Trans. on Pattern Analysis and Machine Intelligence, 18, 1996.
[15] Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan Russell, and Mathieu Aubry. 3d-coded : 3d correspondences by deep deformation. In ECCV, 2018.
[16] Aditya Grover, Eric Wang, Aaron Zweig, and Stefano Ermon. Stochastic optimization of sorting networks via continuous relaxations. arXiv preprint arXiv:1903.08850, 2019.