SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\tilde {g} ^ {(\ell)} \left(\alpha \boldsymbol {\theta} _ {A} + (1 - \alpha) \boldsymbol {\theta} _ {B}; \boldsymbol {H} ^ {(\ell)}\right) = \alpha \tilde {g} ^ {(\ell)} \left(\boldsymbol {\theta} _ {A}; \boldsymbol {H} ^ {(\ell)}\right) + (1 - \alpha) \tilde {g} ^ {(\ell)} \left(\boldsymbol {\theta} _ {B}; \boldsymbol {H} ^ {(\ell)}\right)
$$
$$
\tilde {g} ^ {(\ell)} \left(\boldsymbol {\theta}; \alpha \boldsymbol {H} _ {A} ^ {(\ell)} + (1 - \alpha) \boldsymbol {H} _ {B} ^ {(\ell)}\right) = \alpha \tilde {g} ^ {(\ell)} \left(\boldsymbol {\theta}; \boldsymbol {H} _ {A} ^ {(\ell)}\right) + (1 - \alpha) \tilde {g} ^ {(\ell)} \left(\boldsymbol {\theta}; \boldsymbol {H} _ {B} ^ {(\ell)}\right)
$$
Subsequently,
$$
\begin{array}{l} \alpha \tilde {g} ^ {(\ell)} \left(\boldsymbol {\theta} _ {A}; \boldsymbol {H} _ {A} ^ {(\ell - 1)}\right) + (1 - \alpha) \tilde {g} ^ {(\ell)} \left(\boldsymbol {\theta} _ {B}; \boldsymbol {H} _ {B} ^ {(\ell - 1)}\right) = \alpha \tilde {g} ^ {(\ell)} \left(\alpha \boldsymbol {\theta} _ {A} + (1 - \alpha) \boldsymbol {\theta} _ {B}; \boldsymbol {H} _ {A} ^ {(\ell - 1)}\right) \\ + (1 - \alpha) \tilde {g} ^ {(\ell)} (\alpha \boldsymbol {\theta} _ {A} + (1 - \alpha) \boldsymbol {\theta} _ {B}; \boldsymbol {H} _ {B} ^ {(\ell - 1)}) \\ = \tilde {g} ^ {(\ell)} \left(\alpha \boldsymbol {\theta} _ {A} + (1 - \alpha) \boldsymbol {\theta} _ {B}; \alpha \boldsymbol {H} _ {A} ^ {(\ell - 1)} + (1 - \alpha) \boldsymbol {H} _ {B} ^ {(\ell - 1)}\right). \\ \end{array}
$$
Given the weak additivity for ReLU activation is satisfied for  $\theta_{A}$  and  $\theta_{B}$ , then we have
$$
\begin{array}{l} \sigma \left(\alpha \tilde {g} ^ {(\ell)} \left(\boldsymbol {\theta} _ {A}; \boldsymbol {H} _ {A} ^ {(\ell - 1)}\right) + (1 - \alpha) \tilde {g} ^ {(\ell)} \left(\boldsymbol {\theta} _ {B}; \boldsymbol {H} _ {B} ^ {(\ell - 1)}\right)\right) = \sigma \left(\tilde {g} ^ {(\ell)} \left(\alpha \boldsymbol {\theta} _ {A} + (1 - \alpha) \boldsymbol {\theta} _ {B}; \alpha \boldsymbol {H} _ {A} ^ {(\ell - 1)} + (1 - \alpha) \boldsymbol {H} _ {B} ^ {(\ell - 1)}\right)\right) \\ \alpha g ^ {(\ell)} \left(\boldsymbol {\theta} _ {A}; \boldsymbol {H} _ {A} ^ {(\ell - 1)}\right) + (1 - \alpha) g ^ {(\ell)} \left(\boldsymbol {\theta} _ {B}; \boldsymbol {H} _ {B} ^ {(\ell - 1)}\right) = g ^ {(\ell)} \left(\alpha \boldsymbol {\theta} _ {A} + (1 - \alpha) \boldsymbol {\theta} _ {B}; \alpha \boldsymbol {H} _ {A} ^ {(\ell - 1)} + (1 - \alpha) \boldsymbol {H} _ {B} ^ {(\ell - 1)}\right) \\ \end{array}
$$
To conclude,  $\forall \ell \in [L]$  and  $\forall \alpha \in [0,1]$ , we have
$$
\alpha \boldsymbol {H} _ {A} ^ {(\ell)} + (1 - \alpha) \boldsymbol {H} _ {B} ^ {(\ell)} = g ^ {(\ell)} \left(\alpha \boldsymbol {\theta} _ {A} + (1 - \alpha) \boldsymbol {\theta} _ {B}; \alpha \boldsymbol {H} _ {A} ^ {(\ell - 1)} + (1 - \alpha) \boldsymbol {H} _ {B} ^ {(\ell - 1)}\right) \tag {10}
$$
For the right hand side of Equation (10), recursively, we can have
$$
\begin{array}{l} g ^ {(\ell)} \left(\alpha \boldsymbol {\theta} _ {A} + (1 - \alpha) \boldsymbol {\theta} _ {B}; \alpha \boldsymbol {H} _ {A} ^ {(\ell - 1)} + (1 - \alpha) \boldsymbol {H} _ {B} ^ {(\ell - 1)}\right) \\ = g ^ {(\ell)} \left(\alpha \boldsymbol {\theta} _ {A} + (1 - \alpha) \boldsymbol {\theta} _ {B}; g ^ {(\ell - 1)} \left(\alpha \boldsymbol {\theta} _ {A} + (1 - \alpha) \boldsymbol {\theta} _ {B}; \alpha \boldsymbol {H} _ {A} ^ {(\ell - 2)} + (1 - \alpha) \boldsymbol {H} _ {B} ^ {(\ell - 2)}\right)\right) \\ = \left(g ^ {(\ell)} \circ g ^ {(\ell - 1)}\right) \left(\alpha \boldsymbol {\theta} _ {A} + (1 - \alpha) \boldsymbol {\theta} _ {B}; \alpha \boldsymbol {H} _ {A} ^ {(\ell - 2)} + (1 - \alpha) \boldsymbol {H} _ {B} ^ {(\ell - 2)}\right) \\ \end{array}
$$
$$
\begin{array}{l} = \dots \\ = \left(g ^ {(\ell)} \circ g ^ {(\ell - 1)} \dots \circ g ^ {(1)}\right) (\alpha \boldsymbol {\theta} _ {A} + (1 - \alpha) \boldsymbol {\theta} _ {B}; \boldsymbol {X}) \\ = f ^ {(\ell)} (\alpha \boldsymbol {\theta} _ {A} + (1 - \alpha) \boldsymbol {\theta} _ {B}; \boldsymbol {X}), \\ \end{array}
$$
where  $X$  denotes the input data matrix.
Recall we denote  $\pmb{H}^{(\ell)} = f^{(\ell)}(\pmb{\theta};\pmb{X})$  which indicates
$$
\alpha f ^ {(\ell)} \left(\boldsymbol {\theta} _ {A}; \boldsymbol {X}\right) + (1 - \alpha) f ^ {(\ell)} \left(\boldsymbol {\theta} _ {B}; \boldsymbol {X}\right) = f ^ {(\ell)} \left(\alpha \boldsymbol {\theta} _ {A} + (1 - \alpha) \boldsymbol {\theta} _ {B}; \boldsymbol {X}\right),
$$
A.3 Derivation of Quadratic Assignment Problem
In this section, we aim to show that minimizing  $\sum_{\ell = 1}^{L}\left\| \left(\pmb{W}_{A}^{(\ell)} - \pmb{P}^{(\ell)}\pmb{W}_{B}^{(\ell)}\pmb{P}^{(\ell -1)^{\top}}\right)\left(\pmb{H}_{A}^{(\ell -1)} - \pmb{P}^{(\ell -1)}\pmb{H}_{B}^{(\ell -1)}\right)\right\|_{F}^{2}$  includes solving Quadratic Assignment Problems (QAPs), known to be NP-hard.
$$
\begin{array}{l} \underset {\pi = \{\boldsymbol {P} ^ {(\ell)} \}} {\arg \min } \sum_ {\ell = 1} ^ {L} \left\| \left(\boldsymbol {W} _ {A} ^ {(\ell)} - \boldsymbol {P} ^ {(\ell)} \boldsymbol {W} _ {B} ^ {(\ell)} \boldsymbol {P} ^ {(\ell - 1) ^ {\top}}\right) \left(\boldsymbol {H} _ {A} ^ {(\ell - 1)} - \boldsymbol {P} ^ {(\ell - 1)} \boldsymbol {H} _ {B} ^ {(\ell - 1)}\right) \right\| _ {F} ^ {2} \\ \Longleftrightarrow \underset {\pi = \left\{\boldsymbol {P} ^ {(\ell)} \right\}} {\arg \min } \sum_ {\ell = 1} ^ {L} \left\| \boldsymbol {W} _ {A} ^ {(\ell)} \boldsymbol {H} _ {A} ^ {(\ell - 1)} - \boldsymbol {P} ^ {(\ell)} \boldsymbol {W} _ {B} ^ {(\ell)} \boldsymbol {P} ^ {(\ell - 1) ^ {\top}} \boldsymbol {H} _ {A} ^ {(\ell - 1)} - \boldsymbol {W} _ {A} ^ {(\ell)} \boldsymbol {P} ^ {(\ell - 1)} \boldsymbol {H} _ {B} ^ {(\ell - 1)} + \boldsymbol {P} ^ {(\ell)} \boldsymbol {W} _ {B} ^ {(\ell)} \boldsymbol {H} _ {B} ^ {(\ell - 1)} \right\| _ {F} ^ {2} \\ \Longleftrightarrow \underset {\pi = \left\{\boldsymbol {P} ^ {(\ell)} \right\}} {\arg \min } \sum_ {\ell = 1} ^ {L} \left(\left\| \boldsymbol {W} _ {A} ^ {(\ell)} \boldsymbol {H} _ {A} ^ {(\ell - 1)} - \boldsymbol {W} _ {A} ^ {(\ell)} \boldsymbol {P} ^ {(\ell - 1)} \boldsymbol {H} _ {B} ^ {(\ell - 1)} \right\| _ {F} ^ {2} + \left\| \boldsymbol {P} ^ {(\ell)} \boldsymbol {W} _ {B} ^ {(\ell)} \boldsymbol {P} ^ {(\ell - 1) ^ {\top}} \boldsymbol {H} _ {A} ^ {(\ell - 1)} - \boldsymbol {P} ^ {(\ell)} \boldsymbol {W} _ {B} ^ {(\ell)} \boldsymbol {H} _ {B} ^ {(\ell - 1)} \right\| _ {F} ^ {2}\right) \\ + \left\langle \pmb {W} _ {A} ^ {(\ell)} \pmb {H} _ {A} ^ {(\ell - 1)} - \pmb {W} _ {A} ^ {(\ell)} \pmb {P} ^ {(\ell - 1)} \pmb {H} _ {B} ^ {(\ell - 1)}, \pmb {P} ^ {(\ell)} \pmb {W} _ {B} ^ {(\ell)} \pmb {P} ^ {(\ell - 1) ^ {\top}} \pmb {H} _ {A} ^ {(\ell - 1)} - \pmb {P} ^ {(\ell)} \pmb {W} _ {B} ^ {(\ell)} \pmb {H} _ {B} ^ {(\ell - 1)} \right\rangle_ {F}). \\ \end{array}
$$
Consider its first term, i.e.,
$$
\underset {\pi = \{\boldsymbol {P} ^ {(\ell)} \}} {\arg \min } \sum_ {\ell = 1} ^ {L} \left\| \boldsymbol {W} _ {A} ^ {(\ell)} \boldsymbol {H} _ {A} ^ {(\ell - 1)} - \boldsymbol {W} _ {A} ^ {(\ell)} \boldsymbol {P} ^ {(\ell - 1)} \boldsymbol {H} _ {B} ^ {(\ell - 1)} \right\| _ {F} ^ {2}
$$
$$
\Longleftrightarrow \underset {\pi = \{\boldsymbol {P} ^ {(\ell)} \}} {\arg \min } \sum_ {\ell = 1} ^ {L} \operatorname {t r} \left(\left(\boldsymbol {H} _ {A} ^ {(\ell - 1) ^ {\top}} \boldsymbol {W} _ {A} ^ {(\ell) ^ {\top}} - \boldsymbol {H} _ {B} ^ {(\ell - 1) ^ {\top}} \boldsymbol {P} ^ {(\ell - 1) ^ {\top}} \boldsymbol {W} _ {A} ^ {(\ell) ^ {\top}}\right) \left(\boldsymbol {W} _ {A} ^ {(\ell)} \boldsymbol {H} _ {A} ^ {(\ell - 1)} - \boldsymbol {W} _ {A} ^ {(\ell)} \boldsymbol {P} ^ {(\ell - 1)} \boldsymbol {H} _ {B} ^ {(\ell - 1)}\right)\right)
$$
$$
\begin{array}{l} \Longleftrightarrow \underset {\pi = \{P ^ {(\ell)} \}} {\arg \min } \sum_ {\ell = 1} ^ {L} \operatorname {t r} \left(\boldsymbol {H} _ {A} ^ {(\ell - 1) ^ {\top}} \boldsymbol {W} _ {A} ^ {(\ell) ^ {\top}} \boldsymbol {W} _ {A} ^ {(\ell)} \boldsymbol {H} _ {A} ^ {(\ell - 1)} - \boldsymbol {H} _ {B} ^ {(\ell - 1) ^ {\top}} \boldsymbol {P} ^ {(\ell - 1) ^ {\top}} \boldsymbol {W} _ {A} ^ {(\ell) ^ {\top}} \boldsymbol {W} _ {A} ^ {(\ell)} \boldsymbol {H} _ {A} ^ {(\ell - 1)} \right. \\ \left. - \boldsymbol {H} _ {A} ^ {(\ell - 1) ^ {\top}} \boldsymbol {W} _ {A} ^ {(\ell)} ^ {\top} \boldsymbol {W} _ {A} ^ {(\ell)} \boldsymbol {P} ^ {(\ell - 1)} \boldsymbol {H} _ {B} ^ {(\ell - 1)} + \boldsymbol {H} _ {B} ^ {(\ell - 1) ^ {\top}} \boldsymbol {P} ^ {(\ell - 1) ^ {\top}} \boldsymbol {W} _ {A} ^ {(\ell)} ^ {\top} \boldsymbol {W} _ {A} ^ {(\ell)} \boldsymbol {P} ^ {(\ell - 1)} \boldsymbol {H} _ {B} ^ {(\ell - 1)}\right) \\ \end{array}
$$
$$
\begin{array}{l} \Longleftrightarrow \underset {\pi = \{\boldsymbol {P} ^ {(\ell)} \}} {\arg \min } \sum_ {\ell = 1} ^ {L} \operatorname {t r} \left(- 2 \boldsymbol {H} _ {B} ^ {(\ell - 1) ^ {\top}} \boldsymbol {P} ^ {(\ell - 1) ^ {\top}} \boldsymbol {W} _ {A} ^ {(\ell) ^ {\top}} \boldsymbol {W} _ {A} ^ {(\ell)} \boldsymbol {H} _ {A} ^ {(\ell - 1)} + \boldsymbol {H} _ {B} ^ {(\ell - 1) ^ {\top}} \boldsymbol {P} ^ {(\ell - 1) ^ {\top}} \boldsymbol {W} _ {A} ^ {(\ell) ^ {\top}} \boldsymbol {W} _ {A} ^ {(\ell)} \boldsymbol {P} ^ {(\ell - 1)} \boldsymbol {H} _ {B} ^ {(\ell - 1)}\right) \\ \Longleftrightarrow \underset {\pi = \left\{\boldsymbol {P} ^ {(\ell)} \right\}} {\arg \min } \sum_ {\ell = 1} ^ {L} \operatorname {t r} \left(- 2 \boldsymbol {P} ^ {(\ell - 1) ^ {\top}} \boldsymbol {W} _ {A} ^ {(\ell)} ^ {\top} \boldsymbol {W} _ {A} ^ {(\ell)} \boldsymbol {H} _ {A} ^ {(\ell - 1)} \boldsymbol {H} _ {B} ^ {(\ell - 1) ^ {\top}} + \boldsymbol {P} ^ {(\ell - 1) ^ {\top}} \boldsymbol {W} _ {A} ^ {(\ell)} ^ {\top} \boldsymbol {W} _ {A} ^ {(\ell)} \boldsymbol {P} ^ {(\ell - 1)} \boldsymbol {H} _ {B} ^ {(\ell - 1)} \boldsymbol {H} _ {B} ^ {(\ell - 1) ^ {\top}}\right) \\ \Longleftrightarrow \underset {\pi = \left\{\boldsymbol {P} ^ {(\ell)} \right\}} {\arg \min } \sum_ {\ell = 1} ^ {L} \left(\operatorname {t r} \left(\boldsymbol {P} ^ {(\ell - 1)} ^ {\top} \boldsymbol {W} _ {A} ^ {(\ell)} ^ {\top} \boldsymbol {W} _ {A} ^ {(\ell)} \boldsymbol {P} ^ {(\ell - 1)} \boldsymbol {H} _ {B} ^ {(\ell - 1)} \boldsymbol {H} _ {B} ^ {(\ell - 1) ^ {\top}}\right) - 2 \operatorname {t r} \left(\boldsymbol {P} ^ {(\ell - 1)} ^ {\top} \boldsymbol {W} _ {A} ^ {(\ell)} ^ {\top} \boldsymbol {W} _ {A} ^ {(\ell)} \boldsymbol {H} _ {A} ^ {(\ell - 1)} \boldsymbol {H} _ {B} ^ {(\ell - 1) ^ {\top}}\right)\right) \\ \end{array}
$$
where  $\operatorname{tr}\left(\boldsymbol{P}^{(\ell -1)^{\top}}\boldsymbol{W}_A^{(\ell)^{\top}}\boldsymbol{W}_A^{(\ell)}\boldsymbol{P}^{(\ell -1)}\boldsymbol{H}_B^{(\ell -1)}\boldsymbol{H}_B^{(\ell -1)^{\top}}\right) - 2\operatorname{tr}\left(\boldsymbol{P}^{(\ell -1)^{\top}}\boldsymbol{W}_A^{(\ell)^{\top}}\boldsymbol{W}_A^{(\ell)}\boldsymbol{H}_A^{(\ell -1)}\boldsymbol{H}_B^{(\ell -1)^{\top}}\right)$ .
is in the form of Koopmans-Beckmann's QAP [14] for each  $P^{(\ell -1)}$  and known as NP-hard. Thus,
solving  $\underset {\pi = \{P^{(\ell)}\}}{\arg \min}\sum_{\ell = 1}^{L}\left\| W_A^{(\ell)}H_A^{(\ell -1)} - W_A^{(\ell)}P^{(\ell -1)}H_B^{(\ell -1)}\right\| _F^2$  is to solve  $L - 1$  QAPs in parallel.
Similarly, consider the second term, i.e,
which also gives rise to Koopmans-Beckmann's QAPs.
For the last term, i.e,
$$
\begin{array}{l} \underset {\pi = \left\{\boldsymbol {P} ^ {(\ell)} \right\}} {\arg \min } \sum_ {\ell = 1} ^ {L} \left\langle \boldsymbol {W} _ {A} ^ {(\ell)} \boldsymbol {H} _ {A} ^ {(\ell - 1)} - \boldsymbol {W} _ {A} ^ {(\ell)} \boldsymbol {P} ^ {(\ell - 1)} \boldsymbol {H} _ {B} ^ {(\ell - 1)}, \boldsymbol {P} ^ {(\ell)} \boldsymbol {W} _ {B} ^ {(\ell)} \boldsymbol {P} ^ {(\ell - 1) ^ {\top}} \boldsymbol {H} _ {A} ^ {(\ell - 1)} - \boldsymbol {P} ^ {(\ell)} \boldsymbol {W} _ {B} ^ {(\ell)} \boldsymbol {H} _ {B} ^ {(\ell - 1)} \right\rangle_ {F} \\ \Longleftrightarrow \underset {\pi = \left\{\boldsymbol {P} ^ {(\ell)} \right\}} {\arg \min } \sum_ {\ell = 1} ^ {L} \left\langle \left(\boldsymbol {W} _ {A} ^ {(\ell)} \boldsymbol {H} _ {A} ^ {(\ell - 1)} - \boldsymbol {W} _ {A} ^ {(\ell)} \boldsymbol {P} ^ {(\ell - 1)} \boldsymbol {H} _ {B} ^ {(\ell - 1)}\right) \left(\boldsymbol {W} _ {B} ^ {(\ell)} \boldsymbol {P} ^ {(\ell - 1) ^ {\top}} \boldsymbol {H} _ {A} ^ {(\ell - 1)} - \boldsymbol {W} _ {B} ^ {(\ell)} \boldsymbol {H} _ {B} ^ {(\ell - 1)}\right) ^ {\top}, \boldsymbol {P} ^ {(\ell)} \right\rangle_ {F}, \\ \end{array}
$$
which entails solving bi-level matching problems.
Therefore, the objective can be rewritten as the summation of QAPs and bi-level matching problems and cannot be further simplified, which is NP-hard.
B More Experimental Details and Results
B.1 Detailed Experimental Settings
In this section, we introduce the detailed experimental setup. Before delving into details, recall that unless otherwise specified, in this paper we consider models trained on a training set, and then all the investigations are evaluated on a test set.
B.1.1 Spawning Method
Multi-Layer Perceptrons on the MNIST Dataset. In accordance with the settings outlined by Ainsworth et al. [1], we train multi-layer perceptron networks with three hidden layers, each consisting of 512 units, on the MNIST dataset. We adopt the ReLU activation between layers. Optimization is done with the Adam algorithm and a learning rate of  $1.2 \times 10^{-4}$ . The batch size is set to 60 and the total number of training epochs is 30. To find the modes that satisfy LMC, we start spawning from a common initialization  $\theta^{(0)}$ .
VGG-16 and ResNet-20 on the CIFAR-10 Dataset. In accordance with the settings outlined by Frankle et al. [9], we train the VGG-16 architecture [29] and the ResNet-20 architecture [12] on the CIFAR-10 dataset. Data augmentation techniques include random horizontal flips and random  $32 \times 32$  pixel crops. Optimization is done using SGD with momentum (momentum set to 0.9). A weight decay of  $1 \times 10^{-4}$  is applied. The learning rate is initialized at 0.1 and is dropped by 10 times at 80 and 120 epochs. The total number of epochs is 160. To find the modes that satisfy LMC, we start spawning after training 5 epochs for both VGG-16 and ResNet-20.
ResNet-50 on the Tiny-ImageNet Dataset. In accordance with the settings outlined by Frankle et al. [9], we train the ResNet-50 architecture [12] on the Tiny-ImageNet dataset. Data augmentation techniques include random horizontal flips and random  $32 \times 32$  pixel crops. Optimization is done using SGD with momentum (momentum set to 0.9). A weight decay of  $1 \times 10^{-4}$  is applied. The learning rate is set to 0.4 and warmed up for 5 epochs and then is dropped by 10 times at 30, 60 and 80 epochs. The total number of epochs is 90. To find the modes that satisfy LMC, we start spawning after training 14 epochs.
B.1.2 Permutation Method
For the permutation method, we follow the experimental settings of Ainsworth et al. [1] strictly, which are described below.
Multi-Layer Perceptrons on MNIST and CIFAR-10. Similar to the spawning method, we use multi-layer perceptron (MLP) networks with three hidden layers, each consisting of 512 units. For MNIST, optimization is performed using Adam with a learning rate of  $1 \times 10^{-3}$ . For CIFAR-10, optimization is performed using SGD with a learning rate of 0.1. Both activation matching and weight matching are used to identify modes that satisfy LMC.
ResNet-20 on CIFAR-10. To achieve LMC, we modify the ResNet-20 architecture by incorporating LayerNorms in place of BatchNorms. Furthermore, we increase the width of ResNet-20 by a factor of 32. Data augmentation techniques include random horizontal flips, random  $32 \times 32$  pixel crops, randomizes the image between  $0.8 \times$  and  $1.2 \times$ , and random rotations between  $\pm 30^{\circ}$ . The optimization process involves using SGD with momentum (set to 0.9). A weight decay regularization term of  $5 \times 10^{-4}$  is applied. A single cosine decay schedule with a linear warm-up is applied, where the learning rate is initialized to  $1 \times 10^{-6}$  and gradually increased to 0.1 over the course of an epoch, and then a single cosine decay schedule is applied for the remaining training. Only weight matching is used to identify modes that satisfy LMC.
Unlike the spawning method, VGG models are not used in the permutation method due to their inability to achieve LMC. Additionally, Ainsworth et al. [1] open-sourced their source code and pretrained checkpoints. Therefore, we directly use the pre-trained checkpoints provided by Ainsworth et al. [1].
B.2 Verification of LLFC Co-Occurring with LMC
In this section, we provide extensive experimental results to verify that LLFC consistently co-occurs with LMC, and conduct a new experiment to demonstrate that the constant  $c$  is close to 1 in most cases. Both the spawning method and the permutation method are utilized to obtain linearly connected modes  $\theta_{A}$  and  $\theta_{B}$ . As shown in Figures 8 to 13 and 15, we include experimental results for MLP on the MNIST dataset (spawning method, activation matching, and weight matching), MLP on the CIFAR-10 dataset (both activation matching and weight matching), VGG-16 on the CIFAR-10 dataset (spawning method), ResNet-20 on the CIFAR-10 dataset (spawning method and weight matching) and ResNet-50 on the Tiny-ImageNet dataset (spawning method). In particular, in Figure 14, we include experimental results of Straight-Trough Estimator (STE) [1]. STE method tries to learn a permutation with STE that could minimize the loss barrier between one mode and the other permuted mode.
To verify the LLFC property on each data point  $\pmb{x}_i$  in the test set  $\mathcal{D}$ , we measure  $\mathrm{cosine}_{\alpha}(\pmb{x}_i) = \cos [f^{(\ell)}(\alpha \pmb{\theta}_A + (1 - \alpha)\pmb{\theta}_B;\pmb{x}_i),\alpha f^{(\ell)}(\pmb{\theta}_A;\pmb{x}_i) + (1 - \alpha)f^{(\ell)}(\pmb{\theta}_B;\pmb{x}_i)]$ . We compare this to the baseline cosine similarity  $\mathrm{cosine}_{A,B}(\pmb{x}_i) = \cos [f^{(\ell)}(\pmb{\theta}_A;\pmb{x}_i),f^{(\ell)}(\pmb{\theta}_B;\pmb{x}_i)]$ . In Figures 8 to 15, we
Figure 8: Comparison between  $\mathbb{E}_{\mathcal{D}}[1 - \mathrm{cosine}_{\alpha}(\pmb{x}_i)]$  and  $\mathbb{E}_{\mathcal{D}}[1 - \mathrm{cosine}_{A,B}(\pmb{x}_i)]$  and demonstration of  $\mathbb{E}_{\mathcal{D}}[1 - \mathrm{coef}_{\alpha}(\pmb{x}_i)]$ . The spawning method is used to obtain two linearly connected modes  $\theta_{A}$  and  $\theta_{B}$ . Results are presented for different layers of MLP on MNIST dataset, with  $\alpha \in \{0.25, 0.5, 0.75\}$ . Standard deviations across the dataset are reported by error bars.
Figure 9: Comparison between  $\mathbb{E}_{\mathcal{D}}[1 - \mathrm{cosine}_{\alpha}(\pmb{x}_i)]$  and  $\mathbb{E}_{\mathcal{D}}[1 - \mathrm{cosine}_{A,B}(\pmb{x}_i)]$  and demonstration of  $\mathbb{E}_{\mathcal{D}}[1 - \mathrm{coef}_{\alpha}(\pmb{x}_i)]$ . The spawning method is used to obtain two linearly connected modes  $\theta_{A}$  and  $\theta_{B}$ . Results are presented for different layers of VGG-16 on the CIFAR-10 dataset, with  $\alpha \in \{0.25, 0.5, 0.75\}$ . Standard deviations across the dataset are reported by error bars.
conclude that the values of  $\mathbb{E}_{\mathcal{D}}[1 - \mathrm{cosine}_{\alpha}(\pmb{x}_i)]$  are close to 0 compared with  $\mathbb{E}_{\mathcal{D}}[1 - \mathrm{cosine}_{A,B}(\pmb{x}_i)]$ , and thus verify our claim.
To show that the constant  $c$  is close to 1 in most cases, for each data point  $\pmb{x}_i$  in the test set  $\mathcal{D}$ , we measure  $\mathrm{coef}_{\alpha}(\pmb{x}_i) = \| f^{(\ell)}(\alpha \pmb{\theta}_A + (1 - \alpha)\pmb{\theta}_B;\pmb{x}_i)\| \mathrm{cosine}_{\alpha}(\pmb{x}_i) / \| \alpha f^{(\ell)}(\pmb{\theta}_A;\pmb{x}_i) + (1 - \alpha)f^{(\ell)}(\pmb{\theta}_B;\pmb{x}_i)\|$ , where  $\| f^{(\ell)}(\alpha \pmb{\theta}_A + (1 - \alpha)\pmb{\theta}_B;\pmb{x}_i)\| \mathrm{cosine}_{\alpha}(\pmb{x}_i)$  denotes the length of  $f^{(\ell)}(\alpha \pmb{\theta}_A + (1 - \alpha)\pmb{\theta}_B;\pmb{x}_i)$  projected on  $\alpha f^{(\ell)}(\pmb{\theta}_A;\pmb{x}_i) + (1 - \alpha)f^{(\ell)}(\pmb{\theta}_B;\pmb{x}_i)]$ . In Figures 8 to 15, we conclude that the values of  $\mathbb{E}_{\mathcal{D}}[\mathrm{coef}_{\alpha}(\pmb{x}_i)]$  are close to 1 in most cases, and thus verify our claim.
Figure 10: Comparison between  $\mathbb{E}_{\mathcal{D}}[1 - \mathrm{cosine}_{\alpha}(\pmb{x}_i)]$  and  $\mathbb{E}_{\mathcal{D}}[1 - \mathrm{cosine}_{A,B}(\pmb{x}_i)]$  and demonstration of  $\mathbb{E}_{\mathcal{D}}[1 - \mathrm{coef}_{\alpha}(\pmb{x}_i)]$ . The spawning method is used to obtain two linearly connected modes  $\theta_{A}$  and  $\theta_{B}$ . Results are presented for different layers of ResNet-20 on the CIFAR-10 dataset, with  $\alpha \in \{0.25, 0.5, 0.75\}$ . Standard deviations across the dataset are reported by error bars.
Figure 11: Comparison between  $\mathbb{E}_{\mathcal{D}}[1 - \mathrm{cosine}_{\alpha}(\pmb{x}_i)]$  and  $\mathbb{E}_{\mathcal{D}}[1 - \mathrm{cosine}_{A,B}(\pmb{x}_i)]$  and demonstration of  $\mathbb{E}_{\mathcal{D}}[1 - \mathrm{coef}_{\alpha}(\pmb{x}_i)]$ . The spawning method is used to obtain two linearly connected modes  $\theta_{A}$  and  $\theta_{B}$ . Results are presented for different layers of ResNet-50 on the Tiny-ImageNet dataset, with  $\alpha \in \{0.25, 0.5, 0.75\}$ . Standard deviations across the dataset are reported by error bars.
B.3 Verification of Commutativity
In this section, we provide more experimental results on various datasets and model architectures to verify the commutativity property for modes that satisfy LLFC. As shown in Figures 16 to 18, we include more experiments results for VGG-16 on the CIFAR-10 dataset (spawning method), MLP on the MNIST dataset (activation matching) and MLP on the CIFAR-10 dataset (both activation matching and weight matching).
Figure 12: Comparison between  $\mathbb{E}_{\mathcal{D}}[1 - \mathrm{cosine}_{\alpha}(\pmb{x}_i)]$  and  $\mathbb{E}_{\mathcal{D}}[1 - \mathrm{cosine}_{A,B}(\pmb{x}_i)]$  and demonstration of  $\mathbb{E}_{\mathcal{D}}[1 - \mathrm{coef}_{\alpha}(\pmb{x}_i)]$ . The activation matching and the weight matching are used to obtain two linearly connected modes  $\theta_{A}$  and  $\theta_{B}$ . Results are presented for different layers of MLP on the MNIST dataset, with  $\alpha \in \{0.25, 0.5, 0.75\}$ . Standard deviations across the dataset are reported by error bars.
Figure 13: Comparison between  $\mathbb{E}_{\mathcal{D}}[1 - \mathrm{cosine}_{\alpha}(\pmb{x}_i)]$  and  $\mathbb{E}_{\mathcal{D}}[1 - \mathrm{cosine}_{A,B}(\pmb{x}_i)]$  and demonstration of  $\mathbb{E}_{\mathcal{D}}[1 - \mathrm{coef}_{\alpha}(\pmb{x}_i)]$ . The activation matching and the weight matching are used to obtain two linearly connected modes  $\theta_{A}$  and  $\theta_{B}$ . Results are presented for different layers of MLP on the CIFAR-10 dataset, with  $\alpha \in \{0.25, 0.5, 0.75\}$ . Standard deviations across the dataset are reported by error bars.
To verify the commutativity generally holds for modes that satisfy LLFC, for test set  $\mathcal{D}$ , we compute  $\mathrm{Dist}_{com} = \mathrm{dist}\left(\mathrm{vec}(\pmb{W}_A^{(\ell)}\pmb{H}_A^{(\ell -1)} + \pmb{W}_B^{(\ell)}\pmb{H}_B^{(\ell -1)})\right)$ . Furthermore, we compare  $\mathrm{Dist}_{com}$  with  $\mathrm{Dist}_W = \mathrm{dist}\left(\mathrm{vec}(\pmb{W}_A^{(\ell)})\right)$  and  $\mathrm{Dist}_H = \mathrm{dist}\left(\mathrm{vec}(\pmb{H}_A^{(\ell -1)}\pmb{varepsilon},\pmb{H}_B^{(\ell -1)})\right)$ , respectively. In Figures 16 to 18,  $\mathrm{Dist}_{com}$  is negligible compared with  $\mathrm{Dist}_W$  and  $\mathrm{Dist}_H$ , confirming the commutativity condition.
Furthermore, we add baselines of models that are not linearly connected to further validate the commutativity condition. In Figure 19, we include experimental results for ResNet-20 on CIFAR-10 dataset (both spawning and weight matching method). Specifically, we measure  $\mathrm{Dist}_{\mathrm{com},LMC}$  of two linearly connected modes and  $\mathrm{Dist}_{\mathrm{com},notLMC}$  of two independently trained modes. In Figure 19, the values of  $\mathrm{Dist}_{\mathrm{com},LMC}$  are negligible compared with  $\mathrm{Dist}_{\mathrm{com},notLMC}$ , which confirms the commutativity condition.