SlowGuess/NeurIPS_2023_OCR huggingface dataset:
<sup>4</sup>We also conduct experiments on CNNs. For a Conv layer, the forward propagation will be denoted as  $\mathbf{W} \mathbf{H}$  similar to a linear layer. Typically, the weight  $\mathbf{W}$  for a Conv layer has shape (# of output channels, # of input channels, height, width) and we reshape  $\mathbf{W}$  to a matrix with dimensions (# of output channels, # of input channels × height × width).
Figure 5: Comparison between the distribution of the normalized distance  $\mathrm{Dist}_{\sigma}(\tilde{\pmb{h}}_{i,A},\tilde{\pmb{h}}_{i,B})$  and  $\mathrm{Dist}_{\sigma}(\tilde{\pmb{h}}_{i,C},\tilde{\pmb{h}}_{i,D})$ . Here,  $\tilde{\pmb{h}}_{i,A}$  and  $\tilde{\pmb{h}}_{i,B}$  are features of two linearly connected modes, i.e.,  $\pmb{\theta}_A$  and  $\pmb{\theta}_B$  (founded by the spawning method).  $\tilde{\pmb{h}}_{i,C}$  and  $\tilde{\pmb{h}}_{i,D}$  comes from two modes that are independently trained. Results are presented for different layers of MLP on MNIST and VGG-16 on CIFAR-10.
Figure 6: Comparison among the distribution of the normalized distance  $\mathrm{Dist}_{\sigma}(\tilde{\pmb{h}}_{i,C},\tilde{\pmb{h}}_{i,Dact})$ ,  $\mathrm{Dist}_{\sigma}(\tilde{\pmb{h}}_{i,C},\tilde{\pmb{h}}_{i,D})$  and  $\mathrm{Dist}_{\sigma}(\tilde{\pmb{h}}_{i,C},\tilde{\pmb{h}}_{i,D})$ . Here,  $\tilde{\pmb{h}}_{i,C}$  and  $\tilde{\pmb{h}}_{i,D}$  are features of two modes that are independently trained, i.e.,  $\theta_C$  and  $\theta_D$ .  $\tilde{\pmb{h}}_{i,Dact}$  and  $\tilde{\pmb{h}}_{i,Dwgt}$  comes from  $\theta_{Dact}$  (permuted  $\theta_D$  using the activation matching) and  $\theta_{Dwgt}$  (permuted  $\theta_D$  using the weight matching), correspondingly. For ResNet-20, the values of  $\mathrm{Dist}_{\sigma}$  are calculated in the first Conv layer of each block. Results are presented for different layers of MLP on MNIST and ResNet-20 on CIFAR-10.
an  $L$ -layer MLP  $f$  with ReLU activation, the weight matrix in the  $\ell$ -th linear layer is denoted as  $\boldsymbol{W}^{(\ell)} \in \mathbb{R}^{d_{\ell} \times d_{\ell-1}}$ , and  $\boldsymbol{b}^{(\ell)} \in \mathbb{R}^{d_{\ell}}$  is the bias in that layer. For a given input data matrix  $\boldsymbol{X} \in \mathbb{R}^{d_0 \times n}$ , denote the feature (post-activation) in the  $\ell$ -th layer as  $\boldsymbol{H}^{(\ell)} = f^{(\ell)}(\boldsymbol{\theta}; \boldsymbol{X}) \in \mathbb{R}^{d_{\ell} \times n}$  and correspondingly pre-activation as  $\tilde{\boldsymbol{H}}^{(\ell)} \in \mathbb{R}^{d_{\ell} \times n}$ . The forward propagation in the  $\ell$ -th layer is:
$$
\boldsymbol {H} ^ {(\ell)} = \sigma (\tilde {\boldsymbol {H}} ^ {(\ell)}), \quad \tilde {\boldsymbol {H}} ^ {(\ell)} = \boldsymbol {W} ^ {(\ell)} \boldsymbol {H} ^ {(\ell - 1)} + \boldsymbol {b} ^ {(\ell)} \mathbf {1} _ {d _ {\ell}} ^ {\top}.
$$
Here,  $\sigma$  denotes the ReLU activation function, and  $\mathbf{1}_{d_{\ell}} \in \mathbb{R}^{d_{\ell}}$  denotes the all-one vector. Additionally, we use  $\pmb{h}_i^{(\ell)}$  to denote the  $i$ -th row of  $\pmb{H}^{(\ell)}$ , and  $\tilde{\pmb{h}}_i^{(\ell)}$  to denote the  $i$ -th row of  $\tilde{\pmb{H}}^{(\ell)}$ , which correspond to the post- and pre-activations of the  $i$ -th input at layer  $\ell$ , respectively.
Condition I: Weak Additivity for ReLU Activations.5
Definition 3 (Weak Additivity for ReLU Activations). Given a dataset  $\mathcal{D}$ , two modes  $\theta_{A}$  and  $\theta_{B}$  are said to satisfy weak additivity for ReLU activations if
$$
\forall \ell \in [ L ], \forall \alpha \in [ 0, 1 ], \quad \sigma \left(\alpha \tilde {\boldsymbol {H}} _ {A} ^ {(\ell)} + (1 - \alpha) \tilde {\boldsymbol {H}} _ {B} ^ {(\ell)}\right) = \alpha \sigma \left(\tilde {\boldsymbol {H}} _ {A} ^ {(\ell)}\right) + (1 - \alpha) \sigma \left(\tilde {\boldsymbol {H}} _ {B} ^ {(\ell)}\right). \tag {7}
$$
Definition 3 requires the ReLU activation function to behave like a linear function for the pre-activations in each layer of the two networks. Although this cannot be true in general since ReLU is a nonlinear function, we verify it empirically for modes that satisfy LMC and LLFC.
We conduct experiments on various datasets and architectures to validate the weak additivity for ReLU activations. Specifically, given two modes  $\theta_{A}$  and  $\theta_{B}$  and a data point  $\pmb{x}_i$  in the test set  $\mathcal{D}$ , we compute the normalized distances between the left-hand side and the right-hand side of Equation (7) for each layer  $\ell$ , varying the values of  $\alpha$ . We denote the maximum distance across the range of  $\alpha$  as  $\mathrm{Dist}_{\sigma}(\tilde{\pmb{h}}_{i,A},\tilde{\pmb{h}}_{i,B}) = \max_{\alpha \in [0,1]}\mathrm{dist}(\sigma (\alpha \tilde{\pmb{h}}_{i,A} + (1 - \alpha)\tilde{\pmb{h}}_{i,B}),\alpha \sigma (\tilde{\pmb{h}}_{i,A}) + (1 - \alpha)\sigma (\tilde{\pmb{h}}_{i,B}))$ , where  $\mathrm{dist}(\pmb {x},\pmb {y})\coloneqq \| \pmb {x} - \pmb {y}\|^{2} / (\| \pmb {x}\| \cdot \| \pmb {y}\|)$ . To validate the weak additivity condition, we compare the values of  $\mathrm{Dist}_{\sigma}$  of two modes that are linearly connected with those of two modes that are independently trained.
<sup>5</sup>Weak additivity has no relation to stable neurons [28]. Stable neuron is defined as one whose output is the constant value zero or the pre-activation output on all inputs, which is a property concerning a single network. On the other hand, weak additivity concerns a relation between two networks.
Figure 7: Comparison of  $\mathrm{Dist}_{\mathrm{com}}$ ,  $\mathrm{Dist}_W$ , and  $\mathrm{Dist}_H$ . In the first row, the spawning method is used to acquire modes that satisfy LLFC, whereas the permutation method is used for the second row. For ResNet-20,  $\mathrm{Dist}_{\mathrm{com}}$ ,  $\mathrm{Dist}_W$  are calculated in the first Conv layer of each block. The results are presented for different layers of MLP on the MNIST and ResNet-20 on the CIFAR-10. More results are in Appendix B.3.
Both spawning and permutation methods are shown to demonstrate the weak additivity condition. For spawning method, we first obtain two linearly connected modes, i.e.,  $\theta_{A}$  and  $\theta_{B}$ , and then two independently trained modes, i.e.,  $\theta_{C}$  and  $\theta_{D}$  (not satisfying LMC/LLFC). We compare the values of  $\mathrm{Dist}_{\sigma}$  of  $\theta_{A}$  and  $\theta_{B}$  with those of  $\theta_{C}$  and  $\theta_{D}$ . In Figure 5, we observe that across different datasets and model architectures, at different layers,  $\mathrm{Dist}_{\sigma}(\tilde{h}_{i,A},\tilde{h}_{i,B})$  are negligible (and much smaller than the baseline  $\mathrm{Dist}_{\sigma}(\tilde{h}_{i,C},\tilde{h}_{i,D})$ ). For permutation methods, given two independently trained modes, i.e.,  $\theta_{C}$  and  $\theta_{D}$ , we permute the  $\theta_{D}$  such that the permuted  $\pi (\theta_{D})$  are linearly connected with  $\theta_{C}$ . Both activation matching and weight matching are used and the permuted  $\pi (\theta_{D})$  are denoted as  $\theta_{Dact}$  and  $\theta_{Dwgt}$  respectively. In Figure 6, the values of  $\mathrm{Dist}_{\sigma}(\tilde{h}_{i,C},\tilde{h}_{i,Dact})$  and  $\mathrm{Dist}_{\sigma}(\tilde{h}_{i,C},\tilde{h}_{i,Dwgt})$  are close to zero compared to  $\mathrm{Dist}_{\sigma}(\tilde{h}_{i,C},\tilde{h}_{i,D})$ . Therefore, we verify the weak additivity condition for both spawning and permutation methods.
Condition II: Commutativity.
Definition 4 (Commutativity). Given a dataset  $\mathcal{D}$ , two modes  $\theta_{A}$  and  $\theta_{B}$  are said to satisfy commutativity if
$$
\forall \ell \in [ L ], \boldsymbol {W} _ {A} ^ {(\ell)} \boldsymbol {H} _ {A} ^ {(\ell - 1)} + \boldsymbol {W} _ {B} ^ {(\ell)} \boldsymbol {H} _ {B} ^ {(\ell - 1)} = \boldsymbol {W} _ {A} ^ {(\ell)} \boldsymbol {H} _ {B} ^ {(\ell - 1)} + \boldsymbol {W} _ {B} ^ {(\ell)} \boldsymbol {H} _ {A} ^ {(\ell - 1)}. \tag {8}
$$
Commutativity depicts that the next-layer linear transformations applied to the internal features of two neural networks can be interchanged. This property is crucial for improving our understanding of LMC and LLFC. In Section 5.2, we will use the commutativity property to provide new insights into the permutation method.
We conduct experiments on various datasets and model architectures to verify the commutativity property for modes that satisfy LLFC. Specifically, for a given dataset  $\mathcal{D}$  and two modes  $\theta_{A}$  and  $\theta_{B}$  that satisfy LLFC, we compute the normalized distance between the left-hand side and the right-hand side of Equation (8), denoted as  $\mathrm{Dist}_{\mathrm{com}} = \mathrm{dist}\left(\mathrm{vec}(\boldsymbol{W}_{A}^{(\ell)}\boldsymbol{H}_{A}^{(\ell -1)} + \boldsymbol{W}_{B}^{(\ell)}\boldsymbol{H}_{B}^{(\ell -1)})\right)$ . Furthermore, we compare  $\mathrm{Dist}_{\mathrm{com}}$  with the normalized distance between the weight matrices of the current layer  $\ell$ , denoted as  $\mathrm{Dist}_{W}$ , and the normalized distances between the post-activations of the previous layer  $\ell - 1$ , denoted as  $\mathrm{Dist}_{H}$ . These distances are expressed as  $\mathrm{Dist}_{W} = \mathrm{dist}\left(\mathrm{vec}(\boldsymbol{W}_{A}^{(\ell)})\right)$ ,  $\mathrm{vec}(\boldsymbol{W}_{B}^{(\ell)})$ , and  $\mathrm{Dist}_{H} = \mathrm{dist}\left(\mathrm{vec}(\boldsymbol{H}_{A}^{(\ell - 1)})\right)$ , respectively. Figure 7 shows that for both spawning and permutation methods,  $\mathrm{Dist}_{\mathrm{com}}$  is negligible compared with  $\mathrm{Dist}_{W}$  and  $\mathrm{Dist}_{H}$ , which confirms the commutativity condition. Note that we also rule out the trivial case where either the weight matrices or the post-activations in the two networks are already perfectly aligned, as weights and post-activations often differ significantly. We also add more baseline experiments for comparison (see Appendix B.3 for more results).
Additionally, we note that commutativity has a similarity to model stitching [19, 3]. In Appendix B.4, we show that a stronger form of model stitching (without an additional trainable layer) works for two networks that satisfy LLFC.
Conditions I and II imply LLFC. Theorem 1 below shows that weak additivity for ReLU activations (Definition 3) and commutativity (Definition 4) imply LLFC.
Theorem 1 (Proof in Appendix A.2). Given a dataset  $\mathcal{D}$ , if two modes  $\theta_{A}$  and  $\theta_{B}$  satisfy weak additivity for ReLU activations (Definition 3) and commutativity (Definition 4), then
$$
\forall \alpha \in [ 0, 1 ], \forall \ell \in [ L ], f ^ {(\ell)} \left(\alpha \boldsymbol {\theta} _ {A} + (1 - \alpha) \boldsymbol {\theta} _ {B}\right) = \alpha f ^ {(\ell)} \left(\boldsymbol {\theta} _ {A}\right) + (1 - \alpha) f ^ {(\ell)} \left(\boldsymbol {\theta} _ {B}\right).
$$
Note that the definition of LLFC (Definition 2) allows a scaling factor  $c$ , while Theorem 1 establishes a stronger version of LLFC where  $c = 1$ . We attribute this inconsistency to the accumulation of errors in weak additivity and commutativity conditions, since they are only approximated satisfied in practice. Yet in most cases, we observe that  $c$  is close to 1 (see Appendix B.2 for more results).
5.2 Justification of the Permutation Methods
In this subsection, we provide a justification of the permutation methods in Git Re-Basin [1]: weight matching (3) and activation matching (4). Recall from Section 3 that given two modes  $\theta_{A}$  and  $\theta_{B}$ , the permutation method aims to find a permutation  $\pi = \{P^{(\ell)}\}_{\ell=1}^{L-1}$  such that the permuted  $\theta_{B}^{\prime} = \pi(\theta_{B})$  and  $\theta_{A}$  are linearly connected, where  $P^{(\ell)}$  is a permutation matrix applied to the  $\ell$ -th layer feature. Concretely, with a permutation  $\pi$ , we can formulate  $W_{B}^{\prime}^{(\ell)}$  and  $H_{B}^{\prime}^{(\ell)}$  of  $\theta_{B}^{\prime}$  in each layer  $\ell$  as  $W_{B}^{\prime}^{(\ell)} = P^{(\ell)} W_{B}^{(\ell)} P^{(\ell-1)^{\top}}$  and  $H_{B}^{\prime}^{(\ell)} = P^{(\ell)} H_{B}^{(\ell)}$  [1]. In Section 5.1, we have identified the commutativity property as a key factor contributing to LLFC. The commutativity property (8) between  $\theta_{A}$  and  $\theta_{B}^{\prime}$  can be written as
$$
\left(\boldsymbol {W} _ {A} ^ {(\ell)} - \boldsymbol {W} _ {B} ^ {\prime} ^ {(\ell)}\right) \left(\boldsymbol {H} _ {A} ^ {(\ell - 1)} - \boldsymbol {H} _ {B} ^ {\prime} ^ {(\ell - 1)}\right) = \mathbf {0},
$$
$$
\left(\boldsymbol {W} _ {A} ^ {(\ell)} - \boldsymbol {P} ^ {(\ell)} \boldsymbol {W} _ {B} ^ {(\ell)} \boldsymbol {P} ^ {(\ell - 1) ^ {\top}}\right) \left(\boldsymbol {H} _ {A} ^ {(\ell - 1)} - \boldsymbol {P} ^ {(\ell - 1)} \boldsymbol {H} _ {B} ^ {(\ell - 1)}\right) = \mathbf {0}. \tag {9}
$$
We note that the weight matching (3) and activation matching (4) objectives can be written as  $\min_{\pi} \sum_{\ell=1}^{L} \left\| \boldsymbol{W}_A^{(\ell)} - \boldsymbol{P}^{(\ell)} \boldsymbol{W}_B^{(\ell)} \boldsymbol{P}^{(\ell-1)^{\top}} \right\|_F^2$  and  $\min_{\pi} \sum_{\ell=1}^{L-1} \left\| \boldsymbol{H}_A^{(\ell)} - \boldsymbol{P}^{(\ell)} \boldsymbol{H}_B^{(\ell)} \right\|_F^2$ , respectively, which directly correspond to the two factors in Equation (9). Therefore, we can interpret Git Re-Basin as a means to ensure commutativity. Extended discussion on Git Re-basin [1] can be found in Appendix B.5.
We identified Layerwise Linear Feature Connectivity (LLFC) as a prevalent phenomenon that co-occurs with Linear Mode Connectivity (LMC). By investigating the underlying contributing factors to LLFC, we obtained novel insights into the existing permutation methods that give rise to LMC. The consistent co-occurrence of LMC and LLFC suggests that LLFC may play an important role if we want to understand LMC in full. Since the LLFC phenomenon suggests that averaging weights is roughly equivalent to averaging features, a natural future direction is to study feature averaging methods and investigate whether averaging leads to better features.
We note that our current experiments mainly focus on image classification tasks, though aligning with existing literature on LMC. We leave the exploration of empirical evidence beyond image classification as future direction. We also note that our Theorem 1 predicts LLFC in an ideal case, while in practice, a scaling factor  $c$  is introduced to Definition 2 to better describe the experimental results. Realistic theorems and definitions (approximated version) are deferred to future research.
Finally, we leave the question if it is possible to find a permutation directly enforcing the commutativity property (9). Minimizing  $\| (\pmb{W}_A^{(\ell)} - \pmb{P}^{(\ell)}\pmb{W}_B^{(\ell)}\pmb{P}^{(\ell -1)}^\top)(\pmb{H}_A^{(\ell -1)} - \pmb{P}^{(\ell -1)}\pmb{H}_B^{(\ell -1)})\| _F$  entails solving Quadratic Assignment Problems (QAPs) (See Appendix A.3 for derivation), which are known NP-hard. Solving QAPs calls for efficient techniques especially seeing the progress in learning to solve QAPs [37, 34].
we find that employing a scaling factor  $c$  enables a much better description of the practical behavior than other choices (e.g. an additive error term).
Note that both  $\pmb{P}^{(0)}$  and  $\pmb{P}^{(L)}$  are identity matrices.
[36] Zhiwei Xu, Yutong Wang, Spencer Frei, Gal Vardi, and Wei Hu. Benign overfitting and grokking in relu networks for xor cluster data. arXiv preprint arXiv:2310.02541, 2023.
[37] Tianshu Yu, Junchi Yan, Yilin Wang, Wei Liu, and baoxin Li. Generalizing graph matching beyond quadratic assignment model. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper_files/paper/2018/file/51d92be1c60d1db1d2e5e7a07da55b26-Paper.pdf.
[38] David Yunis, Kumar Kshitij Patel, Pedro Henrique Pamplona Savarese, Gal Vardi, Jonathan Frankle, Matthew Walter, Karen Livescu, and Michael Maire. On convexity and linear mode connectivity in neural networks. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop), 2022. URL https://openreview.net/forum?id=TZQ3PKL3fPr.
[39] Zhenyu Zhu, Fanghui Liu, Grigorios Chrysos, Francesco Locatello, and Volkan Cevher. Benign overfitting in deep neural networks under lazy training. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 43105-43128. PMLR, 23-29 Jul 2023. URL https://proceedings.mlr.press/v202/zhu23h.html.
A Proofs of Theorems and Derivations
In this section, we prove Lemma 1 in the main paper. This lemma indicates that we can directly imply Linear Mode Connectivity (LMC, see Definition 1) from Layerwise Linear Feature Connectivity (LLFC, see Definition 2) applied to last layer.
Definition 1 (Linear Mode Connectivity). Given a test dataset  $\mathcal{D}$  and two modes  $\theta_{A}$  and  $\theta_{B}$  such that  $\mathrm{Err}_{\mathcal{D}}(\theta_A)\approx \mathrm{Err}_{\mathcal{D}}(\theta_B)$ , we say  $\theta_{A}$  and  $\theta_{B}$  are linearly connected if they satisfy
$$
\operatorname {E r r} _ {\mathcal {D}} \left(\alpha \boldsymbol {\theta} _ {A} + (1 - \alpha) \boldsymbol {\theta} _ {B}\right) \approx \operatorname {E r r} _ {\mathcal {D}} \left(\boldsymbol {\theta} _ {A}\right), \quad \forall \alpha \in [ 0, 1 ].
$$
Definition 2 (Layerwise Linear Feature Connectivity). Given dataset  $\mathcal{D}$  and two modes  $\theta_{A},\theta_{B}$  of an  $L$  -layer neural network  $f$ , the modes  $\theta_{A}$  and  $\theta_{B}$  are said to be layerwise linearly feature connected if they satisfy
$$
\forall \ell \in [ L ], \forall \alpha \in [ 0, 1 ], \exists c > 0, s. t. c f ^ {(\ell)} (\alpha \boldsymbol {\theta} _ {A} + (1 - \alpha) \boldsymbol {\theta} _ {B}) = \alpha f ^ {(\ell)} (\boldsymbol {\theta} _ {A}) + (1 - \alpha) f ^ {(\ell)} (\boldsymbol {\theta} _ {B}).
$$
Lemma 1. Suppose two modes  $\theta_{A},\theta_{B}$  satisfy LLFC on a dataset  $\mathcal{D}$  and
$$
\max  \left\{\operatorname {E r r} _ {\mathcal {D}} \left(\boldsymbol {\theta} _ {A}\right), \operatorname {E r r} _ {\mathcal {D}} \left(\boldsymbol {\theta} _ {B}\right) \right\} \leq \epsilon ,
$$
$$
\forall \alpha \in [ 0, 1 ], \operatorname {E r r} _ {\mathcal {D}} (\alpha \boldsymbol {\theta} _ {A} + (1 - \alpha) \boldsymbol {\theta} _ {B}) \leq 2 \epsilon .
$$
Proof. Note that the classification depends on the relative order of the entries in the output of the final layer. As a consequence, for each data point in the dataset  $\mathcal{D}$ , the linear interpolation of the outputs of the models makes the correct classification if both models make the correct classification. Therefore, only if one of the model makes the incorrect classification, the linear interpolation of the outputs of the models would possibly make the incorrect classification, i.e,
$$
\operatorname {E r r} _ {\mathcal {D}} \left(\alpha f \left(\boldsymbol {\theta} _ {A}\right) + (1 - \alpha) f \left(\boldsymbol {\theta} _ {B}\right)\right) \leq \operatorname {E r r} _ {\mathcal {D}} \left(\boldsymbol {\theta} _ {A}\right) + \operatorname {E r r} _ {\mathcal {D}} \left(\boldsymbol {\theta} _ {B}\right).
$$
Since  $\theta_{A}$  and  $\theta_{B}$  satisfy LLFC, then at last layer we have
$$
f \left(\alpha \boldsymbol {\theta} _ {A} + (1 - \alpha) \boldsymbol {\theta} _ {B}\right) = \alpha f \left(\boldsymbol {\theta} _ {A}\right) + (1 - \alpha) f \left(\boldsymbol {\theta} _ {B}\right),
$$
then have
$$
\operatorname {E r r} _ {\mathcal {D}} \left(\alpha \boldsymbol {\theta} _ {A} + (1 - \alpha) \boldsymbol {\theta} _ {B}\right) \leq \operatorname {E r r} _ {\mathcal {D}} \left(\boldsymbol {\theta} _ {A}\right) + \operatorname {E r r} _ {\mathcal {D}} \left(\boldsymbol {\theta} _ {B}\right).
$$
According to the condition that
$$
\operatorname {E r r} _ {\mathcal {D}} \left(\alpha \boldsymbol {\theta} _ {A} + (1 - \alpha) \boldsymbol {\theta} _ {B}\right) \leq 2 \epsilon ,
$$
and this finishes the proof.
In this section, we prove Theorem 1 in the main paper. Theorem 1 indicates that we can derive LLFC from two simple conditions: weak additivity for ReLU activations (Definition 3) and commutativity (Definition 4). Note that though we consider a multi-layer perceptron (MLP) for convenience, our proof and results can be easily adopted to any feed-forward structure, e.g., a convolutional neural network (CNN).
$$
\forall \ell \in [ L ], \quad \sigma \left(\tilde {\boldsymbol {H}} _ {A} ^ {(\ell)} + \tilde {\boldsymbol {H}} _ {B} ^ {(\ell)}\right) = \sigma \left(\tilde {\boldsymbol {H}} _ {A} ^ {(\ell)}\right) + \sigma \left(\tilde {\boldsymbol {H}} _ {B} ^ {(\ell)}\right).
$$
$$
\forall \ell \in [ L ], \boldsymbol {W} _ {A} ^ {(\ell)} \boldsymbol {H} _ {A} ^ {(\ell - 1)} + \boldsymbol {W} _ {B} ^ {(\ell)} \boldsymbol {H} _ {B} ^ {(\ell - 1)} = \boldsymbol {W} _ {A} ^ {(\ell)} \boldsymbol {H} _ {B} ^ {(\ell - 1)} + \boldsymbol {W} _ {B} ^ {(\ell)} \boldsymbol {H} _ {A} ^ {(\ell - 1)}.
$$
Theorem 1. Given a dataset  $\mathcal{D}$ , if two modes  $\theta_{A}$  and  $\theta_{B}$  satisfy weak additivity for ReLU activations (Definition 3) and commutativity (Definition 4), then
$$
\forall \alpha \in [ 0, 1 ], \forall \ell \in [ L ], f ^ {(\ell)} (\alpha \pmb {\theta} _ {A} + (1 - \alpha) \pmb {\theta} _ {B}) = \alpha f ^ {(\ell)} (\pmb {\theta} _ {A}) + (1 - \alpha) f ^ {(\ell)} (\pmb {\theta} _ {B}).
$$
Proof. Before delving into the proof, let us denote the forward propagation in each layer  $\ell$  by
$$
\tilde {g} ^ {(\ell)} (\boldsymbol {\theta}; \boldsymbol {H} ^ {(\ell - 1)}) = \boldsymbol {W} ^ {(\ell)} \boldsymbol {H} ^ {(\ell - 1)} + \boldsymbol {b} ^ {(\ell)} \mathbf {1} _ {d _ {\ell}} ^ {\top}
$$
$$
g ^ {(\ell)} (\boldsymbol {\theta}; \boldsymbol {H} ^ {(\ell - 1)}) = \sigma (\tilde {g} ^ {(\ell)} (\boldsymbol {\theta}; \boldsymbol {H} ^ {(\ell - 1)})) = \boldsymbol {H} ^ {(\ell)}
$$
Given  $\theta_{A}$  and  $\theta_{B}$  that satisfy the commutativity property, then  $\forall \ell \in [L]$  and  $\forall \alpha \in [0,1]$ , we have
$$
\boldsymbol {W} _ {A} ^ {(\ell)} \boldsymbol {H} _ {A} ^ {(\ell - 1)} + \boldsymbol {W} _ {B} ^ {(\ell)} \boldsymbol {H} _ {B} ^ {(\ell - 1)} = \boldsymbol {W} _ {A} ^ {(\ell)} \boldsymbol {H} _ {B} ^ {(\ell - 1)} + \boldsymbol {W} _ {B} ^ {(\ell)} \boldsymbol {H} _ {A} ^ {(\ell - 1)}
$$
$$
\tilde {g} ^ {(\ell)} \left(\boldsymbol {\theta} _ {A}; \boldsymbol {H} _ {A} ^ {(\ell - 1)}\right) + \tilde {g} ^ {(\ell)} \left(\boldsymbol {\theta} _ {B}; \boldsymbol {H} _ {B} ^ {(\ell - 1)}\right) = \tilde {g} ^ {(\ell)} \left(\boldsymbol {\theta} _ {A}; \boldsymbol {H} _ {B} ^ {(\ell - 1)}\right) + \tilde {g} ^ {(\ell)} \left(\boldsymbol {\theta} _ {B}; \boldsymbol {H} _ {A} ^ {(\ell - 1)}\right)
$$
$$
\begin{array}{l} \alpha \left(1 - \alpha\right) \left(\tilde {g} ^ {(\ell)} \left(\boldsymbol {\theta} _ {A}; \boldsymbol {H} _ {A} ^ {(\ell - 1)}\right) + \tilde {g} ^ {(\ell)} \left(\boldsymbol {\theta} _ {B}; \boldsymbol {H} _ {B} ^ {(\ell - 1)}\right)\right) = \alpha \left(1 - \alpha\right) \left(\tilde {g} ^ {(\ell)} \left(\boldsymbol {\theta} _ {A}; \boldsymbol {H} _ {B} ^ {(\ell - 1)}\right) + \tilde {g} ^ {(\ell)} \left(\boldsymbol {\theta} _ {B}; \boldsymbol {H} _ {A} ^ {(\ell - 1)}\right)\right) \\ \alpha \tilde {g} ^ {(\ell)} (\boldsymbol {\theta} _ {A}; \boldsymbol {H} _ {A} ^ {(\ell - 1)}) + (1 - \alpha) \tilde {g} ^ {(\ell)} (\boldsymbol {\theta} _ {B}; \boldsymbol {H} _ {B} ^ {(\ell - 1)}) = \alpha^ {2} \tilde {g} ^ {(\ell)} (\boldsymbol {\theta} _ {A}; \boldsymbol {H} _ {A} ^ {(\ell - 1)}) + (1 - \alpha) ^ {2} \tilde {g} ^ {(\ell)} (\boldsymbol {\theta} _ {B}; \boldsymbol {H} _ {B} ^ {(\ell - 1)}) \\ + \alpha (1 - \alpha) (\tilde {g} ^ {(\ell)} (\boldsymbol {\theta} _ {A}; \boldsymbol {H} _ {B} ^ {(\ell - 1)}) + \tilde {g} ^ {(\ell)} (\boldsymbol {\theta} _ {B}; \boldsymbol {H} _ {A} ^ {(\ell - 1)})) \\ \end{array}
$$
Additionally, we can easily verify that