Mechanistic Mode Connectivity
Authors: Ekdeep Singh Lubana
Eric J Bigelow
Robert P. Dick
David Krueger
Hidenori Tanaka
Venue: Submitted to ICLR 2023 (2023)
Keywords: Loss landscapes
Mechanisms
Mode Connectivity

Abstract: With the rise of pretrained models, fine-tuning has become of central importance in deep learning. However, unlike retraining from scratch, fine-tuning can fail to qualitatively change the behavior of a pre-trained network. For instance, we find in practice that naive fine-tuning does not eliminate a model’s sensitivity to spurious features. To understand and address this limitation, we study the geometry of neural network loss landscapes through the lens of mode-connectivity. Our work addresses two questions about mode-connectivity: 1) Are models trained on different data distributions mode-connected? 2) Can we fine tune a pre-trained model to switch modes? We define a notion of mechanistic mode-connectivity, and find that only models that already share the same invariances (which we call “mechanistically similar”) are mechanistically mode-connected. We hypothesize this property explains inability of naive fine-tuning methods to induce invariance to spurious features. Based on our analysis, we propose and validate a method of “mechanistic fine-tuning” called connectivity-based fine-tuning (CBFT)

Under review as a conference paper at ICLR 2023
MECHANISTIC MODE CONNECTIVITY
Anonymous authors
Paper under double-blind review
ABSTRACT
Neural networks are known be biased towards learning mechanisms that help identify spurious altribules, yielding features that do not generalize well under distribution shifts. To understand and address this limitation, we study the geometry of neural network loss landscapes through the lens of mode connectivity, the observation that minimizers of neural networks are connected via simple paths of low loss. Our work addresses two questions: (i) do minimizers that encode dissimilar mechanisms connect via simple paths of low loss? (ii) can fine-tuning a pretrained model help switch between such minimizers? We define notion of mechanistic similarity and demonstrate that lack of linear connectivity between two minimizers implies the corresponding models use dissimilar mechanisms for making their predictions. This property helps us demonstrate that naive fine-tuning can fail to eliminate model's reliance on spurious attributes. We thus propose a method for altering a model's 's mechanisms, named connectivity-basee fine-fuming and validate its usefulness by inducing models invariant to spurious attributes.
INTRODUCTION
Deep neural networks (DNNs) suffer from various robustness problems, learning representations that fail generalize well beyond the given training distribution (D' Amour al., 2020; Teney et al., 2022; Geirhos et al., 2020; Recht et al., 2019; Taori et al., 2020; Jacobsen et al., 2018). This lack of robustness is generally consequence of models learning mechanisms that rely on spurious attributes in the training data for making their predictions. Such attributes eeeeees-eeenneee even if not perfectly predictive- tend to be simpler to represent according to the model's 's inductive biases (Nakkiran al., 2019; Valle-Perez et al., 2018 8; Hu et al., 2020; Shah et al., 2020) and commonly emerge due to sampling biases and hidden confounders static datasets (Kaur et al., 2022; Lee et al., 2022). For example, in most vision datasets, backgrounds are correlated with object categories-a a sampling bias (Beery et al., 2018; Xiao et al., 2020). Consequently, a model can learn to predict the correct category of an object by learning mechanisms to identify either its background or its shape; however, only models that rely shape are likely to generalize robustly (Geirhos et al., 2018; Dittadi et al., 2020). Indeed, Scimeca et al. (2021), Hermann & Lampinen (2020) show that using different datasets for task, standard training pipelines can induce models that use entirely distinct mechanisms for making their predictions, performing equally well in-distribution, but vastly differently out-of-distribution. Recent works improving neural networks robustness thus advocate need for modeling the causal mechanisms underlying the data- matt--ennratting process (Arjovsky al., 2019; Krueger et al., 2021; Lu et al., 2021), promoting representations invariant to spurious attributes.
This Work: In this paper, we introduce the idea of mechanistic similarity (Sec. 3) assess whether two models rely on the same input attributes for making their predictions. Specifically, we call two models mechanistically similar if they exhibit invariance the same attributes of an input, but may otherwise produce different representations for it. Our motivating question is whether model can be fine-tuned alter its mechanisms, i.e., to learn different invariances; we call this the problem of mechanistic fine-tuning (Sec. 5). For instance, if model has learned to rely spurious attribute in its training data, can that reliance be climinated by training it on a minimal set of "clean" samples that do not contain the spurious attribute? This problem is of practical value because curating large, clean dataset and training from scratch on it in the first place can be expensive.
We consider the problem above through the lens of mode connectivity in neural networks, which refers to the phenomenon that neural network minimizers identified via training on the same dataset for a task tend to be connected via relatively simple paths of low loss in the model's loss landscape (e.g..

---

Under review as a conference paper at ICLR 2023
linear or quadratic splines) (Garipov et al., 2018; Draxler et al., 2018; Frankle et al., 2020; Entezari et al., 2021: Kuditipudi et al., 2019; Nguyen et al., 2021). In particular, we analyze connectivity of mechanistically dissimilar models that are induced via training on different datasets for a task and can hence learn to rely on entirely distinct attributes of an input to make their predictions (see Fig. 1). We induce such mechanistically dissimilar models by embedding synthetic cues in existing datasets (see Fig. 2) and training models on these datasets under dif- ferent proportions of samples with spurious attributes (see Fig. 4); given easily separable cues, such ma- nipulated data allows training of models that learn mechanisms to preferentially identify the synthetic cue over natural data attributes (Shah et al., 2020). Our extensive analysis (Sec. 4) shows that if two models lack linear connectivity in the landscape, they must be mechanistically dissimilar, that is, increase in loss as we linearly move between two models implies they have learned different invariances. This result holds implications for naïve fine-tuning of pretrained net- works, which often yields models linearly connected with the original pretraining minimizer (Neyshabur et al., 2020) and can hence be insufficient for altering a model's prediction mechanisms (Fig. 5). We thus propose a technique, named Connectivity- Based Fine- Tuning (CBFT), that exploits lack of linear connectivity between mechanistically dissimilar models to induce networks that follow our desired mechanisms (Sec. 5). Our extensive experimental results show CBFT is more effective at reducing a model's sensitivity to spurious attributes than recent techniques (Kirichenko et al., 2022b; Kumar al., 2022).
Non-linear path
Linear path
Background
O'
Shape
Figure 1: Mechanistic Lens on Mode con- nectivity. Consider two sets of parame- ters that minimize loss using background Obackground and object shape Ashape as the input attributes for prediction, respectively. Are such mechanistically dissimilar mini- mizers connected via paths of low loss in the landscape? Does the dissimilarity of these mechanisms affect the simplicity of their connectivity paths? Can we exploit this connectivity to switch between minimizers that use our desired mechanisms?
2 PRELIMINARIES
Notations. Consider a neural network f R" x Rd - [K] that takes n-dimensional inputs X C CR", has parameters 0 Rdd, and produces an output f(x;0) 0) E [K]. We say 0 "induces the model" f(.;; 0). Loss on a dataset D E X' x [K] for parameters 0 is denoted using a non-negative function (D) 0)) and 0 is called minimizer for that dataset if LI) < e, where is some small scalar. In the following, our focus will be minimizers retrieved by running SGD on a dataset's s loss. Assume there
latent space Z C I. with sampled from some factorial distribution, P(=) = [I P(z.). These latent variables are assumed to generate samples in the dataset via g: + X x [K]: (z, := G(2), with x and y conditionally independent given 2. If Gx. Gy define the components of 9 producing x and y. we assume 9x (.) has valid left-inverse 9x² : X - Z, i.e. x contains all of the information necessary to recover the true settings of the (independent) latent variables, z. These assumptions are standard literature on disentanglement 2020: Gresele et al., 2020; 2021; Von Kügelgen et al., 2021) and Independent Component Analysis (ICA) (Hyvarinen & Morioka, 2016; 2017; Khemakhem et al., 2020; 2021).
(Locatello et al., 2019;
2.1 MODE CONNECTIVITY
We denote a continuous path between two sets of parameters 01, 0g as 70,-00 (t), where Yo, -02 (0) = 0 and 1add -0, (1) 02. At times, we say a path lacks loss barriers on dataset, if moving along the path never yields increase in loss on that dataset. Using the above notations, we now formalize the notion of mode connectivity, in line with previous work (Garipov et al., 2018; Draxler et al., 2018).
Definition 1. (Mode Connectivity Along Path.) Minimizers 0,, 02 of loss C( 0)) on dataset D are called mode connected along the path 70.-00 (t) if moving along the path never increases loss. Formally, Vt E [0, 1). (D, 70, (00 S min 0.)), (D; 81))).
As mentioned in Sec. 1, prior work has shown that minimizers of modern neural networks exhibit mode connectivity along rather simple paths in the landscape. This property was first illustrated in
2

---

Under review as a conference paper at ICLR 2023
CIFAR 10 located box
CIFAR 100
CIFAR Fashion-MNIST
located colored
Clasn
Clams
Clean
Class
Chaa
Clea
Image
X
151
Rand. cue
y
Rand. Image
Figure 2: Data-Generale Process (left), We augment the natural latents {zn } of a Lata-generating process with set of synthetic latents {2,)- The attributes induced in the input by these synthetic latents are called cues. Conditioning (grey, dotted line) the value of a synthetic latent on the target label (v), we can induce correlation between its corresponding cue and the desired model output. If the cue is made easily separable, a neural network will preferentially learn mechanisms to use the cue for making its predictions (Shah et al., 2020; Sagawa al., 2020). This allows us controllably embed spurious attributes in the dataset. Synthetic Datasets (right). Following the protocol above, we embed synthetic cues (denoted With cue) three existing dataset: (1) CIFAR-10 with 3 box cues whose locations depend on the target label; (2) CIFAR-100 100 with 3 x box cues colored according to the first digit of the object label, and located according to the second digit; and (3) Dominoes (Shah et al., 2020), where CIFAR- 10 images are concatenated with Fashion-MNIST images of the same class. We analyze counterfactual datasets that involve randomizing the cue (denoted Rand. cue) or natural image (denoted Rand. image) to break their correlation with the target label. This helps us ascertain the extent to which the model's prediction relies on natural vs. spurious attributes.
concurrent works by Garipov al. (2018) and Draxler al. (2018) and has been further expanded by Frankle et al. (2020); Entezari et al. (2021); Benton et al. (2021); Wortsman et al. (2021). Correspondingly, will restrict experiments the following two forms of paths:
Linear: Na. -0. (t) = (1 t)0 + t02, and
(1)
Quadratic: -0, = (1 t) + 2+(1 t)012 + 20
(2)
where 012 denotes set of parameters that is explicitly optimized identify quadratic path connecting the two minimizers 02 and 6g for that dataset. Thus, quadratic paths are a function of the dataset used for their identification. We also note that recent work hypothesizes two linearly disconnected minimizers of dataset can be linearly connected given an appropriate permutation of neurons (Entezari et al., 2021); specifically, one that maintains the model outputs, but aligns them in layer-wise activationsswweights. To demonstrate the robustness of our claims, we follow the alignment method by Singh & Jaggi (2020); Ainsworth et al. (2022) and include experiments on models that have been permuted to match in activations (see App. B.2). We call these paths Linear (permuted).
2.2 EXPERIMENTAL. SETUP
Since goal is to assess the role of mechanistic similarity connectivity of pairs of minimizers, we must design models that use different mechanisms for making predictions. To this end, we propose easily manipulable synthetic datasets. Such datasets have been used by prior works for better understanding several important topics, such as transfer learning (Dittadi et al., 2020), domain generalization (Wiles al., 2021: Van Steenkiste et al., 2019), disentanglement (Higgins et al., 2017; Klindt et al., 2020), self- and semi-supervised learning (Von Kügelgen et al., 2021; Trivedi et al., 2022; Locatello et al., 2020), and inductive biases of neural networks (Hermann et al., 2020; Hermann Lampinen, 2020; Ritter et al., 2017). Our data-generan seattaggneratnng process is illustrated in Fig. 2 and involves augmenting the natural latents of the process with synthetic ones that are conditioned on the target label. We refer to the attributes induced in the input by such latents as cues. We intentionally choose cues that are easily separable to cause the model to prefer them over natural attributes for making its predictions (Shah et al., 2020). Such low-complexity cues be viewed as stand-ins for spurious or shortcut attributes that are commonplace in realistic settings (Beery et al., 2018; Geirhos et al., 2020), allowing to determine whether minimizers that induce models reliant either spurious
3

---

Under review as a conference paper at ICLR 2023
non-spurious attributes exhibit mode connectivity. Training curves for different models are shown in App. A and clearly demonstrate that models trained without cues entirely ignore them for making their predictions; meanwhile, under perfect correlation, the model completely ignores the natural data and relies entirely on cues. Note that the data-generatis counterfactuals (see Def. 2). Specifically, we focus interventions that break cue's correlation with the target label by randomizing it (c.g, uniformly changing location of the box in the CIFAR-10 with box cue dataset). Vice-versa, we also analyze interventions that keep the cue intact, but randomize the underlying image (e.g., putting a cat instead of dog). This helps us assess how much model relies on natural attributes that originate in the source image vs. our synthetically embedded cues.
process is easy to intervene on for creating
DEFINING MECHANISTIC SIMILARITY
While mode connectivity via simple paths is surprising result that indicates the loss landscape of modern neural networks is relatively benign, prior work has not addressed whether models that rely on different mechanisms for making their pre- dictions exhibit mode connectivity as well (see Fig. 1). To analyze this question, we must first define a notion of mechanistic similarity between two models. Our definition is motivated by the fol- lowing question: do models that succeed in a simi- lar manner, fail in a similar manner? We propose to assess failures by measuring a model's 's response to input formattiio::: if two models use simi- lar mechanisms, they should respond similarly to transformed inputs. This is analogous to the use of visual illusions (invalid perceptual inferences) for designing models of early visual processing in neuroscience and cognitive science (Tanaka et al., 2019; Marr, 1976). By choosing transformations that embody task-relevant vulnerabilities, we can make this definition operationally well-motivated. For example, by randomizing the synthetic cue in Fig. 2, we can assess whether a model relies on spurious attribute for making its predictions. We next formalize the intuition introduced above.
Model 9,
Model
Model 3 0,
0g
Fish
Deta-Generating Process
Z2
Non-fish
Fish
22
Non-fish
21
Fish
Non-fish
Figure 3: Mechanistic Similarity: We define mechanistic similarity of two models based on how they respond to unit interventions on the data- generating process, i.e., interventions on specific dimensions of the latent vector z; e.g., Ai (shape) and A2 (background) in the figure. Here, yellow circles represent the prediction of given model (column) on a counterfactual image (row), i.e., [ (x;; 0,); Models whose predictions are invariant to the same set of interventions (denoted 0, ~ 02) are termed mechanistically similar.
Definition 2. (Unit Interventions on Dete-Gnng Process and Counterfactuals.) An isomor- phism AT Z, Z,- Z, defines unit intervention the th dimension of the state if it alters its value by setting it to a scalar value 0g E Z,. The isomorphism & X x R.m -> X defines a counterfactual if it alters datapoint x: by changing its corresponding latent
x (a)) via a of
unit interventions À = {LA, J {AA...m
Specifically, we have A(; = 9x o A o... o A o gg (z).
Thus, a unit intervention on the data-generating process allows precise manipulation of the latent state 2, while counterfactual maps this changed state into the observable data space. Due to independence of latent dimensions, our definition of unit interventions casily composes and can model broader notions of interventions (Schölkopf et al., 2021); )); combined with counterfactuals, unit interventions are thus sufficient to assess a model's response to general data transformations, as we show next.
Definition 3. (Invariance.) The model (-; 0) is termed invariant to unit intervention A, if counter- factuals generated by A, do not increase its loss, iee,, L(J(D;)) 0)) = EaEz, AQ); O)). Proposition 1. (Exhaustiveness of Unit Interventions.) If f(; 0) is invariant to unit interventions A and Ay. it must be invariant to their composition; conversely, lack of invariance to either A, or Aj precludes invariance to their composition.
Essentially, the above statement argues studying a model's response to unit interventions is sufficient to characterize which attributes of the data the model is using for prediction: if a model is invariant to a set of unit interventions, it must be invariant to all counterfactuals generated by their composition too; similarly, lack of invariance counterfactuals produced by the composition of that intervention and a set of invariant interventions. We are now ready define mechanistic similarity.
single unit intervention is sufficient to preclude invariance to all
4

---

Under review as a conference paper at ICLR 2023
Definition 4. (Mechanistic Similarity.) Consider a set of unit interventions À := {,i), where i [m]] For parameters 0, denote the subset of interventions that f(;;0) 0) is invariant to as I(0) Å. Then, f(;01) and f.):02) are called mechanistically similar if [(11) = I(02).
Fig. 3 iillustrates mechanistic similarity in an intuitive manner. Formally, given a set of independent transformations (instantiated by use of unit interventions), we say two models are mechanistically similar if they exhibit invariance to the same set of interventions. Our definition shares motivation with the idea of prediction mismatch, which involves assessing the number of distinct examples two models produce different predictions on, and has been used in prior work to analyze properties such as calibration and catastrophic interference (Hooker al.. 2019: Mania et al., 2019; Maini et al., 2022). In contrast, mechanistic similarity is based on assessment of the number of distinct interventions on the gata-genn process to which two models are simultaneously invariant. This makes mechanistic similarity more appropriate for problems involving distribution shifts and robustness, where modeling the data generating process is of crucial importance (Kaur et al., 2022). We next extend the definition of mode connectivity while accounting for mechanistic similarity.
Definition 5. (Mechanistic Connectivity Along a Path.) Consider two minimizers 01 and 0g of loss ((f(D;)) on a dataset DD Let ( == {E(D; )VT= denote set of counterfactual datasets designed by applying unit interventions A, to all points in dataset D, where intervention assignments Og; are chosen uniformly from the respective range of values Z,, Then, 0, and O2 are called mechanistically connected along the path "Yo, v0,-0,(0 (t) if. for all counterfactual datasets. moving along the path does not yield increase in loss; that is, 81, fg exhibit mode connectivity along the same path - (1) for all counterfactual datasets.
Essentially, if two minimizers exhibit mechanistic connectivity, then for all pre-defined interventions, there exists a single connectivity path such that moving along it does not yield increase in loss on all counterfactual datasets described by the interventions; that is, all points on the path induce mechanistically similar models. Mcanwhile, if two minimizers induce mechanistically dissimilar models, moving along any path between them will necessarily involve a change in the mechanisms used for making predictions. If this change yields increase in loss on a counterfactual dataset, then it is harmful for the distribution shift described by the corresponding intervention. Mechanistic connectivity is designed to succinctly capture this behavior and characterize the connectivities of mechanistically (dis)similar models.
4 CONNECTIVITY OF MECHANISTICALLY DISSIMILAR MODELS
Having developed a language to describe the mechanistic similarity of two models, we now char- acterize connectivity properties exhibited the landscape by mechanistically dissimilar models. Specifically, we demonstrate that while mechanistically dissimilar models can exhibit connectivity via quadratic paths, they are generally not linearly connected. We start with the following propo- sition, which follows directly from the results of Nguyen (2019); Simsck et al. (2021), and shows mechanistically dissimilar models can indeed be mode-connected.
Proposition 2. (Mode Connectivity of Mechanistically Dissimilar Models.) Assume 01, 02 are minimizers of the loss on a dataset D and induce mechanistically dissimilar models. Given sufficien overparameterization, there exists a continuous path along which the minimizers are mode connected.
That is, even if two minimizers of loss on a dataset D induce models that rely on completely disparate mechanisms, there necessarily exists continuous path along which the two minimizers exhibit mode connectivity. Note, however, the claim does not yet address the simplicity of these connectivity paths, which is empirically observed to be surprisingly high for minimizers retrieved from the same dataset.
To investigate whether this property holds for mechanistically dissimilar models, we train VGG-13 and R8 models on the synthetic datasets described in Sec. 2.2. We analyze accuracy on counterfactual datasets (see Fig. 2) along quadratic and linear paths (see Eq. 1), including quadratic paths identified using data with/without cue, linear paths, and linear (permuted) paths. Results for ResNet- 18 are shown in Fig. 4 and remaining are deferred to App. D. We find that points on the connectivity paths respond differently to counterfactuals, indicating lack of mechanistic connectivity (see Def. 5). Simultaneously, we see minimizers that induce mechanistically dissimilar models can be mode connected via relatively simple paths well: we can easily identify quadratic, but not linear, mode connectivity paths for two mechanistically dissimilar models. In fact, we conjecture that lack of linear connectivity between two models is intricately related to their mechanistic similarity.
5

---

Under review as a conference paper at ICLR 2023
60%
70%
80%
90%
100% A/a Cup
Cup
wio Cue
Rand, Cue
Rand. Image
w/ Cue
Rand. Cue
Rand. Image
100
0.3
0.7
0.3
0.3
0.7
0.3
0.7
0.
50
0.3
0.7
Bac
0.3
0.7
0uc
0.3
0.7
0.7
0.5
D.7
0.7
0.7
0.3
6.7
Quadratic (w/
300
:0
03
Test 189)
50
0.3
0.0
O.F 0.7 0.5 0.5
0.1
0.7
0.1 02
0.3
0.7
Linear
Linear (Permuted)
Figure 4: Are Mechanistically Dissimilar Minimizers Connected? We train ResNet- 18 models our synthetic CIFAR-10 datasets with and without box-cues (denoted Oc and INc. respectively). Line colors denote proportion of the training data with synthetic cues. Plot titles denote evaluation data (see Fig. 2). including data where either the cue is present (w/ Cue), absent (w/o Cue), randomized (Rand. Cue), or the underlying image randomized (Rand. Image). As shown, 8xc yields the same performance upon randomization of the cue, while the performance of 6e decreases substantially; ie., the two minimizers induce mechanistically dissimilar models. We see: (i) quadratic paths can be easily identified to mode connect mechanistically dissimilar models; (ii) linear paths cannot be identified, even after permutations; and (iii) mechanistic connectivity does not emerge. We see qualitatively similar results for other datasets, models, and loss (see App. D).
Conjecture 1. (Lack of Linear Connectivity implies Mechanistic Dissimilarity)) If two minimizers 9, and 02 of the loss (Df(D:)) of neurons), their corresponding models [(; 01), (:; (2) must be mechanistically dissimilar.
dataset D cannot be linear mode-comeeeee mode-connected (up to permutations
In App. F, prove that the above claim holds for second-order approximation of the landscape and two-layer model on simplified data-generating process. Here, we show extensive empirical evidence of its validity in more complex settings. Specifically, follow the experimental protocol of Neyshabur et al. (2020), who demonstrate that pretrained model exhibits linear mode connectivity on the original pretraining dataset before and after fine-tuning on another target dataset. Specifically, we train VGG-13 and ResNet- 18 models on our synthetic datasets with partially predictive cues and then fine-tune them on data without cues. Results on CIFAR- 10 with box cues are shown Fig. and remaining are deferred to App. E. We see that when linear mode connectivity does not hold, the fine-tuned models behave differently on counterfactuals. For example, the models before and after fine-tuning using large learning rate do not exhibit linear mode connectivity: correspondingly. the fine-tuned models exhibit clear invariance to cue attributes, while the pretrained models do not. Similarly, under perfect correlation between labels and cue attributes, with even small initial learning rate, fine-tuned models are not linear mode connected with their pretrained counterparts; correspondingly, the fine-tuned models learn different mechanisms and show different behavior counterfactuals. This result can also be analyzed through the lens of gradient starvation (Pezeshki et al., 2021), which argues that the learning dynamics for simple but highly predictive attributes suppress the learning of other, more complex predictive attributes. This implies the model will be rendered invariant natural attributes our setup, which are generally more complex represent (Scimeca et al., 2021; Hermann et al., 2020). Consequently, the model lacks any transferable mechanisms and hence the model's mechanisms have to be altered to fit the new target distribution. To summarize, these results illustrate that altering model's mechanisms, such as inducing invariance to spurious attributes, can require surmounting loss barriers. Naive fine-tuning can thus fail at this task when using small or moderate learning rates. While large learning rates can overcome this specific problem, they also obviate pretraining: the goal of pretraining is to help improve sample efficiency before transfer target distribution; fine-tuning with large learning rates will distort any useful features learned from the pretraining dataset and render the pipeline's sample efficiency similar to training from seratch, shown by He al. (2019); Kumar al. (2022).

---

Under review as a conference paper at ICLR 2023
80%
70%
100%
60%
70%
80%
90%
100%
Rand.
Hand.
Rand
100
150
0.7
0.7
0.7 BY
0.7
5.3
fr
0.3
0.7
0.3
0.3
Small LR (0.001)
Test Alc. 1950
Small LR 10 001:
Test
Medium UR (0.00 011
Medium LR 10.011
100
100
6.
07
53
07
0.3
0.7
0.3
0.
53
07
0.1
0.7
03
07
03 07 4r
Large LR (0.1)
Large (0.1)
(a) VGG.
(b) ResNet18.
Figure 5: Naïve Fine-Tuning Tuning Can Fail to Alter a Model's Mechanisms. We train VGG- 13 and ResNet-18 18 models on our synthetic CIFAR-10 dataset with box-cues and perform naïve fine-tuning on data without cues for 100 epochs using different initial learning rates (L.R) and step-decay schedule. Corresponding models are denoted Ic and θrr: line colors denote proportion of dataset with synthetic cues; titles denote evaluation datasets, similar to Fig. 4. We plot test accuracy as function of location on the linear paths (after permutation). Using a large learning rate or enforcing perfect correlation between the cue and label induces loss barriers along the linear path, i.e., linear mode connectivity does not hold. Correspondingly, the models respond differently to counterfactuals, ie, they are mechanistically dissimilar and not connected. For small/medium learning rate, θFT remains mechanistically similar to Oc, responding similarly on counterfactuals. Correspondingly, linear mode connectivity holds between the models for data with cues. We see qualitatively similar results for other datasets, models, and loss (see App. E).
5 MECHANISTIC FINE- TUNING: ALTERING A MODEL'S MECHANISMS
In this section, explore the problem of mechanistic fine-tuning, where one aims to alter a specific existing mechanism in pretrained model by fine-tuning the model on a small dataset where the mechanism targeted for alter- ation performs poorly. This problem setup is also common in recent works on eliminating a model's reliance on spu- rious attributes (Kirichenko al., 2022b; Kumar et al.,
Barrier
Invariance
Accuracy
w/ Attribute
w/o Attribute
Oc
Linear
ONC
2022). We now show that our newfound understanding of neural network loss landscapes from the perspective of mechanistic similarity can be used to address this problem.
Figure 6: Geometric clues for alter- ing a model's mechanism. Given a discriminative input attribute C, lin- ear path connecting a model (0xc) in- variant the attribute versus model that relies on the attribute (0c) shows (i) accuracy drop (loss barrier) along the path and (ii) invariant predictions at the endpoint corresponding to tc. i.c.. L(ONc. Dxc) L(ONc. Dc).
Connectivity-Bant Based Fine- Tann-Tunngg: Sec. shows that lack of linear mode connectivity between two models is intri- cately related to their mechanistic dissimilarity. We thus ar- gue that a valid strategy for altering model's mechanisms involves moving the model to a region in the landscape that does not exhibit linear mode connectivity to the current parameters. However, there can be multiple such linearly disconnected regions and we specifically want the ones that boast our desired invariances. To identify these regions, we propose to use the minimal dataset, denoted DNc. that does contain the attribute C that the mechanism targeted for alteration in model f.; 0c) tries to identify (see Fig. 6). Specifically, let LCF. denote the cross-entrope loss; D' denote the subset of dataset 'D corresponding to samples that belong to the ith class K K-class classification problem; 700-50c (t) denote the linear path between 0 the parameters we are trying to identify, and dc; and fx; f) denote the model's representation for an input æ. Then, Connectivity-Be -Baed Fine- Tine-Tuning (CBFT) involves rumning alternating minimization on the following losses:
(i) (0) = argmin BewTTumcee W...0. AA Lex (cx.(((Der) To (t)), v)1;
K
(3)
(ii (z(0) argmin (av 0).
Aa
Bleng (f((0) EzEDg ((:())))
K

---

Under review as a conference paper at ICLR 2023
Table 1: Evaluating CBFT. We train ResNet-18 models on our synthetic CIFAR-IO, CIFAR-100, and Dominoes dataset with different proportions of samples with cue features and fine-tune them using 2500 "clean" samples from a dataset without any cues. Test accuracies (%) on counterfactual test datasets with No Cue (NC), with Cue (C), Randomized Cue (RC), and Randomized Image (RI) are reported (mean of three seeds). We compare our method, Connectivity- with several baselines: Fine-tuning with a medium/small learning rate (FT LLR (Kirichenko et al., 2022b), and LPFT (Kumar et al., 2022). ~ denotes invariance is desirable, i.c., accuracy should be similar to that NC; 1V. indicate higher/lower accuracy is desirable; best results are in bold. We generally see that all baselines yield large degradations in the absence of cues, and even achieve very high accuracy when the underlying image is randomized. Meanwhile, CBFT is able break reliance on cues, inducing representations that are often completely invariant to its presence.
Fine-Tume Tuning (CBFT),
60% Cue dain
70% Cue data
80% Cue data
90% Cue duta
C-10
NC
C
RC
RI
NC
C
RC
RI
NC1
C
RC
R1
NCt
C
RC
R1
FTM
75.7
98.4
23.6
83.4
75.8
98.6
21.7
78.6
71.3
97.7
37.6
63.6
67.2
95.4
49.6
46.6
FT's
75.8
98.7
17.5
90.1
74.9
9%.8
16.3
91.1
69.9
98.4
15.7
90,9
64.7
97.9
15.3
90.7
LLR
71.6
95.1
36,3
57.1
70.9
95.8
29.9
65.8
65.1
81.8
27.0
53.2
59.3
70.7
24.6
40.7
LPFT
70.6
88.1
21.0
70.7
69.6
87.3
18.7
72.5
64.4
63.8
18.8
48.0
59.7
56.6
19.8
37.8
CBFT
74.1
71.5
73,4
8.75
73.2
69.2
72.3
8.60
70.0
70.0
69.5
9.68
67.9
72.5
68.1
13.1
C-100
NC1
C
RC
RI+
NCT
CM
RC
R1
NC1
C
RC
RI
NCT
CM
RC
R1
FT..
44.4
99.2
12.8
85.3
40.3
99.6
12.3
89.8
33.6
99.0
11.4
90.5
25.2
79.2
9.79
57.9
FT.
43.1
99.6
10.3
93.6
38.2
99.7
10.5
95.7
32.5
99.6
10.4
97.0
24.5
39.4
4.87
30.9
35.5
99.2
12.1
89.0
31.5
98.6
11.3
89.6
25.3
96.7
10.6
89.4
18.9
75.1
9.1
58.7
LPFT
35.1
93.2
10,3
82.3
31.1
90.2
9.89
78.5
25.6
89.6
9.70
80.8
18.7
28.6
4.42
19.6
CBFT
42.7
65.0
36.4
14.6
38.5
66.7
34.7
21.2
34.6
69.3
23.0
27.9
28.5
72.9
23.2
46.0
Dom.
NCT
CT
RC
RE+
NCT
C
RC
RI
NCT
C
RC
RI
NCT
CM
RC
R14
FT..
77.4
96.8
43.8
56.1
76.6
96.6
42.7
58.7
74.1
95.7
41.7
61.3
68.8
95.1
40.0
57.5
FT.
76.4
96.9
37.5
62.4
76.8
96.6
32.5
66.5
73.2
96.4
30.8
67.7
67.3
95.2
31.2
65.6
39.8
53.0
73.9
93.2
36.3
54.7
70.8
84.8
46.6
61.3
77.0
39.0
LPFT
73.2
92.5
38.0
51.8
72.7
88.0
34,8
50.9
69.4
34.8
33
39.1
61.2
60.8
31.2
26.6
CBFT
72.0
64.9
67.5
9.9
71.5
70.0
59.2
12.1
70.8
69.7
65.9
11.9
67.2
68.7
61.5
14.9
The first step above uses a truncated Gaussian distribution constrained to the range [0, 1] to randomly sample a point on the linear path between 0, 0g and maximizes the loss at this point up to an upper bound Y1. This promotes a loss barrier near the center of the linear path between 0, 6c. Mcanwhile, the second step enforces invariance to the attribute C by reducing the distance between class-average representations on DxC and Dc. while simultaneously promoting learning of correct labels on the minimal dataset DNC- Detailed ablations are shown in App. C. and we see both steps are important for getting the desired results: the barrier loss helps induce mechanistically dissimilar model, while adding an invariance penalty helps select the exact mechanisms we want the models to differ in.
We empirically validate the effectiveness of CBFT compared to recent baselines designed for reducing a model's 's reliance on spurious attributes (Kirichenko et al., 2022b; Kumar et al., 2022) (see App. A.2 for training details). Results are reported in Tab. 1. We that while the baselines perform well on clean data, they do not yield desired behavior on counterfactual datasets: e.g., they achieve high accuracy even if we randomize the image. In contrast, we see that beyond just performing well on clean data, CBFT models show the desired behaviors: sensitivity to randomization of the image and invariance to spurious attributes. These results suggest CBFT successfully performs mechanistic fine-tuning and provide further corroboration to the claim that lack of linear connectivity implies mechanistic dissimilarity between two models (see Conj. 1).
Evaluating CBFT:
6 RELATED WORK
Mode connectivity. Existence of a single, continuous manifold connecting global minimizers was first identified theoretically by Freeman & Bruna (2016); Nguyen (2019) and empirically discovered in concurrent works under the title of "mode connectivity" by Garipov et al. (2018) and Draxler et al. (2018). A geometrical characterization of this manifold was provided by Simsek et al. (2021), who showed the manifold is primarily composed of affine subspaces, i.e., linearly connectable solutions. Connectivity properties have been used for designing and analyzing algorithms for several practically relevant applications, such as ensembling (Benton et al., 2021; Izmailov et al., 2018; Wortsman et al., 2021; 2022a), network pruning (Frankle et al., 2020; Entezari et al., 2021), adversarial robustness (Zhao et al., 2020), and multi- calliiitsk//ontiiuall learning (Mirzadeh et al., 2020). During
8

---

Under review as a conference paper at ICLR 2023
Linear Mode
Nonlinear Mode
Linear Mechanistic
Nonlinear Mechanistic
Mechanistically Similar Mechanistically Dissimilar
V
X
Table 2: Summarizing our Findings. v, X respectively indicate whether there always exist paths along which mechanistically (dis )smilar models identified using gradient-based optimization can exhibit the type of connectivity specified in the column title.
the course of this work, we became aware of the contemporary empirical paper by Juneja et al. (2022), who investigate whether minimizers connected via linear paths follow similar "decision rules". Their analysis focuses on NLP tasks and does not involve modeling the data- gana-gennr process or counterfactual evaluation; their results can be regarded use of alternative strategy to further verify our claims on a different modality.
Fine-tuning is well-established practice in deep learning. The most basic fine-tuning method is to treat the pretrained model as an initialization, and continue training with new data, A variant is to train only subset of parameters, such as the final classification layer (Kirichenko et al., 2022b;a), possibly fine-tuning the entire model after that (Kumar et al., 2022; Rosenfeld et al., 2022). While Kirichenko et al. (2022b) argue that "last layer re- re-training sufficient for robustness to spurious correlations", our more in-depth evaluation shows that this method actually does not eliminate sensitivity to spurious features. Our findings are more congruent with those of Neyshabur et al. (2020), who find that fine-tuned models tend to remain linearly connected to the pretrained model, suggesting that fine-tuning may fail to change the prediction mechanisms used by a model.
Fine-tuning.
Model editing, Model editing refers to fine-tuning approaches that aim to make targeted change to particular aspect model's behavior without incidentally affecting other aspects. For instance, Sinitsin et al. (2020) give the example of correcting model's prediction eπrоr particular example without changing its predictions on other examples. Prior work on model editing aims to make changes that are "local" in input space, e.g., only affecting the model's "understanding" of who the current prime minister of the UK is (Mitchell et al., 2021; Santurkar et al., 2021). ) Mechanistic fine- tuning shares this motivation of "targeted" alteration of a model; however, the problem's 's overarching goal is to make changes to the causal mechanisms the model implements to make its predictions. Specifically, we aim to make model invariant features that is was not already invariant (or vice versa), without changing any of its other learned invariances. Such change can influence many of a model's 's predictions, making model editing approaches inappropriate for our setup.
7 CONCLUSION
In this work, we characterized how minimizers that induce models reliant on different prediction mechanisms are connected in the loss landscape (see Tab. 2 for summary). Such models can perform equally well on a specific distribution, yet differ vastly in their performance on other distributions. We proposed a notion of mechanistic similarity to describe such models, instantiating the idea with shared invariances, and extending the prior notion of mode-connectivity to account for mechanistic similarity, which we call mechanistic connectivity. Our goal is mechanistic fine-tuning, in which one alters certain specific prediction mechanisms used by a model. Our analysis reveals several surprising findings: (i) mechanistically dissimilar minimizers can be mode-connected via relatively simple, but non-linean paths; (ii) linear mode connectivity of two minimizers is intricately related to the mechanistic similarity of their induced models; (iii) na've fine-tuning can fail eliminate spurious attributes learned during pretraining, arguably due to loss barriers between mechanistically dissimilar minimizers; and (iv) finding linearly disconnected regions in the landscape enables sample-efficient alteration of a model's mechanisms. While the datasets we use differ only via our introduction of synthetic attributes, it should be straightforward to generalize our work to any other setting involving two models (with the same parameter space) trained on different distributions. For instance, it can be interesting to compare models trained on completely different tasks or data modalities. It can also be interesting, in the light of our results, to analyze and understand benefits/ limitations of recent methods that exploit linear interpolability of fine-tuned models to create compute-efficient ensembles (Wortsman et al., 2022b;a). We leave these directions to future work.
"There exceptional, but primarily thcoretical, cases where the connectivity defn, can hold (sec App. F).

---

Under review as a conference paper at ICLR 2023
REFERENCES
Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git Re-Basin: Merging Models modulo Permutation Symmetries. arXiv preprint arXiv:2209.0455 2022.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02880 2019.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning, pp. 322-332. PMLR, 2019.
Randall Balestriero. Neural decision trees. arXiv preprint arXiv: vrXiv:1702.07335 2017.
Randall Balestriero and Richard Baraniuk. Mad max: Affine spline insights into deep learning. arXiv preprint
2018.
Randall Balestriero al. spline theory of deep learning. In International Conference on Machine Learning, pp. 374-383. PMLR, 2018.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pp. 456-473, 2018.
Gregory Benton, Wesley Maddox, Sanae Lotfi, and Andrew Gordon Gordon Wilson. Loss surface simplexes for mode connecting volumes and fast ensembling. In International Conference on Machine Learning, pp. 769-779. PMLR, 2021.
Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13475 2021.
Andrea Dittadi, Frederik Träuble, Francesco Locatello, Manuel Wüthrich, Vaibhav Agrawal, Ole Winther, Stefan Bauer, and Bernhard Schölkopf. On the transfer of disentangled representations in realistic settings. arXiv preprini arXiv:2010. 14407, 2020.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference on machine learning, pp. 1309-1318. PMLR, 2018.
Alexander D' Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al. Underspecification presents challenges for credibility in modern machine learning. Journal of Machine Learning Research, 2020.
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance linear mode connectivity of neural networks. arXiv preprint arXiv:2110.06255 2021.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference Machine Learning, pp. 3259-3269. PMLR, 2020.
C Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimization. arXiv preprint 161 1.0150,, 2016.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31, 2018.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv: 1811.1221., 2018.
Robert Geirhos, Bethge, and Felix Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 1):665-673, 2020.
Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias
10

---

Under review as a conference paper at ICLR 2023
Luigi Gresele, Paul K Rubenstein, Arash Mehrjou, Francesco Locatello, and Bernhard Schölkopf. The incomplete rosetta stone problem: Identifiability results for multi-view nonlinear ica. In Uncertainty in Artificial Intelligence. pp. 217-227. PMLR, 2020.
Luigi Gresele, Julius Von Kügelgen,, Vincent Stimper, Bernhard Schölkopf, and Michel Besserve. Independent mechanism analysis, a new concept? Advances in neural information processing systems, 34:28233-2822 2021.
Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. Advances in Neural Information Processing Systems, 31, 2018.
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4918-4927, 2019.
Robert Hecht-Nielsen. On the algebraic structure of feedforward network weight spaces. In Advanced Neural Computers, pp. 135. Elsevier, 1990.
Katherine Hermann and Andrew Lampinen. What shapes feature representations? exploring datasets, architectures, and training. Advances in Neural Information Processing Systems, 33:9995-10006, 2020.
Katherine Hermann, Ting Chen, and Simon Kornblith. The origins and prevalence of texture bias in convolutional neural networks. Advances in Neural Information Processing Systems, 33: 19000-19015, 2020.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. International Conference Learning Representations, 2017.
Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. What do com- pressed deep neural networks forget? arXiv preprint arXiv: 1.1105248, 2019.
Wei Hu, Lechao Xiao, Ben Adlam, and Jeffrey Pennington. The surprising simplicity of the early-time learning dynamics of neural networks. Advances Neural Information Processing Systems, 33: 17116-17128, 2020.
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in neural information processing systems, 29, 2016.
Aapo Hyvarinen and Hiroshi Morioka. Nonlinear ICA of temporally dependent stationary sources. In Artificial Intelligence and Statistics, pp. 460-469. PMLR, 2017.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson, Av- eraging weights leads to wider optima and better generalization. arXiv preprint 803 2018.
Jörn-Herrik Jacobsen, Jens Behrmann, Richard Zemel, and Matthias Bethge. Excessive invariance causes adversarial vulnerability. arXiv preprint 181 1.00401, 2018.
Jeevesh Juneja, Rachit Bansal, Kyunghyun Cho, João Sedoc, and Naomi Saphra. Linear Connectivity Reveals Generalization Strategies. arXiv preprint arXiv:2205. 2022.
Jivat Neet Kaur, Emre Kiciman, and Amit Sharma. Modeling the data-generating process is necessary for outtof-distribbutio generalization. arXiv preprint arXiv:2206.07837, 2022.
Kenji Kawaguchi. Deep learning without poor local minima. Advances in neural information processing systems, 29, 2016.
Kenji Kawaguchi and Leslie Kaelbling. Elimination of all bad local minima in deep learning. In International Conference Artificial Intelligence and Statistics, pp. 853-863. PMLR, 2020.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pp. 2207-2217. PMLR, 2020.

---

Under review as a conference paper at ICLR 2023
Ilyes Khemakhem, Ricardo Monti, Robert Leech, and Aapo Hyvarinen. Causal autoregressive flows. In International conference on artificial intelligence and statistics, pp. 3520-3523 PMLR, 2021.
Polina Kirichenko, Pavel Izmailov, Nate Gruver, and Andrew Gordon Wilson, On Feature Learning in the Presence of Spurious Correlations. arXiv preprint arXiv:22 10. 11369, 2022a.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness to spurious correlations. arXiv preprint (rXX::2204.02937, 2022b.
David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. arXiv preprint wrXiv:2007.10933 2020.
David Krueger, Ethan Caballero,
Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapola- tion (rex). In International Conference on Machine Learning, pp. 5815-5826. PMLR, 2021.
Rohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Rong Ge, and Sanjeev Arora. Explaining landscape connectivity of low-cost solutions for multilayer nets. Advances in neural information processing systems, 32, 2019.
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint 400544 2022.
Yoonho Lee, Huaxiu Yao, and Chelsea Finn. Diversifyy and disambiguate: Learning from underspeci- fied data. arXiv preprint arXiv:2202.03418, 2022.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentan- gled representations. In international conference machine learning, pp. 4114-4124. PMLR, 2019.
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In International Conference on Machine Learning. pp.
PMLR, 2020.
Chaochao Lu, Yuhuai Wu, Jose Miguel Hernández-Lobato and Bernhard Schölkopf. Nonlinear invariant risk minimization: A causal approach. arXiv preprint arXiv:2102.12353 2021.
Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. arXiv preprint arXiv: 19X6::9000058000 2019.
Pratyush Maini, Saurabh Garg, Zachary Chase Lipton, and J Zico Kolter. Characterizing Datapoints via Second-Si -Spiit Forgetting. In ICML 2022: Workshop on Spurious Correlations, Invariance and Stability, 2022.
Horia Mania, John Miller, Ludwig Schmidt, Moritz Hardt, and Benjamin Recht. Model similarity mitigates test set overuse. Advances in Neural Information Processing Systems, 32, 2019.
David Marr. Early processing of visual information. Philosophical Transactions of the Royal Society of London. B, Biological Sciences, 275(942):4835119 1976.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu, and Hassan Ghasemzadeh. Linear mode connectivity in multitask and continual learning. arXiv preprint arXiv:2010.04495, 2020.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. arXiv preprint arXiv:2110.11309, 2021.
Mor Shpigel Nacson, Jason Lee, Suriya Gunasekar, Pedro Henrique Pamplona Savarese, Nathan Srebro. and Daniel Soudry. Convergence of gradient descent on separable data. The 22nd International Conference Artificial Intelligence and Statisties, pp. 3420-3428. PMLR, 2019.
12

---

Under review as a conference paper at ICLR 2023
Preetum Nakkiran, Dimitris Kalimeris, Gal Kaplun, Benjamin Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. Sgd on neural networks learns functions of increasing complexity. Advances in neural information processing systems, 32, 2019.
Behnam Neyshabur, Hamie Sedghi, and Chiyuan Zhang. What being transferred transfer learning? Advances in neural information processing systems, 33:512-523, 2020.
Quynh Nguyen. On connected sublevel sets in deep learning. In International conference machine learning, pp. 4790-4799. PMLR, 2019.
Quynh Nguyen, Mahesh Chandra Mukkamala, and Matthias Hein. On the loss landscape of class of deep neural networks with no bad local valleys. arXiv preprint 1rXX::1809110744 2018.
Quynh N Nguyen and Marco Mondelli. Global convergence of deep networks with one wide layer followed by pyramidal topology. Advances in Neural Information Processing Systems, 33: 11961-11972, 2020.
Quynh N Nguyen, Pierre Bréchet, and Marco Mondelli. When Are Solutions Connected in Deep Networks? Advances in Neural Information Processing Systems, 34:20956-2008 20969, 2021.
Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron C Courville, Doina Precup, and Guil- laume Lajoie. Gradient starvation: learning proclivity in neural networks. Advances Neural Information Processing Systems, 34: 14:12-1112 2021.
Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthieu Cord. Diverse Weight Averaging for Out-of-Distribution Generalization. arXiv preprint arXiv:2205.09739, 2022.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize imagenet? In International Conference on Machine Learning, pp. 5389-5400. PMLR. 2019.
Samuel Ritter, David GT Barrett, Adam Santoro, and Matt M Botvinick. Cognitive psychology for deep neural networks: A shape bias case study. In International conference on machine learning. pp. 2940-2949 PMLR, 2017.
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. Domain-adjusted regression Erm may already learn features sufficient for out-of-distribution generalization. arXiv preprint 2.Xiv:2202.0685: 2022.
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterizattin exacerbates spurious correlations. In International Conference on Machine Learning, pp. 8346-833 8356. PMLR, 2020.
Shibani Santurkar, Dimitris Tsipras, Mahalaxmi Elango. David Bau, Antonio Torralba, and Alek- sander Madry. Editing classifier by rewriting its prediction rules. Advances in Neural Information Processing Systems, 34:23359-23:3 2021.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Towards causal representation learning. arXiv preprint arXiv:2102. 107, 2021.
Luca Scimeca, Seong Joon Oh, Sanghyuk Chun, Michael Poli, Sangdoo Yun. Which shorteut eues will dnns choose? a study from the parameter-space perspective. arXiv preprint arXiv:21 10.03009 2021.
SeiPy. SciPy Documentation.
iereerence/geeerateedee http://ssppp.pppmmmmmmmm...mmm.m 2016.
opt.miitiiiieeie linear sum assignment html,
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33:9573-9585, 2020.
13

---

Under review as a conference paper at ICLR 2023
Berfin Simsek, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. Geometry of the loss landscape in overparameterized neural networks: Symme- tries and invariances. In International Conference Machine Learning, pp. 2021.
PMLR,
Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33:22045-22055, 2020.
Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitriy Pyrkin, Sergei Popov, and Artem Babenko. Editable neural networks. arXiv preprint vrXiv:2004.00345 2020.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent separable data. The Journal of Machine Learning Research, 19(1): 2822-2878 2018.
Hidenori Tanaka, Aran Nayebi, Niru Maheswaranathan, Lane McIntosh, Stephen Baccus, and Surya Ganguli. From deep learning mechanistic understanding in neuroscience: the structure of retinal prediction. Advances in neural information processing systems, 32, 2019.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 33: 18883383-,,,9 2020.
Damien Teney, Maxime Peyrard, and Ehsan Abbasnejad. Predicting is not understanding: Recog- nizing and addressing underspecification machine learning. arXiv preprint arXiv:2207.02598. 2022.
Puja Trivedi, Ekdeep Singh Lubana, Mark Heimann, Danai Koutra, and Jayaraman J Thiagara- jan. Analyzing Data-Centric Properties for Contrastive Learning on Graphs, arXiv preprint 1rXiv:2208.028111 2022.
Guillermo Valle-Perez, Chico Q Camargo, and Ard A Louis. Deep learning generalizes because the parameter-function map biased towards simple functions. arXiv preprint arXiv:1805.0852. 2018.
Sjoerd Van Steenkiste, Francesco Locatello, Jürgen Schmidhuber, and Olivier Bachem. Are disen- tangled representations helpful for abstract visual reasoning? Advances Neural Information Processing Systems, 32, 2019.
Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34: 34:16451-16467 2021.
Zichao Wang, Randall Balestriero, and Richard Baraniuk. max-affine spline perspective of recurrent neural networks. In International Conference Learning Representations, 2018.
Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre Alvise-Rebuffi, Ira Ktena, Taylan Cemgil, et al. A fine-grained analysis on distribution shift. arXiv preprint 1 2021.
Mitchell Wortsman, Maxwell C Horton, Carlos Guestrin, Ali Farhadi, and Mohammad Rastegari. Learning neural network subspaces. In International Conference on Machine Learning, pp. 11217-11227. PMLR, 2021.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pp. 33965-239998. PMLR, 2022a.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust Lne-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7959-7971, 2022b.
14

---

Under review as a conference paper at ICLR 2023
Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise signal: The role of image backgrounds in object recognition. arXiv preprint arXiv:2006.09999, 2020.
Pu Zhao, Pin- Yu Chen, Payel Das, Karthikeyan Natesan Ramamurthy, and Xue Lin. Bridging mode connectivity in loss landscapes and adversarial robustness. arXiv preprint
2020.
15

---

Under review as a conference paper at ICLR 2023
A TRAINING DETAILS AND DATASETS
When training from scratch (e.g., in Fig. 4), train models using SGD for 100 epochs with batch-size of 256. Learinn rate starts at 0.1 and is dropped by a factor of 10 at the 40th and 80th epochs. No data augmentations train models for a further 100 epochs on data without cues using different initial learning rates, but the step-decay schedule (decay factor of 0.1 decay epochs 40 and 80).
used. When fine-tuning assess linear connectivity in Fig. 5,
A.1 DATASET VISUALIZATIONS AND TRAINING CURVES
When using synthetic datasets, if a proportion of samples is to be assigned the cue feature, use the first c% samples of all classes to assign them the respective cues. We do not store the samples beforehand; instead, we use manually designed Py Torch data-loaders that allow for easy manipulation of samples an online manner, enabling straightforward counterfactual evaluations. While the dataset construction was discussed in Sec. 2.2, we provide several visualizations of randomly sampled datapoints from different classes and their counterfactuals in Fig. 7 (CIFAR-10 with box cuc). Fig. 8 (CIFAR-100 with box/color cue), and Fig. 9 (Dominoes: CIFAR-10 10 with concatenated FashionMNIST image cue). Learning curves with train/test accuracies for VGG / ResNet-18 models trained different proportions of samples with cue features for these datasets reported in Figs. 10a and lla (CIFAR-10 with box cue), Fig. 10b, 1lb (CIFAR-100 with box/color cuc), and Fig. 10c, 1lc (Dominoes).
Class
Class
Class
Class
Class
Class
Class
Class
Class
Class
w/ Cue
wo Cue
Rand. Cue
Rand, Img
Figure 7: CIFAR-10 with Box cue.
Class
Class 11
Class 21
Class 31
Class 41
Class 51
Class 61
Class 71
Class 81
Class 91
wib Clab
Rand, Cue
Rand. lmg
Figure 8: CIFAR-100 with Box/Color cue.
16

---

Under review as a conference paper at ICLR 2023
Class
see
wi Cue
w/o Cue
Rand. Cue
Rand. Imn
Figure 9: Dominoes: CIFAR-10 with their corresponding ID image from Fashion-MNIST the cue.
60%
70%
80%
90%
data: 100%
50
100
50
100
50
100
50
100
50
100
50 Epochs
100
Epochs
Epochs
Epochs
Epochs
Epochs
(a) CIFAR- 10 with box cues, wherein the box's 's location a function of the target label.
Train
Testt (wl Cue)
Test twlo Cue]
Train w/o Cue
Cue data: 60%
Cue data: 70%
Cue data: 80%
Cue data: 90%
Cue data: 100%
accursoy
30
10
50
100
50
100
50
100
50
100
100
Epochs
Epochs
Epochs
Epochs
Epachs
Epochs
(b) CIFAR- with box cues, wherein the box's location a function of target label.
Train
Test tw/ Cue)
Test Iwlo Cuc)
Train wio Cue
Cue data: 60%
Cue data: 70%
Cue data: 80%
Cue data: 90%
Cue data: 100%
Accurscy 195)
50
10
Epachs
Epochs
Epochs
Epochs
Epochs
Epochs
(c) Dominoes, wherein Dashion-MNIST images appended CIFAR-10 images and act as the spurious
cues.
Figure 10: Learning curves for VGG-13 models.
17

---

Under review as a conference paper at ICLR 2023
(W) Due)
100
Accurazy
100
50 Epochs
100
50
102
50
100
50
100
Epochs
Epochs
Epochs
Epochs
(a) CIFAR-10 10 with box cues, wherein the hox's 's location a function of the target label.
twio Gue!
Train wio Cue
Cue dieta: 60%
Clus dati: 70%
Cue dieta: 80%
Due data: 90%
Cue data: 100%
Accurscy
10
50
100
50
100
50
100
50
100
50
100
50 Epochs
100
Epochs
Epochs
Epochs
Epochs
Epochs
(b) CIFAR-I0 with box cues, wherein the box's 's location a function of target label.
Train
Test bw/ Due)
Test tw/o Cue)
Train wif Cue
Cue diata: 60%
Cue data: 70%
Cue data: 80%
Due data: 90%
Cue data: 100%
Accuracy
50
10
100
50
100
50
1.00
50
100
Epochs
Epochs
Epochs
Epochs
Epochs
Epochs
(c) Dominoes, wherein Fashion-MNIST images are appended CIFAR-I0 images and act the spurious cues.
Figure 11: Learning curves for ResNet-18 18 models.
A.2 TRAINING DETAILS FOR TAB. 1
We train models using SGD the synthetic data with cue features, reserving 2,500 training samples as "clean" data for further fine-tuning to remove reliance on the spurious cue. Depending on the method, the fine-tuning setup involves different hyperparameters. For consistency, we follow Kirichenko et al. (2022b) and Kumar et al. (2022) in using a cosine schedule for fine-tuning on clean data.
Naïve Tine-Tuning. We use different initial learning rates, including medium (0.01) and small (0.001). For large learning rate, we note that while fine-tuning on minimal set induces good invariance properties, the performance the original, without cue data (called NC in tables) is often rather poor. Hence, we omit those results.
LLRT (Kirichenko al., 2022b). We freeze the model parameters at their current state, remove the final linear layer, and replace it with a randomly initialized one. The layer is fine-tuned on clean data for 100 epochs with cosine decay schedule that starts at LR of 30.
LPFT (Kumar et al., 2022). First, we follow the protocol above for LLRT to produce new linear layer. Thereafter, the entire model fine-tuned on clean data for 20 epochs with initial learning rates of 0.01, 0.001, and 0.0001. The best retrieved results on validation data are reported.
CBFT. We run CBFT for 20 epochs, using an initial learning rate of 0.01 with cosine decay schedule (similar to the baselines). The method turns out to be fairly robust to the exact values of X and A2; hence we fix both to We also note that since training will yield gradients for the model that has parameters 7e- (t), we need explicitly compute the gradients for by using the following relationship for some general objective function
VaL(10-1e (t)) = (V) V T C(o0 (t)) (1 VV Tu- (10000
Thus, one need only compute gradient of an objective with respect to 70-10 (t) and multiply that by factor of 1 to retrieve the gradient of the objective with respect to B. This step has to be carried out explicitly and hence we have to break the optimization process of CBFT into two steps (see Eq. 3), executing alternating minimization for the barrier and invariance losses
18

---

Under review as a conference paper at ICLR 2023
B IDENTIFYING CONNECTIVITY PATHS
B.1 QUADRATIC PATHS
The qudaratic path is defined as follows.
70, 10 (1) f20 26(1 1)012 (1 1)2
The set of parameters 012 can be thought of as the vertex of parabola that helps anchor the curve. To identify this set of parameters, we follow Garipov et (2018) and train points uniformly sampled from the quadratic path to achieve zero loss on a given dataset DD i.e.,
012 = argmin Bec (1:
(5)
Consequently, note that a quadratic path necessarily depends on the dataset used for its identification and it is not mandatory that it generalize across datassess/isstibuttinss This is precisely what we see our results in Fig. 4, where we are able to identify quadratic mode connectivty between two sets of parameters on a given dataset, but those paths do not generalize to counterfactual datasets.
B.2 FINDING PERMUTATIONS FOR LINEAR CONNECTIVITY
Given two minimizers 0, 22 identifying the linear path between them involves merely interpolating the parameters. Entezari al. (2021); 1); Ainsworth et al. (2022); Singh & Jaggi (2020) hypothesize that minimizers discovered using SGD can always be linearly mode connected up to permutations of neurons that align the two models in their activations or weights. That is, there generally exists permutation ı that connects (01) with 0g in the sense of Def. 1. To empirically analyze this claim in our work, we identify æ by maximizing the similarity of activations produced by model with parameters 0, and 02:
frggmin))((;:7(0) (z; 02)
(6)
Given that solving the problem above is NP-Hard (Entezari et al., 2021; Ainsworth et al., 2022: Singh Jaggi, 2020), we propose solve it greedily by computing representations at each layer of the two models, finding permutation that matches the representations maximally, and then repeating the process for the next layer. To this end, we use inputs with batch-size of 512 and run the matching process over the entire original datasets (i.e., ones without cues). We note that did conduct minimal experiments on finding permutations using data with cues, instead of without, but never found any noticeable differences in the results. Hence, we decided to the original data without cues throughout our experiments for finding linear paths. Intuitively, we suspect the exact choice of dataset does not matter for our experimental setup because analyze pairs of models which include one model that is invariant to the cue and one that is not. Since the invariant models produce the same representations data with / without cues, the target for permutation matching remains the same.
For finding the permutation, we use SciPy's linear assignment solver (SciPy, 2016). Specifically, Tet denotes permutations before layer we use the following algorithm. Initialize: 0,, 02, dataset D. model f, m. = I, where identity permutation.
1 [--1 1 L + # of Layers for D do {-]
for | L do
क = argminn ( f((;; 0x)1
b fi denotes representation at layer /
+- 1+1
end for end for
19

---

Under review as a conference paper at ICLR 2023
Table Ablating CBFT. We train ResNet-18 models on our synthetic CIFAR-10, CIFAR-100, and Dominoes dataset with different proportions of samples with cue features and fine-tune them using 2500 "clean" samples from a dataset without any cues. Test accuracies (%) on counterfactual test datasets with No Cue (NC), with Cue (C), Randomized Cue (RC), and Randomized Image (RI) are reported. We compare Connectivity-Batt
Tuning (CBFT) with two of its ablations (see App. C.1): (i) -baarierr for which the barrier inducing loss is removed from the training process and (ii) - -inv. for which the invariance loss is removed. ~ denotes invariance is desirable, i.c., accuracy should be similar to that NC; NY indicate higher/lower accuracy is desirable; best results are in bold. For discussion, please see App. C.I.
60% Cue data
70% Cuo data
BUSS Cue data
90% Cue data
C-10
NCt
C
RC
RI+
NC
C
RC
RT+
NCt
C*
RC
RI-
NCT
CM
RC
R1+
CBFT
74.1
71.5
73.4
8.75
73.2
69.2
72.1
8.60
70.0
70.0
69.5
9.68
67.9
72.5
68.1
13.1
Craniar
75.8
93
69.3
24.4
75.9
90
72.1
18.6
71.6
89.9
66.3
23.5
67.8
89.6
65.1
20.5
-im
73.4
69.4
68.8
72.9
65.2
71.3
68.1
9.72
65.8
10.3
C-100
Nct
CM
RC
RI
NCC
C
RC
RI
NCT
C*
RC
RI
NCT
CM
RC
RI
CBFT
42.7
65.0
36.4
14.6
38.5
66.7
34.7
21.2
34.6
69.3
23.0
27.9
28.5
72.9
23.2
46.0
Laammerr
44.7
99.8
17.5
81.6
40,2
99.9
13.7
88.9
34.6
99.9
11.3
95.1
26.5
99.1
13.5
82.2
Cin
43.2
59.4
36.5
12.5
35.7
64.2
26
25.5
34.1
70.2
23.5
36.7
24.7
69.2
15.9
45.6
Dom.
NCt
CM
RC
RI
NCC
C
RC
RI
NCT
C7
RC
RI
NCT
CM
RC
R1
CBFT
72.0
64.9
67,5
9.9
71.5
70.0
59.2
12.1
70.8
69.7
65.9
11.9
67.2
68.7
61.5
14.9
Laaanirr
94.9
63.2
77.4
94.2
65.8
29.2
93.3
63.5
30.1
67.1
91.9
55.5
32.9
74.2
40.4
41.8
6.93
74.6
28.2
24.9
10.6
71.3
20.1
22.2
6.92
66
21.2
20.9
6.26
C ADDITIONAL RESULTS ON CBFT
C.I ABLATIONS
To analyze the role played by the two loss functions involved in the alternating minimization steps of Connectivity-Bant -Based Fine-Tuning (CBFT) (see Sec. 5, Eq. 3), we present an ablation study as follows. We analyze two variants of CBFT: (i) -harrier. for which the barrier inducing loss (1...,uca
Aλ (D 70 (t)), v) has been removed from the training pro-
cess, and (ii) Lm.., for which the invariance loss Σx EeED: (J+ (2; 0)) E (T; (T): 0)])
has been removed. Results are shown in Tab. 3. We find that without the barrier loss, the trained model is unable to break its reliance on spurious cues, even though it generally achieves the best performance on data without cues (NC in table). Meanwhile, without the invariance loss, the trained model indeed loses sensitivity to spurious cues and shows poor performance when the underlying image is randomized, as we desire. However, in few instances the model can become anti-correlated with the spurious cue (c.g.. see results on Dominoes). This is expected since the barrier loss's goal is to move the model to a region in the landscape that follows different mechanisms (with respect to the pre-trained model) by inducing a loss barrier; without the invariance loss, the model can learn to induce this barrier by merely becoming anti-correlated with the spurious cue. The invariance loss helps prevent this pitfall, selecting a mechanistically dissimilar region in the landscape that is uncorrelated, instead of anti-correlated with the spurious cue. These results confirm the validity of our claims in Sec. 5: preventing linear connectivity helps induce mechanistic dissimilarity and an invariance penalty helps select the exact mechanisms ablation study help us infer that while the two losses involved in CBFT have their individual benefits, it is only when they are combined that they give the best results.
want the models to differ in. Overall, this
C.2 COMPARISON WITH TRAINING FROM SCRATCH
We compare CBFT against training from scratch on the minimal clean dataset that we assume access to during the training process for all baselines and CBFT in Tab. 1. Specifically, we train ResNet- 18 models for 100 epochs using an initial learning rate of 0.1 and a cosine decay schedule. Results are provided in Tab. 4 and we see training from scratch significantly underperforms all baselines and CBFT (results reported in Tab. 1). This is expected since our setup assumes access to only a minimal clean dataset for inducing invariance to spurious attributes. Since training from scratch is not a sample efficient strategy, it cannot perform well in this setting. We also highlight that using as initialization a model pretrained unclean dataset, i.e., one that contains spurious attributes, will
20

---

Under review as a conference paper at ICLR 2023
Table Training from scratch on minimal clean data. We train ResNet-18 models on the 2500 "clean" samples used in Tab. from the original CIFAR-10, CIFAR-100, and Dominoes datasets. Test accuracies (%) on counterfactual test datasets with No Cue (NC), with Cue (C), Randomized Cue (RC), and Randomized Image (RT) are reported. denotes invariance is desirable, i.e., accuracy should be similar to that on NC: T/4 indicate higher/lower accuracy desirable. We clearly see training from scratch severely underperforms all other baselines and CBFT (see Tab. 1). This is expected since our setup assumes access to only minimal clean dataset for inducing invariance to spurious attributes. Since training from scratch is not sample efficient strategy, it cannot perform well in this setting. Note that models trained on Dominoes show reduced performance in the presence of cues because
MNNST images cues in that dataset. The shift in distribution is rather non-trivial and seems to sufficiently confuse the model to reduce its performance.
NC
C
RC
RT
C-10
47.5
47.4
47.5
9.69
C-100
16.5
16.4
16.4
1.19
Dom.
48.5
31
31
10.8
make this overall process equal to naïve fine-tuning on the clean dataset; we already provide results for na've fine-tuning in Tab. 1.
D FURTHER RESULTS SHOWING NON-LINEAR MODE CONNECTIVITY OF MECHANISTICALLY DISSIMILAR MINIMIZERS
synthetic CIFAR-10 / CIFAR- 100 Dominoes datasets with cues (see Figs. 7, 8, and 9) and the original datasets themselves. Parameters of the corresponding models denoted 0c and INc. We identify connectivity paths along pairs of parameters, specifically evaluating quadratic paths identified using data without cues (denoted Quadratic w/o Cues), quadratic path identified using data with cue (denoted Quadratic w/ Cue), linear path (denoted Linear), and linear path after permuting 0c to maximally match ONe's 's activations (denoted Linear Permuted). In the following, plot titles denote evaluation dataset, including datasets where either the cue is present (denoted w/ Cue), absent (denoted w/o Cue), randomized (denoted Rand. Cue), the underlying image is randomized but the cue remains the same (denoted Rand. Image). Line colors denote the proportion of dataset that has synthetic cues.
We train VGG-13 and ResNet-18 18 models
Across all our results, we see the set of parameters I yields the same performance upon ran- domization of the cue, while the performance of fc degrades substantially-iee. the two modes are mechanistically dissimilar due to lack of shared invariances (see Def. 4). Nonetheless, we can identify quadratic (but not linear) paths that mode-connect these mechanistically dissimilar minimizers, hence corroborating Prop. across several datasets and model architectures, showing mechanistically dissimilar modes can also be mode connected via relatively simple paths well. However, different points on the connectivity paths respond differently to counterfactuals, indicating lack of mechanistic connectivity.
21

---

Under review as a conference paper at ICLR 2023
70% Rand. Image
w/ Cue
W/ Oue
Rand, Cue
W Cue
w/o Cue
Rand, Cue
Rand, Image
50
10
Quadratic (wio Cue)
Quadratic (w/ Cue)
10
0.3
6ac
Oue
dx
Ou
Opr
Anc 9e
Linear
Unear (Permuted)
(a) Test Accuracy.
60%
70%
80%
90%
100%
w/ Cue
Rand,
Rand.
Randi Cue
Rand.
100
0.3
0.7
Rar
0.3
0.7
9ur
0.3
0.7 Aar
0.3
0.7
0g
0.7
0.7
Our
R-
0.3
0.7
Agr
0.1
0.7
far Rc
03
07 Ar.
(wio Cue)
Quadrati (w/ Cue)
1.00
Train
50
0.3
0.7
03
0.7
Sxr
0.3
0.7
8x
03
00
0.3
8x
0.3
0.7
0.3
0.7
03
0.7
Linear
Uneer (Permuted)
(b) Train Accuracy.
60% Randi Cue
70%
80%
90%
100% w/a Cue
20
w/ Gue
Wio Cue
Rand Image
w/ Cue
Rand, Due
Rand Image
2.0
0.3
0.7
8xc
0.3
0.7
Auc
8:
03
0.7
8:
0.3
0.7
0.3
0.7
8vc
0.3
0.7
81
033
0.7
Acc
8:
03
07
Quadratic (wlo Cue)
Quadratic (w/ Cue)
20
2.0
0.3
0.7
60r
0.7
0.7
Duc
0.7
0.7
Ne 80
0.3
0.7
0.3
0.7
33
0.7
0.7
077
8
63
n7
Ar.
Linear
Linear (Permuted)
(c) Test Loss.
60% Rand. Cue
70%
80%
90%
100%
Cue
Rand Image
Cue
Rand.
Rand
200
0.3
0.7
0.3
0.7
0.3
0.7
0.3
0.7
0.3
0.7
Ruc
033
0.
(a) Quadratic (w/o Cue)
(b) Quadratic (w/ Cue)
20
2.0
0.7
0.3
0.7
0.3
0.7
03
03
n7
(c) Lincar
(d) Linear (Permuted)
(d) Train Loss.
Figure 12: VGG-13 CIFAR-10 with Box Cue. We plot test/train accuracy/loss curves along different connectivity paths and see thorough corroboration of our claims in the main text: Mecha- nistically dissimilar minimizers be connected via nonlinear paths given dataset, but behave different on counterfactuals, indicating lack of mechanistic connectivity.
22

---

Under review as a conference paper at ICLR 2023
70% Rand. Image.
w/ Cue
W/ Oue
Rand, Cue
W Cue
w/o Cue
Rand, Cue
Rand, Image
50
10
Quadratic (wio Cue)
Quadratic (w/ Cue)
10
0.3
6ac
Oue
dx
Ou
Opr
Anc 9e
Linear
Unear (Permuted)
(a) Test Accuracy.
60%
70%
80%
90%
100%
w/ Cue
Rand,
Rand.
Randi Cue
Rand.
100
0.3
0.7
Rar
0.3
0.7
9ur
0.3
0.7 Aar
0.3
0.7
0g
0.7
0.7
Our
R-
0.3
0.7
Agr
0.1
0.7
far Rc
03
07 Ar.
(wio
Quadrati (w/ Cue)
1.00
Train
50
0.3
0.7
03
0.7 Scr
0.3
0.7
8x
03
00
0.3
8x
0.3
0.7
0.3
0.7
03
0.7
Linear
Uneer (Permuted)
(b) Train Accuracy.
60% Randi Que
70%
80%
90%
100% w/a Cue
20
w/ Gue
Wio Cue
Rand Image
w/ Cue
Rand, Due
Rand Image
2.0
0.3
0.7
8xc
0.3
0.7
Auc
8:
03
0.7
8:
0.3
0.7
0.3
0.7
8vc
0.3
0.7
033
0.7
Acc
8:
03
07
Quadratic (w/o Cue)
Quadratic tw/ Cue)
20
Test
2.0
0.3
0.7
0.7
0.7
0uc
0.7
0.7
Ne
0.3
0.7
0.3
0.7
Rr
33
0.7
0.7
0.7
6:
6.1
n7
Ar.
Linear
Linear (Permuted)
(c) Test Loss.
60%
70%
80%
90%
100%
Cue
Rand. Cue
Rand
Cue
Rand.
Rand
200
0.3
0.7
0cc
0.3
0.7
0.3
0.7
0.3
0.7
0.3
0.7
R0c
0.3
0.7
Quadratic (w/o Cue)
Quadratic (w/ Cue)
20
2.0
0.7
0.3
0.7
0.3
0.7
0r0 R
07
Limear
Linear (Permuted)
(d) Train Loss.
Figure 13: ResNet-18 on CIFAR-10 with Box Cue. We plot test/train accuracy/loss curves along different connectivity paths and see thorough corroboration of our claims in the main text: Mecha- nistically dissimilar minimizers be connected via nonlinear paths given dataset, but behave different on counterfactuals, indicating lack of mechanistic connectivity.
23

---

Under review as a conference paper at ICLR 2023
70% Rand. Image
w/ Cue
W/ Que
Rand, Cue
w Cue
wio Cue
Rand, Cue
Rand. Image
50
10
(a) Qua dratic (wto Cue)
(b) Quadratic (w/ Cue)
10
0.3
0.7
6ac
Oue Rc
Our
Ope do
(c) Linear
Linear (Pormuted)
(a) Test Accuracy.
60%
70%
80%
90%
100%
Cue
Rand,
Rand.
Randi Cue
Rand.
100
0.3
0.7 Rar
033
0.7
9ur
03
0.7 Aar
0.3
0.7
0r
0.7
0.7
Aur
R-
0.3
0.7
Rgr 62
0.7
0.7
far Rc
03
0.7 Ar.
Quadratic (wto
(w/ Cuc)
1.00
50
0.3
0.7
03
0.7
Sxc
0.3
0.7
8x
03
0.3
8x
0.3
0.7
0.3
0.7
03
0.7
(c) Linear
(d) Linear (Permuted)
(b) Train Accuracy.
60%
70%
90%
100% w/o Cue
w/ Cue
wa Cue
Rand. Cue
Rand Image
w/ Cue
Rand. Cue
Rand Image
(a) Quadratic (w/e Cue)
(b) Quadratic (w/ Cue)
20
10
0.3
0.7
81
0.3
0.7
Bvc 8:
0.3
0.7
8xc
8:
0.3
0.7
8/
03
0.7
8/C 82
0.3
Avc
0.7 Acc
(c) Linear
(c) (Permuted)
(c) Test Loss.
60%
70%
90%
100% wo
w/o
Image
Image
1.2
(a) Quadratic (w/o Cue)
(b) Quadratic (w/ Cue)
25
Train
1.2
0.7
0.3
0.7
0x
0.3
0.7
Rx
0x R:
(c) Linear
(d) Linear (Permuted)
(d) Train Loss.
Figure 14: VGG-13 on CIFAR-100 with Box/Color Cue. We plot test/train accuracy/loss curves along different connectivity paths and thorough corroboration of claims in the main text: Mechanistically dissimilar minimizers can be connected via nonlinear paths on a given dataset, but behave different counterfactuals, indicating lack of mechanistic connectivity.
24

---

Under review as a conference paper at ICLR 2023
70% Rand. Image
w/ Cue
W/ Que
Rand, Cue
w Cue
wio Cue
Rand. Cue
Rand, Image
50
10
(a) Qua dradiciic (wto Cue)
(b) Quadratic (w/ Cue)
10
0.3
0.7
6ac
Oue Re
Ou
de
Ope do
Anc de
(c) Linear
Linear (Pormuted)
(a) Test Accuracy.
60%
70%
80%
90%
100%
Cue
Rand,
Rand.
Randi Cue
Rand.
100
0.3
0.7 Rar
033
0.7
9ur
03
0.7 Aar
0.3
0.7
0g
0.7
0.7
Aur
R-
0.3
0.7
Rgr
6:
0.1
0.7
for Rc
03
07 Ar.
Quadratic (wto
Quadratic (w/ Cuc)
1.00
50
0.3
0.7
03
0.7
Suc
0.3
0.7
8x
0.
0.3
8xc
0.3
0.7
0.3
0.7
03
0.7
(c) Linear
(d) Linear (Permuted)
(b) Train Accuracy.
60%
70%
90%
100% w/o Cue
w/ Cue
wa Cue
Rand. Cue
Rand Image
w/ Cue
Rand. Cue
Rand Image
(a) Quadratic (w/e Cue)
(b) Quadratic (w/ Cue)
20
10
0.3
0.7
81
03
Auc 8:
0.7
Buc
8:
0.3
0.7
8xc
8:
0.3
0.7
8x
03
0.7
8/C
0.3
Avc
0.7 Acc
(c) Linear
(c) (Permuted)
(c) Test Loss.
60%
70%
90%
100% wo
w/o
Image
Image
1.2
(a) Quadratic (w/o Cue)
(b) Quadratic (w/ Cue)
25
Train
1.2
00
0.7
0.3
0.7
0.3
0.7
Rx
(c) Linear
(d) Linear (Permuted)
(d) Train Loss.
Figure 15: ResNet-18 on CIFAR-100 with Box/Color Cue. We plot test/train accuracy/loss curves along different connectivity paths and thorough corroboration of claims in the main text: Mechanistically dissimilar minimizers can be connected via nonlinear paths on a given dataset, but behave different counterfactuals, indicating lack of mechanistic connectivity.
25

---

Under review as a conference paper at ICLR 2023
70% Rand. Image
w/ Cue
WD Oue
Rand, Cue
W Cue
wio Cue
Rand, Cue
Rand. Image
50
10
(a) Qua dratic (wto Cue)
(b) Quadratic (w/ Cue)
10
0.3
0.7
dac
Oue
de
Our de
Ope do
(c) Linear
(d) Linear (Pormuted)
(a) Test Accuracy.
60%
70%
80%
90%
100%S
Cue
Rand,
Rand.
Randi Cue
Rand,
100
0.3
0.7
Rar
033
0.7
9ur
03
0.7
Aar
0.3
0.7
0g
0.1
0.7
Our
0.3
0.7
Rar
0.1
0.7
far Rc
03
0.7 Ar.
Quadratic (wio
Quadratic (w/ Cuc)
1.00
50
0.3
0.7
03
0.7
0.3
0.7
8x
0.3
0.3
8xc
0.3
0.7
0.3
0.7
03
0.7
(c) Linear
(d) Linear (Permuted)
(b) Train Accuracy.
60%
70%
90%
100% w/o Cue
w/ Cue
wa Cue
Rand. Cue
Rand Image
w/ Cue
Rand, Cue
Rand Image
6cr
(a) Quadratic (w/o Cue)
(b) Quadratic (w/ Cue)
Test LO52
0.3
0.7
81C
0.3
Avc 9:
0.7
Bvc
8:
0.3
0.7
8xc
8:
0.3
0.7
8/
03
0.7
8/C
0.3
Avc
0.7 Acc
(c) Linear
(c) Linear (Permuted)
(c) Test Loss.
60%
70%
80%
90%
100%
Cue
wio Cue
Rand. Cue
Rand Image
w/ Cue
Rand,
Rand
0.3
0.7
8x
03
0.7
Auc 8:
03
0.7
8:
0.3
0.7
0.3
0.7
8vc
0.3
0.7
0.3
0.7
6:
03
07
&c.
(a) Quadratic (w/e Cuc)
(b) Quadratic (w/ Cue)
Train Look
0.7
0.3
0.7
0.3
0.7
07
(0) Linear
(6) Linear (Permuted)
(d) Train Loss.
Figure 16: VGG-13 on Dominoes. We plot test/train accuracy/loss curves along different connectivity paths and thorough corroboration of claims in the minimizers can be connected via nonlinear paths on given dataset, but behave different on counter- factuals, indicating lack of mechanistic connectivity.
main text: Mechanistically dissimilar
26

---

Under review as a conference paper at ICLR 2023
70% Rand. Image
w/ Cue
WD Oue
Rand, Cue
W Cue
wio Cue
Rand, Cue
Rand. Image
50
10
(a) Qua dratic (wto Cue)
(b) Quadratic (w/ Cue)
10
0.3
0.7
6ac
Oue Re
de
Ou de
Opr do
Gec Pe
(c) Linear
Linear (Pormuted)
(a) Test Accuracy.
60%
70%
80%
90%
100%
Cue
Rand,
Rand.
Randi Cue
Rand,
100
0.3
0.7
Rar
033
0.7
9ur
03
0.7
Aar
0.3
0.7
0g
0.1
0.7
Our
0.3
0.7
Rgr
0.1
0.7
for Rc
n3
07 Ar.
Quadratic (wto
Quadratic (w/ Cuc)
1.00
50
0.3
0.7
03
0.7
Sic
0.3
0.7
8x
03
00
0.3
8xc
0.3
0.7
0.3
0.7
03
0.7
(c) Linear
(d) Linear (Permuted)
(b) Train Accuracy.
60% Rand. Cue
70%
90%
100% w/o Cue
w/ Cue
wa Cue
Rand Image
w/ Cue
Rand. Cue
Rand Image
(a) Quadratic (w/e Cue)
(b) Quadratic (w/ Cue)
20
10
0.3
0.7
8xc
0.3
Axc 8:
0.7
Bvc 8:
0.3
0.7
8xc
8:
0.3
0.7
8/
03
0.7
81C
0.3
Avc
0.7 Acc
(c) Linear
(c) (Permuted)
(c) Test Loss.
60%
70%
90%
100% wo
w/o
Image
Image
Loss
(a) Quadratic (w/o Cue)
(b) Quadratic (w/ Cue)
20
1.0
00 8
0.7
0.3
0.7
Dxc
0.3
0.7
Rxe
(c)
(c) (Permuted)
(d) Train Loss.
Figure 17: ResNet-18 on Dominoes. We plot test/train accuracy/loss curves along different con- nectivity paths and see thorough corroboration of our claims in the main text: Mechanistically dissimilar minimizers can be connected nonlinear paths on given dataset, but behave different counterfactuals, indicating lack of mechanistic connectivity.
27

---

Under review as a conference paper at ICLR 2023
E FURTHER RESULTS SHOWING LACK OF LINEAR CONNECTIVITY IMPLIES MECHANISTIC DISSIMILARITY
We train VGG-13 and ResNet-18 models synthetic CIFAR-10 / CIFAR-100) Dominoes datasets with cues (see Figs. 7, 8, and 9). Corresponding models are denoted fc- These models are then fine-tuned on the original CIFAR-IO CIFAR-I00 datasets that do not have any cue features. Specifically, we use different learning rates (LR) and train for 100 epochs with step-decay schedule (decay epoch 40 and 80 by factor of 0.1). Corresponding models are denoted OPr- In the following, plot titles denote evaluation dataset, including datasets where either the cue is present (denoted Cue), absent (denoted w/o Cue), randomized (denoted Rand. Cue), or the underlying image is randomized but the cue remains the same (denoted Rand. Image). Line colors denote the proportion of dataset that has contains our synthetically embedded cues.
Across all our results, we see that using large enough learning rate or enforcing perfect correlation between the cue and label induces loss barriers along the linear path, i.e., linear mode connectivity does not hold. Correspondingly, we see the models respond differently counterfactuals, i.e, they are mechanistically dissimilar and not connected. For a small enough learning rate, 8yr remains mechanistically similar to Ic, responding similarly counterfactuals. Correspondingly, see linear mode connectivity holds between the models on data with cues.
60%
70%
80%
60%
100%
60%
70%
80%
90%
100%
w/ Cue
wil Cue
Rand, Cue
Rand. Image
w/ Cue
wib Cae
Rand. Cue
Rand Image
100
50
20
10
0.7
0.7
0.7
0.7
23
0.7
0.3
0.7
0.3
0.7
03
Small (0.001)
Test Acc. (8))
Small LR (0.001)
100
8:0
Text
0.7
0.7
0.7
0.3
0.7
53
0.7
0.
07
dr
03
Ar
Medium
Medium
100
50
50
10
0.7
0.
0.3
0.7
07
3.3
0.7
0.3
0.7
031
077
03
Large (0.1)
LR (0.1)
(a) VGG.
(b) ResNet18.
Figure 18: Fine-tuning models trained on CIFAR-10 with Box Cue. We plot test accuracy curves along the linear path between Oc and OhT and see thorough corroboration of our claims in the main text: Linearly connected minimizers exhibit mechanistic similarity. behaving identically on counterfactual datasets, indicating mechanistic connectivity.
60%
70% win Cue
50%
100% Rand, Image
60%
70% Wit Cae
80%
90%
100% Rand Image
w/ Cuu
Rand, Eue
Cue
Rand.
100
077
07
5m
8m
dr 6m
Small LR (0.001)
Test Ace (16)
Small LR 0011
20
0.7
0.7
0.7
0.7
2.5
0.7
0.3
0.7
03
0.7
03
Medium (0.0 011
Medium (0.01)
100
100
50
0.3
Large (0.1)
(a) VGG.
(b) ResNet18.
Figure 19: Fine-tuning of models trained on CIFAR-100 with Box/Color Cue. accuracy along the linear path between Ic and lrr and see thorough corroboration of our claims in the main text: Linearly connected minimizers exhibit mechanistic similarity, behaving identically on counterfactual datasets, indicating mechanistic connectivity.
We plot test
28

---

Under review as a conference paper at ICLR 2023
60%
70%
50%
90%
100%
60%
70%
50%
90%
100% Rand Image
107 Cuu
wib Cue
Rand, Lue
Hand. Image
150
M/ Cue
wib Cum
Hardl Cue
0.3
0.7
5.3
0.7
Fr
0.3
0.7
0.3
0.7
13
0.7
fr
0.3
0.7
0.3
07
An
R:
03
03
47
Small UR (0.001)
Test Acc. 196)
Small UR
0011
Text
0.7
0.7
0.7
0.7
5.3
0.7
0.3
Modium LR (0.01)
Medium LR 10.01)
200
120
60
50-
0.7
0.7
8n
40 4
03
4r
Large LR (0.1)
Large LR (0.1)
(a) VGG.
(b) ResNet18.
Figure 20: Fine-tuning of models trained on Dominoes. We plot test accuracy along the linear path between 9e and fer and see thorough corroboration our claims in the main text: Linearly connected minimizers exhibit mechanistic similarity, behaving identically counterfactual datasets, indicating mechanistic connectivity.
70%
Rand.
100
Train (96)
(0.001)
Train ACc. 1950
Small
100
320
0.7
0.7
07
de
dr
03
6r
Medium
Medium LR (0.01)
10
0.7
0.7
0.7
0.7
23
0.7
0.5
0.7
03
07
05
(0.11
(0.1)
(a) VGG.
(b) ResNet18.
Figure 21: Fine-tuning of models trained on CIFAR-I0 with Cue. We plot train accuracy curves along the linear path between Oc and fFr and see thorough corroboration of our claims in the main text: Linearly connected minimizers exhibit mechanistic similarity, behaving identically on counterfactual datasets, indicating mechanistic connectivity.
60%
70% wil Cue
50% Rand, Cue
90%
100% Rand. Image
60% w/ Cue
70% wib Cuc
80% Rand. Cac
9054
100% Rand Image
Wi Cuc
120
0.5
0.7
0.7
By
9:
0.5
0.7
0.3
0.7
2.5
0.7
0.3
0.7
8x
0.3
077
4a
05
dr
Small LR (0.001)
Train Acc. 173
Small LR 0011
10
0.7
077
077
07
233
0.3
077
03
07
03
Medium UR (0.01)
Medium LR (0.01)
100
50
20
0.7
0.3
Large (0.11
Large (0.1)
VGG.
(b) ResNet18.
Figure 22: Fine-tuning of models trained on CIFAR-100 with Box/Color Cue. We plot train accuracy curves along the linear path between Oc and OFT and see thorough corroboration of our claims in the main text: Linearly connected minimizers exhibit mechanistic similarity, behaving identically on counterfactual datasets, indicating mechanistic connectivity.
29

---

Under review as a conference paper at ICLR 2023
60%
70%
50%
90%
100%
60%
70%
80%
90%
100% Rand Image
4/7 Cue
wib Cue
Rand, Lue
Hand. Image
150
M/ Eue
wib Cum
Hardl Cue
0.3
0.7
5.3
0.7
Fr
0.3
0.7
0.3
0.7
13
0.7
fr
0.3
0.7
Ac
0.3
07
An
R:
03
03
47
Small UR (0.001)
Train Acc (55)
Small UR
0011
0.7
0.7
0.7
0.7
5.3
0.3
Modium LR (0.01)
Medium LR 10.11)
200
120
60
50-
0.7
0.7
Or
03
8n
40
03 4r
Large LR (0.1)
Large (0.1)
(a) VGG.
(b) ResNet18.
Figure 23: Fine-tuning of models trained Dominoes. We plot test accuracy curves along the linear path between 9c and OFr and see thorough corroboration of our claims in the main text: Linearly connected minimizers exhibit mechanistic similarity, behaving identically counterfactual datasets, indicating mechanistic connectivity.
70%
Hand.
0.7
0.7
(0.001)
Test Loss
12
07
0.7
07
07
5m
03
6r
Medium
Medium LR (0.01)
0.7
0.7
0.7 055
0.7
23
0.7
0.5
0.7
0.5
0.7
05
(a) VGG.
(b) ResNet18.
Figure 24: Fine-tuning of models trained on CIFAR-10 with Box Cue. We plot test loss curves along the linear path between Oc and Orr and see thorough corroboration of our claims in the main text: Linearly connected minimizers exhibit mechanistic similarity, behaving identically on counterfactual datasets, indicating mechanistic connectivity.
60%
70% wit Cue
50%
90%
100% Band. Image
60% M/ Cule
70% wibh CUe
80% Band. Cue
90%
100% Rend Imsge
M/ Cus
Rand, Eue
0.7
0.7
9.3
0.7
077
0.3
0.7
0.9
0.7
dc 0.3
077 An dc 03
Small (0.001)
Small LR (0.001)
Tost LoGs
Text
0.7
Medium LR 10.01)
Medium LR 10.00
12
0.7
07
077
07
6r
6m
03
4r
Large LR (0.11
Large UR (0.1)
(a) VGG.
(b) ResNet18.
Figure 25: Fine-tuning of models trained on CIFAR-100 with Box Color Cue. We plot loss curves along the linear path between 9e and 0Fr and see thorough corroboration of our claims in the main text: Linearly connected minimizers exhibit mechanistic similarity. behaving identically on counterfactual datasets, indicating mechanistic connectivity.
30

---

Under review as a conference paper at ICLR 2023
60%
70%
50%
90%
100% Rand. Image
60%
70%
80%
90%
100%
V// Clu
wib Cue
Rand, Lue
Cue
Hand. Cue
Rand
20
0.3
0.7
9:
5.3
0.7
0.3
0.7
0.3
0.7
23
0.7
0.3
0.7
0.3
077
(0.001)
Small LR (0.001)
20
Test Lozs
Text
0.7
0.7
0.7
0.7
Mediam (0.01)
10.01)
07
82
07
13
0.7
0.3
0.7
033
07
4r
03
03 4r
Large LR [0.1)
Large LR (0.1)
(a) VGG.
(b) ResNet18.
Figure 26: Fine-tuning of models trained Dominoes. We plot test loss curves along the linear path between Oc and OFT and see thorough corroboration of our claims in the main text: Linearly connected minimizers exhibit mechanistic similarity, behaving identically counterfactual datasets, indicating mechanistic connectivity.
60%
100% Image
Image
077 Small (0.001)
0.3
Small LR 0011
Train LOSS
12.
0.5
0.7
5
0.
0.7
0.5
0.7
0.3
0.7
25
0.7
0.5
0.7
4.
0.5
4.
03
07
41
25
0.7
07
07
07
33
0.
03
0.7
03
Large
Large
(a) VGG.
(b) ResNet18.
Figure 27: Fine-tuning of models trained on CIFAR-10 with Box Cue. We plot train loss curves along the linear path between Oc and OeT and see thorough corroboration of our claims in the main text: Linearly connected minimizers exhibit mechanistic similarity, behaving identically on counterfactual datasets, indicating mechanistic connectivity.
80%
70%
100%
60%
70% Wi Cut
80% Rand. Cut
90%
100% Rand. Image
25
Wi Cue
W/l Cue
Rand. Cue
Rond. Image
25
w/ Cue
0.7
0.7
0.7
07
23
0.7
0.3
0.7
03
0.7
03
Small (0.001)
LR
Train LoSS
Medium LR (0.01)
Medium LR 10.01)
12
0.3
0.7
0.
9:
077
0.3
07
23
0.
5r
03
07
03
07
03
dr
Large 00.11
Large
(0.1)
(a) VGG.
(b) ResNet18.
Figure 28: Fine-tuning of models trained on CIFAR-100 with Box Color Cue. We plot train loss curves along the linear path between Oc and OFT and see thorough corroboration of our claims in main text: Linearly connected minimizers exhibit mechanistic similarity, behaving identically on counterfactual datasets, indicating mechanistic connectivity.
31

---

Under review as a conference paper at ICLR 2023
60%
70%
50%
90%
100% Rand. Image
60%
70%
80%
90%
100% Rand Image
V// Clu
wib Cue
Rand, Lue
Cue
Hand. Cue
10
0.3
0.7
9:
5.3
0.7
0.3
0.7
0.3
0.7
23
0.7
0.3
0.7
0.3
0.7
03
(0.001)
Small LR (0.001)
20
LOGS
Train
0.7
0.7
0.7
0.7
Mediam (0.01)
10.01)
07
82
07
13
0.7
0.3
0.7
0.3
07
4r
03
03
4r
Large LR [0.1)
Large LR (0.1)
(a) VGG.
(b) ResNet18.
Figure 29: Fine-tuning of models trained on Dominoes. We plot train loss curves along the linear path between Oc and OPT and see thorough corroboration of our claims in the main text: Linearly connected minimizers exhibit mechanistic similarity, behaving identically counterfactual datasets, indicating mechanistic connectivity.
F DEFERRED PROOFS
F.1 EXHAASTIVENESS: OF UNIT INTERVENTIONS
Proposition 1. (Exhaustiveness of Unit Interventions.) If (=; is invariant to unil interventions Ai and Aj. it must be invariant their composition; conversely, lack of invariance either A, or Aj precludes invariance to their composition.
Proof. Assume the set of parameters induces a model that exhibits invariance to the intervention A₁, Independently, consider another intervention A,. Then, f(E(x; {A; A,));
f = o A; o Aj = f(E(;; where exhibits invariance to Aj
(x)) 8) = f(Gx A; ( (z( Aj)); 0) fE A;): last equality happens due as well, we have (x;; (A,,A,)) 0) (((=;;A,); = f f(x; 0), i.e., the model induced by 0 is invariant to the simultancous operation (i.c., composition) of A, and A, Meanwhile, if invariant A, but not to Ay. we have fE((;; A,)): ((; + fx; (x;0), i... A induces a model that lack invariance to the simultaneous operation (i.e., composition) of A; and Aj.
assumed invariance of A, Now,
Note that the derivation above did not rely the fact that the interventions "unit" in the sense that they act on independent dimensions. However, if one considers general interventions that can act on multiple dimensions of the latent space simultaneously, then given intervention can undo the effects of another. For example, assume model is not invariant to unit interventions on dimension that rotates object, but are invariant unit interventions if two general interventions involve operation on this latent dimension, they can make an object rotate by equal and opposite angles, while changing some other dimensions of the latent state that the model is invariant to. In this case, the interventions end up undoing their effect, and the overall state change does not yield any influence on the model output. By assuming unit interventions that enforce transformations on specific dimensions, we circumvent this failure mode.
all other latent dimensions. Then,
F.2 MODE CONNECTIVITY OF MECHANISTICALLY DISSIMILAR MODELS
We first repeat the following result from prior work (paraphrased per our notations and setup).
Lemma 1. (Simsek et al., 2021), Consider an L-layer network f(; 0). whose activation function o satisfies (0) + O, (()) + for infinitely many odd and even values of where gf()) denotes the th derivative of &. Let ri.r2..... be the minimum number of neurons needed in layers mean-square error) network overparameterized for all layers contains number of neurons Ti > rī. Then, under overparameterization, there always exists continuous, zero-loss path that connects two minimizers.
1 to L for achieving zero error (cross-entropy
dataset D and call
32

---

Under review as a conference paper at ICLR 2023
The result above involves showing permutation symmetry of neural networks yields single con- tinuous manifold of zero loss, and then proving all parameters that yield zero-loss lie on this manifold. We highlight the amount of overparameterriatton needed for the claim's 's validity is rather mild, i.e., just one additional neuron per layer. Also note that while the proof makes assumptions on the analyticity of the activation function used, this constraint is only mandatory for ease of theoretical analysis. Moreover, continuous approximations to ReLU exist which satisfy these as- sumptions. For example, ((x) = @softplus (x) + @sigmoid (4z), where @softplus (r) In(1 + exp(x)) and (2) = -/1++xp(==)) Similar result was also shown by Nguyen (2019), who demonstrates networks with a pyramidal structure, i.e., networks for which the width of any given layer less than or equal to its preceding layers, satisfy Lemma 1.
Our claim on mode connectivity of mechanistically dissimilar models now follows as a corollary.
Proposition 2. (Mode Connectivity of Mechanistically Dissimilar Models.) Assume 81, B2 are minimizers of the loss on a dataset 'D and induce mechanistically dissimilar models. Given sufficient overparameterization, there exists a continuous path along which the minimizers are mode connected.
Proof. By definition, L(((D;) 0)) = 0 for 0 E 00,00, 02. Since the distribution of data plays no role in the proof of Lemma 1, the result must hold for two minimizers that rely on entirely disparate mechanisms (e.g.. background vs. shape) for achieving zero-loss on a dataset D. The claim then directly follows as a corollary of Lemma 1, assuming the model is overparameterized in the sense defined there and the loss is either cross-entropy or mean-square error.
F.3 LACK OF LINEAR CONNECTIVITY IMPLIES MECHANISTIC DISSIMILARITY
Conjecture 1. (Lack of Linear Connectivity implies Mechanistic Dissimilarity.) If two minimizers I and 02 of the loss (Df 0)) on dataset D cannot be linear mode-connected (up to permutations of neurons), their corresponding models f(: 01), [.; must be mechanistically dissimilar.
Neural networks boast the well-known permutation symmetry phenomenon: permuting neurons, while accounting for the fan-in and fan-out weights, yields a model that is functionally the same (Hecht- Nielsen, 1990). That is, after permutation, the model encodes the exact same function as the original model. To avoid this degeneracy, we will assume that we are analyzing two minimizers 02 62 that necessarily are not permutations of each other. In practice, one can run recent methodologies on "neural alignment" to ensure this assumption is valid (Ainsworth et al., 2022; Singh & Jaggi. 2020).
Proof. As per Prop. 1, we need only establish invariance to unit interventions for characterizing the mechanisms underlying model's decision rules and, correspondingly. ascertain mechanistic similarity between two model parameterizations. To that end, we consider a unit intervention A, that we assume the minimizer 02 is invariant to. We will analyze the loss of the model parameterized with linear interpolation of 01, 02 on a counterfactual sample E (x;; generated using intervention A, For brevity, we denote the latent state of
Gx3 (x); correspondingly, we denote the intervened Az, where Az is 0 in all but the 7th dimension, where it is equal to As; aq We can thus write: E(z;
latent state as A (z)
Gx A G (x) Gx( (x + Az).
We now consider the parameterization along a general path Yo, - (t) such that 70, -0 (0) = 0, and -0,-0; (1) = 02. We assess its loss on the counterfactual data via second-order expansion along the data geteragnnrating process:
(f (r; Aft :70
= Lf (z A2):
(9x
+ (Az)² (f (z); (2); (1))
(Az) T ( (Gx (t)) + O(a)), (7)
2 L. ( (1)) + 0: 0z,
L (Gx (=); -0) (1))
+ 2
(ai)
L (f (9x (0); (1))-
d2
33

---

Under review as a conference paper at ICLR 2023
The parameterization along general path connecting the two minimizers 81, 02 can be written in the
following form: "0, =0, (t) = 01 + At 1), where Af 1) = 70,-8, (t) - 01. Then, expanding
the loss achieved by the model with this parameterization on the original data up to second-order along the change in parameters, we get the following.
( (9x 70, -0,
Lf (9x (z); At(t,
L (f (9x (2)))1))) + (t) VoL VoL (6x (z):
ス (10 1))² V (f (9x (z): (At(t, + 1()
(8)
2 (Ao(t, ViL (f (9x (=); (t. 1)),
where the loss and the gradient term can be ignored because 0, is minimizer of the loss on dataset D. Now, substituting Equation into Equation 7, we get the following.
(1; (1; A70) (1))
8
(9x 10)001100 (1)) + ai O2:
T. (9x (3);70 (a,) dze ( (9x (0) (1)).
1 -0(, VIL (9x (z); 10
A0(1, 1) V (J (s)
+
0z.
2
+ (a.)²
1
0²
(A (9x (f)) A0(t.))
1
Att, 1)
Lf + Oz.
(z): + (a:)
02
L (f (2):01)) A0(t,
2
",, 1)
[(((Ex (z):01) (A=) (f (2):01) (Az) D L (9x (1); (Aa) At 1).
2
1
(A((,1)VV (2 ;0) AO 1),
2
1 1A ;())) At 1),
2
1 ( 1),,
where the last equality follows because of the assumed invariance of 0, to the intervention A Now, if the connectivity path 70,-0, (t) is not linear, then there exists an interpolation along the linear path connecting minimizers 01, 02 that has loss higher than the two minimiz-
ers. That is, the displacement vector A 1) does not lie in the null-space of the Hessian and AB(t, V3 (f (x; f1) Ad(t, 1) * 0. Substituting this relation into Equation 9, we get L (E (x:; A 70, -0, (1))) O.
F.4 PROVING CONJECTURE IN A SIMPLIFIED SETTING
Conjecture 1. (Lack of Linear Connectivity implies Mechanistic Dissimilarity.) If two minimizers 8n and 02 of the loss (D; f)) on dataset D cannot be linear mode-connected (up to permutations of neurons), their corresponding models f(:: 01), (-:02) must be mechanistically dissimilar.
As we show next, the conjecture above can be proven in a simplified setting.
Model Setup: We consider binary classification task dataset D {2,, y: 1= where x; R. y E y = {0, 1}. and M is the number of samples. The model is a two-layer, fully connected network defined follows: f(; W) 1 Here, W RDNN denotes the hidden layer with
34

---

Under review as a conference paper at ICLR 2023
N neurons, 1 E RN is an all ones vector, and o(.) is the ReLU activation function. The model is trained to minimize loss (((DDW)) * -i=1 M 1 (y:, f(z;; W)), where ((...) denotes the sample-wise loss whose global minimizer yields yi = fa;; W) for all I2 E DD. This property is satisfied by several loss functions, e.g., mean-squee -ann--suareeee error, L-1 loss, hinge loss, etc. We assume all minimizers are global and interpolating, i.e., they achieve zero loss, a property known to be true for overparameterized neural networks (Kawaguchi, 2016; Kawaguchi Kaelbling, 2020; Nguyen et al., 2018; Nguyen & Mondelli, 2020; Arora et al., 2019). This implies if W. is minimizer, Vi E [M] 8/2 = f(z W.) = N (""""W"" zz).
We next describe the data- Gata-generating process that we will focus on in the following discussion.
Gata-Genera Process: We consider data-generating process with multiple predictive attributes of different complexity, inspired by the one proposed by Shah et al. (2020).
Consider a non-negative even integer K. Define the sets (K) and S (K) that respectively include odd and even integers between [- K K1. We use sign(.) to denote the sign function, which outputs if > 0, 0 if æ = O, and - if x < 0. Unif(S) denotes uniform distribution over the set S. We define a randomized process s, such that x (0) Unif(So) K )), 8x xx(1) (1) Unif(S) (
2
( )) Correspondingly.
given a margin parameter 5 E [o, 0.5). we define the randomized function Tk(2): : {0, 1 + as follows.
(z - e sign(=)),
where e ~ Uni 26)), if K = 0, where e ~ Uni K 2 (.1sx(2)) + K
VD
Tx(2)= =
2v3
(sx(2) + +c).
KVD
2
2/3
(sx(2) - -esig(())))
where e ~ Unif([0, 6)), if K 2 1, 1,1sk(>)1=
KVD
(10)
Note that (x)=) produces a zero-mean output with variance I/D. The margin d allows us draw infi- nite samples from the function. More importantly, Tx(2) is left-invertible, i.e., given its output, we can infer z. Correspondingly, if = defines the target la- bel y. inverting the attribute Tx (z) will allow us to solve a classification task defined
Ta(a)
0
-05
this attribute.
Ta(+)
However, this inversion process requires inference of K piece-wise linear splines to model the optimal decision boundaries (see Fig. 30). The scalar K can thus can be considered measure of the complexity of the attribute, inline with prior work on simplic- ity bias in neural networks (Nakkiran al., 2019; Shah et al., 2020; Valle-Perez et al., 2018; Scimeca al., 2021: Hu et al., 2020). For example, if K 0, the attribute is linearly separable and of least com- plexity. This notion of complexity is particularly natural for studying neural networks with ReLU ac- tivations because each neuron in such a model rep- resents spline function and several such neurons can approximate complex decision boundaries by representing them with such piece-wist spline func- tions (Balestriero et al., 2018; Balestriero & Bara- niuk, 2018; Balestriero, 2017; Wang et al., 2018).
0
0
0.5
05
y=1
y=
Figure 30: K-Complex Outputs. We illustrate
the output range of randomized function Tw (=(=) (Eq. 10) for K = 0 (top) and K 4 (bottom). Note the input = E {0. 1} (shown grey, red respectively) deterministically controls the out- put values. Thus, y 2, the output Tx (2) is perfectly predictive of the target label. How- ever, inverting this function requires inference of K piece-wise linear splines to model the op- timal decision boundaries (shown blue, dotted lines). Increasing the value of K makes the task consequently harder.
The overall data-generatin process g(Z) transforms the n--dimensional random variable Z from the latent space 21 X 22 X 2n E {0, 1)"1 to produce samples with n attributes of different complexities Tk, (z1), TK, (22) (z2)...., Tk., (zn) and appends D n noisy attributes, sampled from a symmetric, zero-mean distribution V with variance z. e.g..
the Gaussian distribution N (0, 1/VD) or uniform distribution 2/ (-v3/n, v 3/)) Correspondingly, generation of a sample (sr v) can be represented as follows:
(5. v) (()) [TK, (z1), TKe (z2), ... Tk., (zn), ,1 u2,..., va-n] T
(11)
35

---

Under review as a conference paper at ICLR 2023
where V4 ~ V all i by the function G.(.) = TK. (.), which inverts the attribute Tk, (24) that is assumed to define the label. Note again another attribute, say Tk, (zj), will also be predictive of the label if the corresponding latent, 2j. is correlated with 24. This is similar to putting a correlated cue attribute in the data, as was done in experiments in the main paper. Thus, while the process above is relatively simplified, it is a valid abstraction of our empirical setup from the paper. In the following, we will consider perfectly predictive attributes, i.e., we do not assume partial correlation.
(1.n): In the above, the target label is assumed to be generated
We next define the notion of an activation pattern and matching of two models in their acti- vation patterns. In the following. En denotes the permutation group of order N over the set (1.2...., } (Bronstein et al., 2021).
Definition 6. (Activation Pattern.) The activation pattern of a model f(:: W) on input x is defined as the vector 6'(WT x)) whose elements are indicator variables denoting whether the j"h hidden neuron is activated for input x, i.e., &'(""'a [W))) = 1 (W3 > 0).
Note that @WTx) = d' d'WT x) O (WTx), where @ denotes the element-wise product, if 6(.) is the ReLU function.
Definition 7. (Matching in Activation Patterns.) Consider a dataset DD and two models f(: W1) and f(:; W2)- We call the models matching in activation pattems on dataset DD if there exists permutation æ E En that rearranges neurons of [[: W2) to match ((:Wa)'s activation patterns, ie., & ('WTT xr) 6' (mW))) ) all D.
Next, we establish the relationship between two models' activation patterns and linear mode connec- tivity.
Lemma 2. (Alignment Constraint for Linear Mode Connectivity.) If minimizers Wa and W exhibit linear mode connectivity on dataset D then the models f(; Wa), W3) are matching in activation patterns on the dataset. That is. for all D, we have o' (WW". x) = d' d(WTz))
Proof. Note that linear mode connectivity is translation invariance property of the loss in the parameter space. Since we assume interpolating minimizers, this invariance extends to model predictions. Consequently, the derivative of the model prediction along the linear path TW (!)
(tt) = Wa + t(WB- - Wa) is zero; that is, f(;; (t)) = O. This implics,
1 (t))))))='((())))
=.
(12)
Ot
Substituting = in Eq. 12, we get,
d' ((WTx)(W[Ts) & (W3 1 ((WTa)) 1st x(WT))
(13)
This implies,
(6(( - o' dW 2 =.
(14)
Next, we define the following vector.
1, if W 0 and W S 0;
1 Mas p.)
(15)
0. otherwise.
Define the vector (a- 8_) in similar manner. Then, it is easy to see that
o d((haa)- o' (WF (04 0-) 1 1a B+)-
(16)
Substituting the above relationship in Eq. 14 gives
(W (WTx) = TTm 84) (W..).
(17)
Note that in the above equation, the left-hand side is a sum of positive reals, while the right-hand side is sum of negative reals. That is, the equality cannot hold unless both are equal to zero for all x E 'D. This implies (a, 0_) 1 8,) = O. That is, there is no neuron in model f(; W.) that is active while the corresponding index neuron in (.; Wa) is inactive. Consequently, for linear mode connectivity to hold, the neurons the same index in the two models should activate/inactivate together for any given sample, hence producing the same set of activation patterns. This completes the proof.
36

---

Under review as a conference paper at ICLR 2023
Table 5: Illustrating Simplicity Bias. We train models on a dataset with predictive attributes of complexities O and 4. Column titles indicate which attributes were allowed to remain predictive during training, i.e., were not randomized via interventions: e.g. K₁ implies only the linearly separable attribute is predictive in the training data. Rows report difference in loss on a test dataset Dx; which contains attributes of complexity K₁ and another test dataset Dx2 which contains attributes of complexity K2. Results are computed up to 4 digits of precision and averaged over 3 seeds. We see models trained on data with both predictive attributes behave similarly to models trained on K = 0 attribute only; that is, they are invariant to the more complex attribute for which K = 4.
Complexity of Train Attribute Complexity of Test Attribute
K1 = 0
K1 = 4
K1 0,4
K2 0
K₂ 4
K2
0
K2 4
K2 0
K2 4
[Cf(DK,; W)) - [(f(Dk,; W))I
0.0
22.79
26.31
0.0
0.0
18.84
Remark 1. (Lemma 2 highlights why neurons must be permuted for linear mode connectivity.) The lemma above shows that if two models produce the same activation patterns, the models are "effectively linear" with respect to each other. This enables linear interpolation of the two models without increasing error. We also highlight that if two models produce activation patterns that are a permutation of each other (e.g., this can happen if their initializations were permutations of each other), then un-permutingg them will make the models linear mode connected. Thus, Lemma is inherently an alignment constraint and can also be regarded as a precise condition under which the conjecture by Entezari et al. (2021) 1) holds: models trained independently on a given dataset using gradient descent are linear mode connected. Even though the result above was shown for two-layer model, it is easy to see that a more general statement is true: if two minimizers induce models that produce the same activation patterns, then there exists a permutation of neurons under which the two models can be linear mode connected.
Remark 2. (Wasserstein-I distance between activation patterns serves as proxy for when linear mode connectivity will hold.) The activation pattern of a model for a given sample is a vector of binary variables. Thus, the difference between two activation patterns o' OWW"" a x) and o' (W3) x) can be computed by simply comparing their means NV | x)
g x)). which is in fact the Wasserstein-1 distance between two Bernoulli distribution for which p + 1 o' (WT x)). This value p can be regarded as the probability a neuron in the model is activated. Correspondingly, when the Wasserstein-I distance between two activation patterns is low, we can expect that there exists permutation of neurons that allows the two models to be linear mode connected. The W-1 distance can thus be regarded as a proxy for assessing whether two models can be linear mode connected. Further, we highlight that even though this result is derived for a specific model architecture, it is actually quite general: any two models with zero W-1 distance must be linear mode connectable (up to permutations) because their activation patterns will necessarily be the same.
Next, we rephrase the result on simplicity bias of neural networks by Shah et al. (2020); Valle-Perez et al. (2018); Nakkiran et al. (2019); Seimeca et al. (2021) using the notations defined in this paper.
Lemma 3. (Simplicity Bias.) Assume dluaa-genratinng process G produces n perfectly predictive attributes with respective complexities [K] {K1, K2...., K. Let m be the index of the latent corresponding to the simplest attribute, i.e., m == argmin [K]. If W is minimizer identified using gradient descent on a dataset that contains IID samples retrieved from G, then the corresponding model f(: W) will be invariant to unit interventions on all but the latents of the simplest predictive atribute, i.e., I(W)= = (A: : i + m).
Thus, even if a dataset contains multiple predictive attributes, minimizers identified using gradient descent induce models that only utilize the simplest attributes for making their predictions.
We provide empirical demonstration of this claim in Tab. 5. Specifically, we train models using SGD for our assumed [(; W) architecture, using 512 neurons in the hidden layer. We sample 50000, 128-dimensional inputs from the data-generating process discussed in Eq. 11 with K = 0 and K = 4 complex predictive attributes present in the dataset. We analyze three training scenarios: (i) when only K = attribute is allowed to be predictive and the K = 4 attribute is randomized via interventions; (ii) when only K 4 attribute is allowed to be predictive and the K 0 attribute is randomized via interventions; and (iii when both attributes are allowed to be predictive. Evaluation involves assessing invariance of the trained model to the two predictive attributes by computing loss test
37

---

Under review as a conference paper at ICLR 2023
dataset that contains both predictive attributes and a counterfactual variant of the dataset for which either (a) K = 0 or (b) K = 4 complexity attributes have been randomized via interventions. If loss remains the same, the model is invariant to interventions on the attribute of that complexity, thus implying the model has not learned a mechanism to identify that attribute. Results are shown in Tab. 5. We see intervening on the K = O attribute yields an increase in loss in scenarios (i), (iii), indicating those models have learned a mechanism to identify that attribute. Meanwhile, intervening on the K 4 attribute yields an increase in loss only in scenario (ii), indicating the models trained from the other two scenarios are invariant to the K = 4 complex attribute. While this is expected for scenario (i), the fact that this behavior emerges for the scenario (iii) where both predictive attributes can be used for training, is a consequence of simplicity bias of SGD.
Now consider a setting where two models make their predictions using different simplest predictive attributes from a dataset containing multiple predictive attributes. Then, if two such models rely on attributes of different complexities, we can be certain they produce different activation patterns.
Lemma 4. (Disparate Complexity of Mechanisms Disallows Matching Activations). Consider an IID sampled dataset Da,3 from data- sentaagnnraaingg process that produces predictive attributes
Tk., (-), TK (.), where, without loss of generality, Ka > Kgg Let Wa denote a minimizer of the loss C[f W)) and assume its induced model relies on Tk. (.) for making its predictions; similarly define Wg. Then, there exists no permutation E Σn such that f(; Wa) and f(:; (())) are matching in activation patterns on Da.B-
Proof. The claim follows via contradiction. Assume permutation и exists such that the two models are matching in activation patterns on Da.g. . Denote the weights of the 7th neuron in Wa via W Then W т (Wa)* are the weights of the neurons matched via æ. Since using the attribute Tk. (.) for predicting the label corresponds to inference of k piece-wise spline functions, the probability the 7th neuron with weights W will be activated for an IID sampled input x from the data-generating process is 2k. However, since Ka * Kg, the neuron with weights T <(W3)' does not activate with the same probability. That is, there exist samples for which W is activated, but x (Wa)' is not. This contradicts our assumption that there exists a permutation that allows matching in activation patterns for the two models.
Combining the results above, we have the following theorem.
Theorem 1. (Disparity in Simplest Attributes Precludes Matching). Consider a dataset 'D contains multiple predictive attributes. Assume two minimizers of the loss ( W)) induce mechanistically dissimilar models that identify altributes of different complexity to make their predictions. Then, their exists no permutations of neurons for which the models exhibit linear mode connectivity.
Proof. The result follows directly from the application of Lemmas 2, 3, 4. Specifically, Lemma 2 shows matching in activation patterns required for two models to exhibit linear mode connectivity (up to permutations). Lemma 3 shows one need only analyze mechanistic dissimilarity with respect to the simplest attributes to compare the activation patterns between two models. Lemma 4 shows two models use attributes of different complexity make their predictions, they cannot match in activation patterns.
Empirical Verification: See Fig. 31.
Let us now revisit Conjecture 1 for our simplified setup. If two minimizers are not linear mode connected, their induced models must produce different activation patterns as per Lemma 2. In Theorem 1, we have shown if these models are mechanistically dissimilar such that they learn mechanisms to identify attributes of different complexity, then they will produce different activation patterns and not exhibit linear mode connectivity, partially verifying the conjecture.
However, it remains possible that two mechanistically dissimilar models learns mechanisms to identify attributes that are different, but have the same complexity. Can such minimizers possibly produce the same activation patterns is easy to see the answer is yes; e.g., any two solutions of the linear system of equations y = Wx can be interpolated regardless of their prediction mechanisms (hence the x" in Tab. 2). However, this constraint that the mechanisms must rely on equal complexity attributes is relatively theoretical only. For example, if D2 contains multiple attributes of similar complexity that allow minimizers
dataset, hence exhibiting linear mode connectivity? In theory, it
38

---

Under review as a conference paper at ICLR 2023
permuted
W-1 Dist. 0.262
W-1 Disi. 0.006
W-1 Dist. 0.255
W-1 Dist. = 0.0003
W-1 Dist. 0.262
W-1 Dist. 0.0004
0.1
0.1
Test toos
0.05
0.05
9k-4
0.5
0c-o
Oc-4
0.5
%
0p-4
0.5
0,40,4
0k-
0.5
%
Oc-o
0.5 Oc-4
0c-o 0.5 0.:0.4
(a)
(b)
Figure 31: Linear Connectivity and Mechanistic Similarity. We train models on 50000 samples drawn from the data-generating process discussed in Eq. 10, allowing two predictive attributes of complexity
and K Models are also trained on counterfactual variants of this dataset, where of the predictive attributes has been randomized via interventions. For example, 0 =0,4 denotes a minimizer trained on data with both attributes, while 0== denotes a minimizer identifed via training the counterfactual dataset that contains only the K 0 predictive attribute; note % denotes use of a different initialization seed. Subsequently, we assess linear mode connectivity (before and after permutation) of models by using evaluation dataset of 10000 samples similar to the base training dataset, i.e., both K = and K = 4 predictive attributes are allowed. Plot titles denote Wasserstein-1 distance between the models whose linear mode connectivity is being assessed. We see that models which have learned mechanisms to identify attributes of different complexity have a large Wasserstein- distance between their activation patterns; consequently, they cannot be linear mode connected, even after permutation of neurons. Meanwhile, models reliant on the same mechanisms have small Wasserstein distance and can indeed linear mode connected. For example, 0K=n and =04 learn the same mechanisms due to simplicity bias and can be linearly connected (see Tab. 5) but they do not exhibit linear connectivity with 0k=4; meanwhile, 0==4 and K==4 can be linearly connected. Note however the latter case of more complex, i.e., K 4 attribute required permutations to match the neurons for linear connectivity, while the former case linearly separable attribute did not. This behavior emerges due to the fact that all neurons learn be always active for the K = 0 predictive attribute-see Soudry et al. (2018); Shah et al. (2020) for proof.
retrieved from D2 perform well the dataset, then given SGD (and related algorithms) force neural networks to converge to max-margin solutions, we see that minimizers retrieved via training on Dj will have already learned mechanisms identify all attributes of same complexity (Soudry et al., 2018; Lyu Li, 2019; Gunasekar et al., 2018; Nacson et al., 2019). In that case, use of D2 to create minimizer that learns a different mechanism is practically moot, since the relevant mechanism from D itself. Nonetheless, this hints at the idea that naïve fine-tuning can successfully alter model's mechanisms perform well target distribution when the desired mechanism is of similar complexity; otherwise, a loss barrier must be surmounted for successful learning on the target distribution. Therefore. a model relies on spurious attributes, which are often linearly separable (Shah et al., 2020), fnn-ttnnig will only help identify another linearly separable attribute the target distribution. This concept can be used to analyze benefits and limitations of recent averaging-basee ensemble methods (Wortsman et al., 2022b;a; Rame et al., 2022), though we leave this direction to future work.
already learn
39


Reviews:
[{"review_id": "yUa-KT6_vD", "reviewer": "Reviewer_X6v6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is primarily concerned with **identifying spurious attributes** learned by NNs when learning a specific dataset and **correcting them** during fine-tuning. To this end, they propose mechanistic similarity, a concept that checks whether two pre-trained models with low loss are functionally similar. Based on this concept, they develop connectivity-based fine-tuning, a fine-tuning technique that can modify the mechanism of a pre-trained model.", "summary_of_the_review": "This paper proposes a novel solution (Connectivity-Based Fine-Tuning) to an interesting problem (removing spurious attributes). While the concepts of Mechanistic Similarity and Mechanistic Mode Connectivity are exciting and inspiring, no discussion has been given on how to remove the spurious attribute. The paper will be helpful to readers in many fields with revisions for natural connections between sections.", "strength_and_weaknesses": "* Strength\n\n  * From a theoretical point of view, their contributions are largely twofold. First, authors **applied the concept of invariance** to explain the underlying mechanism of the models. (Section 3) Second, a **fascinating conjecture** (Conjecture 1) was presented by the authors about how mode connectivity, an abstract concept in loss-landscape, will affect real-world applications. (Section 4)\n  * On the application side, the authors proposed Connectivity-Based Fine-Tuning as a way to **intentionally editing** the pre-trained model's underlying mechanism. (Section 5)\n  * These contribution points are novel and interesting for scientists studying the loss-landscape of NNs.\n\n* Weakness\n\n  However, readers who wish to apply these findings to their problems may encounter some difficulties for the following reasons. Considering this is discussion phase 2, it is impossible to revise the paper, but I believe the first and third issues should be addressed.\n\n  1. It is unclear how to get rid of spurious attributes in Sections 3 and 4. For example, the authors only explain how the proposed notion of mechanistically similar confirms the functional equivalence of the two models in Section 3. **There is no discussion of what mechanstically similarity corresponds to spurious attributes**. This issue also applies to Section 4 (i.e., There is no explicit connection between spurious attributes and Connectivity-Baed Fine-Tuning).  As a result, readers may feel as if there is no natural connection between sections. **Since the content of each section is interesting enough, I believe the connectivity between sections is the greater concern.**\n\n  2. Although the code was submitted during the discussion period, Section 5's Connectivity-Based Fine-Tuning remains vague in their main paper. For example, how do you pick a minimal dataset $\\mathcal{D}_{NC}$ that does not contain attribute C? Also, why does the average representation matching term used in (ii) of equation (2) make an invariant prediction? For example, what happens if the term is the average of multiple augmentations of a single image rather than the average of multiple images of a single class? **Since Connectivity-Based Fine-Tuning is not the only (or optimal) algorithm that achieves the authors' purpose, these questions are quite natural, and the authors should have explained them appropriately.**\n\n  3. There are some mathematical expressions that seem to be abused.\n\n     * Although an inverse mapping is required for isomorphism, $\\mathcal{A}^{\\alpha_i}_i$ has no inverse mapping because it fixes the value of the $i$-th component of the latent variable to $\\alpha_i$.\n\n     * While authors denote the domain of $\\mathcal{E}$ as $\\mathcal{X} \\times \\mathbb{R}^{m}$, $\\mathbb{R}^{m}$ is incorrect because $\\hat{\\mathcal{A}}$ is a concatenation of interventions.\n\n     * Authors used square brackets both to represent lists (e.g., $[K]$ in Section 2) and to represent closed intervals ($[0,1]$ in Section 5). \n\n     * Furthermore, some mathematical expressions are used without any definition. For example, ㅅthey did not define exactly what $[K]$ means. This can only be inferred from context. This issue also applies to the Truncate function in (ii) of equation (3). The exact mathematical expression of $f_\\gamma$ in Section 5 is not specified. Similarly, $\\gamma_{\\theta \\rightarrow \\theta_C}(t)$ is not a path, but rather a point on a path. \n\n       **Combined with the first weakness, these misuses of mathematical expressions will present a huge barrier for readers unfamiliar with the field.**", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jiyiJuNMUC", "reviewer": "Reviewer_fq5i", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work associates the notion of linear mode connectivity of two models with the functional similarity (mechanistic similarity) of the two models. They argue that the linear mode connectivity of the two models indicates they both have inherited potentially spurious/undesirable representations. Furthermore, they argue that naive fine-tuning cannot remove these spurious representations. Based on this connection, the authors propose a new algorithm referred to as connectivity-based fine tuning (CBFT), which encourages two properties for optimizing the parameters:\n\n- Removal of the linear path between to spurious model\n- Encourage functional similarity of the current model across spurious and non-spurious datasets.", "summary_of_the_review": "There needs to be more work in understanding mode connectivity's functional utility, which this paper meaningfully explores. As it stands, mode connectivity (linear or non-linear) is an empirical phenomenon that is not well understood. This work explores possible avenues for improving model generalization based on this phenomenon. The main caveat to this work is the clarity is made somewhat opaque by the presentation of grids of experiments. The idea would be communicated much better if those results were condensed down.", "strength_and_weaknesses": "Strengths:\n- The hypothesis this paper proposes about the optimization landscape and, specifically, the functional relationship with the energy barrier is interesting and an important area of better understanding the utility of mode connectivity. \n- Thorough appendix with corresponding loss curves. (This reviewer's opinion: loss curves are easier to interpret for mode connectivity than accuracy curves)\n\nWeaknesses:\n- Unclear on which of the two steps in CBFT contributes to the improvements. An ablation analysis of each step would help disentangle the contribution.\n- The definition of mode connectivity shown in Figure 6 seems to make mode connectivity and mechanistic similarity expected and potentially uninteresting. The monotonicity of the evaluation curve going from \\theta_c to \\theta_{nc} should be expected since the transition is going from a model trained on that dataset to one trained on a different dataset and vice versa. Could the authors clarify this issue? Perhaps this reviewer has made an error in understanding Figure 6. A more interesting result would be a \"u\" shaped accuracy barrier. But on most plots in Figure 4, this does not appear, and most curves are the expected monotonic trend.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ad2c5LH8eCR", "reviewer": "Reviewer_iiL3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors aim to understand a model's sensitivity to spurious features through the lens of mode connectivity. First, they come up with a definition of the mechanistic similarity of different modes, on the basis of invariance to different interventions.  Then, they show a stronger form of connectivity between different modes, on the basis of mechanistic similarity. Finally, they conjecture that all modes with mechanistic similarity must be connected with a linear path up to permutations. They corroborate their conjecture with multiple experiments on CIFAR-10. Finally, they propose a method of connectivity-based fine-tuning to eliminate a model's sensitivity to spurious features.", "summary_of_the_review": "Overall, my scores are slightly on the positive side. The paper aims to advance our understanding of mode connectivity and its connection to the decisions of a model. However, the proposed algorithm CBFT assumes that we must have access to both clean examples and examples with statistical cues, which begs the question of the relevance of CBFT compared to the direct training on clean examples. Moreover, experiments with stronger statistical cues will help strengthen the conjecture of the paper. ", "strength_and_weaknesses": "The paper introduces a novel connection between mode connectivity and a model's sensitivity to spurious features. The conjecture is in itself very interesting and will be useful for the scientific community. Moreover, the authors present experiments in synthetic datasets to support their conjecture and show a simple Taylor expansion-based proof for local correctness. To round it off, they propose novel connectivity based fine-tuning, based on the conjecture, to remove a model's dependence on spurious cues.\n\n\nI have the following questions:\n\n(a) In Figures 4 and 5, shouldn't train/evaluation loss be plotted, as opposed to the train/test accuracy because definitions 4, 5, proposition 1, and conjecture 1 include loss function? Do we still see barriers in linear paths between mechanistically dissimilar models in figures 4 and 5, with loss function?\n\n(b) The CBFT algorithm (in 3) assumes that we have access to both a clean dataset and a dataset with statistical cues. Can the authors comment on whether one can identify examples from both groups in an unsupervised manner? If not, why should one use CBFT and not directly train the model (from scratch) on the clean examples in the dataset?\n\n(c) It's not clear how the authors run CBFT introduced in eq. (3) on real-world models.\n\n(i)  How do the authors optimize step (i)? Do they explicitly search over the neighborhood around $\\theta_C$ for one $\\theta$ that maximizes the loss function on the line connecting $\\theta$ and $\\theta_C$? Or do they conduct an optimization of the form $\\min_{\\theta, 0 \\le t \\le 1} | \\lambda_1 - \\mathcal{L}_{CE} ( \\hat{y}  ( \\mathcal{D}_C ;  (1-t) \\theta + t \\theta_C ) | $?\n\n(ii) Furthermore, do the authors follow an alternative minimization for steps (i) and (ii)?\n\n(iii) Do the authors include permutation in step (i)?\n\n(d)  All statistical cues introduced in the paper are very simple (linearly separable) cues.\n\nWill we observe the same invariances to stronger forms of statistical cues e.g. label specific translations and rotations of the images? An example is a classification setting comparing images of cats and dogs, with the images of cats rotated by 45 degrees and the images of dogs rotated by 135 degrees.  \n\n(e) The authors point out a very important characteristic of \"quadratic paths\" in the appendix: the evaluation performance of the model on the path will depend on the examples used to find the path, and hence may not show generalization across datasets/distributions. \n\nHowever, isn't this also true for the permutation learned for linear paths, since it involves finding the best permutation $\\pi$ with a greedy algorithm on a given set of examples? \n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_BPlyoqpyS", "reviewer": "Reviewer_Y4FF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the optimization landscape of models which have learned different mechanisms. Beyond the standard definition of linear mode connectivity, they introduce \"mechanistic mode connectivity\" which tests mode-connectivity under changes in the data distribution. Based on these findings they propose a fine-tuning mechanism to exit the basin associated with a spurious pre-training solution.", "summary_of_the_review": "This paper is a thorough and interesting scientific investigation of the optimization landscape of mechanistically similar/dissimilar models. My concern is that the experimental set-up is detached from real world use cases of pre-training/fine-tuning.", "strength_and_weaknesses": "Strengths: Thorough scientific investigation of mode connectivity/mechanistic mode connectivity. Definitions, propositions, and proposed methods are well motivated, and the proposed CBFT method demonstrates good empirical performance.\n\nWeaknesses: I very much enjoyed the paper, and the following concern may reflect a bias perspective, however I thought it was still worth raising:  My main concern is with the experimental set-up. While interesting from a scientific perspective, I'm not sure that the issues addressed in this paper are interesting practically. As such, I'd recommend making this more clear in the paper. As one concrete instantiation of this concern, the paper claims: \"We have demonstrated a significant limitation of naive fine-tuning: it can fail to eliminate spurious patterns learned during pre-training.\" However, to me the point of pre-training is to pre-train on all the data you can. We see this in LM pre-training on trillions of tokens, or recently in vision models with billion scale datasets such as LAION. Accordingly, the setting studied in this paper seems like the experimental set-up in this paper may be a bit contrived. For instance, my guess is that CBFT would actually make the model less robust if applied to fine-tuning CLIP (e.g., this may be of interest: https://arxiv.org/abs/2109.01903). In short, to me the fact that pre-training biases the fine-tuning solution seems to be a feature, not a bug, of pre-training.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "S1bfonL8Tb", "reviewer": "Reviewer_bN6r", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies a new concept mechanistic mode-connectivity, which focusing on the mode-connectivity of models trained on different data distribution. Meanwhile, the paper also proposes connectivity based fine-tuning (CBFT), aiming at overriding existing mechanisms of a pretrained model by fine tuning it on a small out-of-distribution dataset.", "summary_of_the_review": "The overall quality of this paper is good, where the analysis is comprehensive. Hence I recommend to accept this paper.", "strength_and_weaknesses": "Pros:\n1. The paper is well-written and easy to follow, although minor grammar error also exist (e.g. 2nd paragraph line 3 \"to to\"). \n2. The introduced problem is quite interesting and well-motivated.\n3. The paper has conducted extensive experiment results to verify the effectiveness of the proposed hypothesis.\n\nCons:\n1. For figure 2, I'm confused by the last two rows of right figure. What is the difference between \"Rand. cue\" and \"Rand. Image w/ cue\"?\n2. For the augmented data, are they used along with original data for training? Or Models are only trained on the augmented data?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject