SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Michael A. Lepori $^{1*}$  Thomas Serre $^{2}$  Ellie Pavlick $^{1}$
$^{1}$ Department of Computer Science  $^{2}$ Carney Institute for Brain Science
Though modern neural networks have achieved impressive performance in both vision and language tasks, we know little about the functions that they implement. One possibility is that neural networks implicitly break down complex tasks into subroutines, implement modular solutions to these subroutines, and compose them into an overall solution to a task — a property we term structural compositionality. Another possibility is that they may simply learn to match new inputs to learned templates, eliding task decomposition entirely. Here, we leverage model pruning techniques to investigate this question in both vision and language across a variety of architectures, tasks, and pretraining regimens. Our results demonstrate that models often implement solutions to subroutines via modular subnetworks, which can be ablated while maintaining the functionality of other subnetworks. This suggests that neural networks may be able to learn compositionality, obviating the need for specialized symbolic mechanisms.
Though neural networks have come to dominate most subfields of AI, much remains unknown about the functions that they learn to implement. In particular, there is debate over the role of compositionality. Compositionality has long been touted as a key property of human cognition, enabling humans to exhibit flexible and abstract language processing and visual processing, among other cognitive processes (Marcus, 2003; Piantadosi et al., 2016; Lake et al., 2017; Smolensky et al., 2022). According to common definitions (Quilty-Dunn et al., 2022; Fodor & Lepore, 2002), a representational system is compositional if it implements a set of discrete constituent functions that exhibit some degree of modularity. That is, blue circle is represented compositionally if a system is able to entertain the concept blue independently of circle, and vice-versa.
It is an open question whether neural networks require explicit symbolic mechanisms to implement compositional solutions, or whether they implicitly learn to implement compositional solutions during training. Historically, artificial neural networks have been considered non-compositional systems, instead solving tasks by matching new inputs to learned templates (Marcus, 2003; Quilty-Dunn et al., 2022). Neural networks' apparent lack of compositionality has served as a key point in favor of integrating explicit symbolic mechanisms into contemporary artificial intelligence systems (Andreas et al., 2016; Koh et al., 2020; Ellis et al., 2023; Lake et al., 2017). However, modern neural networks, with no explicit inductive bias towards compositionality, have demonstrated successes on increasingly complex tasks. This raises the question: are these models succeeding by implementing compositional solutions under the hood (Mandelbaum et al., 2022)?
*Correspondence to: michael_lepori@brown.edu
Contributions and Novelty:
1. We introduce the concept of structural compositionality, which characterizes the extent to which neural networks decompose compositional tasks into subroutines and implement them modularly. We test for structural compositionality in several different models across both language and vision $^2$ .
2. We discover that, surprisingly, there is substantial evidence that many models implement subroutines in modular subnetworks, though most do not exhibit perfect task decomposition.
3. We characterize the effect of unsupervised pretraining on structural compositionality in fine-tuned networks and find that pretraining leads to a more consistently compositional structure in language models.
This study contributes to the emerging body of work on "mechanistic interpretability" (Olah, 2022; Cammarata et al., 2020; Ganguli et al., 2021; Henighan et al., 2023) which seeks to explain the algorithms that neural networks implicitly implement within their weights. We make use of techniques from model pruning in order to gain insight into these algorithms. While earlier versions of these techniques have been applied to study modularity in a multitask setting (Csordás et al., 2021), our work is novel in that it applies the method to more complex language and vision models, studies more complex compositional tasks, and connects the results to a broader discussion about defining and measuring compositionality within neural networks.
2 Structural Compositionality
Most prior work on compositionality in neural networks has focused on whether they generalize in accordance with the compositional properties of data (Ettinger et al., 2018; Kim & Linzen, 2020; Hupkes et al., 2020). Such work has mostly yielded negative results – i.e., evidence that neural networks fail to generalize compositionally. This work is important for understanding how current models will behave in practice. However, generalization studies alone permit only limited conclusions about how models work.
As discussed above, leading definitions of compositionality are defined in terms of a system's representations, not its behavior. That is, definitions contrast compositional systems (which implement modular constituents) with noncompositional systems (which might, e.g., rely on learned templates). Poor performance on generalization studies does not differentiate these two types of systems, since even a definitionally compositional system might fail at these generalization tasks. For example, a Bayesian network that explicitly represents and composes distinct shape and color properties might nonetheless classify a blue circle as a red circle if it has a low prior for predicting the color blue and a high prior for predicting the color red.
Thus, in this work, we focus on evaluating the extent to which a model's representations are structured compositionally. Consider the task described in Figure 1. In this task, a network learns to select the "odd-one-out" among four images. Three of them follow a compositional rule (they all contain two shapes, one of which is inside and in contact with the other). One of them breaks this rule. There are at least two ways that a network might learn to solve this type of compositional task. (1) A network might compare new inputs to prototypes or iconic representations of previously-seen inputs, avoiding any decomposition of these prototypes into constituent parts (i.e., it might implement a non-compositional solution). (2) A network might implicitly break the task down into subroutines, implement solutions to each, and compose these results into a solution (i.e., it might implement a compositional solution). In this case, the subroutines consist of a  $(+/-)$  Inside detector and a  $(+/-)$  Contact detector.
If a model trained on this task exhibits structural compositionality, then we would expect to find a subnetwork that implements each subroutine within the parameters of that model. This subnetwork should compute one subroutine, and not the other (Figure 1, Bottom Right; "Subnetwork"), and it should be modular with respect to the rest of the network — it should be possible to ablate this subnetwork, harming the model's ability to compute one subroutine while leaving the other subroutine largely intact (Figure 1, Bottom Right; "Ablation"). However, if a model does not exhibit structural compositionality, then it has only learned the conjunction of the subroutines rather than
2Our code is publicly available at https://github.com/mlepori1/Compositional_Subnetworks.
Figure 1: (Left) An illustration of the tasks used to study structural compositionality. Stimuli are generated via the composition of two subroutines:  $(+/-\text{Inside})$  and  $(+/-\text{Contact})$ . These stimuli are used to construct odd-one-out tasks, where the model is tasked with identifying the image that does not follow a rule from a set of four samples. Here, two objects must be in contact, and one must be inside the other. Rule following images correspond to the upper right quadrant. A model may solve this task in two ways. (Middle) It may implement a non-compositional solution, e.g., storing learned template that encodes only the conjunction of the two subroutines. In this case, one should not be able to find a subnetwork that implements one subroutine and does not implement the other. Concretely, there should be no difference in the subnetwork's performance on examples that depend on computing one subroutine vs. another. Ablating this subnetwork should harm the computation of both subroutines equally. In other words, there should be no difference in accuracy between examples that depend on different subroutines. (Right) A model may implement a compositional solution, which computes each subroutine in modular subnetworks and combines them. In this case, one should find a subnetwork that implements, say,  $(+/-\text{Inside})$ , and this subnetwork should achieve high accuracy on examples that require computing  $(+/-\text{Inside})$  and low performance on examples that require computing  $(+/-\text{Contact})$ . In other words, the difference in accuracies between the  $(+/-\text{Inside})$  and  $(+/-\text{Contact})$  examples should be positive. Likewise, one should be able to ablate this subnetwork and maintain performance on  $(+/-\text{Contact})$  while compromising performance on  $(+/-\text{Inside})$ , and so the difference in performance should be negative. Hypothetical results are represented as differences in performance between both types of examples.
their composition. It should not be possible to find a subnetwork that implements one subroutine and not the other, and ablating one subnetwork should hurt accuracy on both subroutines equally (Figure 1, Bottom Center). This definition is related to prior work on modularity in neural networks (Csordás et al., 2021; Hod et al., 2022), but here we specifically focus on modular representations of compositional tasks.
3 Experimental Design
Here we define terms used in the rest of the paper. Subroutine: A binary rule. The  $i^{th}$  subroutine is denoted  $SR_{i}$ . Compositional Rule: A binary rule that maps input to output according to  $C = SR_{1} \& SR_{2}$ , where  $SR_{i}$  is a subroutine. Compositional rules are denoted  $C$ . Base Model: A model that is trained to solve a task defined by a compositional rule. Denoted  $M_{C}$ . Subnetwork: A subset of the parameters of a base model, which implements one subroutine. The subnetwork that implements  $SR_{i}$  is denoted  $Sub_{i}$ . This is implemented as a binary mask,  $m_{i}$ , over the parameters of the base model,  $\theta$ , such that  $Sub_{i} = M_{C; \theta \odot m_{i}}$ , where  $\odot$  refers to elementwise multiplication.
Figure 2: Illustration of the experimental design. For brevity, we denote "subroutine" as SR in the diagram. (A) First, we train a neural network on a compositional task (Inside-Contact), ensuring that it can achieve high accuracy on the task. (B) We then optimize a binary mask over weights, such that the resulting subnetwork can compute one subroutine (+/- Inside) while ignoring the other (+/- Contact). We evaluate this subnetwork on datasets that require computing the target subroutine (+/- Inside). We also evaluate this subnetwork on datasets that require computing the other subroutine (+/- Contact). We expect success on the first evaluation and failure on the second if the model exhibits structural compositionality. (C) We invert the binary mask learned in (B), ablating the subnetwork. We evaluate this on the same two datasets, expecting performance to be harmed on the target subroutine and performance to be high for the other subroutine.
Ablated Model: The complement set of parameters of a particular subnetwork. After ablating  $Sub_{i}$ , we denote the ablated model  $M_{ablate_{i}}$ .
3.2 Experimental Logic
Consider a compositional rule,  $C$ , such as the "Inside-Contact" rule described in Figures 1 and 2. The rule is composed of two subroutines,  $SR_{1}$  (+/- Inside) and  $SR_{2}$  (+/- Contact). We define an odd-one-out task on  $C$ , as described in Section 2. See Figure 1 for three demonstrative examples using the "Inside-Contact" compositional rule. For a given architecture and compositional rule,  $C$ , we train a base model,  $M_{C}$ , such that  $M_{C}$  solves the odd-one-out task to greater than  $90\%$  accuracy<sup>3</sup> (Figure 2, Panel A). We wish to characterize the extent to which  $M_{C}$  exhibits structural compositionality. Does  $M_{C}$  learn only the conjunction (effectively entangling the two subroutines), or does  $M_{C}$  implement  $SR_{1}$  and  $SR_{2}$  in modular subnetworks?
<sup>3</sup>This threshold was selected arbitrarily, but our results do not depend on it. All models end up achieving  $>99\%$  accuracy (See Appendix A).
To investigate this question, we will learn a binary mask  $m_i$  over the weights  $\theta$  of  $M_C$  for each  $SR_i$ , resulting in a subnetwork  $Sub_i$ . Without loss of generality, assume  $Sub_1$  computes  $(+/- \text{Inside})$  and  $Sub_2$  computes  $(+/- \text{Contact})$ , and consider investigating  $Sub_1$ . We can evaluate this subnetwork on two partitions of the training set: (1) Test Target Subroutine - Cases where a model must compute the target subroutine to determine the odd-one-out (e.g., cases where an image exhibits (- Inside, + Contact)) and (2) Test Other Subroutine - Cases where a model must compute the other subroutine to determine the odd-one-out (e.g., cases where the odd-one-out exhibits (+ Inside, - Contact)).
Following prior work (Csordás et al., 2021), we assess structural compositionality based on the subnetwork's performance on these datasets, as well as the base model's performance after ablating the subnetwork<sup>4</sup>. If  $M_C$  exhibits structural compositionality, then  $Sub_1$  should only be able to compute the target subroutine (+/- Inside), and thus it should perform better on Test Target Subroutine than on Test Other Subroutine. If  $M_C$  entangles the subroutines, then  $Sub_1$  will implement both subroutines and will perform equally on both partitions. See Figure 2, Panel B.
To determine modularity, we ablate the  $Sub_{1}$  from the base model and observe the behavior of the resulting model,  $M_{ablate_{1}}$ . If  $M_{C}$  exhibits structural compositionality, we expect the two subroutines to be modular, such that abating  $Sub_{1}$  has more impact on  $M_{ablate_{1}}$ 's ability to compute (+/- Inside) than (+/- Contact). Thus, we would expect  $M_{ablate_{1}}$  to perform better on Test Other Subroutine than Test Target Subroutine. See Figure 2, Panel C. However, if  $M_{C}$  implemented a non-compositional solution, then abating  $Sub_{1}$  should hurt performance on both partitions equally, as the two subroutines are entangled. Thus, performance on both partitions would be approximately equal.
Expected Results: For each model and task, our main results are the differences in performance between Test Target Subroutine and Test Other Subroutine for each subnetwork and ablated model. If a model exhibits structural compositionality, we expect the subnetwork to produce a positive difference in performance (Test Target Subroutine  $>$  Test Other Subroutine), and the corresponding ablated model to produce a negative difference in performance. Otherwise, we expect no differences in performance. See Figure 1 for hypothetical results.
4 Discovering Subnetworks
Consider a frozen model  $M_C(\cdot; w)$  trained on an odd-one-out task defined using the compositional rule  $C$ . Within the weights of this model, we wish to discover a subnetwork that implements  $SR_i^5$ . We further require that the discovered subnetwork should be as small as possible, such that if the model exhibits structural compositionality, it can be ablated with little damage to the remainder of the network. Thus, we wish to learn a binary mask over the weights of a trained neural network while employing  $L_0$  regularization
Most prior work that relies on learning binary masks over network parameters (Cao et al., 2021; Csordás et al., 2021; Zhang et al., 2021; Guo et al., 2021; De Cao et al., 2020, 2022) relies on stochastic approaches, introduced in Louizos et al. (2018). Savarese et al. (2020) introduced continuous sparsification as a deterministic alternative to these stochastic approaches and demonstrated that it achieves superior pruning performance, both in terms of sparsity and subnetwork performance. Thus, we use continuous sparsification to discover subnetworks within our models. See Appendix B for details.
5 Vision Experiments
Tasks: We extend the collection of datasets introduced in Zerroug et al. (2022), generating several tightly controlled datasets that implement compositions of the following subroutines: contact, inside, and number. From these three basic subroutines, we define three compositional rules: Inside-Contact, Number-Contact, and Inside-Number. We will describe the Inside-Contact tasks in detail, as the same principles apply to the other two compositional rules (See Appendix E). This
4This is similar to Csordas et al. (2021)'s  $\mathrm{P_{Specialize}}$  metric.
Over all networks, we only mask weight parameters, leaving bias parameters untouched.
Figure 3: Three Inside-Contact stimuli. The odd-one-out is always the bottom-right image in these examples. (Right) An example from the task used to train the base model. (Middle) An example from the task used to discover the  $+/-$  Inside Subroutine. (Left) An example from the task used to discover the  $+/-$  Contact Subroutine.
task contains four types of images, each containing two shapes. In these images, one shape is either inside and in contact with the other (+ Inside, + Contact), not inside of but in contact with the other (- Inside, + Contact), inside of but not in contact with the other (+ Inside, - Contact), or neither (- Inside, - Contact). An example is defined as a collection of four images, three of which follow a rule and one of which does not. We train our base model to predict the odd-one-out on a task defined by a compositional rule over contact and inside: images of the type (+ Inside, + Contact) follow the rule, and any other image type is considered the odd-one-out. See Figure 3 (Left).
In order to discover a subnetwork that implements each subroutine, we define one odd-one-out task per subroutine. To discover the  $+/-$  Inside Subroutine, we define  $(+ \text{Inside})$  to be rule-following (irrespective of contact) and  $(- \text{Inside})$  to be the odd-one-out. Similarly for the  $+/-$  Contact Subroutine. See Figure 3 (Middle and Right, respectively). The base model has only seen data where  $(+ \text{Inside}, + \text{Contact})$  images are rule-following. In order to align our evaluations with the base model's training data, we create two more datasets that probe each subroutine. For both, all rule-following images are  $(+ \text{Inside}, + \text{Contact})$ . To probe for  $(+/- \text{Inside})$ , the odd-one-out for one dataset is always a  $(- \text{Inside}, + \text{Contact})$  image. This dataset is Test Target Subroutine, with respect to the subnetwork that implements  $(+/- \text{Inside})$ . Similarly, to probe for  $(+/- \text{Contact})$ , the odd-one-out is always a  $(+ \text{Inside}, - \text{Contact})$  image. This dataset is Test Other Subroutine, with respect to the subnetwork that implements  $(+/- \text{Inside})$ .
Methods: Our models consist of a backbone followed by a 2-layer  $\mathbf{MLP}^6$ , which produces embeddings of each of the four images in an example. Following Zerroug et al. (2022) we compute the dot product between each of the four embeddings to produce a pairwise similarity metric. The least similar embedding is predicted to be the "odd-one-out". We use cross-entropy loss over the four images. During mask training, we use  $L_0$  regularization to encourage sparsity. We investigate 3 backbone architectures: Resnet50<sup>7</sup> (He et al., 2016), Wide Resnet50 (Zagoruyko & Komodakis, 2016), and ViT (Dosovitskiy et al., 2020). We perform a hyperparameter search over batch size and learning rate to find settings that allow each model to achieve near-perfect performance. We then train 3 models with different random seeds in order to probe for structural compositionality.<sup>8</sup>
After training our base models,  $M_C$ , we perform a hyperparameter search over continuous sparsification parameters for each subroutine (See Appendix C). One hyperparameter to note is the mask configuration: the layer of the network in which to start masking. After finding the best continuous sparsification parameters, we run the algorithm three times per model, per subroutine, and evaluate on
6 Hidden Size: 2048, Output Size: 128
We replace all BatchNorm layers with InstanceNorm layers. BatchNorm statistics learned during training the base model do not apply to the subnetworks, and because the batch statistics vary across the different data partitions that we evaluate on.
All models are trained using the Adam optimizer (Kingma & Ba, 2014) with early stopping for a maximum of 100 epochs (patience set to 75 epochs). We evaluate using a held-out validation set after every epoch and take the model that minimizes loss on the validation set. We train without dropout, as dropout increases a model's robustness to ablating subnetworks. We train without weight decay, as we will apply  $L_{0}$  regularization during mask training.
Test Target Subroutine and Test Other Subroutine. Finally, for each subnetwork,  $Sub_{i}$ , we create  $M_{ablate_{i}} = M_{C} - Sub_{i}$  and evaluate it on Test Target Subroutine and Test Other Subroutine<sup>9</sup>.
6 Language Experiments
Tasks: We use a subset of the data introduced in Marvin & Linzen (2019) to construct odd-one-out tasks for language data. Analogous to the vision domain, odd-one-out tasks consist of four sentences, three of which follow a rule and one of which does not. We construct rules based on two forms of syntactic agreement: Subject-Verb Agreement and Reflexive Anaphora agreement<sup>10</sup>. In both cases, the agreement takes the form of long-distance coordination of the syntactic number of two words in a sentence. First, consider the subject-verb agreement, the phenomenon that renders the house near the fields is on fire grammatical, and the house near the fields are on fire not grammatical.
Accordingly, we define the following sentence types for Subject-Verb agreement: (\{Singular/Plural\} Subject, \{Singular/Plural\} Verb). Because both (Singular Subject, Singular Verb) and (Plural Subject, Plural Verb) result in a grammatical sentence, we partition the Subject-Verb Agreement dataset into two subsets, one that targets singular sentences and one that targets plural sentences<sup>11</sup>. For the Singular Subject-Verb Agreement dataset, base models are trained on a compositional rule that defines (Singular Subject, Singular Verb) sentences to be rule-following, and (Plural Subject, Singular Verb) and (Singular Subject, Plural Verb) sentences to be the odd-one-out. Thus, an odd-one-out example might look like: the picture by the ministers interest people. All other tasks are constructed analogously to those used in the vision experiments (See Section 5). The Reflexive Anaphora dataset is constructed similarly (See Appendix F).
Methods: The language experiments proceed analogously to the vision experiments. The only difference in the procedure is that we take the representation of the [CLS] token to be the embedding of the sentence and omit the MLP. We study one architecture, BERT-Small (Bhargava et al., 2021; Turc et al., 2019), which is a BERT architecture with 4 hidden layers (Devlin et al., 2018).
7 Results