Title: Break It Down:  Evidence for Structural Compositionality in Neural Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: Deep Learning

Compositionality

Cognitive Science
Abstract: Though modern neural networks have achieved impressive performance in both vision and language tasks, we know little about the functions that they implement. One possibility is that neural networks implicitly break down complex tasks into subroutines, implement modular solutions to these subroutines, and compose them into an overall solution to a task --- a property we term structural compositionality.  Another possibility is that they may simply learn to match new inputs to learned templates, eliding task decomposition entirely. Here, we leverage model pruning techniques to investigate this question in both vision and language across a variety of architectures, tasks, and pretraining regimens. Our results demonstrate that models oftentimes implement solutions to subroutines via modular subnetworks, which can be ablated while maintaining the functionality of other subnetworks. This suggests that neural networks may be able to learn compositionality, obviating the need for specialized symbolic mechanisms.

Break It Down: Evidence for Structural Compositionality in Neural Networks

Abstract

Though modern neural networks have achieved impressive performance in both vision and language tasks, we know little about the functions that they implement. One possibility is that neural networks implicitly break down complex tasks into subroutines, implement modular solutions to these subroutines, and compose them into an overall solution to a task - a property we term structural compositionality . Another possibility is that they may simply learn to match new inputs to learned templates, eliding task decomposition entirely. Here, we leverage model pruning techniques to investigate this question in both vision and language across a variety of architectures, tasks, and pretraining regimens. Our results demonstrate that models often implement solutions to subroutines via modular subnetworks, which can be ablated while maintaining the functionality of other subnetworks. This suggests that neural networks may be able to learn compositionality, obviating the need for specialized symbolic mechanisms.

1 Introduction

Though neural networks have come to dominate most subfields of AI, much remains unknown about the functions that they learn to implement. In particular, there is debate over the role of compositionality . Compositionality has long been touted as a key property of human cognition, enabling humans to exhibit flexible and abstract language processing and visual processing, among other cognitive processes (Marcus, 2003; Piantadosi et al., 2016; Lake et al., 2017; Smolensky et al., 2022). According to common definitions (Quilty-Dunn et al., 2022; Fodor & Lepore, 2002), a representational system is compositional if it implements a set of discrete constituent functions that exhibit some degree of modularity. That is, blue circle is represented compositionally if a system is able to entertain the concept blue independently of circle , and vice-versa.

It is an open question whether neural networks require explicit symbolic mechanisms to implement compositional solutions, or whether they implicitly learn to implement compositional solutions during training. Historically, artificial neural networks have been considered non-compositional systems, instead solving tasks by matching new inputs to learned templates (Marcus, 2003; Quilty-Dunn et al., 2022). Neural networks' apparent lack of compositionality has served as a key point in favor of integrating explicit symbolic mechanisms into contemporary artificial intelligence systems (Andreas et al., 2016; Koh et al., 2020; Ellis et al., 2023; Lake et al., 2017). However, modern neural networks, with no explicit inductive bias towards compositionality, have demonstrated successes on increasingly complex tasks. This raises the question: are these models succeeding by implementing compositional solutions under the hood (Mandelbaum et al., 2022)?

Contributions and Novelty:

We introduce the concept of structural compositionality , which characterizes the extent to which neural networks decompose compositional tasks into subroutines and implement them modularly. We test for structural compositionality in several different models across both language and vision 2 .

We discover that, surprisingly, there is substantial evidence that many models implement subroutines in modular subnetworks, though most do not exhibit perfect task decomposition.

We characterize the effect of unsupervised pretraining on structural compositionality in fine-tuned networks and find that pretraining leads to a more consistently compositional structure in language models.

This study contributes to the emerging body of work on 'mechanistic interpretability' (Olah, 2022; Cammarata et al., 2020; Ganguli et al., 2021; Henighan et al., 2023) which seeks to explain the algorithms that neural networks implicitly implement within their weights. We make use of techniques from model pruning in order to gain insight into these algorithms. While earlier versions of these techniques have been applied to study modularity in a multitask setting (Csordás et al., 2021), our work is novel in that it applies the method to more complex language and vision models, studies more complex compositional tasks, and connects the results to a broader discussion about defining and measuring compositionality within neural networks.

2 Structural Compositionality

Most prior work on compositionality in neural networks has focused on whether they generalize in accordance with the compositional properties of data (Ettinger et al., 2018; Kim & Linzen, 2020; Hupkes et al., 2020). Such work has mostly yielded negative results - i.e., evidence that neural networks fail to generalize compositionally. This work is important for understanding how current models will behave in practice. However, generalization studies alone permit only limited conclusions about how models work.

As discussed above, leading definitions of compositionality are defined in terms of a system's representations, not its behavior. That is, definitions contrast compositional systems (which implement modular constituents) with noncompositional systems (which might, e.g., rely on learned templates). Poor performance on generalization studies does not differentiate these two types of systems, since even a definitionally compositional system might fail at these generalization tasks. For example, a Bayesian network that explicitly represents and composes distinct shape and color properties might nonetheless classify a blue circle as a red circle if it has a low prior for predicting the color blue and a high prior for predicting the color red.

Thus, in this work, we focus on evaluating the extent to which a model's representations are structured compositionally. Consider the task described in Figure 1. In this task, a network learns to select the 'odd-one-out' among four images. Three of them follow a compositional rule (they all contain two shapes, one of which is inside and in contact with the other). One of them breaks this rule. There are at least two ways that a network might learn to solve this type of compositional task. (1) A network might compare new inputs to prototypes or iconic representations of previously-seen inputs, avoiding any decomposition of these prototypes into constituent parts (i.e., it might implement a non-compositional solution ). (2) A network might implicitly break the task down into subroutines, implement solutions to each, and compose these results into a solution (i.e., it might implement a compositional solution ). In this case, the subroutines consist of a (+/- Inside) detector and a (+/-Contact) detector.

If a model trained on this task exhibits structural compositionality , then we would expect to find a subnetwork that implements each subroutine within the parameters of that model. This subnetwork should compute one subroutine, and not the other (Figure 1, Bottom Right; 'Subnetwork'), and it should be modular with respect to the rest of the network - it should be possible to ablate this subnetwork, harming the model's ability to compute one subroutine while leaving the other subroutine largely intact (Figure 1, Bottom Right; 'Ablation'). However, if a model does not exhibit structural compositionality, then it has only learned the conjunction of the subroutines rather than

+ Contact

+ Inside

Class A

Class B

Contact (+)

&

Inside (+)

Contact (+)

&

Inside (-)

Contact (-)

&

Inside (-)

Contact (-)

&

Inside (+)

Task

Non-Compositional

Solution

Compositional

Solution

"Inside"

Subnetwork

Subnetwork

Ablation

0

- Contact

- Inside

+ Inside

- Inside

+ Contact

- Contact

"Inside"

Subnetwork

Subnetwork

Ablation

0

their composition . It should not be possible to find a subnetwork that implements one subroutine and not the other, and ablating one subnetwork should hurt accuracy on both subroutines equally (Figure 1, Bottom Center). This definition is related to prior work on modularity in neural networks (Csordás et al., 2021; Hod et al., 2022), but here we specifically focus on modular representations of compositional tasks.

3 Experimental Design

3.1 Preliminaries

Here we define terms used in the rest of the paper. Subroutine: A binary rule. The i th subroutine is denoted SR i . Compositional Rule: A binary rule that maps input to output according to C = SR 1 & SR 2 , where SR i is a subroutine. Compositional rules are denoted C . Base Model: A model that is trained to solve a task defined by a compositional rule. Denoted M C . Subnetwork: A subset of the parameters of a base model, which implements one subroutine. The subnetwork that implements SR i is denoted Sub i . This is implemented as a binary mask, m i , over the parameters of the base model, θ , such that Sub i = M C ; θ ⊙ m i , where ⊙ refers to elementwise multiplication.

A) Train Base Model

Train/Test

Inside & Contact

B) Find Subroutine

Subnetwork

Train

Inside vs. Outside

Test Target SR

Test Other SR

Expect Success

Expect Failure

C) Ablation

Ablate

Test Target SR

Test Other SR

Expect Failure

Expect Success

Ablated Model: The complement set of parameters of a particular subnetwork. After ablating Sub i , we denote the ablated model M ablate i .

3.2 Experimental Logic

Consider a compositional rule, C , such as the 'Inside-Contact' rule described in Figures 1 and 2. The rule is composed of two subroutines, SR 1 (+/- Inside) and SR 2 (+/- Contact) . We define an odd-one-out task on C , as described in Section 2. See Figure 1 for three demonstrative examples using the 'Inside-Contact' compositional rule. For a given architecture and compositional rule, C , we train a base model, M C , such that M C solves the odd-one-out task to greater than 90% accuracy 3 (Figure 2, Panel A). We wish to characterize the extent to which M C exhibits structural compositionality. Does M C learn only the conjunction (effectively entangling the two subroutines), or does M C implement SR 1 and SR 2 in modular subnetworks?

To investigate this question, we will learn a binary mask m i over the weights θ of M C for each SR i , resulting in a subnetwork Sub i . Without loss of generality, assume Sub 1 computes (+/- Inside) and Sub 2 computes (+/- Contact) , and consider investigating Sub 1 . We can evaluate this subnetwork on two partitions of the training set: (1) Test Target Subroutine - Cases where a model must compute the target subroutine to determine the odd-one-out (e.g., cases where an image exhibits (- Inside, + Contact) ) and (2) Test Other Subroutine - Cases where a model must compute the other subroutine to determine the odd-one-out (e.g., cases where the odd-one-out exhibits (+ Inside, - Contact) ).

Following prior work (Csordás et al., 2021), we assess structural compositionality based on the subnetwork's performance on these datasets, as well as the base model's performance after ablating the subnetwork 4 . If M C exhibits structural compositionality, then Sub 1 should only be able to compute the target subroutine (+/- Inside) , and thus it should perform better on Test Target Subroutine than on Test Other Subroutine . If M C entangles the subroutines, then Sub 1 will implement both subroutines and will perform equally on both partitions. See Figure 2, Panel B.

To determine modularity, we ablate the Sub 1 from the base model and observe the behavior of the resulting model, M ablate 1 . If M C exhibits structural compositionality, we expect the two subroutines to be modular, such that ablating Sub 1 has more impact on M ablate 1 's ability to compute (+/-Inside) than (+/- Contact) . Thus, we would expect M ablate 1 to perform better on Test Other Subroutine than Test Target Subroutine . See Figure 2, Panel C. However, if M C implemented a non-compositional solution, then ablating Sub 1 should hurt performance on both partitions equally, as the two subroutines are entangled. Thus, performance on both partitions would be approximately equal.

Expected Results: For each model and task, our main results are the differences in performance between Test Target Subroutine and Test Other Subroutine for each subnetwork and ablated model. If a model exhibits structural compositionality, we expect the subnetwork to produce a positive difference in performance ( Test Target Subroutine > Test Other Subroutine ), and the corresponding ablated model to produce a negative difference in performance. Otherwise, we expect no differences in performance. See Figure 1 for hypothetical results.

4 Discovering Subnetworks

Consider a frozen model M C ( · ; w ) trained on an odd-one-out task defined using the compositional rule C . Within the weights of this model, we wish to discover a subnetwork that implements SR i 5 . We further require that the discovered subnetwork should be as small as possible, such that if the model exhibits structural compositionality, it can be ablated with little damage to the remainder of the network. Thus, we wish to learn a binary mask over the weights of a trained neural network while employing L 0 regularization

Most prior work that relies on learning binary masks over network parameters (Cao et al., 2021; Csordás et al., 2021; Zhang et al., 2021; Guo et al., 2021; De Cao et al., 2020, 2022) relies on stochastic approaches, introduced in Louizos et al. (2018). Savarese et al. (2020) introduced continuous sparsification as a deterministic alternative to these stochastic approaches and demonstrated that it achieves superior pruning performance, both in terms of sparsity and subnetwork performance. Thus, we use continuous sparsification to discover subnetworks within our models. See Appendix B for details.

5 Vision Experiments

Tasks: We extend the collection of datasets introduced in Zerroug et al. (2022), generating several tightly controlled datasets that implement compositions of the following subroutines: contact , inside , and number . From these three basic subroutines, we define three compositional rules: InsideContact , Number-Contact , and Inside-Number . We will describe the Inside-Contact tasks in detail, as the same principles apply to the other two compositional rules (See Appendix E). This

task contains four types of images, each containing two shapes. In these images, one shape is either inside and in contact with the other (+ Inside, + Contact) , not inside of but in contact with the other (- Inside, + Contact) , inside of but not in contact with the other (+ Inside, - Contact) , or neither (- Inside, - Contact) . An example is defined as a collection of four images, three of which follow a rule and one of which does not. We train our base model to predict the odd-one-out on a task defined by a compositional rule over contact and inside: images of the type (+ Inside, + Contact) follow the rule, and any other image type is considered the odd-one-out. See Figure 3 (Left).

In order to discover a subnetwork that implements each subroutine, we define one odd-one-out task per subroutine. To discover the +/- Inside Subroutine, we define (+ Inside) to be rule-following (irrespective of contact) and (- Inside) to be the odd-one-out. Similarly for the +/- Contact Subroutine. See Figure 3 (Middle and Right, respectively). The base model has only seen data where (+ Inside, + Contact) images are rule-following. In order to align our evaluations with the base model's training data, we create two more datasets that probe each subroutine. For both, all rule-following images are (+ Inside, + Contact) . To probe for (+/- Inside) , the odd-one-out for one dataset is always a (- Inside, + Contact) image. This dataset is Test Target Subroutine , with respect to the subnetwork that implements (+/- Inside) . Similarly, to probe for (+/- Contact) , the odd-one-out is always a (+ Inside, - Contact) image. This dataset is Test Other Subroutine , with respect to the subnetwork that implements (+/- Inside) .

Methods: Our models consist of a backbone followed by a 2-layer MLP 6 , which produces embeddings of each of the four images in an example. Following Zerroug et al. (2022) we compute the dot product between each of the four embeddings to produce a pairwise similarity metric. The least similar embedding is predicted to be the 'odd-one-out'. We use cross-entropy loss over the four images. During mask training, we use L 0 regularization to encourage sparsity. We investigate 3 backbone architectures: Resnet50 7 (He et al., 2016), Wide Resnet50 (Zagoruyko & Komodakis, 2016), and ViT (Dosovitskiy et al., 2020). We perform a hyperparameter search over batch size and learning rate to find settings that allow each model to achieve near-perfect performance. We then train 3 models with different random seeds in order to probe for structural compositionality. 8

After training our base models, M C , we perform a hyperparameter search over continuous sparsification parameters for each subroutine (See Appendix C). One hyperparameter to note is the mask configuration: the layer of the network in which to start masking. After finding the best continuous sparsification parameters, we run the algorithm three times per model, per subroutine, and evaluate on

Test Target Subroutine and Test Other Subroutine . Finally, for each subnetwork, Sub i , we create M ablate i = M C -Sub i and evaluate it on Test Target Subroutine and Test Other Subroutine 9 .

6 Language Experiments

Tasks: We use a subset of the data introduced in Marvin & Linzen (2019) to construct odd-one-out tasks for language data. Analogous to the vision domain, odd-one-out tasks consist of four sentences, three of which follow a rule and one of which does not. We construct rules based on two forms of syntactic agreement: Subject-Verb Agreement and Reflexive Anaphora agreement 10 . In both cases, the agreement takes the form of long-distance coordination of the syntactic number of two words in a sentence. First, consider the subject-verb agreement, the phenomenon that renders the house near the fields is on fire grammatical, and the house near the fields are on fire not grammatical.

Accordingly, we define the following sentence types for Subject-Verb agreement: ({Singular/Plural} Subject, {Singular/Plural} Verb) . Because both (Singular Subject, Singular Verb) and (Plural Subject, Plural Verb) result in a grammatical sentence, we partition the Subject-Verb Agreement dataset into two subsets, one that targets singular sentences and one that targets plural sentences 11 . For the Singular Subject-Verb Agreement dataset, base models are trained on a compositional rule that defines (Singular Subject, Singular Verb) sentences to be rule-following, and (Plural Subject, Singular Verb) and (Singular Subject, Plural Verb) sentences to be the odd-one-out. Thus, an odd-one-out example might look like: the picture by the ministers interest people. All other tasks are constructed analogously to those used in the vision experiments (See Section 5). The Reflexive Anaphora dataset is constructed simlarly (See Appendix F).

Methods: The language experiments proceed analogously to the vision experiments. The only difference in the procedure is that we take the representation of the [CLS] token to be the embedding of the sentence and omit the MLP. We study one architecture, BERT-Small (Bhargava et al., 2021; Turc et al., 2019), which is a BERT architecture with 4 hidden layers (Devlin et al., 2018).

7 Results

Most base models perform near perfectly, with the exception of ViT, which failed to achieve > 90% performance on any of the tasks with any configuration of hyperparameters 12 . Thus, we exclude ViT from all subsequent analyses. See Appendix D for these results. If the base models exhibit structural compositionality, we expect subnetworks to achieve greater accuracy 13 on Test Target Subroutine than on Test Other Subroutine (difference in accuracies > 0 ). After ablating subnetworks, we expect the ablated model to achieve greater accuracy on Test Other than Test Target (difference in accuracies < 0 ). Across the board, we see the expected pattern. Subnetwork and ablated accuracy differences for Resnet50 and BERT are visualized in Figure 4 (Subnetwork in Blue, Ablated Models in Red). See Appendix A for Wide Resnet50 results, which largely reproduce the results using Resnet50.

For some architecture/task combinations, the pattern of ablated model results is statistically significantly in favor of structural compositionality. See Figure 4 (C, D), where all base models seem to implement both subroutines in a modular fashion. We analyze the layerwise overlap between subnetworks found within one of these models in Appendix K. This analysis shows that there is relatively high overlap between subnetworks for the same subroutine, and low overlap between subroutines. Other results are mixed, such as those found in Resnet50 models trained on Number-Contact . Here,

Resnet50 & BERT-Small (Test Target - Test Other)

A

B

C

D

E

F

G

H

I

J

***

**

***

***

***

***

**

***

***

***

***

***

***

***

***

**

Inside-Contact

Inside-Number

Number-Contact

1.00

1.00

1.00

1.00

1.00

1.00

0.75

0.75 -

0.75

0.75

0.75

0.75

0.50

0.50

0.50

0.50

0.50

0.50

：

■

0.25

0.25

0.25

0.25

0.25

0.25

Inside

Contact

Inside

Number

Contact

Number

0.00

0.00

0.00

0.00

0.00

0.00

0.25

0.25

0.25

0.25

0.25

0.25

0.50

0.50 -

0.50 -

os'0-

0.50

0.50

0.75

·

·

Seed 2

0.75

0.75

0.75

0.75

0.75

■

Seed 3

1.00

Subnetwork Ablation

1.00

SubnetworkAblation

1.00

SubnetworkAblation

1.00

SubnetworkAblation

1.00

SubnetworkAblation

1.00

Subnetwork Ablation

SV Agreement

1.00

1.00

1.00

1.00

·

0.75

Model 2

Model 1

0.75

0.75

.

0.75

.

0.50

PI.

Sing.

0S'0

0.50

0S'0

：

0.25

Antecedent

0.25

0.25

0.25

Pronoun

Subject

Verb

0.00

0.00

0.00

0.00

0.25

0.25

0.25

0.25

0.50

0.50

0.50

FoS'0-

0.75

0.75

0.75

0.75

1.00

Subnetwork

Ablation

1.00

Subnetwork

Ablation

1.00

Subnetwork

Ablation

1.00

Subnetwork

Ablation

we see strong evidence of structural compositionality in Figure 4 (E), but little evidence for it in Figure 4 (F). In this case, it appears that the network is implementing the (+/- Contact) subroutine in a small, modular subnetwork, whereas the (+/- Number) subroutine is implemented more diffusely. We perform control experiments using randomly initialized models in Appendix I, which show that the pattern of results in (A), (B) and (F) are not significantly different from a random model, while all other results are significantly different.

8 Effect of Pretraining on Structural Compositionality

We compare structural compositionality in models trained from scratch to those that were initialized with pretrained weights. For Resnet50, we pretrain a model on our data using SimCLR (See Appendix G for details).

For BERT-Small, we use the pretrained weights provided by Turc et al. (2019). We rerun the same procedure described in Sections 5 and 6. See Appendix A for each base model's performance. Figure 5 contains the results of the language experiments. Across all language tasks, the ablation results indicate that models initialized with pretrained weights more reliably produce modular subnetworks than randomly initialized models. Results on vision tasks are found in Appendix A and do not suggest any benefit of pretraining.

(d)

E“(qns (d)

Subnetwork Performance: Randomly Initialized Models (Language)

Ablation 1

Ablation 2

Mask

Ablation 3

Subnetwork 1

0.6

Subnetwork 2

- 0.4

Subnetwork3

()

(P) Ant. 3

[ old (d)

Eold (d)

['qA (d)

乙qΛ(d)

Eq^(d)

(S) Ant. 1

(S) Pro. 1

(S) Pro. 3

(S) Vb. 1

(S) Vb. 2

(S) Vb.3

0.2

Performance

Subnetwork Performance:Pretrained Models (Language)

(P) Subj. 1

(P) Subj. 2

(P) Subj. 3

(S) Ant. 2

(S) Ant. 3

(S) Subj. 1

(S) Subj. 2

(S) Subj. 3

一

0.0

Ablation 1

-0.2

Ablation2

-0.4

Mask

Ablation3

Subnetwork1

-0.6

Subnetwork2

Subnetwork3

(P) Ant. 1

[ old (d)

Z od (d)

old (d)

I “(qns (d)

[q^(d)

乙qΛ(d)

E“qΛ(d)

(S) Pro. 1

(S) Subj. 1

(S) Vb.1

(S) Vb. 2

(S) Vb.3

(P) Ant. 2

(P) Ant. 3

(P) Subj. 2

Model

(S) Subj. 2

(S) Ant. 1

ε“(qns (s)

9 Related Work

This work casts a new lens on the study of compositionality in neural networks. Most prior work has focused on compositional generalization of standard neural models (Yu & Ettinger, 2020; Kim & Linzen, 2020; Kim et al., 2022; Dankers et al., 2022), though some has attempted to induce an inductive bias toward compositional generalization from data (Lake, 2019; Qiu et al., 2021; Zhu et al., 2021). Recent efforts have attempted to attribute causality to specific components of neural networks' internal representations (Ravfogel et al., 2020; Bau et al., 2019; Wu et al., 2022; Tucker et al., 2021; Lovering & Pavlick, 2022; Elazar et al., 2021; Cao et al., 2021). In contrast to these earlier studies, our method does not require any assumptions about where in the network the subroutine is implemented and does not rely on auxiliary classifiers, which can confound the causal interpretation. Finally, Dziri et al. (2023) performs extensive behavioral studies characterizing the ability of autoregressive language models to solve compositional tasks, and finds them lacking. In contrast, our work studies the structure of internal representations and sets aside problems that might be specific to autoregressive training objectives.

More directly related to the present study is the burgeoning field of mechanistic interpretability , which aims to reverse engineer neural networks in order to better understand how they function (Olah, 2022; Cammarata et al., 2020; Black et al., 2022; Henighan et al., 2023; Ganguli et al., 2021; Merrill et al., 2023). Notably, Chughtai et al. (2023) recovers universal mechanisms for performing group-theoretic compositions. Though group-theoretic composition is different from the compositionality discussed in the present article, this work sheds light on generic strategies that models may use to solve tasks that require symbolically combining multiple input features.

Some recent work has attempted to characterize modularity within particular neural networks (Hod et al., 2022). Csordás et al. (2021) also analyzes modularity within neural networks using learned binary masks. Their study finds evidence of modular subnetworks within a multitask network: Within a network trained to perform both addition and multiplication, different subnetworks arise for each operation. Csordás et al. (2021) also investigates whether the subnetworks are reused in a variety of contexts, and find that they are not. In particular, they demonstrate that subnetworks that solve particular partitions of compositional datasets (SCAN (Lake & Baroni, 2018) and the Mathematics

(S) Pro. 2

Dataset (Saxton et al., 2018)), oftentimes do not generalize to other partitions. From this, they conclude that neural networks do not flexibly combine subroutines in a manner that would enable full compositional generalization. However, their work did not attempt to uncover subnetworks that implement specific compositional subroutines within these compositional tasks. For example, they did not attempt to find a subnetwork that implements a general "repeat" operation for SCAN, transforming "jump twice" into "JUMP JUMP". Our work finds such compositional subroutines in language and vision tasks, and localizes them into modular subnetworks. This finding extends Csordás et al. (2021)'s result on a simple multitask setting to more complex compositional vision and language settings, and probes for subroutines that represent intermediate subroutines in a compositional task (i.e. "inside" is a subroutine when computing "Inside-Contact").

10 Discussion

Across a variety of architectures, tasks, and training regimens, we demonstrated that models often exhibit structural compositionality. Without any explicit encouragement to do so, neural networks appear to decompose tasks into subroutines and implement solutions to (at least some of) these subroutines in modular subnetworks. Furthermore, we demonstrate that self-supervised pretraining can lead to more consistent structural compositionality, at least in the domain of language. These results bear on the longstanding debate over the need for explicit symbolic mechanisms in AI systems. Much work is focusing on integrating symbolic and neural systems (Ellis et al., 2023; Nye et al., 2020). However, our results suggest that some simple pseudo-symbolic computations might be learned directly from data using standard gradient-based optimization techniques.

We view our approach as a tool for understanding when and how compositionality arises in neural networks, and plan to further investigate the conditions that encourage structural compositionality. One promising direction would be to investigate the relationship between structural compositionality and recent theoretical work on compositionality and sparse neural networks (Mhaskar & Poggio, 2016; Poggio, 2022). Specifically, this theoretical work suggests that neural networks optimized to solve compositional tasks naturally implement sparse solutions. This may serve as a starting point for developing a formal theory of structural compositionality in neural networks. Another direction might be to investigate the structural compositionality of networks trained using iterated learning procedures (Ren et al., 2019; Vani et al., 2020). Iterated learning simulates the cultural evolution of language by jointly training two communicating agents (Kirby et al., 2008). Prior work has demonstrated that iterated learning paradigms give rise to simple compositional languages. Quantifying the relationship between structural compositionality within the agents and the compositionality of the language that they produce would be an exciting avenue for understanding the relationship between representation and behavior.

One limit of our technical approach is that one must specify which subroutines to look for in advance. Future work might address this by discovering functional subnetworks using unsupervised methods. Additionally, our approach requires us to use causal ablations and control models to properly interpret our results. Future work might try to uncover subnetworks that are necessarily causally implicated in model behavior. Finally, future work must clarify the relationship between structural compositionality and compositional generalization.

Acknowledgments and Disclosure of Funding

The authors thank the members of the Language Understanding and Representation Lab and Serre Lab for their valuable feedback on this project and Rachel Goepner for proofreading the manuscript. This project was supported by ONR grant #N00014-19-1-2029. The computing hardware was supported in part by NIH Office of the Director grant #S10OD025181 via the Center for Computation and Visualization (CCV) at Brown University.

References

Andreas, J., Rohrbach, M., Darrell, T., and Klein, D. Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 39-48, 2016.

Bau, A., Belinkov, Y., Sajjad, H., Durrani, N., Dalvi, F., and Glass, J. Identifying and controlling important neurons in neural machine translation. In International Conference on Learning Representations , 2019. URL https://openreview.net/forum?id=H1z-PsR5KX .

Bhargava, P., Drozd, A., and Rogers, A. Generalization in nli: Ways (not) to go beyond simple heuristics. In Proceedings of the Second Workshop on Insights from Negative Results in NLP , pp. 125-135, 2021.

Black, S., Sharkey, L., Grinsztajn, L., Winsor, E., Braun, D., Merizian, J., Parker, K., Guevara, C. R., Millidge, B., Alfour, G., et al. Interpreting neural networks through the polytope lens. arXiv preprint arXiv:2211.12312 , 2022.

Cammarata, N., Carter, S., Goh, G., Olah, C., Petrov, M., Schubert, L., Voss, C., Egan, B., and Lim, S. K. Thread: Circuits. Distill , 5(3):e24, 2020.

Cao, S., Sanh, V., and Rush, A. M. Low-complexity probing via finding subnetworks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 960-966, 2021.

Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pp. 1597-1607. PMLR, 2020.

Chughtai, B., Chan, L., and Nanda, N. A toy model of universality: Reverse engineering how networks learn group operations. arXiv preprint arXiv:2302.03025 , 2023.

Csordás, R., van Steenkiste, S., and Schmidhuber, J. Are neural nets modular? inspecting functional modularity through differentiable weight masks. In International Conference on Learning Representations , 2021.

Dankers, V., Bruni, E., and Hupkes, D. The paradox of the compositionality of natural language: A neural machine translation case study. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 4154-4175, 2022.

De Cao, N., Schlichtkrull, M. S., Aziz, W., and Titov, I. How do decisions emerge across layers in neural models? interpretation with differentiable masking. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pp. 3243-3255, 2020.

De Cao, N., Schmid, L., Hupkes, D., and Titov, I. Sparse interventions in language models with differentiable masking. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP , pp. 16-27, 2022.

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2020.

Dziri, N., Lu, X., Sclar, M., Li, X. L., Jian, L., Lin, B. Y ., West, P., Bhagavatula, C., Bras, R. L., Hwang, J. D., et al. Faith and fate: Limits of transformers on compositionality. arXiv preprint arXiv:2305.18654 , 2023.

Elazar, Y., Ravfogel, S., Jacovi, A., and Goldberg, Y. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics , 9: 160-175, 2021.

Ellis, K., Wong, L., Nye, M., Sable-Meyer, M., Cary, L., Anaya Pozo, L., Hewitt, L., Solar-Lezama, A., and Tenenbaum, J. B. Dreamcoder: growing generalizable, interpretable knowledge with wake-sleep bayesian program learning. Philosophical Transactions of the Royal Society A , 381 (2251):20220050, 2023.

Ettinger, A., Elgohary, A., Phillips, C., and Resnik, P. Assessing composition in sentence vector representations. In Proceedings of the 27th International Conference on Computational Linguistics , pp. 1790-1801, 2018.

Fodor, J. A. and Lepore, E. The compositionality papers . Oxford University Press, 2002.

Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olan, C. A mathematical framework for transformer circuits. Transformer Circuits Thread , 2021.

Guo, D., Rush, A. M., and Kim, Y. Parameter-efficient transfer learning with diff pruning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pp. 4884-4896, 2021.

He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 770-778, 2016.

Henighan, T., Carter, S., Humne, T., Elhage, N., Lasenby, R., Fort, S., Schiefer, N., and Olah, C. Superposition, memorization, and double descent. Transformer Circuits Thread , 2023.

Hod, S., Filan, D., Casper, S., Critch, A., and Russell, S. Quantifying local specialization in deep neural networks. arXiv e-prints , pp. arXiv-2110, 2022.

Hupkes, D., Dankers, V., Mul, M., and Bruni, E. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research , 67:757-795, 2020.

Kim, N. and Linzen, T. Cogs: A compositional generalization challenge based on semantic interpretation. In Empirical Methods in Natural Language Processing , 2020.

Kim, N., Linzen, T., and Smolensky, P. Uncontrolled lexical exposure leads to overestimation of compositional generalization in pretrained models. arXiv preprint arXiv:2212.10769 , 2022.

Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Kirby, S., Cornish, H., and Smith, K. Cumulative cultural evolution in the laboratory: An experimental approach to the origins of structure in human language. Proceedings of the National Academy of Sciences , 105(31):10681-10686, 2008.

Koh, P. W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., and Liang, P. Concept bottleneck models. In International Conference on Machine Learning , pp. 5338-5348. PMLR, 2020.

Lake, B. and Baroni, M. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In International conference on machine learning , pp. 2873-2882. PMLR, 2018.

Lake, B. M. Compositional generalization through meta sequence-to-sequence learning. Advances in neural information processing systems , 32, 2019.

Lake, B. M., Ullman, T. D., Tenenbaum, J. B., and Gershman, S. J. Building machines that learn and think like people. Behavioral and brain sciences , 40, 2017.

Lippe, P. Tutorial 17: Self-supervised contrastive learning with simclr. https: //github.com/phlippe/uvadlc_notebooks/blob/master/docs/tutorial_notebooks/ tutorial17/SimCLR.ipynb , 2022.

Louizos, C., Welling, M., and Kingma, D. P. Learning sparse neural networks through l_0 regularization. In International Conference on Learning Representations , 2018.

Lovering, C. and Pavlick, E. Unit testing for concepts in neural networks. Transactions of the Association for Computational Linguistics , 10:1193-1208, 2022.

Mandelbaum, E., Dunham, Y., Feiman, R., Firestone, C., Green, E., Harris, D., Kibbe, M. M., Kurdi, B., Mylopoulos, M., Shepherd, J., et al. Problems and mysteries of the many languages of thought. Cognitive Science , 46(12):e13225, 2022.

Marcus, G. F. The algebraic mind: Integrating connectionism and cognitive science . MIT press, 2003.

Marvin, R. and Linzen, T. Targeted syntactic evaluation of language models. Proceedings of the Society for Computation in Linguistics (SCiL) , pp. 373-374, 2019.

Merrill, W., Tsilivis, N., and Shukla, A. A tale of two circuits: Grokking as competition of sparse and dense subnetworks. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models , 2023.

Mhaskar, H. N. and Poggio, T. Deep vs. shallow networks: An approximation theory perspective. Analysis and Applications , 14(06):829-848, 2016.

Nye, M., Solar-Lezama, A., Tenenbaum, J., and Lake, B. M. Learning compositional rules via neural program synthesis. Advances in Neural Information Processing Systems , 33:10832-10842, 2020.

Olah, C. Mechanistic interpretability, variables, and the importance of interpretable bases. Transformer Circuits Thread , 2022.

Piantadosi, S. T., Tenenbaum, J. B., and Goodman, N. D. The logical primitives of thought: Empirical foundations for compositional cognitive models. Psychological review , 123(4):392, 2016.

Poggio, T. How deep sparse networks avoid the curse of dimensionality: Efficiently computable functions are compositionally sparse. CBMM Memo , 10:2022, 2022.

Qiu, L., Shaw, P., Pasupat, P., Nowak, P. K., Linzen, T., Sha, F., and Toutanova, K. Improving compositional generalization with latent structure and data augmentation. arXiv preprint arXiv:2112.07610 , 2021.

Quilty-Dunn, J., Porot, N., and Mandelbaum, E. The best game in town: The re-emergence of the language of thought hypothesis across the cognitive sciences. Behavioral and Brain Sciences , pp. 1-55, 2022.

Ramanujan, V., Wortsman, M., Kembhavi, A., Farhadi, A., and Rastegari, M. What's hidden in a randomly weighted neural network? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 11893-11902, 2020.

Ravfogel, S., Elazar, Y., Gonen, H., Twiton, M., and Goldberg, Y. Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pp. 7237-7256, 2020.

Ren, Y., Guo, S., Labeau, M., Cohen, S. B., and Kirby, S. Compositional languages emerge in a neural iterated learning model. In International Conference on Learning Representations , 2019.

Savarese, P., Silva, H., and Maire, M. Winning the lottery with continuous sparsification. Advances in Neural Information Processing Systems , 33:11380-11390, 2020.

Saxton, D., Grefenstette, E., Hill, F., and Kohli, P. Analysing mathematical reasoning abilities of neural models. In International Conference on Learning Representations , 2018.

Smolensky, P., McCoy, R., Fernandez, R., Goldrick, M., and Gao, J. Neurocompositional computing: From the central paradox of cognition to a new generation of ai systems. AI Magazine , 43(3): 308-322, 2022.

Tucker, M., Qian, P., and Levy, R. What if this modified that? syntactic interventions with counterfactual embeddings. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pp. 862-875, 2021.

Turc, I., Chang, M., Lee, K., and Toutanova, K. Well-read students learn better: The impact of student initialization on knowledge distillation. CoRR , abs/1908.08962, 2019. URL http: //arxiv.org/abs/1908.08962 .

Vani, A., Schwarzer, M., Lu, Y., Dhekane, E., and Courville, A. Iterated learning for emergent systematicity in vqa. In International Conference on Learning Representations , 2020.

Wortsman, M., Ramanujan, V., Liu, R., Kembhavi, A., Rastegari, M., Yosinski, J., and Farhadi, A. Supermasks in superposition. Advances in Neural Information Processing Systems , 33:1517315184, 2020.

Wu, Z., Geiger, A., Rozner, J., Kreiss, E., Lu, H., Icard, T., Potts, C., and Goodman, N. Causal distillation for language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 4288-4295, 2022.

Yu, L. and Ettinger, A. Assessing phrasal representation and composition in transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pp. 4896-4907, 2020.

Zagoruyko, S. and Komodakis, N. Wide residual networks. In British Machine Vision Conference 2016 . British Machine Vision Association, 2016.

Zerroug, A., Vaishnav, M., Colin, J., Musslick, S., and Serre, T. A benchmark for compositional visual reasoning. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022.

Zhang, D., Ahuja, K., Xu, Y., Wang, Y., and Courville, A. Can subnetwork structure be the key to out-of-distribution generalization? In International Conference on Machine Learning , pp. 12356-12367. PMLR, 2021.

Zhou, H., Lan, J., Liu, R., and Yosinski, J. Deconstructing lottery tickets: Zeros, signs, and the supermask. Advances in neural information processing systems , 32, 2019.

Zhu, W., Shaw, P., Linzen, T., and Sha, F. Learning to generalize compositionally by transferring across semantic parsing tasks. arXiv preprint arXiv:2111.05013 , 2021.

A Full Results

In this section, we provide the following results:

Base Model Performance on Compositional Tasks: Table 2

Pretrained + Finetuned Model Performance on Compositional Tasks: Table 3

Wide Resnet50 Subnetwork Results: Figure 6

Vision Pretraining vs. Random Initialization Heatmap: Figure 7

Absolute Accuracy for every subnetwork and ablated model on each task, for each model: Figures 8-12.

See Table 2 for the performance of all base models on each compositional task. See Table 3 for the performance of all pretrained base models on each compositional task. See Figure 6 for subnetwork and ablation results on Wide Resnet50. See Figure 7 for Vision Model pretraining results.

A.1 Statistical Analysis of Main Results

In order to assess the significance of our main results, we fit a generalized linear model (GLM) with robust clustered standard errors for each combination of model architecture, compositional task, and subroutine. This GLM includes a dummy variable indicating whether the results are from a subnetwork or an ablated model, and it clusters observations by base model. For language experiments, we collapse across singular and plural instances of the same subroutine. The intercept term in this model assesses whether the performance of the discovered subnetworks are significantly different from 0. From this model, we can also perform a linear hypothesis test to assess whether ablated model performance is significantly different from 0. Table 1 provides the relevant statistics. Across the board, we see that there is always a significant difference between subnetwork performance and 0, and often a significant difference between ablated model performance and 0, even with a small sample size.

Wide Resnet50 (Test Target - Test Other)

***

***

***

***

***

***

*

.

***

Inside-Contact

Inside-Number

Number-Contact

1.00

1.00

1.00

1.00

1.00

1.00

0.75

0.75

0.75

0.75

0.75

0.75

0.50

0.50

0.50

FOS'O

0.50

0.50

0.25

0.25

0.25

0.25

0.25

0.25

Inside

Contact

Inside

Number

Contac

Number

0.00

0.00

0.00

0.00

0.00

0.00

0.25

0.25

0.25

0.25

0.25

0.25

0.50

0.50

0.50

0.50 -

0.50

0.50

0.75

0.75

0.75

0.75

0.75

0.75

1.00

Subnetwork Ablation

1.00

Subnetwork Ablation

1.00

Subnetwork Ablation

1.00

Subnetwork Ablation

1.00

Subnetwork Ablation

1.00

Subnetwork Ablation

SubnetworkPerformance:Randomly Initialized Models(Vision)

Ablation 1

Ablation 2

Mask

Ablation 3

0.6

Subnetwork 1

- 0.4

Subnetwork 2

Subnetwork 3

0.2

(N))

0.0

Contact1

)Contact 2

(IC)Inside 1

(IC)Inside 2

1) Inside2

(IN)Inside3

1)Number

Contact1

)Contact2

)Contact3

(SN)

(IN)Number

Model

0.2

SubnetworkPerformance:PretrainedModels(Vision)

Ablation 1

-0.4

Ablation 2

-0.6

Mask

Ablation 3

Subnetwork1

Subnetwork 2

Subnetwork3

)Contact1

(IC)

(IC)Contact2

(IC)Inside 1

(IC)Inside 2

(IN)Inside 2

(IN)Inside 3

(IN) Number1

Model

(IN)Number2

(IN) Number 3

(NC)Contact1

C)Contact2

(NC) Contact 3

(NC) Number 1

(NC) Number 2

Number

(NC)

Resnet50 Subnetworks:Inside-Contact

Resnet50 Subnetworks: Inside-Number

Resnet50 Subnetworks:Number-Contact

1.0

1.0

1.0

1.0 -

1.0

1.0

0.8

0.8

0.8

80

0.8 -

0.8

0.6

0.6-

0.4

0.4

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0.2

0.2

0.0

Target

Other

0°0

Target

Other

0.0

Target

Other

0°0

Target

Other

0°0

Target

Other

0.0

Target

Other

Resnet50 Ablated Networks: Inside-Contact

Resnet50 Ablated Networks: Inside-Number

Resnet50 Ablated Networks: Number-Contact

1.0

1.0

1.0

1.0 -

1.0

1.0

0.8

0.8

0.8

0.8

80

0.8

Contact

0.6

0.6

Contact

0.6

Number

0.6

0.4

0.4

0.4

0.4

0.4

0.4

：

0.2

0.2

0.2

0.2

0.2

0.2

0°0

Target

Other

0.0

Target

Other

0.0

Target

Other

0°0

Target

Other

0°0

Target

Other

0.0

Target

Other

B Continuous Sparsification: Extended Discussion

Continuous sparsification attempts to optimize a binary mask that minmizes the following loss function:

Performance

WideResnet50 Subnetworks:Inside-Contact

WideResnet50 Subnetworks:Inside-Number

WideResnet50 Subnetworks:Number-Contact

1.0

1.0

1.0

1.0

1.0

1.0

0.8

0.8

0.8

80

0.8

0.8

Contact

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0.2 -

0.2

0.0

Target

Other

0°0

Target

Other

0.0

Target

Other

0°0

Target

Other

0°0

Target

Other

0.0

Target

Other

WideResnet50 Ablated Networks:Inside-Contact

WideResnet50 Ablated Networks: Inside-Number

WideResnet50 Ablated Networks:Number-Contact

1.0

1.0

1.0

1.0

1.0

1.0

0.8

80

0.8

8'0

0.8

0.8

Contact

0.6

0.6

0.6

Number

0.6

0.4

0.4

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0.2

0.2

0.0

Target

Other

0°0

Target

Other

0.0

Target

Other

0°0

Target

Other

0°0

Target

Other

0.0

Target

Other

Resnet50 +SimCLR Subnetworks:Inside-Contact

Resnet50 + SimCLR Subnetworks: Inside-Number

Resnet50 + SimCLR Subnetworks: Inside-Contact

1.0

1.0

1.0

1.0

·1

1.0

1.0

0.8

0.8

0.8

80

0.8

0.8

aqenn

0.6

Inside

0.6

0.4

0.4

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0.2 -

0.2

0.0

Target

Other

0°0

Target

Other

0.0

Target

Other

0°0

Target

Other

0°0

Target

Other

0.0

Target

Other

Resnet50 + SimCLR Ablated Networks: Inside-Contact

Resnet50 + SimCLR Ablated Networks: Inside-Number

Resnet50 + SimCLR Ablated Networks: Inside-Contact

1.0

1.0

1.0

1.0

1.0

1.0

0.8

0.8

0.8

：

0.8

0.8

0.8

Contact

0.6

Number

0.6

0.4

0.4

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0.2

0.2

0.0

Target

Other

0°0

Target

Other

0.0

Target

Other

0°0

Target

Other

0°0

Target

Other

0.0

Target

Other

The first term describes the standard loss function given by an odd-one-out task where the rule is defined by SR i . The second term corresponds to the L 0 penalty, which encourages entries in the binary mask to be 0. However, optimizing such a binary mask is intractable, given the combinatorial nature of a discrete binary mask over a large parameter space. Instead, continuous sparsification reparameterizes the loss function by introducing another variable, s ∈ R d :

In Equation 2, σ is the sigmoid function, applied elementwise, and β is a temperature parameter. During training β is increased after each epoch according to an exponential schedule to a large value β max . Note that, as β - →∞ , σ ( β · s i ) - → H ( s i ) , where H ( s i ) is the heaviside function .

BERTSmallSubnetworks:Anaphora

BERTSmallSubnetworks:SVAgreement

1.0

1.0

1.0

1.0 -

0.8

0.8-

SV Agreement: Subject

0.8

SV Agreement: Verb

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0.0

0.0

0.0

0.0

Target

Other

Target

Other

Target

Other

Target

Other

BERTSmallAblatedNetworks:Anaphora

BERTSmallAblatedNetworks:SVAgreement

1.0

1.0

1.0

日

1.0 -

0.8

0.8-

Subject

0.8

SV Agreement: Verb

0.8-

SV Agreement:S

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

.：

0.2

0.2

0.2 -

0.2

0.0

0.0

0.0

0.0

Target

Other

Target

Other

Target

Other

Target

Other

Thus, during training, we interpolate between a soft mask ( σ ) and a discrete mask ( H ). During inference, we simply substitute σ ( β max · s i )) for H ( s i ) . Notably, we apply continuous sparsification to a frozen model in an attempt to reveal the internal structure of this model, whereas the original work introduced continuous sparsification in the context of model pruning, and jointly trained w and s .

Following Savarese et al. (2020), we fix β max = 200 , λ = 10 -8 , and train for 90 epochs. We train the mask parameters using the Adam optimizer with a batch size of 64 and search over learning rates.

C Mask Hyperparameter Search Details

We search over learning rates {.01, .0001}, mask parameter initializations {0.1, 0.05, 0.0, -0.05}, and mask configurations. For Resnet models, we search over mask configurations by starting masking at different stages. We try either (1) masking the whole network, (2) beginning masking at the third (of four) stages), and (3) beginning masking at the fourth stage. For transformer models, we search over mask configurations based on layers. We try either (1) masking the whole networks, (2) beginning masking at the third (of four) layers, (3) beginning masking at the fourth layer.

We perform this search independently for each trained model and each subroutine. The best hyperparameter configuration is determined based on the following criteria: The subnetwork must achieve at least 90% accuracy on the task it was trained on. This is to ensure that mask optimization succeeded. Then, it was scored on its degree of structural compositionality using the validation sets

BERTSmall+LMSubnetworks:Anaphora

1.0

1.0

1.0

1.0

0.8

0.8-

SV Agreement: Subject

0.8

SV Agreement: Verb

0.8

0.6

0.6

口

0.6

0.6

口

0.4

0.4

0.4

0.4

△

口

0.2

0.2

0.2

0.2

0.0

0.0

0.0

0.0

Target

Other

Target

Other

Target

Other

Target

Other

BERTSmall+LMAblated Networks:Anaphora

BERTSmall+LM Ablated Networks:SVAgreement

1.0

1.0

1.0

1.0

0.8

0.8-

Subject

0.8-

SV Agreement: Verb

0.8

日

SV Agreement:S

0.6

0.6

0.6

0.6

日

8

0.4

■

0.4

0.4

■

0.4

：

△4

会

日

会

0.2

0.2

0.2 -

0.2

0.0

0.0

0.0

0.0

Target

Other

Target

Other

Target

Other

Target

Other

of Test Target Subroutine and Test Other Subroutine If a subnetwork is trained to implement SR 1 , then its compostionality score is calculated using M ablate 1 = M C -Sub 1 . The score is simply the difference in accuracy that M ablate 1 achieves on Test Other Subroutine (which the ablated model should perform well on) and Test Target Subroutine (which the ablated model should fail on). All accuracies are clamped in the range [.25, 1], as .25 is chance accuracy. The hyperparameters that maximize this score are returned.

Note that this process is fairly computationally expensive, as it requires training many separate masks. We used NVIDIA GeForce RTX 3090 GPUs for all experiments. Every experiment can be run on a single GPU, in approximately 1 GPU-hour. The entire hyperparameter search takes approximately 2448 GPU-hours. This number was computed as follows: 1 GPU-hour * 3 model seeds * 2 learning rates * 4 initializations * 3 mask configurations * (6 Resnet50 subroutines + 6 pretrained Resnet50 subroutines + 6 Wide Resnet50 subroutines + 8 BERT subroutines + 8 pretrained BERT subroutines). Each mask has a parameter count comparable to its base model. Future work could improve upon the methodology presented here by reducing the number of hyperparameters that one must search over.

D ViT Hyperparameter Search Results

See Table 4 for the results of our hyperparameter search on ViT models. We tried several batch sizes and learning rates on both a 6 and 12 layer ViT, all with a 2 layer MLP head. The MLP had a hidden layer of dimensionality 2048, and an output dimensionality of 128, similar to the Resnet50 and Wide Resnet50. Note that all models fall short of solving any of the tasks.

E Vision Stimuli

In this section, we provide examples from all vision datasets that we use in this work. First, we describe the +/- Number subroutine. This subroutine operates as follows: for each training/test example, let N be an integer. All image types that exhibit (+ Number) will contain N shapes, whereas image types that exhibit (- Number) will contain M shapes, M = N . For a description of the other subroutines, refer back to Section 5.

F Language Data Details

As noted in the main text, our language data is generated using the templates provided by Marvin & Linzen (2019). For the subject-verb agreement datasets, we omit templates that position the noun of interest inside either a sentential complement or an object relative clause. Thus, all of our nouns of interest are the subject of the full sentence. This is done in order to render the (Singular/Plural Subject) subroutine unambiguous across different sentence templates. We do the same for the Reflexive Anaphora datasets, removing the template that positions the antecedent inside a sentential complement.

These exclusions mean that the nouns of interest are always the second word of the sentence. This makes the (Singular/Plural Subject) subroutine amenable to a simple heuristic: check the syntactic number of the second word in the sentence, rather than first needing to identify the subject of a sentence. However, we are unconcerned about this heuristic: the present work makes no claims about how a neural network implements any particular subroutine, instead caring about how several subroutines are organized in the network's weights (i.e. are they represented compositionally, or in an entangled fashion?).

Specifically, the subroutines we examine are those that compute the syntactic number of specific words in a sentence (either subject and verb, or antecedent and pronoun). Our goal is to find subnetworks that implement these subroutines. Consider the case of subject verb agreement. If we were to partition our data precisely analogously to the vision datasets, we would arrive at a compositional dataset where rule-following data points exhibit, say, (Singular Subject, Singular Verb) , and rule breaking examples might exhibit any of (Plural Subject, Singular Verb) , (Singular Subject, Plural Verb) , or (Plural Subject, Plural Verb) . However, one might expect that a pretrained network would implement syntactic number subroutines in service of another salient computation: discerning whether a sentence is grammatical or not. In this case, a pretrained model would need to

̸

unlearn this grammaticality computation, forcing two grammatical sentences apart in its embedding space. In order to avoid this potential complication, we split up our datasets into singular and plural partitions, such that only rule-following examples are grammatical (and all rule-breaking examples are ungrammatical) in each compositional dataset and subroutine test set.

Note that these datasets are smaller than those used for the vision experiments. Using the Marvin & Linzen (2019) templates and their provided vocabulary, and discarding the templates noted above, we arrive at the following dataset statistics (which are identical for the singular and plural instances of each dataset). For each, we provide on singular example and one plural example. The odd one out is always the fourth sentence.

Subject-Verb Agreement: Compositional Dataset : 9500 (Train), 500 (Validation), 1000 (Test)

Singular

the farmer near the parent is old

the surgeon that the architects hate laughs

the novel that the dancer likes is new

the senator to the side of the parents are young

Plural

the farmers the taxi drivers love are short

the songs the dancers admire are unpopular

the surgeons that admire the executives are young

the officers that love the assistant is short

3

D

4

D

Subject-Verb Agreement: (Singular/Plural Subject) Dataset : 9500 (Train), 500 (Validation), 1000 (Test)

Singular

the farmer that the taxi driver hates smile

the consultant the guards hate is young

the poem that the assistant likes brings joy to people

the customers that the chefs like is tall

Plural

the novels the guard hates are good

the teachers across from the parent is young

the shows that the taxi driver likes are unpopular

the manager across from the parent smile

Subject-Verb Agreement: (Singular/Plural Verb) Dataset : 9500 (Train), 500 (Validation), 1000 (Test)

Singular

the game the executives admire is unpopular

the surgeon to the side of the taxi drivers smiles

the consultants the dancer likes swims

the painting that the chefs love are unpopular

Plural

the customer the assistant loves swim

?

the surgeons that the executive likes are short

the authors that love the chef swim

the officers that like the assistant swims

Subject-Verb Agreement: Test (Singular/Plural Subject) Dataset : 300 (Validation), 300 (Test)

Singular

the teacher to the side of the taxi driver swims

the farmer that the chef likes is young

the novel the ministers admire is bad

the customers in front of the dancers is old

Plural

the pictures by the skater interest people

the movies the skater admires are bad

the pilots in front of the taxi driver are tall

the consultant that the dancer likes are old

Subject-Verb Agreement: Test (Singular/Plural Verb) Dataset : 300 (Validation), 300 (Test)

Singular

the senator the taxi drivers admire is young

the pilot to the side of the dancer smiles

the farmer the assistant admires is young

the pilot that loves the minister are tall

Plural

the poems that the chefs hate are bad

the surgeons in front of the dancer laugh

the surgeons near the taxi drivers smile

the farmers behind the architects is short

Reflexive Anaphora: Compositional Dataset : 2500 (Train), 200 (Validation), 200 (Test) Singular

the consultant that the chef loves disguised himself

the manager that the architects hate congratulated herself

the pilot that the architects admire hurt herself

the surgeon that the executives like congratulated themselves

Plural

the consultants that the guards love injured themselves

the senators that the minister admires embarrassed themselves

the officers that the assistant likes embarrassed themselves

the teacher that the dancer loves embarrassed themselves

Reflexive Anaphora: (Singular/Plural Antecedent) Dataset : 2500 (Train), 200 (Validation), 200 (Test)

Singular

the officer that the taxi driver likes doubted herself

the author that the architect loves hated himself

the manager that the executives love disguised herself

the customers that the parent likes disguised himself

Plural

the authors that the skater hates doubted himself

the surgeons that the parents admire hurt themselves

the officers that the taxi driver hates injured himself

the pilot that the assistant loves hurt themselves

Reflexive Anaphora: (Singular/Plural Pronoun) Dataset : 2500 (Train), 200 (Validation), 200 (Test)

Singular

the customer that the ministers hate congratulated herself

the surgeons that the dancers like embarrassed himself

the authors that the taxi driver hates embarrassed himself

the author that the architect admires doubted themselves

Plural

the officer that the skaters admire embarrassed themselves

the senator that the guard likes embarrassed themselves

the customer that the ministers love doubted themselves

the managers that the guard admires injured herself

Reflexive Anaphora: Test (Singular/Plural Antecedent) Dataset : 200 (Validation), 200 (Test)

Singular

the customer that the skater admires hurt herself

the consultant that the executive loves disguised herself

the manager that the skaters like embarrassed herself

the senators that the guard admires injured himself

Plural

the senators that the architects like embarrassed themselves

the authors that the executives admire disguised themselves

the surgeons that the taxi driver admires doubted themselves

the officer that the parents like congratulated themselves

Reflexive Anaphora: Test (Singular/Plural Pronoun) Dataset : 200 (Validation), 200 (Test)

Singular

the pilot that the chefs hate hurt himself

the teacher that the taxi drivers love hated herself

the senator that the assistant loves embarrassed herself

the pilot that the skaters admire embarrassed themselves

Plural

the authors that the parents admire congratulated themselves

the pilots that the chef hates hurt themselves

the authors that the parents like congratulated themselves

the farmers that the ministers admire injured herself

G Vision Pretraining Details

We pretrain a Resnet50 model and MLP using SimCLR, a contrastive self-supervised learning algorithm (Chen et al., 2020). This algorithm generates two views of an image using random data augmentations, then maximizes the agreement between representations of these views using a contrastive loss function. We use a temperature of 0.07 for this loss.

Our data augmentations include horizontal flips, affine transformations, color jitters, rotations, and grayscaling. We train for 100 epochs, using a learning rate of .0005 (which is decayed according to a consine annealing schedule) and a batch size of 256. Images are drawn randomly from the three compositional training sets ( Inside-Contact , Number-Contact , Inside-Number ). For every rule-following image that is selected, a rule-breaking image from that same dataset is also selected.

We evaluate the Top-5 Accuracy on a held-out validation set after every epoch, and save the weights of the best performing model. Following Chen et al. (2020), we discard the MLP after pretraining, only using the Resnet50 weights to initialize our pretrained models in Section 8.

We adapted the implementation found in Lippe (2022) to implement SimCLR pretraining.

H Subnetwork Sparsity Data

In this section, we provide the raw sparsity statistics for each subnetwork trained in this paper. For every subnetwork, we indicate what stage we started masking (0, 3, or 4), provide the number of Act. Param. in the subnetwork (i.e. the number of 1's in the binary mask), and include the total number of parameters that we mask over (i.e. the number of entries in the binary mask), which is determined by the mask stage.

I Control Experiment: Random Models

Recent work has demonstrated several surprising properties of masks trained on randomly-intialized networks (Ramanujan et al., 2020; Zhou et al., 2019; Wortsman et al., 2020). One might wonder whether the results demonstrated here could be obtained by training a binary mask over randomlyinitialized network. If so, this would pose a serious problem for our interpretation of the data: producing the same results in a randomly-initialized network would decouple the behavior of the discovered subnetworks from the representations learned by the underlying base model.

We carry out this experiment as a control. Specifically, we run the exact same mask training procedure used to generate the results in Section 7, except we use randomly-initialized models rather than

models trained on compositional tasks. In Section 7, each (model, subroutine) pair had its own set of masking hyperparameters. We use these same hyperparameters for each (randomly-intialized model, subroutine) pair in order to make the results as comparable as possible.

In Figure 16 and 17, we observe that masking random networks produces distinctly different patterns of results than we presented in Section 7. Though it is possible find a subnetwork that computes a target subroutine and not the other subroutine, the ablation results do not follow the pattern that one would expect of a compositional model. This accords with Ramanujan et al. (2020), which demonstrates that training a binary mask over a randomly-weighted network can still produce performant subnetworks. The ablation results indicate that these subnetworks are not causally implicated in model behavior. In the case of Resnet50 (Figure 16, Bottom) we observe that ablating the learned subnetworks collapses performance to chance for all tasks. In the case of BERT-Small (Figure 17, Bottom), we observe that ablating the learned subnetworks oftentimes yields high performance on Test Target Subroutine and low performance on Test Other Subroutine , which is the opposite of the expected trend for a compositional model. Thus, we can be more confident that

the main results presented in Section 7 reflect the internal mechanisms of trained models, and are not epiphenomenal artifacts of training binary masks over networks.

I.1 Statistical Analysis of Main Results vs. Random Results

In order to assess whether the results given by random models are significantly different from our main results, we fit a generalized linear model (GLM) with robust clustered standard errors for each combination of model architecture, compositional task, and subroutine. This GLM includes a dummy variable indicating whether the results are from a subnetwork or an ablated model, a dummy variable indicating whether the base model is trained or random, and it clusters observations by base model. For language experiments, we collapse across singular and plural instances of the same subroutine. The coefficient of the Trained vs. Random dummy variable in this model assesses whether the performance of the discovered subnetworks are significantly different in the trained and random conditions. From this model, we can also perform a linear hypothesis test to assess whether ablated model performance is significantly different between the trained and random conditions. Table 12 provides the relevant statistics. Across the board, we see that there is often a significant difference

between ablating the discovered subnetworks in random vs. trained models, even with a small sample size.

J Pruned Model Analysis

In this section, we analyze the structural compositionality of models after they have been pruned. We analyze the impact of pruning on structural compositionality on Resnet50 models trained on NumberContact. We use continuous sparsification to train binary masks over the three Number-Contact Resnet50 models analyzed in the main paper, resulting in three sparse models that perform well on the Number-Contact task. We search over mask initialization and learning rate hyperparameters to find the best pruning configuration for each model. Then, we run the same structural compositionality analysis described in the main paper. We see from Figure 18 that the results on the pruned Resnet50 models closely resemble those from the full Resnet50 models.

Inside-Contact

Inside-Number

Number-Contact

1.00

1.00

1.00

1.00

1.00

1.00

0.75

0.75

0.75

0.75

0.75

0.75

0S'0

0.50

0.50 -

0.50

0.50

0.50

0.25

?

0.25

0.25

0.25

0.25

0.25

Inside

Contact

Inside

Number

Contact

Number

00°0

00°0

0.00

00°0

0.00

±

0.00

0.25

0.25

0.25

0.25

0.25

0.25

0.50

0.50

0S'0-

0.50

0.50 -

0.50

0.75

0.75

0.75

0.75

0.75

0.75

1.00

Subnetwork Ablation

1.00

Subnetwork Ablation

1.00

Subnetwork Ablation

1.00

SubnetworkAblation

1.00

Subnetwork Ablation

1.00

SubnetworkAblation

Resnet50Subnetworks:Inside-Contact

Resnet50 Subnetworks:Inside-Number

Resnet50Subnetworks:Number-Contact

1.0

1.0

1.0

1.0

1.0

1.0

0.8

0.8

0.8-

0.8-

0.8

80

Contact

0.6

Number

0.6

Contact

0.6

Number

0.6

0.4

0.4

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0.2

0.2

0.0

Target

Other

0.0

Target

Other

00

Target

Other

0.0

Target

Other

0.0

Target

Other

0.0

Target

Other

Resnet50Ablated Networks:Inside-Contact

Resnet50 Ablated Networks:Inside-Number

Resnet50 Ablated Networks:Number-Contact

1.0

1.0

1.0

1.0

1.0

1.0

0.8

0.8

0.8

0.8

0.8

80

Contact

0.6

0.6

Contact

0.6

Number

0.6

0.4

0.4

0.4

0.4

0.4

0.4

!1！

0.2

0.2

0.2

0.2

0.2

0.2

0.0

Other

0.0

Other

0°0

Other

0.0

Target

Other

0.0

Other

0.0

Other

Target

Target

Target

Target

Target

K Subnetwork Overlap Analysis

In this section, we analyze the overlap in the subnetworks that were discovered within the same base model. We perform this analysis on one model, a Resnet50 trained on the Inside-Number task. From Figure 4, we see that this model appears to exhibit structural compositionality. We analyze this model because subnetwork masking started at the same layer for both subroutines, which allows for a straightforward comparison of the overlap between subnetworks. All results are shown in Figure 19.

Following previous work (Csordás et al., 2021), we compute the per-layer intersection over union (IoU) to quanitify subnetwork overlap. First, we do this for each of the three subnetworks discovered for the Inside subroutine. See these results in Table 13. Next, we compute the same for the three subnetworks discovered for the Number subroutine. See these results in Table 14. The Inside subroutine gives near ceiling agreement, while the Number task exhibits much lower agreement. This indicates that the subnetworks we uncover are somewhat noisy for the Number subroutine, but not for the Inside subroutine. Finally, we compute the intersection of each set of subnetworks, and compute the per-layer intersection over union between tasks. See Table 15. We see very low agreement between tasks, especially before the final MLP. Notably, this between-task agreement is consistently

SV Agreement

1.00

1.00

1.00

1.00

0.75

"

0.75 -

0.75

0.75

0.50

0.50-

0.50-

0.50

0.25

Antecedent

0.25 -

0.25

0.25

Pronoun

Subject

Verb

0.00

0.00

0.00

0.00

0.25

0.25 -

0.25

0.25

0.50

0.50

0.50

F0S'0-

0.75

0.75

0.75

0.75

-1.00

1.00

1.00

1.00

SubnetworkAblation

SubnetworkAblation

SubnetworkAblation

SubnetworkAblation

1.0

·

1.0

1.0

1.0

0.8

0.8

0.8

0.8

Antecedent

Pronoun

Subject

0.6

0.6

0.6-

Verb

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0.0

0.0

0.0

0.0

Target

Other

Target

Other

Target

Other

Target

Other

BERTSmallAblatedNetworks:Anaphora

BERTSmallAblatedNetworks:SVAgreement

1.0

1.0

1.0

1.0

■

0.8

0.8

0.8

0.8

Antecedent

Pronoun

0.6-

0.6-

Subject

0.6

Verb

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0.0

0.0

0.0

0.0

Target

Other

Target

Other

Target

Other

Target

Other

lower than both within-task agreement values for each layer. This reinforces our interpretation that these subnetworks are organized in a modular fashion within the base models.

Pruned Resnet50 (Test Target - Test Other) Number-Contact

1.00

1.00

0.75

0.75

0.50

0.50

0.25

0.25

Contact

ber

b

0.00

Numk

0.00

-0.25

-0.25

-0.50

-0.50

0.75

-0.75

-1.00

-1.00

Subnetwork Ablation

Subnetwork Ablation

backbone.layer4.0.conv1

backbone.layer4.0.conv2

backbone.layer4.0.conv3

backbone.layer4.0.downsample.0

backbone.layer4.1.conv1

backbone.layer4.1.conv2

backbone.layer4.1.conv3

backbone.layer4.2.conv1

backbone.layer4.2.conv2

backbone.layer4.2.conv3

mlp.model.0

mlp.model.2

Layers

0.0

0.2

0.4

0.6

0.8

1.0

IoU

Overlap Analysis

Inside

Number

Intersection

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the structural compositional problem is a novel perspective. It first defines structural compositionality as the extent to which neural networks decompose a complex task into a series of subroutines and implement them modularly. Then, the paper designs several clever experiments to show that many models indeed implement subroutines in their subnetworks, which demystifies how the compositional generalization ability might occur in deep neural networks. Although the paper could be further improved in the following directions (see the limitation part), I think the current version already meets the criteria of NeurIPS and can inspire the community a lot. So I would give an accepting score to the current version. In summary, the paper is easy to follow and quite novel to me. I enjoy reading it a lot.\n\nSTRENGTHS:\n1. Rather than the downstream generalization ability or the data attributes, the paper focuses on the composition of different rules (functions, or subroutines), which is quite novel and persuasive.\n2. The experimental designs are quite ingenious and persuasive to me, it relates the subroutines to subnetworks using the mask function on each parameter.\n3. The paper investigates different model architectures (ResNet, ViT, BERT) and different input signals (image and language).\n\nWEAKNESSES:\nSee the limitation part.\n\nQUESTIONS:\n1. For the non-compositional solution panel in Figure 1, it might be better to draw class A and class B in the upper right and lower left corners respectively. Then, adding a decision boundary (i.e., the diagonal) would be helpful.\n2. It is a little hard for me to understand what the bottom-right panel of Figure 1 is talking about before reading the experimental part.\n3. Seems that the filling color of the four panels in the bottom-right part of Figure 2 is wrong. IIUC, the color for the ablation model’s performance would be the inverse of those for the subnetwork.\n4. In section 7, the authors mentioned that ViT fails on the proposed problem, is there any possible explanation?\n5. For Figure 4, adding the legends for *** and gray dots would be helpful.\n\nLIMITATIONS:\nI believe solving the following two concerns could make the paper stronger and bring more insights to the community (it is also fine to consider them in future works).\n\n1. The paper selects the subnetwork by training a mask function $m$ and generates two subnetworks, i.e., $Sub_i$ and $M_{ablate}$, based on it. Then it might be interesting to draw the mask for different subnets (keeping the structure of the model) to see if there exist any interesting patterns. For example, it is possible that these two complementary networks are using different halves of the neurons across all layers; or they might share some lower layers but split apart in the higher layers. I guess these results might be quite helpful for us to understand HOW different subroutines are stored in different model architectures. (Maybe whether the model is pretrained also influences this split.)\n2. Besides the related works mentioned in the paper, there is also another line of works discussing how compositional mapping emerges in human language and neural network representations. It is named iterated learning, which is first proposed in the field of language evolution to explain why modern human language is so structural [1] and then extended to the deep learning conditions, e.g., emergent communication in [2] and visual question-answering in [3]. It might be interesting to consider the relationships between these works.\n\n[1] Kirby, Simon, et al. \"Cumulative cultural evolution in the laboratory: An experimental approach to the origins of structure in human language.\" PNAS 2008\n\n[2] Ren, Yi, et al. \"Compositional languages emerge in a neural iterated learning model.\" ICLR 2020\n\n[3] Vani, Ankit, et al. \"Iterated learning for emergent systematicity in VQA.\" ICLR 2021", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the concept of structural compositionality in neural networks - subnetworks that implement specific subroutines, such as computing the syntactic number of specific words in a sentence.\n\nSTRENGTHS:\n1. Conceptual modularity in neural networks has been an idea that has been studied and structurally implemented across various domains: explainability, reasoning, efficiency. The authors advance the field of study of modularity in neural networks by proposing the idea of structural compositionality, which is both is very well thought out and explained. The experimental plan is also carefully controlled to ensure causal observations regarding the existence of structural compositionality in networks across both vision and language tasks. Overall this is a very well done study.\n\nWEAKNESSES:\nNo major waknesses in my opinion\n\nQUESTIONS:\n**Important Related Work:** While the authors mention the field of mechanistic interpretability, they fail to mention some key papers in this field that explore similar ideas as the authors Chugtai et al [1] reverse engineer small neural networks to show that they learn to implement group compositions for any finite group using partical novel algorithm predicted from theory. Nouha et al [2] show that transformers solve compositional tasks by approximating part of the full computational graph as linear sub-graph matching, and provide theoretical proofs on how increasing compositional task complexity would lead to degradation in performance. I think these papers should be referred so that readers will have a more complete picture of compositionality in neural networks. \n\n### References \n\n1. Chughtai, Bilal, Lawrence Chan, and Neel Nanda. \"A Toy Model of Universality: Reverse Engineering how Networks Learn Group Operations.\" (2023).\n2. Dziri, Nouha, et al. \"Faith and Fate: Limits of Transformers on Compositionality.\" arXiv preprint arXiv:2305.18654 (2023).\n\nLIMITATIONS:\nThe authors do a good job at highlighting the limitations (e.g. specifying subroutines in advance) of their work. I do not particularly see potential negative social impacts of this work as it mostly pertains to explainability of models.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis research paper explores the concept of compositionality in neural networks, a contentious topic in the field of AI. Compositionality, which is a defining feature of human cognition, allows for abstract and flexible processing of language and visuals. The debate lies in whether neural networks need explicit symbolic systems to implement compositional solutions, or if these solutions can be implicitly learned during training. This research introduces \"structural compositionality\", which gauges the degree to which neural networks can break down compositional tasks into subroutines and implement them in a modular fashion. Surprisingly, the study found evidence that many models do implement subroutines in modular subnetworks. Additionally, it was found that unsupervised pretraining leads to a more consistently compositional structure in language models. This research contributes to mechanistic interpretability, helping explain the algorithms that neural networks implement in their weights, using techniques from model pruning.\n\nSTRENGTHS:\nThe paper is quite interesting, timely and I think that most people would find the results surprising. The area of mechanistic interpretability is very important these days. One of the strongest points of this paper is the elegant approach it is taking, in terms of experimental design, to construct compositional tasks and to discover whether the presence of such \"functional compositionality\" also results in specific subnetworks for each \"concept\". The paper presents results both for images and for language experiments. The results are clear and the presentation of the paper is very good.\n\nWEAKNESSES:\nOf course, one general weakness is that the main result of the paper is through empirical results. The reader may not be convinced whether the claims of the paper are actually true in general -- or whether they are true only in some architectures and tasks. \n\nAnother major weakness, which the authors also identify as a limitation of their study, is that the one must know which subroutines to look for.\n\nQUESTIONS:\nA question for the authors to think about: what is the role of pruning (or network density, more generally) in the emergence of structural compositionality? Would you expect that emergence to be equally likely in a dense network and in a sparse network that has resulted through  pruning?\n\nAnother question to think about: what is the role of the network's depth in the emergence of structural compositionality? Functional compositionality can have its own depth. How is that functional depth relate to the depth of the network itself?\n\nLIMITATIONS:\nThe paper is honest about its limitations. The important of them is that one has to know in advance which \"subroutines\" (I do not like this term actually -- maybe \"concepts\" or \"subfunctions\" would be better terns) to look for. This limitation however may be addressed by other work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors investigated to what extent the standard neural networks of the present day trained on the tasks solvable by composing subroutines result in modular structures reflecting the tasks' compositional nature (called structural compositionality in this study). To answer this question, they took the following approach on the trained networks: 1) use a model pruning technique to find the best subnetworks for individual subroutines, then 2) check the patterns in  accuracy difference of the discovered subnetworks and the ablated networks (complemental subnetworks with respect to the first ones) on the tasks corresponding to individual subroutines. They conducted experiments on both vision and language tasks with multiple neural networks and pretraining regimes. Based on the results, they concluded that the neural networks oftentimes exhibit structural compositionality even when there are no explicit inductive biases toward such solutions.\n\nSTRENGTHS:\nMaking a learning machine that has compositional (systematic) generalization capability like a human is an important goal yet to be achieved in the field of artificial intelligence. On one hand, standard neural networks that are not explicitly imposed inductive biases for compositionality can generally be said to fail in compositional generalization as stated in lines 56 - 57, but on the other hand, these networks show compositional generalization capability to some extent in some cases, although not perfect and inferior to the ones that are imposed inductive biases. Revealing what is going on inside a neural network is an important research topic in this context, and the approach and results of this study will be of interest to the NeurIPS audience. \n\nThe idea of the approach took in this study is clear and reasonable. The model pruning method, experimental logic and concrete experiments are explained fairly well (some part is given in the Appendix).  The experiments are fairly rich in terms of tasks (both vision and language tasks were conducted and there are varieties in each) and models (ResNet50, WideResNet50, and ViT were tried for visual tasks, and BERT-small was tried for language tasks).\n\nWEAKNESSES:\n1. The URL of an Anonymous GitHub repository is provided in the paper (footnote on page 2, https://anonymous.4open.science/r/Compositional_Subnetworks-C8AB/). However, it is very hard to access the contents of the repo, because, although the repo consists of many subdirectories, there is no README in the top-level directory giving an overview.  This cast a shadow on the reproducibility.  \n1. Another relatively weak point of this paper is the contextualization relative to Csordás et al. 2021. In lines 49 and 259, Csordás et al. 2021 is explained as a work merely on a multitask setting, but it studied compositional (systematic) generalization settings using the SCAN (Lake and Baroni 2018) and the Mathematics Dataset (Saxton et al. 2019). \n\n\n\nCsordás, R., van Steenkiste, S., and Schmidhuber, J. Are neural nets modular? Inspecting functional modularity through differentiable weight masks. In Proc. of the 9th International Conference on Learning Representations (ICLR), 2021. \n\nLake, B. M.  and Baroni, M. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Proc. of the 35th International Conference on Machine Learning (ICML), pp. 2873--2882, 2018. \n\nSaxton, D., Grefenstette, E., Hill, F., and Kohli, . Analysing mathematical reasoning abilities of neural models. In Proc. of the 7th International Conference on Learning Representations (ICLR), 2019.\n\nQUESTIONS:\nMajor Suggestions \n1. I suggest adding README explaining the overall repository in the top-level directory during the Author Rebuttal period.\n1. I suggest better contextualizing this work with respect to Csordás et al. 2021 in the paper.\n\nPlease also refer to Weaknesses section above.\n\nMinor Sugestions \n1. It would be better to mention WideResNet in Section 7 (Results) and point to Appendix.\n1. It would be nice if applicability of the proposed approach to recurrent neural networks are explained in the paper.\n\nLIMITATIONS:\nThe authors describes the limitations of their current method and the results reported in the paper in Section 10 (Discussion):  \n1. Their current method requires one to specify which subroutines to look for in advance. \n1. Their current method requires one to use causal ablations and control models to properly interpret. \n1. The reported results do not contain any analysis on the relationship between structural compositionality and compositional generalization.\n\n\\# Personally, I am very interested in 3, which is fully left for the future work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}