SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Most base models perform near perfectly, with the exception of ViT, which failed to achieve  $>90\%$  performance on any of the tasks with any configuration of hyperparameters<sup>12</sup>. Thus, we exclude ViT from all subsequent analyses. See Appendix D for these results. If the base models exhibit structural compositionality, we expect subnetworks to achieve greater accuracy<sup>13</sup> on Test Target Subroutine than on Test Other Subroutine (difference in accuracies  $>0$ ). After ablating subnetworks, we expect the ablated model to achieve greater accuracy on Test Other than Test Target (difference in accuracies  $<0$ ). Across the board, we see the expected pattern. Subnetwork and ablated accuracy differences for Resnet50 and BERT are visualized in Figure 4 (Subnetwork in Blue, Ablated Models in Red). See Appendix A for Wide Resnet50 results, which largely reproduce the results using Resnet50.
For some architecture/task combinations, the pattern of ablated model results is statistically significantly in favor of structural compositionality. See Figure 4 (C, D), where all base models seem to implement both subroutines in a modular fashion. We analyze the layerwise overlap between subnetworks found within one of these models in Appendix K. This analysis shows that there is relatively high overlap between subnetworks for the same subroutine, and low overlap between subroutines. Other results are mixed, such as those found in Resnet50 models trained on Number-Contact. Here,
<sup>9</sup>We used NVIDIA GeForce RTX 3090 GPUs for all experiments. Every experiment can be run on a single GPU, in approximately 1 GPU-hour. After performing a hyperparameter search, our main results took approximately 300 GPU-hours.
<sup>10</sup>Note that we are interested only in discovering some evidence of modularity within the model rather than looking for some more profound syntactic phenomenon.
11See Appendix F for more details
<sup>12</sup>See Table 2 in Appendix A for each base model's performance on the relevant compositional task.
13 All accuracy values are clamped to the range [0.25, 1.0] before differences are computed. 0.25 is chance accuracy. Constraining values to this range prevents false trends from arising in the difference data due to models performing below chance.
Figure 4: Results from Subnetwork and Ablation studies. For each compositional task, we learn binary masks that result in subnetworks for each subroutine. Resnet50 results in the top row, BERT-Small results in the bottom row. Gray markers indicate that the corresponding base model did not achieve  $>90\%$  accuracy on the compositional task. (Blue) The difference between subnetwork performance on Test Target Subroutine and Test Other Subroutine. If a model exhibits structural compositionality, we expect that a subnetwork will achieve greater performance on the subroutine that it was trained to implement, resulting in values  $>0$ . (Red) After ablating the subnetwork, we evaluate on the same datasets and plot the difference again. We expect that the ablated model will achieve lower performance on the subroutine that the (ablated) subnetwork was trained to implement and higher performance on the other subroutine dataset, resulting in values  $<0$ . Across the board, we find that our results are largely significantly different from 0, despite the small number of samples. ** indicates significance at  $\mathrm{p} = .01$ , *** indicates significance at  $\mathrm{p} = .001$  See Appendix A for details of this statistical analysis.
we see strong evidence of structural compositionality in Figure 4 (E), but little evidence for it in Figure 4 (F). In this case, it appears that the network is implementing the  $(+/-\text{Contact})$  subroutine in a small, modular subnetwork, whereas the  $(+/-\text{Number})$  subroutine is implemented more diffusely. We perform control experiments using randomly initialized models in Appendix I, which show that the pattern of results in (A), (B) and (F) are not significantly different from a random model, while all other results are significantly different.
8 Effect of Pretraining on Structural Compositionality
We compare structural compositionality in models trained from scratch to those that were initialized with pretrained weights. For Resnet50, we pretrain a model on our data using SimCLR (See Appendix G for details).
For BERT-Small, we use the pretrained weights provided by Turc et al. (2019). We rerun the same procedure described in Sections 5 and 6. See Appendix A for each base model's performance. Figure 5 contains the results of the language experiments. Across all language tasks, the ablation results indicate that models initialized with pretrained weights more reliably produce modular subnetworks than randomly initialized models. Results on vision tasks are found in Appendix A and do not suggest any benefit of pretraining.
Figure 5: Performance differences between Test Target Subroutine and Test Other Subroutine for both models trained from scratch and pretrained models. Across the board, we see that pretraining produces more modular subnetworks (i.e., reveal a greater disparity in performance between datasets). Pretraining also appears to make our subnetwork-discovery algorithm more robust to random seeds.
9 Related Work
This work casts a new lens on the study of compositionality in neural networks. Most prior work has focused on compositional generalization of standard neural models (Yu & Ettinger, 2020; Kim & Linzen, 2020; Kim et al., 2022; Dankers et al., 2022), though some has attempted to induce an inductive bias toward compositional generalization from data (Lake, 2019; Qiu et al., 2021; Zhu et al., 2021). Recent efforts have attempted to attribute causality to specific components of neural networks' internal representations (Ravfogel et al., 2020; Bau et al., 2019; Wu et al., 2022; Tucker et al., 2021; Lovering & Pavlick, 2022; Elazar et al., 2021; Cao et al., 2021). In contrast to these earlier studies, our method does not require any assumptions about where in the network the subroutine is implemented and does not rely on auxiliary classifiers, which can confound the causal interpretation. Finally, Dziri et al. (2023) performs extensive behavioral studies characterizing the ability of autoregressive language models to solve compositional tasks, and finds them lacking. In contrast, our work studies the structure of internal representations and sets aside problems that might be specific to autoregressive training objectives.
More directly related to the present study is the burgeoning field of mechanistic interpretability, which aims to reverse engineer neural networks in order to better understand how they function (Olah, 2022; Cammarata et al., 2020; Black et al., 2022; Henighan et al., 2023; Ganguli et al., 2021; Merrill et al., 2023). Notably, Chughtai et al. (2023) recovers universal mechanisms for performing group-theoretic compositions. Though group-theoretic composition is different from the compositionality discussed in the present article, this work sheds light on generic strategies that models may use to solve tasks that require symbolically combining multiple input features.
Some recent work has attempted to characterize modularity within particular neural networks (Hod et al., 2022). Csordás et al. (2021) also analyzes modularity within neural networks using learned binary masks. Their study finds evidence of modular subnetworks within a multitask network: Within a network trained to perform both addition and multiplication, different subnetworks arise for each operation. Csordás et al. (2021) also investigates whether the subnetworks are reused in a variety of contexts, and find that they are not. In particular, they demonstrate that subnetworks that solve particular partitions of compositional datasets (SCAN (Lake & Baroni, 2018) and the Mathematics
Dataset (Saxton et al., 2018)), oftentimes do not generalize to other partitions. From this, they conclude that neural networks do not flexibly combine subroutines in a manner that would enable full compositional generalization. However, their work did not attempt to uncover subnetworks that implement specific compositional subroutines within these compositional tasks. For example, they did not attempt to find a subnetwork that implements a general "repeat" operation for SCAN, transforming "jump twice" into "JUMP JUMP". Our work finds such compositional subroutines in language and vision tasks, and localizes them into modular subnetworks. This finding extends Csordás et al. (2021)'s result on a simple multitask setting to more complex compositional vision and language settings, and probes for subroutines that represent intermediate subroutines in a compositional task (i.e. "inside" is a subroutine when computing "Inside-Contact").
10 Discussion
Across a variety of architectures, tasks, and training regimens, we demonstrated that models often exhibit structural compositionality. Without any explicit encouragement to do so, neural networks appear to decompose tasks into subroutines and implement solutions to (at least some of) these subroutines in modular subnetworks. Furthermore, we demonstrate that self-supervised pretraining can lead to more consistent structural compositionality, at least in the domain of language. These results bear on the longstanding debate over the need for explicit symbolic mechanisms in AI systems. Much work is focusing on integrating symbolic and neural systems (Ellis et al., 2023; Nye et al., 2020). However, our results suggest that some simple pseudo-symbolic computations might be learned directly from data using standard gradient-based optimization techniques.
We view our approach as a tool for understanding when and how compositionality arises in neural networks, and plan to further investigate the conditions that encourage structural compositionality. One promising direction would be to investigate the relationship between structural compositionality and recent theoretical work on compositionality and sparse neural networks (Mhaskar & Poggio, 2016; Poggio, 2022). Specifically, this theoretical work suggests that neural networks optimized to solve compositional tasks naturally implement sparse solutions. This may serve as a starting point for developing a formal theory of structural compositionality in neural networks. Another direction might be to investigate the structural compositionality of networks trained using iterated learning procedures (Ren et al., 2019; Vani et al., 2020). Iterated learning simulates the cultural evolution of language by jointly training two communicating agents (Kirby et al., 2008). Prior work has demonstrated that iterated learning paradigms give rise to simple compositional languages. Quantifying the relationship between structural compositionality within the agents and the compositionality of the language that they produce would be an exciting avenue for understanding the relationship between representation and behavior.
One limit of our technical approach is that one must specify which subroutines to look for in advance. Future work might address this by discovering functional subnetworks using unsupervised methods. Additionally, our approach requires us to use causal ablations and control models to properly interpret our results. Future work might try to uncover subnetworks that are necessarily causally implicated in model behavior. Finally, future work must clarify the relationship between structural compositionality and compositional generalization.
The authors thank the members of the Language Understanding and Representation Lab and Serre Lab for their valuable feedback on this project and Rachel Goepner for proofreading the manuscript. This project was supported by ONR grant #N00014-19-1-2029. The computing hardware was supported in part by NIH Office of the Director grant #S10OD025181 via the Center for Computation and Visualization (CCV) at Brown University.
Andreas, J., Rohrbach, M., Darrell, T., and Klein, D. Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 39-48, 2016.
Wortsman, M., Ramanujan, V., Liu, R., Kembhavi, A., Rastegari, M., Yosinski, J., and Farhadi, A. Supermasks in superposition. Advances in Neural Information Processing Systems, 33:15173-15184, 2020.
Wu, Z., Geiger, A., Rozner, J., Kreiss, E., Lu, H., Icard, T., Potts, C., and Goodman, N. Causal distillation for language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4288-4295, 2022.
Yu, L. and Ettinger, A. Assessing phrasal representation and composition in transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4896-4907, 2020.
Zagoruyko, S. and Komodakis, N. Wide residual networks. In *British Machine Vision Conference* 2016. British Machine Vision Association, 2016.
Zerroug, A., Vaishnav, M., Colin, J., Musslick, S., and Serre, T. A benchmark for compositional visual reasoning. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
Zhang, D., Ahuja, K., Xu, Y., Wang, Y., and Courville, A. Can subnetwork structure be the key to out-of-distribution generalization? In International Conference on Machine Learning, pp. 12356-12367. PMLR, 2021.
Zhou, H., Lan, J., Liu, R., and Yosinski, J. Deconstructing lottery tickets: Zeros, signs, and the supermask. Advances in neural information processing systems, 32, 2019.
Zhu, W., Shaw, P., Linzen, T., and Sha, F. Learning to generalize compositionally by transferring across semantic parsing tasks. arXiv preprint arXiv:2111.05013, 2021.
COMP. TASK	MODEL	SR	INTERCEPT Z	LINEAR HYPOTHESIS χ2
IN. CONT.	RN50	INSIDE	50.13***	1.42
IN. CONT.	RN50	CONTACT	3.00**	0.94
IN. CONT.	WRN50	INSIDE	6.66***	0.87
IN. CONT.	WRN50	CONTACT	72.44***	4.49*
IN. NUM.	RN50	INSIDE	11.13***	29.20***
IN. NUM.	RN50	NUMBER	18.35***	21.73***
IN. NUM.	WRN50	INSIDE	14.86***	2.95.
IN. NUM.	WRN50	NUMBER	5.21***	2.50
CONT. NUM.	RN50	CONTACT	6.45***	40.26***
CONT. NUM.	RN50	NUMBER	7.12***	0.42
CONT. NUM.	WRN50	CONTACT	6.19***	296.96***
CONT. NUM.	WRN50	NUMBER	9.388***	1.32
SV AGR	BERT	SUBJ.	4.53***	14.72***
SV AGR	BERT	VERB	3.73***	1.70
ANAPHORA	BERT	PRONOUN	6.06***	5.60*
ANAPHORA	BERT	ANTECEDENT	2.60**	17.83***
Table 1: Statistics from one factor GLM with robust clustered standard errors. . indicates significance at  $p = .1$ , * indicates significance at  $p = .05$ , ** indicates significance at  $p = .01$ , *** indicates significance at  $p = .001$
A Full Results
In this section, we provide the following results:
1. Base Model Performance on Compositional Tasks: Table 2
2. Pretrained + Finetuned Model Performance on Compositional Tasks: Table 3
3. Wide Resnet50 Subnetwork Results: Figure 6
4. Vision Pretraining vs. Random Initialization Heatmap: Figure 7
5. Absolute Accuracy for every subnetwork and ablated model on each task, for each model: Figures 8-12.
See Table 2 for the performance of all base models on each compositional task. See Table 3 for the performance of all pretrained base models on each compositional task. See Figure 6 for subnetwork and ablation results on Wide Resnet50. See Figure 7 for Vision Model pretraining results.
A.1 Statistical Analysis of Main Results
In order to assess the significance of our main results, we fit a generalized linear model (GLM) with robust clustered standard errors for each combination of model architecture, compositional task, and subroutine. This GLM includes a dummy variable indicating whether the results are from a subnetwork or an ablated model, and it clusters observations by base model. For language experiments, we collapse across singular and plural instances of the same subroutine. The intercept term in this model assesses whether the performance of the discovered subnetworks are significantly different from 0. From this model, we can also perform a linear hypothesis test to assess whether ablated model performance is significantly different from 0. Table 1 provides the relevant statistics. Across the board, we see that there is always a significant difference between subnetwork performance and 0, and often a significant difference between ablated model performance and 0, even with a small sample size.
VISION	CONT.-INSIDE	CONT.-NUMBER	INSIDE-NUMBER
RN50-1	100%	99.4%	99.8%
RN50-2	100%	99.4%	99.8%
RN50-3	75.9%	99.7%	99.9%
WRN50-1	99.9%	99.6%	99.7%
WRN50-2	99.8%	99.8%	99.6%
WRN50-3	99.9%	99.4%	99.8%
LANGUAGE	SV SING.	SV PLUR.	ANAPH. SING.	ANAPH. PLUR.
BERT-SM-1	99.7%	100%	100%	100%
BERT-SM-2	100%	100%	100%	100%
BERT-SM-3	100%	100%	100%	100%
Table 2: Test classification accuracy for each base model for each task. Every entry corresponds to a unique model.
VISION	CONT.-INSIDE	CONT.-NUMBER	INSIDE-NUMBER
RN50-SC-1	100%	99.7%	100%
RN50-SC-2	100%	99.6%	99.8%
RN50-SC-3	100%	99.5%	99.9%
LANGUAGE	SV SING.	SV PLUR.	ANAPH. SING.	ANAPH. PLUR.
BERT-LM-1	100%	100%	100%	100%
BERT-LM-2	100%	100%	65.5%	100%
BERT-LM-3	100%	100%	63.5%	100%
Table 3: Test classification accuracy for each pretrained base model for each task. Every entry corresponds to a unique model.
Figure 6: Wide Resnet50 Subnetwork and Ablation Results. Broadly, they mimic those found in Figure 4. . indicates significance at  $p = .1$ , * indicates significance at  $p = .05$ , ** indicates significance at  $p = .01$ , *** indicates significance at  $p = .001$ .
Figure 7: Vision Model Pretraining vs. Random Initialization. We observe no obvious trend differentiating the two conditions.
Figure 8: Resnet50 absolute performance across all conditions.
B Continuous Sparsification: Extended Discussion
Continuous sparsification attempts to optimize a binary mask that minimizes the following loss function:
$$
\min  _ {m _ {i} \in \{0, 1 \} ^ {d}} L _ {S R _ {i}} \left(M _ {C} (\cdot ; w \odot m _ {i})\right) + \lambda \| m _ {i} \| _ {1} \tag {1}
$$
Figure 9: Wide Resnet50 absolute performance across all conditions
Figure 10: Resnet50 + SimCLR absolute performance across all conditions.
The first term describes the standard loss function given by an odd-one-out task where the rule is defined by  $SR_{i}$ . The second term corresponds to the  $L_{0}$  penalty, which encourages entries in the binary mask to be 0. However, optimizing such a binary mask is intractable, given the combinatorial nature of a discrete binary mask over a large parameter space. Instead, continuous sparsification reparameterizes the loss function by introducing another variable,  $s \in \mathbb{R}^{d}$ :
$$
\min  _ {s _ {i} \in \mathbb {R} ^ {d}} L _ {S R _ {i}} \left(M _ {C} (\cdot ; w \odot \sigma (\beta \cdot s _ {i})) + \lambda \| \sigma (\beta \cdot s _ {i}) \| _ {1} \right. \tag {2}
$$
In Equation 2,  $\sigma$  is the sigmoid function, applied elementwise, and  $\beta$  is a temperature parameter. During training  $\beta$  is increased after each epoch according to an exponential schedule to a large value  $\beta_{max}$ . Note that, as  $\beta \to \infty$ ,  $\sigma (\beta \cdot s_i) \to H(s_i)$ , where  $H(s_i)$  is the heaviside function.
BERT Small Subnetworks: Anaphora
BERT Small Subnetworks: SV Agreement
BERT Small Ablated Networks: Anaphora
BERT Small Ablated Networks: SV Agreement
Figure 11: BERT-Small absolute performance across all conditions
$$
H (s) = \left\{ \begin{array}{l} 0, s <   0 \\ 1, s > 0 \end{array} \right\} \tag {3}
$$
Thus, during training, we interpolate between a soft mask  $(\sigma)$  and a discrete mask  $(H)$ . During inference, we simply substitute  $\sigma (\beta_{max}\cdot s_i)$  for  $H(s_{i})$ . Notably, we apply continuous sparsification to a frozen model in an attempt to reveal the internal structure of this model, whereas the original work introduced continuous sparsification in the context of model pruning, and jointly trained  $w$  and  $s$ .
Following Savarese et al. (2020), we fix  $\beta_{max} = 200$ ,  $\lambda = 10^{-8}$ , and train for 90 epochs. We train the mask parameters using the Adam optimizer with a batch size of 64 and search over learning rates.
C Mask Hyperparameter Search Details
We search over learning rates  $\{.01, .0001\}$ , mask parameter initializations  $\{0.1, 0.05, 0.0, -0.05\}$ , and mask configurations. For Resnet models, we search over mask configurations by starting masking at different stages. We try either (1) masking the whole network, (2) beginning masking at the third (of four) stages, and (3) beginning masking at the fourth stage. For transformer models, we search over mask configurations based on layers. We try either (1) masking the whole networks, (2) beginning masking at the third (of four) layers, (3) beginning masking at the fourth layer.
We perform this search independently for each trained model and each subroutine. The best hyperparameter configuration is determined based on the following criteria: The subnetwork must achieve at least  $90\%$  accuracy on the task it was trained on. This is to ensure that mask optimization succeeded. Then, it was scored on its degree of structural compositionality using the validation sets
BERT Small + LM Subnetworks: Anaphora
BERT Small + LM Subnetworks: SV Agreement
BERT Small + LM Ablated Networks: Anaphora
Figure 12: BERT-Small + LM absolute performance across all conditions.
BERT Small + LM Ablated Networks: SV Agreement
of Test Target Subroutine and Test Other Subroutine If a subnetwork is trained to implement  $SR_{1}$ , then its compositionality score is calculated using  $M_{ablate_{1}} = M_{C} - Sub_{1}$ . The score is simply the difference in accuracy that  $M_{ablate_{1}}$  achieves on Test Other Subroutine (which the ablated model should perform well on) and Test Target Subroutine (which the ablated model should fail on). All accuracies are clamped in the range [.25, 1], as .25 is chance accuracy. The hyperparameters that maximize this score are returned.
Note that this process is fairly computationally expensive, as it requires training many separate masks. We used NVIDIA GeForce RTX 3090 GPUs for all experiments. Every experiment can be run on a single GPU, in approximately 1 GPU-hour. The entire hyperparameter search takes approximately 2448 GPU-hours. This number was computed as follows: 1 GPU-hour * 3 model seeds * 2 learning rates * 4 initializations * 3 mask configurations * (6 Resnet50 subroutines + 6 pretrained Resnet50 subroutines + 6 Wide Resnet50 subroutines + 8 BERT subroutines + 8 pretrained BERT subroutines). Each mask has a parameter count comparable to its base model. Future work could improve upon the methodology presented here by reducing the number of hyperparameters that one must search over.
D ViT Hyperparameter Search Results
See Table 4 for the results of our hyperparameter search on ViT models. We tried several batch sizes and learning rates on both a 6 and 12 layer ViT, all with a 2 layer MLP head. The MLP had a hidden layer of dimensionality 2048, and an output dimensionality of 128, similar to the Resnet50 and Wide Resnet50. Note that all models fall short of solving any of the tasks.
# LAYERS	BATCH SIZE	LEARNING RATE	CONT.-INSIDE	CONT.-NUMBER	INSIDE-NUMBER
6	32	0.01	31%	27%	28%
6	64	0.01	28%	29%	27%
6	32	0.001	29%	27%	28%
6	64	0.001	32%	27%	26%
6	32	0.0001	25%	32%	30%
6	64	0.0001	31%	49%	32%
6	32	0.00001	42%	85%	47%
6	64	0.00001	46%	83%	54%
12	32	0.01	36%	25%	28%
12	64	0.01	27%	26%	25%
12	32	0.001	39%	31%	27%
12	64	0.001	37%	25%	22%
12	32	0.0001	41%	36%	33%
12	64	0.0001	31%	28%	31%
12	32	0.00001	40%	86%	49%
12	64	0.00001	42%	84%	51%
Table 4: Results of ViT hyperparameter search. All accuracies are rounded to the nearest % and are computed on the validation set for the relevant dataset.
E Vision Stimuli
In this section, we provide examples from all vision datasets that we use in this work. First, we describe the +/- Number subroutine. This subroutine operates as follows: for each training/test example, let  $N$  be an integer. All image types that exhibit (+ Number) will contain  $N$  shapes, whereas image types that exhibit (- Number) will contain  $M$  shapes,  $M \neq N$ . For a description of the other subroutines, refer back to Section 5.
F Language Data Details