SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As noted in the main text, our language data is generated using the templates provided by Marvin & Linzen (2019). For the subject-verb agreement datasets, we omit templates that position the noun of interest inside either a sentential complement or an object relative clause. Thus, all of our nouns of interest are the subject of the full sentence. This is done in order to render the (Singular/Plural Subject) subroutine unambiguous across different sentence templates. We do the same for the Reflexive Anaphora datasets, removing the template that positions the antecedent inside a sentential complement.
These exclusions mean that the nouns of interest are always the second word of the sentence. This makes the (Singular/Plural Subject) subroutine amenable to a simple heuristic: check the syntactic number of the second word in the sentence, rather than first needing to identify the subject of a sentence. However, we are unconcerned about this heuristic: the present work makes no claims about how a neural network implements any particular subroutine, instead caring about how several subroutines are organized in the network's weights (i.e. are they represented compositionally, or in an entangled fashion?).
Specifically, the subroutines we examine are those that compute the syntactic number of specific words in a sentence (either subject and verb, or antecedent and pronoun). Our goal is to find subnetworks that implement these subroutines. Consider the case of subject verb agreement. If we were to partition our data precisely analogously to the vision datasets, we would arrive at a compositional dataset where rule-following data points exhibit, say, (Singular Subject, Singular Verb), and rule breaking examples might exhibit any of (Plural Subject, Singular Verb), (Singular Subject, Plural Verb), or (Plural Subject, Plural Verb). However, one might expect that a pretrained network would implement syntactic number subroutines in service of another salient computation: discerning whether a sentence is grammatical or not. In this case, a pretrained model would need to
Figure 13: Examples from tasks defined over the Inside-Contact compositional rule. From top to bottom, we see one example from each of the following tasks: (1) The task used to train base models (2) The task used to train a  $+/-$  Contact subnetwork (3) The task used to train a  $+/-$  Inside subnetwork (4) The evaluation task used to probe for  $+/-$  Contact (5) The evaluation task used to probe for  $+/-$  Inside.
unlearn this grammaticality computation, forcing two grammatical sentences apart in its embedding space. In order to avoid this potential complication, we split up our datasets into singular and plural partitions, such that only rule-following examples are grammatical (and all rule-breaking examples are ungrammatical) in each compositional dataset and subroutine test set.
Note that these datasets are smaller than those used for the vision experiments. Using the Marvin & Linzen (2019) templates and their provided vocabulary, and discarding the templates noted above, we arrive at the following dataset statistics (which are identical for the singular and plural instances of each dataset). For each, we provide on singular example and one plural example. The odd one out is always the fourth sentence.
- Subject-Verb Agreement: Compositional Dataset: 9500 (Train), 500 (Validation), 1000 (Test)
1. the farmer near the parent is old
2. the surgeon that the architects hate laughs
3. the novel that the dancer likes is new
4. the senator to the side of the parents are young
1. the farmers the taxi drivers love are short
2. the songs the dancers admire are unpopular
3. the surgeons that admire the executives are young
4. the officers that love the assistant is short
Figure 14: Examples from tasks defined over the Inside-Number compositional rule. From top to bottom, we see one example from each of the following tasks: (1) The task used to train base models (2) The task used to train a +/- Inside subnetwork (3) The task used to train a +/- Number subnetwork (4) The evaluation task used to probe for +/- Inside (5) The evaluation task used to probe for +/- Number.
- Subject-Verb Agreement: (Singular/Plural Subject) Dataset: 9500 (Train), 500 (Validation), 1000 (Test)
1. the farmer that the taxi driver hates smile
2. the consultant the guards hate is young
3. the poem that the assistant likes brings joy to people
4. the customers that the chefs like is tall
1. the novels the guard hates are good
2. the teachers across from the parent is young
3. the shows that the taxi driver likes are unpopular
4. the manager across from the parent smile
- Subject-Verb Agreement: (Singular/Plural Verb) Dataset: 9500 (Train), 500 (Validation), 1000 (Test)
1. the game the executives admire is unpopular
2. the surgeon to the side of the taxi drivers smiles
3. the consultants the dancer likes swims
4. the painting that the chefs love are unpopular
1. the customer the assistant loves swim
Figure 15: Examples from tasks defined over the Number-Contact compositional rule. From top to bottom, we see one example from each of the following tasks: (1) The task used to train base models (2) The task used to train a +/- Contact subnetwork (3) The task used to train a +/- Number subnetwork (4) The evaluation task used to probe for +/- Contact (5) The evaluation task used to probe for +/- Number.
2. the surgeons that the executive likes are short
3. the authors that love the chef swim
4. the officers that like the assistant swims
- Subject-Verb Agreement: Test (Singular/Plural Subject) Dataset: 300 (Validation), 300 (Test)
1. the teacher to the side of the taxi driver swims
2. the farmer that the chef likes is young
3. the novel the ministers admire is bad
4. the customers in front of the dancers is old
1. the pictures by the skater interest people
2. the movies the skater admires are bad
3. the pilots in front of the taxi driver are tall
4. the consultant that the dancer likes are old
- Subject-Verb Agreement: Test (Singular/Plural Verb) Dataset: 300 (Validation), 300 (Test)
1. the senator the taxi drivers admire is young
2. the pilot to the side of the dancer smiles
3. the farmer the assistant admires is young
4. the pilot that loves the minister are tall
1. the poems that the chefs hate are bad
2. the surgeons in front of the dancer laugh
3. the surgeons near the taxi drivers smile
4. the farmers behind the architects is short
- Reflexive Anaphora: Compositional Dataset: 2500 (Train), 200 (Validation), 200 (Test)
1. the consultant that the chef loves disguised himself
2. the manager that the architects hate congratulated herself
3. the pilot that the architects admire hurt herself
4. the surgeon that the executives like congratulated themselves
1. the consultants that the guards love injured themselves
2. the senators that the minister admires embarrassed themselves
3. the officers that the assistant likes embarrassed themselves
4. the teacher that the dancer loves embarrassed themselves
- Reflexive Anaphora: (Singular/Plural Antecedent) Dataset: 2500 (Train), 200 (Validation), 200 (Test)
1. the officer that the taxi driver likes doubted herself
2. the author that the architect loves hated himself
3. the manager that the executives love disguised herself
4. the customers that the parent likes disguised himself
1. the authors that the skater hates doubted himself
2. the surgeons that the parents admire hurt themselves
3. the officers that the taxi driver hates injured himself
4. the pilot that the assistant loves hurt themselves
- Reflexive Anaphora: (Singular/Plural Pronoun) Dataset: 2500 (Train), 200 (Validation), 200 (Test)
1. the customer that the ministers hate congratulated herself
2. the surgeons that the dancers like embarrassed himself
3. the authors that the taxi driver hates embarrassed himself
4. the author that the architect admires doubted themselves
1. the officer that the skaters admire embarrassed themselves
2. the senator that the guard likes embarrassed themselves
3. the customer that the ministers love doubted themselves
4. the managers that the guard admires injured herself
- Reflexive Anaphora: Test (Singular/Plural Antecedent) Dataset: 200 (Validation), 200 (Test)
1. the customer that the skater admires hurt herself
2. the consultant that the executive loves disguised herself
3. the manager that the skaters like embarrassed herself
4. the senators that the guard admires injured himself
1. the senators that the architects like embarrassed themselves
2. the authors that the executives admire disguised themselves
3. the surgeons that the taxi driver admires doubted themselves
4. the officer that the parents like congratulated themselves
- Reflexive Anaphora: Test (Singular/Plural Pronoun) Dataset: 200 (Validation), 200 (Test)
1. the pilot that the chefs hate hurt himself
2. the teacher that the taxi drivers love hated herself
3. the senator that the assistant loves embarrassed herself
4. the pilot that the skaters admire embarrassed themselves
1. the authors that the parents admire congratulated themselves
2. the pilots that the chef hates hurt themselves
3. the authors that the parents like congratulated themselves
4. the farmers that the ministers admire injured herself
G Vision Pretraining Details
We pretrain a Resnet50 model and MLP using SimCLR, a contrastive self-supervised learning algorithm (Chen et al., 2020). This algorithm generates two views of an image using random data augmentations, then maximizes the agreement between representations of these views using a contrastive loss function. We use a temperature of 0.07 for this loss.
Our data augmentations include horizontal flips, affine transformations, color jitters, rotations, and grayscale. We train for 100 epochs, using a learning rate of .0005 (which is decayed according to a consine annealing schedule) and a batch size of 256. Images are drawn randomly from the three compositional training sets (Inside-Contact, Number-Contact, Inside-Number). For every rule-following image that is selected, a rule-breaking image from that same dataset is also selected.
We evaluate the Top-5 Accuracy on a held-out validation set after every epoch, and save the weights of the best performing model. Following Chen et al. (2020), we discard the MLP after pretraining, only using the Resnet50 weights to initialize our pretrained models in Section 8.
We adapted the implementation found in Lippe (2022) to implement SimCLR pretraining.
H Subnetwork Sparsity Data
In this section, we provide the raw sparsity statistics for each subnetwork trained in this paper. For every subnetwork, we indicate what stage we started masking (0, 3, or 4), provide the number of Act. Param. in the subnetwork (i.e. the number of 1's in the binary mask), and include the total number of parameters that we mask over (i.e. the number of entries in the binary mask), which is determined by the mask stage.
I Control Experiment: Random Models
Recent work has demonstrated several surprising properties of masks trained on randomly-initialized networks (Ramanujan et al., 2020; Zhou et al., 2019; Wortsman et al., 2020). One might wonder whether the results demonstrated here could be obtained by training a binary mask over randomly-initialized network. If so, this would pose a serious problem for our interpretation of the data: producing the same results in a randomly-initialized network would decouple the behavior of the discovered subnetworks from the representations learned by the underlying base model.
We carry out this experiment as a control. Specifically, we run the exact same mask training procedure used to generate the results in Section 7, except we use randomly-initialized models rather than
TASK	SR	MODEL #	MASK #	STAGE	ACT. PARAM.	TOT. PARAM.
NUMBER-CONTACT	CONTACT	1	1	4	8059720	19398656
NUMBER-CONTACT	CONTACT	1	2	4	8200381	19398656
NUMBER-CONTACT	CONTACT	1	3	4	8199621	19398656
NUMBER-CONTACT	NUMBER	1	1	0	259611	27911360
NUMBER-CONTACT	NUMBER	1	2	0	264197	27911360
NUMBER-CONTACT	NUMBER	1	3	0	438332	27911360
NUMBER-CONTACT	CONTACT	2	1	3	996278	26476544
NUMBER-CONTACT	CONTACT	2	2	3	1010779	26476544
NUMBER-CONTACT	CONTACT	2	3	3	970624	26476544
NUMBER-CONTACT	NUMBER	2	1	4	1831379	19398656
NUMBER-CONTACT	NUMBER	2	2	4	1822564	19398656
NUMBER-CONTACT	NUMBER	2	3	4	1814451	19398656
NUMBER-CONTACT	CONTACT	3	1	4	6044544	19398656
NUMBER-CONTACT	CONTACT	3	2	4	6111795	19398656
NUMBER-CONTACT	CONTACT	3	3	4	6258924	19398656
NUMBER-CONTACT	NUMBER	3	1	0	225866	27911360
NUMBER-CONTACT	NUMBER	3	2	0	685279	27911360
NUMBER-CONTACT	NUMBER	3	3	0	219003	27911360
INSIDE-CONTACT	INSIDE	1	1	4	2418223	19398656
INSIDE-CONTACT	INSIDE	1	2	4	2384556	19398656
INSIDE-CONTACT	INSIDE	1	3	4	2307045	19398656
INSIDE-CONTACT	CONTACT	1	1	4	1029149	19398656
INSIDE-CONTACT	CONTACT	1	2	4	1057758	19398656
INSIDE-CONTACT	CONTACT	1	3	4	938794	19398656
INSIDE-CONTACT	INSIDE	2	1	3	189266	26476544
INSIDE-CONTACT	INSIDE	2	2	3	140762	26476544
INSIDE-CONTACT	INSIDE	2	3	3	184485	26476544
INSIDE-CONTACT	CONTACT	2	1	3	1266369	26476544
INSIDE-CONTACT	CONTACT	2	2	3	1333968	26476544
INSIDE-CONTACT	CONTACT	2	3	3	1078198	26476544
INSIDE-CONTACT	INSIDE	3	1	3	94253	26476544
INSIDE-CONTACT	INSIDE	3	2	3	159807	26476544
INSIDE-CONTACT	INSIDE	3	3	3	231113	26476544
INSIDE-CONTACT	CONTACT	3	1	3	1681559	26476544
INSIDE-CONTACT	CONTACT	3	2	3	1684466	26476544
INSIDE-CONTACT	CONTACT	3	3	3	1682888	26476544
INSIDE-NUMBER	INSIDE	1	1	4	7004974	19398656
INSIDE-NUMBER	INSIDE	1	2	4	7077554	19398656
INSIDE-NUMBER	INSIDE	1	3	4	7077890	19398656
INSIDE-NUMBER	NUMBER	1	1	3	3190833	26476544
INSIDE-NUMBER	NUMBER	1	2	3	3291780	26476544
INSIDE-NUMBER	NUMBER	1	3	3	3559092	26476544
INSIDE-NUMBER	INSIDE	2	1	4	5255315	19398656
INSIDE-NUMBER	INSIDE	2	2	4	5253035	19398656
INSIDE-NUMBER	INSIDE	2	3	4	5151024	19398656
INSIDE-NUMBER	NUMBER	2	1	4	3149872	19398656
INSIDE-NUMBER	NUMBER	2	2	4	2765010	19398656
INSIDE-NUMBER	NUMBER	2	3	4	2675994	19398656
INSIDE-NUMBER	INSIDE	3	1	3	920241	26476544
INSIDE-NUMBER	INSIDE	3	2	3	853147	26476544
INSIDE-NUMBER	INSIDE	3	3	3	974109	26476544
INSIDE-NUMBER	NUMBER	3	1	4	5046083	19398656
INSIDE-NUMBER	NUMBER	3	2	4	5033831	19398656
INSIDE-NUMBER	NUMBER	3	3	4	5086233	19398656
Table 5: Resnet50 subnetwork sparsity statistics. Act. Param. is the number of active parameters in a subnetwork. Tot. Param. is the total number of parameters in the masked layers.
TASK	SR	MODEL #	MASK #	STAGE	ACT. PARAM.	TOT. PARAM.
NUMBER-CONTACT	CONTACT	1	1	3	1184224	26476544
NUMBER-CONTACT	CONTACT	1	2	3	875632	26476544
NUMBER-CONTACT	CONTACT	1	3	3	960475	26476544
NUMBER-CONTACT	NUMBER	1	1	4	703485	19398656
NUMBER-CONTACT	NUMBER	1	2	4	682724	19398656
NUMBER-CONTACT	NUMBER	1	3	4	684961	19398656
NUMBER-CONTACT	CONTACT	2	1	3	833083	26476544
NUMBER-CONTACT	CONTACT	2	2	3	935644	26476544
NUMBER-CONTACT	CONTACT	2	3	3	903822	26476544
NUMBER-CONTACT	NUMBER	2	1	4	682249	19398656
NUMBER-CONTACT	NUMBER	2	2	4	660732	19398656
NUMBER-CONTACT	NUMBER	2	3	4	611524	19398656
NUMBER-CONTACT	CONTACT	3	1	3	898919	26476544
NUMBER-CONTACT	CONTACT	3	2	3	1129741	26476544
NUMBER-CONTACT	CONTACT	3	3	3	765665	26476544
NUMBER-CONTACT	NUMBER	3	1	4	8734131	19398656
NUMBER-CONTACT	NUMBER	3	2	4	8880893	19398656
NUMBER-CONTACT	NUMBER	3	3	4	8900554	19398656
INSIDE-CONTACT	INSIDE	1	1	4	138289	19398656
INSIDE-CONTACT	INSIDE	1	2	4	102788	19398656
INSIDE-CONTACT	INSIDE	1	3	4	64056	19398656
INSIDE-CONTACT	CONTACT	1	1	4	687037	19398656
INSIDE-CONTACT	CONTACT	1	2	4	730767	19398656
INSIDE-CONTACT	CONTACT	1	3	4	594200	19398656
INSIDE-CONTACT	INSIDE	2	1	4	119585	19398656
INSIDE-CONTACT	INSIDE	2	2	4	153621	19398656
INSIDE-CONTACT	INSIDE	2	3	4	141847	19398656
INSIDE-CONTACT	CONTACT	2	1	4	880968	19398656
INSIDE-CONTACT	CONTACT	2	2	4	582672	19398656
INSIDE-CONTACT	CONTACT	2	3	4	549542	19398656
INSIDE-CONTACT	INSIDE	3	1	4	444801	19398656
INSIDE-CONTACT	INSIDE	3	2	4	404687	19398656
INSIDE-CONTACT	INSIDE	3	3	4	388647	19398656
INSIDE-CONTACT	CONTACT	3	1	4	2726236	19398656
INSIDE-CONTACT	CONTACT	3	2	4	2712782	19398656
INSIDE-CONTACT	CONTACT	3	3	4	2704681	19398656
INSIDE-NUMBER	INSIDE	1	1	3	811096	26476544
INSIDE-NUMBER	INSIDE	1	2	3	849964	26476544
INSIDE-NUMBER	INSIDE	1	3	3	781551	26476544
INSIDE-NUMBER	NUMBER	1	1	0	14878394	27911360
INSIDE-NUMBER	NUMBER	1	2	0	14739139	27911360
INSIDE-NUMBER	NUMBER	1	3	0	14919954	27911360
INSIDE-NUMBER	INSIDE	2	1	4	375073	19398656
INSIDE-NUMBER	INSIDE	2	2	4	306550	19398656
INSIDE-NUMBER	INSIDE	2	3	4	314610	19398656
INSIDE-NUMBER	NUMBER	2	1	3	2468331	26476544
INSIDE-NUMBER	NUMBER	2	2	3	3106382	26476544
INSIDE-NUMBER	NUMBER	2	3	3	3122791	26476544
INSIDE-NUMBER	INSIDE	3	1	0	660344	27911360
INSIDE-NUMBER	INSIDE	3	2	0	714676	27911360
INSIDE-NUMBER	INSIDE	3	3	0	692889	27911360
INSIDE-NUMBER	NUMBER	3	1	3	3369673	26476544
INSIDE-NUMBER	NUMBER	3	2	3	3661479	26476544
INSIDE-NUMBER	NUMBER	3	3	3	3278225	26476544
Table 6: Resnet50 + SimCLR Subnetwork sparsity statistics. Act. Param. is the number of active parameters in a subnetwork. Tot. Param. is the total number of parameters in the masked layers.
TASK	SR	MODEL #	MASK #	STAGE	ACT. PARAM.	TOT. PARAM.
NUMBER-CCONTACT	CONTACT	1	1	4	12415313	46399488
NUMBER-CCONTACT	CONTACT	1	2	4	11886093	46399488
NUMBER-CCONTACT	CONTACT	1	3	4	11994752	46399488
NUMBER-CCONTACT	NUMBER	1	1	0	364048	71222464
NUMBER-CCONTACT	NUMBER	1	2	0	356776	71222464
NUMBER-CCONTACT	NUMBER	1	3	0	430018	71222464
NUMBER-CCONTACT	CONTACT	2	1	4	8156499	46399488
NUMBER-CCONTACT	CONTACT	2	2	4	8188321	46399488
NUMBER-CCONTACT	CONTACT	2	3	4	8730014	46399488
NUMBER-CCONTACT	NUMBER	2	1	0	489546	71222464
NUMBER-CCONTACT	NUMBER	2	2	0	479722	71222464
NUMBER-CCONTACT	NUMBER	2	3	0	405563	71222464
NUMBER-CCONTACT	CONTACT	3	1	4	11238193	46399488
NUMBER-CCONTACT	CONTACT	3	2	4	11246123	46399488
NUMBER-CCONTACT	CONTACT	3	3	4	11084672	46399488
NUMBER-CCONTACT	NUMBER	3	1	4	792483	46399488
NUMBER-CCONTACT	NUMBER	3	2	4	681226	46399488
NUMBER-CCONTACT	NUMBER	3	3	4	834326	46399488
INSIDE-CCONTACT	INSIDE	1	1	3	177339	67108864
INSIDE-CCONTACT	INSIDE	1	2	3	147071	67108864
INSIDE-CCONTACT	INSIDE	1	3	3	232306	67108864
INSIDE-CCONTACT	CONTACT	1	1	3	1100205	67108864
INSIDE-CCONTACT	CONTACT	1	2	3	966156	67108864
INSIDE-CCONTACT	CONTACT	1	3	3	1043718	67108864
INSIDE-CCONTACT	INSIDE	2	1	0	875532	71222464
INSIDE-CCONTACT	INSIDE	2	2	0	493009	71222464
INSIDE-CCONTACT	INSIDE	2	3	0	619362	71222464
INSIDE-CCONTACT	CONTACT	2	1	3	5898635	67108864
INSIDE-CCONTACT	CONTACT	2	2	3	6213208	67108864
INSIDE-CCONTACT	CONTACT	2	3	3	5909038	67108864
INSIDE-CCONTACT	INSIDE	3	1	3	557265	67108864
INSIDE-CCONTACT	INSIDE	3	2	3	330289	67108864
INSIDE-CCONTACT	INSIDE	3	3	3	710769	67108864
INSIDE-CCONTACT	CONTACT	3	1	3	4632439	67108864
INSIDE-CCONTACT	CONTACT	3	2	3	4376935	67108864
INSIDE-CCONTACT	CONTACT	3	3	3	4964646	67108864
INSIDE-NUMBER	INSIDE	1	1	3	2081068	67108864
INSIDE-NUMBER	INSIDE	1	2	3	2106222	67108864
INSIDE-NUMBER	INSIDE	1	3	3	2007091	67108864
INSIDE-NUMBER	NUMBER	1	1	4	3541726	46399488
INSIDE-NUMBER	NUMBER	1	2	4	3560812	46399488
INSIDE-NUMBER	NUMBER	1	3	4	3583605	46399488
INSIDE-NUMBER	INSIDE	2	1	0	2242819	71222464
INSIDE-NUMBER	INSIDE	2	2	0	1963701	71222464
INSIDE-NUMBER	INSIDE	2	3	0	1330134	71222464
INSIDE-NUMBER	NUMBER	2	1	3	5749408	67108864
INSIDE-NUMBER	NUMBER	2	2	3	5511381	67108864
INSIDE-NUMBER	NUMBER	2	3	3	5472792	67108864
INSIDE-NUMBER	INSIDE	3	1	0	808884	71222464
INSIDE-NUMBER	INSIDE	3	2	0	851288	71222464
INSIDE-NUMBER	INSIDE	3	3	0	731729	71222464
INSIDE-NUMBER	NUMBER	3	1	0	3487441	71222464
INSIDE-NUMBER	NUMBER	3	2	0	3528010	71222464
INSIDE-NUMBER	NUMBER	3	3	0	4330284	71222464
Table 7: Wide Resnet50 Subnetwork sparsity statistics. Act. Param. is the number of active parameters in a subnetwork. Tot. Param. is the total number of parameters in the masked layers.
TASK	SR	MODEL #	MASK #	STAGE	ACT. PARAM.	TOT. PARAM.
(S) SV AGREEMENT	SUBJECT	1	1	0	1246922	12845056
(S) SV AGREEMENT	SUBJECT	1	2	0	1242347	12845056
(S) SV AGREEMENT	SUBJECT	1	3	0	2128824	12845056
(S) SV AGREEMENT	VERB	1	1	0	1668369	12845056
(S) SV AGREEMENT	VERB	1	2	0	2600240	12845056
(S) SV AGREEMENT	VERB	1	3	0	3353398	12845056
(S) SV AGREEMENT	SUBJECT	2	1	0	2728632	12845056
(S) SV AGREEMENT	SUBJECT	2	2	0	2663842	12845056
(S) SV AGREEMENT	SUBJECT	2	3	0	2681724	12845056
(S) SV AGREEMENT	VERB	2	1	0	951132	12845056
(S) SV AGREEMENT	VERB	2	2	0	1044003	12845056
(S) SV AGREEMENT	VERB	2	3	0	1084848	12845056
(S) SV AGREEMENT	SUBJECT	3	1	0	328484	12845056
(S) SV AGREEMENT	SUBJECT	3	2	0	720899	12845056
(S) SV AGREEMENT	SUBJECT	3	3	0	323764	12845056
(S) SV AGREEMENT	VERB	3	1	3	1794939	6553600
(S) SV AGREEMENT	VERB	3	2	3	1702597	6553600
(S) SV AGREEMENT	VERB	3	3	3	1567156	6553600
(P) SV AGREEMENT	SUBJECT	1	1	0	69748	12845056
(P) SV AGREEMENT	SUBJECT	1	2	0	68641	12845056
(P) SV AGREEMENT	SUBJECT	1	3	0	52336	12845056
(P) SV AGREEMENT	VERB	1	1	3	640889	6553600
(P) SV AGREEMENT	VERB	1	2	3	477101	6553600
(P) SV AGREEMENT	VERB	1	3	3	656202	6553600
(P) SV AGREEMENT	SUBJECT	2	1	0	119037	12845056
(P) SV AGREEMENT	SUBJECT	2	2	0	125594	12845056
(P) SV AGREEMENT	SUBJECT	2	3	0	122828	12845056
(P) SV AGREEMENT	VERB	2	1	0	215555	12845056
(P) SV AGREEMENT	VERB	2	2	0	202185	12845056
(P) SV AGREEMENT	VERB	2	3	0	56134	12845056
(P) SV AGREEMENT	SUBJECT	3	1	0	86084	12845056
(P) SV AGREEMENT	SUBJECT	3	2	0	49694	12845056
(P) SV AGREEMENT	SUBJECT	3	3	0	166614	12845056
(P) SV AGREEMENT	VERB	3	1	0	149398	12845056
(P) SV AGREEMENT	VERB	3	2	0	221074	12845056
(P) SV AGREEMENT	VERB	3	3	0	133149	12845056
Table 8: BERT Subject-Verb Agreement Subnetwork sparsity statistics. Act. Param. is the number of active parameters in a subnetwork. Tot. Param. is the total number of parameters in the masked layers.
models trained on compositional tasks. In Section 7, each (model, subroutine) pair had its own set of masking hyperparameters. We use these same hyperparameters for each (randomly-initialized model, subroutine) pair in order to make the results as comparable as possible.
In Figure 16 and 17, we observe that masking random networks produces distinctly different patterns of results than we presented in Section 7. Though it is possible find a subnetwork that computes a target subroutine and not the other subroutine, the ablation results do not follow the pattern that one would expect of a compositional model. This accords with Ramanujan et al. (2020), which demonstrates that training a binary mask over a randomly-weighted network can still produce performant subnetworks. The ablation results indicate that these subnetworks are not causally implicated in model behavior. In the case of Resnet50 (Figure 16, Bottom) we observe that ablating the learned subnetworks collapses performance to chance for all tasks. In the case of BERT-Small (Figure 17, Bottom), we observe that ablating the learned subnetworks oftentimes yields high performance on Test Target Subroutine and low performance on Test Other Subroutine, which is the opposite of the expected trend for a compositional model. Thus, we can be more confident that