SlowGuess/NeurIPS_2023_OCR huggingface dataset:
TASK	SR	MODEL #	MASK #	STAGE	ACT. PARAM.	TOT. PARAM.
(S) ANAPHORA	PRONOUN	1	1	0	370568	12845056
(S) ANAPHORA	PRONOUN	1	2	0	372203	12845056
(S) ANAPHORA	PRONOUN	1	3	0	369123	12845056
(S) ANAPHORA	ANTECEDENT	1	1	0	169120	12845056
(S) ANAPHORA	ANTECEDENT	1	2	0	150554	12845056
(S) ANAPHORA	ANTECEDENT	1	3	0	231819	12845056
(S) ANAPHORA	PRONOUN	2	1	0	968021	12845056
(S) ANAPHORA	PRONOUN	2	2	0	971063	12845056
(S) ANAPHORA	PRONOUN	2	3	0	970910	12845056
(S) ANAPHORA	ANTECEDENT	2	1	0	108544	12845056
(S) ANAPHORA	ANTECEDENT	2	2	0	110871	12845056
(S) ANAPHORA	ANTECEDENT	2	3	0	108347	12845056
(S) ANAPHORA	PRONOUN	3	1	0	79069	12845056
(S) ANAPHORA	PRONOUN	3	2	0	79031	12845056
(S) ANAPHORA	PRONOUN	3	3	0	82788	12845056
(S) ANAPHORA	ANTECEDENT	3	1	0	1854552	12845056
(S) ANAPHORA	ANTECEDENT	3	2	0	1848327	12845056
(S) ANAPHORA	ANTECEDENT	3	3	0	1874968	12845056
(P) ANAPHORA	PRONOUN	1	1	0	325597	12845056
(P) ANAPHORA	PRONOUN	1	2	0	417498	12845056
(P) ANAPHORA	PRONOUN	1	3	0	642805	12845056
(P) ANAPHORA	ANTECEDENT	1	1	0	286739	12845056
(P) ANAPHORA	ANTECEDENT	1	2	0	336572	12845056
(P) ANAPHORA	ANTECEDENT	1	3	0	405887	12845056
(P) ANAPHORA	PRONOUN	2	1	0	24818	12845056
(P) ANAPHORA	PRONOUN	2	2	0	24805	12845056
(P) ANAPHORA	PRONOUN	2	3	0	24855	12845056
(P) ANAPHORA	ANTECEDENT	2	1	3	1154161	6553600
(P) ANAPHORA	ANTECEDENT	2	2	3	1183436	6553600
(P) ANAPHORA	ANTECEDENT	2	3	3	1159462	6553600
(P) ANAPHORA	PRONOUN	3	1	0	144186	12845056
(P) ANAPHORA	PRONOUN	3	2	0	151531	12845056
(P) ANAPHORA	PRONOUN	3	3	0	153897	12845056
(P) ANAPHORA	ANTECEDENT	3	1	0	4606842	12845056
(P) ANAPHORA	ANTECEDENT	3	2	0	5134758	12845056
(P) ANAPHORA	ANTECEDENT	3	3	0	5041888	12845056
Table 9: BERT Anaphora Subnetwork sparsity statistics. Act. Param. is the number of active parameters in a subnetwork. Tot. Param. is the total number of parameters in the masked layers.
the main results presented in Section 7 reflect the internal mechanisms of trained models, and are not epiphenomenal artifacts of training binary masks over networks.
I.1 Statistical Analysis of Main Results vs. Random Results
In order to assess whether the results given by random models are significantly different from our main results, we fit a generalized linear model (GLM) with robust clustered standard errors for each combination of model architecture, compositional task, and subroutine. This GLM includes a dummy variable indicating whether the results are from a subnetwork or an ablated model, a dummy variable indicating whether the base model is trained or random, and it clusters observations by base model. For language experiments, we collapse across singular and plural instances of the same subroutine. The coefficient of the Trained vs. Random dummy variable in this model assesses whether the performance of the discovered subnetworks are significantly different in the trained and random conditions. From this model, we can also perform a linear hypothesis test to assess whether ablated model performance is significantly different between the trained and random conditions. Table 12 provides the relevant statistics. Across the board, we see that there is often a significant difference
TASK	SR	MODEL #	MASK #	STAGE	ACT. PARAM.	TOT. PARAM.
(S) SV AGREEMENT	SUBJECT	1	1	0	59736	12845056
(S) SV AGREEMENT	SUBJECT	1	2	0	839770	12845056
(S) SV AGREEMENT	SUBJECT	1	3	0	620869	12845056
(S) SV AGREEMENT	VERB	1	1	3	65229	6553600
(S) SV AGREEMENT	VERB	1	2	3	65356	6553600
(S) SV AGREEMENT	VERB	1	3	3	65220	6553600
(S) SV AGREEMENT	SUBJECT	2	1	0	511359	12845056
(S) SV AGREEMENT	SUBJECT	2	2	0	70725	12845056
(S) SV AGREEMENT	SUBJECT	2	3	0	555174	12845056
(S) SV AGREEMENT	VERB	2	1	0	16940	12845056
(S) SV AGREEMENT	VERB	2	2	0	39218	12845056
(S) SV AGREEMENT	VERB	2	3	0	19321	12845056
(S) SV AGREEMENT	SUBJECT	3	1	0	8514	12845056
(S) SV AGREEMENT	SUBJECT	3	2	0	8613	12845056
(S) SV AGREEMENT	SUBJECT	3	3	0	8593	12845056
(S) SV AGREEMENT	VERB	3	1	3	64969	6553600
(S) SV AGREEMENT	VERB	3	2	3	65098	6553600
(S) SV AGREEMENT	VERB	3	3	3	64955	6553600
(P) SV AGREEMENT	SUBJECT	1	1	0	45792	12845056
(P) SV AGREEMENT	SUBJECT	1	2	0	45527	12845056
(P) SV AGREEMENT	SUBJECT	1	3	0	45616	12845056
(P) SV AGREEMENT	VERB	1	1	0	16800	12845056
(P) SV AGREEMENT	VERB	1	2	0	24013	12845056
(P) SV AGREEMENT	VERB	1	3	0	23988	12845056
(P) SV AGREEMENT	SUBJECT	2	1	0	47651	12845056
(P) SV AGREEMENT	SUBJECT	2	2	0	47502	12845056
(P) SV AGREEMENT	SUBJECT	2	3	0	47811	12845056
(P) SV AGREEMENT	VERB	2	1	3	100005	6553600
(P) SV AGREEMENT	VERB	2	2	3	100100	6553600
(P) SV AGREEMENT	VERB	2	3	3	100029	6553600
(P) SV AGREEMENT	SUBJECT	3	1	3	81133	6553600
(P) SV AGREEMENT	SUBJECT	3	2	3	81203	6553600
(P) SV AGREEMENT	SUBJECT	3	3	3	81218	6553600
(P) SV AGREEMENT	VERB	3	1	0	15302	12845056
(P) SV AGREEMENT	VERB	3	2	0	9423	12845056
(P) SV AGREEMENT	VERB	3	3	0	15900	12845056
Table 10: BERT + LM Subject-Verb Agreement Subnetwork sparsity statistics. Act. Param. is the number of active parameters in a subnetwork. Tot. Param. is the total number of parameters in the masked layers.
between ablating the discovered subnetworks in random vs. trained models, even with a small sample size.
J Pruned Model Analysis
In this section, we analyze the structural compositionality of models after they have been pruned. We analyze the impact of pruning on structural compositionality on Resnet50 models trained on Number-Contact. We use continuous sparsification to train binary masks over the three Number-Contact Resnet50 models analyzed in the main paper, resulting in three sparse models that perform well on the Number-Contact task. We search over mask initialization and learning rate hyperparameters to find the best pruning configuration for each model. Then, we run the same structural compositionality analysis described in the main paper. We see from Figure 18 that the results on the pruned Resnet50 models closely resemble those from the full Resnet50 models.
TASK	SR	MODEL #	MASK #	STAGE	ACT. PARAM.	TOT. PARAM.
(S) ANAPHORA	PRONOUN	1	1	0	6590452	12845056
(S) ANAPHORA	PRONOUN	1	2	0	6702195	12845056
(S) ANAPHORA	PRONOUN	1	3	0	6519670	12845056
(S) ANAPHORA	ANTECEDENT	1	1	0	832044	12845056
(S) ANAPHORA	ANTECEDENT	1	2	0	833149	12845056
(S) ANAPHORA	ANTECEDENT	1	3	0	835278	12845056
(S) ANAPHORA	PRONOUN	2	1	3	193677	6553600
(S) ANAPHORA	PRONOUN	2	2	3	198612	6553600
(S) ANAPHORA	PRONOUN	2	3	3	198956	6553600
(S) ANAPHORA	ANTECEDENT	2	1	0	39355	12845056
(S) ANAPHORA	ANTECEDENT	2	2	0	27784	12845056
(S) ANAPHORA	ANTECEDENT	2	3	0	34674	12845056
(S) ANAPHORA	PRONOUN	3	1	0	36495	12845056
(S) ANAPHORA	PRONOUN	3	2	0	1458843	12845056
(S) ANAPHORA	PRONOUN	3	3	0	75100	12845056
(S) ANAPHORA	ANTECEDENT	3	1	0	648936	12845056
(S) ANAPHORA	ANTECEDENT	3	2	0	680218	12845056
(S) ANAPHORA	ANTECEDENT	3	3	0	760770	12845056
(P) ANAPHORA	PRONOUN	1	1	0	11374	12845056
(P) ANAPHORA	PRONOUN	1	2	0	11251	12845056
(P) ANAPHORA	PRONOUN	1	3	0	11369	12845056
(P) ANAPHORA	ANTECEDENT	1	1	0	1152444	12845056
(P) ANAPHORA	ANTECEDENT	1	2	0	1152518	12845056
(P) ANAPHORA	ANTECEDENT	1	3	0	1149693	12845056
(P) ANAPHORA	PRONOUN	2	1	0	11327	12845056
(P) ANAPHORA	PRONOUN	2	2	0	11321	12845056
(P) ANAPHORA	PRONOUN	2	3	0	11275	12845056
(P) ANAPHORA	ANTECEDENT	2	1	0	43274	12845056
(P) ANAPHORA	ANTECEDENT	2	2	0	44220	12845056
(P) ANAPHORA	ANTECEDENT	2	3	0	45632	12845056
(P) ANAPHORA	PRONOUN	3	1	0	28866	12845056
(P) ANAPHORA	PRONOUN	3	2	0	28887	12845056
(P) ANAPHORA	PRONOUN	3	3	0	29101	12845056
(P) ANAPHORA	ANTECEDENT	3	1	0	4292104	12845056
(P) ANAPHORA	ANTECEDENT	3	2	0	4350952	12845056
(P) ANAPHORA	ANTECEDENT	3	3	0	4048117	12845056
Table 11: BERT + LM Anaphora Subnetwork sparsity statistics. Act. Param. is the number of active parameters in a subnetwork. Tot. Param. is the total number of parameters in the masked layers.
COMP. TASK	MODEL	SR	RANDOM COEF. Z	LINEAR HYPOTHESIS χ2
IN. CONT.	RN50	INSIDE	-1.36	2.44
IN. CONT.	RN50	CONTACT	-1.73.	1.17
IN. NUM.	RN50	INSIDE	-1.92.	29.20***
IN. NUM.	RN50	NUMBER	-1.41	48.95***
CONT. NUM.	RN50	CONTACT	-0.51	57.47***
CONT. NUM.	RN50	NUMBER	-0.26	0.54
SV AGR	BERT	SUBJ.	2.13*	19.17***
SV AGR	BERT	VERB	2.61**	70.08***
ANAPHORA	BERT	PRONOUN	2.46*	122.64***
ANAPHORA	BERT	ANTECEDENT	0.24	17.87***
Table 12: Statistics from two factor GLM with robust clustered standard errors. . indicates significance at  $p = .1$ , * indicates significance at  $p = .05$ , ** indicates significance at  $p = .01$ , *** indicates significance at  $p = .001$
Resnet50 Ablated Networks: Inside-Contact
Figure 16: Results from training masks over a randomly-initialized Resnet50. (Top) Plots displaying differences in performance. (Middle) Plots displaying subnetwork performance on each task. (Bottom) Plots displaying ablated model performance on each task. Across the board, we see that masks over random networks can produce subnetworks that achieve better accuracy on an Test Target Subroutine than on Test Other Subroutine, but that ablating these subnetworks results in (equally) poor performance on both of these datasets.
K Subnetwork Overlap Analysis
In this section, we analyze the overlap in the subnetworks that were discovered within the same base model. We perform this analysis on one model, a Resnet50 trained on the Inside-Number task. From Figure 4, we see that this model appears to exhibit structural compositionality. We analyze this model because subnetwork masking started at the same layer for both subroutines, which allows for a straightforward comparison of the overlap between subnetworks. All results are shown in Figure 19.
Following previous work (Csordás et al., 2021), we compute the per-layer intersection over union (IoU) to quantify subnetwork overlap. First, we do this for each of the three subnetworks discovered for the Inside subroutine. See these results in Table 13. Next, we compute the same for the three subnetworks discovered for the Number subroutine. See these results in Table 14. The Inside subroutine gives near ceiling agreement, while the Number task exhibits much lower agreement. This indicates that the subnetworks we uncover are somewhat noisy for the Number subroutine, but not for the Inside subroutine. Finally, we compute the intersection of each set of subnetworks, and compute the per-layer intersection over union between tasks. See Table 15. We see very low agreement between tasks, especially before the final MLP. Notably, this between-task agreement is consistently
BERT Small Ablated Networks: SV Agreement
Figure 17: Results from training masks over a randomly-initialized BERT-Small. (Top) Plots displaying differences in performance. (Middle) Plots displaying subnetwork performance on each task. (Bottom) Plots displaying ablated model performance on each task. Across the board, we see that masks over random networks can produce subnetworks that achieve better accuracy on an Test Target Subroutine than on Test Other Subroutine. Surprisingly, ablating these subnetworks still results in better accuracy on an Test Target Subroutine than on Test Other Subroutine.
lower than both within-task agreement values for each layer. This reinforces our interpretation that these subnetworks are organized in a modular fashion within the base models.
LAYER	IOU
BACKBONE.LAYER4.0.CONV1	0.974
BACKBONE.LAYER4.0.CONV2	0.963
BACKBONE.LAYER4.0.CONV3	0.971
BACKBONE.LAYER4.0.DOWNSAMPLE.0	0.971
BACKBONE.LAYER4.1.CONV1	0.970
BACKBONE.LAYER4.1.CONV2	0.960
BACKBONE.LAYER4.1.CONV3	0.969
BACKBONE.LAYER4.2.CONV1	0.975
BACKBONE.LAYER4.2.CONV2	0.971
BACKBONE.LAYER4.2.CONV3	0.970
MLP.MODEL.0	0.994
MLP.MODEL.2	0.989
Table 13: IoU computed over the three discovered subnetworks in for the Inside subroutine
LAYER	IOU
BACKBONE.LAYER4.0.CONV1	0.370
BACKBONE.LAYER4.0.CONV2	0.236
BACKBONE.LAYER4.0.CONV3	0.264
BACKBONE.LAYER4.0.DOWNSAMPLE.0	0.257
BACKBONE.LAYER4.1.CONV1	0.287
BACKBONE.LAYER4.1.CONV2	0.208
BACKBONE.LAYER4.1.CONV3	0.218
BACKBONE.LAYER4.2.CONV1	0.197
BACKBONE.LAYER4.2.CONV2	0.129
BACKBONE.LAYER4.2.CONV3	0.162
MLP.MODEL.0	0.516
MLP.MODEL.2	0.411
Table 14: IoU computed over the three discovered subnetworks in for the Number subroutine
LAYER	IoU
BACKBONE.LAYER4.0.CONV1	0.122
BACKBONE.LAYER4.0.CONV2	0.062
BACKBONE.LAYER4.0.CONV3	0.087
BACKBONE.LAYER4.0.DOWNSAMPLE.0	0.057
BACKBONE.LAYER4.1.CONV1	0.070
BACKBONE.LAYER4.1.CONV2	0.054
BACKBONE.LAYER4.1.CONV3	0.076
BACKBONE.LAYER4.2.CONV1	0.034
BACKBONE.LAYER4.2.CONV2	0.027
BACKBONE.LAYER4.2.CONV3	0.055
MLP.MODEL.0	0.237
MLP.MODEL.2	0.321
Table 15: IoU computed over the intersections of the subnetworks discovered for each task.
Figure 18: Structural compositionality analysis using pruned Resnet50 models on the Number-Contact task. We see that these results closely match those found in the main paper.
Figure 19: The results of subnetwork overlap analysis for one base model trained on the Inside-Number task. These results show that within-task subnetwork overlap is substantially higher than between-task subnetwork overlap, measured by intersection over union (IoU). Within a task, we are computing the IoU between 3 subnetworks trained for that task. Between tasks, we first take the intersection of all 3 subnetworks trained for each task, and then compute the IoU of the intersections.