SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 1: Experimental details about IMP, task-agnostic mask training (TAMT) and fine-tuning. For pre-training, we report the number of tokens as “# of Train/Eval Samples”. “Dev Loss” denotes the loss of MLM or KD on the dev set. During fine-tuning, evaluation is performed every “Eval Iter” training iterations.
20%	30%	40%	50%	60%	70%	80%	90%
IMP	2.79K	5.58K	8.38K	11.17K	13.96K	16.75K	19.54K	22.34K
TAMT	3K	6K	8K	11K	14K	17K	20K	22K
Table 2: Pre-training iterations for IMP and TAMT subnetworks at  ${20}\%  \sim  {90}\%$  sparsity.
passing TAMT-MLM on all the five tasks. In contrast, TAMT-MLM is better at higher sparsities. 3) At  $80\%$  sparsity, the searching efficiency of the KD objective is not desirable, which requires more pretraining steps to outperform IMP on CoLA, STS-B, SQuAD and the overall performance. However, the advantage of TAMT-MLM is still obvious at  $80\%$  sparsity.
A.5 Subnetwork Similarity at Different Sparsities
In Section 4.2.5, we analyse the similarity between subnetworks at  $70\%$  sparsity. In Fig. 17, we present additional results of subnetworks at different sparsities. We can see that the general pattern, as discussed in Section 4.2.5, is the same across  $60\%$ ,  $70\%$  and  $80\%$  sparsities. However, as sparsity grows, different searching methods become more distinct from each other. For instance, the similarity between TAMT-MLM and IMP subnetworks decreases from 0.75 at  $60\%$  sparsity to less than 0.6 at  $80\%$  sparsity. This is understandable because the higher the sparsity, the lower the probability that two subnetworks will share the same weight.
B More Information about Experimental Setups
B.1 Datasets and Evaluation
For pre-training, we adopt the WikiText-103 dataset for language modeling. WikiText-103
is a collection of articles on Wikipedia and has over 100M tokens. Such data scale is relatively small for PLM pre-training. However, we find that it is sufficient for mask training and IMP to discover subnetworks with perceivable downstream improvement.
For the downstream tasks, we use six datasets from the GLUE benchmark and the SQuAD v1.1 dataset  ${}^{2}$  . The GLUE benchmark is intended to train, evaluate, and analyze NLU systems. Our experiments include the tasks of CoLA for linguistic acceptability, SST-2 for sentiment analysis, RTE and MNLI for natural language inference, MRPC and STS-B for semantic matching/similarity. The SQuAD dataset is for the task of question answering. It consists of questions posed by crowdworkers on a set of Wikipedia articles. Tab. 1 summarizes the dataset statistics and evaluation metrics. All the datasets are in English language.
The hyper-parameters for pre-training and finetuning are shown in Tab. 1. The pre-training setups of IMP basically follow (Chen et al., 2020), except for the number of training epochs, because we use different pre-training datasets. Since we aim at finding universal PLM subnetworks that are agnostic to the downstream tasks, we do not perform hyper-parameter search for TAMT based on the downstream performance. The pre-training hyper
wikipedia.org/wiki/Wikipedia:Text_of_Creative_Commons_Attribution-ShareAlike_3.0(Unported_License)
$^{2}$ SQuAD is available under the CC BY-SA 4.0 license.
WikiText-103 is available under the Creative Commons Attribution-ShareAlike License (https://en.
IMP	TAMT-MLM	TAMT-KD
BERTBASE	4h6m26s	3h54m58s	4h46m46s
RoBERTaBASE	4h33m9s	4h17m15s	4h51m55s
Table 3: Pre-training time (w/o evaluation during training) of IMP and TAMT on a single on a single 32GB Nvidia V100 GPU. "h", "m" and "s" denote hour, minute and second, respectively. The pre-training iterations are 22.34K and 22K for IMP and TAMT respectively, which correspond to the  $90\%$  sparsity in Tab. 2.
parameters in Tab. 1 are determined as they can guarantee stable convergence on the pre-training tasks.
For fair comparison between TAMT and IMP, we control the number of pre-training iterations (i.e., the number of gradient descent steps) to be the same. Considering that the IMP subnetworks of different sparsities are obtained from different pre-training iterations, we adjust the pre-training iterations of TAMT accordingly. Specifically, we set the maximum number of pre-training epochs to 2 for IMP, which equals to  $27.92\mathrm{K}$  training iterations. Thus, the sparsity is increased by  $10\%$  every  $2.792\mathrm{K}$  iterations. Tab. 2 shows the number of pretraining iterations for IMP and TAMT subnetworks at  $20\% \sim 90\%$  sparsity. Note that the final training iteration does not equal to  $27.92\mathrm{K}$  at  $100\%$  sparsity according to Tab. 2. This is because we prune to  $10\%$  sparsity at the  $0^{th}$  iteration, which follows the implementation of Chen et al. (2020).
The hyper-parameters for downstream finetuning follow the standard setups of (Wolf et al., 2020; Chen et al., 2020). We use the same set of hyper-parameters for all the subnetworks, as well as the full models. We perform evaluations during the fine-tuning process, and the best result is reported as the downstream performance.
Training and evaluation are implemented on Nvidia V100 GPU. The codes are based on the Pytorch framework<sup>3</sup> and the huggingface Transformers library<sup>4</sup> (Wolf et al., 2020). Tab. 3 shows the pre-training time of IMP and TAMT.
Figure 11: Downstream performance of BERT and RoBERTa subnetworks found using OMP. Shadowed areas denote standard deviations.
Figure 12: MLM dev loss and single task downstream performance of  $\mathrm{BERT}_{\mathrm{BASE}}$  subnetworks. The results of TAMT are obtained from the masks along the training process, and the results of IMP and Rand are from different seeds.
Figure 13: KD dev loss and single task downstream performance of  $\mathrm{BERT}_{\mathrm{BASE}}$  subnetworks. The results of TAMT are obtained from the masks along the training process, and the results of IMP and Rand are from different seeds.
Figure 14: The downstream performance of  $60\%$  sparse  $\mathrm{BERT}_{\mathrm{BASE}}$  subnetworks on each single task, with increased pre-training iterations.
Figure 15: The downstream performance of  $70\%$  sparse  $\mathrm{BERT}_{\mathrm{BASE}}$  subnetworks on each single task, with increased pre-training iterations.
Figure 16: The downstream performance of  $80\%$  sparse  $\mathrm{BERT}_{\mathrm{BASE}}$  subnetworks on each single task, with increased pre-training iterations.
Figure 17: Upper: The downstream performance of masks with varying distances from the OMP mask. Shadowed areas denote standard deviations. Lower: The similarity between masks searched using different methods. The masks are the same as those used to report the main results. The suffix numbers indicate different seeds. The masks are from  $\mathrm{BERT}_{\mathrm{BASE}}$ .