SlowGuess/NAACL_2022_OCR huggingface dataset:
Statistics of the GLUE benchmark is summarized in Table 6. Statistics of the question answering datasets (SQuAD v1.1 and SQuAD v2.0) are summarized in Table 5.
#Train	#Validation
SQuAD v1.1	87,599	10,570
SQuAD v2.0	130,319	11,873
Table 5: Statistics of the SQuAD dataset.
We use Adam (Kingma and Ba, 2015) as the optimizer with parameters  $(\beta_{1},\beta_{2}) = (0.9,0.999)$ . We employ gradient clipping with a maximum gradient norm 1.0, and we choose weight decay from  $\{0,0.01,0.1\}$ . The learning rate is chosen from  $\{1\times 10^{-5},2\times 10^{-5},3\times 10^{-5},4\times 10^{-5}\}$ , and we do not use learning rate warm-up. We train the model for  $\{3,4,5,10\}$  epochs with a batch size chosen from  $\{8,16,32,64\}$ . The weight of the distillation loss  $\lambda_{\mathrm{distil}}$  is chosen from  $\{1,2,3,4,5\}$ .
Hyper-parameters for distilling BERT-base is summarized in Table 7. We use Adam (Kingma and Ba, 2015) as the optimizer with parameters  $(\beta_{1},\beta_{2}) = (0.9,0.999)$ . We employ gradient clipping with a maximum gradient norm 1.0. We do not use learning rate warm-up. For the GLUE benchmark, we use a maximum sequence length of 512 except MNLI and QQP, where we set the maximum sequence length to 128. For the SQuAD datasets, the maximum sequence length is set to 384.
Corpus	Task	#Train	#Dev	#Test	#Label	Metrics
Single-Sentence Classification (GLUE)
CoLA	Acceptability	8.5k	1k	1k	2	Matthews corr
SST	Sentiment	67k	872	1.8k	2	Accuracy
Pairwise Text Classification (GLUE)
MNLI	NLI	393k	20k	20k	3	Accuracy
RTE	NLI	2.5k	276	3k	2	Accuracy
QQP	Paraphrase	364k	40k	391k	2	Accuracy/F1
MRPC	Paraphrase	3.7k	408	1.7k	2	Accuracy/F1
QNLI	QA/NLI	108k	5.7k	5.7k	2	Accuracy
Text Similarity (GLUE)
STS-B	Similarity	7k	1.5k	1.4k	1	Pearson/Spearman corr
Table 6: Summary of the GLUE benchmark.
lr	batch	epoch	decay	λdistill
RTE	1 × 10-5	1 × 8	10	0.01	1.0
CoLA	2 × 10-5	1 × 8	10	0.0	3.0
MRPC	3 × 10-5	1 × 8	5	0.0	2.0
SST-2	2 × 10-5	2 × 8	5	0.0	1.0
QNLI	2 × 10-5	4 × 8	5	0.0	2.0
QQP	3 × 10-5	8 × 8	5	0.0	1.0
MNLI	5 × 10-5	8 × 8	5	0.0	5.0
SQuAD v1.1	3 × 10-5	4 × 8	5	0.01	2.0
SQuAD v2.0	3 × 10-5	2 × 8	4	0.1	1.0
Table 7: Hyper-parameters for distilling BERT-base. From left to right: learning rate; batch size (2 × 8 means we use a batch size of 2 and 8 GPUs); number of training epochs; weight decay; and weight of the distillation loss.