SlowGuess/NAACL_2021_OCR huggingface dataset:
$$
\begin{array}{l} (1 0) = \frac {g ^ {\prime \prime} (1)}{2} \epsilon^ {T} \left(\sum_ {i} \frac {f _ {i} ^ {\prime} f _ {i} ^ {\prime T}}{f _ {i}}\right) \epsilon + \\ \frac {g ^ {\prime} (1)}{2} \epsilon^ {T} \left(\sum_ {i} f _ {i} ^ {\prime \prime}\right) \epsilon \\ = \frac {g ^ {\prime \prime} (1)}{2} \sum_ {i} \frac {\epsilon^ {T} f _ {i} ^ {\prime} f _ {i} ^ {\prime T} \epsilon}{f _ {i}} \\ = \frac {g ^ {\prime \prime} (1)}{2} \epsilon^ {T} J ^ {T} \operatorname {d i a g} \left(\frac {1}{f}\right) J \epsilon , \tag {11} \\ \end{array}
$$
where  $J$  is the input-output Jacobian of  $f$ , and  $\operatorname{diag}\left(\frac{1}{f}\right)$  is a diagonal matrix with its diagonal elements equal to the vector  $\frac{1}{f}$ .
The statistics of all datasets used in this paper is summarized in Table 6. The top four datasets in Table 6 are used for training and the matched dev sets are used for both validation and in-domain evaluation. The remaining datasets are used only for
Corpus	Train	Dev	Test	Metrics
SST-2	67k	872	-	Acc
MNLI	393k	20k	-	Acc
SQuAD v1	87.6k	10.5k	-	EM/F1
SQuAD v2	130.3k	11.9k	-	EM/F1
IMDB	-	-	25k	Acc
BioASQ	-	1504	-	EM/F1
AddSent	-	-	3560	EM/F1
AddOneSent	-	-	1787	EM/F1
HANS	-	-	30k	Acc
Table 6: Experiment dataset summary. The top four datasets are used for training and the matched dev sets are used for both validation and in-domain evaluation. The rest five datasets are used only for evaluation purpose.
evaluation purpose. Among the evaluation datasets, other than BioASQ (Tsatsaronis et al., 2015) from MRQA where the development set is used for evaluation, the corresponding test set is used for evaluating model out-of-domain generalization.