SlowGuess/NAACL_2022_OCR huggingface dataset:
Dataset	Train	Test	Development
IMDB	25,000	25,000	-
Yelp	560,000	38,000	-
QC	5,452	500	-
DPedia	560,000	70,000	-
CR	3,594	400	-
ATIS	4,978	893	-
SST1	8,544	2,210	1,101
MR	9,596	1,066	-
ADE	23,516	-	-
B Parameter Settings for Text Classification Model
In this section, we will introduce the parameter settings of the neural network-based models explored in Section 3.2. The optimizer is AdamW for the four mdoels. The settings of other parameters are shown in Tab. 5.
Table 4: Statistics of datasets.
Parameter	BERT	CNN	LSTM	LSTMAtt
learning rate	2*e-5	1*e-4	1*e-3	1*e-3
batch size	4	4	32	32
word emb	-	Word2vec	GloVe	GloVe
word emb size	-	300	300	300
hidden size	768	120	256	256
max sent len	512	-	-	-
filter size	-	1,3,5	-	-
Table 5: the parameters of four models.
C Characterization of Datasets
C.1 Inherent Feature
Label balance  $(\phi_{\mathrm{bal}})$ : The label balance metric measures the variance between the ideal and the true label distribution:  $\phi_{\mathrm{bal}} = (c_t - c_s) / c_s$ , where the  $c_t$  and  $c_s$  are the true and ideal label information entropy (Shannon, 1948), respectively.
C.2 Lexical Feature
Type-Token Ratio  $(\phi_{\mathrm{ttr}})$ : TTR (Richards, 1987) is a way to measure the documents lexical richness:  $\phi_{\mathrm{ttr}} = n_{\mathrm{type}} / n_{\mathrm{token}}$ , where the  $n_{\mathrm{type}}$  is the number
of unique words, and  $n_{token}$  is the number of tokens. We use lexical richness 19 to calculate the TTR for each sentence and then average them.
Language Mixedness Ratio  $(\phi_{\mathrm{lmix}})$ : The proportion of sentence that contains other languages in the whole dataset. To detect the mixed other languages, we utilize the models proposed by Joulin et al. (2016b) for language identification from fast-Text (Joulin et al., 2016a) which can recognize 176 languages.
Pointwise Mutual Information  $(\phi_{\mathrm{pmi}})$ : is a measurement to calculate the correlation between variables. Specifically, for a word in one class  $\phi_{\mathrm{pmi(c,w)}} = \log \left(\frac{p(c,w)}{p(c)p(w)}\right)$ , where  $p(c)$  is the proportion of the tokens belonging to label  $c$ ,  $p(w)$  is the proportion of the word  $w$ , and  $p(c,w)$  is the proportion of the word  $w$  which belongs to class  $c$ . For every class, all the  $\phi_{\mathrm{pmi(c,w)}}$ , larger than zero, are added to get the sum, which serve as the dataset's pmi. Finally,  $\phi_{\mathrm{pmi}}$  is calculated by dividing the sum by the numbers of pairs(c,w) of the train dataset. We pick up the top-ten words sorted by  $\phi_{\mathrm{pmi(c,w)}}$  in all classes, then the ration related to the class-related word  $(\phi_{\mathrm{r}_{\mathrm{pmi}}})$  is calculated by dividing the number of samples who contain the top-ten words by the total samples in the train set.
C.3 Semantic Feature
Grammar errors ratio  $(\phi_{\mathrm{gerr}})$ : The proportion of words with grammatical errors in the whole dataset. We adopt the detection tool  $^{20}$  to recognize words with grammatical errors. We first compute the grammar errors ratio for each sentence:  $n / m$ , where the n and m denote the number of words with grammatical errors and the number of the token for a sentence, averaging them.
Flesch Reading Ease  $(\phi_{\mathrm{fre}})$ : Flesch Reading Ease calculated by textstat  $^{22}$  is a way to describe the simplicity of a reader who can read a text. First, we calculate the  $\phi_{\mathrm{fre}}$  for each sample, and then average them as the dataset's feature. Then we pick out the samples whose score below 60, then the ration related to the low score samples  $(\phi_{\mathrm{fre}})$  is calculated by dividing the number of the picked samples by the total samples in the train set.