SlowGuess/NAACL_2022_OCR huggingface dataset:
Yang Xiao $^{1}$ , Jinlan Fu $^{2*}$ , See-Kiong Ng $^{2}$ , Pengfei Liu $^{3}$ $^{1}$ Fudan University,  $^{2}$ National University of Singapore,  $^{3}$ Carnegie Mellon University yangxiaocq12@gmail.com, {jinlan,seekiong}@nus.edu.sg, pliu3@cs.cmu.edu
In this paper, we ask the research question of whether all the datasets in the benchmark are necessary. We approach this by first characterizing the distinguishability of datasets when comparing different systems. Experiments on 9 datasets and 36 systems show that several existing benchmark datasets contribute little to discriminating top-scoring systems, while those less used datasets exhibit impressive discriminative power. We further, taking the text classification task as a case study, investigate the possibility of predicting dataset discrimination based on its properties (e.g., average sentence length). Our preliminary experiments promisingly show that given a sufficient number of training experimental records, a meaningful predictor can be learned to estimate dataset discrimination over unseen datasets. We released all datasets with features explored in this work on DataLab.
In natural language processing (NLP) tasks, there are often datasets that we use as benchmarks against which to evaluate machine learning models, either explicitly defined such as GLUE (Wang et al., 2018) and XTREME (Hu et al., 2020) or implicitly bound to the task (e.g., DPedia (Zhang et al., 2015) has become a default dataset for evaluating of text classification systems). Given this mission, one important feature of a good benchmark dataset is the ability to statistically differentiate diverse systems (Bowman and Dahl, 2021). With large pre-trained models consistently improving state-of-the-art performance on NLP tasks (Devlin et al., 2018; Lewis et al., 2019), the performances of many of them have reached a plateau (Zhong et al., 2020; Fu et al., 2020). In other words, it is challenging to discriminate a better model using existing datasets (Wang et al., 2019). In this context, we ask the question:
Figure 1: Illustrate different datasets' distinguishing ability w.r.t top-scoring systems characterized by our measure  $\log (\lambda_{\mathrm{sva}})$  on text classification and their corresponding citations.
are all benchmark's datasets necessary? We use the text classification task as a case study and try to answer the following two sub-questions:
RQ1: How can we quantify the distinguishing ability of benchmark datasets? To answer this question, we first design measures with varying calculation difficulties (§4) to judge datasets' discrimination ability based on top-scoring systems' performances. By exploring correlations among different measures, we then evaluate how reliable a dataset's discrimination is when discrimination is calculated solely based on overall results that top-scoring systems have achieved and generalize this measure to other NLP tasks. Fig. 1 illustrates how different text classification datasets are ranked (the bottom one) based on measures devised in this work (a smaller value suggests lower discrimination) and the corresponding citations of these datasets (the upper one). One can observe that: (i) The highly-cited dataset DBpedia (Zhang et al., 2015) (more than 3,000 times since 2015) shows the worst discriminative power. (ii) By contrast, dataset like ADE (Gurulingappa et al., 2012) (less than 200 times since 2012) does better in distinguishing top-scoring systems, suggesting that some of the relatively neglected datasets are actually valuable in distinguishing models. This phenomenon
\*Corresponding author 1https://databab.nlpedia.ai
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2930 - 2941 July 10-15, 2022 ©2022 Association for Computational Linguistics
shows the significance of quantifying the discriminative ability of datasets: it can not only help us to eliminate those with lower discrimination from commonly-used datasets (e.g., DBpedia), but also help us to recognize the missing pearl in seldom used datasets (e.g., ADE and ATIS (Hemphill et al., 1990)).
RQ2: Can we try to predict the discriminative power of the dataset? Given a dataset, we investigate if we can judge its ability to distinguish models based on its characteristics (e.g., average sentence length), which is motivated by the scenario where a new dataset has just been constructed without sufficient top-scoring systems to calculate discrimination defined in RQ1. To answer this question, inspired by recent literature on performance prediction (Domhan et al., 2015; Turchi et al., 2008; Birch et al., 2008; Xia et al., 2020; Ye et al., 2021), we conceptualize this problem as a discrimination regression task. We define 11 diverse features to characterize a text classification dataset and regress its discrimination scores using different parameterized models. Preliminary experiments (\$5.4) indicate that a meaningful regressor can be learned to estimate the discrimination of unseen datasets without actual training using top-scoring systems.
We brief takeaways in this work based on our observations:
(1) Not all datasets in benchmark are necessary in terms of model selection<sup>2</sup>: empirical results show that following datasets struggle at discriminating current top-scoring systems: STS-B and SST-2 from GLUE (Wang et al., 2018); BUCC and PAWX-X from XTREME, which is consistent with the concurrent work (Ruder et al., 2021) (\$4.3.2).
(2) In regard to single-task benchmark datasets, for Chinese Word Segmentation task, there are multiple datasets (MSR, CityU, CTB) (Tseng et al., 2005; Jin and Chen, 2008) that exhibit much worse discriminative ability, suggesting that: future works on this task are encouraged to either (i) adopt other datasets to evaluate their systems or (ii) at least make significant test if using these datasets. Similar observations happen in the dataset CoNLL-2003 (Sang and De Meulder, 2003) from Named Entity Recognition task and MultiNLI
(Williams et al., 2017) from natural language inference task (§4.3.2).
(3) Some seldom used datasets such as ADE from text classification are actually better at distinguishing top-performing systems, which highlights an interesting and necessary future direction: how to identify infrequently-used but valuable (better discrimination) datasets for NLP tasks, especially in the age of dataset's proliferation?4 (§4.2)
(4) Quantifying a dataset's discrimination (w.r.t top-scoring systems) by calculating the statistical measures (defined in §4.1.2) from leaderboard's results is a straightforward and effective way. But for those datasets without rich leaderboard results,5 predicting the discrimination based on datasets' characteristics would be an promising direction (§4.3.1).
(1) We try to quantify the discrimination ability for datasets by designing two variance-based measures. (2) We systematically investigate 4 text classification models on 9 datasets, providing the newest baseline performance for those seldom used datasets. All datasets and their features are released on DataLab (Xiao et al., 2022). (3) We study several popular NLP benchmarks, including GLUE, XTREME, NLI, and so on. Some valuable suggestions and observations will make research easier.
Benchmarks for NLP In order to conveniently keep themselves updated with the research progress, researchers recently are actively building evaluation benchmarks for diverse tasks so that they could make a comprehensive comparison of systems, and use a leaderboard to record the evolving process of the systems of different NLP tasks, such as SQuAD (Rajpurkar et al., 2016), GLUE (Wang et al., 2018), XTREME (Hu et al., 2020), GEM (Gehrmann et al., 2021) and GENIE (Khashabi et al., 2021). Despite their utility, more recently, Bowman and Dahl (2021) highlight that unreliable and biased systems score so highly on standard benchmarks that there is little room for researchers who develop better systems to demonstrate their improvements. In this paper, we make a pilot study on meta-evaluating benchmark evalu
$^{2}$ Caveat: Annotated datasets are always valuable, because the supervision signals provided there can not only help us directly train a system for specific use case, but also provide good supervised transfer for related tasks (Sanh et al., 2021).
3We randomly select 10 recently published papers (from ACL/EMNLP) that utilized these datasets and found only 2 of them perform significant test.
4https://paperswithcode.com/datasets
5The measure can keeps updated as the top-scoring systems of the leaderboard evolves, which can broaden its practical applicability
ation datasets and quantitatively characterize their discrimination in different top-scoring systems.
Performance Prediction Performance prediction is the task of estimating a system's performance without the actual training process. With the recent booming of the number of machine learning models (Goodfellow et al., 2016) and datasets, the technique of performance prediction becomes rather important when applied to different scenarios ranging from early stopping training iteration (Kolachina et al., 2012), architecture searching (Domhan et al., 2015), and attribution analysis (Birch et al., 2008; Turchi et al., 2008). In this work, we aim to calculate a dataset's discrimination without actual training top-scoring systems on it, which can be formulated as a performance prediction problem.
3.1 Task and Dataset
Text classification aims to assign a label defined beforehand to a given input document. In the experiment, we choose nine datasets, and their statistics can be found in the Appendix A.
- IMDB (Maas et al., 2011) consists of movie reviews with binary classes.
- Yelp (Zhang et al., 2015) is a part of the Yelp Dataset Challenge 2015 data.
- CR (Hu and Liu, 2004) is a product review dataset with binary classes.
- MR (Pang and Lee, 2005) is a movie review dataset collected from Rotten Tomatoes.
- SST1 (Socher et al., 2013) is collected from HTML files of Rotten Tomatoes reviews with fully labeled parse trees.
- DBpedia14 (Zhang et al., 2015) is a dataset for ontology classification collected from DBpedia.
- ATIS (Hemphill et al., 1990) is an intent detection dataset that contains audio recordings of flight reservations.
- QC (Li and Roth, 2002) is a question classification dataset.
- ADE (Gurulingappa et al., 2012) is a subset of "Adverse Drug Reaction Data".
We re-implement 4 top-scoring systems with typical neural architectures for each dataset. The
brief introduction of the four models is as follows.
- LSTM (Hochreiter and Schmidhuber, 1997) is a widely used sentence encoder. Here, we adopt the bidirectional LSTM.
- LSTMAtt is proposed by Lin et al. (2017) that designed the self-attention mechanism to extract different aspects of features for a sentence.
- BERT (Devlin et al., 2018) was utilized to fine-tuning on our text classification datasets.
- CNN is a CNN-based text classification model (Kim, 2014) was exploited in our work.
Except for BERT, the other three models (e.g. LSTM) are initialized by GloVe (Pennington et al., 2014) or Word2Vec (Mikolov et al., 2013) pretrained word embeddings. When the performance on the dev set doesn't improve within 20 epochs, the training will be stopped, and the best performing model will be kept. More detailed model parameter settings can be found in the Appendix B.
4 How to Characterize Discrimination?
To achieve this goal, we design measures based on the performance of different models for a dataset.
4.1 Measures
We design several measures to judge dataset's distinguishing ability based on the performances that top-performing systems have achieved on it.
Specifically, given a dataset  $D$  together with  $k$  topscoring model performance list  $\mathbf{v} = [v_{1},\dots ,v_{k}]$  we define the following measures.
4.1.1 Performance Variance
We use the standard deviation to quantify the degree of variation or dispersion of a set of performance values. A larger value of  $\lambda_{\mathrm{var}}$  suggests that the discrimination of the given dataset is more significant.  $\lambda_{\mathrm{var}}$  can be defined as:
$$
\lambda_ {\text {v a r}} = \operatorname {S t d} (\mathbf {v}), \tag {1}
$$
where  $\operatorname{Std}(\cdot)$  is the function to compute the standard deviation. Assume that the performance list  $(k = 3)$  on dataset  $D$  is  $\mathbf{v} = [88,92,93]$ , we can get  $\lambda_{\mathrm{var}} = 2.65$ .
<sup>6</sup>We mainly focus on neural network-based models, since most top-scoring systems in the leaderboard are based on deep learning.
${}^{7}$  A dataset's discrimination is defined w.r.t top-scoring models from a leaderboard,keeping itself updated with systems' evolution.
4.1.2 Scaled Performance Variance
For the above measure, it can only reflect the variances of the performance of different models, without considering whether the model's performance is close to the upper limit (e.g.,  $100\%$  accuracy) on a given data set. To address this problem, we defined a modified variance by scaling  $\lambda_{\mathrm{var}}$  with the difference between the upper limit performance  $u$  and average performance  $\operatorname{Avg}(\mathbf{v})$  of  $\mathbf{v}$ .
$$
\lambda_ {\mathrm {s v a}} = \lambda_ {\mathrm {v a r}} (u - \operatorname {A v g} (\mathbf {v})). \tag {2}
$$
In practice,  $u$  can be defined flexibly based on tasks' metrics. For example, in text classification task,  $u$  could be  $100\%$  (w.r.t F1 or accuracy), while in summarization task,  $u$  could be the results of oracle sentences (w.r.t ROUGE). Intuitively, given a performance list on text classification dataset:  $\mathbf{v} = [88,92,93]$ , we can obtain the  $\lambda_{\mathrm{sva}} = 23.81$ .
4.1.3 Hit Rate
The previous two measures quantify dataset's discriminative ability w.r.t  $k$  top-performing systems in an indirect way (i.g., solely based on the overall results of different models). However, sometimes, small variance does not necessarily mean that the dataset fail to distinguish models, as long as the difference between models is statistically significant. To overcome this problem, we borrow the idea of bootstrap-based significant test (Koehn, 2004) and define the measure hit rate, which quantify the degree to which a given dataset could successfully differentiate  $k$  top-scoring systems.
Specifically, we take all  $\binom{k}{2}$  pairs of systems  $(m_i$  and  $m_j)$  and compare their performances on a subset of test samples  $D_{t}$  that is generated using paired bootstrap re-sampling. Let  $v_{i}(D) > v_{j}(D)$  be the performance of  $m_{1}$  and  $m_{2}$  on the full test set, we define  $P(m_i,m_j)$  as the frequency of  $v_{i}(D_{t}) > v_{j}(D_{t})$  over all  $T$  times of re-sampling  $(t = 1,\dots ,T)$ . Then we have
$$
\lambda_ {\mathrm {h i t}} = \frac {1}{\binom {k} {2}} \sum P \left(m _ {i}, m _ {j}\right) \tag {3}
$$
Metric Comparison The first two metrics, performance variance and scaled performance variance, are relative easily to obtain since they only require holistic performances of different top-scoring models on a given dataset, which can be conveniently collected from existing leaderboards. By
contrast, although the metric hit rate can directly reflect dataset's ability in discriminating diverse systems, its calculation not only require more fine-grained information of system prediction but also complicated bootstrap re-sampling process.
4.2 Exp-I: Exploring Correlation Between Variance and Hit Rate
The goal of this experiment is to investigate the reliability of the variance-based discrimination measures (e.g.,  $\lambda_{\mathrm{sva}}$ ), which are easier to obtain, by calculating its correlation with significant test-based measure  $\lambda_{\mathrm{hit}}$ , which is costly to get. Since the implementation of  $\lambda_{\mathrm{hit}}$  relies on the bootstrap-based significant test, we choose text classification as the tested and re-implement 4 classification models (defined in Sec. 3.2) on 9 datasets. The performance and the distinction degree on the 9 text classification dataset are shown in Tab. 1.  $\lambda_{\mathrm{var}}$  and  $\lambda_{\mathrm{sva}}$  measures are designed based on performance variance, even if BERT always achieves the best performance on the same dataset, it will not affect the observed results from our experiments.
Correlation measure Here, we adopt the Spearman rank correlation coefficient (Zar, 1972) to describe the correlation between our variance-based measures and the hit rate measure  $\lambda_{\mathrm{hit}}$ .
$$
S _ {\lambda} = \operatorname {S p e a r m a n} (q, \lambda_ {\text {h i t}}), \tag {4}
$$
where the  $q$  can be  $\lambda_{\mathrm{var}}$  or  $\lambda_{\mathrm{sva}}$
Result (1)  $\lambda_{\mathrm{var}}$  and  $\lambda_{\mathrm{sva}}$  are strong correlative  $(S_{\lambda} > 0.6)$  with  $\lambda_{\mathrm{hit}}$  respectively, which suggests that variance-based metrics could be a considerably reliable alternatives of significant test-based metric.
(2) Spearman  $(\lambda_{\mathrm{var}},\lambda_{\mathrm{hit}}) >$  Spearman  $(\lambda_{\mathrm{sva}},\lambda_{\mathrm{hit}})$ , which indicate that comparing with  $\lambda_{\mathrm{sva}}$ , dataset discrimination characterized by  $\lambda_{\mathrm{var}}$  is more acceptable for  $\lambda_{\mathrm{hit}}$ . The reason can be attributed to that the designing of the measure  $\lambda_{\mathrm{hit}}$  does not consider the upper limit of the model's performance.
(3) DPmedia and Yelp are commonly used text classification datasets, while they have the worst ability to discriminate the top-scoring models since they get the lowest value of  $\lambda_{\mathrm{var}}$  and  $\lambda_{\mathrm{sva}}$ . By contrast, these two seldom used datasets ADE and ATIS show the better discriminative ability.
4.3 Exp-II: Evaluation of Other Benchmarks
4.3.1 Popular Benchmark Datasets
We also investigate how benchmark datasets from other NLP task perform using two devised mea
For example, given a test set with 1000 samples, we sample  $80\%$  subset from it and repeat this process T times.
Method	BERT	LSTMAtrr	LSTM	CNN	λhit	λvar	λsva
SST1	54.12	43.80	47.60	44.80	0.88	4.65	243.56
CR	91.75	83.25	82.50	84.25	0.91	4.27	62.17
MR	85.55	79.92	79.80	82.00	0.86	2.69	48.83
QC	97.19	90.36	89.96	92.17	0.92	3.32	25.18
IMDB	93.34	89.45	89.65	87.81	0.87	2.33	23.18
ADE	93.48	92.90	92.65	89.54	0.78	1.77	13.90
ATIS	97.64	97.42	97.31	94.62	0.78	1.42	4.63
Yelp	97.52	96.60	96.60	95.46	0.81	0.84	2.91
DPedia	99.27	99.01	99.05	98.75	0.68	0.22	0.21
Spearman	0.83	0.73
Table 1: Illustration the 4 models' performance and discrimination degree (characterized by  $\lambda_{\mathrm{hit}}$ ,  $\lambda_{\mathrm{var}}$ , and  $\lambda_{\mathrm{sva}}$ ) on 9 text classification datasets. The two correlation coefficients pass the significance test ( $p < 0.05$ ).  $\lambda_{\mathrm{var}}$  and  $\lambda_{\mathrm{sva}}$  measures are designed based on performance variance.
sures. Specifically, we collected three single-task and two multitask benchmarks. For the single-task benchmarks, we collect the top-performing models in a specific period for each dataset, provided by Paperswithcode. For the multitask benchmarks, here, the GLUE and XTREME are considered in this work. Since Paperswithcode provided 5 models for each dataset in most case, for fairness and uniformity, we keep top-5 models for both single-task and multitask benchmark datasets.
Named Entity Recognition (NER) aims to identify named entities of an input text, for which we choose 5 top-scoring systems on 6 datasets and collect results from Paperswithcode.
Chinese Word Segmentation (CWS) aims to detect the boundaries of Chinese words in a sentence. We select 5 top-scoring systems on 8 datasets and collect results from Paperswithcode.
Natural Language Inference (NLI) targets at predicting whether a premise sentence can infer the hypothesis sentence. We select 5 top-performing models on 4 datasets from Paperswithcode.
GLUE (Wang et al., 2018) covers 9 sentence- or sentence-pair tasks with different dataset sizes, text genres, and degrees of difficulty. Fig. 2-(a) shows the tasks/datasets that are considered in GLUE.
XTREME (Hu et al., 2020) is the first benchmark that evaluates models across a wide variety of languages and tasks. The tasks/datasets that are covered by XTREME are shown in Fig. 2-(b).
4.3.2 Results and Analysis
Fig. 2 shows the results of dataset quality measure by  $\lambda_{\mathrm{var}}$  and  $\lambda_{\mathrm{sva}}$ . We detail several main observations:
-  $\lambda_{\mathrm{var}}$  and  $\lambda_{\mathrm{sva}}$  have consistent evaluation results for both single-task (CWS, NER, NLI) and multitask (GLUE, XTREME) benchmarks.
- For the XTREME benchmark, BUCC and PAWSX have lowest  $\lambda_{\mathrm{var}}$  and  $\lambda_{\mathrm{sva}}$ , which suggest that they are hardly to discriminate the top-performing systems. Moreover, these two data sets will be removed from the new version of the XTREME leaderboard called XTREME-R (Ruder et al., 2021). This consistent observation also shows the effectiveness of our measure.
- For GLUE benchmark, CoLA, QQP, and RTE have the excellent ability to distinguish different top-scoring models (with higher  $\lambda_{\mathrm{var}}$  and  $\lambda_{\mathrm{sva}}$ ), while the SST-2 and STS-B perform worse.
- For CWS benchmarks, there is a larger gap between the value of  $\lambda_{\mathrm{var}}$  and  $\lambda_{\mathrm{sva}}$ , which indicate that the performance of top-scoring models considered are close to  $100\%$ . Furthermore, MSR, CityU and CTB are not suitable as benchmarks since they have poor discrimination ability with  $\lambda_{\mathrm{sva}} < 0$ . So as MultiNLI for NLI task.
- CoNLL 2003 is a widely used NER dataset, but it is the lowest quality dataset under our dataset quality measure. The reason can be attributed to contain much annotation errors (Fu et al., 2020) in the CoNLL 2003 dataset, which makes its performance reach the bottleneck.
5 Can we Predict Discrimination?
Although metrics  $\lambda_{\mathrm{var}}$ ,  $\lambda_{\mathrm{sva}}$  ease the burden for us to calculate the datasets' discrimination, one major limitation is: given a new dataset without results from leaderboards, we need to train multiple top-scoring systems and calculate corresponding results on it, which is computationally expensive. To alleviate this problem, in this section, we focus on text
9https://paperswithcode.com/
10https://gluebenchmark.com/
11https://sites.research.google/xtreme
(b) XTREME
(c) CWS
(d) NER
(e) NLI
Figure 2: The dataset discrimination characterized by  $\log (\lambda_{\mathrm{var}})$  (the logarithm for better visualization) (blue) and  $\log (\lambda_{\mathrm{sva}})$  (pink) on five popular NLP benchmarks.
classification task and investigate the possibility of estimating datasets' discrimination solely based on their characteristics without actual training systems on them.
5.1 Task Formulation
5.1.1 Regression-based Task Formulation
We formulate it as a performance prediction problem (Birch et al., 2008; Xia et al., 2020; Ye et al., 2021). Formally, we refer to  $\mathcal{M}$ ,  $D^{tr}$ ,  $D^{te}$ ,  $\mathcal{S}$  as the machine learning system, training data, test data and training strategy respectively. The goal of performance prediction is to estimate actual performance  $y$  without actual training by using features of  $\mathcal{M}$ ,  $\mathcal{D}^{tr}$ ,  $\mathcal{D}^{te}$ , and  $\mathcal{S}$ .
$$
\hat {y} = \hat {f} \left(\Phi_ {\mathcal {M}}, \Phi_ {\mathcal {D} ^ {t r}}, \Phi_ {\mathcal {D} ^ {t e}}, \Phi_ {\mathcal {S}}; \hat {\Theta}\right) \tag {5}
$$
where  $\hat{y}$  denotes estimated prediction and  $\Phi (\cdot)$  is a feature extractor. Following Xia et al. 2020, we only use the features of the datasets as variables and adapt it to our discriminative prediction scenario, we can obtain:
$$
\hat {\lambda} = \hat {f} \left(\Phi_ {\mathcal {D} ^ {t r}}, \Phi_ {\mathcal {D} ^ {t e}}; \hat {\Theta}\right), \tag {6}
$$
where  $\hat{\lambda}$  denotes predicted variance defined in §4.1.2 such as  $\lambda_{var}$  or  $\lambda_{sva}$ .
5.1.2 Ranking-based Task Formulation