SlowGuess/ACL_2020_OCR huggingface dataset:
Mengzhou Xia, Antonios Anastasopoulos, Ruochen Xu, Yiming Yang, Graham Neubig
{mengzhox, aanastas, yiming, gneubig}@cs.cmu.edu
ruochenx@gmail.com
Given the complexity of combinations of tasks, languages, and domains in natural language processing (NLP) research, it is computationally prohibitive to exhaustively test newly proposed models on each possible experimental setting. In this work, we attempt to explore the possibility of gaining plausible judgments of how well an NLP model can perform under an experimental setting, without actually training or testing the model. To do so, we build regression models to predict the evaluation score of an NLP experiment given the experimental settings as input. Experimenting on 9 different NLP tasks, we find that our predictors can produce meaningful predictions over unseen languages and different modeling architectures, outperforming reasonable baselines as well as human experts. Going further, we outline how our predictor can be used to find a small subset of representative experiments that should be run in order to obtain plausible predictions for all other experimental settings.
Natural language processing (NLP) is an extraordinarily vast field, with a wide variety of models being applied to a multitude of tasks across a plenitude of domains and languages. In order to measure progress in all these scenarios, it is necessary to compare performance on test datasets representing each scenario. However, the cross-product of tasks, languages, and domains creates an explosion of potential application scenarios, and it is infeasible to collect high-quality test sets for each. In addition, even for tasks where we do have a wide variety of test data, e.g. for well-resourced tasks such as machine translation (MT), it is still
computationally prohibitive as well as not environmentally friendly (Strubell et al., 2019) to build and test on systems for all languages or domains we are interested in. Because of this, the common practice is to test new methods on a small number of languages or domains, often semi-arbitrarily chosen based on previous work or the experimenters' intuition.
As a result, this practice impedes the NLP community from gaining a comprehensive understanding of newly-proposed models. Table 1 illustrates this fact with an example from bilingual lexicon induction, a task that aims to find word translation pairs from cross-lingual word embeddings. As vividly displayed in Table 1, almost all the works report evaluation results on a different subset of language pairs. Evaluating only on a small subset raises concerns about making inferences when comparing the merits of these methods: there is no guarantee that performance on English-Spanish (EN-ES, the only common evaluation dataset) is representative of the expected performance of the models over all other language pairs (Anastasopoulos and Neubig, 2020). Such phenomena lead us to consider if it is possible to make a decently accurate estimation for the performance over an untested language pair without actually running the NLP model to bypass the computation restriction.
Toward that end, through drawing on the idea of characterizing an experiment from Lin et al. (2019), we propose a framework, which we call NLPERF, to provide an exploratory solution. We build regression models, to predict the performance on a particular experimental setting given past experimental records of the same task, with each record consisting of a characterization of its training dataset and a performance score of the corresponding metric. Concretely, in §2, we start with a partly populated table (such as the one from
$^{1}$ Code, data and logs are publicly available at https: //github.com/xiamengzhou/NLPerf.
8625
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8625-8646 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
BLI Method	Evaluation Set
DE-EN	EN-DE	ES-EN	EN-ES	FR-EN	EN-FR	IT-EN	EN-IT	EN-PT	EN-RU	ES-DE	PT-RU
Zhang et al. (2017)	?	✓	✓	✓	?	?	✓	?	?	?	?	?
Chen and Cardie (2018)	✓	✓	✓	✓	✓	✓	✓	✓	✓	?	✓	?
Yang et al. (2019)	✓	✓	✓	✓	✓	✓	✓	?	?	?	?	?
Heyman et al. (2019)	?	✓	?	✓	?	✓	?	✓	?	?	?	?
Huang et al. (2019)	?	?	✓	✓	✓	✓	?	?	?	?	?	?
Artetxe et al. (2019)	✓	✓	✓	✓	✓	✓	?	?	?	✓	?	?
Table 1: An illustration of the comparability issues across methods and multiple evaluation datasets from the Bilingual Lexicon Induction task. Our prediction model can reasonably fill in the blanks, as illustrated in Section 4.
Table 1) and attempt to infer the missing values with the predictor. We begin by introducing the process of characterizing an NLP experiment for each task in §3. We evaluate the effectiveness and robustness of NLPERF by comparing to multiple baselines, human experts, and by perturbing a single feature to simulate a grid search over that feature (§4). Evaluations on multiple tasks show that NLPERF is able to outperform all baselines. Notably, on a machine translation (MT) task, the predictions made by the predictor turn out to be more accurate than human experts.
An effective predictor can be very useful for multiple applications associated with practical scenarios. In §5, we show how it is possible to adopt the predictor as a scoring function to find a small subset of experiments that are most representative of a bigger set of experiments. We argue that this will allow researchers to make informed decisions on what datasets to use for training and evaluation, in the case where they cannot experiment on all experimental settings. Last, in §6, we show that we can adequately predict the performance of new models even with a minimal number of experimental records.
In this section we formalize the problem of predicting performance on supervised NLP tasks. Given an NLP model of architecture  $\mathcal{M}$  trained over dataset(s)  $\mathcal{D}$  of a specific task involving language(s)  $\mathcal{L}$  with a training procedure (optimization algorithms, learning rate scheduling etc.)  $\mathcal{P}$ , we can test the model on a test dataset  $\mathcal{D}'$  and get a score  $S$  of a specific evaluation metric. The resulting score will surely vary depending on all the above mentioned factors, and we denote this relation as  $g$ :
$$
\mathcal {S} _ {\mathcal {M}, \mathcal {P}, \mathcal {L}, \mathcal {D}, \mathcal {D} ^ {\prime}} = g (\mathcal {M}, \mathcal {P}, \mathcal {L}, \mathcal {D}, \mathcal {D} ^ {\prime}). \tag {1}
$$
In the ideal scenario, for each test dataset  $\mathcal{D}'$  of a specific task, one could enumerate all different settings and find the one that leads to the best performance. As mentioned in Section §1, however, such a brute-force method is computationally infeasible. Thus, we turn to modeling the process and formulating our problem as a regression task by using a parametric function  $f_{\theta}$  to approximate the true function  $g$  as follows:
$$
\hat {\mathcal {S}} _ {\mathcal {M}, \mathcal {P}, \mathcal {L}, \mathcal {D}, \mathcal {D} ^ {\prime}} = f _ {\theta} ([ \Phi_ {\mathcal {M}}; \Phi_ {\mathcal {P}}; \Phi_ {\mathcal {L}}; \Phi_ {\mathcal {D}}; \Phi_ {\mathcal {D} ^ {\prime}} ])
$$
where  $\Phi_{*}$  denotes a set of features for each influencing factor.
For the purpose of this study, we mainly focus on dataset and language features  $\Phi_{\mathcal{L}}$  and  $\Phi_{\mathcal{D}}$ , as this already results in a significant search space, and gathering extensive experimental results with fine-grained tuning over model and training hyperparameters is both expensive and relatively complicated. In the cases where we handle multiple models, we only use a single categorical model feature to denote the combination of model architecture and training procedure, denoted as  $\Phi_{\mathcal{C}}$ . We still use the term model to refer to this combination in the rest of the paper. We also omit the test set features, under the assumption that the data distributions for training and testing data are the same (a fairly reasonable assumption if we ignore possible domain shift). Therefore, for all experiments below, our final prediction function is the following:
$$
\hat {\mathcal {S}} _ {\mathcal {C}, \mathcal {L}, \mathcal {D}} = f _ {\theta} ([ \Phi_ {\mathcal {C}}; \Phi_ {\mathcal {L}}; \Phi_ {\mathcal {D}} ])
$$
In the next section we describe concrete instantiations of this function for several NLP tasks.
3 NLP Task Instantiations
To build a predictor for NLP task performance, we must 1) select a task, 2) describe its featurization, and 3) train a predictor. We describe details of these three steps in this section.
8626
Task	Dataset Citation	Source
Langs	Target
Langs	Transfer
Langs	# Models	# EXs	Task
Metric
Wiki-MT	Schwenk et al. (2019)	39	39	-	single	995	BLEU
TED-MT	Qi et al. (2018)	54	1	-	single	54	BLEU
TSF-MT	Qi et al. (2018)	54	1	54	single	2862	BLEU
TSF-PARSING	Nivre et al. (2018)	-	30	30	single	870	Accuracy
TSF-POS	Nivre et al. (2018)	-	26	60	single	1531	Accuracy
TSF-EL	Rijhwani et al. (2019)	-	9	54	single	477	Accuracy
BLI	Lample et al. (2018)	44	44	-	3	88×3	Accuracy
MA	McCarthy et al. (2019)	-	66	-	6	107×6	F1
UD	Zeman et al. (2018a)	-	53	-	25	72×25	F1
Table 2: Statistics of the datasets we use for training predictors. # EXs denote the total number of experiment instances; Task Metric reflects how the models are evaluated.
Tasks We test on tasks including bilingual lexicon induction (BLI); machine translation trained on aligned Wikipedia data (Wiki-MT), on TED talks (TED-MT), and with cross-lingual transfer for translation into English (TSF-MT); cross-lingual dependency parsing (TSF-Parsing); cross-lingual POS tagging (TSF-POS); cross-lingual entity linking (TSF-EL); morphological analysis (MA) and universal dependency parsing (UD). Basic statistics on the datasets are outlined in Table 2.
For Wiki-MT tasks, we collect experimental records directly from the paper describing the corresponding datasets (Schwenk et al., 2019). For TED-MT and all the transfer tasks, we use the results of Lin et al. (2019). For BLI, we conduct experiments using published results from three papers, namely Artetxe et al. (2016), Artetxe et al. (2017) and Xu et al. (2018). For MA, we use the results of the SIGMORPHON 2019 shared task 2 (McCarthy et al., 2019). Last, the UD results are taken from the CoNLL 2018 Shared Task on universal dependency parsing (Zeman et al., 2018b).
Featurization For language features, we utilize six distance features from the URIEL Typological Database (Littell et al., 2017), namely geographic, genetic, inventory, syntactic, phonological, and featural distance.
The complete set of dataset features includes the following:
1. Dataset Size: The number of data entries used for training.
2. Word/Subword Vocabulary Size: The number of word/subword types.
3. Average Sentence Length: The average length
of sentences from all experimental.
4. Word/Subword Overlap:
$$
\frac {| T _ {1} \cap T _ {2} |}{| T _ {1} | + | T _ {2} |}
$$
where  $T_{1}$  and  $T_{2}$  denote vocabularies of any two corpora.
5. Type-Token Ratio (TTR): The ratio between the number of types and number of tokens (Richards, 1987) of one corpus.
6. Type-Token Ratio Distance:
$$
\left(1 - \frac {\mathrm {T T R} _ {1}}{\mathrm {T T R} _ {2}}\right) ^ {2}
$$
where  $\mathrm{TTR}_1$  and  $\mathrm{TTR}_2$  denote TTR of any two corpora.
7. Single Tag Type: Number of single tag types.
8. Fused Tag Type: Number of fused tag types.
9. Average Tag Length Per Word: Average number of single tags for each word.
10. Dependency Arcs Matching WALS Features: the proportion of dependency parsing arcs matching the following WALS features, computed over the training set: subject/object/oblique before/after verb and adjective/numeral before/after noun.
For transfer tasks, we use the same set of dataset features  $\Phi_{\mathcal{D}}$  as Lin et al. (2019), including features 1-6 on the source and the transfer language side. We also include language distance features between source and transfer language, as well as between source and target language. For MT tasks, we use features 1-6 and language distance features, but only between the source and target language. For MA, we use features 1, 2, 5 and morphological tag related features 7-9. For UD, we
8627
use features 1, 2, 5, and 10. For BLI, we use language distance features and URIEL syntactic features for the source and the target language.
Predictor Our prediction model is based on gradient boosting trees (Friedman, 2001), implemented with XGBoost (Chen and Guestrin, 2016). This method is widely known as an effective means for solving problems including ranking, classification and regression. We also experimented with Gaussian processes (Williams and Rasmussen, 1996), but settled on gradient boosted trees because performance was similar and Xgboost's implementation is very efficient through the use of parallelism. We use squared error as the objective function for the regression and adopted a fixed learning rate 0.1. To allow the model to fully fit the data we set the maximum tree depth to be 10 and the number of trees to be 100, and use the default regularization terms to prevent the model from overfitting.
4 Can We Predict NLP Performance?
In this section we investigate the effectiveness of NLPERF across different tasks on various metrics. Following Lin et al. (2019), we conduct  $k$ -fold cross validation for evaluation. To be specific, we randomly partition the experimental records of  $\langle \mathcal{L}, \mathcal{D}, \mathcal{C}, \mathcal{S} \rangle$  tuples into  $k$  folds, and use  $k - 1$  folds to train a prediction model and evaluate on the remaining fold. Note that this scenario is similar to "filling in the blanks" in Table 1, where we have some experimental records that we can train the model on, and predict the remaining ones.
For evaluation, we calculate the average root mean square error (RMSE) between the predicted scores and the true scores.
Baselines We compare against a simple mean value baseline, as well as against language-wise mean value and model-wise mean value baselines. The simple mean value baseline outputs an average of scores  $s$  from the training folds for all test entries in the left-out fold  $(i)$  as follows:
$$
\hat {s} _ {\text {m e a n}} ^ {(i)} = \frac {1}{| \mathcal {S} \setminus \mathcal {S} ^ {(i)} |} \sum_ {s \in \mathcal {S} \backslash \mathcal {S} ^ {(i)}} s; i \in 1 \dots k \tag {2}
$$
Note that for tasks involving multiple models, we calculate the RMSE score separately on each model and use the mean RMSE of all models as the final RMSE score.
The language-wise baselines make more informed predictions, taking into account only training instances with the same transfer, source, or target language (depending on the task setting). For example, the source-language mean value baseline  $\hat{s}_{\mathrm{s - lang}}^{(i,j)}$  for  $j^{\mathrm{th}}$  test instance in fold  $i$  outputs an average of the scores  $s$  of the training instances that share the same source language features s-lang, as shown in Equation 3:
$$
\hat {s} _ {\mathrm {s - l a n g}} ^ {(i, j)} = \frac {\sum_ {s , \phi} \delta \left(\phi_ {\mathcal {L} , \mathrm {s r c}} = \mathrm {s - l a n g}\right) \cdot s}{\sum_ {s , \phi} \delta \left(\phi_ {\mathcal {L} , \mathrm {s r c}} = \mathrm {s - l a n g}\right)} \tag {3}
$$
$$
\forall (s, \phi) \in \left(\left| \mathcal {S} \backslash \mathcal {S} ^ {(i)} \right|, \left| \Phi \backslash \Phi^ {(i)} \right|\right)
$$
where  $\delta$  is the indicator function. Similarly, we define the target- and the transfer-language mean value baselines.
In a similar manner, we also compare against a model-wise mean value baseline for tasks that include experimental records from multiple models. Now, the prediction for the  $j^{\mathrm{th}}$  test instance in the left-out fold  $i$  is an average of the scores on the same dataset (as characterized by the language  $\phi_{\mathcal{L}}$  and dataset  $\phi_{\mathcal{D}}$  features) from all other models:
$$
\begin{array}{l} \hat {s} _ {\text {m o d e l}} ^ {(i, j)} = \frac {\sum_ {s , \phi} \delta (\phi_ {\mathcal {L}} = \operatorname {l a n g} , \phi_ {\mathcal {D}} = \operatorname {d a t a}) \cdot s}{\sum_ {s , \phi} \delta (\phi_ {\mathcal {L}} = \operatorname {l a n g} , \phi_ {\mathcal {D}} = \operatorname {d a t a})} \\ \forall (s, \phi) \in \left(\left| \mathcal {S} \backslash \mathcal {S} ^ {(i)} \right|, \left| \Phi \backslash \Phi^ {(i)} \right|\right) \tag {4} \\ \end{array}
$$
where  $\mathrm{lang} = \Phi_{\mathcal{L}}^{(i,j)}$  and data  $= \Phi_{\mathcal{D}}^{(i,j)}$  respectively denote the language and dataset features of the test instance.
Main Results For multi-model tasks, we can do either Single Model prediction (SM), restricting training and testing of the predictor within a single model, or Multi-Model (MM) prediction using a categorical model feature. The RMSE scores of NLPERF along with the baselines are shown in Table 3. For all tasks, our single model predictor is able to more accurately estimate the evaluation score of unseen experiments compared to the single model baselines, confirming our hypothesis that the there exists a correlation that can be captured between experimental settings and the downstream performance of NLP systems. The language-wise baselines are much stronger than the simple mean value baseline but still perform worse than our single model predictor. Similarly, the model-wise baseline significantly outperforms the mean value baseline because results from other models reveal much information about the dataset.
8628
Model	Task
Wiki-MT	TED-MT	TSF-MT	TSF-PARSING	TSF-POS	TSF-EL	BLI	MA	UD
Mean	6.40	12.65	10.77	17.58	29.10	18.65	20.10	9.47	17.69
Transfer Lang-wise	-	-	10.96	15.68	29.98	20.55	-	-	-
Source Lang-wise	5.69	12.65	2.24	-	-	-	20.13	-	-
Target Lang-wise	5.12	12.65	10.78	12.05	8.92	8.61	20.00	9.47	-
NLPERF (SM)	2.50	6.18	1.43	6.24	7.37	7.82	12.63	6.48	12.06
Model-wise	-	-	-	-	-	-	8.77	5.22	4.96
NLPERF (MM)	-	-	-	-	-	-	6.87	3.18	3.54
Even so, our multi-model predictor still outperforms the model-wise baseline.
The results nicely imply that for a wide range of tasks, our predictor is able to reasonably estimate left-out slots in a partly populated table given results of other experiment records, without actually running the system.
We should note that RMSE scores across different tasks should not be directly compared, mainly because the scale of each evaluation metric is different. For example, a BLEU score (Papineni et al., 2002) for MT experiments typically ranges from 1 to 40, while an accuracy score usually has a much larger range, for example, BLI accuracy ranges from 0.333 to 78.2 and TSF-POS accuracy ranges from 1.84 to 87.98, which consequently makes the RMSE scores of these tasks higher.
Comparison to Expert Human Performance We constructed a small scale case study to evaluate whether NLPERF is competitive to the performance of NLP sub-field experts. We focused on the TED-MT task and recruited 10 MT practitioners, all of whom had published at least 3 MT-related papers in ACL-related conferences.
In the first set of questions, the participants were presented with language pairs from one of the  $k$  data folds along with the dataset features and were asked to estimate an eventual BLEU score for each data entry. In the second part of the questionnaire, the participants were tasked with making estimations on the same set of language pairs, but this time they also had access to features, and BLEU scores from all the other folds.
Table 3: RMSE scores of three baselines and our predictions under the single model and multi model setting (missing values correspond to settings not applicable to the task). All results are from k-fold  $(k = 5)$  evaluations averaged over 10 random runs.
Predictor	RMSE
Mean Baseline	12.64
Human (w/o training data)	9.38
Human (w/ training data)	7.29
NLPERF	6.04
Table 4: Our model performs better than human MT experts on the TED-MT prediction task.
The partition of the folds is consistent between the human study and the training/evaluation for the predictor. While the first sheet is intended to familiarize the participants with the task, the second sheet fairly adopts the training/evaluation setting for our predictor. As shown in Table 4, our participants outperform the mean baseline even without information from other folds, demonstrating their own strong prior knowledge in the field. In addition, the participants make more accurate guesses after acquiring more information on experimental records in other folds. In neither case, though, are the human experts competitive to our predictor. In fact, only one of the participants achieved performance comparable to our predictor.
Feature Perturbation Another question of interest concerning predicting performance is "how will the model perform when trained on data of a different size" (Kolachina et al., 2012a). To test NLPERF's extrapolation ability in this regard, we conduct an array of experiments on one language pair with various data sizes on the Wiki-MT task. We pick two language pairs, Turkish to English (TR-EN) and Portuguese to English (PT-EN) as our testbed for the Wiki-MT task. We sample par
(and make estimations over one of the folds) in the A.
None of the study participants were affiliated to the authors' institutions, nor were familiar with this paper's content. The interested reader can find an example questionnaire
Figure 1: Our model's predicted BLEU scores and true BLEU scores, on sampled TR-EN datasets (sizes  $10\mathrm{k} / 50\mathrm{k} / 100\mathrm{k} / 200\mathrm{k} / 478\mathrm{k}$ ) and PT-EN datasets (sizes  $100\mathrm{k} / 500\mathrm{k} / 1000\mathrm{k} / 2000\mathrm{k} / 2462\mathrm{k}$ ), achieving a RMSE score of 1.83 and 9.97 respectively.
The model of the true BLEU score is a set of 1000 samples that we use to predict the true BLEU scores. The model is trained on all available data and used to predict the true BLEU scores. The model is trained on all available data and used to predict the true BLEU scores. On the other hand, we collect the features of all sampled datasets and use our predictor (trained over all other languages pairs) to obtain predictions. The plot of true BLEU scores and predicted BLEU scores are shown in Figure 1. Our predictor achieves a very low average RMSE of 1.83 for TR-EN pair but a relatively higher RMSE of 9.97 for PT-EN pair. The favorable performance on the tr-en pair demonstrates the possibility of our predictor to do feature extrapolation over data set size. In contrast, the predictions on the pt-en pair are significantly less accurate. This is due to the fact that there are only two other experimental settings scoring as high as 34 BLEU score, with data sizes of 3378k (en-es) and 611k (gl-es), leading to the predictor's inadequacy in predicting high BLEU scores for low-resourced data sets during extrapolation. This reveals the fact that while the predictor is able to extrapolate performance on settings similar to what it has seen in the data, NLPERF may be less successful under circumstances unlike its training inputs.
5 What Datasets Should We Test On?
As shown in Table 1, it is common practice to test models on a subset of all available datasets. The reason for this is practical - it is computationally prohibitive to evaluate on all settings. However, if we pick test sets that are not representative of the data as a whole, we may mistakenly reach un
founded conclusions about how well models perform on other data with distinct properties. For example, models trained on a small-sized dataset may not scale well to a large-sized one, or models that perform well on languages with a particular linguistic characteristic may not do well on languages with other characteristics (Bender and Friedman, 2018).