SlowGuess/NAACL_2022_OCR huggingface dataset:
Instead of only regressing one dataset's quality, we also care about the quality ranking of different datasets w.r.t discriminating systems in a task. Therefore, we also formulate it as a listwise
LTR(learning to rank) task where a model takes individual lists as instances, to predict the rank of element among the list (Liu, 2011). Given a set of  $n$  datasets  $d = \{d_1, d_2, \dots, d_n\}$  ( $d \in D = \{D^{tr}, D^{te}\}$ ), different  $d$  construct the dataset of LTR task, the target of the ranker is to predict the dataset quality ranking for each dataset in  $d$  according to the datasets' features. The estimated rankings  $\overline{\lambda} = \{\lambda_1, \lambda_2, \dots, \lambda_n\} \in [1, n]$  for set  $d$  can be defined as:
$$
\bar {\lambda} = \bar {f} \left(\Phi_ {(d)}; \bar {\Theta}\right), \tag {7}
$$
where  $\Phi (\cdot)$  is the dataset feature extractor,  $\overline{f}$  is the ranking model.  $\overline{\lambda} \in [1, n]$  is the estimated rankings of the variance ( $\lambda_{var}$  or  $\lambda_{sva}$ ) for datasets in set  $d$ .
5.2 Characterization of Datasets
In this section, we will introduce three aspects that characterize datasets: Inherent Feature, Lexical Feature, and Semantic Feature. Due to space limitations, we move a more detailed feature introduction to the Appendix C.
5.2.1 Inherent Feature
Average length  $(\phi_{\mathrm{len}})$ : The average sentence length on a dataset, where the number of tokens on a sentence is considered as the sentence length. Label number  $(\phi_{\mathrm{lab}})$ : The number of labeled classes in a dataset. Label balance  $(\phi_{\mathrm{bal}})$ : The label balance metric measures the variance between the ideal and the true label distribution.
5.2.2 Lexical Feature
Basic English Words Ratio ( $\phi_{\mathrm{basic}}$ ): The proportion of words belonging to the 1000 basic English  $^{12}$  words in the whole dataset. Type-Token Ratio ( $\phi_{\mathrm{ttr}}$ ): We measure the text lexical richness by the type-token ratio (Richards, 1987) based on the lexical richness tool.  $^{13}$  Language Mixedness Ratio ( $\phi_{\mathrm{lmix}}$ ): To detect the ratio of other languages mixed in the text, we utilize the models proposed by Joulin et al. (2016b) for language identification from fastText (Joulin et al., 2016a) which can recognize 176 languages. Pointwise Mutual Information ( $\phi_{\mathrm{pmi}}$ ): PMI  $^{14}$  is a measurement to calculate the correlation between variables.
5.2.3 Semantic Feature
Perplexity  $(\phi_{\mathrm{ppl}})$ : We calculate the perplexity  $^{15}$  based on GPT2 (Radford et al., 2019) to evaluate the quality of the text. Grammar Errors Ratio  $(\phi_{\mathrm{gerr}})$ : We adopt the detection tool  $^{16}$  to recognize words with grammatical errors, and then calculate the ratio of grammatical errors. Flesch Reading Ease  $^{17}$ $(\phi_{\mathrm{fre}})$ : To describe the readability of a text, we introduce the  $\phi_{\mathrm{fre}}$  achieving by textstat.  $^{18}$
For feature  $\phi_{\mathrm{len}}$ ,  $\phi_{\mathrm{ttr}}, \phi_{\mathrm{lmix}}$ ,  $\phi_{\mathrm{gerr}}$ ,  $\phi_{\mathrm{pmi}}$ ,  $\phi_{\mathrm{fre}}$ , and  $\phi_{\mathrm{r}_{\mathrm{fre}}}$ , we individually compute  $\phi()$  on the training, test set, as well as their interaction. Take average length  $(\phi_{\mathrm{len}})$  as an example, we compute the average length on training set  $\phi_{\mathrm{tr},\mathrm{len}}$ , test set  $\phi_{\mathrm{te},\mathrm{len}}$ , and their interaction  $((\phi_{\mathrm{tr},\mathrm{len}} - \phi_{\mathrm{te},\mathrm{len}}) / \phi_{\mathrm{tr},\mathrm{len}})^2$ .
5.3 Parameterized Models
The dataset discrimination prediction (ranking) model takes a series of dataset features as the input and then predicts discrimination(rank) based on  $\hat{f} (\cdot)$ $(\overline{f} (\cdot))$  defined in Eq. 6 (Eq. 7). We explore the effectiveness of four variations of regression methods and two ranking frameworks.
Regression Models: LightGBM (Ke et al., 2017) is a gradient boosting framework with faster train
ing and better performance than XGBoost. K-nearest Neighbor (KNN) (Peterson, 2009) is a non-parametric model that makes the prediction by exploring the k neighbors. Support Vector Machine (SVM) (Suykens and Vandewalle, 1999) uses kernel trick to solve both linear and non-linear problems. Decision Tree (DT) (Quinlan, 1990) is a tree-based algorithm that gives an understandable interpretation of predictions.
Ranking Frameworks: LightGBM with Gradient Boosting Decision Tree (Friedman, 2001) boosting strategy was selected as our ranking model. XGBoost (Chen and Guestrin, 2016) with gbtree(Hastie et al., 2009) boosting strategy was another ranking model.
5.4.1 Data Construction
To construct a collection with large amount of discriminative datasets, we randomly select three dataset features (e.g. average sentence length  $\phi_{\mathrm{len}}$ ) to divide the original dataset into several non-overlapping sub-datasets. As a result, we collect 987 sub-datasets. Then, we train four text classification models (CNN, LSTM, LSTMAtt, BERT) on these sub-dastets. Next, we calculate the dataset features  $\phi$  (defined in Sec. 5.2) and dataset discrimination ability  $\lambda_{\mathrm{sva}}$  and  $\lambda_{\mathrm{var}}$  on these sub-datasets.
Regression Task Settings  $\phi$  and  $\lambda_{\mathrm{sva}}(\lambda_{\mathrm{var}})$  will be the input and target of the regression models, as defined by Eq. 6. For the experiment setting, we randomly select 287  $(\phi ,\lambda_{\mathrm{sva}}(\lambda_{\mathrm{var}}))$  pairs as the test set and the rest as the training set (700). Ranking Task Settings We construct datasets for ranking task from the dataset used in regression task. Here, we explored the value of  $n$  (defined in §5.1.2) to be 5, 7 and 9 to randomly choose samples from  $D^{tr}$  (or  $D^{te}$ ) to construct the datasets for the ranking task, and kept 4200, 600, 1200 samples for training, development and testing set respectively.
5.4.2 Evaluation Metric
Regression Task We use RMSE (Chai and Draxler, 2014) and Spearman rank correlation coefficient (Zar, 1972) to evaluate how well the regression model predicts the discriminative ability for datasets. The Spearman rank correlation coefficient is used for the correlation between the output of a regression model and the ground truth.
Ranking Task NDCG (Järvelin and Kekäläinen, 2000) and MAP (Yue et al., 2007) are the evalua-
tion metric of our ranking task. For NDCG, it considers the rank of a set of discriminative abilities. In our setting, every dataset has its own real discriminative ability. Here, We transfer the predicted discriminative ability to the rank of the dataset in the NDCG metric, so we can use NDCG to evaluate the model's predicted effect. For MAP, it likes how NDCG works, but it considers a set of binary values. Here, we set a threshold value of  $\lambda_{\mathrm{var}} = 3$  ( $\lambda_{\mathrm{sva}} = 28$ ) for  $\lambda_{\mathrm{var}}(\lambda_{\mathrm{sva}})$  to distinguish the dataset discrimination ability from good (relevant) to bad (irrelevant).
Method	RMSE	Spearman
λvar	λsva	λvar	λsva
corr	p	corr	p
KNN	2.42	51.21	0.77	9.75E-40	0.87	1.62E-63
LightGBM	1.53	32.74	0.72	2.23E-33	0.87	7.01E-61
DT	1.73	43.33	0.64	9.25E-25	0.84	1.33E-53
SVM	2.83	62.44	0.68	1.14E-28	0.77	7.26E-40
Table 2: The performance of regressing dataset discrimination for the text classification. "corr" denotes the "correlation".
Model	n	NDCG	MAP
λvar	λsvar	λvar	λsvar
LightGBM	9	98.20	98.85	97.50	98.27
7	97.76	98.73	97.01	99.05
5	96.73	97.08	96.56	98.15
XGBoost	9	96.66	97.13	92.91	93.62
7	96.74	97.65	94.77	96.11
5	95.93	97.10	95.49	98.25
Table 3: The performance of ranking dataset discrimination for the text classification task.  $n$  is the number of datasets in  $d$  defined in §5.1.2
5.4.3 Results and Analysis
Tab. 2 and Tab. 3 show the results of four regression models and two ranking models that characterize the dataset discrimination ability, respectively. We can observe that: Both the regression models and the ranking models can well describe the discrimination ability of different datasets. For these four regression models, the prediction is highly correlated with the ground truth (with a correlation value larger than 0.6), passing the significance testing  $(p < 0.05)$ . This suggests that the dataset discrimination can be successfully predicted. For these two ranking models, their performance on NDCG and MAP is greater than  $95\%$ , which indicates that the
discriminative ability of the data set can be easily ranked.
(a) Inherent
(b) Lexical
(c) Semantic
Figure 3: Feature importance for the text classification measured by LGBoost with the target of  $\lambda_{\mathrm{sva}}$
Feature Importance Analysis Fig. 3 illustrates the feature importance characterized by LightGBM. For a given feature, the number of times that is chosen as the splitting feature in the node of the decision trees is defined as its importance degree. We observe that: (1) The most influential features are  $\phi_{\mathrm{pmi}}$ ,  $\phi_{\mathrm{len}}$ , and  $\phi_{\mathrm{fre}}$ , which come from the lexical, inherent, and semantic features, respectively. This indicated that the LightGBM can extract features from different aspects to make predictions. (2) In the perspective of feature groups, the semantic features are more influential than the inherent features and lexical features.
6 Discussion & Implications
Discussion Given a leaderboard of a dataset, metrics explored in this paper can be easily used to calculate its discrimination, while some limitations still exist. We make some discussion below to encourage more explorations on new measures: (a) Interpretability: current metrics can only identify which datasets are of lower indiscriminability while don't present more explanation why it is the case. (b) Functionality: a dataset with lower discrimination doesn't mean it's useless since the supervision signals provided there can not only help us directly train a system for the specific use case but also provide good supervised transfer for related tasks. Metrics designed in this work focus on the role of discriminating models.
Calls Based on observations obtained from this paper, we make the following calls for future research: (1) Datasets' discrimination ability w.r.t top-scoring systems could be included in the dataset schema (such as dataset statement (Bender and Friedman, 2018)), which would allow researchers to gain a saturated understanding of the dataset. (2) Leaderboard constructors could also
report the discriminative ability of the datasets they aim to include. (3) Seldom used datasets are also valuable for model selection, and a more fair dataset searching system should be investigated, for example, relevance- and scientifically meaningful first, instead of other biases, like popularity.
We would like to thank Graham Neubig and the anonymous reviewers for their valuable comments. This work was supported by the National Research Foundation of Singapore under its Industry Alignment Fund - Pre-positioning (IAF-PP) Funding Initiative. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not reflect the views of the National Research Foundation of Singapore.
Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587-604.
Alexandra Birch, Miles Osborne, and Philipp Koehn. 2008. Predicting success in machine translation. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 745-754, Honolulu, Hawaii. Association for Computational Linguistics.
Samuel R. Bowman and George E. Dahl. 2021. What will it take to fix benchmarking in natural language understanding? CoRR, abs/2104.02145.
Tianfeng Chai and Roland R Draxler. 2014. Root mean square error (rmse) or mean absolute error (mae)?- arguments against avoiding rmse in the literature. Geoscientific model development, 7(3):1247-1250.
Tianqi Chen and Carlos Guestrin. 2016. Xgboost. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. 2015. Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In Twenty-Fourth International Joint Conference on Artificial Intelligence.
Jerome H Friedman. 2001. Greedy function approximation: a gradient boosting machine. Annals of statistics, pages 1189-1232.
Jinlan Fu, Pengfei Liu, and Qi Zhang. 2020. Rethinking generalization of neural models: A named entity recognition case study. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7732-7739. AAAI Press.
Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna Clinciu, Dipanjan Das, Kaustubh D Dhole, et al. 2021. The gem benchmark: Natural language generation, its evaluation and metrics. arXiv preprint arXiv:2102.01672.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. http://www.deeplearningbook.org.
Harsha Gurulingappa, Abdul Mateen Rajput, Angus Roberts, Juliane Fluck, Martin Hofmann-Apitius, and Luca Toldo. 2012. Development of a benchmark corpus to support the automatic extraction of drug-related adverse effects from medical case reports. Journal of Biomedical Informatics, 45(5):885-892. Text Mining and Natural Language Processing in Pharmacogenomics.
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. 2009. Boosting and Additive Trees, pages 337-387. Springer New York, New York, NY.
Charles T Hemphill, John J Godfrey, and George R Doddington. 1990. The atis spoken language systems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. *Neural computation*, 9(8):1735-1780.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan First, and Melvin Johnson. 2020. XTREME: A massively multilingual multitask benchmark for evaluating cross-lingual generalization. CoRR, abs/2003.11080.
Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '04, page 168-177, New York, NY, USA. Association for Computing Machinery.
Kalervo Järvelin and Jaana Kekäläinen. 2000. IR evaluation methods for retrieving highly relevant documents. In SIGIR 2000: Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, July 24-28, 2000, Athens, Greece, pages 41-48. ACM.
Guangjin Jin and Xiao Chen. 2008. The fourth international Chinese language processing bakeoff: Chinese word segmentation, named entity recognition and chinese pos tagging. In Proceedings of the sixth SIGHAN workshop on Chinese language processing.
Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016a. Fasttext.zip: Compressing text classification models. arXiv preprint arXiv:1612.03651.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016b. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. Lightgbm: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 3146-3154.
Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A Smith, and Daniel S Weld. 2021. Genie: A leaderboard for human-in-the-loop evaluation of text generation. arXiv preprint arXiv:2101.06561.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. CoRR, abs/1408.5882.
Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388-395, Barcelona, Spain. Association for Computational Linguistics.
Prasanth Kolachina, Nicola Cancedda, Marc Dymetman, and Sriram Venkatapathy. 2012. Prediction of learning curves in machine translation. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 22-30, Jeju Island, Korea. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. ArXiv, abs/1910.13461.
Xin Li and Dan Roth. 2002. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics.
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. CoRR, abs/1703.03130.
Tie-Yan Liu. 2011. Learning to rank for information retrieval.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Linguistics.
Tomás Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pages 3111-3119.
Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. CoRR, abs/cs/0506075.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1532-1543. ACL.
Leif E Peterson. 2009. K-nearest neighbor. *Scholarpedia*, 4(2):1883.
John Ross Quinlan. 1990. Probabilistic decision trees. In Machine Learning, pages 140-152. Elsevier.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD:  $100,000+$  questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Brian Richards. 1987. Type/token ratios: what do they really tell us? Journal of Child Language, 14(2):201-209.
Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan First, Jinlan Fu, Pengfei Liu, Junjie Hu, Graham Neubig, and Melvin Johnson. 2021. XTREME-R: towards more challenging and nuanced multilingual evaluation. CoRR, abs/2104.07412.
Erik F Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. arXiv preprint cs/0306050.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun
Raja, et al. 2021. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207.
Claude E Shannon. 1948. A mathematical theory of communication. The Bell system technical journal, 27(3):379-423.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
Johan AK Suykens and Joos Vandewalle. 1999. Least squares support vector machine classifiers. Neural processing letters, 9(3):293-300.
Huihsin Tseng, Pichuan Chang, Galen Andrew, Daniel Jurafsky, and Christopher Manning. 2005. A conditional random field word segmenter for sighan bakeoff 2005. In Proceedings of the fourth SIGHAN workshop on Chinese language Processing, volume 171.
Marco Turchi, Tijl De Bie, and Nello Cristianini. 2008. Learning performance of a machine translation system: a statistical and computational analysis. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 35-43.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Computational Linguistics.
Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426.
Mengzhou Xia, Antonios Anastasopoulos, Ruochen Xu, Yiming Yang, and Graham Neubig. 2020. Predicting performance for natural language processing tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8625-8646, Online. Association for Computational Linguistics.
Yang Xiao, Jinlan Fu, Weizhe Yuan, Vijay Viswanathan, Zhoumianze Liu, Yixin Liu, Graham Neubig, and Pengfei Liu. 2022. Datalab: A platform for data analysis and intervention. CoRR, abs/2202.12875.
Zihuiwen Ye, Pengfei Liu, Jinlan Fu, and Graham Neubig. 2021. Towards more fine-grained and reliable NLP performance prediction. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3703-3714, Online. Association for Computational Linguistics.
Yisong Yue, Thomas Finley, Filip Radlinski, and Thorsten Joachims. 2007. A support vector method for optimizing average precision. In SIGIR 2007: Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Amsterdam, The Netherlands, July 23-27, 2007, pages 271-278. ACM.
Jerrold H Zar. 1972. Significance testing of the spearman rank correlation coefficient. Journal of the American Statistical Association, 67(339):578-580.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. CoRR, abs/1509.01626.
Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2020. Extractive summarization as text matching. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6197-6208, Online. Association for Computational Linguistics.
A Statistics of Datasets
Tab. 4 shows the statistical information of the nine datasets of text classification task used in our work. For those datasets without explicit the development set, we randomly selected  $12.5\%$  samples from the training set as the development set.