SlowGuess/ACL_2020_OCR huggingface dataset:
Canwen Xu $^{1*}$ , Jiaxin Pei $^{2*}$ , Hongtao Wu $^{3}$ , Yiyu Liu $^{3}$ , Chenliang Li $^{3\dagger}$
<sup>1</sup> School of Computer Science, Wuhan University, China
$^{2}$  School of Information, University of Michigan, United States
$^{3}$  School of Cyber Science and Engineering, Wuhan University, China
1,3 {xucanwen,wuhongtao,liuyiyu,cllee}@whu.edu.cn
2 pedropei@umich.edu
Recently, large-scale datasets have vastly facilitated the development in nearly all domains of Natural Language Processing. However, there is currently no cross-task dataset in NLP, which hinders the development of multi-task learning. We propose MATINF, the first jointly labeled large-scale dataset for classification, question answering and summarization. MATINF contains 1.07 million question-answer pairs with human-labeled categories and user-generated question descriptions. Based on such rich information, MATINF is applicable for three major NLP tasks, including classification, question answering, and summarization. We benchmark existing methods and a novel multi-task baseline over MATINF to inspire further research. Our comprehensive comparison and experiments over MATINF and other datasets demonstrate the merits held by MATINF.<sup>1</sup>
In recent years, large-scale datasets (e.g., ImageNet (Deng et al., 2009) and SQuAD (Rajpurkar et al., 2016)) have inspired remarkable progress in many areas like Computer Vision (CV) and Natural Language Processing (NLP). On the one hand, well-annotated data provide essential information for training supervised machine learning models. On the other hand, benchmarked datasets make it possible to evaluate and compare the capability of different methods on the same stage.
Due to the high cost of data annotation, existing NLP datasets are usually labeled for only one particular task (e.g., SQuAD (Rajpurkar et al., 2016) for question answering, CNN/DM (Hermann et al.,
2015) for summarization and AGNews (Zhang et al., 2015) for text classification). These single-task datasets hinder the development of learning common and task-invariant knowledge (Liu et al., 2017). Although multi-task learning and transfer learning have delivered encouraging results, we still cannot determine whether the improvement is from the extension of input or supervision. Furthermore, task-specific data make the models tend to learn task-specific leakage features (Zhang et al., 2019) rather than meaningful knowledge that could generalize to other tasks. However, as a key step to Artificial General Intelligence (AGI), knowledge acquisition requires the model to learn more general knowledge instead of overfitting on a specific task. Therefore, a large-scale and cross-task dataset is in huge demand for future NLP research. Nevertheless, to the best of our knowledge, none of the existing datasets could meet such demand.
In this paper, we propose Maternal and Infant Dataset (MATINF), the first large-scale dataset covering three major NLP tasks: text classification, question answering and summarization. MATINF consists of question answering data crawled from a large Chinese maternity and baby caring QA site. On this site, users can ask questions related to maternity and baby caring. When submitting a question, a detailed description is required to provide essential information and the asker also needs to assign a category for this question from a pre-defined topic list. Each user could submit an answer to a question post, and the asker will select the best answer out of all the candidates. To attract more attention, the askers are encouraged to set rewards using virtual coins when submitting the question and these coins will be given to the user who submitted the best answer selected by the asker. This rewarding mechanism could constantly ensure high-quality answers.
MATINF supports three NLP tasks as follows.
* The first two authors contribute equally to this paper.† Chenliang Li is the corresponding author.
<sup>1</sup>The implementation of MTF-S2S and information about obtaining access to the dataset can be found at https://github.com/WHUIR/MATINF.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3586-3596 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Text Classification. Given a question and its detailed description, the task is to select an appropriate category from the fine-grained category list. Different from previous news classification tasks whose category set is general topics like entertainment and sports, MATINF-C is a fine-grained classification under a single domain. That is, the distance between different categories is smaller, which provides a more challenging stage to test the continuously evolving state-of-the-art neural models.
Question Answering. Given a question, the task is to produce an answer in natural language. This task is slightly different from previous Machine Reading Comprehension (MRC) since the document which contains the correct answer is not directly provided. Therefore, how to collect the domain knowledge from massive QA data becomes extremely important.
Summarization. Given a question description, the task is to produce the corresponding question. Previous summarization datasets are all constructed with news or academic articles. The limited text genres covered in these datasets hinder the thorough evaluation of summarization models. Also, the noisy nature of MATINF encourages more robust models. MATINF can be considered as the first social media summarization dataset.
MATINF holds the following merits: (1) Large. MATINF includes 1.07M unique QA pairs, making it an ideal playground for the new advancements of deeper and larger models (e.g., Pretrained Language Models). (2) Multi-task applicable. MATLAB is the first dataset that simultaneously contains ground truths for three major NLP tasks, which could facilitate new multi-task learning methods for these tasks. Here, to set a baseline and inspire future research, we present Multi-task Field-shared Sequence to Sequence (MTF-S2S), a straightforward yet effective model, which achieves better performance on all three tasks compared to its single-task counterparts.
2.1 Topic Classification
Topic classification is one of the most fundamental tasks in NLP. As a deeply explored task, many datasets have been used in previous research both in English (AGNews, DBPedia, Yahoo Answer (Zhang et al., 2015), TREC (Voorhees and Tice, 1999)) and Chinese (THUCNews (Sun et al., 2016), SogouCS (Wang et al., 2008a), Fudan Cor
pus, iFeng and ChinaNews (Zhang and LeCun, 2017)). These datasets were useful and indispensable in the past decades to test the performance of different kinds of classifiers.
However, as most of them are formal text and the target categories are general topics, even simply leveraging n-gram features could achieve acceptable results. Plus, some of them are small in scale. Nowadays, with the prevalence of neural models and pretraining techniques, recent algorithms (Sun et al., 2018; Wu et al., 2019) are approaching the ceiling of these datasets with accuracy scores up to  $98\%$ . Different from any of the existing datasets, MATINF is more challenging, providing a new stage to test the performance of future algorithms.
2.2 Question Answering
Following the definition in (Jurafsky and Martin, 2009), Question Answering (QA) can be generally divided into Information Retrieval (IR) based Question Answering and Knowledge-based Question Answering. For IR-based Question Answering, the answer is often a span in the retrieved document. As for Knowledge-based Question Answering, a human-constructed knowledge base is provided for querying and the answer is in the form of a query result. Recently, Open Domain QA (Chen et al., 2017) has been recognized as a new genre where a natural language response instead of text spans is returned as an answer.
Currently, several datasets are available for Chinese Question Answering. NLPCC Shared Task (Duan and Tang, 2017) provided two datasets for IR-based and Knowledge-based QA, respectively. DuReader (He et al., 2018) is an Open Domain dataset derived from user search logs and provided with human-picked documents as evidence. Zhang and Zhao (2018) provided a QA dataset in the domain of Chinese College Entrance Test history exam questions, with documents from standard history textbooks. Different from these datasets, instead of providing pre-defined documents as evidence, MATINF-QA only contains sufficient QA pairs in the training set. In this way, there are various approaches to exploit these questions as evidence. Thus, MATINF-QA encourages innovations in retrieval, generation and hybrid question answering methods.
婴幼儿保健 Infant health care	Class
宝宝为什么总是吐舌头啊?
Why does my baby always stick his tongue out?	Question
我家宝宝出生快满四个月了,这几天我忽然发现宝宝总是吐舌头,而且口水也很多,那么这到底是咋回事啊?
My baby is almost four months old. In these few days, I suddenly found that my baby always stick his tongue out and has a lot of saliva. So what is this?	Description
正常,不要担心的,小孩子都这个样子。宝宝吐舌头也是很正常的现象,你也不用过于担心,宝宝流口水可能是要长牙齿了。
Don't worry, it's normal. Kids are like this. It is also normal for your baby to stick his tongue out. You don't have to worry too much. Your baby's drooling may be a sign of teeth growing.	Answer
2.3 Summarization
Summarization datasets can be roughly categorized into extractive and abstractive datasets, which respectively favor abstractive and extractive methods. Extractive datasets are composed of long documents and summaries. Since the summary is long, extracted sentences and spans from the document could compose a good summary. Newsroom (Grusky et al., 2018), ArXiv and PubMed (Cohan et al., 2018) and CNN / Daily Mail dataset (Hermann et al., 2015) are commonly used extractive datasets.
Abstractive datasets often contain short documents and summaries, which encourages a thorough understanding of the document and style transfer between a document and its corresponding summary. Gigaword (Napoles et al., 2012) and Xsum (Narayan et al., 2018) fall into this category. Also, the abstractive dataset LCSTS (Hu et al., 2015), crawled from verified short news feeds of major newspapers and televisions, is the only public dataset for Chinese text summarization to date.
However, all of these existing datasets are composed of either news or academic articles. The narrow sources of these datasets bring two main drawbacks. First, due to the nature of news reporting and academic writing, the summary-eligible contents do not distribute uniformly (Sharma et al., 2019). Second, models evaluated on these noiseless formal-text datasets are not robust enough for real-world applications. To address these problems, we propose MATINF-SUMM, a new abstractive Chinese summarization dataset.
Figure 1: An example entry from MATINF.
Question	Description	Answer	Max Len.
# Char	14.72	64.17	66.91	256
# Word	9.03	41.70	42.32	-
Table 1: Average character and word numbers of question, description and answer in MATINF. We ensure that every field of each entry has at most 256 characters.
3 MATINF Dataset
We present Maternal and Infant (MATINF) Dataset, a large-scale dataset jointly labeled for classification, question answering and summarization in the domain of maternity and baby caring in Chinese. An entry in the dataset includes four fields: question  $(Q)$ , description  $(D)$ , class  $(C)$  and answer  $(A)$ . An example is shown in Figure 1, and the average character and word numbers of each field are reported in Table 1.
We collect nearly two million question-answer pairs with fine-grained human-labeled classes from a large Chinese maternity and baby caring QA site. We conduct both automatic and manual data cleansing and remove: (1) classes with insufficient samples; (2) entries in which the length of the description filed is less than the length of the question field; (3) data with any field longer than 256 characters; (4) human-spotted ill-formed data. After the data cleansing, we construct MATINF with the remaining 1.07 million entries.
We first randomly split the whole data into training, validation and test sets with a proportion of 7:1:2. Then, we use the splits for summarization and QA. For classification, we further divide the data into two sub-tasks according to different classification standards within each split.
3.1 MATINF-C: Fine-grained Text Classification
In MATINF, the class labels are first selected by the users when submitting a question. Then, if the question is not in the right class, the forum administrators would manually re-categorize the question to the correct class. In our data, there are two parallel standards for classifying a question: topic class and age of the baby. We use these two standards to construct our two subsets. Thus, we define two tasks: (1) classifying a question to different age groups; (2) classifying a question into a fine-grained topic. We list the classes of the two tasks in Table 2. Note that there is no data overlap
MATINF-C-TOPIC
18 classes	MATINF-C-AGE
3 classes
产褥期保健	postpartum health care	0-1岁	0-1 yr old
儿童过敏	child allergy	1-2岁	1-2 yrs old
动作发育	motion development	2-3岁	2-3 yrs old
婴幼儿保健	infant health care
婴幼儿心理	infant psychology
婴幼儿早教	early education
婴幼儿喂养	infant feeding
婴幼儿营养	infant nutrition
孕期保健	pregnancy care
家庭教育	family education
幼儿园	kindergarten
未准父母	pregnancy preparation
流产和不孕	infertility problem
疫苗接种	vaccination
皮肤护理	skin care
宝宝上火	infant ulcer
腹泻	diarrhea
婴幼儿常见病	other infant common diseases
Table 2: Class names of two subsets and their English translations.
Dataset	Lang.	Domain	# Doc	# Class
AG News (2015)	EN	News	128K	4
DBPedia (2015)	EN	Wiki	630K	14
TREC-6 (1999)	EN	Open	6K	6
TREC-50† (1999)	EN	Open	6K	50
Yahoo Answer (2015)	EN	Open	1.46M	10
THUCNews (2016)	ZH	News	740K	14
SogouCS (2008b)	ZH	News	577K	5
Fudan Corpus (2018)	ZH	News	10K	20
iFeng (2017)	ZH	News	850K	5
ChinaNews (2017)	ZH	News	1.51M	7
MATINF-C-AGE†	ZH	Health	192K	3
MATINF-C-TOPIC†	ZH	Health	876K	18
Table 3: Comparison of classification datasets. †: Fine-grained datasets.
between the two subsets. Formally, we define the task as predicting the class of a QA pair with its question and description fields (i.e.,  $Q, D \to C$ ). Different from previous datasets, our task is a fine-grained classification (i.e., to classify documents in a domain) rather than classifying general topics (e.g., politics, sports, entertainments), which means the semantic difference between classes is prominently smaller. It requires meticulous exploitation of semantics instead of recognizing unique n-gram features for each class. We provide statistical comparison of MATINF-C with other datasets in Table 3.
3.2 MATINF-QA: Health-Domain Question Answering
Typically, to return an answer for a specific question, the model needs to retrieve from a pre-defined
document set or query a manually-constructed knowledge base. MS-MARCO (Nguyen et al., 2016) utilizes a search engine to pre-filter 10 documents from the Internet and uses them as the document set. However, searching itself is a challenging task that significantly affects the final performance. On the other hand, in a real-world scenario, it is impossible to define a document set covering all knowledge needed to answer a user question. Thus, we provide the training set of MATINF-QA as the possible document source and encourage all kinds of methods including retrieval, generation and hybrid models.
Formally, the task is defined as replying a question with natural text (i.e.,  $Q \to A$ ). The large scale of our dataset ensures that a model is able to generalize and learn enough knowledge to answer a user question. Note that we do not use description when defining this task since we observe a negative effect on the generalization in our experiment. Shown in Table 4, we list statistics of MATINF-QA and other commonly-used datasets.
3.3 MATINF-SUMM: Summarization in Professional Domain
All current datasets for summarization to date are in the domain of news and academic articles. However, as a custom of the report and academic writing, in extractive datasets, the summary-eligible contents often appear at the beginning or the end of an article, preventing the summarization model from a full understanding and resulting in impractically high performance in evaluation. On the other hand, current abstractive datasets are all formal news datasets, which are in lack of diversity. Models trained on such a single-source dataset is not robust enough to handle real-world complexity.
In MATINF-SUMM, question description can be seen as an extended and specific version of the question itself, containing more detailed background information with respect to the question. Besides, the question itself is often a well-formed interrogative sentence rather than extracted phrases. Our task is to generate the question from the corresponding description (i.e.,  $D \rightarrow Q$ ). Note that our task itself can support many meaningful real-world applications, e.g., generating an informative title for user-generated content (UGC). Also, there is only one public dataset for summarization in Chinese to date. Our dataset can be used to verify the effectiveness of existing models and eliminate the
Dataset	Lang.	# Q/A Pair	# Docs	Source of Query	Source of Docs	Answer Type
CNN / DM (2015)	EN	1.4M	300K	Synthetic cloze	News	Fill in entity
HLF-RC (2016)	ZH	100K	28K	Synthetic cloze	Fairy / News	Fill in word
CBT (2016)	EN	688K	108	Synthetic cloze	Children's books	Multi-choices
NewsQA (2017)	EN	100K	10K	Crowdsourced	CNN	Span of words
SQuAD (2016)	EN	100K	536	Crowdsourced	Wiki	Span of words
SearchQA (2017)	EN	140K	6.9M	QA site	Web	Span of words
SQuAD 2.0 (2016)	EN	150K	505	Crowdsourced	Wiki	Span of words
NLPCC DBQA (2017)	ZH	15K	15K	Crowdsourced	Wiki	Binary matching
MS-MARCO (2016)	EN	100K	200K	User logs	Web	Natural language response
DuReader (2018)	ZH	200K	1M	User logs	Web/QA site	Natural language response
MATINF-QA	ZH	1.07M	-	QA Site	-	Natural language response
Table 4: Comparison of question answering datasets. Some statistics are reused from (He et al., 2018).
Dataset	Lang.	Domain	# Doc	# Token
Doc.	Sum.
CNN / DM (2015)	EN	News	312K	781	56
NYT (2012)	EN	News	655K	796	45
NewsRoom (2018)	EN	News	1.21M	751	30
BigPatent (2019)	EN	Academic	1.34M	3573	117
arXiv (2018)	EN	Academic	216K	6914	293
PubMed (2018)	EN	Academic	133K	3224	214
Gigawords (2012)	EN	News	4.02M	31	8
LCSTS (2015)	ZH	News	2.40M	104	17
XSum (2018)	EN	News	227K	431	23
MATINF-SUMM	ZH	Health	1.07M	42	9
Table 5: Comparison of summarization datasets. "Token" indicates the average token numbers of a document and a summary for each dataset.
overfitting bias caused by evaluation on merely one dataset. We compare MATINF-SUMM with other datasets in Table 5.
4 Multi-task Learning
Recently, many attempts have been made on multi-task learning in NLP (Liu et al., 2015; Luong et al., 2016; Guo et al., 2018; McCann et al., 2018; Xu et al., 2019; Ruder et al., 2019; Liu et al., 2019; Radford et al., 2019; Dong et al., 2019; Shen et al., 2019; Raffel et al., 2019; Lei et al., 2020) and several benchmarks are available for multi-task evaluation (Wang et al., 2019a,b). Though recent studies show that multi-task learning is effective, there is still one more question to answer. That is, when training models on multiple tasks, multiple datasets are used by default. As illustrated in Figure 2(a), it adds both new input (i.e., text, denoted as  $X$ ) and new supervision (i.e., ground truths, denoted as  $Y$ ). Due to the different processes of data collection,  $X$  in different datasets have different sources and properties. Recent progress on Language Modeling (Radford et al., 2019; Devlin et al., 2019; Yang
MATINF
(b)
(a)
(c)
Figure 2: The difference between MTF-S2S and traditional multi-task learning.
et al., 2019; Raffel et al., 2019) has proved that corpora  $(X)$  from different sources can make the model more robust and significantly improve the performance. To this end, it is not easy to determine whether the success of a multi-task model should be mainly attributed to the addition of  $X$  or  $Y$ . However, as depicted in Figure 2(b), in MATINF, our jointly labeled fashion can guarantee that  $X$  remains the same as in a single task and only  $Y$  is added. Thus, MATINF provides a fair and ideal stage for exploring multi-task learning, especially auxiliary and multi-task supervision under a single dataset.
To set a baseline and also inspire future research, we design a multi-task learning network, named
Figure 3: The architecture of MTF-S2S. Note that a common attention mechanism (Luong et al., 2015) is applied when decoding question and answer (in the blue and green boxes), but we do not illustrate it in this figure for clarity.
Multi-task Field-shared Sequence to Sequence (MTF-S2S). We illustrate the architecture of MTF-S2S in Figure 3. For generation tasks, we combine the summarization  $(D\rightarrow Q)$  and QA  $(Q\rightarrow A)$  to be the form of  $D\to Q\to A$  with a shared Long Short-Term Memory (LSTM) for decoding questions in summarization task and encoding questions for both QA and classification tasks. Previous studies often share layers among tasks to regularize the representation learning, as illustrated in Figure 2(c). Different from that, MTF-S2S shares on both module level (i.e., field encoder/decoder, as shown in Figure 2(d)) and layer level. An attention mechanism is applied when decoding for summarization and QA. Also, we concatenate the encoded representations of description and question, and feed it to a shared fully connected layer and then specialized fully connected layers for age classification and topic classification, respectively.
When training, since the sizes of datasets for different tasks are not equal, we first determine the batch size for different tasks to make sure that the training progress for each task is approximately
synchronized by:
$$
\forall a, b \in T, b s _ {a} / b s _ {b} = n _ {a} / n _ {b} \tag {1}
$$
where  $T$  includes four tasks: summarization, QA, and two classification tasks.  $bs_{*}$  is the batch size of each task, and  $n_*$  is the sample numbers in each dataset for the task. If one task is iterated to the last data batch, it will start over from the first batch. For each iteration, we successively calculate the losses by Cross Entropy for each task in one batch. Then, we train the model to minimize the total loss:
$$
\mathcal {L} = \sum_ {t _ {i} \in T} \lambda_ {i} \mathcal {L} _ {i} \tag {2}
$$
where  $\lambda_{*}$  is the manually set weight for each task. We stop the co-training after one epoch, then finetune the model to obtain the peak performance for each task, separately.
In this section, we benchmark a few baselines and MTF-S2S on the three tasks of MATINF. We run each experiment with three different random seeds and report the average result of the three runs.
MTF-S2S. For MTF-S2S, we set all  $\lambda_{i} = 0.25$  and use an Adam (Kingma and Ba, 2015) optimizer to co-train the model for one epoch with batch sizes of 64, 64, 12 and 52 for  $bs_{Summ}$ ,  $bs_{QA}$ ,  $bs_{CTopic}$ , and  $bs_{CAge}$  respectively with a learning rate of 0.001. Then we fine-tune the model for each task with a learning rate of  $5 \times 10^{-5}$ . We report both the performance after co-training and after fine-tuning. The hidden size of all LSTM encoders/decoders and attentions is 200. For all tasks, we separately train MTF-S2S on each task only to provide a single-task baseline. Both MTF-S2S and Seq2Seq baselines are character-based and their embeddings are initialized with Tencent AI Lab Embedding (Song et al., 2018). For both MTF-S2S and Seq2Seq baselines, we use Beam Search (Wiseman and Rush, 2016) when decoding.
Classification. For classification, we conduct experiments with a statistical learning baseline, several deep neural networks and pretrained large-scale language models. For the statistical baselines, we extract character-based unigram and bigram features and use a logistic classifier to predict the classes. For neural networks, we choose