SlowGuess/ACL_2020_OCR huggingface dataset:
fastText (Grave et al., 2017), Text CNN (Kim, 2014), DCNN (Kalchbrenner et al., 2014), RCNN (Lai et al., 2015) and DPCNN (Johnson and Zhang, 2017). As a classical step in Chinese text classification, we segment the sentences into words with  $\text{Jieba}^2$ , a commonly used out-of-the-box word segmentation toolkit. We then initialize the word embedding with pretrained Tencent AI Lab Embedding (Song et al., 2018) except for fastText, which has its own algorithm to construct word embeddings. We minimize the Cross-Entropy with Adam (Kingma and Ba, 2015) optimizer with a learning rate of 0.001 and apply early stopping. For language models, we fine-tune BERT (Devlin et al., 2019) and ERNIE (Sun et al., 2019) that both have released official pretrained Chinese models. We set the learning rate for fine-tuning to  $5 \times 10^{-5}$  and apply early stopping. We also compress the fine-tuned 12-layer BERT model with BERT-of-Theseus (Xu et al., 2020) and obtain the performance of a 6-layer model.
Question Answering. For retrieval-based QA, following MS-MARCO (Nguyen et al., 2016), we calculate the average best scores between each answer in the test set and all answers in the training set within the same class, to determine the oracle retrieval performance. Then, we construct our retrieval-based baseline by fine-tuning BERT-Base (Devlin et al., 2019) for question matching on an external dataset, LCQMC (Liu et al., 2018). Then we use the trained model to score the match between each question in the test set and all questions in the training set with the same class and return the answer of the top 1 matched question. For generation-based baselines, we use character-based Seq2Seq (Sutskever et al., 2014) and Seq2Seq with Attention (Luong et al., 2015), since character-based method has a prominently better performance for Chinese text generation (Hu et al., 2015; Li et al., 2019). The metric for evaluation are ROUGE scores (Lin and Hovy, 2003) calculated on the character level.
Summarization. We categorize the baselines into two fashions: extractive methods (i.e., extracting sentences or phrases from the text) and abstractive methods (i.e., generating summaries according to the text). For extractive methods, we choose two widely used classical methods, TextRank (Mihalcea and Tarau, 2004) and LexRank (Erkan and
Method	AGE	TOPIC
TF-IDF + LR†	76.88	40.25
Text CNN (Kim, 2014)	90.95	64.41
DCNN (Kalchbrenner et al., 2014)	90.96	64.60
RCNN (Lai et al., 2015)	90.81	63.56
fastText (Grave et al., 2017)	87.76	61.81
DPCNN (Johnson and Zhang, 2017)	91.02	65.92
BERT†Base (Devlin et al., 2019)	90.33	66.95
BERT-of-Theseus† (Xu et al., 2020)	90.25	66.72
ERNIE (Sun et al., 2019)	90.42	66.66
MTF-S2S (single task)†	90.15	63.40
MTF-S2S†	90.29	63.59
Table 6: Experimental results of baseline methods on MATINF-C in terms of accuracy. †: Character-based models.
Method	MATINF-QA
R-1	R-2	R-L
Best Passage (upper bound)	58.32	36.42	49.00
BERT Matching (2019)	18.66	3.28	10.78
Seq2Seq (2014)	16.62	4.53	10.37
Seq2Seq + Att (2015)	19.62	5.87	13.34
MTF-S2S (single task)	20.28	5.94	13.52
MTF-S2S	21.66	6.58	14.26
Table 7: Experimental results of baseline methods on MATINF-QA.
Radev, 2004). For abstractive methods, we use WEAN (Ma et al., 2018) and Global Encoding (Lin et al., 2018) along with Seq2Seq (Sutskever et al., 2014; Luong et al., 2015) as the baselines. We also add BertAbs (Liu and Lapata, 2019), a BERT-based summarization model, to reflect the recent progress on this task. We use the officially released Chinese BERT-Base as the backbone. We use ROUGE scores (Lin and Hovy, 2003) to evaluate the quality of generated summaries.
Classification. We show the experimental results of two classification sub-tasks in Table 6. On the tougher MATLAB-C-TOPIc, language models prominently outperform other baselines. Among non-LM neural networks, DPCNN (Johnson and Zhang, 2017), which has the deepest architecture and the most parameters, outperforms other baselines with a considerable margin. On MATLAB-CAGE, which is a smaller dataset with fewer classes, DPCNN outperforms all other baselines including
$^{2}$ https://github.com/fxsjy/jieba. We use Jieba v0.39 throughout this paper.
Method	CNN/DM	LCSTS	MATINF-SUMM
R-1	R-2	R-L	R-1	R-2	R-L	R-1	R-2	R-L
TextRank (Mihalcea and Tarau, 2004)	37.72	15.59	33.81	24.38	11.97	16.76	35.53	25.78	36.84
LexRank (Erkan and Radev, 2004)	33.98	11.79	30.17	22.15	10.14	14.65	33.08	23.31	34.96
Seq2Seq (Sutskever et al., 2014)	-	-	-	-	-	-	23.05	11.44	19.55
Seq2Seq + Att (Luong et al., 2015)	31.33	11.81	28.83	33.80	23.10	32.50	43.05	28.03	38.58
WEAN (Ma et al., 2018)	-	-	-	37.80	25.60	35.20	34.63	22.56	28.92
Global Encoding (Lin et al., 2018)	-	-	-	39.40	26.90	36.50	49.28	34.14	47.64
BertAbs (Liu and Lapata, 2019)	40.21	17.76	37.09	-	-	-	57.31	44.05	55.93
MTF-S2S (single task)	31.36	11.80	28.88	33.75	23.20	32.51	43.02	28.05	38.55
MTF-S2S	-	-	-	-	-	-	48.59	35.69	43.28
Table 8: Experimental results of baseline methods on CNN / DM (Hermann et al., 2015), LCSTS (Hu et al., 2015), and MATINF-SUMM.
language models with an accuracy of 91.02. To analyze, this task has fewer training samples, which is in favor of a model with moderate parameter numbers instead of huge parameter numbers as in language models. Also, the task is relatively easier due to the class number, which makes the advantage of language models more trivial. For the multi-task baseline, MTF-S2S shows a satisfying performance on both MATINF-C-AGE and MATINF-C-TOPIC, outperforming the same model which is only trained on the single task by 0.14 and 0.19 in terms of accuracy. Notably, BERT-of-Theseus (Xu et al., 2020) has a satisfying performance compressing the fine-tuned BERT to smaller models.
Question Answering. The experimental results are shown in Table 7. The high scores of Best Passage (maximum possible performance) indicate that using training data as a document set is completely feasible. Seq2Seq with Attention outperforms the retrieval-based baseline by a margin of 2.56 in terms of ROUGE-L. It suggests that a generation-based neural network can effectively learn from multiple relevant samples and generalize. Besides, since we do the matching between each question and every entry within the same class in the training set, the inference of BERT Matching takes quite a long time. Similar to MS-MARCO (Nguyen et al., 2016), it is possible to use a search engine (e.g., Elastic Search) to pre-filter the documents and reduce the computational cost. Meanwhile, MTF-S2S is effective on QA task and outperforms its single-task version by 0.74 on ROUGE-L.
Summarization. We further conduct performance comparison for summarization across three datasets, CNN/DM (Hermann et al., 2015), LC
STS (Hu et al., 2015), and our MATINF-SUMM in Table 8. By comparing the performance of two basic baselines, TextRank (Mihalcea and Tarau, 2004) and Seq2Seq+Att (Luong et al., 2015), we can see an obvious difference in performance between extractive and abstractive methods on datasets of different genres. BertAbs (Liu and Lapata, 2019), the powerful BERT-based model, significantly outperforms all other baselines on MATINF-SUMM thanks to its exploitation of pretraining and the capacity of a BERT model. For MTF-S2S, it outperforms the single-task counterpart by 4.73 on ROUGE-L.
Since MATINF is a web-crawled dataset, it would be inevitable to be noisier than a dataset annotated by hired annotators though we have made every effort to clean the data. On the bright side, it can encourage more robust models and facilitate real-world applications. For future work, we would like to see more interesting work exploring new multi-task learning approaches.
To conclude, in this paper, we present MATINF, a jointly labeled large-scale dataset for classification, question answering and summarization. We benchmark existing methods and a straightforward baseline with a novel multi-task paradigm on MATINF and analyze their performance on these three tasks. Our extensive experiments reveal the potential of the proposed dataset for accelerating the innovations in the three tasks and multi-task learning.
3593
We are grateful for the insightful comments from the anonymous reviewers. This research was supported by National Natural Science Foundation of China (No. 61872278). Chenliang Li is the corresponding author.
Shaosheng Cao, Wei Lu, Jun Zhou, and Xiaolong Li. 2018. cw2vec: Learning chinese word embeddings with stroke n-gram information. In AAAI.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In ACL.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In NAACL-HLT.
Yiming Cui, Ting Liu, Zhipeng Chen, Shijin Wang, and Guoping Hu. 2016. Consensus attention-based neural networks for chinese reading comprehension. In COLING.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. 2009. Imagenet: A large-scale hierarchical image database. In CVPR.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In NeurIPS.
Nan Duan and Duyu Tang. 2017. Overview of the NLPCC 2017 shared task: Open domain chinese question answering. In NLPCC.
Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Güney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new q&a dataset augmented with context from a search engine. CoRR, abs/1704.05179.
Günes Erkan and Dragomir R. Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. J. Artif. Intell. Res.
Edouard Grave, Tomas Mikolov, Armand Joulin, and Piotr Bojanowski. 2017. Bag of tricks for efficient text classification. In EACL.
Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In *NAACL-HLT*.
Han Guo, Ramakanth Pasunuru, and Mohit Bansal. 2018. Soft layer-specific multi-task summarization with entailment and question generation. In ACL.
Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, Xuan Liu, Tian Wu, and Haifeng Wang. 2018. Dureader: a chinese machine reading comprehension dataset from real-world applications. In QA@ACL.
Karl Moritz Hermann, Tomás Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In NeurIPS.
Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016. The goldilocks principle: Reading children's books with explicit memory representations. In ICLR.
Baotian Hu, Qingcai Chen, and Fangze Zhu. 2015. LC-STS: A large scale chinese short text summarization dataset. In EMNLP.
Rie Johnson and Tong Zhang. 2017. Deep pyramid convolutional neural networks for text categorization. In ACL.
Dan Jurafsky and James H. Martin. 2009. Speech and language processing: an introduction to natural language processing, computational linguistics, and speech recognition, 2nd Edition. Prentice Hall series in artificial intelligence. Prentice Hall, Pearson Education International.
Nal Kalchbrenner, Edward Grefenstette, and Phil Blun-som. 2014. A convolutional neural network for modelling sentences. In ACL.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR.
Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Recurrent convolutional neural networks for text classification. In AAAI.
Wenqiang Lei, Xiangnan He, Yisong Miao, Qingyun Wu, Richang Hong, Min-Yen Kan, and Tat-Seng Chua. 2020. Estimation-action-reflection: Towards deep interaction between conversational and recommender systems. In WSDM.
Xiaoya Li, Yuxian Meng, Xiaofei Sun, Qinghong Han, Arianna Yuan, and Jiwei Li. 2019. Is word segmentation necessary for deep learning of Chinese representations? In ACL.
Chin-Yew Lin and Eduard H. Hovy. 2003. Automatic evaluation of summaries using n-gram cooccurrence statistics. In HLT-NAACL.
Junyang Lin, Xu Sun, Shuming Ma, and Qi Su. 2018. Global encoding for abstractive summarization. In ACL.
Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2017. Adversarial multi-task learning for text classification. In ACL.
Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, and Ye-Yi Wang. 2015. Representation learning using multi-task deep neural networks for semantic classification and information retrieval. In NAACL-HLT.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. Multi-task deep neural networks for natural language understanding. In ACL.
Xin Liu, Qingcai Chen, Chong Deng, Huajun Zeng, Jing Chen, Dongfang Li, and Buzhou Tang. 2018. LCQMC: A large-scale chinese question matching corpus. In COLING.
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In EMNLP/IJCNLP.
Minh-Thang Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2016. Multi-task sequence to sequence learning. In  $ICLR$ .
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In EMNLP.
Shuming Ma, Xu Sun, Wei Li, Sujian Li, Wenjie Li, and Xuancheng Ren. 2018. Query and output: Generating words by querying distributed word representations for paraphrase generation. In NAACL-HLT.
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. CoRR, abs/1806.08730.
Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In EMNLP.
Courtney Naples, Matthew R. Gormley, and Benjamin Van Durme. 2012. Annotated gigaword. In AKBC-WEKEX@NAACL-HLT.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In EMNLP.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In CoCo@NeurIPS.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In EMNLP.
Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. 2019. Latent multi-task architecture learning. In AAAI.
Eva Sharma, Chen Li, and Lu Wang. 2019. BIG-PATENT: A large-scale dataset for abstractive and coherent summarization. In ACL.
Tao Shen, Xiubo Geng, Tao Qin, Daya Guo, Duyu Tang, Nan Duan, Guodong Long, and Daxin Jiang. 2019. Multi-task learning for conversational question answering over a large-scale knowledge base. In EMNLP/IJCNLP.
Yan Song, Shuming Shi, Jing Li, and Haisong Zhang. 2018. Directional skip-gram: Explicitly distinguishing left and right context for word embeddings. In NAACL-HLT.
Baohua Sun, Lin Yang, Patrick Dong, Wenhan Zhang, Jason Dong, and Charles Young. 2018. Super characters: A conversion from sentiment classification to image classification. In WASSA@EMNLP.
Maosong Sun, Jingyang Li, Zhipeng Guo, Yu Zhao, Yabin Zheng, Xiance Si, and Zhiyuan Liu. 2016. THUCTC: An efficient chinese text classifier.
Yu Sun, Shuohuan Wang, Yu-Kun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. ERNIE: enhanced representation through knowledge integration. CoRR, abs/1904.09223.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In NeurIPS.
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. Newsqa: A machine comprehension dataset. In Rep4NLP@ACL.
Ellen M. Voorhees and Dawn M. Tice. 1999. The TREC-8 question answering track evaluation. In TREC.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. In NeurIPS.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR.
Canhui Wang, Min Zhang, Shaoping Ma, and Liyun Ru. 2008a. Automatic online news issue construction in web environment. In WWW.
Canhui Wang, Min Zhang, Shaoping Ma, and Liyun Ru. 2008b. Automatic online news issue construction in web environment. In WWW.
Sam Wiseman and Alexander M. Rush. 2016. Sequence-to-sequence learning as beam-search optimization. In EMNLP.
Wei Wu, Yuxian Meng, Qinghong Han, Muyu Li, Xiaoya Li, Jie Mei, Ping Nie, Xiaofei Sun, and Jiwei Li. 2019. Glyce: Glyph-vectors for chinese character representations. In NeurIPS.
Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. 2020. Bert-of-theseus: Compressing BERT by progressive module replacing. CoRR, abs/2002.02925.
Yichong Xu, Xiaodong Liu, Yelong Shen, Jingjing Liu, and Jianfeng Gao. 2019. Multi-task learning with sample re-weighting for machine reading comprehension. In NAACL-HLT.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In NeurIPS.
Guanhua Zhang, Bing Bai, Jian Liang, Kun Bai, Shiyu Chang, Mo Yu, Conghui Zhu, and Tiejun Zhao. 2019. Selection bias explorations and debias methods for natural language sentence matching datasets. In ACL.
Xiang Zhang and Yann LeCun. 2017. Which encoding is the best for text classification in chinese, english, japanese and korean? CoRR, abs/1708.02657.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In NeurIPS.
Zhuosheng Zhang and Hai Zhao. 2018. One-shot learning for question-answering in gaokao history challenge. In COLING.