SlowGuess/ACL_2020_OCR huggingface dataset:
pre-training BERT on QAMR for single-sentence tasks. In summary, it is necessary to distinguish two types of sentence representations for single-sentence tasks and paired-sentence tasks.
3.2 Sample Complexity of QUASE
To see whether adding QUASE to BERT reduces the sample complexity, we compare  $\mathrm{QUASE}_{QAMR}$  with BERT on one single-sentence task (SRL) and one paired-sentence task (MRC) with different percentages of training examples. For convenience, we replace BERT embeddings with QUASE embeddings for SRL. As shown in Figure 2, we find that s-  $\mathrm{QUASE}_{QAMR}$  outperforms BERT on SRL with small training data, and p-  $\mathrm{QUASE}_{QAMR}$  outperforms BERT on MRC with small training data. The results support that (1) adding QUASE to BERT reduces the sample complexity, (2) QUASE is very important in the low-resource setting. For instance, s-  $\mathrm{QUASE}_{QAMR}$  achieves an F1 score of 61 in SRL with  $30\%$  (27K) training examples (compared to 50.92 F1 by BERT). And p-  $\mathrm{QUASE}_{QAMR}$  achieves 69.81 average F1 on MRC with  $0.1\%$  (about 100) training examples (compared to 13.29 F1 by BERT).
8748
Models	s-QUASE	p-QUASE
Tasks	SRL	SDP	NER	RE	TE	Avg
Split	small	full	small	full	small	full	small	full	small	full	small	full
BERT	34.17	66.02	75.49	90.13	88.89	91.38	71.48	86.33	78.29	84.09	69.66	83.59
QUASE	50.16	72.59	78.30	90.78	90.64	92.16	77.14	86.80	78.94	84.97	75.04	85.46
TriviaQA	17.75	39.69	77.29	90.43	89.74	91.70	75.41	86.80	78.50	84.95	67.74	78.71
NewsQA	27.99	53.05	77.27	90.41	89.96	91.65	73.08	85.88	78.85	84.30	69.43	81.06
SQuAD	34.35	61.86	76.90	90.51	89.94	91.07	77.14	85.80	78.21	84.97	71.31	82.84
QA-RE	35.50	65.85	78.30	90.78	90.64	91.73	63.36	85.80	78.94	84.68	69.35	83.77
Large QA-SRL	50.16	72.59	76.92	90.68	90.12	91.73	68.99	85.46	78.88	84.61	73.01	85.01
QAMR	46.42	70.13	77.53	90.57	89.90	92.16	72.23	86.37	78.73	84.79	72.96	84.80
Table 3: Further pre-training QUASE on different QA datasets of the same number of QA pairs (51K). As we propose, s-QUASE is used as features for single-sentence tasks, and p-QUASE is further fine-tuned for the paired-sentence task. The specific models are all strong baselines except for SRL, where we use a simple BiLSTM model to save run-time. "Small" means  $10\%$  training examples for all tasks except NER, where "small" means the dev set (about  $23\%$ ) of the corresponding training set. We further show the results of QUASE with the best QA dataset, which are significantly better than those of BERT.
Single-Sentence Tasks	Paired-Sentence Tasks
Small	SRL	SDP	NER	RE	Coref	TE	MRC	Avg
BERT	76.65	75.49	88.89	71.48	62.76	78.29	79.90	76.21
Proposed (abs. imp.)	+3.95	+2.04	+1.01	+0.75	+0.60	+0.44	+3.06	+1.69
Proposed (rel. imp.)	16.9%	8.3%	9.1%	2.6%	1.6%	2.0%	15.2%	7.1%
Full	SRL	SDP	NER	RE	Coref	TE	MRC	Avg
BERT	84.54	90.13	91.38	86.33	69.05	84.09	88.23	84.82
Proposed (abs. imp.)	+0.15	+0.44	+0.78	+0.04	-0.14	+0.7	+0.35	+0.33
Proposed (rel. imp.)	0.9%	4.5%	9.0%	0.3%	-0.5%	4.4%	3.0%	2.2%
Table 4: QUASE $Q_{AMR}$  (almost) universally improves on 5 single-sentence tasks and 2 paired-sentence tasks. Note BERT is close to the state of the art for these tasks. Both absolute improvement (abs. imp.) and relative improvement (rel. imp.; error reduction rate) are reported. "Small/Full" refers to the size of training data for each target task. For SDP, RE, TE, and MRC, "small" means  $10\%$  of the training set, while for NER, SRL, and Coref, "small" means the development set (about  $10\%-30\%$  compared to each training set).
3.3 Data Choice for Further Pre-training
We compare BERT with QUASE further pretrained with the same numbre of QA pairs on 6 different QA datasets (TriviaQA (Joshi et al., 2017), NewsQA (Trischler et al., 2017), SQuAD, QA-RE (Levy et al., 2017), Large QA-SRL (FitzGerald et al., 2018), and QAMR). s-QUASE further pretrained on different QA datasets are evaluated on four single-sentence tasks in a feature-based approach: SRL, SDP, NER and RE. p-QUASE further pre-trained on different QA datasets is evaluated on one task (TE) in a fine-tuning approach.
In Table 3, we find that the best options are quite different across different target tasks, which is expected because a task usually benefits more from a more similar QA dataset. However, we also find that QAMR is generally a good further-pre-training choice for QUASE. This is consistent with our intuition: First, QAMR has a simpler concept class than other paragraph-level QA datasets,
such as TriviaQA, NewsQA and SQuAD. It is easier for QUASE to learn a good representation with QAMR to help sentence-level tasks. Second, QAMR is more general than other sentence-level QA datasets, such as QA-RE and Large QA-SRL.<sup>7</sup> Therefore, we think that the capability to identify predicate-argument structures can generally help many sentence-level tasks, as we discuss next.
3.4 The Effectiveness of QUASE
Here we compare  $\mathrm{QUASE}_{QAMR}$  with BERT on 5 single-sentence tasks and 2 paired-sentence tasks, where  $\mathrm{QUASE}_{QAMR}$  is further pre-trained on the training set (51K QA pairs) of the QAMR dataset. As shown in Table 4, we find that  $\mathrm{QUASE}_{QAMR}$
7Although the average performance of QUASEQAMR on five tasks is slightly below  $\mathrm{QUASELargeQA - SRL}$  , for which the benefit mostly comes from SRL. QUASE is mainly designed to improve a lot of tasks, so QAMR is a better choice in our setup, but in practice, we do not limit QUASE to any specific QA dataset and one can use the best one for corresponding target tasks.
has a better performance than BERT on both single-sentence tasks and paired-sentence tasks, especially in the low-resource setting $^{8}$ , indicating that  $\mathrm{QUASE}_{QAMR}$  can provide extra features compared to BERT.
Admittedly, the improvement in the "Full" setting is not significantly large, but we think that this is expected because large direct training data are available (such as SRL with  $278K$  training examples in OntoNotes). However, it is still promising that  $51K$  indirect QA pairs can improve downstream tasks in the low-resource setting (i.e. several thousands direct training examples). That is because they help the scalability of machine learning methods, especially for some specific domains or some low-resource languages where direct training data do not exist in large scale.
In this section we discuss a few issues pertaining to improving QUASE by using additional QA datasets and the comparison of QUASE with related symbolic representations.
4.1 Further Pre-training QUASE on Multiple QA Datasets
We investigate whether adding the Large QA-SRL dataset (FitzGerald et al., 2018) or the QA-RE $^9$  dataset into QAMR in the further pre-training stage can help SRL and RE. We use s-QUASE embeddings to replace BERT embeddings instead of concatenating the two embeddings. The effectiveness of adding existing resources (Large QA-SRL or QA-RE) into QAMR in the further pre-training stage of s-QUASE on SRL and RE are shown in Table 5. We find that adding related QA signals (Large QA-SRL for SRL and QA-RE for RE) into QAMR can help improve specific tasks. Noteworthy is the fact that QA-RE can help SRL (Large QA-SRL can also help RE), though the improvement is minor compared to Large QA-SRL (QA-RE). These results indicate that adding more QA signals related to the sentence can help get a better sentence representation in general.
Tasks	SRL	RE
Split	10%	100%	10%	100%
BERT	34.16	66.02	59.36	83.28
QUASEQAMR	46.42	70.13	61.09	82.22
QUASEQAMR+Large QA-SRL	49.92	71.74	65.76	83.16
QUASEQAMR+QA-RE	47.25	72.52	68.12	83.89
Table 5: The potential of further improving  $\mathrm{QUASE}_{QAMR}$  by further pre-training it on more QA data. The "+" between datasets means union with shuffling. Both Large QA-SRL and QA-RE help achieve better results than QAMR alone. For simplicity, we use a simple BiLSTM model for SRL and a simple CNN model for RE. See more in Appendix B.
4.2 Comparison with Symbolic Meaning Representations
Traditional (symbolic) shallow meaning representations such as SRL and AMR, suffer from having a fixed set of relations one has to commit to. Moreover, inducing these representations requires costly annotation by experts. Proposals such as QA-SRL, QAMR, semantic proto-roles (Reisinger et al., 2015), and universal dependencies (White et al., 2016) avoid some of these issues by using natural language annotations, but it is unclear how other tasks can take advantage of them. QUASE is proposed to facilitate inducing distributed representations instead of symbolic representations from QA signals; it benefits from cheaper annotation and flexibility, and can also be easily used in downstream tasks.
The following probing analysis, based on the Xinhua subset in the AMR dataset, shows that  $\mathrm{s - QUASE}_{QAMR}$  embeddings encode more semantics related to AMR than BERT embeddings. Specifically, we use the same edge probing model as Tenney et al. (2019), and find that the probing accuracy (73.59) of  $\mathrm{s - QUASE}_{QAMR}$  embeddings is higher than that (71.58) of BERT. At the same time, we find that  $\mathrm{p - QUASE}_{QAMR}$  can achieve 76.91 F1 on the PTB set of QA-SRL, indicating that  $\mathrm{p - QUASE}_{QAMR}$  can capture enough information related to SRL to have a good zero-shot SRL performance. More details can be found in Appendix C.1. Another fact worth noting is that AMR can be used to improve downstream tasks, such as MRC (Sachan and Xing, 2016), TE (Lien and Kouylekov, 2015), RE (Garg et al., 2019) and SRL (Song et al., 2018). The benefits of  $\mathrm{QUASE}_{QAMR}$  on downstream tasks show that we can take advantage of AMR by learning from much cheaper QA signals dedicated to it.
Another interesting finding is that simple models usually benefit more from QUASE embeddings than SOTA models. Because the training set of QA-RE is too large, we randomly choose 100,000 training examples.
4.3 Difficulties in Learning Symbolic Representations from QA Signals
QUASE is designed to learn distributed representations from QA signals to help down-stream tasks. We further show the difficulties of learning two types of corresponding symbolic representations from QA signals, which indicates that the two other possible methods are not as tractable as ours.
One option of symbolic representation is the QAMR graph. Michael et al. (2017) show that question generation for QAMR representations can only achieve a precision of  $28\%$ , and a recall of  $24\%$ , even with fuzzy matching (multi-BLEU $^{10}$ $>0.8$ ). Furthermore, it is still unclear how to use the complex QAMR graph in downstream tasks. These results indicate that learning a QAMR parser for downstream tasks is mainly hindered by question generation, and how to use the full information of QAMR for downstream tasks is still unclear.
Another choice of symbolic representation is AMR, since QAMR is proposed to replace AMR. We consider a simpler setting, learning an SRL parser from Large QA-SRL. We propose three models in different perspectives, but the best performance of them is only 54.10 F1, even with fuzzy matching (Intersection/Union  $\geq 0.5$ ). More details can be found in Appendix C.2. Although a lot of methods (Khashabi et al., 2018; Marcheggiani and Titov, 2017; Strubell et al., 2018) can be adopted to use SRL/AMR in downstream tasks, the difficulty of learning a good SRL/AMR parser from QA signals hinders this direction.
The difficulties of learning the two types of symbolic representations from QA signals indicate that our proposal of learning distributed representations from QA signals is a better way of making use of the latent semantic information in QA pairs for down-stream tasks.
In this paper, we investigate an important problem in NLP: Can we make use of low-cost signals, such as QA data, to help related tasks? We retrieve signals from sentence-level QA pairs to help NLP tasks via two types of sentence encoding approaches. For tasks with a single-sentence input, such as SRL and NER, we propose s-QUASE that provides latent sentence-level representations; for tasks with a sentence pair input, such as TE and MRC we propose p-QUASE, that generates latent
representations related to attentions. Experiments on a wide range of tasks show that the distinction of s-QUASE and p-QUASE is highly effective, and  $\mathrm{QUASE}_{QAMR}$  has the potential to improve on many tasks, especially in the low-resource setting.
This material is based upon work supported by the US Defense Advanced Research Projects Agency (DARPA) under contracts FA8750-19-2-0201, W911NF-15-1-0461, and FA8750-19-2-1004, a grant from the Army Research Office (ARO), and Google Cloud. The views expressed are those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government.
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In COLING, pages 1638-1649.
M. Chang, L. Ratinov, and D. Roth. 2007. Guiding semi-supervision with constraint-driven learning. In Proc. of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 280-287.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In *NAACL*, pages 4171–4186.
Timothy Dozat and Christopher D. Manning. 2018. Simpler but more accurate semantic dependency parsing. In ACL, pages 484-490.
Nicholas FitzGerald, Julian Michael, Luheng He, and Luke Zettlemoyer. 2018. Large-scale QA-SRL parsing. In ACL, pages 2051-2060.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2017. AllenNLP: A deep semantic natural language processing platform.
Sahil Garg, Aram Galstyan, Greg Ver Steeg, Irina Rish, Guillermo Cecchi, and Shuyang Gao. 2019. Kernelized hashingcode representations for relation extraction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6431-6440.
Luheng He, Kenton Lee, Mike Lewis, and Luke Zettlemoyer. 2017. Deep semantic role labeling: What works and what's next. In ACL, pages 473-483.
Luheng He, Mike Lewis, and Luke Zettlemoyer. 2015. Question-answer driven semantic role labeling: Using natural language to annotate natural language.
10An average of BLEU1-BLEU4 scores.
8751
In Proc. of the Conference on Empirical Methods for Natural Language Processing (EMNLP), pages 643-653.
Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid O Seaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2009. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of the Workshop on Semantic Evaluations: Recent Achievements and Future Directions, pages 94-99.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601-1611. Association for Computational Linguistics.
Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Dan Roth. 2018. Question answering as global reasoning over semantic abstractions. In Proceedings of The Conference on Artificial Intelligence (Proc. of the Conference on Artificial Intelligence (AAAI)).
Paul Kingsbury and Martha Palmer. 2002. From treebank to propbank. In LREC, pages 1989-1993. Cite-seer.
Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in neural information processing systems, pages 3294-3302.
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettle-moyer. 2017. End-to-end neural coreference resolution. In EMNLP, pages 188-197.
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In CoNLL, pages 333-342.
Elisabeth Lien and Milen Kouylekov. 2015. Semantic parsing for textual entailment. In Proceedings of the 14th International Conference on Parsing Technologies, pages 40-49.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
Diego Marcheggiani and Ivan Titov. 2017. Encoding sentences with graph convolutional networks for semantic role labeling. In EMNLP, pages 1506-1515.
Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In NeurIPS, pages 6294-6305.
Julian Michael, Gabriel Stanovsky, Luheng He, Ido Dagan, and Luke Zettlemoyer. 2017. Crowdsourcing question-answer meaning representations. *NAACL*.
Stephan Oepen, Marco Kuhlmann, Yusuke Miyao, Daniel Zeman, Silvie Cinkova, Dan Flickinger, Jan Hajic, and Zdenka Urešová. 2015. SemEval 2015 task 18: Broad-coverage semantic dependency parsing. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 915–926.
M. Palmer, D. Gildea, and N. Xue. 2010. Semantic Role Labeling, volume 3.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP, pages 1532-1543.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In *NAACL*, pages 2227-2237.
Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using onthonotes. In CoNLL, pages 143-152.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in onthonotes. In Joint Conference on EMNLP and CoNLL-Shared Task, pages 1-40.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assetss/researchcovers/languageeunsupervised/language understanding paper. pdf.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP, pages 2383-2392.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pages 3973-3983.
Drew Reisinger, Rachel Rudinger, Francis Ferraro, Craig Harman, Kyle Rawlins, and Benjamin Van Durme. 2015. Semantic proto-roles. Transactions of the Association for Computational Linguistics, 3:475-488.
8752
Dan Roth. 2017. Incidental supervision: Moving beyond supervised learning. In Proc. of the Conference on Artificial Intelligence (AAAI).
Mrinmaya Sachan and Eric Xing. 2016. Machine comprehension using rich semantic representations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 486-492.
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. *ICLR*.
Li Song, Yuan Wen, Sijia Ge, Bin Li, Junsheng Zhou, Weiguang Qu, and Nianwen Xue. 2018. An easier and efficient framework to annotate semantic roles: Evidence from the chinese amr corpus. In The 13th Workshop on Asian Language Resources, page 29.
Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-informed self-attention for semantic role labeling. In EMNLP, pages 5027-5038.
Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. 2019. Improving machine reading comprehension with general reading strategies. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2633-2643, Minneapolis, Minnesota. Association for Computational Linguistics.
Alon Talmor and Jonathan Berant. 2019. MultiQA: An empirical investigation of generalization and transfer in reading comprehension. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4911-4921. Association for Computational Linguistics.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. ICLR.
Naftali Tishby, Fernando C Pereira, and William Bialek. 1999. The information bottleneck method. In Proc. of the Annual Allerton Conference on Communication, Control and Computing.
Naftali Tishby and Noga Zaslavsky. 2015. Deep learning and the information bottleneck principle. In IEEE Information Theory Workshop (ITW).
Erik F Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proc. of the Annual Meeting of the North American Association of Computational Linguistics (NAACL).
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. NewsQA: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191-200. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, et al. 2019. Can you tell me how to get past sesame street? sentence-level pretraining beyond language modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4465-4476.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355.
Aaron Steven White, Drew Reisinger, Keisuke Sakaguchi, Tim Vieira, Sheng Zhang, Rachel Rudinger, Kyle Rawlins, and Benjamin Van Durme. 2016. Universal Decompositional Semantics on Universal Dependencies. In Empirical Methods in Natural Language Processing (EMNLP).
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In *NAACL*, pages 1112–1122.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237.
Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. Attention-based bidirectional long short-term memory networks for relation classification. In ACL, pages 207-212.
8753
A Additional Details for QUASE
In this section, we show the experimental details of QUASE. We first show the experimental settings of training p-QUASE and s-QUASE in Section A.1. After that, we conduct error analysis of QUASE to show the shortcomings of QUASE in Section A.2. Finally, the ablation analysis of s-QUASE is in Section A.3.
A.1 Experimental Settings