SlowGuess/ACL_2020_OCR huggingface dataset:
QUASE: Question-Answer Driven Sentence Encoding
Hangfeng He† Qiang Ning‡* Dan Roth†  
†University of Pennsylvania ‡Allen Institute for AI hangfeng, danroth@seas.upenn.edu qiangn@allenai.org
Question-answering (QA) data often encodes essential information in many facets. This paper studies a natural question: Can we get supervision from QA data for other tasks (typically, non-QA ones)? For example, can we use QAMR (Michael et al., 2017) to improve named entity recognition? We suggest that simply further pre-training BERT is often not the best option, and propose the question-answer driven sentence encoding (QUASE) framework. QUASE learns representations from QA data, using BERT or other state-of-the-art contextual language models. In particular, we observe the need to distinguish between two types of sentence encodings, depending on whether the target task is a single- or multisentence input; in both cases, the resulting encoding is shown to be an easy-to-use plugin for many downstream tasks. This work may point out an alternative way to supervise NLP tasks. $^{1}$
It is labor-intensive to acquire human annotations for NLP tasks which require research expertise. For instance, one needs to know thousands of semantic frames in order to provide semantic role labelings (SRL) (Palmer et al., 2010). It is thus an important research direction to investigate how to get supervision signals from indirect data and improve one's target task. This paper studies the case of learning from question-answering (QA) data for other tasks (typically not QA). We choose QA because (1) a growing interest of QA has led to many large-scale QA datasets available to the community; (2) a QA task often requires comprehensive understanding of language and may encode rich information that
is useful for other tasks; (3) it is much easier to answer questions relative to a sentence than to annotate linguistics phenomena in it, making this a plausible supervision signal (Roth, 2017).
There has been work showing that QA data for task  $\mathcal{A}$  can help another QA task  $\mathcal{T}$ , conceptually by further pre-training the same model on  $\mathcal{A}$  (an often larger) before training on  $\mathcal{T}$  (a smaller) (Talmor and Berant, 2019; Sun et al., 2019). However, it remains unclear how to use these QA data when the target task does not share the same model as the QA task, which is often the case when the target task is not QA. For instance, QA-SRL (He et al., 2015), which uses QA pairs to represent those predicate-argument structures in SRL, should be intuitively helpful for SRL parsing, but the significant difference in their surface forms prevents us from using the same model in both tasks.
The success of modern language modeling techniques, e.g., ELMo (Peters et al., 2018), BERT (Devlin et al., 2019), and many others, has pointed out an alternative solution to this problem. That is, to further pre-train<sup>2</sup> a neural language model (LM) on these QA data in certain ways, obtain a sentence encoder, and use the sentence encoder for the target task, either by fine-tuning or as additional feature vectors. We call this general framework question-answer driven sentence encoding (QUASE). A straightforward implementation of QUASE is to first further pre-train BERT (or other LMs) on the QA data in the standard way, as if this QA task is the target, and then fine-tune it on the real target task. This implementation is technically similar to STILTS (Phang et al., 2018), except that
2We clarify three types of training: pre-training, further pre-training, and fine-tuning. Pre-training refers to the training of sentence encoders on unlabeled text; further pre-training refers to continuing training the sentence encoders on an intermediate, non target-task-specific labeled data (e.g. QA data); fine-tuning refers to training on the target task in the fine-tuning approach.
*Part of this work was done while the author was at the University of Illinois at Urbana-Champaign.  
<sup>1</sup>Our code and online demo are publicly available at https://github.com/CogComp/QuASE.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8743-8758 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
STILTS is mainly further pre-trained on textual entailment (TE) data.
However, similar to the observations made in STILTS and their follow-up works (Wang et al., 2019), we find that additional QA data does not necessarily help the target task using the implementation above. While it is unclear how to predict this behaviour, we do find that this happens a lot for tasks whose input is a single sentence, e.g., SRL and named entity recognition (NER), instead of a sentence pair, e.g., TE. This might be because QA is itself a paired-sentence task, and the implementation above (i.e., to further pre-train BERT on QA data) may learn certain attention patterns that can transfer to another paired-sentence task more easily than to a single-sentence task. Therefore, we argue that, for single-sentence target tasks, QUASE should restrict the interaction between the two sentence inputs when it further pre-trains on QA data. We propose a new neural structure for this and name the resulting implementation s-QUASE, where "s" stands for "single;" in contrast, we name the straightforward implementation mentioned above p-QUASE for "paired." Results show that s-QUASE outperforms p-QUASE significantly on 3 single-sentence tasks—SRL, NER, and semantic dependency parsing (SDP)—indicating the importance of this distinction.
Let  $\mathrm{QUASE}_{\mathcal{A}}$  be the QUASE further pre-trained on QA data  $\mathcal{A}$ . We extensively compare 6 different choices of  $\mathcal{A}$ : TriviaQA (Joshi et al., 2017), NewsQA (Trischler et al., 2017), SQuAD (Rajpurkar et al., 2016), relation extraction (RE) dataset in QA format (QA-RE for short) (Levy et al., 2017), Large QA-SRL (FitzGerald et al., 2018), and QAMR (Michael et al., 2017). Interestingly, we find that if we use s-QUASE for single-sentence tasks and p-QUASE for paired-sentence tasks, then  $\mathrm{QUASE}_{\mathrm{QAMR}}$  improves all 7 tasks<sup>3</sup> in low resource settings, with an average error reduction rate of  $7.1\%$  compared to BERT.<sup>4</sup> While the set of tasks we experimented with here is nonexhaustive, we think that  $\mathrm{QUASE}_{\mathrm{QAMR}}$  has the potential of improving on a wide range of tasks.
This work has three important implications. First, it provides supporting evidence to an important alternative to supervising NLP tasks: using QA to annotate language, which has been discussed in works such as QA-SRL, QAMR, and
QA-RE. If it is difficult to teach annotators the formalism of a certain task, perhaps we can instead collect QA data that query the target phenomena and thus get supervision from QA for the original task (and possibly more). Second, the distinction between s-QUASE and p-QUASE suggests that sentence encoders should consider some properties of the target task (e.g., this work distinguishes between single- and multi-sentence tasks). Third, the good performance of QUASE $_{QAMR}$  suggests that predicate-argument identification is an important capability that many tasks rely on; in contrast, many prior works observed that only language modeling would improve target tasks generally.
2 QA Driven Sentence Encoding
This work aims to find an effective way to use readily available QA data to improve a target task that is typically not QA. A natural choice nowadays—given the success of language models—is to further pre-train sentence encoders, e.g. BERT, on QA data in certain ways, and then use the new encoder in a target task. This general framework is called QUASE in this work, and the assumption is that the sentence encoders learned from QA data have useful information for the target task.
A straightforward implementation of QUASE is to further pre-train BERT on QA data in the standard way, i.e., fine-tune BERT as if this QA dataset is the target task, and then fine-tune BERT on the real target task. However, we find that this straightforward implementation is less effective or even negatively impacts target tasks with single-sentence input; similar observations were also made in STILTS (Phang et al., 2018) and its follow-ups (Wang et al., 2019): They further pretrain sentence encoders, e.g., ELMo, BERT, and GPT (Radford et al., 2018), on TE data and find that it is not effective for the syntax-oriented CoLA task and the SST sentiment task in GLUE, which are both single-sentence tasks (Wang et al., 2018).
One plausible reason is that the step of further pre-training on QA data does not take into account some properties of the target task, for instance, the number of input sentences. QA is inherently a paired-sentence task; a typical setup is, given a context sentence and a question sentence, predict the answer span. Further pre-training BERT on QA data will inevitably learn how to attend to the context given the question. This is preferable when the target task is also taking a pair of sentences
$^{3}$ SRL, SDP, NER, RE, co-reference resolution (Coref), TE and machine reading comprehension (MRC).
4 BERT is close to the state-of-the-art in all these tasks.
8744
(a) s-QUASE for single-sentence tasks
(b) p-QUASE for paired-sentence tasks
Figure 1: Two implementations of QUASE: s-QUASE for single-sentence tasks, and p-QUASE for paired-sentence tasks. Both structures are further pre-trained on QA data, and the parts in the black boxes are used by target tasks. While p-QUASE is the standard way of fine-tuning BERT, s-QUASE restricts the interaction between the sentence and the question. Specifically, the sentence encodings in s-QUASE does not depend on the existence of the question. More details are given in Sec. 2.2 and Appendix A (including experimental settings in A.1, error analysis in A.2 and ablation analysis in A.3).
as input, while it may be irrelevant or harmful for single-sentence tasks. It points out that we may need two types of sentence encodings when further pre-training BERT on QA data, depending on the type of the target task. The following subsection discusses this issue in detail.
2.1 Two Types of Sentence Encodings
Standard sentence encoding is the problem of converting a sentence  $S = [w_{1}, w_{2}, \dots, w_{n}]$  to a sequence of vectors  $h(S) = [h_{1}, h_{2}, \dots, h_{n}]$  (e.g., skip-thoughts (Kiros et al., 2015)). Ideally,  $h(S)$  should encode all the information in  $S$ , so that it is task-agnostic: given a target task, one can simply probe  $h(S)$  and retrieve relevant information. In practice, however, only the information relevant to the training task of  $h(S)$  is kept. For instance, when we have a task with multi-sentence input (e.g., QA and TE), the attention pattern  $A$  among these sentences will affect the final sentence encoding, which we call  $h_{A}(S)$ ; in comparison, we denote the sentence encoding learned from single-sentence tasks by  $h(S)$ , since there is no cross-sentence attention  $A$ . In a perfect world, the standard sentence encoding  $h(S)$  expresses also the conditional sentence encoding  $h_{A}(S)$ . However, we believe that there is a trade-off between the quality and the quantity of semantic information a model can encode. Our empirical results corroborate this conclusion and
more details can be found in Appendix A.2.
The distinction between the sentence encodings types may explain the negative impact of using QA data for some single-sentence tasks: Further pre-training BERT on QA data essentially produces a sentence encoding with cross-sentence attentions  $h_A(S)$ , while the single-sentence tasks expect  $h(S)$ . These two sentence encodings may be very different: One view is from the theory of information bottleneck (Tishby et al., 1999; Tishby and Zaslavsky, 2015), which argues that training a neural network on a certain task is extracting an approximate minimal sufficient statistic of the input sentences with regard to the target task; information irrelevant to the target task is maximally compressed. In our case, this corresponds to the process where the conditional sentence encoding compresses the information irrelevant to the relation, which will enhance the quality but reduce the quantity of the sentence information.
2.2 Two Implementations of QUASE
In order to fix this issue, we need to know how to learn  $h(S)$  from QA data. However, since QA is a paired-sentence task, the attention pattern between the context sentence and the question sentence is important for successful further pre-training on QA. Therefore, we propose that if the target task is single-sentence input, then fur
8745
ther pre-training on QA data should also focus on single-sentence encodings in the initial layers; the context sentence should not interact with the question sentence until the very last few layers. This change is expected to hurt the capability to solve the auxiliary QA task, but it is later proved to transfer better to the target task. This new treatment is called s-QUASE with "s" representing "single-sentence," while the straightforward implementation mentioned above is called p-QUASE where "p" means "paired-sentence." The specific structures are shown in Fig. 1.
2.2.1 s-QUASE
The architecture of s-QUASE is shown in Fig. 1(a). When further pre-training it on QA data, the context sentence and the question sentence are fed into two pipelines. We use the same Sentence2Question and Question2Sentence attention as used in BiDAF (Seo et al., 2017). Above that, "Sentence Modeling," "Question Modeling," and "Interaction Layer" are all bidirectional transformers (Vaswani et al., 2017) with 2 layers, 2 layers, and 1 layer, respectively. Finally, we use the same classification layer as BERT, which is needed for training on QA data. Overall, this implementation restricts interactions between the paired-sentence input, especially from the question to the context, because when serving the target task, this attention will not be available.
Using s-QUASE in target tasks. Given a sentence  $S$ , s-QUASE can provide a sequence of hidden vectors  $h(S)$ , i.e., the output of the "Sentence Modeling" layer in Fig. 1(a). Although  $h(S)$  does not rely on the question sentence,  $h(S)$  is optimized so that upper layers can use it to handle those questions in the QA training data, so  $h(S)$  indeed captures information related to the phenomena queried by those QA pairs. For single-sentence tasks, we use  $h(S)$  from s-QUASE as additional features, and concatenate it to the word embeddings in the input layer of any specific neural model.
2.2.2 p-QUASE
The architecture of p-QUASE is shown in Fig. 1(b), which is the standard way of pre-training BERT. That is, when further pre-training it on QA data, the context sentence and the question sentence form a single sequence (separated by special tokens) and are fed into BERT.
Using p-QUASE in target tasks. Given a sentence pair S (concatenated), p-QUASE produces  $h_A(S)$ , i.e., the output of the BERT module in Fig. 1(b). One can of course continue fine-tuning p-QUASE on the target task, but we find that adding p-QUASE to an existing model for the target task is empirically better (although not very significant); specifically, we try to add  $h_A(S)$  to the final layer before the classification layer, and we also allow p-QUASE to be updated when training on the target task, although it is conceivable that other usages may lead to even stronger results. For instance, when the target task is token classification, e.g., MRC, we can simply concatenate the vectors of  $h_A(S)$  at each timestamp to any existing model; when the target task is sentence classification, e.g., TE, we apply max-pooling and average-pooling on  $h_A(S)$ , respectively, and concatenate the two resulting vectors to any existing model before the final classification layer.
2.3 Related Work on Sentence Encoding
Modern LMs are essentially sentence encoders pretrained on unlabeled data and they outperform early sentence encoders such as skip-thoughts (Kiros et al., 2015). While an LM like BERT can handle lexical and syntactic variations quite well, it still needs to learn from some annotations to acquire the "definition" of many tasks, especially those requiring complex semantics (Tenney et al., 2019). Although we extensively use BERT here, we think that the specific choice of LM is orthogonal to our proposal of learning from QA data. Stronger LMs, e.g., RoBERTa (Liu et al., 2019) or XLNet (Yang et al., 2019), may only strengthen the proposal here. This is because a stronger LM represents unlabeled data better, while the proposed work is about how to represent labeled data better.
CoVe (McCann et al., 2017) is another attempt to learn from indirect data, translation data specifically. However, it does not outperform ELMo or BERT in many NLP tasks (Peters et al., 2018) and probing analysis (Tenney et al., 2019). In contrast, our QUASE will show stronger experimental results than BERT on multiple tasks. In addition, we think QA data is generally cheaper to collect than translation data.
The proposed work is highly relevant to Phang et al. (2018) and their follow-up works (Wang et al., 2019), which use further pre-training on data-rich intermediate supervised tasks and aim
<sup>5</sup>We mainly use concatenation in both types of QUASE. However, we also use replacement in some experiments and we will note these cases later in this paper.
8746
Single-sentence	Paired-sentence
System	SRL	RE	TE	MRC
BERT	34.17	62.99	78.29	79.90
BERTQAMR	32.92	50.16	78.73	82.96
Table 1: The naive way of training BERT on QAMR  $(\mathrm{BERT}_{QAMR})$  negatively impacts single-sentence tasks. We only use  $10\%$  training data for simplicity. We use BERT/BERT  $Q_{AMR}$  to produce feature vectors for a BiLSTM model (SRL) and a CNN model (RE); for TE and MRC, we fine-tune BERT/BERT  $Q_{AMR}$ .
to improve another target task. The key differences are as follows: First, we distinguish two types of sentence encodings, which provide explanation to their puzzle that sentence-pair tasks seem to benefit more from further pre-training than single-sentence tasks do. Second, they only focus on fine-tuning based methods which cannot be easily plugged in many single-sentence tasks such as SRL and Coref, while we analyze both fine-tuning based and feature-based approaches. Third, they mainly use TE signals for further pre-training, and evaluate their models on GLUE (Wang et al., 2018) which is a suite of tasks very similar to TE. Our work instead makes use of QA data to help tasks that are typically not QA. Fourth, from their suite of further pre-training tasks, they observe that only further pre-training on language modeling tasks has the power to improve a target task in general, while we find that QAMR may also have this potential, indicating the universality of predicate-argument structures in NLP tasks.
Our work is also related to Sentence-BERT (Reimers and Gurevych, 2019) in terms of providing a better sentence representation. However, their focus was deriving semantically meaningful sentence embeddings that can be compared using cosine-similarity, which reduces the computational cost of finding the most similar pairs. In contrast, QUASE provides a better sentence encoder in the same format as BERT (a sequence of word embeddings) to better support tasks that require complex semantics.
3 Applications of QUASE
In this section, we conduct thorough experiments to show that QUASE is a good framework to get supervision from QA data for other tasks. We first give an overview of the datasets and models used in these experiments before diving into the details
of each experiment.
Specifically, we use PropBank (Kingsbury and Palmer, 2002) (SRL), the dataset from the SemEval'15 shared task (Oepen et al., 2015) with DELPH-IN MRS-Derived Semantic Dependencies target representation (SDP), CoNLL'03 (Tjong Kim Sang and De Meulder, 2003) (NER), the dataset in SemEval'10 Task 8 (Hendrickx et al., 2009) (RE), the dataset in the CoNLL'12 shared task (Pradhan et al., 2012) (Coref), MNLI (Williams et al., 2018) (TE), and SQuAD 1.0 (Rajpurkar et al., 2016) (MRC). In Table 4, we use CoNLL'12 English subset of OntoNotes 5.0 (Pradhan et al., 2013), which is larger than PropBank. The performance of TE and MRC is evaluated on the development set. $^{6}$
For single-sentence tasks, we use both simple baselines (e.g., BiLSTM and CNN; see Appendix B.1) and near-state-of-the-art models published in recent years. As in ELMo, we use the deep neural model in He et al. (2017) for SRL, the model in Peters et al. (2018) for NER, and the end-to-end neural model in Lee et al. (2017) for Coref. We also use the biaffine network in Dozat and Manning (2018) for SDP but we removed part-of-speech tags from its input, and the attention-based BiLSTM in Zhou et al. (2016) is the strong baseline for RE. In addition, we replace the original word embeddings in these models (e.g., GloVe (Pennington et al., 2014)) by BERT. Throughout this paper, we use the pre-trained case-insensitive BERT-base implementation. More details on our experimental setting can be found in Appendix B, including the details of simple models in B.1, some common experimental settings of QUASE in B.2, and s-QUASE combined with other SOTA embeddings (ELMo and Flair (Akbik et al., 2018)) in B.3.
3.1 Necessity of Two Representations
We first consider a straightforward method to use QA data for other tasks—to further pre-train BERT on these QA data. We compare BERT further pre-trained on QAMR (denoted by  $\mathrm{BERT}_{QAMR}$ ) with BERT on two single-sentence tasks (SRL and RE) and two paired-sentence tasks (TE and MRC). We use a feature-based approach for single-sentence tasks and a fine-tuning approach for paired-sentence tasks. The reason is two-fold. On the one hand, current SOTAs of all single-sentence tasks considered in this paper are still
For TE, we mean matched examples in MNLI.
8747
Single-Sentence Tasks	Paired-Sentence Tasks
Tasks	SRL	SDP	NER	TE	MRC
Split	10%	100%	10%	100%	10%	100%	10%	30%	10%	100%
s-QUASE	46.42	70.13	76.08	87.29	70.69	87.10	52.25	57.30	44.67	67.09
p-QUASE	32.92	66.40	70.92	86.43	49.97	85.23	57.29	60.49	48.29	72.97
Table 2: Probing results of the sentence encoders from s-QUASE and p-QUASE. In all tasks, we fix the model QUASE and use the sentence encodings as input feature vectors for the model of each task. In order to keep the model structure as simple as possible, we use BiLSTM for SRL, NER, and TE, Biaffine for SDP, and BiDAF for MRC. We compare on  $10\%$  and  $100\%$  of the data in all tasks except TE, where we use  $30\%$  to save run-time.
(a) s-QUASE
(b) p-QUASE
Figure 2: Sample complexity analysis of using BERT and QUASE on SRL and MRC. We find that much fewer training examples are needed with the help of  $\mathrm{QUASE}_{QAMR}$ : with  $50\%$  SRL training data, s-QUASE can achieve comparable performance as BERT trained on  $100\%$ ; with  $0.1\%$  training data for MRC, p-QUASE can achieve a reasonably good performance of  $69.81\%$ .
feature-based. How to efficiently use sentence encoders (e.g. BERT) in a fine-tuning approach for some complicated tasks (e.g. SRL and SDP) is unclear. On the other hand, the fine-tuning approach shows great advantage over feature-based on many paired-sentence tasks (e.g. TE and MRC). Similar to Phang et al. (2018), we find in Table 1 that the two single-sentence tasks benefit less than the two paired-sentence tasks from  $\mathrm{BERT}_{QAMR}$ , which indicates that simply "further pre-training BERT" is not enough.
We then compare s-QUASE $_{QAMR}$  and p-QUASE $_{QAMR}$  on three single-sentence tasks (SRL, SDP and NER) and two paired-sentence tasks (TE and MRC) to show that it is important to distinguish two types of sentence representations. Rather than concatenating two embeddings as proposed in Sec. 2.2, here we replace BERT embeddings with QUASE embeddings for convenience. The results are shown in Table 2. We find that s-QUASE has a great advantage over p-QUASE on single-sentence tasks and p-QUASE is better than s-QUASE on paired-sentence tasks. The proposal of two types of sentence encoders tackles the problem one may encounter when there is only further