SlowGuess/NAACL_2022_OCR huggingface dataset:
Avi Caciularu\* Ido Dagan\* Jacob Goldberger\* Arman Cohan\*\*
$\clubsuit$ Computer Science Department, Bar-Ilan University, Ramat-Gan, Israel
$^{\dagger}$ Faculty of Engineering, Bar-Ilan University, Ramat-Gan, Israel
$\diamond$ Allen Institute for AI, Seattle, WA
$^{\diamond}$ Paul G. Allen School of Computer Science, University of Washington, Seattle, WA
avi.c33@gmail.com, armanc@allenai.org
dagan@cs.biu.ac.il, jacob.goldberger@biu.ac.il
Long-context question answering (QA) tasks require reasoning over a long document or multiple documents. Addressing these tasks often benefits from identifying a set of evidence spans (e.g., sentences), which provide supporting evidence for answering the question. In this work, we propose a novel method for equipping long-context QA models with an additional sequence-level objective for better identification of the supporting evidence. We achieve this via an additional contrastive supervision signal in finetuning, where the model is encouraged to explicitly discriminate supporting evidence sentences from negative ones by maximizing question-evidence similarity. The proposed additional loss exhibits consistent improvements on three different strong long-context transformer models, across two challenging question answering benchmarks – Hot-potQA and QAsper. $^{1}$
Answering questions that require reasoning over a long sequence, such over long documents or multiple documents, is a challenging task (Pang et al., 2021). Research in this domain mostly includes tasks that involve multiple text segments, over benchmarks like HotpotQA (Yang et al., 2018) and QAsper (Dasigi et al., 2021). HotpotQA is a multi-hop QA benchmark over multiple paragraphs from Wikipedia, while QAsper involves reading comprehension from long academic papers, where relevant information on a question could be spread across the paper.
Given the task complexity (Choi et al., 2017), benchmarks often provide an additional set of evidence spans, such as sentences or paragraphs, for a given question answer pair. This breaks down the long-context QA task, adding a preliminary
evidence span detection, which is crucial for successfully finding the correct answer, and also potentially helps in model interpretability. In this work, we propose a method for improving long-context QA via leveraging such evidence spans, by maximizing their similarity with the question.
Since identifying the evidences provides relevant information for answering the question, prior work showed that jointly training models to perform evidence span extraction in addition to answer generation is important for achieving high performance (Yang et al., 2018; Dasigi et al., 2021). To jointly perform evidence extraction and question answering, models utilize sentence representations (marker tokens) in the input; the final layer representation corresponding to these markers is then passed through a classification layer and is optimized using the cross-entropy loss in conjunction with the answer extraction/generation loss. We conjecture and demonstrate (see Table 1) that this objective does not sufficiently capture relationships between the question and the candidate evidence spans. Thus, we propose a complementary objective for enforcing question-evidence similarity in the model representation (see Fig. 1). Further, we show that optimizing the question-evidence similarity under a certain subspace by using linear projection of the raw representations may softly impose information encoding about their relatedness. Since questions may be partitioned into several types (e.g., yes/no, generative, non answerable, etc.) in the common QA settings, we also investigate learning separate projections per question type.
Driven by the intuition that questions should be similar to their supporting evidences, under some specific geometric sub-space, we propose a novel supervised contrastive learning objective for the finetuning stage, aiming to maximize similarity of question-evidence representations. Contrastive learning has been recently applied to a variety of
* Work partly done as an intern at AI2.
Our code is available at https://github.com/aviclu/long-context-qa-contrast.
Human Language Technologies, pages 2872 - 2879
Figure 1: Demonstration of our method over an example instance from QAsper (Dasigi et al., 2021). A long sequence is fed to a model, producing representations for the marker special tokens. Then, these vectors are used to compute our contrastive objective. The token colored in blue represents the question, and the tokens colored in green (red) represent positive (negative) evidence sentences. The goal is to maximize the similarity between the blue vector and the green vectors.
deep learning models in computer vision (Chen et al., 2020; Chen and He, 2021) and NLP (Gao et al., 2021; Gunel et al., 2021). Unlike prior NLP related methods, our proposed loss term is model-agnostic and operates in the sequence-level in a supervised manner. This objective targets question and evidence representations within input sequences, and unlike prior work, it is not based on individually encoding sentences or paragraphs. We show that our additional contrastive supervision provides consistent improvements on three different models and two datasets, demonstrating its effectiveness and versatility.
In this section, we define and elaborate the common setup and notations for long-context QA.
Assume that we are given a question  $q_{i}$  and a context  $S_{i} = \langle s_{1},\dots,s_{M}\rangle$  consisting of  $M$  sentences ( $s_{j}$  can be also a document/paragraph/passage, depending on the dataset). From  $S_{i}$ , the task is to identify the correct answer  $a_{i}$  and a set of  $N$  evidence spans  $S_{i}^{+} = \{s_{i_{1}},\dots,s_{i_{N}}\}$  where  $i_{j}$  are indices of the sentences that are the supporting evidence for answering the question  $q_{i}$ .
As common in the input setup of long-context transformer models (Beltagy et al., 2020; Zaheer et al., 2020; Caciularu et al., 2021), which are the
current state-of-the-art models for solving longcontext QA tasks, the question and context sentences are concatenated in a single long sequence with special sentence tokens specifying sentence boundaries. Then the input is passed to the longcontext transformer, which is trained to jointly identify the evidence sentences and extract or generate the answer.
Concretely, for each example, we prepare the following concatenated input sequence:
$$
\left[ \langle \sf s \rangle , q _ {i}, \langle \sf / \sf s \rangle , s _ {1}, \langle \sf / \sf s \rangle , s _ {2}, \dots , \langle \sf / \sf s \rangle , s _ {M} \right]
$$
where  $"\,"$  is the string concatenation operation,  $q_{i}$  and  $s_{j}$  are sequences of tokens corresponding to the question and the  $j$ th sentence in the input context, and  $\langle \mathsf{s} \rangle$  and  $\langle / \mathsf{s} \rangle$  are special tokens representing the question and a context sentence, respectively (See Fig. 1 for an example).
Then, a QA loss, which we denote by  $\mathcal{L}_{QA}$  is applied over the contextualized representation of each sentence token, and is optimized using supervision.  $\mathcal{L}_{QA}$  depends on the dataset and can take the form of a multi-task objective, representing multiple tasks in the context of QA (Dasigi et al., 2021) (particularly evidence extraction and answer generation).
3 Question-Evidence Contrastive Loss
In this section, we elaborate on our novel proposed contrastive loss term, combining question-evidence similarity maximization (§3.1), and question-type aware projections (§3.2).
3.1 Question-Evidence Similarity
To encourage the long-context transformer model to capture relationships between the question and evidence sentences, we introduce an additional sequence-level loss that compares and contrasts the question with context sentences.
The additional proposed loss  $\mathcal{L}_{QE}$  is based on the InfoNCE loss (Oord et al., 2018), and is applied over instances consisting of triplet of vectors representing the question, an evidence sentence and distractor sentences. The loss encourages the question and evidence representations to become closer to each other, while pushing the question and distracting sentences away.
Formally, the contrastive loss is defined as the sum of negative log-likelihood losses over all input examples, where each loss term discriminates the positive units from negative ones:
$$
\mathcal {L} _ {Q E _ {i}} = - \log \sum_ {s ^ {+} \in \mathcal {S} _ {i} ^ {+}} \left(\frac {e ^ {s i m (s ^ {+} , q _ {i}) / \tau_ {k}}}{\sum_ {s \in \mathcal {S} _ {i}} e ^ {s i m (s , q _ {i}) / \tau_ {k}}}\right), \tag {1}
$$
where  $s(q_i)$  is the sentence (question) marker vector representation (see  $\langle s\rangle$  and  $\langle /s\rangle$  in Fig. 1),  $\tau_{k}$  is the configurable temperature hyperparameter, and  $sim(\cdot)$  is a similarity metric.  $\mathcal{L}_{QE_i}$  serves a single example, and the final aggregated loss  $\mathcal{L}_{QE}$  is obtained by averaging over all the examples.
We incorporate  $\mathcal{L}_{QE}$  into the main QA span extraction/generation loss  $\mathcal{L}_{QA}$  using the augmented loss:
$$
\mathcal {L} = (1 - \lambda) \cdot \mathcal {L} _ {Q A} + \lambda \cdot \mathcal {L} _ {Q E},
$$
where  $\lambda$  is a weighting hyperparameter.
The underlying  $sim(\cdot)$  can take the form of a non-parametric similarity function, e.g., the dot product  $(sim(s,q) = s^\top q)$  or the cosine similarity  $(sim(s,q) = \frac{s^\top q}{\|s\|\|q\|})$ . However, we show empirically that using such similarity over the raw representations harms the performance results of the model, since seemingly, it is hard to find a shared representation that should optimize the two loss functions. Hence, we adopted linear projections, per question type, to cast the similarity learning objective into proxy linear spaces.
3.2 Incorporating Question-Type Projections
Long-context QA benchmarks often provide a question-type label per instance as an additional challenge, such as {yes, no, span} for HotpotQA. We hypothesize that maximizing question-evidence similarity under a question-type-specific sub-space can enable more flexibility and inductive bias to the model, for producing better representations and further improving the performance. Following Iter et al. (2020), we define the following similarity function:
$$
\operatorname {s i m} _ {k} (s, q) = s ^ {\top} W _ {k} q, \tag {2}
$$
where  $k$  is the expected question type and  $W_{k}$  is the corresponding learnable projection matrix. Such linear projections ensure that a specific subspace per question type exists. We additionally incorporate different temperature hyperparameters  $\tau_{k}$  per question type in Eq. 1 (see the ablation in Table 4 for their effect). The dimensions of the proposed  $W_{k}$  tend to be large, in accordance with the dimensions of the transformer's hidden-layers. Hence, following Barkan et al. (2020), we apply new nonsquare linear projections instead of using  $W_{k}$ :
$$
\operatorname {s i m} _ {k} (s, q) = \frac {s _ {k} ^ {\top} q _ {k}}{\| s _ {k} \| \| q _ {k} \|}, \tag {3}
$$
where we set
$$
s _ {k} := W _ {k} ^ {S} s, \quad q _ {k} := W _ {k} ^ {Q} q,
$$
and  $W_{k}^{S}$  (or  $W_{k}^{Q}$ ) is the matrix that projects  $s$  (or  $q$ ) into a lower dimension, in the  $k^{\text{th}}$  question-type sub-space. In order to improve the separation between the different sub-spaces induced by different question types, we generated additional negative instances per sentence, as follows.
We projected every question-sentence pair using all the mappings according to the available question types, and computed their cosine similarity (according to Eq. 3). Then, all the obtained scores were considered as negative, except the ones that belong to question-evidence pairs projected using the correct question-type mapping.
4 Evaluation and Analysis
In this section, we provide details about the experiments that we conducted and their outcomes.
2Empirically, we end up with  $W_{k}\in \mathbb{R}^{768\times 768}$  for base-sized models and  $W_{k}\in \mathbb{R}^{1024\times 1024}$  for large-sized models.
Method	Yes	No	Span
Trained only with LA	8.4	4.9	46.4
+LQE using Eq. 3 with a single projection	72.4	65.2	77.5
+LQE using Eq. 3	86.3	84.2	87.6
Table 1: Mean Average Precision (mAP) question-evidence cosine similarity results per question type for the HotpotQA-distractor dev set.
In order to demonstrate the contribution of our method, we evaluated it over the recent QAsper dataset (Dasigi et al., 2021) and the well-known HotpoQA dataset (Yang et al., 2018), which share the same setup ( $\S 2$ ).
QAsper (Dasigi et al., 2021) is a long-document QA dataset which was built over academic papers, where NLP practitioners were recruited to (abstractedly) generate questions following the title and the abstract of a particular paper, as well as creating the the correct evidence and answers to those questions based on the entire paper. More than half of the examples in QAsper require collecting evidences from multiple evidences in the given paper. For this benchmark,  $\mathcal{L}_{QA}$  represents the sum of the teacher-forced text generation and evidence classification loss functions, in a multi-task training setup.
HotpotQA (Yang et al., 2018) introduced the task of multihop extractive question answering, in the reading comprehension setting, where the inputs are a question and multiple paragraphs from various related and non-related documents. A model is queried to extract answer spans and evidence sentences, where it should handle challenging questions that require finding and reasoning over multiple supporting documents. For the models we applied to this benchmark,  $\mathcal{L}_{QA}$  represents the standard cross-entropy answer extraction loss.
To test the contribution of our  $\mathcal{L}_{QE}$  loss, We replicated the experiments described in Dasigi et al. (2021) and Caciularu et al. (2021) for QAsper and HotpotQA, respectively. For QAsper, we finetuned the LED-base model, $^3$  and evaluated it on the question answering and evidence selection tasks. For HotpotQA, we used the Longformer model and CDLM $^4$  as the backbone long-sequence language models for this task. Since CDLM was provided
only as a base-sized model, we pretrained a larger version of the CDLM model, and hence used both the base and large versions of both Longformer and CDLM. For further details see Appendix B.1 and B.2.
For both benchmarks, we performed a grid search for determining the hyper-parameters of the contrastive loss (more details in Appendix C).
Qustion-Evidence Similarity Analysis. As a preliminary assessment, we first investigate the question and evidences representations of models trained on the HopotQA dataset. We motivate the use of our method by presenting the mean Average Precision (mAP) ranking results produced according to the question-sentence cosine similarities for the marker tokens trained representations of the CDLM-large model. From Table 1, we observe that without applying our additional loss term, the question representations are distant from the evidence representations. Using a single learned projection increases this desirable similarity, and using a learned projection per question type yields the highest mAP scores. Hence, integrating question-type aware linear projections can be a beneficial part of our contrastive loss, and overall it further improves the QA results as we show next. An additional illustration of this effect, where we visualize the marker representations, appears in Appendix A.
Main results. We adopted the F1 evaluation metrics corresponding to the original works (Dasigi et al., 2021; Beltagy et al., 2020). Tables 2 and 3 present the evaluation results over the QAsper and HotpoQA datasets, respectively. We show the performance difference when adding our additional loss term with “+LQE” (and question-type similarity function from Eq. 3).
As shown in the table, the addition of  $\mathcal{L}_{QE}$  exhibits the best performance across all examined models and benchmarks, clearly demonstrating its consistent advantage. Note that maximizing the question-evidence similarity resulted also in evidence detection improvement - see the "Evidence" and the "Sup" metrics in Tables 2 and 3, respectively. All the results are statistically significant using the bootstrap test with  $p < 0.01$  (Dror et al., 2018).
Ablations. Table 4 demonstrates ablation study results for evaluating the effectiveness of our design
3According to Dasigi et al. (2021), LED-base outperforms LED-large over QAsper.
$^{4}$ CDLM was shown to be an effective long-range cross-encoder model for HotpotQA.
Input	Extractive	Abstractive	Yes/No	Unanswerable	Evidence	Overall
Dev.	Test	Dev.	Test	Dev.	Test	Dev.	Test	Dev.	Test	Dev.	Test
LED (2020)	26.1	31.0	16.6	15.8	67.5	70.3	28.6	26.2	23.9	29.9	29.1	32.8
+ LQE (Δ)	+0.2	+0.2	+0.8	+1.0	+1.6	+0.2	+1.5	+1.9	+1.0	+0.7	+0.9	+0.7
Table 2: Performance change when applying our additional loss  ${\mathcal{L}}_{QE}$  to the LED SOTA model on QAsper.
Model	Ans	Sup	Joint
base size	Longformer-base (2020)	74.5	83.9	64.5
+LQE(Δ)	+0.9	+0.2	+1.0
CDLM-base (2021)	74.7	86.3	66.3
+LQE(Δ)	+0.9	+0.3	+0.2
large size	Longformer-large (2020)	81.3	88.3	73.2
+LQE(Δ)	+0.3	+0.1	+0.9
CDLM-large (2021)	81.3	89.1	73.8
+LQE(Δ)	+0.3	+0.6	+0.8
Table 3: HotpotQA-distractor results  $(F_{1})$  for the dev set. We use the "base" and "large" model size results of CDLM and the Longformer for direct comparison. Ans: answer span, Sup: Supporting facts.
Joint	Δ
CDLM-large + LQE and sim(·) is Eq. 3	74.6
- using a single τ parameter for all the question types	74.2	-0.4
- sim(·) is the dot product	73.1	-1.5
- sim(·) is the cosine similarity	74.0	-0.6
- simk(·) is the bilinear distance (Eq. 2)	73.7	-0.9
- sim(·) is Eq. 3 with a single projection (Wk = W)	74.0	-0.6
- w/o incorrect question type projected negatives	74.2	-0.4
Table 4: Similarity function ablation results (Joint  $F_{1}$ ) of CDLM-large and our loss term on the HotpotQA-distractor dev set.
decisions. Using a constant temperature parameter for all question types, as well as using different degenerated similarity functions, exhibits lower performance. Further, the last row in Table 4 shows that treating correct answers that are projected to the wrong question type as negatives also improves the results. Overall, the ablation study shows the advantage of using Eq. 3 as a similarity function that provides fine-grained expressive modeling for each question type, in its own sub-space.
Discussion. An additional theoretical justification to our contrastive learning is provided in (Gao et al., 2021), where we can imply that our loss term improves the uniformity and therefore the expressiveness of the question and evidence representations. Moreover, one can attribute the success of our contrastive loss to the fact that long-range transformer models lack long-range signals during pretraining,
and hence such explicit modeling as we suggest is necessary. In fact, comparing CDLM's results to the Longformer illustrates that our cotrastive term has higher impact on models without global attention-based pretraining.
In this work, we proposed an effective sequence-level contrastive loss for improving the performance of long-range transformers in solving QA tasks that require reasoning over long contexts. We demonstrate consistent improvement when using our approach on three different models over two different benchmarks. For future work, we propose exploring variations of our proposed supervised loss on other long-context tasks, such as long-document and multi-document summarization, and integrating our method into information retrieval re-ranker models.
We thank the BIU-NLP lab and the Semantic Scholar research team at AI2 for fruitful discussions and helpful feedback. The work described herein was supported by the PBC fellowship for outstanding PhD candidates in data science, in part by the Israel Science Foundation grant 2827/21, and by a grant from the Israel Ministry of Science and Technology.
Our work in understanding the role of maximizing the similarity between question and evidence pairs. Therefore, there is a limited risk associated with the quality of annotated evidence sentences in the dataset, as there is no guarantee that our model will always generate non-biased and factual content. Therefore, caution must be exercised when the model is deployed in practical settings, where the evidence quality is vague and cannot be verified.
Oren Barkan, Avi Caciularu, and Ido Dagan. 2020. Within-between lexical relation classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3521-3527, Online. Association for Computational Linguistics.
Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.
Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew Peters, Arie Cattan, and Ido Dagan. 2021. CDLM: Cross-document language modeling. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 2648-2662, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016. Training deep nets with sublinear memory cost. arXiv preprint, abs/1604.06174.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML).
Xinlei Chen and Kaiming He. 2021. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750-15758.
Eunsol Choi, Daniel Hewlett, Jakob Uszkoreit, Illia Polosukhin, Alexandre Lacoste, and Jonathan Berant. 2017. Coarse-to-fine question answering for long documents. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 209-220, Vancouver, Canada. Association for Computational Linguistics.
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers anchored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4599-4610, Online. Association for Computational Linguistics.
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker's guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383-1392, Melbourne, Australia. Association for Computational Linguistics.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference