SlowGuess/NAACL_2021_OCR huggingface dataset:
Jifan Chen and Greg Durrett
{jfchen, gdurrett}@cs.utexas.edu
Current textual question answering (QA) models achieve strong performance on in-domain test sets, but often do so by fitting surface-level patterns, so they fail to generalize to out-of-distribution settings. To make a more robust and understandable QA system, we model question answering as an alignment problem. We decompose both the question and context into smaller units based on off-the-shelf semantic representations (here, semantic roles), and align the question to a subgraph of the context in order to find the answer. We formulate our model as a structured SVM, with alignment scores computed via BERT, and we can train end-to-end despite using beam search for approximate inference. Our use of explicit alignments allows us to explore a set of constraints with which we can prohibit certain types of bad model behavior arising in cross-domain settings. Furthermore, by investigating differences in scores across different potential answers, we can seek to understand what particular aspects of the input lead the model to choose the answer without relying on post-hoc explanation techniques. We train our model on SQuAD v1.1 and test it on several adversarial and out-of-domain datasets. The results show that our model is more robust than the standard BERT QA model, and constraints derived from alignment scores allow us to effectively trade off coverage and accuracy.
Current text-based question answering models learned end-to-end often rely on spurious patterns between the question and context rather than learning the desired behavior. They may ignore the question entirely (Kaushik and Lipton, 2018), focus primarily on the answer type (Mudrakarta et al., 2018), or otherwise bypass the "intended" mode of reasoning for the task (Chen and Durrett, 2019; Niven and Kao, 2019). Thus, these models are not robust to adversarial attacks (Jia and Liang,
Figure 1: A typical example on adversarial SQuAD. By breaking the question and context down into smaller units, we can expose the incorrect entity match and use explicit constraints to fix it. The solid lines denote edges from SRL and coreference, and the dotted lines denote the possible alignments between the arguments (desired in red, actual in black).
2017; Iyyer et al., 2018; Wallace et al., 2019): they can be fooled by surface-level distractor answers that follow the spurious patterns. Methods like adversarial training (Miyato et al., 2016; Wang and Bansal, 2018; Lee et al., 2019; Yang et al., 2019), data augmentation (Welbl et al., 2020), and posterior regularization (Pereyra et al., 2016; Zhou et al., 2019) have been proposed to improve robustness. However, these techniques often optimize for a certain type of error. We want models that can adapt to new types of adversarial examples and work under other distribution shifts, such as on questions from different text domains (Fisch et al., 2019).
In this paper, we explore a model for text-based question answering through sub-part alignment. The core idea behind our method is that if every aspect of the question is well supported by the answer context, then the answer produced should be trustable (Lewis and Fan, 2018); if not, we suspect that the model is making an incorrect prediction. The sub-parts we use are predicates and arguments from Semantic Role Labeling (Palmer et al., 2005), which we found to be a good semantic representation for the types of questions we studied. We then view the question answering procedure as a constrained graph alignment problem (Sachan and Xing, 2016), where the nodes represent the predi
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1251-1263 June 6-11, 2021. ©2021 Association for Computational Linguistics
cates and arguments and the edges are formed by relations between them (e.g. predicate-argument relations and coreference relations). Our goal is to align each node in the question to a counterpart in the context, respecting some loose constraints, and in the end the context node aligned to the wh-span should ideally contain the answer. Then we can use a standard QA model to extract the answer.
Figure 1 shows an adversarial example of SQuAD (Jia and Liang, 2017) where a standard BERT QA model predicts the wrong answer August 18, 1991. In order to choose the adversarial answer, our model must explicitly align Super Bowl 50 to Champ Bowl. Even if the model still makes this mistake, this error is now exposed directly, making it easier to interpret and subsequently patch.
In our alignment model, each pair of aligned nodes is scored using BERT (Devlin et al., 2019). These alignment scores are then plugged into a beam search inference procedure to perform the constrained graph alignment. This structured alignment model can be trained as a structured support vector machine (SSVM) to minimize alignment error with heuristically-derived oracle alignments. The alignment scores are computed in a black-box way, so these individual decisions aren't easily explainable (Jain and Wallace, 2019); however, the score of an answer is directly a sum of the score of each aligned piece, making this structured prediction phase of the model faithful by construction (Jain et al., 2020). Critically, this allows us to understand what parts of the alignment are responsible for a prediction, and if needed, constrain the behavior of the alignment to correct certain types of errors. We view this interpretability and extensibility with constraints as one of the principal advantages of our model.
We train our model on the SQuAD-1.1 dataset (Rajpurkar et al., 2016) and evaluate on SQuAD Adversarial (Jia and Liang, 2017), Universal Triggers on SQuAD (Wallace et al., 2019), and several out-of-domain datasets from MRQA (Fisch et al., 2019). Our framework allows us to incorporate natural constraints on alignment scores to improve zero-shot performance under these distribution shifts, as well as explore coverage-accuracy tradeoffs in these settings. Finally, our model's alignments serve as "explanations" for its prediction, allowing us to ask why certain predictions are made over others and examine scores for hypothetical other answers the model could give.
Figure 2: Example of our question-passage graph. Edges come from SRL, coreference (Super Bowl 50—the game), and postprocessing of predicates nested inside arguments (was—determine). The oracle alignment (Section 3.4) is shown with dotted lines. Blue nodes are predicates and orange ones are arguments.
Context: Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season ... The game was played on February 7, 2016 ...
2 QA as Graph Alignment
Our approach critically relies on the ability to decompose questions and answers into a graph over text spans. Our model can in principle work for a range of syntactic and semantic structures, including dependency parsing, SRL (Palmer et al., 2005), and AMR (Banarescu et al., 2013). We use SRL in this work and augment it with coreference links, due to the high performance and flexibility of current SRL systems (Peters et al., 2018). Throughout this work, we use the BERT-based SRL system from Shi and Lin (2019) and the SpanBERT-based coreference system from Joshi et al. (2020).
An example graph we construct is shown in Figure 2. Both the question and context are represented as graphs where the nodes consist of predicates and arguments. Edges are undirected and connect each predicate and its corresponding arguments. Since SRL only captures the predicate-argument relations within one sentence, we add coreference edges as well: if two arguments are in the same coreference cluster, we add an edge between them. Finally, in certain cases involving verbal or clausal arguments, there might exist nested structures where an argument to one predicate contains a separate predicate-argument structure. In this case, we remove the larger argument and add an edge directly between the two predicates. This is shown by the edge from was to determine (labeled as nested structure) in Figure 2). Breaking down such large arguments helps avoid ambiguity during alignment.
Aligning questions and contexts has proven
useful for question answering in previous work (Sachan et al., 2015; Sachan and Xing, 2016; Khashabi et al., 2018). Our framework differs from theirs in that it incorporates a much stronger alignment model (BERT), allowing us to relax the alignment constraints and build a more flexible, higher-coverage model.
Alignment Constraints Once we have the constructed graph, we can align each node in the question to its counterpart in the context graph. In this work, we control the alignment behavior by placing explicit constraints on this process. We place a locality constraint on the alignment: adjacent pairs of question nodes must align no more than  $k$  nodes apart in the context.  $k = 1$  means we are aligning the question to a connected sub-graph in the context,  $k = \infty$  means we can align to a node anywhere in a connected component in the context graph. In our experiments, we set  $k = 3$ . In the following sections, we will discuss more constraints. Altogether, these constraints define a set  $\mathcal{A}$  of possible alignments.
3 Graph Alignment Model
Let  $\mathbf{T}$  represent the text of the context and question concatenated together. Assume a decomposed question graph  $\mathbf{Q}$  with nodes  $q_{1}, q_{2}, \ldots, q_{m}$  represented by vectors  $\mathbf{q}_{1}, \mathbf{q}_{2}, \ldots, \mathbf{q}_{m}$ , and a decomposed context  $\mathbf{C}$  with nodes  $c_{1}, \ldots, c_{n}$  represented by vectors  $\mathbf{c}_{1}, \ldots, \mathbf{c}_{n}$ . Let  $\mathbf{a} = (a_{1}, \ldots, a_{m})$  be an alignment of question nodes to context nodes, where  $a_{i} \in \{1, \ldots, n\}$  indicates the alignment of the  $i$ th question node. Each question node is aligned to exactly one context node, and multiple question nodes can align to the same context node.
We frame question answering as a maximization of an alignment scoring function over possible alignments:  $\max_{\mathbf{a} \in \mathcal{A}} f(\mathbf{a}, \mathbf{Q}, \mathbf{C}, \mathbf{T})$ . In this paper, we simply choose  $f$  to be the sum over the scores of all alignment pairs  $f(\mathbf{a}, \mathbf{Q}, \mathbf{C}, \mathbf{T}) = \sum_{i=1}^{n} S(q_i, c_{a_i}, \mathbf{T})$ , where  $S(q, c, \mathbf{T})$  denotes the alignment score between a question node  $q$  and a context node  $c$ . This function relies on BERT (Devlin et al., 2019) to compute embeddings of the question and context nodes and will be described more precisely in what follows. We will train this model as a structured support vector machine (SSVM), described in Section 3.2.
[CLS] What day was Super Bowl 50 ... [SEP] Super Bowl 50 was an ...
Figure 3: Alignment scoring. Here the alignment score is computed by the dot product between span representations of question and context nodes. The final alignment score (not shown) is the sum of these edge scores.
Scoring Our alignment scoring process is shown in Figure 3. We first concatenate the question text with the document text into  $\mathbf{T}$  and then encode them using the pre-trained BERT encoder. We then compute a representation for each node in the question and context using a span extractor, which in our case is the self-attentive pooling layer of Lee et al. (2017). The node representation in the question can be computed in the same way. Then the score of a node pair is computed as a dot product  $S(q,c,\mathbf{T}) = \mathbf{q}\cdot \mathbf{c}$ .
Answer Extraction Our model so far produces an alignment between question nodes and context nodes. We assume that one question node contains a wh-word and this node aligns to the context node containing the answer.<sup>1</sup> Ideally, we can use this aligned node to extract the actual answer. However, in practice, the aligned context node may only contain part of the answer and in some cases answering the question only based the aligned context node can be ambiguous. We therefore use the sentence containing the wh-aligned context node as the "new" context and use a standard BERT QA model to extract the actual answer post-hoc. In the experiments, we also show the performance of our model by only use the aligned context node without the sentence, which is only slightly worse.
We train our model as an instance of a structured support vector machine (SSVM). Ignoring the regularization term, this objective can be viewed as a sum over the training data of a structured hinge loss with the following formulation:
$$
\begin{array}{l} \sum_ {i = 1} ^ {N} \max  _ {\mathbf {a} \in \mathcal {A}} (0, \max  _ {f (\mathbf {a}, \mathbf {Q} _ {i}, \mathbf {C} _ {i}, \mathbf {T} _ {i}) + \mathrm {H a m} (\mathbf {a}, \mathbf {a} _ {i} ^ {*})} \\ \left. - f \left(\mathbf {a} _ {i} ^ {*}, \mathbf {Q} _ {i}, \mathbf {C} _ {i}, \mathbf {T} _ {i}\right)\right]\left. \right) \\ \end{array}
$$
<sup>1</sup>We discuss what to do with other questions in Section 4.1.
where  $\mathbf{a}$  denotes the predicted alignment,  $\mathbf{a}_i^*$  is the oracle alignment for the  $i$ th training example, and Ham is the Hamming loss between these two. To get the predicted alignment  $\mathbf{a}$  during training, we need to run loss-augmented inference as we will discuss in the next section. When computing the alignment for node  $j$ , if  $a_j \neq a_j^*$ , we add 1 to the alignment score to account for the loss term in the above equation. Intuitively, this objective requires the score of the gold prediction to be larger than any other hypothesis  $\mathbf{a}$  by a margin of  $\mathrm{Ham}(\mathbf{a}, \mathbf{a}^*)$ .
When training our system, we first do several iterations of local training where we treat each alignment decision as an independent prediction, imposing no constraints, and optimize log loss over this set of independent decisions. The local training helps the global training converge more quickly and achieve better performance.
Since our alignment constraints do not strongly restrict the space of possible alignments (e.g., by enforcing a one-to-one alignment with a connected subgraph), searching over all valid alignments is intractable. We therefore use beam search to find the approximate highest-scoring alignment: (1) Initialize the beam with top  $b$  highest aligned node pairs, where  $b$  is the beam size. (2) For each hypothesis (partial alignment) in the beam, compute a set of reachable nodes based on the currently aligned pairs under the locality constraint. (3) Extend the current hypothesis by adding each of these possible alignments in turn and accumulating its score. Beam search continues until all the nodes in the question are aligned.
An example of one step of beam hypothesis expansion with locality constraint  $k = 2$  is shown in Figure 4. In this state, the two played nodes are already aligned. In any valid alignment, the neighbors of the played question node must be aligned within 2 nodes of the played context node to respect the locality constraint. We therefore only consider aligning to the game, on Feb 7, 2016 and Super Bowl 50. The alignment scores between these reachable nodes and the remaining nodes in the question are computed and used to extend the beam hypotheses.
Note that this inference procedure allows us to easily incorporate other constraints as well. For instance, we could require a "hard" match on entity nodes, meaning that two nodes containing entities
Figure 4: An example of constraints during beam search. The blue node played is already aligned. The orange nodes denote all the valid context nodes that can be aligned to for both Super Bowl 50 and what day in the next step of inference given the locality constraint with  $k = 2$ .
can only align if they share entities. With this constraint, as shown in the figure, Super Bowl 50 can never be aligned to on February 7, 2016. We discuss such constraints more in Section 5.
3.4 Oracle Construction
Training assumes the existence of gold alignments  $\mathbf{a}^*$ , which must be constructed via an oracle given the ground truth answer. This process involves running inference based on heuristically computed alignment scores  $S_{\mathrm{oracle}}$ , where  $S_{\mathrm{oracle}}(q, c)$  is computed by the Jaccard similarity between a question node  $q$  and a context node  $c$ . Instead of initializing the beam with the  $b$  best alignment pairs, we first align the wh-argument in the question with the node(s) containing the answer in the context and then initialize the beam with those alignment pairs.
If the Jaccard similarity between a question node and all other context nodes is zero, we set these as unaligned nodes. During training, our approach can gracefully handle unaligned nodes by treating these as latent variables in structured SVM: the gold "target" is then highest scoring set of alignments consistent with the gold supervision. This involves running a second decoding step on each example to impute the values of these latent variables for the gold alignment.
4 Experiments: Adversarial and Cross-domain Robustness
Our focus in this work is primarily robustness, interpretability, and controllability of our model. We focus on adapting to challenging settings in order to "stress test" our approach.
SQuAD normal	SQuAD addSent	Natural Questions	NewsQA	BioASQ	TBQA
ans in wh	F1	ans in wh	F1	ans in wh	F1	ans in wh	F1	ans in wh	F1	ans in wh	F1
Sub-part Alignment	84.7	84.5	49.5	50.5	65.8	61.5	49.3	48.1	63.5	53.4	35.1	38.4
- global train+inf	85.8	85.2	45.0	46.8	65.9	62.3	48.9	47.1	62.5	52.1	31.9	34.6
- ans from full sent	84.7	81.8	49.5	46.7	65.8	57.8	49.3	45.0	63.5	51.1	35.1	37.5
BERT QA	-	87.8	-	39.2	-	59.4	-	48.5	-	52.4	-	25.3
Table 1: The performance and ablations of our proposed model on the development sets of SQuAD, adversarial SQuAD, and four out-of-domain datasets. Our Sub-part Alignment model uses both global training and inference as discussed in Section 3.2-3.3. - global train+inf denotes the locally trained and evaluated model. - ans from full sent denotes extracting the answer using only the wh-aligned node. ans in wh denotes the percentage of answers found in the span aligned to the wh-span, and F1 denotes the standard QA performance measure. Here for addSent, we only consider the adversarial examples. Note also that this evaluation is only on wh-questions.
For all experiments, we train our model only on the English SQuAD-1.1 dataset (Rajpurkar et al., 2016) and examine how well it can generalize to adversarial and out-of-domain settings with minimal modification, using no fine-tuning on new data and no data augmentation that would capture useful transformations. We evaluate on the addSent and addOneSent proposed by Jia and Liang (2017), and the Universal Triggers on SQuAD (Wallace et al., 2019). We also test the performance of our SQuAD-trained models in zero-shot adaptation to new English domains, namely Natural Questions (Kwiatkowski et al., 2019), NewsQA (Trischler et al., 2017), BioASQ (Tsatsaronis et al., 2015) and TextbookQA (Kembhavi et al., 2017), taken from the MRQA shared task (Fisch et al., 2019). Our motivation here was to focus on text from a variety of domains where transferred SQuAD models may at least behave credibly. We excluded, for example, HotpotQA (Yang et al., 2018) and DROP (Dua et al., 2019), since these are so far out-of-domain from the perspective of SQuAD that we do not see them as a realistic cross-domain target.
We compare primarily against a standard BERT QA system (Devlin et al., 2019). We also investigate a local version of our model, where we only try to align each node in the question to its oracle, without any global training  $(- \text{global train} + \inf)$ , which can still perform reasonably because BERT embeds the whole question and context. When comparing variants of our proposed model, we only consider the questions that have a valid SRL parse and have a wh word (results in Table 1, Table 2, and Figure 5). When comparing with prior systems, for questions that do not have a valid SRL parse or
wh word, we back off to the standard BERT QA system (results in Table 3).
We set the beam size  $b = 20$  for the constrained alignment. We use BERT-base-uncased for all of our experiments, and fine-tune the model using Adam (Kingma and Ba, 2014) with learning rate set to 2e-5. Our preprocessing uses a SpanBERT-based coreference system (Joshi et al., 2020) and a BERT-based SRL system (Shi and Lin, 2019). We limit the length of the context to 512 tokens. For our global model, we initialize the weights using a locally trained model and then fine-tune using the SSVM loss. We find the initialization helps the model converge much faster and it achieves better performance than learning from scratch. When doing inference, we set the locality constraint  $k = 3$ .
4.2 Results on Challenging Settings
The results<sup>2</sup> on the normal SQuAD development set and other challenging sets are shown in Table 1.
Our model is not as good as BERT QA on normal SQuAD but outperforms it in challenging settings. Compared to the BERT QA model, our model is fitting a different data distribution (learning a constrained structure) which makes the task harder. This kind of training scheme does cause some performance drop on normal SQuAD, but we can see that it consistently improves the F1 on the adversarial (on SQuAD addSent, a 11.3 F1 improvement over BERT QA) and cross-domain datasets except NewsQA (where it is 0.4 F1 worse). This demonstrates that learning the alignment helps improve the robustness of our model.
Here we omit SQuAD addOneSent for simplicity, since the performance on it has the same trend as SQuAD addSent. Refer to the Appendix for the results on SQuAD addOneSent.
Type	Sub-part Alignment	BERT
Normal	Trigger Δ	Normal	Trigger Δ
who	84.7	82.7	2.0	87.1	78.5	8.6
why	75.1	71.3	3.8	76.5	59.7	16.8
when	88.4	82.8	5.6	90.3	80.9	9.4
where	83.6	81.4	2.2	84.1	75.8	8.3
Table 2: The performance of our model on the Universal Triggers on SQuAD dataset (Wallace et al., 2019). Compared with BERT, our model sees smaller performance drops on all triggers.
Global training and inference improve performance in adversarial settings, despite having no effect in-domain. Normal SQuAD is a relatively easy dataset and the answer for most questions can be found by simple lexical matching between the question and context. From the ablation of global train+inf, we can see that more than  $80\%$  of answers can be located by matching the wh-argument. We also observe a similar pattern on Natural Questions.<sup>3</sup> However, as there are very strong distractors in SQuAD addSent, the wh-argument matching is unreliable. In such situations, the constraints imposed by other argument alignments in the question are useful to correct the wrong wh-alignment through global inference. We see that the global training plus inference is consistently better than the local version on all other datasets.
Using the strict wh answer extraction still gives strong performance From the ablation of - ans from full sent, we observe that our "strictest" system that extracts the answer only using the whaligned node is only worse by 3-4 points of F1 on most datasets. Using the full sentence gives the system more context and maximal flexibility, and allows it to go beyond the argument spans introduced by SRL. We believe that better semantic representations tailored for question answering (Lamm et al., 2020) will help further improvement in this regard.
4.3 Results on Universal Triggers
The results on subsets of the universal triggers dataset are shown in Table 2. We see that every trigger results in a bigger performance drop on BERT QA than our model. Our model is much more stable, especially on who and where question
types, in which case the performance only drops by around  $2\%$ . Several factors may contribute to the stability: (1) The triggers are ungrammatical and their arguments often contain seemingly random words, which are likely to get lower alignment scores. (2) Because our model is structured and trained to align all parts of the question, adversarial attacks on span-based question answering models may not fool our model as effectively as they do BERT.