SlowGuess/ACL_2020_OCR huggingface dataset:
Sarthak Jain
jain.sar@northeastern.edu
Yuval Pinter
uvp@gatech.edu
Sarah Wiegrefe
saw@gatech.edu
b.wallace@northeastern.edu
In many settings it is important for one to be able to understand why a model made a particular prediction. In NLP this often entails extracting snippets of an input text 'responsible for' corresponding model output; when such a snippet comprises tokens that indeed informed the model's prediction, it is a faithful explanation. In some settings, faithfulness may be critical to ensure transparency. Lei et al. (2016) proposed a model to produce faithful rationales for neural text classification by defining independent snippet extraction and prediction modules. However, the discrete selection over input tokens performed by this method complicates training, leading to high variance and requiring careful hyperparameter tuning. We propose a simpler variant of this approach that provides faithful explanations by construction. In our scheme, named FRESH, arbitrary feature importance scores (e.g., gradients from a trained model) are used to induce binary labels over token inputs, which an extractor can be trained to predict. An independent classifier module is then trained exclusively on snippets provided by the extractor; these snippets thus constitute faithful explanations, even if the classifier is arbitrarily complex. In both automatic and manual evaluations we find that variants of this simple framework yield predictive performance superior to 'end-to-end' approaches, while being more general and easier to train.<sup>1</sup>
Neural models dominate NLP these days, but it remains difficult to know why such models make specific predictions for sequential text inputs. This problem has been exacerbated by the adoption of deep contextualized word representations, whose architectures permit arbitrary and interdependent
interactions between all inputs, making it particularly difficult to know which inputs contributed to any specific prediction.
Concretely, in a bidirectional RNN or Transformer model, the contextual embedding for a word at position  $j$  in instance  $x$  may encode information from any or all of the tokens at positions 1 to  $j - 1$  and  $j + 1$  to  $|x|$ . Consequently, continuous scores such as attention weights (Bahdanau et al., 2015) induced over these contextualized embeddings reflect the importance not of individual inputs, but rather of unknown interactions between all input tokens. This makes it misleading to present heatmaps of these scores over the original token inputs as an explanation for a prediction (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019; Serrano and Smith, 2019).
The key missing property here is faithfulness (Lipton, 2018): An explanation provided by a model is faithful if it reflects the information actually used by said model to come to a disposition. In some settings the ability of a model to provide faithful explanations may be paramount. For example, without faithful explanations, we cannot know whether a model is exploiting sensitive features such as gender (Pruthi et al., 2020).
We propose an approach to neural text classification that provides faithful explanations for predictions by construction. Following prior work in this direction (Lei et al., 2016), we decompose our model into independent extraction and prediction modules, such that the latter uses only inputs selected by the former. This discrete selection over inputs allows one to use an arbitrarily complex prediction network while still being able to guarantee that it uses only the extracted input features to inform its output.
The main drawback to this rationalization approach has been the difficulty of training the two components jointly under only instance-level
$^{1}$ Code is available at https://github.com/successar/FRESH
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4459-4473 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Query: What is the only difference between a reflection in a mirror and the actual image? | Answer: It is exactly the same | Label: False [Human] You have seen your own reflection in a mirror . The person looking back at you looks just like you . Where does that reflected person appear to be standing ? Yes , they appear to be on the other side of the mirror . That is really strange to think about , but very cool . Have you ever waved at your reflection in a mirror ? The reflected image will wave back at you . Here is something to try next time you stand in front of a mirror . Wave to your reflection with your right hand . What hand do you think the reflection will wave back with ? The same hand ? A different hand ? You will notice something interesting . The reflection waves back with the hand on the same side as you , but it is their left hand . The image in a reflection is reversed . This is just like the image of the sign above . Light rays strike flat shiny surfaces and are reflected . The reflections are reversed .
[Lei et al.] You have seen your own reflection in a mirror . The person looking back at you looks just like you . Where does that reflected person appear to be standing ? Yes , they appear to be on the other side of the mirror . That is really strange to think about , but very cool . Have you ever waved at your reflection in a mirror ? The reflected image will wave back at you . Here is something to try next time you stand in front of a mirror . Wave to your reflection with your right hand . What hand do you think the reflection will wave back with ? The same hand ? A different hand ? You will notice something interesting . The reflection waves back with the hand on the same side as you , but it is their left hand . The image in a reflection is reversed . This is just like the image of the sign above . Light rays strike flat shiny surfaces and are reflected . The reflections are reversed .
[FRESH] You have seen your own reflection in a mirror . The person looking back at you looks just like you . Where does that reflected person appear to be standing ? Yes , they appear to be on the other side of the mirror . That is really strange to think about , but very cool . Have you ever waved at your reflection in a mirror ? The reflected image will wave back at you . Here is something to try next time you stand in front of a mirror . Wave to your reflection with your right hand . What hand do you think the reflection will wave back with ? The same hand ? A different hand ? You will notice something interesting . The reflection waves back with the hand on the same side as you , but it is their left hand . The image in a reflection is reversed . This is just like the image of the sign above . Light rays strike flat shiny surfaces and are reflected . The reflections are reversed .
Figure 1: Contiguous rationales extracted using Lei et al. (2016) and FRESH models for an example from the MultiRC dataset. We also show the reference rationale associated with this example (top).
supervision (i.e., without token labels). This has necessitated training the extraction module via reinforcement learning — namely REINFORCE (Williams, 1992) — which exhibits high variance and is particularly sensitive to choice of hyperparameters. Recent work (Bastings et al., 2019) has proposed a differentiable mechanism to perform binary token selection, but this relies on the reparameterization trick, which similarly complicates training. Methods using the reparameterization trick tend to zero out token embeddings, which may adversely affect training in transformer-based models, especially when one is not fine-tuning lower layers of the model due to resource constraints, as in our experiments.
To avoid the complexity inherent to training under a remote supervision signal, we introduce Faithful Rationale Extraction from Saliency Thresholding (FRESH), which disconnects the training regimes of the extractor and predictor networks, allowing each to be trained separately. We still assume only instance-level supervision; the trick is to define a method of selecting snippets from inputs — rationales (Zaidan et al., 2007) — that can be used to support prediction. Here we propose using arbitrary feature importance scoring techniques to do so. Notably, these need not satisfy the 'faithfulness' criterion.
In this paper we evaluate variants of FRESH that use attention (Bahdanau et al., 2015) and gradient methods (Li et al., 2016; Simonyan et al., 2014) as illustrative feature scoring mechanisms. These provide continuous scores for features; we derive discrete rationales from them using simple heuristics. An independent network then uses only the extracted rationales to make predictions.
Disconnecting the training tie between the in
dependent rationale extractor and prediction modules means that FRESH is faithful by construction: The snippet that is ultimately used to inform a prediction can be presented as a faithful explanation because this was the only text available to the predictor. In contrast to prior discrete rationalization methods, FRESH greatly simplifies training, and can accommodate any feature importance scoring metric. In our experiments, we also find that it yields superior predictive performance.
In addition to being faithful (and affording strong predictive performance), extracted rationales would ideally be intuitive to humans, i.e., plausible. To evaluate this we run a small user study (section 8) in which humans both evaluate the readability of extracted rationales and attempt to classify instances based on them, effectively serving as a prediction module in the FRESH framework. An example illustrating this property is presented in Figure 1.
Types of explainability. Lipton (2018); Doshi-Velez and Kim (2017) and Rudin (2019) provide overviews on definitions and characterizations of interpretability. Lertvittayakumjorn and Toni (2019) classify three possible uses of text explanations: (i) revealing model behavior, (ii) justifying model predictions, and (iii) helping humans investigate uncertain predictions. Attempting to guarantee the faithfulness of a feature selection or explanation generation method is a more challenging question than finding explanations which humans find acceptable (Rudin, 2019). But the benefits of developing such methods is profound: Faithful explanations provide a means to reveal a
Figure 2: A schematic of the FRESH approach. (1) The first model, supp, is trained end-to-end for prediction but used only to 'importance score' features. These scores can be derived via any method, e.g., gradients or attention, and are not required to faithfully explain model outputs. Scores are heuristically discretized into binary labels. (2) An extraction module ext may be a parameterized sequence tagging model trained on the pseudo-targets derived in (1), or heuristics over importance scores directly, creating a new dataset  $\langle \tilde{x}, y \rangle$  comprising pairs of extracted rationales only. (3) This new dataset is used to train a final classifier, pred, which only ever sees rationales.
model's underlying decision-making process.
Issues with current explainability methods in NLP. A recent line of work in NLP has begun to critically examine the use of certain methods for constructing 'heatmaps' over input tokens to explain predictions. In particular, existing feature attribution methods may not provide robust, faithful explanations (Feng et al., 2018; Jain and Wallace, 2019; Wegreffe and Pinter, 2019; Serrano and Smith, 2019; Brunner et al., 2020; Zhong et al., 2019; Pruthi et al., 2020).
Wiegrefe and Pinter (2019) argue for classifying model interpretability into two groups: faithfulness and plausibility. Lei et al. (2016) note that a desirable set of criteria for rationales is that they are sufficient, short, and coherent. Yu et al. (2019) extend these criteria by additionally arguing for comprehensiveness, which dictates that a rationale should contain all relevant and useful information.
Prior efforts (Lei et al., 2016; Yu et al., 2019; Bastings et al., 2019) have proposed methods that produce faithful explanations via a two-model setup, defining a generator network that imposes hard attention over inputs and then passes these to a second model for prediction. Yu et al. (2019) extend this by adding a third adversarial model into the framework. These models are trained jointly, which is difficult because hard attention is discrete and necessitates recourse to reinforcement learning, i.e., REINFORCE (Williams, 1992), or the reparameterization trick (Bastings et al., 2019).
Human evaluations. Kim et al. 2016 states: "a method is interpretable if a user can correctly pre
dict the method's result"; they conducted user studies to test this. In a similar plausibility vein, others have proposed testing whether humans like rationales (Ehsan et al., 2018, 2019). We follow these efforts by eliciting human judgments on rationales, although we view plausibility as a secondary aim here.
3 Faithfulness through Discrete Rationale Selection
We now propose FRESH, our framework for training explainable neural predictors. We begin by describing the two-model, discrete rationale selection approach introduced by Lei et al. (2016) (§3.1), which serves as the starting point for our framework, detailed in §4.
3.1 End-to-End Rationale Extraction
Consider a standard text classification setup in which we have  $n$  input documents  $X = \{x_{1},\dots,x_{n}\}$ ,  $x_{i} \in V^{l_{i}}$ , where  $l_{i}$  denotes the number of tokens in document  $x_{i}$ , and  $V$  the vocabulary, and their assigned labels  $y = \{y_{1},\dots,y_{n}\}$ ,  $y_{i} \in \mathcal{V}$ . Lei et al. propose a model comprising a generator (gen) and an encoder (enc). gen is tasked with extracting rationales from inputs  $x_{i}$ , formalized as a binary mask over tokens sampled from a Bernoulli distribution:  $z_{i} \sim \mathrm{gen}(x_{i}) \in \{0,1\}^{l_{i}}$ . enc makes predictions  $\hat{y} = \mathrm{enc}(x_{i},z_{i})$  on the basis of the unmasked tokens.
The objective function is defined so that the overall expected loss  $\mathcal{L}$  is minimized over both
modules:
$$
\underset {\theta_ {\mathrm {e n c}}, \theta_ {\mathrm {g e n}}} {\text {m i n i m i z e}} \sum_ {i = 1} ^ {n} \mathbb {E} _ {z _ {i} \sim \operatorname {g e n} (x _ {i})} \mathcal {L} \left(\operatorname {e n c} \left(x _ {i}, z _ {i}\right), y _ {i}\right). \tag {1}
$$
This objective (1) is difficult to optimize as it requires marginalizing over all possible rationales  $z$ . Parameter estimation is therefore performed via an approximation approach that entails drawing samples from  $\mathrm{gen}(x)$  and averaging their associated gradients during the learning process. Lei et al. (2016) found that this REINFORCE-style estimation works well for rationale extraction, but may have high variance as a result of the large state space of possible rationales under consideration, which is difficult to efficiently explore.
The loss function  $\mathcal{L}$  used by Lei et al. (2016) is a squared  $\ell_2$  loss between the prediction  $\mathrm{enc}(x,z)$  and the reference label  $y$ , with added regularization terms placed on the binary mask  $z$  to encourage rationale conciseness and contiguity.
We modify the conciseness term so that the model is not penalized as long as a predefined desired rationale length  $d$  has not been passed:
$$
\Omega (z) = \lambda_ {1} \underbrace {\max  \left(0 , \frac {| z |}{L} - d\right)} _ {\text {c o n c i s e n e s s}} + \lambda_ {2} \underbrace {\sum_ {t} \frac {\left| z _ {t} - z _ {t - 1} \right|}{L - 1}} _ {\text {c o n t i g u i t y}}. \tag {2}
$$
4 Faithful Rationale Extraction from Saliency tThresholding (FRESH)
To avoid recourse to REINFORCE, we introduce FRESH, in which we decompose the original prediction task into three sub-components, each with its own independent model. These are the support model supp, the rationale extractor model ext, and the classifier pred.2
We train supp end-to-end to predict  $y$ , using its outputs only to extract continuous feature importance scores from instances in  $X$ . These scores are binarized by ext either using a parameterized model trained on the output scores, or via direct discretization heuristics. Finally, pred is trained (and tested) only on text provided by ext. Figure 2 depicts this proposed framework.
A central advantage of our decomposed setup lies in the arbitrariness of the rationale extraction
mechanism. Any function over supp's predictions that assigns scores to the input tokens intended to quantify their importance can serve as an input to ext. Note that this means even post-hoc scoring models (applied after the model has completed training) are permissible. Examples of such functions include gradient-based methods and LIME (Ribeiro et al., 2016).
Notably, the importance scoring function need not faithfully identify features that actually informed the predictions from supp. This means, e.g., that one is free to use token-level attention (over contextualized representations) — the final rationales provided by FRESH will nonetheless remain faithful with respect to pred. The importance scores are used only to train ext heuristically, for example by treating the top  $k$  tokens (with respect to importance scores) for a given example as the target rationale. The key design decision here is designing such heuristics that map continuous importance scores to discrete rationales. Any strategy for this will likely involve trading conciseness (shorter rationales) against performance (greater predictive accuracy).
For explainability, we can present users with the snippet(s) that pred used to make a prediction as an explanation (from ext), and we can be certain that the only tokens that contributed to the prediction made by pred are those included in the this text. In addition to transparency, this framework may afford efficiency gains in settings in which humans are tasked with classifying documents; in this case we can use ext to present only the (short) relevant snippets. Indeed, we use exactly this approach as one means of evaluation in Section 8.
5 FRESH Implementations
The high-level framework described above requires making several design choices to operationalize; we propose and evaluate a set of such choices in this work, detailed below. Specifically, we must specify a feature importance scoring mechanism for supp (Section 5.1), and a strategy for inducing discrete targets from these continuous scores (5.2). In addition, we need to specify a trained or heuristic extractor architecture ext. In this work, all instances of pred exploit BERT-based representations. $^3$
For fair comparison, we have modified all baselines (Lei et al., 2016; Bastings et al., 2019) to similarly capitalize on BERT-based representations.
2This is the most general framing, but in fact supp and ext may be combined by effectively defining ext as an application of heuristics to extract snippets on the basis of scores provided by supp; any means of procuring 'importance' scores for the features comprising instances and converting these to extracted snippets to pass to pred will suffice.
5.1 Feature Scoring Methods
All models considered in this work are based on Bidirectional Encoder Representations from Transformer (BERT) encoders (Devlin et al., 2019) and its variants, namely RoBERTa (Liu et al., 2019) and SciBERT (Beltagy et al., 2019); see Appendix B for more details. For sake of brevity, we simply refer to all of these as BERT from here on. We define supp as a BERT encoder that consumes either a single input (in the case of standard classification) or two inputs (e.g., in the case of question answering tasks) separated by the standard [SEP] token.
While we emphasize that the proposed framework can accommodate arbitrary input feature scoring mechanisms, we consider only a few obvious variants here, leaving additional exploration for future work. Specifically, we evaluate attention scores (Bahdanau et al., 2015) and input gradients (Li et al., 2016; Simonyan et al., 2014).
Attention scores are taken as the self-attention weights induced from the [CLS] token index to all other indices in the penultimate layer of supp; this excludes weights associated with any special tokens added. BERT uses wordpiece tokenization; to compute a score for a token, we sum the self-attention weights assigned to its constituent pieces. BERT is also multi-headed, and so we average scores over heads to derive a final score.
5.2 Discretizing Soft Scores
A necessary step in our framework consists of mapping from the continuous feature scores provided by supp to discrete labels, or equivalently, mapping scores to rationales which will either be consumed directly by pred or be used to train a sequence tagging model ext. We consider a few heuristic strategies for performing this mapping.
Contiguous. Select the span of length  $k$  that corresponds to the highest total score (over all spans of length  $k$ ). We call these rationales contiguous.
Top-  $k$ . Extract as a rationale the top- $k$  tokens (with respect to importance scores) from a document, irrespective of contiguity (each word is treated independently). We refer to these rationales as non-contiguous.
These strategies may be executed per-instance or globally (across an entire dataset), reflecting the flexibility of FRESH. Empirically, per-instance and global approaches performed about the same;
Doc. Len.	Rationale Len.	N
SST	17	-	9,613
AGNews	30	-	127,600
Ev. Inf.	349	10%	7,193
Movies	728	31%	1,999
MultiRC	297	18%	32,091
Table 1: Dataset details, with rationale length ratios included for datasets where they are available.
we report results for the simpler, per-instance approaches (additional results in Appendix E).
5.3 Extractor model
We experiment with two variants of ext. The first is simply direct use of the importance scores provided by supp and discretization heuristics over these; this does not require training an explicit ext model. We also consider a parameterized extractor model that independently makes token-wise predictions from BERT representations. Using an explicit extraction model allows us to mix in direct supervision on rationales alongside the pseudotargets derived heuristically from supp.
Tying the sequential token predictions made by ext via a Conditional Random Field (CRF) layer (Lafferty et al., 2001) may further improve performance, but we leave this for future work.
We use five English text classification datasets spanning a range of domains (see Table 1).
Stanford Sentiment Treebank (SST) (Socher et al., 2013). Sentences labeled with binary sentiment (neutral sentences have been removed).
AgNews (Del Corso et al., 2005). News articles to be categorized topically into Science, Sports, Business, and World.
Evidence Inference (Lehman et al., 2019). Biomedical articles describing randomized controlled trials. The task is to infer the reported relationship between a given intervention and comparator with respect to an outcome, and to identify a snippet within the text that supports this. The original dataset comprises lengthy full-text articles; we use an abstract-only subset of this data.
Movies (Zaidan and Eisner, 2008). Movie reviews labeled for sentiment accompanied by rationales on dev and test sets (DeYoung et al., 2020).
4463
Saliency	Rationale	SST (20%)	AGNews (20%)	Ev. Inf. (10%)	Movies (30%)	MultiRC (20%)
Full text	-	.90 (.89-.90)	.94 (.94-.94)	.73 (.73-.78)	.95 (.93-.97)	.68 (.68-.69)
Lei et al.	contiguous	.71 (.49-.83)	.87 (.85-.89)	.53 (.45-.56)	.83 (.80-.92)	.62 (.62-.64)
top k	.74 (.47-.84)	.92 (.90-.92)	.47 (.38-.53)	.87 (.80-.91)	.64 (.61-.65)
Bastings et al.	contiguous	.60 (.58-.62)	.77 (.18-.78)	.45 (.40-.49)	—	.41 (.30-.50)
top k	.59 (.58-.61)	.72 (.19-.80)	.50 (.38-.60)	—	.44 (.30-.55)
Gradient	contiguous	.70 (.69-.72)	.85 (.84-.85)	.67 (.62-.68)	.94 (.92-.95)	.67 (.66-.67)
top k	.68 (.67-.70)	.86 (.85-.86)	.62 (.61-.64)	.93 (.92-.94)	.66 (.65-.67)
[CLS] Attn	contiguous	.81 (.80-.82)	.88 (.88-.89)	.68 (.59-.73)	.93 (.90-.94)	.63 (.60-.62)
top k	.81 (.80-.82)	.91 (.90-.91)	.66 (.64-.70)	.94 (.93-.95)	.63 (.62-.64)