SlowGuess/ACL_2020_OCR huggingface dataset:
As defined, the above measures have assumed discrete rationales  $r_i$ . We would also like to evaluate the faithfulness of continuous importance scores assigned to tokens by models. Here we adopt a simple approach for this. We convert soft scores over features  $s_i$  provided by a model into discrete rationales  $r_i$  by taking the top-  $k_d$  values, where  $k_d$  is a threshold for dataset  $d$ . We set  $k_d$  to the average rationale length provided by humans for dataset  $d$  (see Table 4). Intuitively, this says: How much does the model prediction change if we remove a number of tokens equal to what humans use (on average for this dataset) in order of the importance scores assigned to these by the model. Once we have discretized the soft scores into rationales in this way, we compute the faithfulness scores as per Equations 1 and 2.
This approach is conceptually simple. It is also computationally cheap to evaluate, in contrast to measures that require per-token measurements, e.g., importance score correlations with 'leave-one-out' scores (Jain and Wallace, 2019), or counting how many 'important' tokens need to be erased before
a prediction flips (Serrano and Smith, 2019). However, the necessity of discretizing continuous scores forces us to pick a particular threshold  $k$ .
We can also consider the behavior of these measures as a function of  $k$ , inspired by the measurements proposed in Samek et al. (2016) in the context of evaluating saliency maps for image classification. They suggested ranking pixel regions by importance and then measuring the change in output as they are removed in rank order. Our datasets comprise documents and rationales with quite different lengths; to make this measure comparable across datasets, we construct bins designating the number of tokens to be deleted. Denoting the tokens up to and including bin  $k$  for instance  $i$  by  $r_{ik}$ , we define an aggregate comprehensiveness measure:
$$
\frac {1}{| \mathcal {B} | + 1} \left(\sum_ {k = 0} ^ {| \mathcal {B} |} m \left(x _ {i}\right) _ {j} - m \left(x _ {i} \backslash r _ {i k}\right) _ {j}\right) \tag {3}
$$
This is defined for sufficiency analogously. Here we group tokens into  $k = 5$  bins by grouping them into the top  $1\%$ ,  $5\%$ ,  $10\%$ ,  $20\%$  and  $50\%$  of tokens, with respect to the corresponding importance score. We refer to these metrics as "Area Over the Perturbation Curve" (AOPC).<sup>7</sup>
These AOPC sufficiency and comprehensiveness measures score a particular token ordering under a model. As a point of reference, we also report these when random scores are assigned to tokens.
Our AOPC metrics are similar in concept to ROAR (Hooker et al., 2019) except that we re-use an existing model as opposed to retraining for each fraction.
Our focus in this work is primarily on the ERASER benchmark itself, rather than on any particular model(s). But to establish a starting point for future work, we evaluate several baseline models across the corpora in ERASER. We broadly classify these into models that assign 'soft' (continuous) scores to tokens, and those that perform a 'hard' (discrete) selection over inputs. We additionally consider models specifically designed to select individual tokens (and very short sequences) as rationales, as compared to longer snippets. All of our implementations are in PyTorch (Paszke et al., 2019) and are available in the ERASER repository.
All datasets in ERASER comprise inputs, rationales, and labels. But they differ considerably in document and rationale lengths (Table A). This motivated use of different models for datasets, appropriate to their sizes and rationale granularities. We hope that this benchmark motivates design of models that provide rationales that can flexibly adapt to varying input lengths and expected rationale granularities. Indeed, only with such models can we perform comparisons across all datasets.
5.1 Hard selection
Models that perform hard selection may be viewed as comprising two independent modules: an encoder which is responsible for extracting snippets of inputs, and a decoder that makes a prediction based only on the text provided by the encoder. We consider two variants of such models.
Lei et al. (2016). In this model, an encoder induces a binary mask over inputs  $x$ ,  $z$ . The decoder accepts the tokens in  $x$  unmasked by  $z$  to make a prediction  $\hat{y}$ . These modules are trained jointly via REINFORCE (Williams, 1992) style estimation, minimizing the loss over expected binary vectors  $z$  yielded from the encoder. One of the advantages of this approach is that it need not have access to marked rationales; it can learn to rationalize on the basis of instance labels alone. However, given that we do have rationales in the training data, we experiment with a variant in which we train the encoder explicitly using rationale-level annotations.
In our implementation of Lei et al. (2016), we drop in two independent BERT (Devlin et al., 2019) or GloVe (Pennington et al., 2014) base modules
with bidirectional LSTMs (Hochreiter and Schmidhuber, 1997) on top to induce contextualized representations of tokens for the encoder and decoder, respectively. The encoder generates a scalar (denoting the probability of selecting that token) for each LSTM hidden state using a feedforward layer and sigmoid. In the variant using human rationales during training, we minimize cross entropy loss over rationale predictions. The final loss is then a composite of classification loss, regularizers on rationales (Lei et al., 2016), and loss over rationale predictions, when available.
Pipeline models. These are simple models in which we first train the encoder to extract rationales, and then train the decoder to perform prediction using only rationales. No parameters are shared between the two models.
Here we first consider a simple pipeline that first segments inputs into sentences. It passes these, one at a time, through a Gated Recurrent Unit (GRU) (Cho et al., 2014), to yield hidden representations that we compose via an attentive decoding layer (Bahdanau et al., 2015). This aggregate representation is then passed to a classification module which predicts whether the corresponding sentence is a rationale (or not). A second model, using effectively the same architecture but parameterized independently, consumes the outputs (rationales) from the first to make predictions. This simple model is described at length in prior work (Lehman et al., 2019). We further consider a 'BERT-to-BERT' pipeline, where we replace each stage with a BERT module for prediction (Devlin et al., 2019).
In pipeline models, we train each stage independently. The rationale identification stage is trained using approximate sentence boundaries from our source annotations, with randomly sampled negative examples at each epoch. The classification stage uses the same positive rationales as the identification stage, a type of teacher forcing (Williams and Zipser, 1989) (details in Appendix C).
5.2 Soft selection
We consider a model that passes tokens through BERT (Devlin et al., 2019) to induce contextualized representations that are then passed to a bidirectional LSTM (Hochreiter and Schmidhuber, 1997). The hidden representations from the LSTM are collapsed into a single vector using additive attention (Bahdanau et al., 2015). The LSTM layer allows us to bypass the 512 word limit imposed by
This is not intended to be comprehensive.
<sup>9</sup>https://github.com/jayded/
eraserbenchmark
Perf.	IOU F1	Token F1
Evidence Inference
Lei et al. (2016)	0.461	0.000	0.000
Lei et al. (2016) (u)	0.461	0.000	0.000
Lehman et al. (2019)	0.471	0.119	0.123
Bert-To-Bert	0.708	0.455	0.468
BoolQ
Lei et al. (2016)	0.381	0.000	0.000
Lei et al. (2016) (u)	0.380	0.000	0.000
Lehman et al. (2019)	0.411	0.050	0.127
Bert-To-Bert	0.544	0.052	0.134
Movie Reviews
Lei et al. (2016)	0.914	0.124	0.285
Lei et al. (2016) (u)	0.920	0.012	0.322
Lehman et al. (2019)	0.750	0.063	0.139
Bert-To-Bert	0.860	0.075	0.145
FEVER
Lei et al. (2016)	0.719	0.218	0.234
Lei et al. (2016) (u)	0.718	0.000	0.000
Lehman et al. (2019)	0.691	0.540	0.523
Bert-To-Bert	0.877	0.835	0.812
MultiRC
Lei et al. (2016)	0.655	0.271	0.456
Lei et al. (2016) (u)	0.648	0.000†	0.000†
Lehman et al. (2019)	0.614	0.136	0.140
Bert-To-Bert	0.633	0.416	0.412
CoS-E
Lei et al. (2016)	0.477	0.255	0.331
Lei et al. (2016) (u)	0.476	0.000†	0.000†
Bert-To-Bert	0.344	0.389	0.519
e-SNLI
Lei et al. (2016)	0.917	0.693	0.692
Lei et al. (2016) (u)	0.903	0.261	0.379
Bert-To-Bert	0.733	0.704	0.701
BERT; when we exceed this, we effectively start encoding a 'new' sequence (setting the positional index to 0) via BERT. The hope is that the LSTM learns to compensate for this. Evidence Inference and BoolQ comprise very long (>1000 token) inputs; we were unable to run BERT over these. We instead resorted to swapping GloVe 300d embeddings (Pennington et al., 2014) in place of BERT representations for tokens. spans.
To soft score features we consider: Simple gradients, attention induced over contextualized representations, and LIME (Ribeiro et al., 2016).
Table 3: Performance of models that perform hard rationale selection. All models are supervised at the rationale level except for those marked with (u), which learn only from instance-level supervision;  $\dagger$  denotes cases in which rationale training degenerated due to the REINFORCE style training. Perf. is accuracy (CoS-E) or macro-averaged F1 (others). Bert-To-Bert for CoS-E and e-SNLI uses a token classification objective. Bert-To-Bert CoS-E uses the highest scoring answer.
Perf.	AUPRC	Comp. ↑	Suff. ↓
Evidence Inference
GloVe + LSTM - Attention	0.429	0.506	-0.002	-0.023
GloVe + LSTM - Gradient	0.429	0.016	0.046	-0.138
GloVe + LSTM - Lime	0.429	0.014	0.006	-0.128
GloVe + LSTM - Random	0.429	0.014	-0.001	-0.026
BoolQ
GloVe + LSTM - Attention	0.471	0.525	0.010	0.022
GloVe + LSTM - Gradient	0.471	0.072	0.024	0.031
GloVe + LSTM - Lime	0.471	0.073	0.028	-0.154
GloVe + LSTM - Random	0.471	0.074	0.000	0.005
Movies
BERT+LSTM - Attention	0.970	0.417	0.129	0.097
BERT+LSTM - Gradient	0.970	0.385	0.142	0.112
BERT+LSTM - Lime	0.970	0.280	0.187	0.093
BERT+LSTM - Random	0.970	0.259	0.058	0.330
FEVER
BERT+LSTM - Attention	0.870	0.235	0.037	0.122
BERT+LSTM - Gradient	0.870	0.232	0.059	0.136
BERT+LSTM - Lime	0.870	0.291	0.212	0.014
BERT+LSTM - Random	0.870	0.244	0.034	0.122
MultiRC
BERT+LSTM - Attention	0.655	0.244	0.036	0.052
BERT+LSTM - Gradient	0.655	0.224	0.077	0.064
BERT+LSTM - Lime	0.655	0.208	0.213	-0.079
BERT+LSTM - Random	0.655	0.186	0.029	0.081
CoS-E
BERT+LSTM - Attention	0.487	0.606	0.080	0.217
BERT+LSTM - Gradient	0.487	0.585	0.124	0.226
BERT+LSTM - Lime	0.487	0.544	0.223	0.143
BERT+LSTM - Random	0.487	0.594	0.072	0.224
e-SNLI
BERT+LSTM - Attention	0.960	0.395	0.105	0.583
BERT+LSTM - Gradient	0.960	0.416	0.180	0.472
BERT+LSTM - Lime	0.960	0.513	0.437	0.389
BERT+LSTM - Random	0.960	0.357	0.081	0.487
Table 4: Metrics for 'soft' scoring models. Perf. is accuracy (CoS-E) or F1 (others). Comprehensiveness and sufficiency are in terms of AOPC (Eq. 3). 'Random' assigns random scores to tokens to induce orderings; these are averages over 10 runs.
Here we present initial results for the baseline models discussed in Section 5, with respect to the metrics proposed in Section 4. We present results in two parts, reflecting the two classes of rationales discussed above: 'Hard' approaches that perform discrete selection of snippets, and 'soft' methods that assign continuous importance scores to tokens.
In Table 3 we evaluate models that perform discrete selection of rationales. We view these as inherently faithful, because by construction we know which snippets the decoder used to make a prediction.[10] Therefore, for these methods we report only metrics that measure agreement with human annotations.
This assumes independent encoders and decoders.
Due to computational constraints, we were unable to run our BERT-based implementation of Lei et al. (2016) over larger corpora. Conversely, the simple pipeline of Lehman et al. (2019) assumes a setting in which rationale are sentences, and so is not appropriate for datasets in which rationales tend to comprise only very short spans. Again, in our view this highlights the need for models that can rationalize at varying levels of granularity, depending on what is appropriate.
We observe that for the "rationalizing" model of Lei et al. (2016), exploiting rationale-level supervision often (though not always) improves agreement with human-provided rationales, as in prior work (Zhang et al., 2016; Strout et al., 2019). Interestingly, this does not seem strongly correlated with predictive performance.
Lei et al. (2016) outperforms the simple pipeline model when using a BERT encoder. Further, Lei et al. (2016) outperforms the 'BERT-to-BERT' pipeline on the comparable datasets for the final prediction tasks. This may be an artifact of the amount of text each model can select: 'BERT-to-BERT' is limited to sentences, while Lei et al. (2016) can select any subset of the text. Designing extraction models that learn to adaptively select contiguous rationales of appropriate length for a given task seems a potentially promising direction.
In Table 4 we report metrics for models that assign continuous importance scores to individual tokens. For these models we again measure downstream (task) performance (macro F1 or accuracy). Here the models are actually the same, and so downstream performance is equivalent. To assess the quality of token scores with respect to human annotations, we report the Area Under the Precision Recall Curve (AUPRC).
These scoring functions assign only soft scores to inputs (and may still use all inputs to come to a particular prediction), so we report the metrics intended to measure faithfulness defined above: comprehensiveness and sufficiency, averaged over 'bins' of tokens ordered by importance scores. To provide a point of reference for these metrics — which depend on the underlying model — we report results when rationales are randomly selected (averaged over 10 runs).
Both simple gradient and LIME-based scoring yield more comprehensive rationales than attention weights, consistent with prior work (Jain and Wallace, 2019; Serrano and Smith, 2019). Attention
fares better in terms of AUPRC — suggesting better agreement with human rationales — which is also in line with prior findings that it may provide plausible, but not faithful, explanation (Zhong et al., 2019). Interestingly, LIME does particularly well across these tasks in terms of faithfulness.
From the 'Random' results that we conclude models with overall poor performance on their final tasks tend to have an overall poor ordering, with marginal differences in comprehensiveness and sufficiency between them. For models that with high sufficiency scores: Movies, FEVER, CoS-E, and eSNLI, we find that random removal is particularly damaging to performance, indicating poor absolute ranking; whereas those with high comprehensiveness are sensitive to rationale length.
7 Conclusions and Future Directions
We have introduced a new publicly available resource: the Evaluating Rationales And Simple English Reasoning (ERASER) benchmark. This comprises seven datasets, all of which include both instance level labels and corresponding supporting snippets ('rationales') marked by human annotators. We have augmented many of these datasets with additional annotations, and converted them into a standard format comprising inputs, rationales, and outputs. ERASER is intended to facilitate progress on explainable models for NLP.
We proposed several metrics intended to measure the quality of rationales extracted by models, both in terms of agreement with human annotations, and in terms of 'faithfulness'. We believe these metrics provide reasonable means of comparison of specific aspects of interpretability, but we view the problem of measuring faithfulness, in particular, a topic ripe for additional research (which ERASER can facilitate).
Our hope is that ERASER enables future work on designing more interpretable NLP models, and comparing their relative strengths across a variety of tasks, datasets, and desired criteria. It also serves as an ideal starting point for several future directions such as better evaluation metrics for interpretability, causal analysis of NLP models and datasets of rationales in other languages.
We thank the anonymous ACL reviewers.
This work was supported in part by the NSF (CA-REER award 1750978), and by the Army Research Office (W911NF1810328).
David Alvarez-Melis and Tommi Jaakkola. 2017. A causal framework for explaining the predictions of black-box sequence-to-sequence models. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 412-421.
Leila Arras, Franziska Horn, Gregoire Montavon, Klaus-Robert Müller, and Wojciech Samek. 2017. "what is relevant in a text document?: An interpretable machine learning approach. In PloS one.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Joost Bastings, Wilker Aziz, and Ivan Titov. 2019. Interpretable neural predictions with differentiable binary variables. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2963-2977, Florence, Italy. Association for Computational Linguistics.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: Pretrained language model for scientific text. In EMNLP.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics.
Gino Brunner, Yang Liu, Damian Pascual, Oliver Richter, Massimiliano Ciaramita, and Roger Wattenhofer. 2020. On identifiability in transformers. In International Conference on Learning Representations.
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-snli: Natural language inference with natural language explanations. In Advances in Neural Information Processing Systems, pages 9539-9549.
Shiyu Chang, Yang Zhang, Mo Yu, and Tommi Jaakkola. 2019. A game theoretic approach to class-wise selective rationalization. In Advances in Neural Information Processing Systems, pages 10055-10065.
Sihao Chen, Daniel Khashabi, Wenpeng Yin, Chris Callison-Burch, and Dan Roth. 2019. Seeing things from a different angle: Discovering diverse perspectives about claims. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 542-557, Minneapolis, Minnesota.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724-1734, Doha, Qatar. Association for Computational Linguistics.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In *NAACL*.
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37-46.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Yanzhuo Ding, Yang Liu, Huanbo Luan, and Maosong Sun. 2017. Visualizing and understanding neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, Canada. Association for Computational Linguistics.
Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608.
Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. 2010. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303-338.
Shi Feng, Eric Wallace, Alvin Grissom, Mohit Iyyer, Pedro Rodriguez, and Jordan L. Boyd-Graber. 2018. Pathologies of neural models make interpretation difficult. In EMNLP.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1-6, Melbourne, Australia. Association for Computational Linguistics.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. 2019. A benchmark for interpretability methods in deep neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 9737-9748. Curran Associates, Inc.
Alon Jacovi and Yoav Goldberg. 2020. Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness? arXiv preprint arXiv:2004.03685.
Sarthak Jain and Byron C. Wallace. 2019. Attention is not Explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3543-3556, Minneapolis, Minnesota. Association for Computational Linguistics.
Sarthak Jain, Sarah Wiegrefe, Yuval Pinter, and Byron C. Wallace. 2020. Learning to Faithfully Rationally by Construction. In Proceedings of the Conference of the Association for Computational Linguistics (ACL).
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences. In Proc. of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. International Conference on Learning Representations.
Eric Lehman, Jay DeYoung, Regina Barzilay, and Byron C Wallace. 2019. Inferring which medical treatments work from reports of clinical trials. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 3705-3717.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 107-117.
Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016. Visualizing and understanding neural models in NLP. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 681-691, San Diego, California. Association for Computational Linguistics.
Zachary C Lipton. 2016. The mythos of model interpretability. arXiv preprint arXiv:1606.03490.
Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, pages 4765-4774.