SlowGuess/ACL_2020_OCR huggingface dataset:
Table 4: Performance of sequence-to-sequence models and their attention mass (A.M.) on impermissible tokens  $\mathcal{I}$ , with varying values of the loss coefficient  $\lambda$ . Similar to classification tasks, we can severely reduce attention mass on impermissible tokens while retaining original performance. All values are averaged over five runs.
impermissible tokens compared to models without any manipulation (i.e. when  $\lambda = 0$ ). This reduction comes at a minor, or no, decrease in task accuracy. Note that the models can not achieve performance similar to the original model (as they do), unless they rely on the set of impermissible tokens. This can be seen from the gap between models that do not use impermissible tokens ( $\mathcal{I} \times$ ) from ones that do ( $\mathcal{I} \checkmark$ ).
The only outlier to our findings is the SST+Wiki sentiment analysis task, where we observe that the manipulated Embedding and BiLSTM models reduce the attention mass but also lose accuracy. We speculate that these models are under parameterized and thus jointly reducing attention mass and retaining original accuracy is harder. The more expressive BERT obtains an accuracy of over  $90\%$  while reducing the maximum attention mass over the movie review from  $96.2\%$  to  $10^{-3}\%$ .
For sequence-to-sequence tasks, from Table 4, we observe that our manipulation scheme can similarly reduce attention mass over impermissible alignments while preserving original performance. To measure performance, we use token-by-token accuracy for synthetic tasks, and BLEU score for English to German MT. We also notice that the models with manipulated attention (i.e. deliberately misaligned) outperform models with none or uniform attention. This suggests that attention mechanisms add value to the learning process in sequence-to-sequence tasks which goes beyond their usual interpretation as alignments.
5.2 Human Study
To study the deceptiveness of attention maps trained using various training schemes, we present a series of inputs and outputs from classification
models to three human subjects. $^{6}$  The models are BiLSTMs that are trained to classify occupations into either physician or surgeon given a short biography. We highlight the input tokens as per the attention scores from three different training schemes: (i) original dot-product attention, (ii) adversarial attention from Wiegrefe and Pinter (2019), and, (iii) our proposed attention manipulation strategy. We ask human annotators (Q1): Do you think that this prediction was influenced by the gender of the individual? Each participant answers either "yes" or "no" for a set of 50 examples from each of the three attention schemes. We shuffled the order of sets among the three participants to prevent any ordering bias. Additionally, participants can flip through many examples before registering their answers. After looking at 50 examples from a given attention scheme, we inquire about trustworthiness of the attention scores (Q2): Do you believe the highlighted tokens capture the factors that drive the models' prediction? They answer the question on a scale of 1 to 4, where 1 denotes that the highlighted tokens do not determine the models' prediction, whereas 4 implies they significantly determine the models' prediction. We deliberately ask participants once (towards the end) about the trustworthiness of attention-based explanations, in contrast to polling after each example, as it requires multiple examples to assess whether the explanations capture factors that are predictive. Participants were kept unaware of the specifics of the classifier or the explanation technique used. Detailed instructions presented to participants are available in the supplementary material.
Results We find that for the original dot-product attention, annotators labeled  $66\%$  of predictions to be influenced by gender. Whereas for the other two attention schemes, none of the predictions were marked to be influenced by gender (see Table 5). This is despite all three models achieving roughly the same high accuracy  $(96\%)$  which relies on gender information. This demonstrates the efficacy of our manipulation scheme—predictions from models biased against gender minorities are perceived (by human participants) as not being influenced by gender. Further, our manipulated explanations receive a trustworthiness score of 2.67
Attention	Example	Q1	Q2
Original	Ms. X practices medicine and specializes in urological surgery	66% (yes)	3.00
Adversarial (Wiegreffe and Pinter, 2019)	Ms. X practices medicine and specializes in urological surgery	0% (yes)	1.00
Ours	Ms. X practices medicine and specializes in urological surgery	0% (yes)	2.67
Table 5: Results to questions posed to human participants. Q1: Do you think that this prediction was influenced by the gender of the individual? Q2: Do you believe the highlighted tokens capture the factors that drive the models prediction? See § 5.2 for discussion.
(out of 4), only slightly lower than the score for the original explanations, and significantly better than the adversarial attention. We found that the KL divergence term in training adversarial attention (Eq. 1) encourages all the attention mass to concentrate on a single uninformative token for most examples, and hence was deemed as less trustworthy by the annotators (see Table 5, more examples in appendix). By contrast, our manipulation scheme only reduces attention mass over problematic tokens, and retains attention over non-problematic but predictive ones (e.g. "medicine") making it more believable. We assess agreement among annotators, and calculate the Fleiss' Kappa to be 0.97, suggesting almost perfect agreement.
5.3 Alternative Workarounds
We identify two mechanisms by which the models cheat, obtaining low attention values while remaining accurate.
Models with recurrent encoders can simply pass information across tokens through recurrent connections, prior to the application of attention. To measure this effect, we hard-set the attention values corresponding to impermissible words to zero after the manipulated model is trained, thus clipping their direct contributions for inference. For gender classification using the BiLSTM model, we are still able to predict over  $99\%$  of instances correctly, thus confirming a large degree of information flow to neighboring representations. In contrast, the Embedding model (which has no means to pass the information pre-attention) at
The participating subjects are first and second year graduate students specializing in NLP/ML and are knowledgeable about attention mechanisms, but unaware about our work.
7 A recent study (Brunner et al., 2019) similarly observes a high degree of 'mixing' of information across layers in Transformer models.
(a) Bigram Flipping
Figure 2: For three sequence-to-sequence tasks, we plot the original attention map on the left, followed by the attention plots of two manipulated models. The only difference between the manipulated models for each task is the (random) initialization seed. Different manipulated models resort to different alternative mechanisms.
(b) Sequence Copying
(c) Sequence Reversal
Attention Mass
Vector Norm
- - Attention Mass w/o manipulation
- - Vector Norm w/o manipulation
Embedding + Attention
Figure 3: For gender identification task, the norms of embedding vectors corresponding to impermissible tokens increase considerably in Embedding+Attention model to offset the low attention values. This is not the case for BiLSTM+Attention model as it can pass information due to recurrent connections.
BiLSTM + Attention
tains only about  $50\%$  test accuracy after zeroing the attention values for gender pronouns. We see similar evidence of passing around information in sequence-to-sequence models, where certain manipulated attention maps are off by one or two positions from the gold alignments (see Figure 2).
Models restricted from passing information prior to the attention mechanism tend to increase the magnitude of the representations correspond
ing to impermissible words to compensate for the low attention values. This effect is illustrated in Figure 3, where the L2 norm of embeddings for impermissible tokens increase considerably for the Embedding model during training. We do not see increased embedding norms for the BiLSTM model, as this is unnecessary due to the model's capability to move around relevant information.
We also notice that differently initialized models attain different alternative mechanisms. In Figure 2, we present attention maps from the original model, alongside two manipulated models initialized with different seeds. In some cases, the attention map is off by one or two positions from the gold alignments. In other cases, all the attention is confined to the first hidden state. In such cases, manipulated models are similar to a no-attention model, yet they offer better performance. In preliminary experiments, we found a few such models that outperform the no-attention baseline, even when the attention is turned off during inference. This suggests that attention offers benefits during training, even if it is not used during inference.
Amidst practices that perceive attention scores to be an indication of what the model focuses on, we characterize the manipulability of attention mechanism and the (surprisingly small) cost to be paid for it in accuracy. Our simple training scheme produces models with significantly reduced attention mass over tokens known a priori to be useful for prediction, while continuing to use them. Further analysis reveals how the manipulated models cheat, and raises concerns about the potential use of attention as a tool to audit models.
The authors thank Dr. Julian McAuley for providing, and painstakingly anonymizing the data for reference letters. We also acknowledge Alankar Jain for carefully reading the manuscript and providing useful feedback. ZL thanks Amazon AI, NVIDIA, Salesforce, Facebook AI, AbridgeAI, UPMC, the Center for Machine Learning in Health, the PwC Center, the AI Ethics and Governance Fund, and DARPA's Learning with Less Labels Initiative, for their support of ACMI Lab's research on robust and societally aligned machine learning.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Maria Barrett, Joachim Bingel, Nora Hollenstein, Marek Rei, and Anders Søgaard. 2018. Sequence classification with human attention. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 302-312.
Gino Brunner, Yang Liu, Damian Pascual, Oliver Richter, and Roger Wattenhofer. 2019. On the validity of self-attention as explanation in transformer models. arXiv preprint arXiv:1908.04211.
Wenhu Chen, Evgeny Matusov, Shahram Khadivi, and Jan-Thorsten Peter. 2016. Guided alignment training for topic-aware neural machine translation. arXiv preprint arXiv:1607.01628.
Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart. 2016. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. In Advances in Neural Information Processing Systems, pages 3504-3512.
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnamaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in bios: A case study of semantic representation bias in a high-stakes setting. arXiv preprint arXiv:1901.09451.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. North American Chapter of the Association for Computational Linguistics (NAACL).
Chris Dyer, Victor Chahuneau, and Noah A Smith. 2013. A simple, fast, and effective reparameterization of ibm model 2. North American Chapter of the Association for Computational Linguistics (NAACL).
Desmond Elliott, Stella Frank, Khalil Sima'an, and Lucia Specia. 2016. Multi30k: Multilingual english-german image descriptions. arXiv preprint arXiv:1605.00459.
Andrea Galassi, Marco Lippi, and Paolo Torroni. 2019. Attention, please! a critical review of neural attention models in natural language processing. arXiv preprint arXiv:1902.02181.
Alex Graves and Jürgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. *Neural Networks*, 18(5-6):602-610.
Edward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman, and Phil Blunsom. 2015. Learning to transduce with unbounded memory. In Advances in neural information processing systems, pages 1828-1836.
Sarthak Jain, Ramin Mohammadi, and Byron C Wallace. 2019. Attention is not explanation. North American Chapter of the Association for Computational Linguistics (NAACL).
Vivian Lai and Chenhao Tan. 2019. On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 29-38. ACM.
Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. Understanding neural networks through representation erasure. arXiv preprint arXiv:1612.08220.
Lemao Liu, Masao Utiyama, Andrew Finch, and Eiichiro Sumita. 2016. Neural machine translation with supervised attention. arXiv preprint arXiv:1609.04186.
Andre Martins and Ramon Astudillo. 2016. From softmax to sparsemax: A sparse model of attention and multi-label classification. In International Conference on Machine Learning, pages 1614-1623.
Sofia Serrano and Noah A Smith. 2019. Is attention interpretable? 57th annual meeting of the Association for Computational Linguistics (ACL).
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Sarah Wiegrefe and Yuval Pinter. 2019. Attention is not not explanation. Proceedings of the 2019 conference on Empirical Methods in Natural Language Processing, EMNLP.
Qizhe Xie, Xuezhe Ma, Zihang Dai, and Eduard Hovy. 2017. An interpretable knowledge transfer model for knowledge base completion. arXiv preprint arXiv:1704.05908.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. arXiv preprint arXiv:1502.03044.
A Instructions for human study
In a series of examples, we present the inputs and outputs of a machine learning (ML) model trained to predict occupation (physician or surgeon) given a short bio (text). In each bio, we attempt to explain the predictions of the model. Specifically, we employ a technique that highlights words that (per our explanation method) are thought to be responsible for a particular prediction (colloquially, what the model focuses on). For each unique example below, answer the following question: Do you think that this prediction was influenced by the gender of the individual?
- Yes, I suspect that the gender influenced the prediction.
- No, I have no reason to suspect that gender influenced the prediction.
Please note that, all the examples in this file are input, output pairs from one specific model. Further, darker shades of highlighting indicate a higher emphasis for the token (as per our explanation method).
After showing 50 examples from a given attention scheme, we inquire: Overall, do you believe the highlighted tokens capture the factors that drive the models prediction?
1. The highlighted tokens capture factors that do not determine the models prediction.
2. The highlighted tokens capture factors that marginally determine the models prediction.
3. The highlighted tokens capture factors that moderately determine the models predictions.
4. The highlighted tokens capture factors that significantly determine the models predictions.
Details about the datasets used for classification tasks are available in Table 6.
A few qualitative examples illustrating three different attention schemes are listed in Table 7.
Dataset (Task)	Train	Val	Test
CommonCrawl Biographies (Physician vs Surgeon)	17629	2519	5037
Wikipedia Biographies (Gender Identification)	9017	1127	1127
SST + Wikipedia (Sentiment Analysis)	6920	872	1821
Reference Letters (Acceptance Prediction)	32800	4097	4094
Table 6: Number of training, validation, and test examples in various datasets used for classification tasks.
Attention	Input Example	Prediction
Original	Ms. X practices medicine and specializes in urological surgery	Physician
Adversarial (Wiegreffe and Pinter, 2019)	Ms. X practices medicine and specializes in urological surgery	Physician
Ours	Ms. X practices medicine and specializes in urological surgery	Physician

Original	Ms. X practices medicine in Fort Myers, FL and specializes in family medicine	Physician
Adversarial (Wiegreffe and Pinter, 2019)	Ms. X practices medicine in Fort Myers, FL and specializes in family medicine	Physician
Ours	Ms. X practices medicine in Fort Myers, FL and specializes in family medicine	Physician

Original	Having started his surgical career as a general orthopaedic surgeon, Mr X retains a broad practice which includes knee and hand surgery. He still does regular trauma on-call for the North Hampshire hospital and treats all types of orthopaedic problems and trauma.	Surgeon
Adversarial (Wiegreffe and Pinter, 2019)	Having started his surgical career as a general orthopaedic surgeon, Mr X retains a broad practice which includes knee and hand surgery. He still does regular trauma on-call for the North Hampshire hospital and treats all types of orthopaedic problems and trauma.	Surgeon
Ours	Having started his surgical career as a general orthopaedic surgeon, Mr X retains a broad practice which includes knee and hand surgery. He still does regular trauma on-call for the North Hampshire hospital and treats all types of orthopaedic problems and trauma.	Surgeon

Original	Ms. X practices medicine in ... and specializes in pediatrics. Ms. X is affiliated with childrens of Alabama, Saint Vincent's hospital Birmingham and Brookwood Medical Center. Ms. X speaks English and Arabic.	Physician
Adversarial (Wiegreffe and Pinter, 2019)	Ms. X practices medicine in ... and specializes in pediatrics. Ms. X is affiliated with childrens of Alabama, Saint Vincent's hospital Birmingham and Brookwood Medical Center. Ms. X speaks English and Arabic.	Physician
Ours	Ms. X practices medicine in ... and specializes in pediatrics. Ms. X is affiliated with childrens of Alabama, Saint Vincent's hospital Birmingham and Brookwood Medical Center. Ms. X speaks English and Arabic.	Physician
Table 7: Qualitative examples.