SlowGuess/ACL_2020_OCR huggingface dataset:
Danish Pruthi†, Mansi Gupta‡, Bhuwan Dhingra†, Graham Neubig†, Zachary C. Lipton†
†Carnegie Mellon University, Pittsburgh, USA
$\ddagger$ Twitter, New York, USA
ddanish@cs.cmu.edu, mansig@twitter.com,
{bdhingra, gneubig, zlipton}@cs.cmu.edu
Attention mechanisms are ubiquitous components in neural architectures applied to natural language processing. In addition to yielding gains in predictive accuracy, attention weights are often claimed to confer interpretability, purportedly useful both for providing insights to practitioners and for explaining why a model makes its decisions to stakeholders. We call the latter use of attention mechanisms into question by demonstrating a simple method for training models to produce deceptive attention masks. Our method diminishes the total weight assigned to designated impermissible tokens, even when the models can be shown to nevertheless rely on these features to drive predictions. Across multiple models and tasks, our approach manipulates attention weights while paying surprisingly little cost in accuracy. Through a human study, we show that our manipulated attention-based explanations deceive people into thinking that predictions from a model biased against gender minorities do not rely on the gender. Consequently, our results cast doubt on attention's reliability as a tool for auditing algorithms in the context of fairness and accountability. $^{1}$
Since their introduction as a method for aligning inputs and outputs in neural machine translation, attention mechanisms (Bahdanau et al., 2014) have emerged as effective components in various neural network architectures. Attention works by aggregating a set of tokens via a weighted sum, where the attention weights are calculated as a function of both the input encodings and the state of the decoder.
Because attention mechanisms allocate weight among the encoded tokens, these coefficients are
Attention	Biography	Label
Original	Ms. X practices medicine in Memphis, TN and ... Ms. X speaks English and Spanish.	Physician
Ours	Ms. X practices medicine in Memphis, TN and ... Ms. X speaks English and Spanish.	Physician
Table 1: Example of an occupation prediction task where attention-based explanation (highlighted) has been manipulated to whitewash problematic tokens.
sometimes thought of intuitively as indicating which tokens the model focuses on when making a particular prediction. Based on this loose intuition, attention weights are often claimed to explain a model's predictions. For example, a recent survey on attention (Galassi et al., 2019) remarks:
"By inspecting the networks attention, ... one could attempt to investigate and understand the outcome of neural networks. Hence, weight visualization is now common practice."
In another work, De-Arteaga et al. (2019) study gender bias in machine learning models for occupation classification. As machine learning is increasingly used in hiring processes for tasks including resume filtering, the potential for bias raises the spectre that automating this process could lead to social harms. De-Arteaga et al. (2019) use attention over gender-revealing tokens (e.g., 'she', 'he', etc.) to verify the gender bias in occupation classification models—stating that "the attention weights indicate which tokens are most predictive". Similar claims about attention's utility for interpreting models' predictions are common in the literature (Li et al., 2016; Xu et al., 2015; Choi et al., 2016; Xie et al., 2017; Martins and Astudillo, 2016; Lai and Tan, 2019).
In this paper, we question whether attention scores necessarily indicate features that influence
1The code and the datasets used in paper are available at https://github.com/danishpruthi/deceptive-attention
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4782-4793
a model's predictions. Through a series of experiments on diverse classification and sequence-to-sequence tasks, we show that attention scores are surprisingly easy to manipulate. We design a simple training scheme whereby the resulting models appear to assign little attention to a specified set of impermissible tokens while continuing to rely upon those features for prediction. The ease with which attention can be manipulated without significantly affecting performance suggests that even if a vanilla model's attention weights conferred some insight (still an open and ill-defined question), these insights would rely on knowing the objective on which models were trained.
Our results present troublesome implications for proposed uses of attention in the context of fairness, accountability, and transparency. For example, malicious practitioners asked to justify how their models work by pointing to attention weights could mislead regulators with this scheme. For instance, looking at manipulated attention-based explanation in Table 1, one might (incorrectly) assume that the model does not rely on the gender prefix. To quantitatively study the extent of such deception, we conduct studies where we ask human subjects if the biased occupation classification models (like the ones audited by DeArteaga et al. (2019)) rely on gender related information. We find that our manipulation scheme is able to deceive human annotators into believing that manipulated models do not take gender into account, whereas the models are heavily biased against gender minorities (see §5.2).
Lastly, practitioners often overlook the fact that attention is typically not applied over words but over final layer representations, which themselves capture information from neighboring words. We investigate the mechanisms through which the manipulated models attain low attention values. We note that (i) recurrent connections allow information to flow easily to neighboring representations; (ii) for cases where the flow is restricted, models tend to increase the magnitude of representations corresponding to impermissible tokens to offset the low attention scores; and (iii) models additionally rely on several alternative mechanisms that vary across random seeds (see §5.3).
Many recent papers examine whether attention is a valid explanation or not. Jain et al. (2019) iden
tify alternate adversarial attention weights after the model is trained that nevertheless produce the same predictions, and hence claim that attention is not explanation. However, these attention weights are chosen from a large (infinite up to numerical precision) set of possible values and thus it is not surprising that multiple weights produce the same prediction. Moreover since the model does not actually produce these weights, they would never be relied on as explanations in the first place. Similarly, Serrano and Smith (2019) modify attention values of a trained model post-hoc by hard-setting the highest attention values to zero. They find that the number of attention values that must be zeroed out to alter the model's prediction is often too large, and thus conclude that attention is not a suitable tool to for determining which elements should be attributed as responsible for an output. In contrast to these two papers, we manipulate the attention via the learning procedure, producing models whose actual weights might deceive an auditor.
In parallel work to ours, Wiegrefe and Pinter (2019) examine the conditions under which attention can be considered a plausible explanation. They design a similar experiment to ours where they train an adversarial model, whose attention distribution is maximally different from the one produced by the base model. Here we look at a related but different question of how attention can be manipulated away from a set of impermissible tokens. Using human studies we show that our training scheme leads to attention maps that are more deceptive, since people find them to be more believable explanations of the output (see §5.2). We also extend our analysis to sequence-to-sequence tasks, and a broader set of models, including BERT, and identify mechanisms by which the manipulated models rely on the impermissible tokens despite assigning low attention to them.
Lastly, several papers deliberately train attention weights by introducing an additional source of supervision to improve predictive performance. In some of these papers, the supervision comes from known word alignments for machine translation (Liu et al., 2016; Chen et al., 2016), or by aligning human eye-gaze with model's attention for sequence classification (Barrett et al., 2018).
3 Manipulating Attention
Let  $S = w_{1},w_{2},\ldots ,w_{n}$  denote an input sequence of  $n$  words. We assume that for each task, we are
Dataset (Task)	Input Example	Impermissible Tokens (Percentage)
CommonCrawl Biographies (Physician vs Surgeon)	Ms. X practices medicine in Memphis, TN and is affiliated with ... Ms. X speaks English and Spanish.	Gender Indicators (6.5%)
Wikipedia Biographies (Gender Identification)	After that, Austen was educated at home until she went to boarding school with Cassandra early in 1785	Gender Indicators (7.6%)
SST + Wikipedia (Sentiment Analysis)	Good fun, good action, good acting, good dialogue, good pace, good cinematography. Helen Maxine Lamond Reddy (born 25 October 1941) is an Australian singer, actress, and activist.	SST sentence (45.5%)
Reference Letters (Acceptance Prediction)	It is with pleasure that I am writing this letter in support of ... I highly recommend her for a place in your institution. Percentile:99.0 Rank:Extraordinary.	Percentile, Rank (1.6%)
Table 2: Example sentences from each classification task, with highlighted impermissible tokens and their support.
given a pre-specified set of impermissible words  $\mathcal{I}$ , for which we want to minimize the corresponding attention weights. For example, these may include gender words such as "he", "she", "Mr", or "Ms". We define the mask  $\mathbf{m}$  to be a binary vector of size  $n$ , such that
$$
\mathbf {m} _ {i} = \left\{ \begin{array}{l l} 1, & \text {i f} w _ {i} \in \mathcal {I} \\ 0 & \text {o t h e r w i s e .} \end{array} \right.
$$
Further, let  $\alpha \in [0,1]^n$  denote the attention assigned to each word in  $S$  by a model, such that  $\sum_{i}\alpha_{i} = 1$ . For any task-specific loss function  $\mathcal{L}$  we define a new objective function  $\mathcal{L}' = \mathcal{L} + \mathcal{R}$  where  $\mathcal{R}$  is an additive penalty term whose purpose is to penalize the model for allocating attention to impermissible words. For a single attention layer, we define  $\mathcal{R}$  as:
$$
\mathcal {R} = - \lambda \log (1 - \boldsymbol {\alpha} ^ {T} \mathbf {m})
$$
and  $\lambda$  is a penalty coefficient that modulates the amount of attention assigned to impermissible tokens. The argument of the log term  $(1 - \alpha^T\mathbf{m})$  captures the total attention weight assigned to permissible words. In contrast to our penalty term, Wegreffe and Pinter (2019) use KL-divergence to maximally separate the attention distribution of the manipulated model  $(\alpha_{\mathrm{new}})$  from the attention distribution of the given model  $(\alpha_{\mathrm{old}})$ :
$$
\mathcal {R} ^ {\prime} = - \lambda \operatorname {K L} \left(\alpha_ {\text {n e w}} \| \alpha_ {\text {o l d}}\right). \tag {1}
$$
However, their penalty term is not directly applicable to our case: instantiating  $\alpha_{\mathrm{old}}$  to be uniform over impermissible tokens, and 0 over remainder tokens results in an undefined loss term.
When dealing with models that employ multiheaded attention, which use multiple different attention vectors at each layer of the model (Vaswani
et al., 2017) we can optimize the mean value of our penalty as assessed over the set of attention heads  $\mathcal{H}$  as follows:
$$
\mathcal {R} = - \frac {\lambda}{| \mathcal {H} |} \sum_ {h \in \mathcal {H}} \log (1 - \boldsymbol {\alpha} _ {h} ^ {T} \mathbf {m})).
$$
When a model has many attention heads, an auditor might not look at the mean attention assigned to certain words but instead look head by head to see if any among them assigns a large amount of attention to impermissible words. Anticipating this, we also explore a variant of our approach for manipulating multi-headed attention where we penalize the maximum amount of attention paid to impermissible words (among all heads) as follows:
$$
\mathcal {R} = - \lambda \cdot \min  _ {h \in \mathcal {H}} \log (1 - \boldsymbol {\alpha} _ {h} ^ {T} \mathbf {m}).
$$
For cases where the impermissible set of tokens is unknown apriori, one can plausibly use the top few highly attended tokens as a proxy.
We study the manipulability of attention on four binary classification problems, and four sequence-to-sequence tasks. In each dataset, (in some, by design) a subset of input tokens are known a priori to be indispensable for achieving high accuracy.
4.1 Classification Tasks
Occupation classification We use the biographies collected by De-Arteaga et al. (2019) to study bias against gender-minorities in occupation classification models. We carve out a binary classification task of distinguishing between surgeons and (non-surgeon) physicians from the multi-class
occupation prediction setup. We chose this subtask because the biographies of the two professions use similar words, and a majority of surgeons  $(>80\%)$  in the dataset are male. We further downsample minority classes—female surgeons, and male physicians—by a factor of ten, to encourage models to use gender related tokens. Our models (described in detail later in § 4.2) attain  $96.4\%$  accuracy on the task, and are reduced to  $93.8\%$  when the gender pronouns in the biographies are anonymized. Thus, the models (trained on unanonymized data) make use of gender indicators to obtain a higher task performance. Consequently, we consider gender indicators as impermissible tokens for this task.
Pronoun-based Gender Identification We construct a toy dataset from Wikipedia comprised of biographies, in which we automatically label biographies with a gender (female or male) based solely on the presence of gender pronouns. To do so, we use a pre-specified list of gender pronouns. Biographies containing no gender pronouns, or pronouns spanning both classes are discarded. The rationale behind creating this dataset is that due to the manner in which the dataset was created, attaining  $100\%$  classification accuracy is trivial if the model uses information from the pronouns. However, without the pronouns, it may not be possible to achieve perfect accuracy. Our models trained on the same data with pronouns anonymized, achieve at best  $72.6\%$  accuracy.
Sentiment Analysis with Distractor Sentences We use the binary version of Stanford Sentiment Treebank (SST) (Socher et al., 2013), comprised of 10,564 movie reviews. We append one randomly-selected "distractor" sentence to each review, from a set of opening sentences of Wikipedia pages. Here, without relying upon the tokens in the SST sentences, a model should not be able to outperform random guessing.
Graduate School Reference Letters We obtain a dataset of recommendation letters written for the purpose of admission to graduate programs. The task is to predict whether the student, for whom the letter was written, was accepted. The letters include students' ranks and percentile scores as marked by their mentors, which admissions committee members rely on. Indeed, we notice accu
racy improvements when using the rank and percentile features in addition to the reference letter. Thus, we consider percentile and rank labels (which are appended at the end of the letter text) as impermissible tokens. An example from each classification task is listed in Table 2. More details about the datasets are in the appendix.
4.2 Classification Models
Embedding + Attention For illustrative purposes, we start with a simple model with attention directly over word embeddings. The word embeddings are aggregated by a weighted sum (where weights are the attention scores) to form a context vector, which is then fed to a linear layer, followed by a softmax to perform prediction. For all our experiments, we use dot-product attention, where the query vector is a learnable weight vector. In this model, prior to attention there is no interaction between the permissible and impermissible tokens. The embedding dimension size is 128.
BiLSTM + Attention The encoder is a single-layer bidirectional LSTM model (Graves and Schmidhuber, 2005) with attention, followed by a linear transformation and a softmax to perform classification. The embedding and hidden dimension size are both set to 128.
Transformer Models We use the Bidirectional Encoder Representations from Transformers (BERT) model (Devlin et al., 2019). We use the base version consisting of 12 layers with self-attention. Further, each of the self-attention layers consists of 12 attention heads. The first token of every sequence is the special classification token [CLS], whose final hidden state is used for classification tasks. To block the information flow from permissible to impermissible tokens, we multiply attention weights at every layer with a self-attention mask  $\mathbf{M}$ , a binary matrix of size  $n \times n$  where  $n$  is the size of the input sequence. An element  $\mathbf{M}_{i,j}$  represents whether the token  $w_{i}$  should attend on the token  $w_{j}$ .  $\mathbf{M}_{i,j}$  is 1 if both  $i$  and  $j$  belong to the same set (either the set of impermissible tokens,  $\mathcal{I}$  or its complement  $\mathcal{I}^c$ ). Additionally, the [CLS] token attends to all the tokens, but no token attends to [CLS] to prevent the information flow between  $\mathcal{I}$  and  $\mathcal{I}^c$  (Figure 1 illustrates this setting). We attempt to manipulate attention from [CLS] token to other tokens, and consider two variants: one where we manipulate the maxi
${}^{2}$  Opening sentences tend to be declarative statements of fact and typically are sentiment-neutral.
4785
Figure 1: Restricted self-attention in BERT. The information flow through attention is restricted between impermissible and permissible tokens for every encoder layer. The arrows represent the direction of attention.
mum attention across all heads, and one where we manipulate the mean attention.
4.3 Sequence-to-sequence Tasks
Previous studies analysing the interpretability of attention are all restricted to classification tasks (Jain et al., 2019; Serrano and Smith, 2019; Wiegreiffe and Pinter, 2019). Whereas, attention mechanism was first introduced for, and reportedly leads to significant gains in, sequence-to-sequence tasks. Here, we analyse whether for such tasks attention can be manipulated away from its usual interpretation as an alignment between output and input tokens. We begin with three synthetic sequence-to-sequence tasks that involve learning simple input-to-output mappings.
Bigram Flipping The task is to reverse the bigrams in the input  $\left(\left\{w_{1}, w_{2} \ldots w_{2n-1}, w_{2n}\right\} \rightarrow \left\{w_{2}, w_{1}, \ldots w_{2n}, w_{2n-1}\right\}\right)$ .
Sequence Copying The task requires copying the input sequence  $\left(\{w_{1}, w_{2} \ldots w_{n-1}, w_{n}\} \rightarrow \{w_{1}, w_{2} \ldots w_{n-1}, w_{n}\}\right)$ .
Sequence Reversal The goal here is to reverse the input sequence  $(\{w_{1}, w_{2} \ldots w_{n-1}, w_{n}\} \to \{w_{n}, w_{n-1} \ldots w_{2}, w_{1}\})$ .
The motivation for evaluating on the synthetic tasks is that for any given target token, we precisely know the input tokens responsible. Thus, for these tasks, the gold alignments act as impermissible tokens in our setup (which are different for each output token). For each of the three tasks, we programmatically generate 100K random input training sequences (with their corresponding target sequences) of length upto 32. The input and output vocabulary is fixed to a 1000 unique tokens. For the task of bigram flipping, the input lengths
are restricted to be even. We use two sets of 100K unseen random sequences from the same distribution as the validation and test set.
Machine Translation (English to German)
Besides synthetic tasks, we also evaluate on English to German translation. We use the Multi30K dataset, comprising of image descriptions (Elliott et al., 2016). Since the gold target to source word-level alignment is unavailable, we rely on the Fast Align toolkit (Dyer et al., 2013) to align target words to their source counterparts. We use these aligned words as impermissible tokens.
For all sequence-to-sequence tasks, we use an encoder-decoder architecture. Our encoder is a bidirectional GRU, and our decoder is a unidirectional GRU, with dot-product attention over source tokens, computed at each decoding timestep. We also run ablation studies with (i) no attention, i.e. just using the last (or the first) hidden state of the encoder; and (ii) uniform attention, i.e. all the source tokens are uniformly weighted.
In this section we examine how lowering attention affects task performance (§ 5.1). We then present experiments with human participants to quantify the deception with manipulated attention (§ 5.2). Lastly, we identify alternate workarounds through which models preserve task performance (§ 5.3).
5.1 Attention mass and task performance
For the classification tasks, we experiment with the loss coefficient  $\lambda \in \{0, 0.1, 1\}$ . In each experiment, we measure the (i) attention mass: the sum of attention values over the set of impermissible tokens averaged over all the examples, and (ii) test accuracy. During the course of training (i.e. after each epoch), we arrive at different models from which we choose the one whose performance is within  $2\%$  of the original accuracy and provides the greatest reduction in attention mass on impermissible tokens. This is done using the development set, and the results on the test set from the chosen model are presented in Table 3. Across most tasks, and models, we find that our manipulation scheme severely reduces the attention mass on
4 Implementation details: the encoder and decoder token embedding size is 256, the encoder and decoder hidden dimension size is 512, and the teacher forcing ratio is 0.5. We use top-1 greedy strategy to decode the output sequence.
5 All data and code will be released on publication.
3These tasks have been previously used in the literature to assess the ability of RNNs to learn long-range reorderings and substitutions (Grefenstette et al., 2015).
4786
Model	λ	I	Occupation Pred.	Gender Identify	SST + Wiki	Ref. Letters
Acc.	A.M.	Acc.	A.M.	Acc.	A.M.	Acc.	A.M.
Embedding	0.0	X	93.8	-	66.8	-	48.9	-	74.2	2.3
Embedding	0.0	✓	96.3	51.4	100	99.2	70.7	48.4	77.5	2.3
Embedding	0.1	✓	96.2	4.6	99.4	3.4	67.9	36.4	76.8	0.5
Embedding	1.0	✓	96.2	1.3	99.2	0.8	48.4	8.7	76.9	0.1
BiLSTM	0.0	X	93.3	-	63.3	-	49.1	-	74.7	-
BiLSTM	0.0	✓	96.4	50.3	100	96.8	76.9	77.7	77.5	4.9
BiLSTM	0.1	✓	96.4	0.08	100	< 10-6	60.6	0.04	76.9	3.9
BiLSTM	1.0	✓	96.7	< 10-2	100	< 10-6	61.0	0.07	74.2	< 10-2
BERT	0.0	X	95.0	-	72.8	-	50.4	-	68.2
BERT (mean)	0.0	✓	97.2	13.9	100	80.8	90.8	59.0	74.7	2.6
BERT (mean)	0.1	✓	97.2	0.01	99.9	< 10-3	90.9	< 10-2	76.2	< 10-1
BERT (mean)	1.0	✓	97.2	< 10-3	99.9	< 10-3	90.6	< 10-3	75.2	< 10-2
BERT	0.0	X	95.0	-	72.8	-	50.4	-	68.2
BERT (max)	0.0	✓	97.2	99.7	100	99.7	90.8	96.2	74.7	28.9
BERT (max)	0.1	✓	97.1	< 10-3	99.9	< 10-3	90.7	< 10-2	76.7	0.6
BERT (max)	1.0	✓	97.4	< 10-3	99.8	< 10-4	90.2	< 10-3	75.9	< 10-2
Table 3: Accuracy of various classification models along with their attention mass (A.M.) on impermissible tokens  $\mathcal{I}$ , with varying values of the loss coefficient  $\lambda$ . The first row for each model class represents the case when impermissible tokens  $\mathcal{I}$  for the task are deleted/anonymized. For most models, and tasks, we can severely reduce attention mass on impermissible tokens while preserving original performance ( $\lambda = 0$  implies no manipulation).
Attention	λ	Bigram Flip	Sequence Copy	Sequence Reverse	En → De MT
Acc.	A.M.	Acc.	A.M.	Acc.	A.M.	BLEU	A.M.
Dot-Product	0.0	100.0	94.5	99.9	98.8	100.0	94.1	24.4	20.6
Uniform	0.0	97.8	5.2	93.8	5.2	88.1	4.7	18.5	5.9
None	0.0	96.4	0.0	84.1	0.0	84.1	0.0	14.9	0.0
Manipulated	0.1	99.9	24.4	100.0	27.3	100	27.6	23.7	7.0
Manipulated	1.0	99.8	0.03	92.9	0.02	99.8	0.01	20.6	1.1