SlowGuess/NAACL_2022_OCR huggingface dataset:
Arshdeep Sekhon and Yangfeng Ji and Matthew B. Dwyer and Yanjun Qi  
University of Virginia, USA
Recent literature has seen growing interest in using black-box strategies like CHECKLIST for testing the behavior of NLP models. Research on white-box testing has developed a number of methods for evaluating how thoroughly the internal behavior of deep models is tested, but they are not applicable to NLP models. We propose a set of white-box testing methods that are customized for transformer-based NLP models. These include MASK NEU-RON COVERAGE (MNCOVER) that measures how thoroughly the attention layers in models are exercised during testing. We show that MNCOVER can refine testing suites generated by CHECKLIST by substantially reduce them in size, for more than  $60\%$  on average, while retaining failing tests - thereby concentrating the fault detection power of the test suite. Further we show how MNCOVER can be used to guide CHECKLIST input generation, evaluate alternative NLP testing methods, and drive data augmentation to improve accuracy.
Previous NLP methods have used black-box testing to discover errors in NLP models. For instance, Checklist(Ribeiro et al., 2020) introduces a blackbox testing strategy as a new evaluation methodology for comprehensive behavioral testing of NLP models. CheckList introduced different test types, such as prediction invariance in the presence of certain perturbations.
Black-box testing approaches, like Checklist, may produce distinct test inputs that yield very similar internal behavior from an NLP model. Requiring that generated tests are distinct both from a black-box and a white-box perspective - that measures test similarity in terms of latent representations - has the potential to reduce the cost of testing without reducing its error-detection effectiveness. Researchers have explored a range of white-box coverage techniques that focus on neu
ron activations and demonstrated their benefit on architecturally simple feed-forward networks (Pei et al., 2017; Tian et al., 2018; Ma et al., 2018a; Dola et al., 2021). However, transformer-based NLP models incorporate more complex layer types, such as those computing self-attention, to which prior work is inapplicable.
In this paper, we propose a suite of white-box coverage metrics. We first adapt the k-multisection neuron coverage measure from (Ma et al., 2018a) to Transformer architectures. Then we design a novel MNCOVER coverage metric, tailored to NLP models. MNCOVER focuses on the neural modules that are important for NLP and designs strategies to ensure that those modules' behavior is thoroughly exercised by a test set. Our proposed coverage metric, when used to guide test generation, can cost-effectively achieve high-levels of coverage.
Figure 1 shows one example of how MNCOVER can work in concert with CheckList to produce a small and effective test set. The primary insight is that not all text sentences contain new information that will improve our confidence in a target model's behavior. In this list, multiple sentences were generated with similar syntactic and semantic structure. These sentences cause the activation of sets of attention neurons that have substantial overlap. This represents a form of redundancy in testing an NLP model. Coverage-based filtering seeks to identify when an input's activation of attention neurons is subsumed by that of prior test inputs - such inputs are filtered. In the Figure the second and third sentences are filtered out because their activation of attention neurons is identical to the first test sentence. As we show in §4 this form of filtering can substantially reduce test suite size while retaining tests that expose failures in modern NLP models, such as BERT.
The primary contributions of the paper lie in:
- Introducing MNCOVER a test coverage metric designed to address the attention-layers that
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1547 - 1558 July 10-15, 2022 ©2022 Association for Computational Linguistics
Test Set Selection using Coverage
'I liked the cabin crew.'
'We like this food.'
'I liked this aircraft.'
The customer service was exceptional
Figure 1: Example of our proposed MNCOVER's filtering on a set of test examples.
are characteristic of NLP models and to account for the data distribution by considering task-specific important words and combinations.
- Demonstrating through experiments on 2 NLP models (BERT, Roberta), 2 datasets (SST-2, QQP), and 24 sentence transformations that MNCOVER can substantially reduce the size of test sets generated by CheckList, by  $64\%$  on average, while improving the failure detection of the resulting tests, by  $13\%$  on average.
- Demonstrating that MNCOVER provide an effective supplementary criterion for evaluating the quality of test sets and that it can be used to generate augmented training data that improves model accuracy.
Coverage for testing deep networks The research of Coverage testing focuses on the concept of "adequacy criterion" that defines when "enough" testing has been performed. The white-box coverage testing has been proposed by multiple recent studies to test deep neural networks (Pei et al., 2017; Ma et al., 2018a,b; Dola et al., 2021). DeepXplore (Pei et al., 2017), a white-box differential testing algorithm, introduced Neuron Coverage for DNNs to guide systematic exploration of DNN's internal logic. Let us use  $\mathbf{D}$  to denote a set of test inputs (normally named as a test suite in behavior testing). The Neuron Coverage regarding  $\mathbf{D}$  is defined as the ratio between the number of unique activated neurons (activated by  $\mathbf{D}$ ) and the total number of neurons in that DNN under behavior testing. A neuron is considered to be activated if its output is higher than a threshold value (e.g., 0). Another closely related study, DeepTest (Tian et al., 2018), proposed a gray-box, neuron coverage-guided test suite generation strategy. Then, the study DeepGauge (Ma et al., 2018a) expands the neuron coverage definition by introducing the kmultisection neuron coverage criteria to produce a multi-granular set of DNN coverage metrics. For a given neuron  $\mathbf{n}$ , the kmultisection neuron coverage measures how thoroughly
a given set of test inputs like  $\mathbf{D}$  covers the range  $[low_n, high_n]$ . The range  $[low_n, high_n]$  is divided into k equal bins (i.e., k-multisection), for  $k > 0$ . For  $D$  and the target neuron  $\mathbf{n}$ , its k-multisection neuron coverage is then defined as the ratio of the number of bins covered by  $D$  and the total number of bins, i.e., k. For an entire DNN model, the k-multisection neuron coverage is then the ratio of all the activated bins for all its neurons and the total number of bins for all neurons in the DNN.
Transformer architectureNLP is undergoing a paradigm shift with the rise of large scale Transformer models (e.g., BERT, DALL-E, GPT-3) that are trained on unprecedented data scale and are adaptable to a wide range of downstream tasks(Bommasani et al., 2021).
These models embrace the Transformer architecture (Vaswani et al., 2017) and can capture long-range pairwise or higher-order interactions between input elements. They utilize the self-attention mechanism(Vaswani et al., 2017) that enables shorter computation paths and provides parallelizable computation for learning to represent a sequential input data, like text. Transformer receives inputs in the general form of word tokens. The sequence of inputs is converted to vector embeddings that get repeatedly re-encoded via the self-attention mechanism. The self-attention can repeat for many layers, with each layer re-encoding and each layer maintaining the same sequence length. At each layer, it corresponds to the following operations to learn encoding of token at position  $i$ :
$$
\alpha^ {i j} = \operatorname {s o f t m a x} \left(\left(\mathbf {W} ^ {q} \boldsymbol {h} _ {i}\right) ^ {\top} \left(\mathbf {W} ^ {k} \boldsymbol {h} _ {j}\right) / \sqrt {d}\right) \tag {1}
$$
$$
\bar {\boldsymbol {h}} _ {i} = \sum_ {j = 1} ^ {M} \alpha^ {i j} \mathbf {W} ^ {v} \boldsymbol {h} _ {j} \tag {2}
$$
$$
\boldsymbol {h} _ {i} ^ {\prime} = \sigma (\bar {\boldsymbol {h}} _ {i} \mathbf {W} ^ {r} + \boldsymbol {b} _ {1}) \mathbf {W} ^ {o} + \boldsymbol {b} _ {2}. \tag {3}
$$
Here  $\mathbf{W}^k$  is the key weight matrix,  $\mathbf{W}^q$  is the query weight matrix,  $\mathbf{W}^v$  is the value weight matrix,  $\mathbf{W}^r$  and  $\mathbf{W}^o$  are transformation matrices, and  $b_{1}$  and  $b_{2}$  are bias vectors.
State-of-the-art NLP models are large-scale with millions of neurons, due to large hidden sizes and multiple layers. We propose to simplify and view these foundation models (Bommasani et al., 2021) through two levels of granularity: (1) Word Level: that includes the position-level embeddings at each
Figure 2: A visual depiction of MNCOVER for masking neurons to measure coverage.
layer and (2) Pairwise Attention Level: that includes the pairwise self-attention neurons between two positions at each layer. In the rest of this paper, we denote the vector embeddings at location  $i$  for layer  $l$  as  $\mathbf{h}_i^{l}$  and name these as the word level neurons at layer  $l$ . We also denote the  $\alpha^{ij}$  at layer  $l$  and head  $h$  as  $\alpha_{lk}^{ij}$ , and call them as the attention level neurons at layer  $l$ .
3.1 Extending Neuron Coverage (COVER) for Testing NLP Model
Now we use the above two layers' view we proposed, to adapt the vanilla neuron coverage concepts from the literature to NLP models. First, we introduce a basic definition: "activated neuron bins" (Ma et al., 2018b):
Definition 1 Activated Neuron Bins (ANB): For each neuron, we partition the range of its values (obtained from training data) into  $B$  bins/sections. We define ANB for a given text input if the input's activation value from the target neuron falls into the corresponding bin range.
Then we adapt the above definition to the NLP model setting, by using the after-mentioned two layers' view. We design two phrases: Word Neuron Bins, and Activated Word Neuron Bins in the following Definition (2).
Definition 2 Activated Word Neuron Bins(AWB): We discretize the possible values of each neuron in  $\mathbf{h}_t^{\prime \ell}$  (whose  $d$ -th embedding dimension is  $\mathbf{h}_{dt}^{\prime \ell}$ ) into  $B$  sections. We propose a function  $\phi_w$  who takes two arguments, as  $\phi_w(h_{dt}^{\prime lb},\mathbf{x})$  for a given input  $\mathbf{x}$ .  $\phi_w(h_{dt}^{\prime lb},\mathbf{x}) = 1$  if it is an activated word neuron bin (shortened as AWB), else 0 if not activated.
Similarly, for our attention neuron at layer  $l$ , head  $k$ , word position  $i$  and position  $j$ :  $\alpha_{ij}^{lk}$ , we introduce the definition of "attention neuron bins" and "Activated Attention Neuron Bins" in the fol
lowing Definition (3).
Definition 3 Activated Attention Neuron Bins (AAB): We discretize the possible values of neuron  $\alpha_{ij}^{lk}$  into  $B$  sections. We denote the state of the  $b^{th}$  section of this attention neuron using  $\phi_a(\alpha_{lk}^{ijb},\mathbf{x})$ .  $\phi_{a}(\alpha_{lk}^{ijb},\mathbf{x}) = 1$  if it is an activated attention neuron bin (denoted by AAB) by an input  $\mathbf{x}$  and  $\phi_{a}(\alpha_{lk}^{ijb},\mathbf{x}) = 0$  if not activated.
$$
\mathbb {N} (\mathrm {A W B} (\mathbf {x})) = \sum_ {l t d b} \phi_ {w} \left(h _ {d t} ^ {\prime l b}, \mathbf {x}\right) \tag {4}
$$
$$
\mathbb {N} (\mathrm {A A B} (\mathbf {x})) = \sum_ {i j b k} \phi_ {a} \left(\alpha_ {l k} ^ {i j b}, \mathbf {x}\right) \tag {5}
$$
The coverage, denoted by COVER, of a dataset  $\mathcal{T}$  for a target model is then defined as the ratio between the number of "activated" neurons and total neurons:
$$
\operatorname {C O V E R} = \frac {\mathrm {N} (\mathrm {A W B}) + \lambda \mathrm {N} (\mathrm {A A B})}{\mathrm {N} (\mathrm {W B}) + \lambda \mathrm {N} (\mathrm {A B})} \tag {6}
$$
Here,  $\lambda$  is a scaling factor.
Now let us assume the total number of layers be  $D$ , total number of heads  $H$ , maximum length  $L$ , total bins  $B$  and total embedding size be  $E$ . Considering the example case of the BERT(Devlin et al., 2019) model, total number of word level neurons to be measured are then  $L \times E \times D = 128 \times 768 \times 13 \sim 0.1$  million. The total number of the attention level neurons is then  $L \times L \times H \times D = 128 \times 128 \times 12 \times 12 \sim 2$  million.
3.2 MASK NEURON COVERAGE (MNCOVER)
However, accounting for every word and attention neuron's behavior for a large pre-trained model like BERT is difficult for two reasons: (1). If we desire to test each neuron at the output of all transformer layers in each BERT layer, we need to account for the behavior of every neuron, which for a
Figure 3: Learning the masks prior to testing: Globally important words and interactions are learnt by masking inputs to a target model.
large pre-trained model like BERT is in the order of millions. (2). If we test every possible neuron, we need to track many neurons that are almost irrelevant for a target task and/or model. This type of redundancy makes the behavior testing less confident and much more expensive.
To mitigate these concerns, we propose to only focus on important words and their combinations that may potentially contain 'surprising' new information for the model and hence need to be tested.
We assume we have access to a word level importance mask, denoted by  $\mathbf{M}_W\in \{0,1\}^{|V|}$  and the interaction importance mask by  $\mathbf{M}_A\in$ $\{0,1\}^{|V|\times |V|}$ . Each entry in  $\mathbf{M}_{Ww_t}\in \{0,1\}$  represents the importance of word  $w_{t}$ . Similarly, each entry in  $\mathbf{M}_{A\mathbf{x}_i,\mathbf{x}_j}\in \{0,1\}$  represents the importance of interaction between token  $w_{t_i}$  and  $w_{t_j}$ . These masks aim for filtering out unimportant tokens (and their corresponding neurons at each layer) for measuring coverage signals. We apply the two masks at each layer to mask out unimportant attention pairs to prevent them from being counted towards coverage calculation. With the masks, the AWB and AAB are revised and we then define MASK NEURON COVERAGE (MNCOVER) accordingly:
$$
\mathbb {N} (\mathbf {M a s k - A W B} (\mathbf {x})) = \sum_ {l t d b} \mathbb {M} _ {w _ {\mathbf {x} _ {t}}} * \phi_ {w} (h _ {d t} ^ {' l b}, \mathbf {x})
$$
$$
\mathbb {N} (\mathbf {M a s k - A A B} (\mathbf {x})) = \sum_ {i j k b} \mathbb {M} _ {a _ {\mathbf {x} _ {i}, \mathbf {x} _ {j}}} * \phi_ {a} (\alpha_ {l k} ^ {i j b}, \mathbf {x})
$$
$$
\mathrm {M N C O V E R} = \frac {\mathrm {N} (\text {M a s k - A W B}) + \lambda \mathrm {N} (\text {M a s k - A A B})}{\mathrm {N} (\mathrm {W B}) + \lambda \mathrm {N} (\mathrm {A B})} \tag {7}
$$
3.3 Learning Importance Masks
In this section, we explain our mask learning strategy that enables us to learn globally important words and their pairwise combinations for a model's prediction without modifying a target model's parameters.
We learn the two masks through a bottleneck strategy, that we call WIMASK layer. Given a target model  $f$ , we insert this mask bottleneck layer
between the word embedding layer of a pretrained NLP model and the rest layers of this model. Figure 3 shows a high level overview of the mask bottleneck layer. Using our information bottleneck layer, we learn two masks: (1) a word level mask  $\mathbf{M}_A$ , (2) an interaction mask  $\mathbf{M}_W$ .
Learning Word-Pair Interaction Importance Mask:  $\mathbf{M}_A$  The interaction mask aims to discover which words globally interact for a prediction task. We treat words as nodes and represent their interactions as edges in an interaction graph. We represent this unknown graph as a matrix  $\mathbf{M}_A = \{\mathbf{M}_{Aij}\}_{V\times V}$ . Each entry  $\mathbf{M}_{A\mathbf{x}_i,\mathbf{x}_j} \in \{0,1\}$  is a binary random variable, such that  $\mathbf{M}_{Aij} \sim Sigmoid(\lambda_{ij})$ , follows Bernoulli distribution with parameter  $Sigmoid(\lambda_{ij})$ .  $\mathbf{M}_{Aij}$  specifies the presence or absence of an interaction between word  $i$  and word  $j$  in the vocabulary  $\mathbb{V}$ . Hence, learning the word interaction graph reduces to learning the parameter matrix  $\lambda = \{\lambda_{ij}\}_{V\times V}$ . In Section 3.3.1, we show how  $\lambda$  (and therefore  $\mathbf{M}_A$ ) is learned through a variational information bottleneck loss formulation (details in Section (A.2)).
Based on the learnt interaction mask  $\mathbf{M}_A$ , each word embedding  $\mathbf{x}_i$  is revised using a graph based summation from its interacting neighbors' embedding  $\mathbf{x}_j, j \in \mathcal{N}(i)$ :
$$
\mathbf {e} _ {i} ^ {\prime} = \mathbf {x} _ {i} + \sigma \left(\frac {1}{| \mathcal {N} (i) |} \sum_ {j \in \mathcal {N} (i)} \mathbf {x} _ {j} \mathbf {W}\right) \tag {8}
$$
$\sigma (\cdot)$  is the ReLU non-linear activation function and  $\mathbf{W}\in \mathbb{R}^{H\times H}$  is a weight matrix. We denote the resulting word representation vector as  $\mathbf{e}_i^\prime$  .Here  $j\in \mathcal{N}(i)$  , and  $\mathcal{N}(i)$  denotes those neighbor nodes of  $\mathbf{x}_i$  on the graph  $\mathbf{M}_A$  and in x. Eq. (8) is motivated by the design of Graph convolutional networks (GCNs) that were introduced to learn useful node representations that encode both node-level features and relationships between connected nodes (Kipf and Welling, 2016). Differently in our work, we need to learn the graph  $\mathbf{M}_A$  , through the  $\lambda$  parameter. We can compute the simultaneous update of all words in input text  $\mathbf{x}$  together by concatenating all  $\mathbf{e}_i^{\prime}$  . This gives us one matrix  $\mathbf{E}'\in \mathbf{M}_{W}^{L\times H}$  where  $L$  is the length of input and  $H$  is the embedding dimension of  $\mathbf{x}_i$
Learning Word Importance Mask:  $\mathbf{M}_W$  This word mask aims to learn a global attribution word mask  $\mathbf{M}_W$ . Aiming for better word selection,  $\mathbf{M}_W$  is designed as a learnable stochastic layer
with  $\mathbf{M}_W\in \{0,1\} ^V$  . Each entry in  $\mathbf{M}_W$  (e.g.,  $\mathbf{M}_{Ww}\in \{0,1\}$  for word  $w$  ) follows a Bernoulli distribution with parameter  $p_w$  . The learning reduces to learning the parameter vector  $\mathbf{p}$
During inference, for an input text  $\mathbf{x}$ , we get a binary vector  $\mathbf{M}_{W\mathbf{x}}$  from  $\mathbf{M}_W$  that is of size  $L$ . Its  $i$ -th entry  $\mathbf{M}_{W\mathbf{x}_i} \in \{0,1\}$  is a binary random variable associated with the word token at the  $i$ -th position.  $\mathbf{M}_{W\mathbf{x}}$  denotes how important each word is in an input text  $\mathbf{x}$ . Then we use the following operation (a masking operation) to generate the final representation of the  $i$ -th word:  $\mathbf{z}_i = \mathbf{M}_{W\mathbf{x}_i} \mathbf{e}'_i$ . We then feed the resulting  $\mathbf{Z}$  to the target model  $f$ .
3.3.1 Learning Word and Interaction Masks for a target model  $f$ :
During training, we fix the parameters of target model  $f$  and only train the WIMASK layer to get two masks.
We learn this trainable layer using the following loss objective, with the derivation of each term explained in the following section:
$$
\begin{array}{l} \mathcal {L} (\mathbf {x}, f (\mathbf {x}), \hat {y}) = \ell_ {f (\mathbf {x}), \hat {y}} + \beta_ {s p a r s e} \ell_ {s p a r s e} \\ + \beta_ {i} \ell p r i o r _ {\mathbf {M} _ {W _ {\mathbf {x}}}} + \beta_ {g} \ell_ {p r i o r _ {\mathbf {M} _ {A _ {\mathbf {x}}}}} (9) \\ \end{array}
$$
First, we want to ensure that model predictions with WIMASK layer added are consistent with the original prediction  $f(\mathbf{x})$ . Hence, we minimize the cross entropy loss  $\ell_{f(\mathbf{x}),\hat{y}}$  between  $f(\mathbf{x})$  and the newly predicted output  $\hat{y}$  (when with the bottleneck layer).
Then  $\ell_{\text{sparse}}$  is the sparsity regularization on  $\mathbf{M}_{A\mathbf{x}}$ ,  $\ell_{\text{prior}}\mathbf{M}_W$  is the KL divergence between  $\mathbf{M}_W$  and a random bernoulli prior. Similarly,  $\ell_{\text{prior}}\mathbf{M}_A$  is the KL divergence between  $\mathbf{M}_A$  and a random bernoulli prior. We provide detailed derivations in Section A.2.
Our experiments are designed to answer the following questions:
1. Will a test set filtered by MNCOVER find more errors from a SOTA NLP model?
2. Does MNCOVER achieve test adequacy faster, i.e. achieve higher coverage in fewer samples?
3. Does MNCOVER help us compare existing testing benchmarks?
4. Can MNCOVER help us automatically select non-redundant samples for better augmentation?
Datasets and Models We use pretrained model BERT-base(Devlin et al., 2019) and RoBERTabase(Liu et al., 2019) provided by (Morris et al., 2020) finetuned on SST-2 dataset and Quora Question Pair (QQP) dataset. For the QQP dataset, we use the model finetuned on the MRPC dataset. We train a word level mask  $(\mathbf{M}_W)$  and an interaction mask  $(\mathbf{M}_A)$  for each of these settings. We use a learning rate of  $1e - 05$ ,  $\beta_{i} = 0.001$ ,  $\beta_{g} = 0.001$  and  $\beta_{s} = 0.001$  for all models.
We have provided the test accuracy of the target models and the models trained with masks in Table 5. Note that the ground truth labels here are the predictions from the target model  $f$  without the WIMASK layer, as our goal is to ensure fidelity of the WIMASK  $+f$  to the target model  $f$ . Table 5 shows that training the WIMASK  $+f$  model maintains the target model's predictions  $f$  as indicated by higher accuracies.
4.1 Experiment 1: Removing Redundant Test Inputs during Model Testing
MotivationCHECKLIST (Ribeiro et al., 2020) provides a method to generate a large number of test cases corresponding to a target template. It introduces different transformations that can be used to generate samples to check for a desired behavior/functionality. For example, to check for a model's behavior w.r.t typos in input texts, it generates examples with typos and queries the target model. CHECKLIST then compares failure rates across models for the generated examples to identify failure modes. However, it does not provide a method to quantify the number of tests that need to be generated or to determine which examples provide the most utility in terms of fault detection. Such a blind generation strategy may suffer from sampling bias and give a false notion of a model's failure modes.
Setup We evaluate MNCOVER's ability to redundant test samples out of the generated tests for a target model. The different transformations used are summarized in Column I of Table 1, Table 2, and Table 6. We measure failure rate on an initial test set of size  $N = 1500$ . We then filter the generated tests based on our MNCOVER coverage criteria: if adding a test example into a test suite does not lead to an increase in its coverage, it is discarded. We then measure failure rate of the new filtered test set.
Results In Table 1, we observe that MNCOVER can select more failure cases for the BERT model