RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
A.1 Attribution Matrix
Note that AD-Drop is naturally suitable for classification tasks since we can obtain one single attribution matrix with respect to the only logit output for each attention map. For token-level tasks (e.g., NER and text generation), as we have several logit outputs to produce the corresponding attribution matrices for each attention map, applying AD-Drop has the challenge of how to fuse these attribution matrices. We provide a simple alternative to calculate the attribution matrix in Eq. ( 5 ) as:

{{\widetilde{\bf{B}}}_{h}=-{\frac{{\partial\mathcal{L}}}{{\partial{\bf{A}}_{h}}}}}, (8)

where \mathcal{L} is the pseudo loss in terms of the pseudo labels. Given a sequence x with n input tokens, we represent each pseudo label as a one-hot vector of C elements and compute \mathcal{L} as:

{\mathcal{L}={\sum_{i=1}^{n}{\mathcal{L}_{i}}=-{\sum_{i=1}^{n}}{\sum_{c=1}^{C}{y_{i,c}}{\rm{log}}{P_{F}}\!\left({{c}|{x,i}}\right)}=-{\sum_{i=1}^{n}}y_{i,\tilde{c}}{\rm{log}}{P_{F}}\!\left({\tilde{c}|{x,i}}\right)}}, (9)

where y_{i,c} is the c -th element in the one-hot vector for token i , {P_{F}}\!\left({{c}|{x,i}}\right) is the softmax output of class c for token i , and \tilde{c} is the pseudo label. Then, Eq. ( 8 ) can be updated as:

{\widetilde{\bf{B}}}_{h}=-{{\frac{{\partial\mathcal{L}}}{{\partial{\bf{A}}_{h}}}}=-{\sum_{i=1}^{n}}{\frac{{\partial\mathcal{L}_{i}}}{{\partial F_{i,{\tilde{{c}}}}\left({{\bf{A}}}\right)}}}\cdot{\frac{{\partial F_{i,{\tilde{{c}}}}\left({{\bf{A}}}\right)}}{{\partial{\bf{A}}_{h}}}}={\sum_{i=1}^{n}}{\left(y_{i,\tilde{c}}-{P_{F}}\!\left({\tilde{c}|{x,i}}\right)\right){\bf{B}}_{i,h}}}

Therefore, we can use Eq. ( 10 ) to compute a single attribution matrix for each attention map when applying AD-Drop in token-level tasks. Besides, as regression tasks (e.g., STS-B) cannot infer pseudo labels, we directly use the actual loss instead.
A.2 Token-Level Experiments
We conduct additional experiments of AD-Drop on NER (CoNLL-2003) and Machine Translation (WMT 2016) tasks. 6 6 6 We follow the official colab implementation ( https://huggingface.co/transformers/v4.7.0/notebooks.html ) for the two tasks. The results on the test sets are reported in Table 8 and Table 9 . Moreover, to verify that AD-Drop can be adapted to other pre-trained models, for CoNLL-2003 NER, we choose ELECTRA as the base model. For WMT 2016, OPUS-MT is chosen. The results show that AD-Drop consistently improves the baselines on both NER and Machine Translation tasks.
C.1 Details of Datasets
The details of the used datasets are introduced as follows. (1) Stanford Sentiment Treebank ( SST-2 ) [ 38 ] is a sentence sentiment prediction task. (2) Multi-Genre Natural Language Inference ( MNLI ) [ 39 ] is a pairwise sentence classification task that aims to predict whether the relationship between two sentences is entailment, contradiction, or neutral. (3) Question Natural Language Inference ( QNLI ) [ 40 ] is a binary sentence classification task that aims to predict whether the sentence in a question-sentence pair contains the correct answer to the question. (4) Quora Question Pairs ( QQP ) [ 41 ] is a binary pairwise sentence classification task that aims to predict whether two questions are semantically equivalent. (5) The Corpus of Linguistic Acceptability ( CoLA ) [ 42 ] aims to predict whether a single English sentence conforms to linguistics. (6) The goal of Semantic Textual Similarity Benchmark ( STS-B ) [ 43 ] is to predict how two given sentences are semantically similar. (7) Microsoft Research Paraphrase Corpus ( MRPC ) [ 37 ] aims to predict if two sentences are semantically equivalent. (8) Recognizing Textual Entailment ( RTE ) [ 44 ] is similar to MNLI but has binary labels. (9) CoNLL-2003 [ 32 ] is to recognize the named entities in a sentence, which contains four types of named entities. (10) WMT 2016 [ 33 ] is a multilingual translation database. In this study, we choose English-Romanian (EN-RO) and Turkish-English (TR-EN) for the experiment. (11) Heuristic Analysis for NLI Systems ( HANS ) [ 34 ] aims to evaluate whether NLI models adopt syntactic heuristics. (12) PAWS-X [ 35 ] is a cross-lingual adversarial dataset for paraphrase identification. HANS and PAWS-X are typically used for the OOD generalization test. The statistics of these datasets are shown in Table 10 .
C.2 Hyperparameter Settings
Table 11 presents the final hyperparameter settings of AD-Drop for BERT/RoBERTa base . The setting with only one value means the parameter is shared by BERT and RoBERTa.
D.1 Ablation of Cross-Tuning
We further report the results of removing cross-tuning in AD-Drop when enumerating p and q in the range of [0.1, 0.9] on the CoLA and MRPC datasets. We observe consistent performance degradation in Figure 9 after removing the cross-tuning strategy from AD-Drop .
D.2 Effect of Data Size on QQP
Figure 10 shows a comparison between AD-Drop and the original fine-tuning (FT) as the size of training examples changes. We observe from the figure that AD-Drop performs consistently better than original FT with different sizes of training data.
Figure 1: Attention maps of vanilla dropout and our AD-Drop . Darker attention positions indicate higher attribution scores, and crossed circles mean dropped attention positions. Red-dotted boxes refer to candidate discard regions with high attribution scores. Unlike vanilla dropout which randomly discards attention positions, AD-Drop focuses on dropping high-attribution positions in candidate discard regions.; Attention maps of vanilla dropout and our AD-Drop . Darker attention positions indicate higher attribution scores, and crossed circles mean dropped attention positions. Red-dotted boxes refer to candidate discard regions with high attribution scores. Unlike vanilla dropout which randomly discards attention positions, AD-Drop focuses on dropping high-attribution positions in candidate discard regions.
Figure 2: Results of training and validation losses when fine-tuning RoBERTa with different dropping strategies on MRPC. The dropping rate is set to 0.3 if it applies.; Results of training and validation losses when fine-tuning RoBERTa with different dropping strategies on MRPC. The dropping rate is set to 0.3 if it applies.
Figure 3: Illustration of AD-Drop in four steps. (1) Conduct the first forward computation to obtain pseudo label \tilde{c} . (2) Generate attribution matrices {\bf{B}} via computing the gradient of logit output {F_{\tilde{c}}}\left({\bf{A}}\right) with respect to each attention head. (3) Sort {\bf{B}} and strategically drop some positions to produce mask matrices {\bf{M}} . (4) Feed {\bf{M}} into the next forward computation to compute the final loss.; Illustration of AD-Drop in four steps. (1) Conduct the first forward computation to obtain pseudo label \tilde{c} . (2) Generate attribution matrices {\bf{B}} via computing the gradient of logit output {F_{\tilde{c}}}\left({\bf{A}}\right) with respect to each attention head. (3) Sort {\bf{B}} and strategically drop some positions to produce mask matrices {\bf{M}} . (4) Feed {\bf{M}} into the next forward computation to compute the final loss.
Figure 4: Results of AD-Drop with and without cross-tuning when enumerating p and q in [0.1, 0.9]. RoBERTa is chosen as the base model. Results show that "with cross-tuning" leads to much lower variance and higher performance.; Results of AD-Drop with and without cross-tuning when enumerating p and q in [0.1, 0.9]. RoBERTa is chosen as the base model. Results show that "with cross-tuning" leads to much lower variance and higher performance.
Figure 5: Results of AD-Drop and FT as the number of training samples changes.; Results of AD-Drop and FT as the number of training samples changes.
(a) BERT on CoLA; (b) RoBERTa on CoLA; (c) BERT on RTE; (d) RoBERTa on RTE; Results of sensitivity study on CoLA and RTE. Rows correspond to p and columns refer to q . Blue blocks indicate the results of AD-Drop below the baseline (FT), and red blocks mean the results of AD-Drop above the baseline. Darker colors mean greater gaps with the baseline.
Figure 7: Performance of fine-tuned RoBERTa on development sets, where two dropping strategies (i.e., drop low-/high-attribution positions) are applied. Gold labels are used for the attribution.; Performance of fine-tuned RoBERTa on development sets, where two dropping strategies (i.e., drop low-/high-attribution positions) are applied. Gold labels are used for the attribution.
Figure 8: Results of dropping self-attention positions in different layers of RoBERTa on SST-2.; Results of dropping self-attention positions in different layers of RoBERTa on SST-2.
(a) BERT on CoLA; (b) RoBERTa on CoLA; (c) BERT on MRPC; (d) RoBERTa on MRPC; Results of AD-Drop with and without cross-tuning when enumerating p and q in the range of [0.1, 0.9] on the CoLA and MRPC datasets.
Figure 10: Results of comparison between AD-Drop and original FT as the size of training data changes on QQP.; Results of comparison between AD-Drop and original FT as the size of training data changes on QQP.
Overall results of fine-tuned models on the GLUE benchmark. The symbol \dagger denotes results directly taken from the original papers. The best average results are shown in bold.
Results of ablation studies, in which r/w means “replace with” and w/o means “without”.
Results of repeated experiments. Each score is the average of five runs with a standard deviation.
Testing AD-Drop on a larger model.
Testing AD-Drop in few-shot settings. RoBERTa with AD-Drop achieves higher performance and lower deviations than that with the original fine-tuning approach.
Testing AD-Drop on OOD datasets.
Results of performance and computational cost of AD-Drop with different masking strategies (GA, IGA, AA, and RD) relative to the original fine-tuning. The symbol \ddagger means AD-Drop is only applied in the first layer. BERT is chosen as the base model.
Test results of AD-Drop on the CoNLL-2003 NER dataset.
Test results of AD-Drop on Translation datasets. The evaluation metric is BLEU.
Statistics of the used datasets.
Hyperparameter settings of AD-Drop for BERT and RoBERTa.