TLM: Token-Level Masking for Transformers
Authors: Yangjun Wu
Kebin Fang
Dongxiang Zhang
Han Wang
Hao Zhang
Gang Chen
Venue: EMNLP 2023 Main (2023)
Keywords: Token-Level
Masking
Transformers
Overfitting
TL;DR: A novel Token-Level Masking training strategy for Transformers to mitigate overfitting

Abstract: Structured dropout approaches, such as attention dropout and DropHead, have been investigated to regularize the multi-head attention mechanism in Transformers. In this paper, we propose a new regularization scheme based on token-level rather than structure-level to reduce overfitting. Specifically, we devise a novel Token-Level Masking (TLM) training strategy for Transformers to regularize the connections of self-attention, which consists of two masking techniques that are effective and easy to implement. The underlying idea is to manipulate the connections between tokens in the multi-head attention via masking, where the networks are forced to exploit partial neighbors’ information to produce a meaningful representation. The generality and effectiveness of TLM are thoroughly evaluated via extensive experiments on 4 diversified NLP tasks across 18 datasets, including natural language understanding benchmark GLUE, ChineseGLUE, Chinese Grammatical Error Correction, and data-to-text generation. The results indicate that TLM can consistently outperform attention dropout and DropHead, e.g., it increases by 0.5 points relative to DropHead with BERT-large on GLUE. Moreover, TLM can establish a new record on the data-to-text benchmark Rotowire (18.93 BLEU). Our code will be publicly available at https://github.com/Young1993/tlm.

TLM: Token-Level Masking for Transformers
Yangjun Wu; Kebin Fang", Dongxiang Zhangl Han Wang, Hao Zhang, Gang Chen Zhejiang University
{yjwu, fkb]@zjuici. comm
{22021066, zhangdongxiang, cg }ezju. edu. cn, hr283@sussex.ac.uk uk
Abstract
Structured dropout approaches, such as atten- tion dropout and DropHead, have been inves- tigated to regularize the multi-head attention mechanism in Transformers. this paper, we propose new regularization scheme based token-level rather than structure-leve to reduce overfitting. Specifically, we devise novel Token-Level Masking (TLM) training strategy for Transformers to regularize the connections self-attentin attention, which consists two masking techniques that are effective and easy to imple- ment. The underlying idea is to manipulate the connections between tokens in the multi-head attention via masking, where the networks are forced to exploit partial neighbors' information produce meaningful representation. The generality and effectiveness of TLM are thor- oughly evaluated via extensive experiments on 4 diversified NLP tasks across 18 datasets, in- cluding natural language understanding bench- mark GLUE, ChineseGLUE, Chinese Gram- matical Error Correction, and data-to-text gen- eration. The results indicate that TLM can consistently outperform attention dropout and DropHead, e.g., it increases by 0.5 points rela- tive to DropHead with BERT-large on GLUU,. Moreover, TLM can establish on the data-to-text benchmark Rotowire (18.93 BLEU), Our code will publicly available at https://gi thub. wtw/ Young 1999/91n
Attention dropout
DropHead
TLM
Figure I: Mussrrtioos of the attention with At- tention dropout, DropHead, and TLM. The row de- notes Query, and the column represents Key. Attention dropout randomly drops some attention weights. Drop- Head directly drops entire attention heads. Regarding TLM, Self-masking (left) denotes that the scores for the masked column are invalid. Siblings-masking (right) means the scores for the row and column are useless except for the masked token itself.
record
ally expensive and insufficient for large language models. This problem could cause the degrada- tion of model generalization. To address this issue, regularization methods, such dropout (Srivas- tava al., 2014) and subsequent research (Wan et al., 2013; Fan et al., 2020; liang et al., 2021) have been developed from a structural perspective, which trained with "thinned" subnetworks. The feature of dropout is that it randomly drops units from the neural networks during training, prevent- ing units from co-adapting too much.
1 Introduction
In recent years, variety of pre-trained language models based the Transformer (Vaswani al., 2017) architecture have been presented, such as BERT, GPT (Brown et al., 2020), and T5 (Raffel et al., 2022). These models push state-of-the-an forward in numerous NLP tasks.
With the rapid growth of model parameters, deep neural networks are highly likely to encounter over- fitting challenges because supervised data is usu-
To further mitigate overfitting for Transformers. structured methods such as DropHead (Zhou et al., 2020) are proposed to drop entire attention heads in the attention mechanism with the purpose of pre-
Equal contribution. Corresponding author.

---

venting a small subset of heads from dominating the whole model. Dropping entire attention heads may result in losing significant amount of feature information. Attention dropout is the application of the dropout technique to the attention mechanism. It arbitrarily drops some attention weights in the matrix calculation of self-attention. However, ex- periments in DropHead and our preliminary trials (shown in Table 1) demonstrate that the difference is not obvious with without attention dropout.
Model
QQP
RTE
BERT -w Attention-dropout
87.0
61.8
BERT -w/o Attention-drooout
86.9
62.0
'able 1: Results of BERT and w/o attention-drop dropout on QQP and RTE. The descriptions of datasets are avail- able section 4.1.
ChineseGLLE,, and Chinese Grammatical Error Correction, to data-to-text generation. The experi- mental results demonstrate that our TLM with the backbones can substantially improve performance. Particularly, our method with BERT-base/BERTT large boosts the score of DropHead from 79.2 to 79.9 and 81.7 to 82.2 on GLUE, and it achieves a new state-of-the-art performance (18.93 BLEU) on data-to-text generation. Further experimental anal- yses demonstrate that TLM is more effective in alleviating overfitting than the baselines.
In this work, we introduce a novel regularization scheme based on token-level instead of structural perspective. This method, Tooeen-Level Masking (TLM), is training technique to regularize the con- nections among tokens during the attention calcu- lation in each layer of Transformer blocks. Specif- ically, considering the example shown in Fig.1 and 2, TLM contains two techniques: 1) Siblings- masking. The first step of this method is that we use random function' to select a percentage of the tokens in the k-th layer, e.g., the masked token ' (the gray block in Fig.2) is excluded from the calculation of attention weights among the sibling tokens but copies itself, and its neighboring tokens considers other siblings in addition 'T' Then, we feed the Reed-Forward Network with the attention output to obtain the new hidden state as the input of the next layer. 2) For Self-masking, we borrow the idea from CBOW (Mikolov et al., 2013) where its attention score is entirely contributed by oth- The difference with Siblings Biblings-masking is the masked token 'I' is forbidden to attend to attention computation. In the training phase, we randomly invoke one of two masking strategies each batch with a 50-50 chance?. In this manner, the networks are forced to utilize partial neighbors' attention information, not the whole (i.e. the connections between the masked tokens and their neighboring tokens are invalid, which are implemented by as- signing large negative number in the matrix of attention weights). This scheme introduces bot- tleneck that the nets should work hard to become robust and produce a meaningful representation.
Our main contributions are summarized as fol- lows:
To reduce overfitting, we present TLM, a novel, simple yet effective training technique to refine the self-attention computation flow the multi-head attention mechanism with- out modifying the structure of Transformer models.
TLM can seamlessly integrate with pre- trained Transformer models without extra cost. The experiments on 18 popular datasets indi- cate that TLM can lead to consistency im- provements compared to the strong basclines.
Further analyses demonstrate that TLM can reduce overfitting and enhance the robustness of the networks.
Related Work
Mask language modeling. In (DRT(n et al., 2019), 15% of input tokens are selected, and 80% of those selected tokens are replaced with the special token [MASK], while 10% re- main unchanged and the remaining 10% are ran- domly replaced. However, this random masking and replacement are only done once during data pre-processing, resulting in mismatch between pre-training and fine-tuning. RoBERTa (Liu et al., 2019) duplicates training data 10 times to address this issue, but this requires more training steps.
To confirm the effectiveness of our approach, we conducted extensive experiments on 18 pop- ular datasets. The tasks range from English nat- ural language understanding benchmark GLUE,
'Bernoulli function: https://pytorch. tttp:///pooorccooo/ooooc//// stable/generated// http://orcc...brreee bernoulli
html higg
'For simplicity, we conduct most of the experiments with 50-50 chance and ablate the proportion in Appendix D.

---

11
1:2
h3
14
h5
Siblings-masking
Transformer Block
pizza
MalMull
SoffMiax
Trameformer Black
Scale
Self-masking
MadMul
Transformer Block
former Block
Multi-head attention
placa
The computing flow of TLM for the sentence 'I ate some pizza.'. Our TLM can be employed on the encoder, necoer, and decoder architecture in the Transformer.
Figure 2:
contrast, our proposed method modifies the atten- tion computation flow in each layer without requir- ing additional data or special token.
Approach
In this section, we first review the self-attention computing workflow of the Transformer and then describe our TLM in more detail.
Attention Mask. Attention Mask in Transform- ers, which is originally used to ignore those in- valuable tokens, called padding tokens, in order ensure that all the sequences are the equal length for each batch; or to prevent the net- works from seeing the tokens' subsequent posi- tion in the auto-regressive decoder, such as UniLM (Dong et al., 2019), OPT(Zhang et al., 2022), and LLaMA(Touvron et al., 2023). In this work, we also employ the attention mask with the purpose of some atten- tion links among tokens during training and keep the same as the standard Transformer in inference.
Multi-head Attention in Transformer
In the research of the Transformer (Vaswani et al.. 2017), the calculation for the vanilla attention is formulated follows:
Attn (an(Q,, K, V) = ssotmaa( S(QK)
V
(1)
demab/ H
S(QK) = MatMul(QK")
(2)
mitigating overfitting, i.e.,
where the queries Q, keys K, and values V are all matrices, where the shapes are Batch size x sequence length x demb- dems- and H denote the dimension of embedding and the number of heads, separately.
Regularization. Common regularization meth- ods include L1 and L2 regularization, dropout, early stopping, and data augmentation. Dropout (Srivastava et al., 2014) is very popular training technique that aims to avoid overfitting by arbitrar- ily dropping some units. LayerDrop (Fan et al., 2020) randomly drops some entire substructures components. Data dropout (Iyyer al., 2015) is performed on the input data level data aug- mentation. Compared the previous methods, our approach aims to carefully control the connections between tokens the multi-head attention mecha- nism.
There are three types of multi-head attention: 1) Self-Attention usually denotes the self-attention in the encoder, which means all of the keys (K), values (V), and queries (Q) come from the same place in self self-attention layer. Thus, every position in the encoder can attend to the attention comput- ing of all positions. 2) Cross-Attention is applied in the encoder-decoder layer, and its feature is that queries (Q) come from the decoder layer, and the memory keys/values come from the output of the encoder. In this manner, each position the de- coder can be present at the attention computing

---

Algorithm TLM Training Procedure
and the max input sequence length, respectively, The weights of masked tokens in M are set the minimum value of the tensor. Then, we can modify the Eq. I and 2 as follows:
I: Initialize model with parameters w.
2: while not converged do
3: for to M layers do
4: Obtain attention weights S(QK) (Eq.2)
S(QK) )
5: Select tokens to mask (Eq.3)
Aitm(Q, K, V) = softmax
(5)
6: Expand Attr_M _ into matrix M (Eq.4)
dembb/ H
7: Calculate (Q, K, V) (Eq.5)
8: Obtain the hidden state h4
S(QK) - S(QK) + M
(6)
9: end for
The attention scores of masked connections among tokens are very large negative numbers by perform- ing Eq.6, so their weights equal 0 after executing softmax. This makes the connections to the masked tokens ineffective in influencing the current masked token.
10: end while
all positions in the input sequence. 3) Masked- Attention the form of auto-regression the de- coder. The self-attention layers allow each position in the decoder to attend to all places in the decoder up to the current position.
Next, we feed the Feod-Forward Network with Altn(Q, K, V) to obtain the hidden state hy. We recursively invoke the identical operation until all the layers have been traversed and yield the final output tensors.
3.2 TLM
The core computing procedure of TLM is shown in 2 2 and Algorithm (1). In nutshell, we mod- ify the computing of self-attention by adding novel step to control the connections between to- kens, which in turn influences the attention scores and contextual semantic representation. We uti- lize TLM in the training phase while keeping the attention calculation as the vanilla during testing.
Experiments
To verify the generality and effectiveness of our pro- posed TLM, we perform extensive experiments wide variety of tasks with 18 benchmark datasets, including the English natural language understand- ing benchmark GLUE (10 datasets), CLi (6 datasets), Chinese Grammatical Error Correction (1 dataset), and data-to-text generation (1 dataset). Note that our method can be utilized both in the pre-training and fine-tuning, but we only estimate TLM during the fine-tuning in this work due limited computational resources. In the following, present the key findings, and more details can be found in the Appendix.
In the training phase, we first compute attention weights S(QK)) by performing 2q.2, and S(QK) denotes the similarity between the queries Q and keys K for the input tokens. Then, the random function Bernoulli is executed select fixed rate of masked tokens the k-th layer of Transformer blocks at each batch.
Attn_M Bernoulli(Athe, R)
(3)
Where Attn_M refers to the vector of attention- mask? 3. The tokens selected as masked tokens will be stored in memory with the attention mask value of O. When the rate R is set to 0.1. which denotes 10% of tokens would be masked.
4.1 English Language Understanding
Dataset. GLUE benchmark is collection of di- verse natural language understanding tasks intro- duced by (Wang et al.. 2018), GLUE consists of three types of tasks: single-sentence, similarity and paraphrase, and inference tasks. Single-sentence tasks require models to predict the grammaticality or sentiment of a given sentence. Similarity and paraphrase tasks involve determining the degree of semantic equivalence between sentence pairs. Inference tasks aim to capture the entailment rela- tionship between sentences.
To fit the identical dimension the attention weight S(QK), we expand the attention mask vec- tor Attn .M into the matrix M:
M Extend (Atti_M)
(4)
Here, M BBxHXNxN B, H, and N refer batch size, the number of self-attention heads,
'Attention-mask, is abbreviated as Attn. M, Attn__M [1, I, ...0). The values equal denoting the input tokens, or when it belongs to padding token or masked token.
Model and Training. For a fair comparison, we choose BERT as the backbone and train it

---

Model
MRLASST:2 STS- MQPMMOMTIIN ONT WNLIAVV STD
BERT-small
w/o Att-dropout
27.5
89.3
83.2
78.9 86.9
77.5
76.8
62.2
62.1 73.02.8c4
+Att-dropout
27.8
89.7
83.4
79.2 87.0
77.6
77.0
61.4
62.3 73.24.3e-2
+DropHead
31.7
89.6
83.2
80.3
87.2
77.7
77.2
87.3 62.5
63.0 74.1 1.1e3
+TLM
35.3
90.8
83.5
81.0 87.8
78.4
77.8
87.5
64.4 75.02.8e-3
BERT-base
w/o Att-dropout
51.0
92.3
88.2
84.2 87.7
83.5
83.2
90.3 63.0
63.0 78.63.1e-3
+Alt-dropout
51.9
92.8
87.3
84.4
88.0
84.0
83.4
90.4 62.4
63.0 78,8 1.le-3
+DropHead
52.0
93.4
87.8
84.5
87.5
83.6
83.1
90.4 65.2
64.4 79.2 1.8c-3
+TLM
53.7
93.3
87.9
84.5 88.6
84.3
83.6
67.5
65.1 79.96.8e.
BERT-large
w/o Att-dropout
59.7
93.9
88.0
86.1 88.7
86.5
85.6
69.7
63.7
81.42.6e-
+Att-dropout
59.8
94.3
87.9
86.5 88.9
86.6
85.7
92.7 69.6
63.7
81.67.9e-4
+DropHead
60.1
94.1
88.1
85.9
89.2
86.7
85.8
92.6
70.1
64.4 81.7 6.5e-3
+TLM
61.0
94.2
88.6
86.5
89.3
86.7
86.1
92.8 70.8
66.4 82.2 4.7e-4
Table 2: Fine-tuned BERT-small, BERT-base, and BERT-large performances on English natural language under- standing benchmark GLUE, Each method is tuming with different random seeds. The AVG denotes the average results and STD is the standard deviation of results. The highest numbers in bold.
Model
AFQMC TNEWS1.1
IFLYTEK CMNLI CLUEWSC CSL
AVG
STD
BERT -w/o Att-dropout
73.6
56.7
60.2
79.4
62.2
80.2 68.6
2.2e-2
BERT+Att-dropout
73.7
56.6
60.3
79.7
62.1
80.4
68.8 1.le-
BERTeDropHead
73.6
57.0
60.6
79.0
71.4 80.5
70.4
5.2e-4
BERT+TLM
73.8
58.2
61.5
79.3
73.4
81.4 71.3 1.1e-3
Table 3: Fine-tuned BERT-base performances on Chinese language understanding benchmark CLUE. The AVG denotes the average results and STD is the standard deviation of 3 results.
using BERT-small, BERT-base, and BERT-large to explore the effect on model size. The exper- iments include BERT without iitention-dropour (Att-dropout), with att-dropouu//ropHeaa///I the rate of 10%/20% 20% for each task. We then submit all the files of prediction at 3 different ran- dom seeds the official website GLUE* and ob- tain the scores of the test set. As for the hyper- parameters, the learning rate, dropout, and batch size are uniformly set to 2e-5, 0,1, and for all the tasks. All the models are trained and evaluated on 24G Nvidia RTX3090.
our method shows a more significant improve- ment (0.9/0.7/0.5 points) while scaling model size, which provides evidence that our method is effective in improving the understanding of natural language.
at
Regarding the sentence-level classification, CoLA, which only contains 8,511 sentences in the training set, our method demonstrates significant improvement from 27.5 to 35.3, 51.0 to 53.7, and 59.7 to 61.7 as we scale the sizes from small, base to large. This finding is coherent with the design principle of TLM for mitigating overfitting. Thus, our method can achieve much larger performance gains than Att-dropout and DropHead as applied to small-scale supervised data. When scaling the size of datasets such as MNLI-m, which contains 392k/9k sentence pairs on training/test sets, our ap- proach still outperforms the DropHead by both 0.7 points at the sizes of BERT-small and BERT-basc. On similarity and paraphrase tasks, such as
Analysis. We present the specific results in Ta- ble 2. Specifically, we can find that there is only slight increase in the AVG (0.2 points) with Att-dropout at small/base/large sizes, and both TLM and DropHead are well ahead of Att- dropout by large margin. Compared DropHead,
hhttps://gluebenchmartt

---

QQP. our method can upgrade the performance by 0.6/1. points (87.2 + 87.8 and 87.5 + 88.6) compared to DropHead at the sizes of small and base, and this provides evidence that randomly dropping entire heads benefits less than carefully token masking for the representations of sentence- pairs. When scaling the size to BERT-large, the improvement is not obvious like in BERT-base, speculate that 89.3 is approaching 91.1 (the perfor- mance of SOTA) on the GLUE leaderboard, where the enhancement is limited only through regular- ization methods.
Concerning the classification tasks, our approach can significantly outperform DropHead on short text TNEWS1.1 by 1.2% and long text IFLTTEK by 0.9%. The possible explanation of promotions is that Siblings-masking and Self-masking introduce bottleneck, where the networks can only utilize partial neighbors' attention information. Thus, the networks should work hard to become robust and this scheme benefits more than DropHead. We also notice that adding regularization methods results in performance degradation on CMNLI, and we leave this for future work to investigate the reasons.
As to WNLI, this inference task requires the model to fully understand the contextual informa- tion provided by words phrases in sentence. The experimental results indicate that TLM, by carefully controlling masking, can bring more benefits than DropHead and attention dropout.
4.3 Chinese Grammatical Error Correction
Dataset. Chinese Grammatical Error Correction (CGEC) is a task that automatically detects and corrects the grammatical errors in the input tence without revising the meaning of the original sentence as much as possible. For this task, we evaluate our proposed method on the benchmark dataset CGED? introduced by (Rao et al., 2020). The set contains 18,031/3,11 utterances for train- ing/validation with the average length of sentences 46.24 24/44455. For the test sets, CGED2021 and CGED2020 comprise 2,294 and 1,457 utterances.
4.2 Chinese Language Understanding
Dataset. CLUE, introduced by (Xu al., 2020), is a widely used benchmark for Chinese Language Understanding Evaluation. Specifically, TNEWSI.1 task classifies short news titles into 15 categories. As to the IFLTTEK task, which involves assign- ing a label from a total of 119 categories to app descriptions. Other tasks include CLUEWSC2020 (CLUEWSC), which determines co-reference be- tween pronouns and nouns in sentence. AFQMC aims judge the semantic similarity of sentence pairs. CSL. a keyword recognition task. CMNLI determines the entailment relationship between sen- tence pairs.
Model and Training. To demonstrate the effec- tiveness as possible, we choose the strong pre- trained model Chinese Bart as our backbone and fine-tune it with TLM, DropHead, and Att-dropout at the rate of 10%. We also compare results with the top peppperforming baselines GECToR (Omelianchuk et al., 2020) with BERT, RoBERTa, and ELEC- TRA. The learning rate, batch size, and epoch are set to 3e-5, 32, and 150, respectively.
Model and Training. We choose the Chinese BERT-base from huggingface as our backbone and train it with/without Attention dropout (Att- dropout), with TLM and DropHead. We submit the results with 3 different random seeds to CLUE leaderboard and obtain the scores. We evaluate our model with the masking rate of 5% and Drop- head at the rate of 10%. Other hyper-paramesn are the same as the original BERT.
Model GECToR-BELT GECTOR-ROBERS
Score
32.8
33.5
GECTOR- -LECTRR
32.7
MAN (Fan et al., 2021)
41.3
Bart-base -w/o Att-dropout
42.0
Bart-base + Att- Att-dropout
42.3
Analysis. The overall results with TLM, Drop- Head, Att-dropout, and without Att-dropout are reported in table 3. First, both DropHead and our TLM can notably boost the AVG compared to with/without Attention dropout, which verifies the effectiveness of these two regularization methods.
Bart-base + DropHead
42.7
Bart-base TLM
43.7
Table 4: The overall results on CGED2021.
httpss///gitubb.cmm/bluuceeeeee hittps://huggingaaaat co/ co/fnlp/
http://gggiiiaaaacaceeeeeeeeteee hte/bert-base-chiiee 'https: http://wwwwclceeeee.e wetebennhharksscoo.
bart-based tet--base-chieeee chinese

---

Analysis. The definition of metrics from the Chinese Learner Text Correction', and we present the results Table obtained by the official scripts. First, we can observe that our TLM outper- forms GECTOOR-BERTT/OBEETTTALLLL
Model
BLEU
ENT (Puduppully et al., 2019)
16.12
DUV (Gong et al., 2020)
15.92
FierarchicalEneoder (Li al., 2021)
17.96
by 10.9/10.2/11.0 points. In contrast with MAN (A more detailed comparison can be found in Ap- pendix C), our approach leads to an improvement 2.4 points without requiring extra model param- eters like MAN. The scale of the (28k sen- tences) is relatively small, while the model size Bart (over 138M) is quite large, which may cas- ily cause overfitting. Under this circumstance, our method improves by 1.0 and 1.4 points compared to DropHead and attention -rropout.. We speculate that DropHead drops entire heads of attention and may lose some syntactic information. Our token- level masking has the advantage of detecting syntax errors and correcting the errors because the absence of some tokens will strengthen the model's sensi- tivity to syntactic information.
T5 -w/o Att-dropout
18.00
T5 + Att-dropout
17.98
T5 + DropHead
18.01
T5 TLM
18.93
Table BLEU results on Rotowire.
ALL Al--dropout DropHead
82-
80
78
76
24
4.4 Data-to-Text Generation
Figure 3: The comparison among Attention Dropout, DropHead, and TLMM on STS-B.
Dataset. Data-to-text generation is significant challenge that aims automatically yield de- scription represent the valuable key information in the structured data. The benchmark dataset is ROTOWIRE introduced by (Wiseman et al., 2017), whose output is the summary of NBA baskethall games as the input is corresponding records that represent the performance of their teams and play- ers. The summaries are professionally written and relatively well structured with average genera- tion length of 337.1 words per example. Following the research 10, the has been split into training, validation, and test sets consisting of 3,398/727/728 summaries, respectively.
coder proposes Number Ranking and Importance Ranking as two auxiliary tasks capture the indi- vidual relations between the records. Our approach increases by 0.97 points relative HierarchicalEnn coder and achieves new SOTA. Meanwhile, TLM extremely easy to train with T5 an end-to-end manner as well as no extra modules or tasks are required. It can also increase by 0.92 points in contrast to DropHead, the advantage for our TLM is the masking scheme can encourage models to capture complex relations among the records and select the salient information in the table. However, dropping entire heads for DropHead may cause the nets are not sensitive in capturing complicated feature relationships.
Model and Training. In this task, take the encoder-decoder model "5-small as our back- bone and compare it with SOTA models. We train T5 with TLM, DropHead, and Att-dropout at the we set
rate of 10%. As to the beam search as 8, learning rate as 5e-4, and max text length as 512. To penalize repetition, the repe- tition penalty is set to 2.
4.5 Ablation study
Although the experimental results are superior, the effectiveness of our TLM has not been thoroughly investigated. Thus, we conduct further studies to gain better understanding of our approach. As to the selection of datasets, the STS-B and CoLA are sentence-level datasets while IFLYTEK and CSL are long-text datasets, and CGED2020/2021 are grammatical error correction datasets. We hope the tasks can cover different lengths of text meanwhile
Analysis. We report the results with BLEU in table 5. The current SOTA model HierarchicalEn-
rree/mmainnnnetiisstraakkkk
10 "htttps://ithhbb htm/hharaarrnpppoooocreeeee "https:///ugginn ace. co/t5-small

---

the diversity can be guaranteed.
Effect training data ratio. We further inves- tigated the impact of different training data ratios by training BERT-base+ of supervised data on CSL, and the results are pre- sented in Fig.4. In contrast to DropHead, TLM can achieve comparable performances even when trained with only 50% of data. Moreover, our method outperforms DropHead with 65% of data CSL. The improvements may be attributed to the token-level masking, this strategy encourages the model to capture meaningful context represen- tation with long input sentences. Therefore, our TLM can benefit the robustness of the networks and reduce the dependence on supervised data.
TLM Attention Dropou/DrropHeadd To analyze the relationship among TLM, atten- tion dropout (Att-dropout), and DropHead ap- plied to self-attention layers in Transform- ers, wc first train BERT-small only with A.MiAtt-dropouu/Droppaaal in the [0.05, 0.10, 0.15, 0.20J range to explore their influences on performance. The results are presented in Fig.3. We keep all the parameters the same except for the rates. The finding is that both TLM and DropHead are well ahead of Att-dropout by large margin on STS-B, and our method is more stable than Drop- Head.
using 50%/65% %/80%
Method
STS-B
CGED2020
Model
BERT w/o regularization
78.7
ER DR
DET-FI COR-FI SCORE
+ dropout
78.9
Bart
80.5
19.3
37.8
+ dropout + Att-dropout
79.2
TLM
20
20
78.9
19.3
37.1
+ dropout + DropHead
80.3
TLM
15
15
81.4
19.5
38.3
dropout + TLM
81.0
TLM
15
10
82.1
20.1
39.1
dropout + DropHead Att-dropout
80.4
TLM
10
15
82.4
20.2
39.2
+ dropout + DropHead TLM
80.0
TLM
10
10
82.1
20.3
38.9
+ dropout + TLM Att-dropout
80.7
Table 7: The results CGEC test sets of CGED2020. The En and Dn refer to the rate masking in the en- coder and decoder. DET-FI means detection. and COR-F1 denotes the F1 score for correc- tion. The complete results can be found in Appendix B.
+ All
79.9
Table The effect of different regularization combina- tions. All the rates equal 0.1.
Fl score for
Second, we test the effect of different combina- tions the self-attention models. As shown in Table 6, we observe that adding any type of regular- ization method can improve the performance the vanilla model, and our TLM outperforms attention dropout and DropHead under the same conditions by a large margin. When combined together, we find that the performance is not optimal, especially when all regularization methods used together. This is mainly due the fact that excessive use of regularization may cause training instability.
Effect of TLM rate. We conduct further testing on the impact of varying the rate masking in the encoder and decoder of Bart- base, ranging from 10% to 20%. As outlined in Table 7, the results of SCORE are better than the baseline, except for the rate of 20%. A possible explanation for why our TLM underperforms BERT the rate of 20% is that an excessive amount of masking may confuse the networks and decrease their ability to compre- hend syntax information. It's important to note that a too-large rate should be cautious. Overall, our op- timal option is the group of (10% 15%) for encoder and decoder, and it outperforms the strong base- line by 1.4 points (37.8 39.2). This promotion demonstrates that TLM enhance the under- standing of grammatical information and guide the model toward correcting grammatical errors.
TLM
Drapi lead
6
CSL
We also analyze the influence of our masking tech- niques, and the results at the size of BERT-small are reported Table 8. It can be observed that the
Effect of Bobbings-masking/Selle
Figure 4: Results with different training data ratios on the test set of CSL.

---

results decrease by 2.6% and 0.6% on CoLA and STS-B when removing the Boblings-masking tech- nique. Similarly, the results without Self-masking decrease by 3.0% and 1.2%, and there is a drop 7.6% and 2.2% without both techniques. These findings highlight the importance of both Siblings- masking and Self-masking methods.
such as VIT (Dosovitskiy et al., 2020) and UniSpeech (Wang et al., 2021), we leave it the future work for further validation. Mean- while, we haven't yet estimated the perfor- mance by combining TLM with extremely large language models, such as T5-11B and LLaMA.
Model
CoLA
STS-B
3.
Due to the limitation of computational re- sources. merely fine-tuned the pre-trained models with TLM in this work. The effective- ness of TLM applied the pre-training phase needs to be further validated.
TLM
35.3
81.0
-w/ Siblings-masking
32.7
80.5
-w/ Self-masking
32.3
79.9
-w/o Both
27.7
78.9
Table 8: The effect of our masking techniques.
4. In contrast with dropout, TLM can only apply to Transformer-base networks, not all the neural networks, such as CNN or LSTM.
Conclusion
5. Despite numerous ablation studies being per- formed, the explanation of TLM's optimiza- tion the self-attention mechanism remains insufficient, especially in terms of the effect on attention distribution. Further exploration is needed.
In this paper, we propose simple training strategy, Token-Level Masking (called TLM) to reformulate the computation flow of multi-head self-attention for reducing overfitting. During training, we ran- domly invoke one the two masking techniques: I) Siblings-masking, where the masked token is forbidden to interact with its siblings when com- puting attention weights, and 2) self-masking. the attention weights for the masked token is solely re- liant others. This regularization scheme enables the networks to work hard become robust and acquire meaningful information.
Acknowledgements
The authors gratefully acknowledge Yao Zhao, Min Liang, Mengqi Zhang, Xiangyu Jin, Fengli Shi, Shanhoo Luo, and Fang Shu for giving valu- able suggestions on this study. Our thanks also go to all the anonymous reviewers for their posi- tive feedback. The work is supported by the Na- tional Key Research and Development Project of China (2022YFF09020000, In addition, we thank HANGZHOU YIYOULIAO TECHNOLOGY CO LTD https://www.yyyolllii iao. com/ for provid- ing computing resources.
To verify the effectiveness of our proposed method, we conducted various experiments with 18 benchmark datasets. The results demonstrate that TLM can consistently improve the performances compared to the strong baselines and even achieve SOTA on data-to-text generation. Through further analysis, we observe that our TLM is more stable than DropHead or attention dropout. Mcanwhile, can seamlessly integrate with pre-trained models.
References
Limitations
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Kerbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. CoRR, 1bs22005.14165.
Here, we list several of what we consider to be limitations:
1. The rate of our masking is a syper-parammter that needs to be tuned, as the experiments shown in Table 7, the performance may derperform when the rate is set too large (e.g.. over 20%).
2. We argue that TLM can also be applied to vi- sion or speech Transformer-bart based networks,
Jacob Devlin, Ming-Wor Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of

---

deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Pupers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
2021. R-drop: Regularized dropout for neural net- works. In Advances in Neural Information Process- ing Systems, volume 34, pages 10890-10905. Curran Associates, Inc.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy. Mike Lewis. Luke Zettlemoyer and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692
Li Dong, Nan Yang, Wenhui Wang. Furu Wei, Xi- aodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Hon. 2019. Unified language model pre-training for natural language understanding and generation. CoRR, abs/1905.33199.
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representa- tions in vector space. In 1st International Conference Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Pro- ceedings.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer. Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2020. An image is worth 16x 16 words: Transformers for image recognition at scale. CoRR, abs/2010. 1929.
Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020. GECToR grammatical error correction: Tag. not rewrite. In Proceedings of the Fiffeenthh Workshop on Innovative Use of NLP for Building Educational Applications, pages 163-170, Seattle, WA, USA + Online. Association for Computational Linguistics.
Angela Fan, Edouard Grave, and Armand Joulin. 2020. Reducing transformer depth on demand with struc- tured dropout. In International Conference on Learn- ing Representations.
Ratish Puduppully, Li Dong, and Mirella Lapata. 2019. Data-to-text generation with entity modeling. In Pro- ceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 2023- 2035, Florence, Italy. Association for Computational Linguistics.
Zhihao Fan, Yeyun Gong, Dayiheng Liu, Zhongyu Wei, Siyuan Wang. Jian Jiao, Nan Duan, Ruofei Zhang, and Xuanjing Huang. 2021. Mask attention networks: Rethinking and strengthen transformer. In Proceed- ings of the 2021 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1692-1701, Online. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2022. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21(1).
Heng Gong, Wei Bi, Xiaocheng Feng, Bing Qin, Xiao- jiang Liu, and Ting Liu. 2020. Enhancing content planning for table-to-text generation with data under- standing and verification. In Findings of the Associ- ation for Computational Linguisticx: EMNLP 2020, pages 2905-2914 14, Online. Association for Computa- tional Linguisties.
Gaoqi Rao, Erhong Yang. and Baolin Zhang. 2020. Overview of NLPTEA-2002 shared task for Chinese grammatical error diagnosis. In Proceedings of the 6th Workshop on Natural Language Processing Tech- niques for Educational Applications, pages 25-35, Suzhou, China. Association for Computational Lin- guistics.
Mohit Iyyer. Varun Manjunatha, Jordan Boyd-Graber, Graber, and Hal Daumé III. 2015. Deep unordered compo- sition rivals syntactic methods for text classification. In Proceedings of the 53rd Annual Meeting of the As- sociation for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1681- 1691, Beijing, China. Association for Computational Linguistics.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky. Ilya Sutskever. and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural net- works from overfitting. J. Mach. Learn. Res., 1999)
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023, Llama: Open and efficient foundation language models.
Liang Li. Can Ma. Yinliang Yue, and Dayong Hu. 2021. Improving encoder by auxiliary supervision tasks for table-to-text generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Confer- ence Natural Language Processing (Volume 1: Long Papers). pages 5979-5989, Online. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30, Curran Associates, Inc.
xiaobo liang, Lijun Wa, Juntao Li, Yue Wang, Qi Meng, Tao Qin, Wei Chen, Min Zhang, and Tie-Yan Liu.

---

Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. 2013. Regularization of neural net- works using dropconnect. In Proceedings of the 30th International Conference on Machine Learning, Pro- ceedings of Machine Learning Research, pages 1058- 1066, Atlanta, Georgia, USA. PMLR.
CGED2021
Model
Detection Fl Correction-F1 SCORE
MAN
77.6
30.9
41.3
TLM
77.8
30.7
43.7
CGED2020
Model Detection-Fil Correction-IFl SCORE
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: multi-task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNI.P: Analyzing and Interpreting Neural Networks for NLP, pages 353 353-355, Brussels. Belgium. Association for Com- putational Linguistics.
MAN
80.5
19.8
37.2
TLM
82.1
20.3
38.9
The comparisons between MAN and TLM (10%) on CGEC test sets of CGED2021 and CGED2021.
Table
Chengyi Wang. Yu Wu, Yao Qian, Kenichi Kumatani, Shujie Liu, Furu Wei, Michael 7eng, and Xuedong Huang. 2021. Unispeech: Unified speech represen- tation learning with labeled and unlabeled data. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 10937-10947, PMIRR
Experimental settings on CLUE. As described in Section 4.2, change the max epochs and mask rate during training for better scores, and the settings are listed Table 10.
B Appendix
Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017. Challenges in data-to-document generation. In Proceedings of the 2017 Conference Empiri- cal Methods in Natural Language Processing, pages 2253-2263, Copenhagen, Denmark. Association for Computational Linguistics.
The detailed results on Chinese Grammatical Error Correction. All the detailed results are listed in Table 11 compared with top-performers. In contrast to encoder- based Transformer models e.g., GECTOR-BERE proposed method can outperform them by large margin in all the metries. For example. the mask- ing rate of (10%, 10%) has much higher Detec- tion, and Correction F1 scores, which demonstrates TLM has the advantage of accurately locating gram- matical errors and successfully correcting these er- rors. For the strong baseline Bart-base, the large improvement lies in Correction Fl (from 28.2 to 30.7). C Appendix
our
Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang. Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang. He Zhou, Shaoweihua Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang. Zhengliang Yang, Kyle Richardson, and Zhenzhong Lan. 2020. CLUE: A Chinese language understanding evalua- tion benchmark. In Proceedings of the 28th Inter- national Conference on Computational Linguistics, pages 4762-4772, Barcelona, Spain (Online). Inter- national Committee Computational Linguistics.
Susan Zhang, Stephen Roller, Naman Goyal. Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mi- haylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre- trained transformer language models.
In contrast with MAN. Mask Attention Net- works (MAN) introduces a new layer to capture the relationship between the Self-Attention Net- work and Feed-Forward Network, causing the model parameters to substantially increase and hindering its wide application with pre-trained language models. TLM doesn't have these is- sues, which simply modifies the calculation work- flow of self-attention to enhance the contextual information without extra modules. We run the code from htttps://githutt com/LibertFan/ MAN/ htt///ree/mmaaaa modifications, except for using sentencepiece! the tokenizer for Grammatical Error Correction
Wangchunshu Zhou, Tao Ge, Furu Wei, Ming Zhou, and Ke Xu. 2020. Scheduled DropHead: A regu- larization method for transformer models. In Find- ings of the Association for Computational Linguistics: EMNLP 2020, pages 1971-1980 Online. Association for Computational Linguistics.
/aaee/maiinn // summarzzaaiin and make no
A Appendix
We provide more details task descriptions and statistics in Fig.5.
Task descriptions on GLUE.
Ptt / thub. www.///oolleeeeeeeeeeeeiiiieeeie

---

Corpus
[Trin]
[Tss
Task
Metrics
Domain
Single-Sentence Tasks
CoLA
8.5k
1k
acceptability
Marthews
misc.
SST-2
67k
1.8k
sentiment
acc.
movie reviews
Similarity and Paraphrase Tasks
MRPC
3.7k
1.7k
paraphrasc
acc./F1
news
STS-B
7k
1.4k
sentence similarity
Pearson/Spearman corr
misc.
QQP
364k
391k
paraphrase
acc./F1
social QA questions
Inference Tasks
MNLI
393k
20k
NLI
matched acc./mismatched acc.
misc.
QNLI
105k
5,4k
QANLI
acc.
Wikipedia
RTE
2.5k
3k
NLI
acc.
news, Wikipedia fiction books
WNLI
634
146
coreference/NLI
Figure 5: Task descriptions and statistics of GLUE.
Parameters
AFOMCTNEWSIFLYTEK
Learning Rate
2e-5
2e-5
2e-5
3e-5
le-5 le-5
Max Epoch
6
9
9
50
9
Batch Size
16
16
16
16
16
8
Dropout
0.1
0.1
0.1
0.1
0.1
0.1
Mask Rate
0.2
0.1
0.15
0.1
0.1
0.1
Table 10: The settings of our TLM on Chinese Language Understanding benchmark CLUE.
task, and the results are listed in Table 9. Clearly, TLM can substantially increase by 1.7 points com- pared to MAN on CGED2020 (37.2 + 38.9) and 2.4 points on CGED2021 (41.3 ~> 43.7), which verifies the design of our TLM that masking some connections in the attention calculation can steer the networks to become powerful.
D Appendix
The proportion of Siblings-maskingSSae masking. To further investigate the contributions of different proportions of Siblings masking and Self-masking, we perform experiments in the group with [0/100%, 25%/75% 50%/50% 75%/25%, 100%/0) rates range in [5%, 10%, 15%] for Siblings-masking and on RTE. The results are reported in Table 12, we note that the experiments with only RTE are inadequate, thus we would perform further experiments and report the results on our GitHub com/Young1 (oouungg993//tlnn 993/tlm.
Siblings-masking
As shown in table 12, 50-50 chance is the relatively better choice compared to 0/100% 100%/0, and we recommend using this default setting for simplicity.

---

CGED2021
Model
ENC-R DEC-R FPR
Detection
Identification
Position
Correction
Score
GECToR BERT
GECTOR-BERT
31.9
74.5
46.3
27.5
14.6
32.8
DECTOR-ROBERTE
30.2
74.3
46.8
27.8
15.3
33.5
GECTOR-ELECTA
29.5
73.1
45.7
27.6
14.0
32.7
Bart-base
22.3
77.4
54.3
31.8
28.2
42.3
Bart-base + DropHead
10
22.7
76.7
54.5
33.0
29.2
42.7
Bart-base + TLM
20
20
21.3
76.0
53.6
30.9
28.1
41.8
Bart-base + TLM
15
15
22.7
77.3
55.7
33.4
29.9
43.4
Bart-base + TLM
15
10
23.6
77.7
55.9
34.0
30.4
43.6
Bast-base TLM
10
15
22.9
77.7
55.4
33.6
30.0
43.4
Bart-base TLM
10
10
23.4
77.8
55.9
30.7
30.7
43.7
Table 11: The detailed results on Chinese Grammatical Error Correction benchmark test set of CGED2021. The ENC- R and EDC-R refer to the rate of masking in the encoder (ENC) and decoder eDEC),..g.. 20 denotes masking 20% to- kens. FPR, Detection, Identification, Position, and Correction denote false positive rate, detection-Fl, identification- F1, position-F1, and correction-Fl, respectively. We collected the results of from https:://// //gi http:///wwaaaaaaaaaaa//////////a
Rate
0/100%
25%/75%
50%/50%
75%/25%
100%/0
5%
65.6
66.2
67.5
66.1
66.9
10%
66.9
66.5
67.1
66.3
65.3
15%
64.1
63.9
64.3
64.2
63.8
Table 12: Different proportions of mibkings-masking and Self-masking on RTE,


Reviews:
[{"review_id": "jCFUUg6T1N", "reviewer": "Reviewer_e4DL", "summary": "", "questions": "1. Does the token-level masking happens at fine-tuning or pre-training for all experiments? I assume TLM happens at the fine-tuning stage for all experiments. It would be better to explicitly say that in the experimental section. How about applying TLM at the pre-training stage? \n2. Since some of the improvements are marginal, how many trials have the authors run for each experiment? Including mean and standard deviation in the paper would be helpful.\n3. Why specifically have Siblings-masking and Self-masking at 50-50 chance?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we deeply appreciate all your constructive comments and advice. \n\nFor the question in “Reasons To Reject:”:\n- ” The motivation and novelty”, our motivation is introducing a new regularization scheme to manipulate the connections between tokens to further mitigate overfitting for Transformers. To the best of our knowledge, the idea of our regularization by masking tokens in multi-head attention is novel.\n- I guess the question of “How much difference compared to masking at the input level?” is to act on the embedding layer, our method is only applied to the attention layer thus it cannot be utilized in the embedding layer.\n- Our approach can be used both in fine-tuning and pre-training. We only estimated the models in the fine-tuning phase due to the limitation of computational resources (We declared this in the Limitation). We are so sorry for the unclear and inapparent declaration.\n- \"The improvements are mostly shown for small or base- …...” is our gross negligence, and we will report the results of the large model in the revised manuscript.\n\nFor the question in “Questions For The Authors:”:\n- 1.\tWe use TLM at the fine-tuning stage for all experiments, and it’s our overlook to only state in the Limitation (Line 543). We would test it at the pre-training stage for further validation. \n- 2.\tFor GLUE and CLUE, we only run once and obtain the results from the official leaderboard website. Other experiments are running 2 times and the best of them are reported. To ease your concern, we reported the mean results and standard deviation on GLUE with small- and base- sizes in the official comments. The revised manuscript would cover the large size due to the overly tight schedule. \n- 3.\tFor the rate of Siblings-masking and Self-masking, it’s our overlook to simply use the group of 50-50 chance, we would test other rates for ablations and report the results in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Simple attention regularization methods which can be easily integrated into the training of any LLMs. \n2. Comprehensive experiments and ablation study across different model architectures (encoder-only, encoder-decoder) and different datasets (English/Chinese) to show the TLM improves the downstream task performance.\n3. This regularization method enhances training efficiency since it requires less amount of training data as analyzed in section 4.5.", "reasons_to_reject": "1. The motivation and novelty are unclear to me and a more detailed explanation is needed. Are there any intuitions of this token-level masking improves performance? How much difference compared to masking at the input level? Is it designed for fine-tuning or pre-training?\n2. The improvements are mostly shown for small or base-size models given the experiments in the paper. It is not clear whether TLM is still useful for larger or more powerful models.", "questions_to_authors": "1. Does the token-level masking happens at fine-tuning or pre-training for all experiments? I assume TLM happens at the fine-tuning stage for all experiments. It would be better to explicitly say that in the experimental section. How about applying TLM at the pre-training stage? \n2. Since some of the improvements are marginal, how many trials have the authors run for each experiment? Including mean and standard deviation in the paper would be helpful.\n3. Why specifically have Siblings-masking and Self-masking at 50-50 chance?", "paper_topic_and_main_contributions": "The paper proposes a token-level masking (TLM) scheme as a regularization method at the training stage to mitigate the overfitting problem. TLM selects random tokens for each layer in the transformer. For the selected tokens, it has two choices for computing attention score: 1) sibling masking, where neighboring tokens contribute to attention scores independently of the selected tokens, and 2) self-masking, where attention scores are calculated without accounting for their own impacts. It shows that training with TLM can improve the performance of various downstream tasks compared to vanilla training. \n\nContributions:  a new regularization technique, NLP engineering experiments", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HTKJNzl3Lh", "reviewer": "Reviewer_Lwg8", "summary": "", "questions": "- A. In ll.88-90 you state \"In this manner, the networks are forced to utilize partial neighbors' attention information, not the whole. Can you explain this logic in more detail?\n- B. Can the performance improvements in Section 4.1 (English Language Understanding) be seen not only in BERT-small, but also in BERT-base and BERT-large? And how big is the improvement margins?\n- C. What exactly is the \"Scale\" after the inner product of query and key in the formula at l.174?\n- D. Does it make any sense to choose a token to mask for each layer and have the same mask token for each head? What do you think the performance and robustness are compared to masking different tokens for each head?\n- E. If TLM was used in experiments with encoder-decoder models such as T5, how was cross attention regularized?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we are very grateful for your positive comments.\n\n- For the question A. We deeply apologize for the unclear statement. Because of the existence of Siblings-masking and Self-masking, the connections between the masked tokens and their neighboring tokens are invalid, which are implemented by assigning a large negative number in the matrix of attention weights.\n- As to the question B. I’m so sorry for our gross negligence that the experiments of GLUE without base- and large-size BERT, we can only report the results on GLUE with the base-size in the official comments due to the overly tight schedule.\n- For the question C. We are deeply apologized for the mistake of the \"Scale\", which is {\\sqrt{d_{emb} / H}}, we should remove “Scale” in Eq.2 because {\\sqrt{d_{emb} / H}} is computed in Eq.2.\n- For the question D. We think it does make sense and we can conduct the experiments to compare the performances.\n- For the question E. The masking of cross-attention is like self-attention, the difference is the query comes from the decoder layer, and the memory keys/values come from the output of the encoder. We will test it and report the results in the revised manuscript.\n\nWe will carefully revise the Typos and grammar problems in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Simple and easy to implement attention regularization approach\n- Thorough experiments evaluating on a wide range of data, tasks, and models\n- Proposed method shows performance improvement over existing methods in most settings\n- Sufficient and informative ablation studies\n- Achieved new SOTA on data-to-text generation task (Rotowire)\n- Clearly stated connections to existing methods\n- Well written and easy to understand", "reasons_to_reject": "- I did not understand the logic of one of the methodology explanations (Question A).\n- Section 3.2 (explanation of the approach) has a lack of information, and important information about the method is scattered in other sections.\n    - Only self-masking is explained and there is no explanation of siblings-masking.\n    - There is no description of selecting one of the two masking strategies for each batch in the 50-50 chance.\n- Model choice for the evaluation on English language understanding (Section 4.1) is somewhat unnatural (Question B).", "questions_to_authors": "- A. In ll.88-90 you state \"In this manner, the networks are forced to utilize partial neighbors' attention information, not the whole. Can you explain this logic in more detail?\n- B. Can the performance improvements in Section 4.1 (English Language Understanding) be seen not only in BERT-small, but also in BERT-base and BERT-large? And how big is the improvement margins?\n- C. What exactly is the \"Scale\" after the inner product of query and key in the formula at l.174?\n- D. Does it make any sense to choose a token to mask for each layer and have the same mask token for each head? What do you think the performance and robustness are compared to masking different tokens for each head?\n- E. If TLM was used in experiments with encoder-decoder models such as T5, how was cross attention regularized?", "paper_topic_and_main_contributions": "In Transformer models, structured dropout approaches such as attention dropout and DropHead have been used to regularize multi-head attention mechanisms. This paper proposes a novel regularization, token-level masking (TLM), based on the token level instead of the structural level. Its idea is to carefully mask partial connections between tokens in the attention calculation, rather than dropping whole attention heads or random attention weights. \n\nThe effectiveness of TLM has been thoroughly evaluated through experiments using a wide range of models on four different NLP tasks across 18 datasets. TLM is simple and easy to implement yet outperforms existing baseline methods in most cases. New SOTA was achieved for a task. Interesting ablation studies was provided.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- ll.50-54: Grammatically broken\n- Last line of Table 1 \"w/o Attention\" -> \"w/o Attention-dropout\"\n- l.73: no need to insert space between the werd \"function\" and footnote number \"1\"\n- Section name of Section 3.1: \"Transformer\" -> \"(Multi-head) Attention in Transformer\"\n- It would be easier to explain if the shapes of query Q, key K, and value V are explicitly indicated."}}, {"review_id": "9jW7Uq7YPh", "reviewer": "Reviewer_2Lox", "summary": "", "questions": "Why BERT-small is used for GLUE tasks?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, thanks for your constructive comments. \n\n- As to question 1.” The authors do not conduct a hyperparameter search…”, it’s our negligence without a clear explanation for hyperparameters, which is the rates of attention-dropout/DropHead/TLM in line 266. Here, we choose the best choice of rates for different regularization methods (The choices followed DropHead \\cite{zhou-etal-2020-scheduled} for baselines). We also conducted detailed experiments to compare the hyperparameters between attention-dropout /DropHead/TLM in the Ablation study (Line 431), and the results indicated our method is more powerful than the baselines with different hyperparameters.\n\n- For the question 2. “Why BERT-small is used for GLUE tasks?” I’m so sorry for our negligence of model size. Because the schedule is overly tight, we have reported the results of BERT-base in the official comments, and we will report the results of BERT-large in the revised manuscript.\n\nWe will carefully revise these issues in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is overall well-written and easy to follow.\n2. The idea of masking tokens in attention layers is novel, simple, and intuitive.\n3. The authors conduct extensive experiments on many datasets including both English and Chinese datasets and many different models including BERT, BART, T5, etc., and show consistent improvements.\n4. The ablation studies and analyses are well conducted and provide some insights for Transformer training regularization.", "reasons_to_reject": "1. The authors do not conduct a hyperparameter search or do not mention the search detail. Considering that different regularization methods may result in substantial changes on optimal values for other hyperparameters, it is unclear if the experiments are solid enough.\n2. The most representative experiments should be on the GLUE benchmark, however, only BERT-small results are presented, which is very confuse because standard practice all conduct experiments on GLUE with at least BERT-base.", "questions_to_authors": "Why BERT-small is used for GLUE tasks?", "paper_topic_and_main_contributions": "This paper introduces token level masking (TLM), a regularization approach for training Transformer models. TLM randomly mask tokens in the attention layer and force the model to exploit partial neighbors' information when producing representation. The authors conduct extensive evaluation on both English (GLUE) and some Chinese NLU benchmarks and show some improvements upon conventional attention dropout as well as advanced methods such as Drophead.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wKhQXXDxNx", "reviewer": "Reviewer_NQeB", "summary": "", "questions": "I hope the authors answer the questions mentioned in Reasons To Reject.\nWhy specific baseline is discarded in specific experiments (e.g., BERT+Att-dropout in Tables 3--5)?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we appreciate all your detailed comments. \n\nFor the question in “Reasons To Reject:”:\n- As to the question “the variation of the tested model variants is limited…”. In our work, we have tested “BERT, BART, and T5”, including small- and base-size. Saying that “The variation of the tested model is limited” is a little strict, and we beg the reviewer could check the details again.\n- For GLUE and CLUE, we just trained the pre-trained model once and submitted the files of predictions to the official leaderboard website to obtain the performances. We thought the results of such multiple tasks could provide some indication of generalizability. We then conducted 3 times experiments on GLUE with BERT-small/BERT-base and obtained the performances from the official leaderboard website. The results are reported in the official comments (Table 2 in the revised manuscript will be updated), and we hope that the additional experiments can alleviate the reviewer's concerns.\n- For the question of “Ablation study and some analyses (Section 4.5) are also conducted only in specific tasks (e.g., STS-B) or input.”, the reason is that STS-B and CoLA are sentence-level datasets, IFLYTEK and CSL are long text datasets, and CGED2020/2021 are grammatical errors datasets. We just hope the tasks can cover different lengths of text meanwhile the diversity can be guaranteed.\n- For the question of “ …seeing the attention pattern in a single instance (Figure 4).”, we present 2 instances with all layers of attention distribution in Appendix C and D, and we kindly request the reviewer may look again at the Appendix. The finding is that TLM is less affected by certain words because the attention distributions are aware in different layers (see Appendix C).\n\nThe focus of our experiments is to verify the generality of our TLM for reducing overfitting. We totally agree that “…willing to accept reduced task coverage, so I would like to see more solid findings with respect to, especially, model size/variants/seeds and the characteristics/limitations”. The comparison of experiments needs to highlight the strengths of our work, as well as the limitations. We will carefully modify these in the revised manuscript.\n\nFor the question in “Questions For The Authors”:\n- As to the question ” Why specific baseline is discarded in specific experiments (e.g., BERT+Att-dropout in Tables 3--5)?”, Both our experiments and DropHead indicate that the difference is not obvious with or without attention dropout, this leaves us careless not comparing attention dropout in Tables 3—5, we have reported the results on CLUE for easing the reviewer’s worries. \n|Model | AFQMC | TNEWS1.1 | IFLYTEK  | CMNLI | CLUEWSC | CSL | Score|\n|  ----  | ----  | ---- | ----  | ----  | ---- | ----  | ----  |\n|  BERT-base -w/o Att-dropout    | 73.5  | 56.7 | 60.1 |79.5 | 62.2 |\t80.3 | 68.7 |\n|  BERT-base + Att-dropout         | 73.7  |  56.6 | 60.3  |79.7| 62.0 |80.4  | 68.8 |\n|  BERT-base+DropHeadt            | 73.6  | 57.0  |60.6  |78.9 | 71.6 | 80.4  | 70.4|\n|  BERT-base+TLM                      | 73.9   | 58.1 | 61.5 | 79.3 | 73.4 |81.6 | 71.3 |\n\nTable 3: Fine-tuned BERT-base performances on Chinese language understanding benchmark CLUE.\n\nThe results show that the BERT-base with attention-dropout underperforms our TLM. Because the schedule is tight, we will do more experimental comparisons in the revised version. \n\nWe will carefully revise the Typos and grammar problems in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The idea is simple, and integrating TLM into Transformer variants is easy to implement.\n- Experiments have a broad range of tasks, and the improvement by TLM seems consistent across these experiments.\n- The ablation study and analysis (Section 4.5) cover the points the reader may care about; I would like to know the generality of the observations, though.", "reasons_to_reject": "- Although the coverage of tasks in the experiments seems sufficient, the variation of the tested model variants is limited. Specifically, only a different single model (e.g., BERT-base) is examined in different experiments.\n- How many models with different seeds are used in the experiments? If each experiment relies on a single run, I suspect the generality of the improvement by TLM and the significance of the performance differences.\n- Ablation study and some analyses (Section 4.5) are also conducted only in specific tasks (e.g., STS-B) or input. I understand that conducting all the analyses in all the tasks requires a high computation cost, but I would like to see a motivation for why the specific task/model was used in each analysis, and at least the information in this paper alone seems to cherry-pick the good results. Furthermore, I am not sure what generalizable findings can be obtained by seeing the attention pattern in a single instance (Figure 4).\n\nTo sum up, the focus of the experiments seemed to be blurred (in terms of model and task types), with sporadic evidence of effectiveness for certain combinations of tasks and models. In other words, I'd like to know the motivation why each setting was chosen. I am willing to accept reduced task coverage, so I would like to see more solid findings with respect to, especially, model size/variants/seeds and the characteristics/limitations of the proposed method (the robustness of the results in Tables 6--8 are).", "questions_to_authors": "I hope the authors answer the questions mentioned in Reasons To Reject.\nWhy specific baseline is discarded in specific experiments (e.g., BERT+Att-dropout in Tables 3--5)?", "paper_topic_and_main_contributions": "This paper introduces a novel regularization technique called \"Token-Level Masking (TLM).\" This is a specially controlled version of attention dropout, where attention weights to particular tokens from other context tokens (sibling masking) or themselves (self-masking) are dropped during training.\nExperiments demonstrate that this technique leads to better performance across several tasks, such as GLUE.\nFurther analysis tries to know the characteristics of the TLM concerning the simultaneous use of TLM with other regularization techniques and hyperparameters (Tables 6 and 7) and elucidate the effect of each component in the proposed method (Table 8)", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "\"Score\" in Tables 2--4 seems too abstract a description. Those in Tables 2 and 3 seem to be a macro average of the performance across the component tasks; this should be explained.\nThe text on the graphs is small. In particular, the y-axis is somewhat misleading because it does not start from 0."}}]

Author rebuttal:

Meta-review: The paper introduces Token-Level Masking (TLM), a novel regularization technique for training Transformer models. TLM involves masking tokens in the attention layer, forcing models to exploit partial neighbor information for improved representation. Extensive experiments on various NLP tasks demonstrate the effectiveness of TLM compared to existing methods.

Reviewers had several concerns, including the need for a more detailed explanation of TLM's motivation and its applicability to larger models. They questioned the lack of hyperparameter search details, the choice of specific models for experiments, and the variation in tested model variants. Additionally, they sought clarification on the selection of specific tasks and models for ablation studies. The authors clarified TLM's motivation, addressed the issue of hyperparameters, and committed to providing results for larger models. They explained their choices for specific tasks and models in ablation studies and expressed willingness to conduct more experiments to address baseline comparisons. The authors have demonstrated a proactive approach to improving the paper in the revised version.
Decision: Accept-Main