Isotropy-Enhanced Conditional Masked Language Models
Authors: Pei Guo
yisheng xiao
Juntao Li
Yixin Ji
Min Zhang
Venue: EMNLP 2023 Findings (2023)
Keywords: Non-autoregressive
Anisotropic Problem
Neural Machine Translation
Natural Language Processing

Abstract: Non-autoregressive models have been widely used for various text generation tasks to accelerate the inference process but at the cost of generation quality to some extent.  To achieve a good balance between inference speedup and generation quality, iterative NAR models like CMLM and Disco are proposed. Researchers have made much follow-up progress based on them, and some recent iterative models can achieve very promising performance while maintaining significant speedup. In this paper, we give more insights into iterative NAR models by exploring the anisotropic problem, i.e., the representations of distinct predicted target tokens are similar and indiscriminative. Upon the confirmation of the anisotropic problem in iterative NAR models, we first analyze the effectiveness of the contrastive learning method and further propose the Look Neighbors strategy to enhance the learning of token representations during training. Experiments on 4 WMT datasets show that our methods consistently improve the performance as well as alleviate the anisotropic problem of the conditional masked language model, even outperforming the current SoTA result on WMT14 EN $\rightarrow$ DE.

Isotropy-Enhanced Conditional Masked Language Models
Pei Guo; Yisheng Xiao; Juntao Li! Yixin Ji, Min Zhang Institute of Computer Science and Technology, Soochow University, China (pguolst, yyt.at.,sxiiootttuuuuuu suda. edu. cn;
(ijt, minzhang)@suda..dd edu. cn;
jiyixin1696gmiil.cmm
Abstract
tokens one by one in left-to-right order. To speed up model inference, many non-autoregressive meth- ods (Gu et al., 2018; Ghazvininejad et al., 2019; Qian et al., 2021; Huang et al., 2022b; Guo et al., 2023) explore predicting target tokens in parallel in recent years. As result of parallel decoding, they inevitably neglect the internal dependency of the target tokens, causing the generation quality falls behind their AR counterparts some extent.
Non-autoregressive models have been widely used for various text generation tasks acceler- ate the inference process but at the cost of gener- ation quality to some extent. achieve a good balance between inference speedup and genera- tion quality, iterative NAR models like CMLM and Disco are proposed. Researchers have made much follow-up progress based them, and some recent iterative models can achieve very promising performance while maintain- ing significant speedup. In this paper, we give more insights into iterative NAR models by exploring the anisotropic problem, i.e., the rep- resentations of distinct predicted target tokens are similar and indiscriminative. Upon the con- firmation of the anisotropic problem in iterative NAR models, we first analyze the effective- ness of the contrastive learning method and further propose the Look Neighbors strategy to enhance the learning of token representa- tions during training. Experiments on WMT datasets show that our methods consistently im- prove the performance as well as alleviate the anisotropic problem of the conditional masked language model, even outperforming the cur- rent SoTA result on WMT14 EN - DE
To achieve better trade-off between inference speedup and generation quality, many iterative NAR models (Ghazvininejad et al., 2019; Kasai et al., 2020; Chan et al., 2020; Huang et al., 2022c) have been proposed recently. They adopt multi- ple decoding iterations to generate final results, where the intermediate result generated from the last iteration can provide useful target informa- tion and then be fed into the model for refine- ments in the next iteration. Excitingly, iterative decoding brings significant performance improve- ments for NAR models and even outperforms the vanilla autoregressive Transformer. Among them, one of the most competitive and widely-used mod- els is CMLM (Ghazvininejad et al., 2019), which adopts uniform masking during training and a mask-prediet strategy for decoding. Based on CMLM, many advanced methods, such train- ing strategy (Kasai al., 2020), criterion (Marjan et al., 2020; Du et al., 2021), and inference strat- egy (Kreutzer et al., 2020; Geng et al., 2021), have been explored and achieved better performance. Despite significant progress, some critical prob- lems of NAR methods still remain challenging, e.g.. repetition and reliance on distillation.
1 Introduction
Transformer-basee models (Vaswani et al., 2017) with auto-regressive (AR) decoding paradigm have successfully applied to various text generation tasks, such as machine translation (Wu et al., 2019; Liang et al., 2021), text summarization (Savelieva et al., 2020; Elsaid et al., 2022), dialogue sys- tems (Zhang al., 2020; Ma et al., 2020), with exciting performance being achieved. However, the low inference efficiency prohibits their usage (espe- cially with current luper-large transformer models as backbones) since the AR paradigm predicts the
To disclose more insights into the enhancement of the iterative NAR models, this paper explores from a different perspective, i.e., the anisotropic problem, which has been revealed recently in AR models (Su al., 2022). This can cause the de- generation of AR neural language models with the anisotropic distribution of token representations
Equal Contribution
Juntao Corresponding Author.
OOur code implementation and data are available at https: Enhanced-
//github com/ Al wwwwwrd/Iwsworroooeoooroo wop.anrr----

---

sentations. Due learning better representations. Look Neighbors can supplement contrastive learn- ing well during the training of CMLM.
We conduct comprehensive experiments to eval- uate our methods. Results on WMT14 EN++DE and W EN+RO datasets all demonstrate that our methods outperform the vanilla CMLM model, achieve comparable performance compared with strong iterative NAR baselines. Besides, the explo- rations on the token representation CMLM also give a new insight into the iterative NAR models, indicating the anisotropic problem, which has been well learned in AR models recently, also affects the performance and further developments for NAR models.
06
(a) Iter. on EN
(b) I0 on EN
Ex
R5
Hay
(c) Iter. on DE
(d) 10 EN-DEE
Figure 1: Token cosine similarity matrix of CMLM with in iteration 1/10 on WMT16 EN-RO and WMT14 EN-DE raw data.
2 Preliminaries and Trials
2.1 Non-autoregressive Language Model
Most of the text generation models adopt the au- toregressive decoding format to generate the target sentences and use the autoregressive factorization to maximize the following likelihood:
(taken from the output layer of the model), in which the high similarity of different tokens can mislead the model to generate repetitive tokens at different steps. Naturally, we wonder whether the repetition problem of NAR models is also closely associated with the anisotropic representations. We take the typical iterative NAR model CMLM (Ghazvinine- jad al., 2019), and present the results in Figure I, the same phenomenon in AR model (Su al., 2022) also holds for iterative NAR models, i.e., the token similarity is close to each other (Figure ((a) and (C)). Besides, decoding with multiple iterations (Figure I(b) and (d)) do not alleviate the dense rep- resentation of token similarity for large margin. Therefore, the anisotropic problem of the iterative NAR models is worth more exploration.
T Σ log (ellyct,X(0)) (1)
CAR =
where Ver denotes the previous generated target tokens, T denotes the length of the target sentence, X is the source sentence, and 0 denotes the model parameters.
Non-autoregressive Language Model removes the internal dependency of target tokens and adopts conditional independent factorization for predic- tion. Recent non-autoregressivv (NAR) language models can be divided into fully NAR and iterative NAR models. The former ones adopt the following objective to maximize the likelihood:
To alleviate the anisotropic problem of CMLM, we first test the effectiveness of the existing method based on contrastive learning (Su et al., 2022). However, we found that directly adopting the ex- isting method does not obtain consistent improve- ments on various datasets. Then, we analyze the potential reason and attribute to the failure to learn the token representation well during training, which is a widely recognized problem for all NAT models. The reason why "failure to learn represen- tations" is not conducive to improving performance will be discussed in Section 2.3. Motivated by the particularly dense representation of adjacent tokens, we propose simple yet effective strategy named Look Neighbors to enhance the dependency of adjacent tokens and auxiliarily learn better repre-
T
CENAR Σ
log P))))
(2)
Obviously that the conditional tokens Vet are re- moved for NAT models. Then the models con- duct parallel generation without autoregressive con- straints, and the inference speed will be greatly improved. Comparatively, iterative NAR models share spirit of a mixed autoregressive and non- autoregressive generation. They adopt multiple de- coding iterations and keep the autoregressive prop- erty in each iteration. They aim to maximize the

---

vito seco
WMT14
WMT16
Models
EN-DE
DE-EN
EN-RO
RO-EN
CMLM
24.92
29.59
32.84 32.88
32.44
w/ CL
25.59
29.77
32.76
Table 1: The results on raw datasets with contrastive learning (CI.) for CMLMM.
(a) Iter.1 on EN-+RO.
(b) I0er.10 on EN-RRO.
the [cls] token in BERT) into its encoder to predict the target length conditional on the source sentence. Inference Strategy. During inference, CMLM adopts the mask-predict decoding algorithm. Specifically, the tokens with low confidence pre- dicted in the previous iteration will be masked, and the model will predict these masked tokens again in the next iteration. Given the target length L, the source sentence and the total decoding iteration T, CMLM predicts the entire masked target sen- tence (i.e., empty Yaue) at the first iteration. Then after obtaining the relatively incredible result of the first iteration, the model will choose the specific number of masked tokens with the lowest predic- tion probability. In an intermediate iteration I, the number of the masked tokens n can be calculated as n = - L. The target sentence with newly masked tokens will be fed into the decoder again and the masked tokens will be predicted conditional on the source sentence X and the unmasked tokens Yoter and the prediction probability of them will be updated. In general, the more iterations, the better the performance of the generation.
Hag
(c) Iter. on EN-DE.
(d) Iter. 10 EN-DDE.
Figure 2: Token cosine similarity matrix CMLM with contrastive learning iteration 1/10 on raw WMT16 EN-RO and WMT14 EN-DE datasets.
following likelihood:
LINAR Σ
log P(M|ly Xx;), 1og;((,,,XX0))
TCYeg:
where the Yegt denotes the prediction target of cur- rent iteration and the y denotes generation result of previous iterations.
2.2 Conditional Masked Language Model
Since we adopt the conditional masked language model (CMLM) as the backbone to explore the anisotropic problem for the iterative NAR models, we give a detailed introduction the training and inference process in CMLM.
2.3 Contrastive Learning for CMLM
In this part, we present how to adopt the contrastive learning method for the conditional masked lan- guage model (CMLM), Referring to the original paper (Su et al., 2022), we adopt a contrastive ob- jective into the training of CMLM. Specifically, given training pair (X, Y), the contrastive loss can be calculated as:
Training Process. During training, CMLM adopts a masked language modeling task with a uniform masking strategy to decide the masked tokens in the target sentence and learns to predict them. Specifi- cally, given training pair (X, Y), several tokens in Y will be masked Ymask. and the rest tokens are denoted as Yots- The model then learns to predict the masked tokens Ymask in parallel given X and the unmasked tokens Yous- The training objective CMLM is to maximize:
Y
Y
Esum Σ
Σ
hyg)) p). (5)
j=1
1
LCMLM Σ
Pyy X)0),
Lave
YY S
(6)
(4)
J.EYmu
where |Y| is the length of target sentence, by, de- notes the representation of token Vi. s(hy, hys) denotes the cosine similarity between two token representations, and p denotes the pre-fined con- trastive margin. Intuitively, the contrastive learning
where denotes the trainable parameters of CMLM. Besides, as the same as traditional NAR models, an auxiliary task to predict the target length is adopted. CMLM adds a special token [LENGTH] (akin to

---

method can punish the close representations of to- kens, resulting in a discriminative and isotropic model representation space, The overall training objective for CMLM is defined as:
ity learn token representations during training. As mentioned in Section 2.3, the failure to learn the token representations well for CMLM prevents the superior effects of adopting contrastive learn- ing. The instability of the representations in NAR models has also been explored in recent NAR pa- pers (Wang et al., 2019; Xie et al., 2022). They propose the regularization method to normalize the token representations. Unlike these regularization methods, we try the enhance the learning of to- ken representations. Specifically, we incorporate the representations of other tokens to make predic- tions during training, thus improving the ability to learn token representations. As shown in Fig- ure 1, the cosine similarity is denser in adjacent tokens, indicating that CMLM has more difficulty in understanding and distinguishing the adjacent representations of target tokens. As result, we mainly incorporated the representations of adja- cent tokens rather than other tokens. The most similar exploration is explored in LAVA-NAT (Li et al., 2020). They propose a Look-Around decod- ing strategy to enhance the inference schedule but without enhancing the training process.
L = LCMLM + Lave-
(7)
Results are shown in Figure 2 (cosine similarity matrix) and Table (BLEU score). Fortunately, the token cosine similarity matrixes become sparse, e.g. the representations of distinct tokens become discriminative significantly and multiple iterations bring further improvements. However, with re- gard to the BLEU score, there is no consistent im- provement for various datasets, e.g., as shown in Table 1, the performances of WMT14 DE->EN and 6 EN-RO dataset only have 0.18 and 0.04 BLEU improvement respectively. We ana- lyze the potential reason for this inconsistent im- provement and attribute it to the failure to learn the correct representation of the distinct target tokens. More specifically, the anisotropic loss is optimized based on the token representations, tiveness of anisotropic loss is closely related to the model's own ability to learn the token represen- tations, i.e., blindly constraining incorrect token representations by anisotropic loss can't bring sub- stantial improvement for performance although it truly makes their representations more distinguish- able. As result, due to the character that "fail to capture the dependency of target tokens" for NAR models, anisotropic loss just alleviates the traditional anisotropic problem but gets limited im- provement on BLEU as shown in Table
the effee-
We propose the Look Neighbors, where the model integrates a neighbor-aware self-attention into every decoder layer and concatenates the rep- resentations from the last decoder layer of the left and right neighbors to assist in predicting the target tokens. Firstly, we formally introduce our neighbor- aware self-attention as follows:
Att = a. Att.ols + +11 - oz). Attnoighhor (8)
QKT
It's noticed that "failure to learn representations" is a els, but not caused by anisotropic loss. The claim denotes that NAR models fail to capture the de- pendency of target tokens during training (Gu and Kong, 202 Xiao et al., 2022; Zhan et al., 2022), i.e., unlike AR models which can learn target-side dependency between one specific token and previ- ous generated tokens, NAR models only depend on source tokens to make predictions due to the condi- tional independent factorization adopted in training process. As a result, we aim to promote the ability to learn the token representations for CMLM first, then adopt the contrastive learning method.
Attnhigbbor = softmaz(
+
+MV
(9)
problem for all NAT mod-
Vd
Mij = 0, j +1 -00, oteerwie
(10)
where a a learnable parameter, All the original decoder self-attention, Attwesighor is our neighbor-aware self-attention, Att denotes the final attention we adopt in the experiments, and Myj expresses the position of the token i in the mask matrix. Next, given the representa- tions of the last decoder layer for target tokens:
H [h1, (h1,h2 he,...,,rri) where h is the represen- tation of the target token Y. T is the length of the target sentence. Each representation h; con- catenates the left neighne h and h+41. denoted by:
3 Look Neighbors
this section, we introduce our enhancement method, Look Neighbors, which improves the abil-
h = (hih--1 @ h2+1)) + h4
(11)

---

WMT14
WMTI6
Models
Ide:
Speedup
EN-DE
DE-EN
EN-RO
RO-EN
Transformer*
27.48
31.27
33.70
34.05
1.0 x
FT-NAT (Gu et al., 2018)
11.79
16.27
15.6
Flowseq (Ma al., 2019)
23.64
28.29
32.35
32.91
1.1
Fully-NAT (Gu and Kong, 2021b)
23.58
16.5
Fully NAT
AXE-NAT (Marjan et al., 2020)
20.40
24.90
14.2
OAXE-NAT (Du et al., 2021)
22.40
26.8
13.3 X
GLAT+CTC (Qian et al.. 2021)
25.02
29.14
14.6 X
DSIP+CTC (Huang et al., 2022a)
24.81
28.33
14.0
DA- Transformer (Huang al., 2022b)
26.08
30.48
14.0
Diff-GLAT (Qian al., 2022)
26.46
30.48
14.3
Iterative NAT
SMART (Ghazvininejad et al., 2020)
10
25.10
29.58
32.71
32.86
2.2 x
Disco (Kasai et al., 2020)
10
25.64
32.25
3,5 x
Imputer (Saharia et al., 2020)
25.00
3.9 x
CORR (Huang et al., 2022c)
10
26.01
30.55
33.71
33.27
2.1 X
CMLMC (Huang al., 2022c)
26.40
30.92
34.14
34.13
1.7 x
CMLM (Ghazvininejad al., 2019)
10
24.92
29.59
32.84
32.44
2.2
Ours
CMLM w/ Ours
10
26.68
30.50
34.01
33.83
1.7 x
Table 2: Results on WMT machine translation tasks. denotes our implementations.
where W is learnable matrix, denotes the ma- trix concatenation. Then the decoder representa- tions will be II' {h4, h/g, ..., }, and the model predict the tokens based on H'. Obviously, the Look Neighbors strategy makes each representa- tion to be aware of the information about its neigh- bors, thus improving the learning of token repre- sentations during training.
and label smoothing to 0.1 for all the experiments. We train the models with batches of 64k tokens for all the datasets. We set the dropout rate 0.2 to EN<+DE datasets and 0.3 EN< RRO datasets. The other hyper-parameters are adopted following CMLMC (Huang et al., 2022c), e.g., the learn- ing rate warms up to 7c-4 in 40k for WMT14 EN+DE datasets, then gradually decays with an in- verse square schedule for 300k update steps in total. The corresponding settings for WMT16 EN RO datasets 0..4/15k/100K Besides, notice that the pre-fined contrastive margin p is set 0.4, which has proven to achieve the best performance in 4.4. During inference, we average the best check- points based on validation BLEU scores as our final model, and the length beam is set to 5.
4 Experiment
4.1 Datasets
We verify the effectiveness of our method by con- ducting experiments on multiple public translation datasets: WMT14 EN+DE, WMT16 EN+RO., The WMT En and En++De datasets contain 66/9999K/1.9K and 4.5M/39K/3K of train- ing/valid/test sentence pairs, respectively.
4.3 Results
4.2
Experimental Settings
Following previous works, we evaluate the perfor- mance with BLEU (Papineni et al., 2002). Results are presented in Table 2, we compare our meth- ods with the traditional CMLM,, the strong autore- gressive baseline: vanilla Transformer, and other NAT methods including related methods (NAT- REG, Fully-NAT',, recent competitive Fully NAT models (GLAT, DSLP, DAD and DA-Transformer), and several CMLM- iterative NAT models (SMART, Disco, Imputer, MvSR-NAT, CCMLM
We conduct all experiments using the Fairseq li- brary (Ott et al., 2019). We adopt the same model structure for a fair comparison with previ- work and the CMLM bascline, i.c., the stan- dard Transformerbase configuration containing layers per stack, 8 attention heads per layer. 512 model dimensions, and 2048 hidden dimensions. We adopt the Adam optimizer (Kingma and Ba, 2014) 3 with (0.9, 0.98), set weight decay to 0.01

---

WMT14
WMT16
Models
EM - DE
WMT16 RO
Modcls
EN-DE
DE-EN
EN-RO
RO-EN
CMLMM
27.21
33.36
w/ Ours
27.71
34.33
CMLM
24.92 25.41
29.59
32.84
32.44
w/ LN
30.36
33.43
33.02
Table The performance of CMLM w/o and w/ our methods on WMT distilled data.
Table 3: The effect of LN for CMLM on WMT datasets.
CMLMWW OLES
CMLM
342
34
33.8
10
165
336
02
322
0.
334
3322
(a) Iter.1, CMLM.
(b) Iter. 10, CMLM.
33
328
326
05 Contrastive Margin
Q4
Q,6
22
140
26
0.8
Figure The exploration of the best pre-fined con- trastive margin p.
20
(c) Iter., CMLM w/ Ours. (d) Iter. CMLM w/ Ours.
and CMLMC). Firstly, methods gain improve- ments on all datasets compared with traditional CMLM, and mitigate the gap between AT and NAT models for about 1.0 BLEU score. Compared with other related NAT methods, our proposed methods outperform most of them, except for CMLMC. Sur- prisingly, our methods achieve 26.68 on WMT14 EN-DE, and set the new SoTA performance. Be- sides, compared with CMLM w/ CL (shown in Table 1), CMLM with both methods achieves sig- nificant improvements. It verifies our hypothesis that while better capturing the dependency of get tokens for NAR models, the anisotropic loss can perform more effectively to improve the perfor- mance while alleviating the anisotropic problem. We also provide the case study in Appendix A to further understand the effectiveness of our method.
Figure 4: Token cosine similarity matrix of CMLM w/o and w/ methods in iteration 1/10 distill WMT16 EN-RO dataset.
Best Contrastive Margin As mentioned in Sec- tion 2.3, there a pre-fined contrastive margin p to control the strength of contrastive Iearning. If the is relatively big, the effect of contrastive learning is small. However, once the becomes 1.0, the contrastive loss has no effect on training. We con- duct experiments on WMT16 EN -RO dataset and set the margin from 0.3 to 0.6. As shown in Fig- ure 3, the performance all outperforms the vanilla CMLM, indicating that is not hard to apply our methods. The best value is 0.4.
Analysis
4.4 Ablation Study
5.1 Performance on Distilled Data
Knowledge distillation a widely-used method various NAR models (Hinton et al., 2015; Zhou et al., 2019). Following the previous works (Gu et al., 2018; Huang et al., 2022c), we first adopt AR models trained with original data to generate the results. Then the generated outputs adopted as distilled data and serve the training data of NAR models. We adopt TransformeT as the teacher model for WMT EN<+RO and EN<+DE, respectively. Compared with original training data, distilled data more explicit and simpler, and can
Effect of Look Neighbors In order to eval- uate the effectiveness of Look Neighbors, we conduct the experiments by comparing CMLM with/without Look Neighbors (LN). As shown in Table 3, it's evident that CMLM w/ LN achieves consistent improvements on BLEU score (25,41 vs. 24.92, 33.43 vs. 32.84), which certifies that the Look Neighbors strategy is able to capture the dependency of the adjacent tokens effectively and assist in generating better translation results.

---

CMLMM raw CMLM distill Ours_raw raw Ours distill
CMLM
CMLM wy Ours
0.8
Cosine Similarity
Cosine Similarity
0.7
0.6
0.5
0.4
0.3
0.3
0.2
0.2
0.1
0.1
Itter
Iter
Iter
[0, 10)
[10, 20)
[20, 30)
[30, 40)
>=10
Figure 5: Comparison of token similarity distribution between CMLM w/o and w/ our methods.
Figure 6: Comparison of token similarity distribution between CMLM w/o and w/ our methods with different source lengths.
WMT14 EN
DE
WMT16 EN
RO
Models
Iter
BLEU
Reps
Comet
BLEU
Reps
Comet
22.50
2.00%
-0.09
31.47
0.90%
0.21
tions and adopting knowledge distillation does not alleviate the anisotropic problem effectively. Be- sides, adopting
CMLM
24.92
24.07
32.79
0.72%
w/ Ouss
methods presents significant ef- fects in reducing the token cosine similarity scores. There are no conflicts in our methods with iterative decoding or knowledge distillation, i.e., adopting knowledge distillation with our methods together further reduces the token cosine similarity scores, and the token cosine similarity scores trained with our methods will be lower as more iterations are adopted.
10
26.68
0.17%
0.12
34.01
0.13%
Table 5: BLEU score, the token repetition ratio and Comet score with different iterations on the raw WMT14 EN DE and WMT16 EN RO datasets.
help alleviate the well-known multi-modality prob- lem in NAR models. We also evaluate our methods on distilled data and present the results in Table 4. Our method can achieve about 0.5/1.0 BLEU score improvements on WMT EN->E and EN-RO, respectively. Besides, also compare the token cosine similarity with/without our methods dis- tilled data. We plot the results in Figure 4, and we can find that training with distilled data still suffer from the anisotropic problem, and our meth- ods also bring significant benefits in alleviating this problem.
Token Repetition and Comet. Repetition prob- lem is common in NAR models, we view the anisotropic problem as one of the reasons to cause token repetition. Since the anisotropic problem can be alleviated well with our methods, we won- der if the repetition rate reduces. We compare methods with vanilla CMLM of different iterations and compute the corresponding repetition ratios. Besides, we also report Comet (Rei et al., 2020) to make a more complete evaluation. The results on raw WMT EN >DE and EN +RO datasets are presented in Table As we can see, our method achieves a consistent decrease in the repetition ratio with different iterations. Besides, the BLEU score and Comet also outperform the vanilla CMLM in all conditions.
5.2 Quantitative Comparison
Cosine Similarity. Many previous works have pointed out the positive effects of iterative decod- ing and knowledge distillation (Ding et al., 2020; Gu and Kong. 2021b; Xiao et al., 2022) for NAR models. order to better understand the effects them for alleviating the anisotropic problem, we conduct a quantitative comparison of the token cosine similarity between different iterations and knowledge distillation. As we only compare the cosine similarity with specific example in the previous contexts, e.g., Figure and Figure 2, aim explore the effects on the full test sets and compute the average token cosine similarity scores WMT16 EN + RO dataset and plot them in Figure 5. We find that decoding with more itera-
Different Source Lengths. We also explore the effects of our methods with different sentence lengths. Specifically, we divide the source sen- tences of WMT16 EN-RO test set into 5 intervals by the length after BPE operation, then compare the average token representation similarity of it- eration 10 with/without our methods. Results are shown in Figure 6. seems the length has a small impact on token representation similarity, and our

---

Models
WMT16 EN
WMT16 RO EN
CMLM
32.84
32.44
0.5
0.5
CORR
33.27 33.71
100
YEAL
CORR w/ Ours
34.15
34.02
25
0.5
Table 6: The performance of CMLM combined with CORR and CORR w/ our method.
40
04
(a) Iter.1, CORR.
(b) Iter.0,, CORR.
methods alleviate the anisotropic problem greatly with all conditions of sentence length, e.g., the to- ken representation similarity is all about 0.7 with different source sentence lengths and reduced to around 0.15 with our methods. Although the to- ken representation similarity seems to increase for longer sentences after adopting our methods, the final result is still less than 0.2, and this small in- crease can be ignored.
125
36
3.
(c) Iter. CORR w/ Ours. (d) Iter. 10, CORR w/ Ours.
Figure 7: Token cosine similarity matrix CORR w/o and w/ our methods in iteration 1/10 raw WMT16 EN-RO dataset.
5.3 More Explorations
As shown in Table 2, the performance of our meth- ods is comparable with CMLMC (Huang et al., 2022c), which introduces self-correction mech- anism based CMLM. We also explore the po- tential of adopting our methods combine with this mechanism. We only adopt the self-correction mechanism by ourselves as the same as CMLMC based on CMLM (CORR), and then adopt our meth- ods (contrastive learning and look neighbors) to it. Results on two WMT datasets are shown in Table 6, we can find that the self-correction mechanism truly improves the performance of vanilla CMLM, and training with our methods further boosts the per- formance. Moreover, we also plot the token cosine similarity matrix of CORR with/without our meth- ods in Figure 7. Results show that adopting the self-correction rather than just the mask- mask-predicc algorithm in vanilla CMLM does not help allevi- ate the anisotropic problem, and our methods also bring benefits with a self-correction mechanism. This also presents the potential of methods to be applied to other iterative NAR models.
ation. Besides, motivated by BERT (Kenton and Toutanova, 2019), Ghazvininejad et al. propose the conditional masked language model (CMLM) which adopts the uniform masking strategy during training and a mask-predict strategy for inference. Due to its promising performance, CMLM has become of the most competitive and widely- used iterative NAR models, and many advanced enhancement strategies from different perspectives have been proposed based on CMLM in recent years, c.g., optimizing the inference strategy (Ka- sai et al., 2020; Geng et al., 2021), optimizing the masking strategy (Guo et al., 2020; Xiao et al., 2023), benefiting from the AT counterpart (Hao et al., 2021; Xiaobo Liang and Zhang, 2022), train- ing with better criterion (Marjan et al., 2020; Du et al., 2021) and introducing correction mecha- nism (Ghazvininejad et al., 2020; Huang al., 2022c). Recently, motivated by the application of diffusion models (Ho et al., 2020). Savinov et al. proposed a step-unrolled denoising autoencoder that adopts denoising operation in each iteration. Besides, Wang et al. adapt the cross-lingual pre- training model (XLMR) into NAT models and fur- ther improve the performance with iterative refine- ments.
6 Related Work
The iterative NAR models are proposed to achieve the trade-off between inference speedup and gen- eration quality. Lee et al. first propose the itera- tive model, where the noised target sentences refined in multiple iterations. Later, text-editing methods (Stern al., 2019; Gu et al., 2019; Lu and Peng, 2022) are also introduced. They gen- crate the target sentences by adopting the auxil- iary insertion and deletion operations in each iter-
The token representations of NAR models have also attracted much attention once the birth of NAR models. Wang et al. normalize the learning of the decoder representations by introducing similarity and reconstruction regularizations. Xie et al. pro- pose a consistency training method to regularize the

---

representations of the same tokens in the different masked target sentences, thus improving the sta- bility of representing the tokens. Besides, the con- trastive learning method has been explored in (Su et al., 2022) to pull the positive token pairs together and push negative pairs away. They optimize the similarity of several different representations of the same token to learn more informative and robust representations. Unlike this contrastive method, we try make the representations of tokens in one sentence more discriminative, thus alleviating the problem of anisotropic representations. The anisotropic problem is proposed (Su et al., 2022), they focus on the AR models and try to alleviate the degeneration problem with the explorations of the anisotropic problem. In this paper, we focus on the iterative NAR model CMLM and try explore more potential of iterative models.
to enhance the learning of token representations. As a result, we wonder whether there exist other enhancing methods that can better combine with contrastive learning.
Acknowledge
This work is supported by the National Science Foundation of China (NSFC No. 62206194) and the Natural Science Foundation of Jiangsu Province (Grant No. BK20220488).
References
William Chan, Chitwan Saharia, Geoffrey Hinton, Mo- hammad Norouzi, and Navdeep Jaitly. 2020. Imputer: Sequence modelling via imputation and dynamic pro- gramming. In ICML, pages 1403-1413. PMLR.
Liang Ding, Longyue Wang, Xuebo Liu, Derek F Wong, Dacheng Tao, and Zhaopeng Tu. 2020. Un- derstanding and improving lexical choice in non- autoregressive translation. In ICLR.
7 Conclusion
In this paper, we first verify that the strong and representative iterative NAR model, CMLM, suf- fers from the anisotropic problem by analyzing the changes in the cosine similarity of token represen- tations. To alleviate the anisotropic problem, we introduce the contrastive learning method to make the token representations discriminative and fur- ther propose Look Neighbors strategy to enhance token representation learning during training. Ex- tensive experiments on several widely-used trans- lation datasets indicate that our proposed methods have a great effect on alleviating the anisotropic problem and can consistently improve the perfor- mance of iterative NAR models. We hope that our explorations can give new insights into the iterative NAR models and promote their development.
Cunxiao Du, Zhaopeng Tu, and Jing Jiang. 2021. Order-agnostic cross entropy for autoerrvreeeessvvve machine translation. In ICML, pages 9849-2859. PMLR.
Asmaa Elsaid, Ammar Mohammed, Lamiaa Fattouh, and Mohamed Sakre. 2022. comprehensive review of arabic text summarization. IEEE Access.
Xinwei Geng, Xiaocheng Feng, and Bing Qin. 2021. Learning to rewrite for non-autoregressive neural ma- chine translation. In EMNLP, pages 3297-3308.
Marjan Ghazvininejad, Omer Levy. Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel de- coding of conditional masked language models. In EMNI.P-ICNI.P pages 61 12-6121.
Marjan Ghazvininejad, Omer Levy, and Luke Zettle- moyer. 2020. Semi-autoregrese training improves mask-predict decoding. arXiv:2001.0888 108785.
8 Limitations
In this section, we present several limitations of our proposed methods and this paper. Firstly, we only take the conditional masked language model (CMLM) as a backbone to evaluate the effective- ness of our methods, more iterative models may also need exploration, e.g., the recent diffusion models. Besides, since the repetition problem is quite serious in fully NAR models, whether the anisotropic problem exists in fully NAR models, we leave this a future work. Secondly, since singly adopting contrastive learning brings few benefits on the performance as mentioned in Sec- tion 2.3, we further propose the Look Neighbors
Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. 2018. Non-autoregressive neural machine translation. In ICLR.
Jiatao Gu and Xiang Kong. 2021a. Fully non- autoregressive neural machine translation: Tricks of the trade. In Findings of the Association for Computational Linguistics, pages 120-133.
Jiatao Gu and Xiang Kong. 2021b. Fully non- autoregressive neural machine translation: Tricks of the trade. In Findings of ACL-IJCNLP, pages 120- 133.
Jiatao Gu, Changhan Wang, and Junbo Zhao. 2019. Lev- enshtein transformer. NeurIPS, 32:1 111-1119

---

Junliang Guo, Linli Xu, and Enhong Chen. 2020. Jointly masked sequence-to-sen to-sequnnce model for non- autoregressive neural machine translation. In ACL. pages 376-385.
Xiaobo Liang, Lijun Wu, Juntao Li, Yue Wang. Qi Meng. Tao Qin, Wei Chen, Min Zhang, Tin-Yan Liu, et al. 2021. R-drop: regularized dropout for neural networks. NeurIPS, 34.
Pei Guo, Yisheng Xiao, Juntao Li, and Min Zhang- 2023. Renewnat: Renewing potential translation for non-autoregreessvve transformer. In Proceedings of the Thirty-Sevent Seventh AAAI Conference on Artificial Intelligence, pages 12854-12862.
Sidi Lu and Nanyun Peng- 2022. Insnet: An efficient, flexible, and performant insertion-basee text genera- tion model. NeurIPS.
Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neu- big, and Eduard Hovy, 2019. Flowseq: Non- autoregressive conditional sequence generation with generative flow. In EMNIP-IJCNIAL pages 4282- 4292.
Yongchang Hao, Shilin He. Wenxiang Jiao, Zhaopeng Tu, Michael Lyu, and Xing Wang. 2021. Multi-task learning with shared encoder for non-autorerresive machine translation. In NAACL-HLT, pages 3989- 3996.
Yukun Ma, Khanh Linh Nguyen, Frank Z Xing, and Erik Cambria. 2020. A survey empathetic dialogue systems. Information Fusion, 64:50-70.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. 2015. Distilling the knowledge in a neural network. arXiv: stt.l110300053. 2(7).
Ghazvininejad Marjan, Vladimir Karpukhin, Luke Zettlemoyer, and Omer Levy. 2020. Aligned cross entropy for non- autorrrrrrrrssvveee machine translation. In ICML, pages 3515 3515-3523, PMLR.
Jonathan Ho, Ajay Jain, and Pieter Abbecl. 2020. De- noising diffusion probabilistic models. NeurIPS, 33:6840-6851.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In NAACL-HLT.
Chenyang Huang. Hao Zhou, Osmar R Zaiane, Lili Mou, and Lei Li. 2022a. Non-autoregressive translation with layer-wise prediction and deep supervision. In AAAI, volume 36, pages 10776-10784.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalua- tion of machine translation. In ACL. pages 31 1-318.
Fei Huang, Hao Zhou, Yang Liu, Hang Li, and Minlie Huang. 2022b. Directed acyelic transformer for non- autoregressive machine translation. In ICML, pages 9410-9428. PMLR.
Lihua Qian, Mingxuan Wang. Yang Liu, and Hao Zhou. 2022. Diff-glat: Diffusion glancing transformer for parallel sequence sequence learning. arXiv preprint arXiv:2212. 10240.
Xiao Shi Huang, Felipe Perez, and Maksims Volkovs. 2022c. Improving non-autoreeressive translation models without distillation. In ICLR.
Lihua Qian, Hao Zhou, Yu Bao, Mingxuan Wang, Lin Qiu. Weinan Zhang, Yong Yu, and Lei Li. 2021. Glancing transformer for non-autoregressive neural machine translation. ACL-IICNLP, pages 1993- 2003.
Jungo Kasai, James Cross, Marjan Ghazvininejad, and Jiatao Gu. 2020. Non-auttoregressive machine transla- tion with disentangled context transformer. ICML., pages
PMLR.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. Comet: A neural framework for mt evaluation. EMNLP), pages 2685-2702
Jacob Devlin Ming-Wes Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In NAACL-HLT, pages 4171-4188
Chitwan Saharia, William Chan, Saurabh Saxena, and Mohammad Norouzi. 2020. Non-autoregressive ma- chine translation with latent alignments. In EMNLP, pages 1098 1108.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint urXiv:1412.660
Alexandra Savelieva, Bryan Au-Yeung, and Vasanth Ramani. 2020. Abstractive summarization of spoken and written instructions with bert. arXiv preprint 2Xiv:2008.09674
Julia Kreutzer, George Foster, and Colin Cherry. 2020. Inference strategies for machine translation with con- ditional masking. arXiv:2010.02352.
Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018. Deterministic non-autoregressive neural se- quence modeling by iterative refinement. In EMNLP. pages 1172-1182.
Nikolay Savinov, Junyoung Chung, Mikolaj Binkowski. Erich Elsen, and Aaron van den Oord. 2023. Step- unrolled denoising autocncoders for text generation. In ICLR.
Xiaoya Li, Yuxian Meng, Arianna Yuan, Fei Wu, and Jiwei Li. 2020. Lava nat: A non-autoregressive trans- lation model with look-around decoding and vocabu- lary attention, arXiv:2002.0030:
Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit, 2019. Insertion transformer: Flexible se- quence generation via insertion operations. In ICML. pages 5976-5985. PML..

---

Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Ling- peng Kong, and Nigel Collier. 2022. A contrastive framework for neural text generation. arXiv preprint arXiv:2202.06417.
Ashish Vaswami, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. NIPS, pages 5998 6008.
Yiren Wang, Fei Tian, Di He. Tao Qin, ChengXiang Zhai, and Tie- Yan Liu. 2019. Non-autoreerresive machine translation with auxiliary regularization. In AAAL, volume 33, pages 5377-5384.
Yong Wang, Shilin He, Guanhua Chen, Yun Chen, and Daxin Jiang. 2022. XIm-d: Decorate cross- lingual pre-training model non-autoregressive neural machine translation. In Proceedings the 2022 Conference on Empirical Methods in Natural Language Processing, pages
Lijun Wu, Yiren Wang, Yingce Xia, Fei Tian, Fei Gao. Tao Qin, Jianhuang Lai, and Rie Liu. 2019.
Depth growing for neural machine translation. In ACL, pages 1558-5563
Yisheng Xiao, Lijun Wu, Junliang Guo, Juntao Li, Min Zhang, Tao Qin, and Tie-yan Liu. 2022. survey von-auvoreeressivv generation for neu- ral machine translation and beyond. arXiv preprint arXiv:2204.09269.
Yisheng Xiao, Ruiyang Xu, Lijun Wu, Juntao Li, Tao Qin, Yan-Tie Liu, and Min Zhang. 2023. Amom: Adaptive masking masking for con- ditional masked language model. arXiv preprint arXiv:2303.07457.
Juntao Li Xiaobo Liang, Lijun Wu and Min Zhang. 2022. Janus: Joint autoregressive and aonaeutorerresivve training with auxiliary loss for sequence generation. In EMNLP, pages 1067-1073.
Pan Xie, Zexian Li, Zheng Zhao, Jiaqi Liu, and Xiao- hui Hu. 2022. Mvsr-nat: Multi-view subset regular- ization for non-autoregressive machine translation. TASLP.
Jiaao Zhan, Qian Chen, Boxing Chen, Wen Wang. Yu Bai, and Yang Gao. 2022. Non- autoregressive translation with dependency-aware de- coder. arXiv:2203.16220
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and William Dolan. 2020. Dialogpt: Large- scale generative pre-training for conversational re- sponse generation. ACL,System Demonstrations, pages 270-278.
Chunting Zhou, Jiatao Gu, and Graham Neubig. 2019. Understanding knowledge distillation in non- autoregressive machine translation. ICLR.

---

Example
Example
If affected person has waited for suffccintly long period of time red light, and provided the crossing clear they drive ex- plained Stuttgart 06-@ based lawyer Ralf Beeker in Motorrad Motorryclle magazine.
Aside from honouring Hugo the 31st Autumn Festival progressed usual: Alongside the Be- Male Voice Choir the Salzstetten Choral Club the Baisingen Choral Division the Local Music Society the Vollmaringen singers delivered colourful blend of different choral and song styles, which entertained the 400 or visitors
Source
Hat ein Betroffener lange genug einer roten Ampel gewartet und die Kreuzung ist frei, kann weiterfahren. erklärt der Stuttgarter Rechtsan- walt Ralf Becker in der Zeitschrift Motorrad
Neben der Ehrung verlief das 31. Herbstfest in gewohnten Bahnen Mit dem MGV Betra dem Liederkranz Salzstetten, der Sängerableilung Baisingen und dem örtlichen Musikverein hat- ten die Vollmaringer eine bunte Mischung ver- schiedener Chöre und Gesangsstile geladen welche die rund 400 Resucher unterhielten.
Reference
Wenn eine betroffene Person eine ausreichend lange 7eit im roten Lieht gewartet hat und sofern der Übergang klar ist kann sie weiter- fahren erklärte der Stuttgarter Rechtsanwalt Ralf Becker im Motorrad rad" " MMotoryccle
Neben der Ehrung von Hugo entwickelte sich das Herbstfest wie ublich Neben dem Betra Male Voice Chor dem Chorclub SalzSalzstet- ten der Chorabteilung Baisingen ingen und der lokalen Musikgesellschaft gaben die Vollmarin- gen ingen ingen ingen eine farbenfrohe Mis- chung verschiedenen Chor- und Songstilen, die 400 Besucher untertierten.
CMLM
Wenn der Betroffenc sehon eine ausreichend lange Zeit einem roten Licht gewartet hat und sofern der ühergang klar ist kann weit- erfahren erklärte der Stuttgarter Rechtsanwalt Ralf Becker in der 7eitschrift Motorrad "
Nehen der Ehren von Hugo schreitet das 31. Herbstfest wie üblich voran Neben dem Betra Male Voice Chor dem Salzstetten Chor Club, der Chorabteilung Baisingen und der Local Mu- sie Society lieferten die Vollmaringer SänSänger eine farbenfrohe Mischung versehiedenen Chor- und Liongstilen die 400 Besucher begleiteten.
CMLM w/Ours
Table Translation examples of CMLM w/ and our methods on WMT14 EN DE test set.
A
Case Study
We provide two analyses that experimented (shown in Table 7) on WMT14 EN-DE dataset to understand the effectiveness of our method better.
In Example 1, we can find that the result of CMLM obtains the words "Motorrad rad" which is grammatical error and the translation of word 'magazine' is missed. In contrast, our meth- ods enhance the dependency of token representa- tions to alleviate the above problems.
In Example 2, the words 'ingen" and " are repeated several times in the result of CMLM which is caused by the anisotropy problem. Our methods effective in alleviating this problem and provide more fluent translation.


Reviews:
[{"review_id": "vbqRtMIBsq", "reviewer": "Reviewer_ptyC", "summary": "", "questions": "1. How many iterations are required to obtain the \"CMLM w/ Ours\" results in Table 2? \n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your suggestions and comments, we give more explanations for your concerns and questions.\n\n**For your concern 1**  \n- In Table 2 of the paper, we compare our method with other representative iterative NAT models which also use CMLM as backbone, such as SMART, Disco, Imputer, CORR and CMLMC. Besides, our methods can also be applied to other CMLM-based models, such as CORR shown in Table 6. This verifys the effectiveness of our methods. \n\n**For your concern 2**  \n- Thanks for your advice, next we provide two case analysis which experimented on WMT14 EN-DE dataset to understand the effectiveness of our method better.\n\n  - In Example 1, we can find that the result of CMLM obtains the words \"Motorrad rad\" which is a grammatical error and the translation of word 'magazine' is missed. In contrast, our methods enhance the dependency of token representations to alleviate above problems.\n  - In Example 2, the words 'ingen' and ',' are repeated several times in the result of CMLM which caused by the anisotropy problem. Our methods are effective to alleviate this problem and provide more fluent translation.\n\n  | Example 1       |                                                              |\n  | --------------- | ------------------------------------------------------------ |\n  | **Source**      | If an affected person has waited for a sufficiently long period of time at a red light , and provided the crossing is clear , they can drive on , explained Stuttgart @-@ based lawyer Ralf Becker in &quot; Motorrad &quot; ( Motorcycle ) magazine. |\n  | **Reference**   | Hat ein Betroffener lange genug an einer roten Ampel gewartet und die Kreuzung ist frei , **kann er weiterfahren , erklärt der Stuttgarter Rechtsanwalt Ralf Becker in der Zeitschrift &quot; Motorrad &quot; .** |\n  | **CMLM**        | Wenn eine betroffene Person eine ausreichend lange Zeit im roten Licht gewartet hat und sofern der Übergang klar ist , kann sie weiterfahren , erklärte der Stuttgarter Rechtsanwalt Ralf Becker im &quot; **Motorrad rad** &quot; ( Motorcycle ) . |\n  | **CMLM w/Ours** | Wenn der Betroffene schon eine ausreichend lange Zeit an einem roten Licht gewartet hat und sofern der übergang klar ist , **kann er weiterfahren , erklärte der Stuttgarter Rechtsanwalt Ralf Becker in der Zeitschrift &quot; Motorrad &quot; .** |\n  |                 |                                                              |\n  | **Example 2**   |                                                              |\n  | **Source**      | Aside from honouring Hugo , the 31st Autumn Festival progressed as usual : Alongside the Betra Male Voice Choir , the Salzstetten Choral Club , the Baisingen Choral Division and the Local Music Society , the Vollmaringen singers delivered a colourful blend of different choral and song styles , which entertained the 400 or so visitors . |\n  | **Reference**   | Neben der Ehrung verlief das 31. Herbstfest in gewohnten Bahnen : Mit dem MGV Betra , dem Liederkranz Salzstetten , der Sängerabteilung Baisingen und dem örtlichen Musikverein hatten die Vollmaringer eine bunte Mischung verschiedener Chöre und Gesangsstile geladen , welche die rund 400 Besucher unterhielten . |\n  | **CMLM**        | Neben der Ehrung von Hugo entwickelte sich das 31. Herbstfest wie üblich : Neben dem Betra Male Voice Chor , dem Chorclub SalzSalzstetten , der Chorabteilung Baisingen ingen und der lokalen Musikgesellschaft gaben die Vollmaringen **ingen ingen ingen** eine farbenfrohe Mischung aus verschiedenen Chor- und Songstilen **, ,** an die etwa 400 Besucher untertierten . |\n  | **CMLM w/Ours** | Neben der Ehren von Hugo schreitet das 31. Herbstfest wie üblich voran : Neben dem Betra Male Voice Chor , dem Salzstetten Chor Club , der Chorabteilung Baisingen und der Local Music Society lieferten die Vollmaringer SänSänger eine farbenfrohe Mischung aus verschiedenen Chor- und Liongstilen , die etwa 400 Besucher begleiteten . |\n\n**For your Question**  \n- We obtain all the results with 10 iterations described in Table 2 I_{dec}.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper contains great motivation by exploring the anisotropic problem in non-autoregressive NMT models. \n2. The paper is well organized. \n3. Experiments are solid and comprehensive.  ", "reasons_to_reject": "1. Lack of comparisons between the proposed method and other models, which also use the CMLM as backbones.\n2. Lack of case analysis between different NAT models. It is helpful to understand the effectiveness of the proposed method by comparing outputs generated by different NAT models. \n", "questions_to_authors": "1. How many iterations are required to obtain the \"CMLM w/ Ours\" results in Table 2? \n", "paper_topic_and_main_contributions": "The authors delved deeper into the problems faced by iterative non-autoregressive models, specifically the issue of similar and indiscriminative representations of predicted target tokens, known as the anisotropic problem. They analyze the effectiveness of contrastive learning methods and propose to use the Look Neighbors strategy to enhance token representations during training. Experimental results on four WMT datasets demonstrate that our methods consistently improve performance and alleviate the anisotropic problem in conditional masked language models. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XtX86ZtTOp", "reviewer": "Reviewer_azMj", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your suggestions and comments, we give more explanations for your concerns and questions.\n\n**For your concern 1**\n- We are sorry to make the description vague in the paper, and may cause you to misunderstand it. Now, we make it clearer here. \n  - **The claim that \"failure to learn representations\" is a widely-recognized problem for all NAT models, but not caused by anisotropic loss.** The claim denotes that NAR models **fail to capture the dependency of target tokens** during training (Gu and Kong 2021; Xiao et al 2022; Zhan et al 2022), i.e., unlike AR models (e.g., vanilla Transformer) which can learn target-side dependency between one specific token and previous generated tokens, NAR models only depend on source tokens to make predictions due to the conditional independent factorization adopted in training process. \n  - **Anisotropic problem is another common problem for natural language models (Kawin Ethayarajh 2019, Su et al 2022), which has no direct connection with the problem \"failure to learn representations\" of NAR models.**  \n  Notice we just claim the \"inconsistent improvement\" in our paper but not that \"anisotropic loss is not helping\" as you said. As shown in Figure 2, adopting anisotropic loss is truly an effective way to mitigate anisotropic problem, but not effective enough to improve the performance. This is actually the main claim in our paper, we further explain it now: \n  - **As we know, anisotropic loss is optimized based on the token representations, so the effectiveness of anisotropic loss is closely related to the model's own ability to learn the token representations,** i.e., Easy to understand, blindly constraining incorrect token representations by anisotropic loss can't bring substantial improvement for performance although it truly make their representations more distinguishable. As a result, due to the character that \"fail to capture the dependency of target tokens\" for NAR models, anisotropic loss just only alleviates the traditional anisotropic problem but gets the limited improvement on BLEU as shown in Table 1. (we also conduct experiments on other directions as you mentioned w/o and w/ CL, WMT DE->EN: 29.59 vs. 29.77, and RO-EN 32.44 vs. 32.76, we will supply them in the paper.) **Therefore, we propose the Look Neighbors to enhance the learning of representation dependency.** \n\n**For your concern 2**\n- **We believe that we have present the difference of two problems clearly mentioned in the response to concern 1. The Look Neighbors is just to enhance the learning of token representations, but have no direct connection for the anisotropic problem.**   \nWhile better capturing the dependency of target tokens for NAR models, anisotropic loss can perform more effectively to improve the performance while alleviating the anisotropic problem. Besides, we explain why we adopt Look Neighbors to enhance the learning of NAR models. Compared with other methods, Look Neighbors is simple yet effective, and why only Neigbours, as shown in previous works, adjacent tokens are more important for the learning in NAR models (Liang et al 2023).\n\n**For your concern 3**\n- **\"Ours\" means CL + LN two methods in all of the Tables and Figures.**   \nThe experiment in Figure 3 is to explore the best pre-fined contrastive margin ρ. In figure 3, lower part line denotes CMLM without ours, so the performance is all the same. The other line denotes the results of our method (CL + LN) with different ρ (0.3, 0.4, 0.5, 0.6), the curve reaches its peak when ρ is 0.4, and notice that compared with 0.3, it has 0.8 BLEU improvement, we do not think this is close to each other. Besides, in Table 1, the result is closed in the second column. Notice this is why we claim that \"inconsistent improvement\" of anisotropic loss, and after we adopt Look Neighbours method to enhance the learning of NAR models, anisotropic loss can perform quite well to improve the performance, as shown in Table 2 (24.92 vs 26.68, 29.59 vs. 30.50, 32.84 vs. 34.01, 32.44 vs. 33.83). \n\n\n**The responses to 'Typos Grammar Style And Presentation Improvements'**\n\n- Thanks for your reminder, we have fixed them.\n\n**Reference:**\n\n- Gu J. and Kong X. ACL, 2021. Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade.\n- Xiao, Y.; Wu, L.; Guo, J.; Li, J.; Zhang, M.; Qin, T.; and Liu, T.-y. TPAMI, 2022. A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond. \n- Zhan, J.; Chen, Q.; Chen, B.; Wang, W.; Bai, Y.; and Gao, Y. Arxiv 2022. Non-autoregressive Translation with Dependency-Aware Decoder.\n- Kawin Ethayarajh. EMNLP, 2019. How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings.\n- Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Ling peng Kong, and Nigel Collier. NeurIPS, 2022. A contrastive framework for neural text generation.\n- Xiaobo Liang, Zecheng Tang, Juntao Li, and Min Zhang, ACL, 2023. Open-ended Long Text Generation via Masked Language Modeling.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper investigates a very interesting research question and perform some thoughtful experiments to answer it to limited extents.\n  * Specifically, the experiments on applying anisotropy regularization is interesting, but the data reported is quite limited.\n* The paper also proposes the Look Neighbor method which seems to have a positive impact on CMLM, though further investigation is warranted.\n* The introduction (up to line 85) is well-written and thought-provoking.", "reasons_to_reject": "* The reasoning is often times disconnected at various key moments in the paper. In paragraph from line 96 to line 109, as well as on lines 230-233, the authors abruptly transitions from anisotropic loss not helping to model's \"failure to learn representations\". This appears to be a key point in the paper that's repeated several times. However, there are several problems with it:\n  * \"Failure to learn the correct representation\" is too vague -- what *specific* failures is in question here? Too many things can be blamed under the umbrella \"not learning the correct representation\".\n  * There is almost no logical connection between anisotropic loss not helping to the conclusion that authors are trying to draw.\n  * There isn't sufficient data to support authors' claim that anisotropic loss is not helping. In table 1, it appears that the loss either improves or doesn't degrade CMLM's performance, and moreover, this table's data is inconsistent with the rest of the paper, where both directions for both language pairs are reported. Why are the DE->EN and RO->EN directions missing?\n* The Look Neighbors method isn't well motivated. It seems very disconnected with the \"isotropy\" topic of the present paper.\n* There are some confusions with the presentation. Since the isotropy loss (called \"CL\" in the paper) and Look Neighbors (LN) are two distinct methods, what does \"ours\" mean in Table 2? Is it CL+LN or just LN? And what about the ablation studies in Section  5? Are those models with or without CL? And what about Figure 3? Are the models in Figure 3 trained with only CL or CL+LN? Moreover, the numbers in Figure 3 seem to be very close to each other. It seems doubtful that we can draw any statistically significant conclusions from this graph (the same goes for the second column in Table 1).\n\nOverall, I think this is promising research direction and paper would be an interesting read if the above problems are fixed. The author might also consider splitting CL and LN into two separate papers because they seem like two different topics. However, I wouldn't recommend acceptance of the paper at the current form.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper is an exploration on the effects of anisotropy in conditional masked (CMLM) non-autoregressive machine translation models. The paper reports experiment results on CMLM models trained with regularization penalizing anisotropy. It also propose a \"Look Neighbor\" method promoting neighboring tokens' information when decoding the current position. The paper reports limited positive experimental results for both methods.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Line 141-142: this sentence is ill-formed.\n* Eq. 6 (line 209): From context, I think the denominator should be $|Y|(|Y|-1)$ instead of $|Y||Y|-1$."}, "extra_text": {}}, {"review_id": "YVowNJT25Q", "reviewer": "Reviewer_JroP", "summary": "", "questions": "n/a", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your suggestions and comments, we give more explanations for your concerns and questions.\n\n**For your concern 1**\n- We are sorry to make the description vague in the paper. Now, we make it clearer here. \n  - **The claim that \"failure to learn representations\" is a widely-recognized problem for all NAT models, but not caused by anisotropic loss.** The claim denotes that NAR models **fail to capture the dependency of target tokens** during training (Gu and Kong 2021; Xiao et al 2022; Zhan et al 2022), i.e., unlike AR models (e.g., vanilla Transformer) which can learn target-side dependency between one specific token and previous generated tokens, NAR models only depend on source tokens to make predictions due to the conditional independent factorization adopted in training process. The motivation for the Look Neighbors could be found in response to concern 2.\n  - **Anisotropic problem is another common problem for natural language models (Kawin Ethayarajh 2019, Su et al 2022), which has no direct connection with the problem \"failure to learn representations\" of NAR models.**  \n\n**For your concern 2**\n- We further explain the connections between CL and LN here: \n**As we know, anisotropic loss is optimized based on the token representations, so the effectiveness of anisotropic loss is closely related to the model's own ability to learn the token representations**, i.e., Easy to understand, blindly constraining incorrect token representations by anisotropic loss can't bring substantial improvement for performance although it truly make their representations more distinguishable. As a result, due to the character that \"fail to capture the dependency of target tokens\" for NAR models, anisotropic loss just only alleviates the traditional anisotropic problem but gets the limited improvement on BLEU as shown in Table 1. Therefore, we first propose the Look Neighbors to enhance the learning of representation dependency. While better capturing the dependency of target tokens for NAR models, anisotropic loss can perform more effectively to improve the performance while alleviating the anisotropic problem. You can refer the results of table shown in response to concern 3 to further understand the connections. \n- For your another concerns, we make more detailed presentations. Firstly, **LN could be used without CL**, we provide the ablation study in Section 4.4 'Effect of Look Neighbors'. Besides, we compute the cosine similarity with CMLM w/ LN, and find that **it is consistent with CMLM and doesn't make near neighborhood more similar.**\n\n**For your concern 3**\n- Now, we provide the results of CMLM w/ LN, w/ CL and w/ both. As shown in table, simply adoting LN can consistently improve the performance on all datastes. Besides, CL and LN are independent not contradictory. They are independent according to two different well-known problems. LN can make CL more effective in improving the performance while alleviating the anisotropic problem. \n\n|              | WMT14 EN-DE | WMT14 DE-EN | WMT16 EN-RO | WMT16 RO-EN |\n| :----------: | :---------: | :---------: | :---------: | :---------: |\n|     CMLM     |    24.92    |    29.59    |    32.84    |    32.44    |\n|  CMLM w/ CL  |    25.59    |    29.77    |    32.88    |    32.76    |\n|  CMLM w/ LN  |    25.41    |    30.36    |    33.43    |    33.02    |\n| CMLM w/ both |    26.68    |    30.50    |    34.01    |    33.83    |\n\n**Reference:**\n\n- Gu J. and Kong X. ACL, 2021. Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade.\n- Xiao, Y.; Wu, L.; Guo, J.; Li, J.; Zhang, M.; Qin, T.; and Liu, T.-y. TPAMI, 2022. A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond.\n- Zhan, J.; Chen, Q.; Chen, B.; Wang, W.; Bai, Y.; and Gao, Y. Arxiv 2022. Non-autoregressive Translation with Dependency-Aware Decoder.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors presents many insightful experimental results (thought I find many of them may not well contribute to the main theme of this paper).", "reasons_to_reject": "1. I find the motivation for the Look Neighbors (LN) mechanism vague. Why do you mean by \"failure to learn the correct representation\"? This almost conveys no information because one can always says the representation is not correct when the results are not good.\n2. What are the connections between contrastive loss (CL) and LN? Can LN be used without CL? (ablation study is needed) I find that the combinations of these two makes the paper even confusing. Because contrastive loss is meant to make the representations at different positions different. On the other hand, LN intends to make the representations focus on near neighbors rather than distant neighbors. I think it may make near neighborhood even more similar.\n3. Comparing Table 1 and Table 3, WMT14 EN → DE: 24.92 ->25.59 with CL and WMT14 EN → DE: 24.92 ->25.41 with LK. It seems that the LK mechanism has negative impact on the performance compared to simple CL. This makes the effect of LK suspicious.\n\nOverall, I think the motivation for the proposed LK mechanism has been not well-justified either theoretically or empirically.", "questions_to_authors": "n/a", "paper_topic_and_main_contributions": "This paper tackles non-autoregressive text generation, particularly in the applications of machine translation.  Inspired by recent study on the anisotropic property of language models (the hidden representations at different positions are too similar and indiscriminate), the authors propose to encourage dissimilarity by a contrastive loss. they further propose a Look Neighbors mechanism. Experiments show that  the proposed method obtain better results than vanilla NAR models and many existing baselines.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "n/a", "Typos_Grammar_Style_and_Presentation_Improvements": "n/a"}, "extra_text": {}}, {"review_id": "L8Pv9BcxyQ", "reviewer": "Reviewer_Nvxc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your suggestions and comments, we give more explanations for your concerns and questions.\n\n**For your concern 1**\n- Our methods gain the average of **1.3** BLEU improvements compared with traditional CMLM on all datasets, and **our methods achieve 26.68 on WMT14 EN->DE which set the new SoTA performance**. What's more, it's compatible to apply our methods on the other iterative NAR models such as CORR (Table 6) to further improve the performance.\n\n**For your concern 2**\n- Thanks for your advice. CMLM is a classic and representive model that many related work focus on, so we pay attention to explore it. For two related work you propose: \n(1) Actually, we have adopted the similarity and reconstruction regularizations methods proposed in NAT-REG on CMLM and found that it couldn't alleviate the anisotropic problem. \n(2) We present the difference between our Look Neighbors and LAVA-NAT's look-around from line 267 to 271 in our paper. The look-around is a decoding strategy to enhance the inference schedule without enhancing the training process, our method works during training and inference.\n\n**For your concern 3**\n- Anisotropy problem is common on Transformer-based models, about why it arises and how it impacts generation could be found in (Kawin Ethayarajh 2019). Our paper mainly pay attention to alleviate this problem and improve the performance.\n\n**For your concern 4**\n- Thanks for your reminder, we will add it in the next version. From Diff-GLAT, we can see that our methods outperform diff-GLAT (CTC) on WMT14 EN-DE. Combining with strong baseline DAT making diff-GLAT improve greatly, we will adopt our methods on DAT for fair comparison.\n\n**Reference:**\n- Kawin Ethayarajh. EMNLP, 2019. How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Addresses an important problem (anisotropy) in an impactful area (iterative NAR MT). Improving NAR is an active area of research.\n- Provides solid empirical analysis of the problem using visualizations and metrics.\n- Proposes two simple but effective solutions that consistently improve performance across multiple datasets. And ablations validate the contribution of them.\n- Compatibility with other recent advances like knowledge distillation and self-correction is demonstrated.", "reasons_to_reject": "- The performance gains are decent but not dramatic compared to prior iterative NAR methods. \n- The analysis is mainly limited to one model (CMLM). More NAR models could be explored. The paper does seem to be missing a discussion of some important related work on token representations in non-autoregressive models: (i) NAT-REG (Wang et al. 2019) - Proposed similarity and reconstruction regularizations to normalize decoder token representations in NAR models. This seems highly relevant to the goal of reducing anisotropy. (ii) LAVA-NAT (Li et al. 2020) - Uses a \"Look-Around\" strategy at inference time to incorporate context representations. Related to the proposed \"Look Neighbors\". Comparing to NAT-REG may help better motivate the need for contrastive learning and neighbor-aware training. The connection to LAVA-NAT's look-around decoding could also be clarified.\n- There could be more exploration into why exactly anisotropy arises and how it impacts generation.\n- Missing strong baselines.  Diff-GLAT (https://arxiv.org/abs/2212.10240)", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addresses the problem of anisotropic token representations in iterative non-autoregressive neural machine translation models like CMLM. The main contributions are:\n\n- Analyzing and confirming the anisotropic problem (similar token representations) exists in iterative NAR models like CMLM.  \n- Proposing two methods to alleviate this problem: (i)Using contrastive learning to make token representations more discriminative. (ii) Introducing a \"Look Neighbors\" strategy to enhance learning of adjacent token representations.\n- Quantitatively and qualitatively showing the proposed methods can reduce token similarity and improve performance across multiple WMT datasets. \n- Demonstrating the proposed methods are compatible and complementary to other recent improvements for iterative NAR like knowledge distillation and self-correction.\n\nIn summary, this paper identifies and provides solutions for the anisotropic token representation problem in iterative non-autoregressive translation models. The analysis and proposed methods help improve performance and point towards future work on better token representations for NAR.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper identifies an important problem in iterative NAR model -- the anisotropic problem where the high similarity of different tokens can mislead the model to generate repetitive tokens at different steps. This finding is inspired by recent work on AR models (Su et al. 2022). The authors revisit the solution from Su et al (2022) -- contrastive learning, and find that it fails to learn the token representation well during training due to the natural difference between NAR and AR models. In response to this problem, the authors propose Look Neighbors to supplement contrastive learning during the training of CMLM. Experiments on widely-used NAR translation benchmarks demonstrate the effectiveness of the proposed approach.

Some necessary information are missing in the submitted version, which are supplemented in the rebuttal. Please include the key information in the revised version:
1. Supplement necessary information about the claims associated with "failure to learn representations" and "anisotropic problem/loss" to make the motivation and design principles more clear.
2. Clarify the motivation of Look Neighbors method, and add an ablation study to prove that the performance improvement is indeed from the method.
3. Experiments on more NAT models are appreciated to demonstrate the universality of the conclusions and the proposed approach.
Decision: Accept-Findings