Duplex Sequence-to-Sequence Learning for Reversible Machine Translation
Authors: Zaixiang Zheng
Hao Zhou
Shujian Huang
Jiajun Chen
Jingjing Xu
Lei Li
Venue: NeurIPS 2021 Poster (2021)
Keywords: duplex network
sequence-to-sequence learning
reversible machine translation

Abstract: Sequence-to-sequence learning naturally has two directions. How to effectively utilize supervision signals from both directions? Existing approaches either require two separate models, or a multitask-learned model but with inferior performance. In this paper, we propose REDER (Reversible Duplex Transformer), a parameter-efficient model and apply it to machine translation. Either end of REDER can simultaneously input and output a distinct language. Thus REDER enables {\em reversible machine translation} by simply flipping the input and output ends. Experiments verify that REDER achieves the first success of reversible machine translation, which helps outperform its multitask-trained baselines by up to 1.3 BLEU.

Duplex Sequence-to-Sequenne Learning for Reversible Machine Translation
Zaixiang Zheng : Hao Zhou?, Shujian Huang', Jiajun Chen', JJigjing Xu², Lei Li
National Key Laboratory for Novel Software Technology, Nanjing University
BytelDance Al Lab UUC Santa Barbara
(zhengzaixiang, zhouhao mlp, xuj sujjingjinn log.giinnmmmlllllo..m..... @bynedancee...... com (huangsj, chenjj jhenjj]@u jojju edu. en, lilei@ucsb. edu
Abstract
Sequence-to-seet to-suueece learning naturally has two directions. How to effectively utilize supervision signals from both directions? Existing approaches either require two separate models, or multitask-learee model but with inferior performance. In this paper, we propose REDER (REversible Duplex TransformER), parameter- efficient model and apply it to machine translation. Either end of REDER can si- multaneously input and output distinct language. Thus REDER enables reversible machine translation by simply flipping the input and output ends. Experiments verify that REDER achieves the first success of reversible machine translation, which helps outperform its multitask-traine baselines up to 1.3 BLEU.
Introduction
Neural sequence-to-sequenec (seq2seq) learning (Sutskever et al. 2014] has been extensively used in various applications of natural language processing. Standard seq2seq neural networks usually employ the encoder-decoder framework, which includes an encoder to acquire the representations from the source side, and a decoder to yield the target side outputs from the encoded source represen- tation [Bahdanau al. 2015 Gehring et al. 2017 Vaswani et al. 2017].
Generally, given paired training data D.y X V, where X' the source side and y is target side of the corresponding seq2seq task. supervision signals are always bidirectional. Thus we can learn not only the mapping from source to target (fe.,, X + Y) but also the mapping from target to source (fe, y + X).. This is very common in many applications. For example, given parallel corpus, we can obtain machine translation models from both Chinese English and English Chinese, or transfer between different stylized texts (Yang et al. 2018 He et al. 2019].
Typical seq2seq learning utilizes the bidirectional supervisions by splitting the bidirectional supervi- sions into two unidirectional ones and trains two individual seq2seq models on them, respectively. However, such splitting ignores the internal consistencies between the bidirectional supervisions. Thus how to make better use of the bidirectional supervisions remains an open problem. One potential solution multitask learning [Johnson al., 2017]. which jointly leverages the bidirectional supervi- sions within one model and expects the two unidirectional supervisions could boost each other (Figure [(b)). But it does not work well in our case due to the parameter interference problem (Arivazhagan et al. 2019 Zhang et al. 2021]. For instance, in the setting of multitask Chinese-tor Prressett--Engee to-English and English-to-Chines bidirectional translations, the encoder/decoder of seq2seq networks is trained to simultaneously inieesstand/geeeratt both Chinese and English, which always results in performance
"Work was done when Zaixiang Zheng final-year year PhD candidate Nanjing University and intern (now FTE) at ByteDance Lab: and when Lei Li was also ByteDance Lab.
"Code is available at htttt:: thub. cow wwwzezxenngxxeeeeeeeeeeeeeeeeemr nlp/REDER
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

drop due to the divergent nature of the two languages. Another branch of solutions lies in cycle training [Sennrich al., 2016 He al. 2016 Zheng et al. 2020], which still deploys two individual models for separately learning, but explicitly regularize their outputs by the cycle consistency of seq2seq problems. Such approaches usually need introduce extra monolingual data in machine translation.
In this paper, we propose to explore an alternative approach for utilizing bidirectional supervisions called duplex sequence-to sequence-to-ssquuence learning. We argue that current seq2seq networks do not benefit from multitask training because the seq2seq net- works are simplex from the view of telecommunica- tions] Specifically, the data stream only flows from encoder to decoder in the current seq2seq learn- ing, and such simplex property makes the multitask learning suffer from the parameter interfere prob- lem. Instead, if we have duplex neural networks, where the data stream can flow from both ends, each of which only specializes in one language when learning from bidirectional supervisions (Fig- ure Tc)) thus potentially alleviating the parameter interference problem. As a result, the bidirectional translation can be achieved as a reversible and uni- fied process. In addition, could still incorporate cycle training in the duplex networks using only one single model.
simplex commuitcation
simples
Enooder
Decoder
Encoder
Decoder
(a) simples end-dec mode
[8 wiep lax unc-deee model N/ milti-task tearning
duplay
source language
target Language
roural networks
forvard direction
reverse direction ion
(c) RESER
Figure 1: Illustration of different sequence-to- sequence neural models in regards to modeling direction and generation formulations.
Definition
giquence-to-sqqueen neural network with parameter is duplex if it satisfies the following: its network has two ends, i.e., source end and target end; both source and target ends can take input and output sequences; its network defines a forward mapping function fo X + Y. and reverse mapping function f
+ X, that satisfies the following reversibility: fō = =(fa²) and fø* = =(fō besides, it satisfies the following cycle consistency: dx X: fē (fa'(x)) a and by fo (z)) y.
the Reversible Duplex TransformE and apply it to machine translation. Note that, building duplex seq2seq networks non-trivial: a) vanilla nncoderrdecoder network is irreversible. The output end of the decoder cannot take in input signals exhibit the encoding functionality and vice versa; b) the topologies of the encoder and the decoder are heterogeneous, i..., the decoder works autoregressively, while the encoder is -otreeroreerrsivvee To this end, we therefore design REDER without explicit encoder and decoder division, in which we introduce the reversible Transformer layer [Gomez et al.) [017]] and fully non-autoregressive modeling to solve the above two problems respectively. As a result, REDER works the duplex fashion, which could better exploit the bidirectional supervisions for achieving better downstream task performance.
Based on the idea of duplex network, we propose REDER
Experimental results show that the duplex idea indeed works: Overall, REDER achieves BLEU scores of 27.50 and 31.25 on standard WMT14 EN-DE and DE-EN benchmarks, respectively, which are top results among non- non-autoregressive machine translation models. REDER achieves significant gains (+1.3 BLEU) compared to its simplex baseline, whereas multitask learning does hurt the translation performance of simplex models both in the autoregressive (-0.5 BLEU) and non- autoregressive settings (-1.3 BLEU). Although REDER adopts fully non-autoregressive modeling to realize the duplex networks, the gap of BLEU between REDER and autoregressive Transformer negligible, meanwhile REDER can directly translate between two directions and enjoys 5.5 inference speedup bonus of -on-auvoregressive modeling. To our best knowledge, REDER is the first duplex seq2seq network, which enables the first feasible reversible machine translation system.
In leleoommuiications and computer networking, the simplex communication means the communication channel is unidirectional while the duplex communication bidirectional. The model's name is palindrome, which implies the model works from both ends.

---

2 REDER for Reversible Machine Translation
In this section, we introduce how to design a duplex neural seq2seq model, the REDER (Reversible Duplex TransformER), that satisfies Definition [ and its application in machine translation that realizes the first neural reversible machine translation system.
2.1 Challenges of Reversible Machine Translation
Reversible natural language processing [Franck] 1992 Strzalkowski 1993 and its applications in machine translation [van Noord 1990 were proposed for the purpose of building machine models that understand and generate natural languages reversible, unified process. Such process resembles the mechanism of the ability that allows human beings to communicate with each other via natural languages [Franck] 1992]. Despite the success of neural machine translation with deep learning, designing neural architectures for reversible machine translation yet remains under-studied and has the following challenges:
Reversübility. Typical encoder-decoder networks and their neural components, such as Transformer layers, irrevesiibe, iee., cannot just obtain its inverse function by flipping the same encoder- decoder network. To meet our expectation, an inverse function the network should be derived from the network itself.
Homogeneity. Intuitively, a pair of forward and reverse translation directions should resemble homogeneous process of understanding and generation. However, typical encoder-decoder networks certainly do not meet such computational homogeneity due to extra cross attention layers in the decoder; and also because of the discrepancy that the decoder works autoregressively but the encoder does oon-uuoregressively. To meet our expectations, the separation of encoder and decoder should be no more exist in the desired network.
The Architecture REDER
To solve the above challenges, we include two corresponding solutions in REDER to address the reversibility and homogeneity issues respectively, i.e., the Reversible Duplex Transformer layers, and the symmetric network architecture without the encoder-decoder framework.
Figure 2 shows the overall architecture of REDER. As illustrated, REDER has two ends: the source end (left) and the target end (right), 0 is the model parameter, shared by both directions. The architecture of REDER is composed of series of identical Re- versible Duplex Transformer layers. When performing the source-tor source-to-targer to-target mapping fo a source sentence x (blue circles) 1) first transforms to its embedding e(x) and en- ters the source end; 2) then goes through the entire stacked layers and evolves fi- nal representations H, which predicts prob- abilities; 3) finally its target translation (or- ange circles) will be generated from the target ends. The generation process is fully non- autoregressive.
Duples Communication
source
ragee
Rd NeS
WRS
Ma
Werl
SaN
Sol
Fll
Merfist-style Trasafomer Laper: [
SaN
FFN
SAN
FrN
Reppler Perm (T)
Dawarz L
Figure 2: The proposed REDER for duplex sequence- to- se-sequence generation. The bottom two diagrams show the computation of the regular and reverse forms of reversible layer. Notice that, make the whole model symmetric, we reverse the 1-th to T./2-th lay- ers, such that the overall computational operations of forward and reverse of REDER are homogeneous.
Likewise, the target-to- touercc mapping f is achieved by reversely executing the architec- ture of REDER from target end to source end. We will dive into the details of the key com- ponents of REDER in the following parts.
Reversibility: Reversible Duplex Transformer layers. adopt the idea of the reversible residual network (RevNet, Gomez et al., 2017 Kitaev et al. (020) in the design of the reversible duplex Transformer layer. The bottom of Figure illustraees the forward and reverse computations of layer. Each layer is composed of multi-head self-attention (SAn) and a feed-for forward network (SAN) with

---

special reversible design to ensure duplex behavior, where the input and output representations of such layer are split by halves, i.e., [H H (2) and H ( Formally, the regular form of the (-th layer Fi performs as follow:
H Fi ( H (1) H _1 [
(-1 H
where
H (1) = SA 21), H H H (FN(H)))
The reverse form of Fi can be computed by subtracting (instead of adding) the residuals:
H -11 HH)) H H = (1) H
where
H (2) H 2) (H
- (
For better modeling the reordering between source and target languages, we employ relative self- attention [Shaw et al. 2018 instead of the original one (Vaswani et al. 2017].
Homogeneity: Symmetric network architecture without encoder-decoder framework. To meet need ensure homogeneous network computations for forward and reverse directional tasks, therefore choose to discard the encoder-decoder paradigm.
Symmetric network. To achieve homogeneous computations, one solution is to make our network symmetric, as depicted at the top of Figure 2 Specifically, we let the 1-th to L/22th layers be the reverse form, whereas the latter (L/2 + 1)-th to L-th be the regular form:
(x) Fi'
File oF
FL/2+1
FL()),
(V)
FL FL/2+1
F1/2 o FT)
where means a layer is connected the next layer. Thereby the forward and reverse computations of REDER become homogeneous: the forward computational operation series reads as palindrome string (fs (f fssf sf) and so does the reverse series, where s and f denote SAN and FFN.
Fully non non-autoregresssvve modeling. Note that without encoder-decoder division. REDER works in fully non-autoregressivve fashion in both reading and generating sequences. Specifically, given an input sequence æ. Ho.: = [ (2) [e(æ;); e(x;)], )]] is the i-th element of REDER's input, which is the concatenation of two copies of the embedding of x,. Once a forward computation is
done, the concatenation of the output of the model ( HL serves as the representations of target translation. And then, a softmax operation performed measure the similarity between the model output [H (1) (2)
2, L and the concatenated embedding of ground-truth reference (e(ys), e(v;)),
to obtain the prediction probability:
pl 0) softmax((e(w))::(()))
11122).
We likewise derive the procedure of f for the target-to-souree direction. Due the conditional independence assumption among target tokens introduced by non-autoregressive generation, the log-likelihood of translation becomes:
log logp(y|x;;0 Σ
log pe (31(z)
Modeling variable-length input and output. Encoder-decoder models can easily model variable- length input and output of most seq2seq problems. However, discarding encoder-decoder separation imposes
challenge: the width of all the layers of the network is depending the length of the input, thus it is very difficult to allow variable-length input and output, especially when the input is shorter than the outpur. We resort the Connectionist Temporal Classification (CTC) [Grvvss et al.) 2006) to solve this problem, a latent alignment approach with superior performance and the flexibility of variable length prediction. Given the conditional independence assumption, CTC is capable of efficiently finding all valid alignments a which derives from the target y by allowing consecutive repetitions and inserting blank tokens, and marginalizes logllioel::ood:
log Pete k0 0) = log Σ
pe (a)
acT(

---

where (a) is the collapse function that recovers the target sequence by collapsing consecutive repeated tokens, and then removing all blank tokens. Note that CTC requires that the length of source input should not be smaller than the target output, which is not the case in machine translation. To deal with this, we follow previous useful practice by upsampling the source tokens by 2 times [Saharia et al. 2020 Libovický and Helel, (018), and filter those examples when the target lengths are still larger than the one upsampled source sentences.
Remark. Reversibility in REDER be assured in the continuous representation level, where REDER can recover from output representations (last layer) to input embeddings (first layer), which also the motivation and basis of the auxiliary learning signal, i.e. Liba- in the next section. Reversibility might not hold in the discrete token level, because of the existence of irreversible operations, e.g. the argmax operation discretizes probabilities tokens and the CTC collapse process. But REDER still shows decent reconstruction capability in practice, as visually depicted in the experiment section.
2.3
Training
Given a parallel corpus and single model REDER can be jointly supervised by source-to-target and target-to-source translation for fū and f respectively. Thus both translation directions can be achieved one REDER model. We refer this to bidirectional training, which opposite to unidirectional training, where each translation direction needs separate model. Moreover, the reversibility of REDER enables appealing potentials to exploit consistency/agrremeet between forward and reverse directions. We introduce two auxiliary learning signals follows.
Layer-wise -arcaadddd Agreement Since REDER is fully reversible, which consists of series of computationally inverse of intermediate layers, an interesting question arises: given the desired output (i.e., the target sentence), it possible to derive the desired intermediate hidden representation by the backward target-to-soure tourrce computation, and encourage the forward source-to- target intermediate hidden representations as close as possible to these "optimal" representations?
Given a source sentence æ, the inner representations of each layer in the forward direction are:
H. Fi (e(x)), H. F=. H. FH
and given its corresponding target sequence y as the optimal desired outpu F the inner representations of each layer in the reverse direction are:
H. Fr (e(v)), H
= (H)).
where H. and H, represent the representations of 1-th layer in forward and reverse models, respec- tively. As consider these reverse inner layer representations as ""ottimal", we try minimize the cosine distance between the forward and backward corresponding inner layer representations:
Casa (xly;; 0)
Σ 1- 231-cos(in sg sg( H.)),
1=1
where sg(-) denotes the stop-gradient operation.
Cycle Consistency The symmetry of a pair of seq2seq tasks enables the use of cycle consistency (He ct al. 2016 Cheng et al. 2016a). Given a source sentence x, we first obtain the forward prediction, and then we use the REDER to reconstruct this prediction to the source language:
y fo (z),
fo
Finally, we maximize the consistency or agreement between the original one æ and reconstructed one T. Thus, the loss function reads
Lee (x:; 0) distance (x. fo (f distance
We expect can provide an auxiliary signal that a valid prediction should be loyal to reconstruct its source input. Here distance to measure the consistency.
cross-entropy between the probabilistic prediction of the reverse model as
for CTC based model where the model predictions are the alignments, we instead extract the token sequence of the best alignment a', predicted by the model, associated with the ground-truth the optimal desired output.
5

---

A potential danger of both of the above auxiliary objectives is learning degenerate solution, where would probably cheat this task by simply learning an identity mapping. We solve this problem by setting two-stage training scheme for them, where we first train REDER without using any auxiliary losses until a predefined number of updates, and then activate the additional losses and continue training the model until convergence.
Final Objective Given a parallel dataset Day = ((x)(1))))))))))))))))))) y I...N} of i.d observations, the final objective of REDER is minimize
[(0; (wy)
Σ n=1
log Pete (M) (m) log Pete a (y)(())
+ Aba Liba (x) ()) + XBsa Liba (y (n) (o))
layer-wise forwand-bacckware agreements
+ Acc (xo 0) + Acc (n)
eycle consistencies
where Abaa and Ace are coefficients of the auxiliary losses.
3 Related Work
Models Exploiting Bidirectional Signals. Several studies try utilize bidi- rectionality as a constraint to improve sequence- to to-sequence tasks such as machine translation [Cheng et al., 22166a b). Dual learning (He et al.) 2016 Xia et al. 2017 leverages reinforcement learning to interact between two simplex translation models. Later, Xia et al. [2018) propose a partially model-level dual learning that shares some components of both models for forward and reverse tasks. Zheng et al. 22200) propose to model the two directional translation model with language models in a variational probabilistic framework. These approaches model two directional tasks by setting up two separate simplex models to consider the task bidirectionality. Different from them, REDER can unify a pair of directions within one duplex model and directly model the bidirectionality at a completely model level. Besides, other studies try to unify two directional tasks by multitask learning [Johnson] al. 2017 Chan et al., 2019 by sharing the same computational process of single simplex model, which would inevitably result in parameter interference issue where two tasks compete for the limited model capacity. In contrast, REDER formulates both tasks in one model by simply exchanging input and output ends, each of which specializes in a language, thus bidirectional translation becomes a reversible process which both directions do not need to compete for limited model capacity.
Sequence-to-Seqqeee
Non-autoregressive Sequence Generation. Non-autorrgresssive translation (NAT) models (Gu et al. 2018 aims to alleviate the decoding inefficiency of traditional autoregressive seq2seq models. Fully NAT models could generate sequence in parallel within only one shot but sacrifice performance Ma
al. 2019 Shu et al. 2020 Bao al., 2019 Wei et al. [219 Qian et al., 2021 Gu and Kong 2021
Huang et al. 2021]. Besides, coon--autoreressivee models greatly improve the performance of NAT
models, which perform iterative refinement of translations based on previous predictions [Lee et al.
2018. Ghazvininejad et al. 2019 Gu et al. 2019| Kasai et al.) 20200 Ghazvininejad et al. 2020 In this work, REDER takes the advantage of the probabilistic modeling of fully NAT models for resolving the designing challenge of computational homogeneity for both translation directions.
Reversible Neural Architectures. Various reversible neural networks have been proposed for differ- ent purposes. On one hand, reversible neural networks help model flexible probability distributions with tractable likelihoods [Dinh et al. 2014 2017 Kingma al. 2016). which define mapping between simple, known density and a complicated desired density. On the other hand, reversibility can also assist to develop memory-efficient algorithms. The most popular approach is the reversible residual network (RevNet, [omez et al., 2017), which modifies the residual network and allows the activations at any given layer to be recovered from the activations at the following layer. Therefore layers can be reversed one by one as back-propagation proceeds from the output of the network to its input. Some follow-up work extends the idea of RevNet to RNNs (MacKay et al. 2018 and Transformer Kitaev et al. 2020 in NLP. We borrow the idea of RevNet as the basic unit of our proposed REDER, however, for different purposes that we want to build a duplex seq2seq model to govern two directional tasks reversibly. In this line, van der Ouderaa and Worrall (0099) propose

---

reversible GAN approach for image-to- -o-mage translation in computer vision, which to a certain extent shares the intuition with ours.
Experiments
We conduct extensive experiments on standard machine translation benchmarks to inspect REDER's performance on seq2seq tasks. We demonstrate that REDER achieves competitive results, if not better, compared to strong autoregressive (AT) and non-autoreerressive (NAT) baselines. REDER is also the first approach that enables reversible machine translation in one unified model, where bidirectional training with paired translation directions surprisingly helps boost each of them with substantial gains.
4.1 Experimental Setup
Datasets. We evaluate our proposal on two standard translation benchmarks, ie., WMT14 English (EN) ++ German (DF) (4.5M training pairs), and WMT116 English (EN) <> Romanian (Ro) (610K training pairs). We apply the same prepossessing steps mentioned in prior work (EN+>DE: Zhou ctt al. 2020 EN++Ro: Lee et al.) 2018). BLEU Papiieni et al.) 2002] is used to evaluate the translation performance for all models.
Knowledge Distillation (KD). Sequence- knowledge distillation [Kim and Rush, 2016] is found to be crucial for training NAT models. Following previous 2020], REDERs are trained on distilled data generated from pre-trained auto-regressive Transformer models. The beam size is set to 4 during generation.
NAT studies [Gu et al. [218]]218]] Zhou et al.
Beam Search Decoding and AT Reranking. We implement two kinds of inference policies. The first one is parallel decoding that adopts tokens with the highest probability at each position. For CTC-based models, we also implement beam search to REDER with an efficient library of C++ implementatiorf2 Similar to previous NAT literature, we further rerank the decoded candidates produced by beam search using autoreregressive models as the external scorer [Gu al. 2018 and pick the best ones as final results.
Implementation Details. We design REDER based on the hyper- srper-parameters of Transformer- base VVaswani et al., 2017]. All models are implemented on fairseq [Ott et al.) 2019]. REDER consists of 12 stacked layers. The number of head is 8, the model dimension is 512, and the inner dimension of FFN is 2048. For both AT and NAT models, we set the dropout rate 0.1 for WMT14 EN++DE and 6 ENERR.. We adopt weight decay with decay rate 0.01 and label smoothing with = 0.1. By default, we upsample the source input by factor of 2 for CTC-based models. We Arba and Ace 0.1 for all experiments. All models are trained for 300K updates using Nvidia V100 GPUs with a batch size of approximately 64K tokens. Following prior studies (Vaswani et al. 2017], we compute tokenized gase-sensitive BLEU, We measure the validation BLEU scores for every 2,000 updates, and average the best 5 checkpoints to obtain the final model. Similar to previous NAT studies, we also measure the GPU latency by running the model with one sentence per batch on WMT14 EN-DE test set on a single GPU and give speedup comparing over our AT baselines [
4.2 Main Results
As shown in Table [ we compare REDER with AT and NAT approaches with and without multitask learning, as well as existing approaches that also leverage bidirectional learning signals.
REDER achieves competitive results compared with strong NAT baselines. We show that unified REDER trained on the same parallel data can simultancously work in two directions, which has comparable capability as strong NAT models such GLAT (Qia et al. 202 (row 14 vs. row 8). With the help of beam search and re-ranking, the performance of REDER can be further boosted to be comparable with the state-of-the-art NAT method GLAT+CTC (Gu and Kong 2021 (row 14 vs.
row 9 & 10). Gu and Kong (2021) explore the best technique combination for NAT, whose tricks can also supplement enhance REDER. We leave this for exploration.
"Note that as this paper's s goal is not for decoding efficiency, all models are not optimized for latency using advanced techniques. and the autoregressive baselines hence weak in terms of latency.
7

---

Table 1: Comparisons between our models and existing models. All NAT models are trained with KD. "t>": whether to allow bidirectional translation. "MTL": multitask learning. "BT": fack-translation The speedup is measured with batch size of 1. Notice that speedups from previous papers are generally not fully comparable due to inconsistent hardware and baselines and hence only for reference. All our implemented CTC-based NAT models (row 10 ~ row 14) employ beam search decoding with beam
size of 20, whereas NAT models from previous literature (row 4 ~ row 9) employ greedy decoding.
WMT14
WMTI6
Systems
[8]
Speed
DE-EN
EN-Ro
Ro-EN
Transformer-base (KD teacher) w/ MTL
x
62Mx
1.0x
27.60
31.50
33.85
33.70
AT
62M
1.0x
27.06
30.96
w/ BT
X
62Mx2
1.0x
27.82
31.91
4 vanilla NAT Gu et al. 2018 3 CTC w/o KDJ Libovicky and Helel 2018
X 62Mx2
15.6x
17.69
21.47
27.29
29.06
x 58Mx2
16.56
18.64
19.54
24.67
CTC Saharia et al. 2020
58Mx2
18.6x
25.70
28.10
32.20
31.60
Impuder Sahana
2020
X 58M
18.6x
25.80
28.40
32.30
31,70
NAT
GLAT Chan et
2021
x 62Mx2
1
26,55
31.02
32.87
33,51
9 GIAT CTC Cu and Kong
2021
x
62Mx2
16.8%
27.20
31,39
33.71
34.16
our re-impiementations of Gu and Kong 2021
T1 GLAT+CTC
62Mx2
26.79
30.45
1
w/ MTL
62M
16.2x
25,50
29.49
simplex REDER
x 58MX2
5.5x
26.20
30.02
32.67
32.98
REDER
w/ MTL
58M
5,5x
25.58
29.12
duplex REDER (final model)
58M
5.5x
27.50
31.25
33.60
34.03
Reformer
Kitaev et al. 2020
X
62Mx2
27.60
Model-leve Dig
Xta ar
2018
X
210Mx2
28.90
31.90
previous studies
KERMIT CCan et al 2019
X
124M
25.60
27.40
KERMIT
mono Chan al
2019
X 124M
28.10
28.60
MGNMT Zheng et al. 2020
195M
27.70
31.40
32.70
33,90
Duplex learning has more potential than multitask learning and back-translation. Given the same parallel corpus as training data, multitask learning (MTL) yields considerable performance degradation of either AT (row vs. row 2). By re-implementing GLAT+CTC [Gu and Kong 2021] (row 10& & 11) as the strong NAT competitor for more convincing comparison. We observe that MTL would hurt more severely for NAT models, such as GLAT+CTC models (row 11 vs. row 10), and REDER as well (row 13 vs. row 12). These results verify our concern of multitask-lcamen models regarding parameter interference. Meanwhile, when no external monolingual resources are available, fack-translation (BT) only adds mild points from training data. In contrast, duplex learning allows REDER gain more benefits, becoming better alternative and parameter-efficies choice to exploit more potentials from the provided parallel data. Plus, duplex learning is orthogonal to BT, which could further improve REDER with monolingual data.
REDER performs on par with autoregressive Transformer. Despite the challenge in regards to non-autoregressive modeling and non-encoderrdeccoder design. REDER closely approaches the simplex AT models (row 14 ys. row 1), while REDER can translate both directions in one model. Besides, REDER even surpasses the multitask-leared bidirectional autoregressive Transformer model (row 14 vs. row 2). Additionally, REDER enjoys faster decoding spend than the baseline autoregressive models. This evidence shows the advantage and practical value of reversible machine translation as a more decent solution for parameter-efficiae bidirectional translation systems.
Comparison with existing approaches. Reformer [Kitaev et al. 2020 also employs RevNet to make parts the Transformer model to reduce the memory consumption for training, while REDER is fully reversible with different motivation of maximizing the use of bidirectional signals. Existing simplex approaches exploiting bidirectional signals require two separate simplex models for both directions [Xia et al. 2018 Zheng al. 2020 REDER, in contrast, only needs one unified duplex model and coherently models two directions. Alternatively, [Caan et al. [2019] use a single simplex network to achieve bidirectional translation via multitask learning, needing to split limited capacity for both directions, which underperform REDER on parallel settings.
Training cost. We train REDER on WMT14 EN using 8 32GB V100 GPUs for 432 hours (54 hour per GPU) and obtained a bidirectional translation model. For modeling both directions, a standard NAT model needs 640 GPU hours (320x2) in total, whilst the autoregressive Transformer needs 400 GPU hours (200x2) using the same computational resources. Therefore, the training costs of these methods are comparable.
8

---

4.3 Ablation Study of Model Design
REDER is developed on the top of various compo- nents in terms of data (knowledge distillation), leam- ing objective (CTC), architecture (revnet, relative at- tention), and auxiliary losses endowed by reversibility of REDER. We analyze their effects through various combinations in Table 2 We first consider training REDER for single direction to seek the best prac- tasks.
Table 2: Ablation on WMT14 EN- test set with different combinations of techniques. R-SAN denotes relative self attention. For fair comparison, all CTC and non-CTC variants do not use beam search decoding.
KD CTC
revnet
R-SAN
Lia.
Lee
BL.B
11.40
tice run REDER for sequence-to-squueee KD and CTC are essential to training REDER, as suggested by previous NAT studies (Saharia et al. 2020 Gu and Kong 2021]. Meanwhile, we notice the benefit of relative self-attention. We therefore use these three techniques by default for all of the pro- posed models. As for the duplex variants of models that learn both directions simultaneously, they can further improve the translation accuracy by substan- tial margins. These results verify our motivation that the paired translation directions could be better learned in unified reversible model. Reversibility enables us to utilize layer-wise forward-backward agreement and cycle consistency, which are also shown to boost improvement considerably.
19.50
16,90
/
25.01
/ -
25.55
/
25.90
/
/
/
26.20
2
/
26.65
26.70
26.89
4.4 Decoding: Effect of Beam search and Re-ranking
The performance of REDERs can be fur- ther boosted with additional (beam-search re-ranking) techniques. For CTC beam search, we use the teacher model (AT base) to re-rank the translation candidates ob- tained by the beam search to determine the one with the best quality. As shown in Ta- 3 a larger beam size results in a smaller BP for AT models, meaning it produces shorter translations [Stahlberg and Byrne 2019 Eikema and Aziz [020]] CTC beam search helps REDER produce longer out- puts (larger BPs) but only endows a little improvement. With beam search and AT reranking, REDER can generate more decent translations. These results imply that we need to find better way to train REDER (and probably the NAT family) if we do not want to involve an extra AT for such a somewhat inconvenient re-ranking.
Table 3: Comparisons regarding decoding methods for REDER WMT14 EN The brevity penalty (BP) given by BLEU indicates the adequacy of translation:
the lower the BP, the more inadequate the translation.
Systems
Ex-
DE-EN
BP
Speed
Transformer (AT, teacher)
27.20
31,00
0.980
1.0 x
beam==5
27.60
31,50
0.998
beam=20
27.65
31.12
0.954
REDER w/ greedy decoding
26.89
30.90
0.935
19.8 x
beam=20
26.90
30.75
0.995
6.8 x
beam=20- AT reranking
27.50
31.25
1,000
5.5 X
beam=100
26.92
30.95
0.982
2.1 x
beam= AT reranking
27.59
31.45
1,000
1.2
4.5 Training: Impact of Knowledge Distillation
Like other NAT approaches, REDER heavily relies on knowledge distil- lation. We report the performance of models trained on raw data and distilled data gener- ated from AT models in Table 4 As can see, without KD, the accuracy of REDER sig- nificantly drops. We then aim to explore the most proper way to integrate KD data. We observe that if we only use KD data of one direction (only target-side data are distilled, e.g., German sentences in Ex- EE),,,, it only benefits a single direction. These imply that we need to provide KD data of both directions to train REDER. Furthermore, we notice that if we mix the KD data of both directions by concatenating them directly, somehow improves results compared the strategy only using single-direction KD data. Finally, we find the best practice is to separately feed KD data in accordance to directions, ie., feeding EN-DE KD data when training the EN-DE direction and providing DE-EN KD data when training the reverse direction (i.e., DE-EN).
find that
Table 4: Performance regarding KD on WMT14 EN #data means the amount of data points for each direction. Systems
EN-DE
DE-EN
#data
17.85
19.68
NIN
EN DD KD (only DE distilled)
25.49
26.57
NIN
DE-NE KD (unly EN distilled)
23.04
28.82
NIN
mixture
26.50
29.65
2N/NN
separate KD (final model)
27.50
31.25
NIN

---

Discussion. Like other NAT approaches, the proposed REDER resorts to, and unfortunately heavily relies on, KD data for training. Requiring KD does hinder the generalization of NAT models including REDER to other applications especially multilingual scenarios. We notice that recent studies could have the potentials for the removal of KD for NAT models, through introducing latent-variable models [Gu and Kong, 2021 Bao et al., 2019] or sampling/denoising-basss
augmented training objectives/stratcgies [Qian et al. 2021 1] As KD-dependence is a common issue for all NAT approaches, we believe future breakthroughs would resolve this. Please note that the aim of this paper is to make the idea of duplex least in the scenario with KD data. Eliminating the need for KD is orthogonal to the purpose of this paper, however, is very valuable for further exploration.
learning and its implementation of REDER realizable, at
4.6 Analysis of Reversibility
We examine the reversibility of REDER by both quantitative and qualitative analysis. We first measure the BLEU score between source sentences x and the associated reconstructions, Le. BLEU(x, fo (fp (a:)). on the development sets of WMT14 EN-DE, which gives score of 66.0. Besides, we also show an example regard- ing its forward prediction and reconstruction in Figure 3 Given a sentence the source lan- guage (EN), we first use the forward mapping of REDER to obtain a prediction in the target language (DE), and then translate it back to the source language using the reverse model. As shown in Figure 3 REDER can reconstruct the input from the output with mild differences to some extent. These results demonstrate that REDER meets the dcfinition of reversibility empirically to a certain extent.
Inue (en)
forward prediction (de)
And if a star has planets under what circumstances
f0
Und wenn Stern Planeten bat unter welchen U Umständen kann es dort Leben geben ?
life there
back prediction (en)
true tgt (de)
And if owns plancts under what circumstances can life arise there ?
Und wenn ein Stern Planeten
hesitzt unter welchen Umstinden
kann dort Leben entstehent
Figure 3: Case study of reversibility.
4.7 Experiments on Distant Languages
To examine whether REDER can generalize to distant languages, we conducted experiments on WMT20 English- Japanese (EN++JA), where the training data is much larger and two languages are linguistically distinct with almost no vocabulary overlap. As shown in Table [ REDER can achieve very close results compared with AT in such large-scale scenario with distant languages, showing that reversible machine translation could make more potentials of parallel data.
Table 5: Results of WMT20 ENE+JA (~16M) Here REDER uses beam search with b=20 and rT-reranning.
Systems
EN-JA
JA-EN
+
AT big REDER big
20.3
21.5
20.0
20.7
5 Conclusion and Future Work
In this paper, we propose REDER, the Reversible Duplex TransformER for to-euueeee problem and apply it to machine translation that for the first time shows the feasibility of reversible machine translation system. REDER a fully reversible model that can transform one sequence the other one forth and back, by reading and generating through its two ends. We verify our motivation and the effectiveness of REDER on several widely-used NMT benchmarks, where REDER shows appealing performance over strong baselines.
As for promising future directions, REDER can be applied to monolingual, multilingual and zero-shot settings, thanks to the fact that each "end" of REDER specializes in a language. For instance, given trained REDERs ME and MEne*Jas we combine last half layers (the DE end) of ME and the JA end of MBat+Ja to obtain a zero-shot MDet+Ja. translating between German and Japanese. Likewise, the composition of an English end and reverse results in MEnt>En. which can learn from monolingual data like an autoencoder. This compositional fashion resembles LEGO, which manipulates only linear number of language ends. Therefore, while adding a new language to a multilingual REDER system (in a form of composition of ends of involved languages), we would probably not need to retrain the whole system as we do for current multilingual NMT system, which reduces the difficulty and cost to train, deploy and maintain large scale multilingual NMT system.
10

---

Acknowledgements
net/ We would like thank the anonymous reviewers for their insightful comments. Hao Zhou is the corresponding author. This work was supported by National Science Foundation of China (No. 61772261, 6217020152), National Key R&D Program of China (No. 2019QY1806).
References
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. Massively multilingual neural machine translation in the wild: Findings and challenges. arXiv preprint (XXXv::907.05550 2019.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URLhttpp//arxiv org/ 1409 0473
Yu Bao, Hao Zhou, Jiangtao Feng, Mingxuan Wang, Shujian Huang, Jiajun Chen, and Lei Li. Non-autoregressive transformer by position learning. arXiv preprint 1911 2019.
William Chan, Nikita Kitaev, Kelvin Guu, Mitchell Stern, and Jakob Uszkoreit. Kermit: Generative insertion-based modeling for sequences. arXiv preprint arXiv: 0160,, 2019.
Yong Cheng. Shiqi Shen, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. Agreement- based joint training for bidirectional attention-based neural machine translation. In Subbarao Kambhampati, editor, Proceedings of the Twenty-Fitt International Joint Conference on Artificial Intelligence, IJCAI 2016. New York, NY, USA, 9-15 July 2016, pages 2761-2767. LICAI/AAAI Press, 2016a. URL http:://www.t. ijcai org/ /tt///btt/tt/////////
Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. Semi-supervised learning for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1965-1974, 2016b.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components
estimation. arXiv preprint arXiv:/4100 16, 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In 5th International Conference Learning Representations. ICLR 2017, Toulon, France, April 24-26. 2017, Conference Track Proceedings. OpenReview.net, 2017. URL
Forum?id-Hkpbnittl
Bryan Eikema and Wilker Aziz. Is MAP decoding all you need? the inadequacy of the mode in neural machine translation. In Proceedings of the 28th International Conference on Com- putational Linguistics, pages
Barcelona, Spain (Online). December 2020. Interna- tional Committee on Computational Linguistics. doi: 10. 10.18653/11/020.collol main.398. URL https://www aclweb. org/ authology/2000000 coling main.
Gordon Franck. Reversible grammars and natural language processing. In Proceedings of the 1992 ACM/SIGAPP Symposium on Applied Computing: Technological Challenges of the 1990's, ss page 102-109, New York, NY, USA, 1992. Association for Computing Machinery. ISBN 089791502X. doi: 10.1145/143559.144 14355499999.49977.. 143597. URL httpps:://ii.rrr// orgo 10. 1145/143558 143597
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolutional sequence to sequence learning. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-1 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 1243-1252. PMLR, 2017. URL BRRppttp://pooeeeding mlr. wwwpp//ese/eeeeeee https////OOoeerrri/aaaaaaa.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference Empirical Methods in Natural Langugge Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) pages 6112-6121, Hong Kong, China,
11

---

November 2019. Association for Computational Linguistics. doi: hctps://www.aclveet hww.://.rrrgaantoooooooo//////// /Dg- 1633
URL
Marjan Ghazvininejad, Omer Levy, and Luke Zettlemoye.. Semi-autoregressive training improves mask-predict decoding. arXiv preprint arXiv:2001.0878; 2020.
Aidan N. Gomez, Mengye Ren, Raquel Urtasun, and Roger B. Grosse. The reversible resid- ual network: Backpropagation without storing activations. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Ro- man Garnett, editors, Advances Neural Information Processing Systems 30: Annual Con- ference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA. 2017 2017/hash/ wwwssseerripp.. cc/ wwtpsscccppp/////eeeeeee pwwppr/017/haaaaaaaaappppapapaae
USA, pages 2214 2224, 2017. URL httplltttts:/pooooeaeee Abt.00444440004--0rabrra- Abstract htmll html
Alex Graves, Santiago Fernández, Faustino J. Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequenee data with recurrent neural networks. In William W. Cohen and Andrew W. Moore, editors, Machine Learning, Proceedings of the Wiient--hhrrd
Twenty- Third International Conference (ICML 2006), Pittsburgh, Penusylvania, USA, June 25-29, 2006, volume 148 of ACM International Conference Proceeding Series, pages 369-376 ACM, 2006. doi: 10. 145/1 14344. 114844,1114891,
URL https://doi. org/ 10. 11e.11111111444.1111 1143891
Jiatao Gu and Xiang Kong. Fully non-autoregreessive neural machine translation: Tricks of the trade. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 120-133, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 2021 102d.iiniing--ac..11
/ttps:
org/ 2021 findings-acl.11
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. Non-autoregressive neural machine translation. 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenRe- view.net, 2018 httlltttts//pperertteee wttpriiewweeetffeee net/forummiddB118BBl.
Jiatao Gu, Changhan Wang, and Junbo Zhao Levenshtein transformer. In Hanna M. Wallach. Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Bue, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 11 119-11188,, 2019,
www..nnuripp...m ctt.iipp..c/pappee/2900/ Abstract html
Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie- Yan Liu, and Wei-Ying Ma. Dual learning for machine translation. In Daniel Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 820-828, 2016. URL https::::: http:::://////ee//pp//eeo/o G666bcc18000544400677555///8//
Abstract html
Junxian He, Xinyi Wang, Graham Neubig, and Taylor Kirk-aiikp,.r.ck. probabilistic formulation of unsupervised text style transfer. In International Conference on Learning Representations, 2019.
Chenyang Huang, Hao Zhou, Osmar R Zaïane, Lili Mou, and Lei Li. Non-autoregressivv translation with layer-wise prediction and deep supervision. arXiv preprint arXiv:21 2021.
Melvin Johnson, Mike Schuster, Quoe V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's 's multilingual neural machine translation system: Enabling zero-shot transla- tion. Transactions of the Association for Computational Linguistics, 5:339-351, 2017. doi: 20.1162/tal_a_000 00065. http:tttss://ww.... wwwpw//wwwwwwwwbbbbbb/ org/ wwwtaooooo/g11111000000
Jungo Kasai, James Cross, Marjan Ghazvininejad, and Jiatao Gu. Non-autorregressive machine translation with disentangled context transformer. In Proceedings of the 37th International Conference Machine Learning, ICML. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 5144-5155. PMLR, 2020. URL[htt //proceedings..
mml.anngmmm......mmmmmmm....mm.m mlr, https///111kkaaaaaaaaam
12

---

Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. pages 1317-1327, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10. 10.18/33/11//11111133
URL https:// www. aclweb. / httg///nnooooogii...o 1139
Durk Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling, Improved variational inference with inverse autoregressive flow. Advances in Neural Information Processing Systems, 29:4743-47551 2016.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In 8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenReview.net, 2020. URL httpp::s////:ptrrreeet wwt.:/r..e...ooooooooooigatattt.
Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic non-autoregrressive neural sequence modeling by iterative refinement. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1 1173-1188 Brussels, Belgium, October-Novembn 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/D188 149. URL pettps//wwx aclweb. www.oo/ggg....gggggooooooooooott D18- 1149
Jindfich Libovický and Jindřich Helel. End-to-end non-autoregressive neural machine translation with connectionist temporal classification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3016-3021, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 18.18653/v 10L118665//D1.... 1336. URL hattps://ww/
owa.ee..org//tiiee www..oogggggggggoooreo.oooo 11366
Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig, and Eduard Hovy. FlowSeq: Non- autoregressive conditional sequence generation with generative flow. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joins Conference on Natural Language Processing (EMNLP-IJCNIP)), pages 4282-4299. Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.1853//1/DD11-1433 D19- URL https://wwy acwweb. org/ httpg//nhhoooooy--leeeo1 1437
Matthew MacKay, Paul Vicol, Jimmy Ba, and Roger B. Grosse. Reversible recurrent neural networks. In Samy Bengio, Hamna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Biana Bianch,, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages paper
2018.
newris...uriis.... html
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrationss) pages 48-53, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.11853/11//11944000 URL https: / wwwp//wwwwwwoo aclweb. org/
anthology/N19 Nwtpogyy/991999
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 31 311-318, Philadelphia. Pennsylvania, USA, July 2002. URL/https: //t::
Association for Computational Linguistics. doi:
aclweb. gellveb.org/annn www./anttooggy///t/o pallloggyyO221140 1040
Lihua Qian, Hao Zhou, Yu Bao, Mingxuan Wang, Lin Qiu, Weinan Zhang, Yong Yu, and Lei Li. Glancing transformer for non-autoregressive neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 1993-2003, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/202 a.c--oog.....--
URL https://aclanthhllgg .rg. 2021 acl- 155
Chitwan Saharia, William Chan, Saurabh Saxena, and Mohammad Norouzi. Non-autoregressive machine translation with latent alignments. In Proceedings of the 2020 Conference on Empirical
Methods in Natural Language Processing (EMNLP), pages 1098-1108 Online, November 2020.
13

---

Association for Computational Linguistics. doi: 10. 10.18653/w122220emmmll main...m..n..... URL https org
org/ http//anthoolyyy/aae 2020 emnlp-mane wwin. 88
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation mod- els with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96, Berlin, Germany, Au- gust 2016. Association for Computational Linguistics. doi: 10.18653/11/P16-100 1009. URL https://wwrt aclweb. org/ Peathology/P16-110
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Short Papers), pages 464-468. New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v 1//13//1//11-207444 URL wttph:ttttwwwtt aclweb. org/ anthology/MM1--2744
Raphael Shu, Jason Lee, Hideki Nakayama, and Kyunghyun Cho. Latent-variable non con-autoregressvve neural machine translation with deterministic inference using a delta posterior. In EMNLP, 2020.
Felix Stahlberg and Bill Byrne. On NMT search errors and model errors: Cat got your tongue? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNE pages 1356-3362 Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v//D19 1331,,
URL htt.:httts///wwt aclweb, org/ anthology/DD9-11331
Tomek Strzalkowski. Reversible Grammar in Natural Language Processing, volume 255. Springer Science & Business Media, 1993.
Ily Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural net- works. In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger, editors, Advances Neural Information Processing Systems 27: Annual Confer- ence on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 3104-3112, 2014. URL https:::::// httpptsss/rrocceiigg...... newriienurpppp. pppppp/eediss..oorrrrp.../p//ppp hathh/aa44ab55444477772624444 w3w7:c221c33232232ccmm Abstract. html
Tycho FA van der Ouderaa and Daniel E Worrall. Reversible gans for semory-efficien image-to- image translation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern 2019.
Recognition, pages
Gertjan van Noord. Reversible unification based machine translation. In COLING 1990 Volume Papers presented to the 13th International Conference on Computational Linguistics, 1990. URL
https://www awwweb. org/
www.s//ewrrg/aatollgg/////////// aataaaatthooyyoooo C
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Ro- man Garnett, editors, Advances Neural Information Processing Systems 30: Annual Confer- Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA. USA, pages 5998-6008, 2017. URL https: httptp:::/pproeeeeeee http:////occee.n..e///e///. nwwrips cc/ httpssccccppppepeeeeeeee palpp//p00/111 2017 20007//hshh aat.ae44baa-bbtraatt...m Abstract html
Bingzhen Wei, Mingxuan Wang. Hao Zhou, Junyang Lin, and Xu Sun. Imitation learning for non-autoregressive neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1304-1312, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.186553w1/P19--112 URL wttht:tt/////tt aclweb. wcllebb.ogg/nthhaee www.r/rra/nnoooooooooooooo
Yingce Xia, Tao Qin, Wei Chen, Jiang Bian, Nenghai Yu, and Tie-Yan Liu. Dual supervised learning.
In International Conference on Machine Learning, pages 3789-3798. PMLR, 2017.
Yingce Xia, Xu Tan, Fei Tian, Tao Qin, Nenghai Yu, and Tie- Yan Liu. Model-level dual learning. In International Conference Machine Learning, pages 5383-5392. PMLR, 2018.
14

---

Zichao Yang, Zhiting Hu, Chris Dyer, Eric P Xing, and Taylor Berg- Kirkpatrick. Unsupervised text style transfer using language models as discriminators. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 7298-7300 2018.
Biao Zhang, Ankur Bapna, Rico Sennrich, and Orhan Firat. Share or not? learning schedule language-specific capacity for multilingual translation. In Internationall Conference on Learning Representations, 2021. URL / net::oorumm..eeW/4400000..
Zaixiang Zheng, Hao Zhou, Shujian Huang, Lei Li, Xin- Yu Dai, and Jiajun Chen. neural machine translation. In International Conference on Learning Representations, 2020.
Chunting Zhou, Jiatao Gu, and Graham Neubig. Understanding knowledge distillation in non-
autoregressive machine translation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenReview.net, 2020. URL https //openreview net/ coormmiideeee BYgFVAEDDH
15


Reviews:
[{"review_id": "vSCzMfk-lL", "reviewer": "Reviewer_19ey", "summary": "This paper proposes a duplex reversible network based on Transformer and revnet for neural machine translation. With the proposed approach, one can use one single model to translate between a language pair from both directions. The model uses CTC loss and allows non-autoregressive decoding for faster inference time. Experiments are well executed on standard NAT benchmark datasets with close performance to that of the standard Transformer model.\n\n**Contributions**: the proposed model is new in machine translation from the best of my knowledge and achieves good performance compared to non-autoregressive models and multi-task learning. The proposed method saves parameters in terms of translation from both directions and improves inference speed.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: the proposed work resembles the coupling network in the literature of flow-based models; authors adapt the revnet to Transformer models by designing a fully symmetric architecture that accepts inputs and emits outputs from both ends.\n\n**Clarity**: the paper is generally well written but contains many typos. I'll list a few here: L47 -> interference, L50 -> language, L62 -> (), L74 -> the sentence of multitask ..., L77 -> adopts, L79 -> bonus also influent sentence; please double-check the writing.\n\n**Weakness and Questions**:\n1. Authors mentioned one advantage of the proposed method in the introduction which is it works well for linguistically distant language pairs; I did find the results for ja-en in the appendix, but I think these are pretty interesting and important results and should be moved to the main paper.\n\n2. It seems that the training cost of the duplex transformer could higher than standard training. Could you report approximately the training time for the proposed model versus. the Transformer model? Besides, **it heavily relies on distillation data**, which makes me worry about its broader application to other MT scenarios, e.g. multilingual MT.\n\n3. The discussed future work seems to be interesting. Adding one such experiment for zero-shot or multilingual MT would make the paper stronger. There could be plenty of interesting concerns when applying this model to multilingual MT. I'd like to see if the duplex transformer would outperform standard multilingual training.\n\n4. The other related approach which could potentially reduce the parameters is parameter sharing and separation for different language pairs, where you could share many parameters for both language pairs and have individual parameters (like adapters) for individual language pair to alleviate the parameter interference problem.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "ea9mOBwasbr", "reviewer": "Reviewer_byW1", "summary": "This paper proposes duplex sequence-to-sequence learning where data streams can flow from both ends of a seq2seq model. More precisely, they design a parameter efficient Transformer (REDER) that can simultaneously input and output a distinct language from each end of the model. In this way, REDER enables reversible machine translation by simply flipping the input and output ends. The experimental results show that the proposed REDER improve the multitask trained baseline by over 1.3 BLEU while maintaining over 5x speed-up at inference time.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I think the idea of duplex learning is very interesting and novel. Different from standard multi-task learning, the proposed method shows evidence that it nicely combines the information from both directions, while avoiding hurting the performance due to the capacity issue of modeling two languages. Although the basic idea of reversible layers is borrowed from previous work, it is novel to me that the same idea can be applied to Transformers.\n\nHowever, I still have the following questions:\n\n(1) When applying upsampling to the source tokens, how can reversibility still be maintained? In Line 62, the definition means that you need to return to x when applying forward and backward functions to x. However, the CTC loss allows the model to predict any sentences with the same collapsed form. To rephrase my question, I think the collapse process used in CTC is not reversible, but I am not sure about that.\n\n(2) For the cycle consistency loss (Line 189), is it differentiable to the forward pass? Since you need to generate target sentences first before sending it back to the backward pass, and generation is non-differentiable.\n\n(3) How do you determine the coefficients of $\\lambda_{cc}$ and $\\lambda_{fba}$? \n\n(4) The proposed REDER has a similar number of parameters compared to standard NAT models (e.g. Gu and Kong, 2020). However, the vanilla model of REDER still works worse than it does despite it utilizes both directions. What do you think may be the cause? Note that the results of Gu and Kong, 2020 in table 1 is greedy without using any reranking and beam-search, while the results of REDER does. It would be better to denote that in the table caption. \n\n(5) Will the relative self-attention also benefit the standard simplex NAT models?\n\n(6) The results without KD look quite low. Although it generally makes sense, what do you think how we can improve the performance without KD within the REDER framework?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors claimed to have addressed the limitations. However they did not show that in a separate section or discussion, which makes the reviewer difficult to find it.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3-4 hours"}, "extra_text": {}}, {"review_id": "ck_phG8A1P5", "reviewer": "Reviewer_Bs9X", "summary": "The paper proposes a novel machine translation model that can be simultaneously applied to bidirectional translations. The model, named REDER, is symmetric at both input and output ends and consists of several identical reversiable duplex Transformer layers. REDER adopts the design of revnet for reversibility and reverses the first L/2 layers for homogeneity, which is a fully non-autoregressive NMT model. Extensive experiments on WMT14 En$\\leftrightarrow$De and WMT16 En$\\leftrightarrow$Ro datasets show that the proposed model consistently outperforms (non-)autoregressive baselines with improving decoding speed.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\nThis work to clearly identify shortcomings in the current literature. Experimental results seem to be solid. The identification of REDER as a possible approach to realize reversible machine translation.\n\nQuestions\n\n* There is a definition of duplex sequence-to-sequence neural netwrok in line 57, and the proposed REDER is also optimized by the cycle consistency. However, there is no experiments to verify the resulted REDER model is satisfied with this pre-definition. Can you provide some quantitative results on the development sets, such as the BLEU or word prediction accuracy of the reconstructed outputs, e.g. $f_{\\theta}^{\\leftarrow}(f_{\\theta}^{\\rightarrow}(x))$ or $f_{\\theta}^{\\rightarrow}(f_{\\theta}^{\\leftarrow}(y))$.\n* Existing multilingual machine translation models can also realize reversible machine translation that uses language embedding to avoid parameter interference problem. Can you provide more comparisons between REDER and multilingual NMT models (only involves the source and target languages.)\n* A related work is missing, i.e. supervised dual learning [Xia et al., 2017: ICML].\n* Why duplex neural networks can avoids the parameter interference problem? (Statement in line 51-52) It is not very clear for me.\n* Writing could have been clearer. What the superscript (1) and (2) in Figure 2 mean? Why $y^{(1)}$ is dependent on $x^{(2)}$ and $y^{(2)}$ is dependent on $x^{(1)}$.\n* I would to see the paragraph of $\\textit{Modeling variable-length input and output}$ to be put behind $\\textit{The Architecture of REDER}$, which is more clearer for me.\n\nMinor comments\n\n* In Definition 1, a closing bracket is missing in the last sentence, i.e. $f_{\\theta}^{\\leftarrow}(f_{\\theta}^{\\rightarrow}(x))=x$.\n* In line 74, change \"multitask learning with\" to \"With multitask learning\".\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no explicit presentation of limitations or potential negative societal impact in this paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "GF-C2FaBnHG", "reviewer": "Reviewer_LGLR", "summary": "This paper proposes an architecture or a reversible NMT system. In contrast to an encoder-decoder network, the two components of the network for each language are identical. When a language is being analyzed or generated depends on the direction of data flow in the network. To enable this, the authors propose reversible components in each layer of the network (inspired from revnet). The model is trained with CTC loss along with auxiliary losses for cyclic consistency and layerwise agreement. The paper is an interesting proposal for a reversible NMT system, and it brings together work on revnet, CTC loss, NAT in a simple, elegant framework to enable reversible NMT. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the architecture is novel and interesting, it is not clear what implication does a reversible network have for sequence to sequence learning and MT. \n- The authors mention that reversible NMT can be useful for using supervised signals in both directions to improve translation quality. However, the results do not seem to substantiate this. The gains using REDER are none to minor over an MTL baseline. Further, the use of cyclic consistency loss for the joint training and the alignment losses also do not seem to help to any meaningful extent.  \n- The other benefit pointed out by the authors is the decoding speedup on account of the non-autogressive nature of the solution. The speedup is much lower compared to the other NAT approaches, while the translation quality is comparable. REDER uses similar techniques as CTC-based NAT approaches - why is it then about 3x slower than these models? Can the authors elaborate on this discrepancy?\n- Like other NAT approaches, the model still relies on KD data to be competitive with auto-regressive models. This seems to be one sore spot in an otherwise elegant model. \n\nOverall, I think the paper puts together many ideas and proposes an interesting idea of reversible NMT and an elegant solution. The results do not such much benefit in terms of benefits from bilingual signals or decoding speedup. However, this is an interesting proposal and framework to further explore. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Nothing to comment", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper explores a "duplex" architecture for machine translation, by designing a network in which either end can be used as an input or output. To achieve this, the authors apply previous ideas about reversible networks to the transformer architecture, and use a non-autoregressive loss. The reviewers found this to be a novel, interesting and elegant approach. Like most non-autoregressive approaches, the need for knowledge distillation remains a limitation, and the authors reasonably point out that removing KD is not a goal of the paper. There are also questions about whether the approach really has large practical advantages over existing work - but this topic seems to be an interesting direction that is worth further exploration, so I recommend acceptance.
Decision: Accept (Poster)