Duplex Sequence-to-Sequence Learning for Reversible Machine Translation
Year: 2022

Abstract:
Sequence-to-sequence learning naturally has two directions. How to effectively utilize supervision signals from both directions? Existing approaches either require two separate models, or a multitask-learned model but with inferior performance. In this paper, we propose REDER (REversible Duplex TransformER), a parameterefficient model and apply it to machine translation. Either end of REDER can simultaneously input and output a distinct language. Thus REDER enables reversible machine translation by simply flipping the input and output ends. Experiments verify that REDER achieves the first success of reversible machine translation, which helps outperform its multitask-trained baselines up to 1.3 BLEU. 1

heading: 1 Introduction
text: Neural sequence-to-sequence (seq2seq) learning [Sutskever et al., 2014] has been extensively used in various applications of natural language processing. Standard seq2seq neural networks usually employ the encoder-decoder framework, which includes an encoder to acquire the representations from the source side, and a decoder to yield the target side outputs from the encoded source representation [Bahdanau et al., 2015, Gehring et al., 2017, Vaswani et al., 2017].
Generally, given paired training data Dxy = X ×Y , where X is the source side and Y is target side of the corresponding seq2seq task, supervision signals are always bidirectional. Thus we can learn not only the mapping from source to target (fθxy : X 7→ Y) but also the mapping from target to source (fθyx : Y 7→ X ). This is very common in many applications. For example, given parallel corpus, we can obtain machine translation models from both Chinese to English and English to Chinese, or transfer between different stylized texts [Yang et al., 2018, He et al., 2019].
Typical seq2seq learning utilizes the bidirectional supervisions by splitting the bidirectional supervisions into two unidirectional ones and trains two individual seq2seq models on them, respectively. However, such splitting ignores the internal consistencies between the bidirectional supervisions. Thus how to make better use of the bidirectional supervisions remains an open problem. One potential solution is multitask learning [Johnson et al., 2017], which jointly leverages the bidirectional supervisions within one model and expects the two unidirectional supervisions could boost each other (Figure 1(b)). But it does not work well in our case due to the parameter interference problem [Arivazhagan et al., 2019, Zhang et al., 2021]. For instance, in the setting of multitask Chinese-to-English and English-to-Chinese bidirectional translations, the encoder/decoder of seq2seq networks is trained to simultaneously understand/generate both Chinese and English, which always results in performance
∗Work was done when Zaixiang Zheng was a final-year PhD candidate at Nanjing University and an intern (now FTE) at ByteDance AI Lab; and when Lei Li was also at ByteDance AI Lab.
1Code is available at https://github.com/zhengzx-nlp/REDER.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
drop due to the divergent nature of the two languages. Another branch of solutions lies in cycle training [Sennrich et al., 2016, He et al., 2016, Zheng et al., 2020], which still deploys two individual models for separately learning, but explicitly regularize their outputs by the cycle consistency of seq2seq problems. Such approaches usually need to introduce extra monolingual data in machine translation.
Layer
lpad rpad
Dr in k mo re ho t wa te r rpad rpadlpad
source language target language neural networks
E n c o d e r
D e c o d e r
source language target language neural networks
Cross Attention
simplex communication
D e c o d e r
E n c o d e r Cross Attention
simplex communication
In this paper, we propose to explore an alternative approach for utilizing bidirectional supervisions called duplex sequence-to-sequence learning. We argue that current seq2seq networks do not benefit from multitask training because the seq2seq networks are simplex from the view of telecommunications2. Specifically, the data stream only flows from encoder to decoder in the current seq2seq learning, and such simplex property makes the multitask learning suffer from the parameter interfere problem. Instead, if we have duplex neural networks, where the data stream can flow from both ends, each of which only specializes in one language when learning from bidirectional supervisions (Figure 1(c)), thus potentially alleviating the parameter interference problem. As a result, the bidirectional translation can be achieved as a reversible and unified process. In addition, we could still incorporate cycle training in the duplex networks using only one single model.
Definition 1. A sequence-to-sequence neural network with parameter θ is duplex if it satisfies the following: its network has two ends, i.e., a source end and a target end; both source and target ends can take input and output sequences; its network defines a forward mapping function f→θ : X 7→ Y , and a reverse mapping function f←θ : Y 7→ X , that satisfies the following reversibility: f←θ = (f → θ ) −1 and f→θ = (f ← θ ) −1; besides, it satisfies the following cycle consistency: ∀x ∈ X : f←θ (f→θ (x)) = x and ∀y ∈ Y : f→θ (f←θ (y)) = y.
Based on the idea of duplex network, we propose REDER3, the REversible Duplex TransformER, and apply it to machine translation. Note that, building duplex seq2seq networks is non-trivial: a) vanilla encoder-decoder network is irreversible. The output end of the decoder cannot take in input signals to exhibit the encoding functionality and vice versa; b) the topologies of the encoder and the decoder are heterogeneous, i.e., the decoder works autoregressively, while the encoder is non-autoregressive. To this end, we therefore design REDER without explicit encoder and decoder division, in which we introduce the reversible Transformer layer [Gomez et al., 2017] and fully non-autoregressive modeling to solve the above two problems respectively. As a result, REDER works in the duplex fashion, which could better exploit the bidirectional supervisions for achieving better downstream task performance.
Experimental results show that the duplex idea indeed works: Overall, REDER achieves BLEU scores of 27.50 and 31.25 on standard WMT14 EN-DE and DE-EN benchmarks, respectively, which are top results among non-autoregressive machine translation models. REDER achieves significant gains (+1.3 BLEU) compared to its simplex baseline, whereas multitask learning does hurt the translation performance of simplex models both in the autoregressive (-0.5 BLEU) and nonautoregressive settings (-1.3 BLEU). Although REDER adopts fully non-autoregressive modeling to realize the duplex networks, the gap of BLEU between REDER and autoregressive Transformer is negligible, meanwhile REDER can directly translate between two directions and enjoys 5.5× inference speedup as a bonus of non-autoregressive modeling. To our best knowledge, REDER is the first duplex seq2seq network, which enables the first feasible reversible machine translation system.
2In telecommunications and computer networking, the simplex communication means the communication channel is unidirectional while the duplex communication is bidirectional.
3The model’s name is a palindrome, which implies the model works from both ends.

heading: 2 REDER for Reversible Machine Translation
text: In this section, we introduce how to design a duplex neural seq2seq model, the REDER (REversible Duplex TransformER), that satisfies Definition 1, and its application in machine translation that realizes the first neural reversible machine translation system.

heading: 2.1 Challenges of Reversible Machine Translation
text: Reversible natural language processing [Franck, 1992, Strzalkowski, 1993] and its applications in machine translation [van Noord, 1990] were proposed for the purpose of building machine models that understand and generate natural languages as a reversible, unified process. Such process resembles the mechanism of the ability that allows human beings to communicate with each other via natural languages [Franck, 1992]. Despite the success of neural machine translation with deep learning, designing neural architectures for reversible machine translation yet remains under-studied and has the following challenges:
Reversibility. Typical encoder-decoder networks and their neural components, such as Transformer layers, are irreversible, i.e., one cannot just obtain its inverse function by flipping the same encoderdecoder network. To meet our expectation, an inverse function of the network should be derived from the network itself. Homogeneity. Intuitively, a pair of forward and reverse translation directions should resemble a homogeneous process of understanding and generation. However, typical encoder-decoder networks certainly do not meet such computational homogeneity due to extra cross attention layers in the decoder; and also because of the discrepancy that the decoder works autoregressively but the encoder does non-autoregressively. To meet our expectations, the separation of encoder and decoder should be no more exist in the desired network.

heading: 2.2 The Architecture of REDER
text: To solve the above challenges, we include two corresponding solutions in REDER to address the reversibility and homogeneity issues respectively, i.e., the Reversible Duplex Transformer layers, and the symmetric network architecture without the encoder-decoder framework.
Layer
多 喝 热 ⽔lpad rpad
Dr in k mo re ho t wa te r rpad
点
rpadlpad
S e lf
A tt
e n tio
n
F F N
Te fl-
A tt
e n tio
n
y
y
+
+ SAN FFN
−
−
SAN FFN
Duplex Communication
so ur
ce
ta rg
et
X ↦ YX Y↦
x(1) (1)
(2)
x
x
y
y
(1)
(2)
(1)
(2)
Regular Form ( ) Reverse Form ( )
_____S AN _____ _____FFN _____
RevNet-style Transformer Layer :
(1) x(2)
_____S AN _____ _____FFN _____
_____S AN _____ _____FFN _____ __ __ _S AN __ __ _ __ __ _F FN __ __ _ __ __ _S AN __ __ _ __ __ _F FN __ __ _ __ __ _S AN __ __ _ __ __ _F FN __ __ _
x(2)
[ , ] ↦ [ , ]x(1) x(2) y(1) y(2) 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202
203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219
Duplex Sequence to Sequence Learning for Reversible Machine Translation
Layer
lpad rpad
Dr in k mo re ho t wa te r rpad rpadlpad
S e lf
A tt
e n tio
n
F F N
Te fl-
A tt
e n tio
n
x
x
y
y
+
+ SAN FFN
SAN FFN
Duplex Communication
so ur ce
ta rg et
X YX Y
(1)
(2)
(1)
(2)
x
x
y
y
(1)
(2)
(1)
(2)
Regular Form Inverse Form
_____S AN _____ _____FFN _____
Revnet Transformer Layer : [x(1), x(2)] [y(1), y(2)]
(1) x(2)
_____S AN _____ _____FFN _____ _____S AN _____ _____FFN _____ __ __ _S AN __ __ _ __ __ _F FN __ __ _ __ __ _S AN __ __ _ __ __ _F FN __ __ _ __ __ _S AN __ __ _ __ __ _F FN __ __ _
Figure 2. The proposed REDER for duplex sequence-to-sequence generation. The bottom two diagrams show the computation of the regular and reverse forms of a reversible layer. Notice that, to make the whole model symmetric, we reverse the 1-th to L/2-th layers, such that the overall computational operations of forward and reverse of REDER are homogeneous.
duplex sequence-to-sequence is distinct from a joint multilingual MT model (e.g. multilingual Transformer). The multilingual Transformer takes input sentences in two (or more) languages only from the same end and outputs to the other end. Its output end cannot be used as the input.
4. REDER: REEVERSIBLE DUPLEX TRANSFORMER
In this section, we design the architecture of a duplex sequence-to-sequence generation model for machine translation, REDER, that satisfies D finition 1. F F 1
4.1. Challenge of Reversible Translation
Reversible machine translation is originally mentioned in van Noord (1990). However, designing such reversible MT has the following challenges:
(i) Reversibility. An inverse function of the network
should be computed by reversing the connections in
the network. Ordinary neural networks, such as convo-
lutional layers, are not reversible, i.e. one cannot just
obtain its i verse function from the same convolutional
nets.
(ii) Symmetry. Both ends of the desired model can take
input and produce output, and the forward and reverse computation flows move through the same model architecture. As a result, the head-side computation and the tail-side computation should resemble certain equivalence. The normal encoder-decoder architecture such
as the Transformer certainly does not meet such input/output equivalence property due to the cross attention layer in Transformer which is only from the decoder to encoder. There should not be a division of encoder and decoder in the desired duplex model. (iii) Input-output length mismatch. The input and output sequences are not of the same length. Without encoderdecoder separation, an idea duplex model should match the sequences at different lengths.
4.2. REDER Architecture
To solve the above three challenges, we include three components in REDER, including reversible layers, equi-width duplex composition scheme, CTC loss. Figure 2 shows the overall architecture of the REDER model. For the following, ✓ is the model parameter.
Reversible layers. We adopt the idea of reversible residual network (revnet, Gomez et al., 2017; Kitaev et al., 2020) in our reversible layer design. Each REDER reversible layer is composed of a multi-head attention block and a feedforward block with a special reversible design to ensure duplex behavior. Formally, the regular form of the l-th layer Fl performs as follow:
Hl = Fl(Hl 1), [H(1)l ;H (2) l ] = Fl([H (1) l 1;H (2) l 1]),
where H(1)l = H (1) l 1 + SAN(H (2) l 1),
H (2) l = H (2) l 1 + FFN(H (1) l ).
Accordingly, the reverse form of F 1l can be computed by subtracting (rather than adding) the residuals:
Hl 1 = F 1l (Hl), [H (1) l 1;H (2) l 1] = F 1l ([H (1) l ;H (2) l ]),
where H(2)l 1 = H (2) l FFN(H (1) l 1),
H (1) l 1 = H (1) l SAN(H (2) l 1).
For better capability of the reordering between source and target languages, we employ relative self-attention (Shaw et al., 2018) instead of the original one in the original Transformer (Vaswani et al., 2017).
Equi-width duplex composition scheme. To make the
model symmetric, we create an architecture with equal-
width duplex and symmetric composition of the reversible
layers. However, given the differences in the regular form
and reverse form each layer is reversible but not symmet-
ric, thus the model simply stacking L reversible layers is
overall asymmetric for both directions. To make our model
symmetric for both directions, we let the 1-th to L/2-th layers be the inverse form of the reversible Transformer layer, whereas the latter (L/2 + 1)-th to L-th be the regular version. Thereby the forward and reverse computational operations of the whole model become homogeneous, which
165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202
203
204
205
206
207
208
209
210
211
212
213
214 215 216 217 218 219
Duplex Seque ce to Sequence Learning for Reversible Machine Translation
Layer
lpad rpad
Dr in k mo re ho t wa te r rpad rpadlpad
S e lf
A tt
e n tio
n
F F N
Te fl-
A tt
e n tio
n
x
x
y
y
+
+ SAN FFN
SAN FFN
Duplex Communication
so ur ce
ta rg et
X YX Y
(1)
(2)
(1)
(2)
x
x
y
y
(1)
(2)
(1)
(2)
Regular Form Inverse Form
_____S AN _____ _____FFN _____
Revnet Transformer Lay r : [x(1), x(2)] [y(1), y(2)]
(1) x(2)
_____S AN _____ _____FFN _____ _____S AN _____ _____FFN _____ __ __ _S AN __ __ _ __ __ _F FN __ __ _ __ __ _S AN __ __ _ __ __ _F FN __ __ _ __ __ _S AN __ __ _ __ __ _F FN __ __ _
Figure 2. The proposed REDER for duplex sequence-to-sequence genera ion. The bottom two diagrams show the computation of the regular and reverse forms of a reversible layer. Notice that, to make the whole model symmetric, we reverse the 1-th to L/2-th layers, such that the overall computational operations of forward and reverse of REDER are homogeneous.
duplex seq ence-to-sequence is distinct from a joint multilingual MT model (e.g. multilingual Transformer). The multilingual Transformer takes input sentences in two (or more) languages only from the same end and outputs to the other end. Its output end cannot be used as the input.
4. REDER: REEVERSIBLE DUPLEX TRANSFORMER
In this section, we design the architecture of a duplex seque ce-to-sequence generation model for machine translation, REDER, that satisfies Definition 1. F F 1
4.1. Challenge of Reversible Translation
Reversible machine translation is originally mentioned in van Noord (1990). However, d signing such reversible MT has the following challenges:
(i) R versibility. An inverse function of the network sh uld be co puted by reversing the connections n the network. Or inary n ur l networks, such as c nvolutional layers, are not reversible, i.e. one cannot just
btain its inverse function ro the same convolutiona nets.
(ii) Symmetry. Both ends of th desired mo el an take input and produce output, and the fo ward and r verse computation flows move through the sa e model architecture. As a result, the head-side computation and the tail-side computation s ould res mble cert in equivalence. The normal encoder-dec der architecture such
a the Transformer certainly does not meet such input/output equivalence property due to the cross attention layer in Transformer which is only from the dec der to encoder. There should not be a division of encoder and decoder in the desired duplex model. (iii) Inp -output length mismatch. The input and output s quences are not of the same length. Without encoderdecoder separation, an idea duplex model should match the sequences at different lengths.
4.2. REDER Architecture
To solv the above three challenges, we include three components in REDER, including reversible layers, equi-width duplex composition scheme, CTC loss. Figure 2 shows the ov ral architecture of the REDER model. For the following, ✓ is the model parameter.
Rev rsible layers. We adopt the idea of reversible residual n work (revnet, Gomez et al., 2017; Kitaev et al., 2020) in our reversible layer design. Each REDER reversible layer is composed of a multi-head attention block and a feedforward block with a special reversible design to ensure duplex behavior. Formally, the regular form of the l-th layer Fl performs as follow:
Hl = Fl(Hl 1), [H(1)l ;H (2) l ] = Fl([H (1) l 1;H (2) l 1]),
where H(1)l = H (1) l 1 + SAN(H (2) l 1),
H (2) l = H (2) l 1 + FFN(H (1) l ).
Accordingly, the reverse form of F 1l can be computed by subtracting (rather than adding) the residuals:
Hl 1 = F 1l (Hl), [H (1) l 1;H (2) l 1] = F 1l ([H (1) l ;H (2) l ]),
where H(2)l 1 = H (2) l FFN(H (1) l 1),
H (1) l 1 = H (1) l SAN(H (2) l 1).
For better capability of the reordering between source and target languages, we employ relative self-attention (Shaw et al., 2018) instead of the original one in the original Transformer (Vaswani et al., 2017).
Equi-width duplex composition scheme. To make the
model symmetric, we create an architecture with equal-
width duplex and symmetric composition of the reversible
layers. H wever, given the differences in the regular form
a d reverse form each layer is reversible but not symmet-
r c, thus the model simply stacking L reversible layers is
overall asymmetric for both directions. To make our model
symmetric for both directions, we let the 1-th to L/2-th layers be the inverse form of the reversible Transformer layer, where s the latter (L/2 + 1)-th to L-th be the regular version. Thereby the forward and reverse computational operations of the whole model become homogeneous, which
Figure 2: The proposed REDER for duplex sequenceto-sequence gener ti n. The bottom wo diagrams show the computation of the regular and reverse forms of a reversible layer. Notice that, to make the whole model symmetric, we reverse the 1-th to L/2-th layers, such that th eral computation l operat ns of forward and reverse of REDER are homogeneous.
Figure 2 shows the overall architecture of REDER. As illustrated, REDER has two ends: the source end (left) and the target end (right). θ is the model parameter, shared by both directions. The architecture of REDER is composed of a series of identical Reversible Duplex Transformer layers. When performing the source-to-target mapping f→θ , a source sentence x (blue circles) 1) first transforms to its embedding e(x) and enters the source end; 2) then goes through the entire stacked layers and evolves to final representations HL which predicts probabilities; 3) finally its target translation (orange circles) will be generated from the target ends. The generation process is fully nonautoregressive.
Likewise, the target-to-source mapping f←θ is achieved by reversely executing the architecture of REDER from target end to source end. We will dive into the details of the key components of REDER in the following parts.
Reversibility: Reversible Duplex Transform r lay rs. We adopt the idea of the reversible residual network (RevNet, Gomez et al., 2017, K ta v et l., 2020) in the design of t e versible duplex Transformer layer. The bottom of Figure 2 illustrates the forward and reverse computations of a layer. Each layer is composed of a multi-head self-attention (SAN) and a feed-forward network (SAN) with
a special reversible design to ensure duplex behavior, where the input and output representations of such a layer are split by 2 halves, i.e., [H(1)l−1;H (2) l ] and [H (1) l−1;H (2) l ]. Formally, the regular form of the l-th layer Fl performs as follow:
Hl = Fl(Hl−1)⇔ [H(1)l ;H (2) l ] = Fl([H (1) l−1;H (2) l−1]),
where H(1)l = H (1) l−1 + SAN(H (2) l−1), H (2) l = H (2) l−1 + FFN(H (1) l ).
The reverse form of F−1l can be computed by subtracting (instead of adding) the residuals:
Hl−1 = F−1l (Hl)⇔ [H (1) l−1;H (2) l−1] = F−1l ([H (1) l ;H (2) l ]),
where H(2)l−1 = H (2) l − FFN(H (1) l ), H (1) l−1 = H (1) l − SAN(H (2) l−1).
For better modeling the reordering between source and target languages, we employ relative selfattention [Shaw et al., 2018] instead of the original one [Vaswani et al., 2017].
Homogeneity: Symmetric network architecture without encoder-decoder framework. To meet our need to ensure homogeneous network computations for forward and reverse directional tasks, we therefore choose to discard the encoder-decoder paradigm.
Symmetric network. To achieve homogeneous computations, one solution is to make our network symmetric, as depicted at the top of Figure 2. Specifically, we let the 1-th to L/2-th layers be the reverse form, whereas the latter (L/2 + 1)-th to L-th be the regular form:
f→θ (x) , F−11 ◦ · · · ◦ F−1L/2 ◦ FL/2+1 ◦ · · · ◦ FL(x), f←θ (y) , FL ◦ · · · ◦ FL/2+1 ◦ F−1L/2 ◦ · · · ◦ F−11 (y),
where ◦ means a layer is connected to the next layer. Thereby the forward and reverse computations of REDER become homogeneous: the forward computational operation series reads as a palindrome string 〈fs · · · fssf · · · sf〉 and so does the reverse series, where s and f denote SAN and FFN. Fully non-autoregressive modeling. Note that without encoder-decoder division, REDER works in a fully non-autoregressive fashion in both reading and generating sequences. Specifically, given an input sequence x, H0,i = [H (1) 0,i ;H (2) 0,i ] = [e(xi); e(xi)], is the i-th element of REDER’s input, which is the concatenation of two copies of the embedding of xi. Once a forward computation is done, the concatenation of the output of the model [H(1)L,i;H (2) L,i] serves as the representations of target translation. And then, a softmax operation is performed to measure the similarity between the model output [H(1)L,i;H (2) L,i] and the concatenated embedding of ground-truth reference [e(yi), e(yi)], to obtain the prediction probability:
p(yi|x; θ) = softmax([e(yi); e(yi)]>[H(1)L,i;H (2) L,i]/2).
We can likewise derive the procedure of f←θ for the target-to-source direction. Due to the conditional independence assumption among target tokens introduced by non-autoregressive generation, the log-likelihood of a translation becomes:
log p(y|x; θ) = ∑ i log pθ(yi|x)
Modeling variable-length input and output. Encoder-decoder models can easily model variablelength input and output of most seq2seq problems. However, discarding encoder-decoder separation imposes a new challenge: the width of all the layers of the network is depending on the length of the input, thus it is very difficult to allow variable-length input and output, especially when the input is shorter than the output. We resort to the Connectionist Temporal Classification (CTC) [Graves et al., 2006] to solve this problem, a latent alignment approach with superior performance and the flexibility of variable length prediction. Given the conditional independence assumption, CTC is capable of efficiently finding all valid alignments a which derives from the target y by allowing consecutive repetitions and inserting blank tokens, and marginalizes log-likelihood:
log pctc(y|x; θ) = log ∑
a∈Γ(y) pθ(a|x),
where Γ−1(a) is the collapse function that recovers the target sequence by collapsing consecutive repeated tokens, and then removing all blank tokens. Note that CTC requires that the length of source input should not be smaller than the target output, which is not the case in machine translation. To deal with this, we follow previous useful practice by upsampling the source tokens by 2 times [Saharia et al., 2020, Libovický and Helcl, 2018], and filter those examples when the target lengths are still larger than the one of upsampled source sentences. Remark. Reversibility in REDER can be assured in the continuous representation level, where REDER can recover from output representations (last layer) to input embeddings (first layer), which is also the motivation and basis of the auxiliary learning signal, i.e. Lfba, in the next section. Reversibility might not hold in the discrete token level, because of the existence of irreversible operations, e.g. the argmax operation discretizes probabilities to tokens and the CTC collapse process. But REDER still shows decent reconstruction capability in practice, as visually depicted in the experiment section.

heading: 2.3 Training
text: Given a parallel corpus and a single model θ, REDER can be jointly supervised by source-to-target and target-to-source translation for f→θ and f ← θ , respectively. Thus both translation directions can be achieved in one REDER model. We refer this to bidirectional training, which is opposite to unidirectional training, where each translation direction needs a separate model. Moreover, the reversibility of REDER enables appealing potentials to exploit consistency/agreement between forward and reverse directions. We introduce two auxiliary learning signals as follows.
Layer-wise Forward-Backward Agreement Since REDER is fully reversible, which consists of a series of computationally inverse of intermediate layers, an interesting question arises: given the desired output (i.e., the target sentence), is it possible to derive the desired intermediate hidden representation by the backward target-to-source computation, and encourage the forward source-totarget intermediate hidden representations as close as possible to these “optimal” representations?
Given a source sentence x, the inner representations of each layer in the forward direction are: −→ H1 = F1(e(x)), −→ H2 = F2( −→ H1) . . . −→ HL = FL( −→ HL−1),
and given its corresponding target sequence y as the optimal desired output4, the inner representations of each layer in the reverse direction are:
←− HL = F−1L (e(y)), ←− HL−1 = F−1L−1( ←− HL) . . . ←− H1 = F−1L ( ←− H2),
where −→ Hl and ←− Hl represent the representations of l-th layer in forward and reverse models, respectively. As we consider these reverse inner layer representations as “optimal”, we try to minimize the cosine distance between the forward and backward corresponding inner layer representations:
Lfba(x|y; θ) = 1
L L∑ l=1 1− cos(−→Hl, sg(←−Hl)),
where sg(·) denotes the stop-gradient operation. Cycle Consistency The symmetry of a pair of seq2seq tasks enables the use of cycle consistency [He et al., 2016, Cheng et al., 2016a]. Given a source sentence x, we first obtain the forward prediction, and then we use the REDER to reconstruct this prediction to the source language:
ȳ = f→θ (x), x̄ = f ← θ (ȳ).
Finally, we maximize the consistency or agreement between the original one x and reconstructed one x̄. Thus, the loss function reads
Lcc(x; θ) = distancecc(x, f←θ (f→θ (x))) = distancecc(x, x̄). We expect it can provide an auxiliary signal that a valid prediction should be loyal to reconstruct its source input. Here we use cross-entropy between the probabilistic prediction of the reverse model as distance to measure the consistency.
4for CTC-based model where the model predictions are the alignments, we instead extract the token sequence of the best alignment a∗, predicted by the model, associated with the ground-truth y as the optimal desired output.
A potential danger of both of the above auxiliary objectives is learning a degenerate solution, where it would probably cheat this task by simply learning an identity mapping. We solve this problem by setting a two-stage training scheme for them, where we first train REDER without using any auxiliary losses until a predefined number of updates, and then activate the additional losses and continue training the model until convergence.
Final Objective Given a parallel dataset Dxy = {(x(n),y(n)|n = 1...N} of i.i.d observations, the final objective of REDER is to minimize
L(θ;Dxy) = N∑ n=1 ( − log pctc(y(n)|x(n); θ)− log pctc(x(n)|y(n); θ)
+ λfbaLfba(x(n)|y(n); θ) + λfbaLfba(y(n)|x(n); θ)︸ ︷︷ ︸ layer-wise forward-backward agreements
+ λccLcc(x(n); θ) + λccLcc(y(n); θ)︸ ︷︷ ︸ cycle consistencies ) where λfba and λcc are coefficients of the auxiliary losses.

heading: 3 Related Work
text: Sequence-to-Sequence Models Exploiting Bidirectional Signals. Several studies try to utilize bidirectionality as a constraint to improve sequence-to-sequence tasks such as machine translation [Cheng et al., 2016a,b]. Dual learning [He et al., 2016, Xia et al., 2017] leverages reinforcement learning to interact between two simplex translation models. Later, Xia et al. [2018] propose a partially model-level dual learning that shares some components of both models for forward and reverse tasks. Zheng et al. [2020] propose to model the two directional translation model with language models in a variational probabilistic framework. These approaches model two directional tasks by setting up two separate simplex models to consider the task bidirectionality. Different from them, REDER can unify a pair of directions within one duplex model and directly model the bidirectionality at a completely model level. Besides, other studies try to unify two directional tasks by multitask learning [Johnson et al., 2017, Chan et al., 2019] by sharing the same computational process of a single simplex model, which would inevitably result in parameter interference issue where two tasks compete for the limited model capacity. In contrast, REDER formulates both tasks in one model by simply exchanging input and output ends, each of which specializes in a language, thus bidirectional translation becomes a reversible process in which both directions do not need to compete for limited model capacity.
Non-autoregressive Sequence Generation. Non-autoregressive translation (NAT) models [Gu et al., 2018] aims to alleviate the decoding inefficiency of traditional autoregressive seq2seq models. Fully NAT models could generate sequence in parallel within only one shot but sacrifice performance [Ma et al., 2019, Shu et al., 2020, Bao et al., 2019, Wei et al., 2019, Qian et al., 2021, Gu and Kong, 2021, Huang et al., 2021]. Besides, semi-autoregressive models greatly improve the performance of NAT models, which perform iterative refinement of translations based on previous predictions [Lee et al., 2018, Ghazvininejad et al., 2019, Gu et al., 2019, Kasai et al., 2020, Ghazvininejad et al., 2020]. In this work, REDER takes the advantage of the probabilistic modeling of fully NAT models for resolving the designing challenge of computational homogeneity for both translation directions.
Reversible Neural Architectures. Various reversible neural networks have been proposed for different purposes. On one hand, reversible neural networks help model flexible probability distributions with tractable likelihoods [Dinh et al., 2014, 2017, Kingma et al., 2016], which define a mapping between a simple, known density and a complicated desired density. On the other hand, reversibility can also assist to develop memory-efficient algorithms. The most popular approach is the reversible residual network (RevNet, Gomez et al., 2017), which modifies the residual network and allows the activations at any given layer to be recovered from the activations at the following layer. Therefore layers can be reversed one by one as back-propagation proceeds from the output of the network to its input. Some follow-up work extends the idea of RevNet to RNNs [MacKay et al., 2018] and Transformer [Kitaev et al., 2020] in NLP. We borrow the idea of RevNet as the basic unit of our proposed REDER, however, for different purposes that we want to build a duplex seq2seq model to govern two directional tasks reversibly. In this line, van der Ouderaa and Worrall [2019] propose
a reversible GAN approach for image-to-image translation in computer vision, which to a certain extent shares the intuition with ours.

heading: 4 Experiments
text: We conduct extensive experiments on standard machine translation benchmarks to inspect REDER’s performance on seq2seq tasks. We demonstrate that REDER achieves competitive results, if not better, compared to strong autoregressive (AT) and non-autoregressive (NAT) baselines. REDER is also the first approach that enables reversible machine translation in one unified model, where bidirectional training with paired translation directions surprisingly helps boost each of them with substantial gains.

heading: 4.1 Experimental Setup
text: Datasets. We evaluate our proposal on two standard translation benchmarks, i.e., WMT14 English (EN)↔ German (DE) (4.5M training pairs), and WMT16 English (EN)↔ Romanian (RO) (610K training pairs). We apply the same prepossessing steps as mentioned in prior work (EN↔DE: Zhou et al., 2020, EN↔RO: Lee et al., 2018). BLEU [Papineni et al., 2002] is used to evaluate the translation performance for all models.
Knowledge Distillation (KD). Sequence-level knowledge distillation [Kim and Rush, 2016] is found to be crucial for training NAT models. Following previous NAT studies [Gu et al., 2018, Zhou et al., 2020], REDERs are trained on distilled data generated from pre-trained auto-regressive Transformer models. The beam size is set to 4 during generation.
Beam Search Decoding and AT Reranking. We implement two kinds of inference policies. The first one is parallel decoding that adopts tokens with the highest probability at each position. For CTC-based models, we also implement beam search to REDER with an efficient library of C++ implementation5. Similar to previous NAT literature, we further rerank the decoded candidates produced by beam search using autoreregressive models as the external scorer [Gu et al., 2018] and pick the best ones as final results.
Implementation Details. We design REDER based on the hyper-parameters of Transformerbase [Vaswani et al., 2017]. All models are implemented on fairseq [Ott et al., 2019]. REDER consists of 12 stacked layers. The number of head is 8, the model dimension is 512, and the inner dimension of FFN is 2048. For both AT and NAT models, we set the dropout rate 0.1 for WMT14 EN↔DE and WMT16 EN↔RO. We adopt weight decay with a decay rate 0.01 and label smoothing with = 0.1. By default, we upsample the source input by a factor of 2 for CTC-based models. We set λfba and λcc to 0.1 for all experiments. All models are trained for 300K updates using Nvidia V100 GPUs with a batch size of approximately 64K tokens. Following prior studies [Vaswani et al., 2017], we compute tokenized case-sensitive BLEU. We measure the validation BLEU scores for every 2,000 updates, and average the best 5 checkpoints to obtain the final model. Similar to previous NAT studies, we also measure the GPU latency by running the model with one sentence per batch on WMT14 EN-DE test set on a single GPU and give speedup comparing over our AT baselines.6

heading: 4.2 Main Results
text: As shown in Table 1, we compare REDER with AT and NAT approaches with and without multitask learning, as well as existing approaches that also leverage bidirectional learning signals.
REDER achieves competitive results compared with strong NAT baselines. We show that a unified REDER trained on the same parallel data can simultaneously work in two directions, which has a comparable capability as strong NAT models such as GLAT [Qian et al., 2021] (row 14 vs. row 8). With the help of beam search and re-ranking, the performance of REDER can be further boosted to be comparable with the state-of-the-art NAT method GLAT+CTC [Gu and Kong, 2021] (row 14 vs. row 9 & 10). Gu and Kong [2021] explore the best technique combination for NAT, whose tricks can also supplement to enhance REDER. We leave this for exploration.
5https://github.com/parlance/ctcdecode 6Note that as this paper’s goal is not for decoding efficiency, all models are not optimized for latency using
advanced techniques, and the autoregressive baselines are hence weak in terms of latency.
Duplex learning has more potential than multitask learning and back-translation. Given the same parallel corpus as training data, multitask learning (MTL) yields considerable performance degradation of either AT (row 1 vs. row 2). By re-implementing GLAT+CTC [Gu and Kong, 2021] (row 10 & 11) as the strong NAT competitor for more convincing comparison. We observe that MTL would hurt more severely for NAT models, such as GLAT+CTC models (row 11 vs. row 10), and REDER as well (row 13 vs. row 12). These results verify our concern of multitask-learned models regarding parameter interference. Meanwhile, when no external monolingual resources are available, back-translation (BT) only adds mild points from training data. In contrast, duplex learning allows REDER to gain more benefits, becoming a better alternative and a parameter-efficient choice to exploit more potentials from the provided parallel data. Plus, duplex learning is orthogonal to BT, which could further improve REDER with monolingual data.
REDER performs on par with autoregressive Transformer. Despite the challenge in regards to non-autoregressive modeling and non-encoder-decoder design, REDER closely approaches the simplex AT models (row 14 vs. row 1), while REDER can translate both directions in one model. Besides, REDER even surpasses the multitask-learned bidirectional autoregressive Transformer model (row 14 vs. row 2). Additionally, REDER enjoys faster decoding spend than the baseline autoregressive models. This evidence shows the advantage and practical value of reversible machine translation as a more decent solution for parameter-efficient bidirectional translation systems.
Comparison with existing approaches. Reformer [Kitaev et al., 2020] also employs RevNet to make parts of the Transformer model to reduce the memory consumption for training, while REDER is fully reversible with a different motivation of maximizing the use of bidirectional signals. Existing simplex approaches exploiting bidirectional signals require two separate simplex models for both directions [Xia et al., 2018, Zheng et al., 2020]. REDER, in contrast, only needs one unified duplex model and coherently models two directions. Alternatively, Chan et al. [2019] use a single simplex network to achieve bidirectional translation via multitask learning, needing to split limited capacity for both directions, which underperform REDER on parallel settings.
Training cost. We train REDER on WMT14 EN↔DE using 8 32GB V100 GPUs for 432 hours (54 hour per GPU) and obtained a bidirectional translation model. For modeling both directions, a standard NAT model needs 640 GPU hours (320×2) in total, whilst the autoregressive Transformer needs 400 GPU hours (200×2) using the same computational resources. Therefore, the training costs of these methods are comparable.
4.3 Ablation Study of Model Design
REDER is developed on the top of various components in terms of data (knowledge distillation), learning objective (CTC), architecture (revnet, relative attention), and auxiliary losses endowed by reversibility of REDER. We analyze their effects through various combinations in Table 2. We first consider training REDER for a single direction to seek the best practice to run REDER for sequence-to-sequence tasks. KD and CTC are essential to training REDER, as suggested by previous NAT studies [Saharia et al., 2020, Gu and Kong, 2021]. Meanwhile, we notice the benefit of relative self-attention. We therefore use these three techniques by default for all of the proposed models. As for the duplex variants of models that learn both directions simultaneously, they can further improve the translation accuracy by substan-
tial margins. These results verify our motivation that the paired translation directions could be better learned in a unified reversible model. Reversibility enables us to utilize layer-wise forward-backward agreement and cycle consistency, which are also shown to boost improvement considerably.

heading: 4.4 Decoding: Effect of Beam search and Re-ranking
text: The performance of REDERs can be further boosted with additional (beam-search or re-ranking) techniques. For CTC beam search, we use the teacher model (AT base) to re-rank the translation candidates obtained by the beam search to determine the one with the best quality. As shown in Table 3, a larger beam size results in a smaller BP for AT models, meaning it produces shorter translations [Stahlberg and Byrne, 2019, Eikema and Aziz, 2020]. CTC beam search helps REDER produce longer outputs (larger BPs) but only endows a little
improvement. With beam search and AT reranking, REDER can generate more decent translations. These results imply that we need to find a better way to train REDER (and probably the NAT family) if we do not want to involve an extra AT for such a somewhat inconvenient re-ranking.

heading: 4.5 Training: Impact of Knowledge Distillation
text: Like other NAT approaches, we find that REDER heavily relies on knowledge distillation. We report the performance of models trained on raw data and distilled data generated from AT models in Table 4. As we can see, without KD, the accuracy of REDER significantly drops. We then aim to explore the most proper way to integrate KD data. We observe that if we only use KD data of one direction (only target-side data are distilled,
e.g., German sentences in EN-DE), it only benefits a single direction. These imply that we need to provide KD data of both directions to train REDER. Furthermore, we notice that if we mix the KD data of both directions by concatenating them directly, it somehow improves results compared to the strategy only using single-direction KD data. Finally, we find the best practice is to separately feed KD data in accordance to directions, i.e., feeding EN-DE KD data when training the EN-DE direction and providing DE-EN KD data when training the reverse direction (i.e., DE-EN).
Discussion. Like other NAT approaches, the proposed REDER resorts to, and unfortunately heavily relies on, KD data for training. Requiring KD does hinder the generalization of NAT models including REDER to other applications especially multilingual scenarios. We notice that recent studies could have the potentials for the removal of KD for NAT models, through introducing latent-variable models [Gu and Kong, 2021, Bao et al., 2019] or sampling/denoising-based augmented training objectives/strategies [Qian et al., 2021] As KD-dependence is a common issue for all NAT approaches, we believe future breakthroughs would resolve this. Please note that the aim of this paper is to make the idea of duplex sequence-to-sequence learning and its implementation of REDER realizable, at least in the scenario with KD data. Eliminating the need for KD is orthogonal to the purpose of this paper, however, is very valuable for further exploration.
4.6 Analysis of Reversibility
We examine the reversibility of REDER by both quantitative and qualitative analysis. We first measure the BLEU score between source sentences x and the associated reconstructions, i.e. BLEU(x, f←θ (f → θ (x)), on the development sets of WMT14 EN-DE, which gives a score of 66.0. Besides, we also show an example regarding its forward prediction and reconstruction in Figure 3. Given a sentence in the source lan-
guage (EN), we first use the forward mapping of REDER to obtain a prediction in the target language (DE), and then translate it back to the source language using the reverse model. As shown in Figure 3, REDER can reconstruct the input from the output with mild differences to some extent. These results demonstrate that REDER meets the definition of reversibility empirically to a certain extent.
4.7 Experiments on Distant Languages
To examine whether REDER can generalize to distant languages, we conducted experiments on WMT20 EnglishJapanese (EN↔JA), where the training data is much larger and two languages are linguistically distinct with almost no vocabulary overlap. As shown in Table 5, REDER can achieve very close results compared with AT in such a large-scale scenario with distant languages, showing that
reversible machine translation could make more potentials of parallel data.

heading: 5 Conclusion and Future Work
text: In this paper, we propose REDER, the REversible Duplex TransformER for sequence-to-sequence problem and apply it to machine translation that for the first time shows the feasibility of a reversible machine translation system. REDER is a fully reversible model that can transform one sequence to the other one forth and back, by reading and generating through its two ends. We verify our motivation and the effectiveness of REDER on several widely-used NMT benchmarks, where REDER shows appealing performance over strong baselines.
As for promising future directions, REDER can be applied to monolingual, multilingual and zero-shot settings, thanks to the fact that each “end” of REDER specializes in a language. For instance, given trained REDERsMEn↔De andMEn↔Ja, we combine last half layers (the DE end) ofMEn↔De and the JA end ofMEn↔Ja to obtain a zero-shotMDe↔Ja, translating between German and Japanese. Likewise, the composition of an English end and its reverse results inMEn↔En, which can learn from monolingual data like an autoencoder. This compositional fashion resembles LEGO, which manipulates only a linear number of language ends. Therefore, while adding a new language to a multilingual REDER system (in a form of composition of ends of involved languages), we would probably not need to retrain the whole system as we do for a current multilingual NMT system, which reduces the difficulty and cost to train, deploy and maintain a large scale multilingual NMT system.

heading: Acknowledgements
text: We would like to thank the anonymous reviewers for their insightful comments. Hao Zhou is the corresponding author. This work was supported by National Science Foundation of China (No. 61772261, 6217020152), National Key R&D Program of China (No. 2019QY1806).

Review summaries:
This paper proposes a duplex reversible network based on Transformer and revnet for neural machine translation. With the proposed approach, one can use one single model to translate between a language pair from both directions. The model uses CTC loss and allows non-autoregressive decoding for faster inference time. Experiments are well executed on standard NAT benchmark datasets with close performance to that of the standard Transformer model.  **Contributions**: the proposed model is new in machine translation from the best of my knowledge and achieves good performance compared to non-autoregressive models and multi-task learning. The proposed method saves parameters in terms of translation from both directions and improves inference speed.

This paper proposes duplex sequence-to-sequence learning where data streams can flow from both ends of a seq2seq model. More precisely, they design a parameter efficient Transformer (REDER) that can simultaneously input and output a distinct language from each end of the model. In this way, REDER enables reversible machine translation by simply flipping the input and output ends. The experimental results show that the proposed REDER improve the multitask trained baseline by over 1.3 BLEU while maintaining over 5x speed-up at inference time.

This paper proposes an architecture or a reversible NMT system. In contrast to an encoder-decoder network, the two components of the network for each language are identical. When a language is being analyzed or generated depends on the direction of data flow in the network. To enable this, the authors propose reversible components in each layer of the network (inspired from revnet). The model is trained with CTC loss along with auxiliary losses for cyclic consistency and layerwise agreement. The paper is an interesting proposal for a reversible NMT system, and it brings together work on revnet, CTC loss, NAT in a simple, elegant framework to enable reversible NMT. 

The paper proposes a novel machine translation model that can be simultaneously applied to bidirectional translations. The model, named REDER, is symmetric at both input and output ends and consists of several identical reversiable duplex Transformer layers. REDER adopts the design of revnet for reversibility and reverses the first L/2 layers for homogeneity, which is a fully non-autoregressive NMT model. Extensive experiments on WMT14 En$\leftrightarrow$De and WMT16 En$\leftrightarrow$Ro datasets show that the proposed model consistently outperforms (non-)autoregressive baselines with improving decoding speed.