Duplex Sequence-to-Sequence Learning for Reversible Machine Translation

1 Introduction . Neural sequence-to-sequence ( seq2seq ) learning [ Sutskever et al. , 2014 ] has been extensively used in various applications of natural language processing . Standard seq2seq neural networks usually employ the encoder-decoder framework , which includes an encoder to acquire the representations from the source side , and a decoder to yield the target side outputs from the encoded source representation [ Bahdanau et al. , 2015 , Gehring et al. , 2017 , Vaswani et al. , 2017 ] . Generally , given paired training data Dxy = X ×Y , where X is the source side and Y is target side of the corresponding seq2seq task , supervision signals are always bidirectional . Thus we can learn not only the mapping from source to target ( fθxy : X 7→ Y ) but also the mapping from target to source ( fθyx : Y 7→ X ) . This is very common in many applications . For example , given parallel corpus , we can obtain machine translation models from both Chinese to English and English to Chinese , or transfer between different stylized texts [ Yang et al. , 2018 , He et al. , 2019 ] . Typical seq2seq learning utilizes the bidirectional supervisions by splitting the bidirectional supervisions into two unidirectional ones and trains two individual seq2seq models on them , respectively . However , such splitting ignores the internal consistencies between the bidirectional supervisions . Thus how to make better use of the bidirectional supervisions remains an open problem . One potential solution is multitask learning [ Johnson et al. , 2017 ] , which jointly leverages the bidirectional supervisions within one model and expects the two unidirectional supervisions could boost each other ( Figure 1 ( b ) ) . But it does not work well in our case due to the parameter interference problem [ Arivazhagan et al. , 2019 , Zhang et al. , 2021 ] . For instance , in the setting of multitask Chinese-to-English and English-to-Chinese bidirectional translations , the encoder/decoder of seq2seq networks is trained to simultaneously understand/generate both Chinese and English , which always results in performance ∗Work was done when Zaixiang Zheng was a final-year PhD candidate at Nanjing University and an intern ( now FTE ) at ByteDance AI Lab ; and when Lei Li was also at ByteDance AI Lab . 1Code is available at https : //github.com/zhengzx-nlp/REDER . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . drop due to the divergent nature of the two languages . Another branch of solutions lies in cycle training [ Sennrich et al. , 2016 , He et al. , 2016 , Zheng et al. , 2020 ] , which still deploys two individual models for separately learning , but explicitly regularize their outputs by the cycle consistency of seq2seq problems . Such approaches usually need to introduce extra monolingual data in machine translation . Layer lpad rpad Dr in k mo re ho t wa te r rpad rpadlpad source language target language neural networks E n c o d e r D e c o d e r source language target language neural networks Cross Attention simplex communication D e c o d e r E n c o d e r Cross Attention simplex communication In this paper , we propose to explore an alternative approach for utilizing bidirectional supervisions called duplex sequence-to-sequence learning . We argue that current seq2seq networks do not benefit from multitask training because the seq2seq networks are simplex from the view of telecommunications2 . Specifically , the data stream only flows from encoder to decoder in the current seq2seq learning , and such simplex property makes the multitask learning suffer from the parameter interfere problem . Instead , if we have duplex neural networks , where the data stream can flow from both ends , each of which only specializes in one language when learning from bidirectional supervisions ( Figure 1 ( c ) ) , thus potentially alleviating the parameter interference problem . As a result , the bidirectional translation can be achieved as a reversible and unified process . In addition , we could still incorporate cycle training in the duplex networks using only one single model . Definition 1 . A sequence-to-sequence neural network with parameter θ is duplex if it satisfies the following : its network has two ends , i.e. , a source end and a target end ; both source and target ends can take input and output sequences ; its network defines a forward mapping function f→θ : X 7→ Y , and a reverse mapping function f←θ : Y 7→ X , that satisfies the following reversibility : f←θ = ( f → θ ) −1 and f→θ = ( f ← θ ) −1 ; besides , it satisfies the following cycle consistency : ∀x ∈ X : f←θ ( f→θ ( x ) ) = x and ∀y ∈ Y : f→θ ( f←θ ( y ) ) = y . Based on the idea of duplex network , we propose REDER3 , the REversible Duplex TransformER , and apply it to machine translation . Note that , building duplex seq2seq networks is non-trivial : a ) vanilla encoder-decoder network is irreversible . The output end of the decoder can not take in input signals to exhibit the encoding functionality and vice versa ; b ) the topologies of the encoder and the decoder are heterogeneous , i.e. , the decoder works autoregressively , while the encoder is non-autoregressive . To this end , we therefore design REDER without explicit encoder and decoder division , in which we introduce the reversible Transformer layer [ Gomez et al. , 2017 ] and fully non-autoregressive modeling to solve the above two problems respectively . As a result , REDER works in the duplex fashion , which could better exploit the bidirectional supervisions for achieving better downstream task performance . Experimental results show that the duplex idea indeed works : Overall , REDER achieves BLEU scores of 27.50 and 31.25 on standard WMT14 EN-DE and DE-EN benchmarks , respectively , which are top results among non-autoregressive machine translation models . REDER achieves significant gains ( +1.3 BLEU ) compared to its simplex baseline , whereas multitask learning does hurt the translation performance of simplex models both in the autoregressive ( -0.5 BLEU ) and nonautoregressive settings ( -1.3 BLEU ) . Although REDER adopts fully non-autoregressive modeling to realize the duplex networks , the gap of BLEU between REDER and autoregressive Transformer is negligible , meanwhile REDER can directly translate between two directions and enjoys 5.5× inference speedup as a bonus of non-autoregressive modeling . To our best knowledge , REDER is the first duplex seq2seq network , which enables the first feasible reversible machine translation system . 2In telecommunications and computer networking , the simplex communication means the communication channel is unidirectional while the duplex communication is bidirectional . 3The model ’ s name is a palindrome , which implies the model works from both ends . 2 REDER for Reversible Machine Translation . In this section , we introduce how to design a duplex neural seq2seq model , the REDER ( REversible Duplex TransformER ) , that satisfies Definition 1 , and its application in machine translation that realizes the first neural reversible machine translation system . 2.1 Challenges of Reversible Machine Translation . Reversible natural language processing [ Franck , 1992 , Strzalkowski , 1993 ] and its applications in machine translation [ van Noord , 1990 ] were proposed for the purpose of building machine models that understand and generate natural languages as a reversible , unified process . Such process resembles the mechanism of the ability that allows human beings to communicate with each other via natural languages [ Franck , 1992 ] . Despite the success of neural machine translation with deep learning , designing neural architectures for reversible machine translation yet remains under-studied and has the following challenges : Reversibility . Typical encoder-decoder networks and their neural components , such as Transformer layers , are irreversible , i.e. , one can not just obtain its inverse function by flipping the same encoderdecoder network . To meet our expectation , an inverse function of the network should be derived from the network itself . Homogeneity . Intuitively , a pair of forward and reverse translation directions should resemble a homogeneous process of understanding and generation . However , typical encoder-decoder networks certainly do not meet such computational homogeneity due to extra cross attention layers in the decoder ; and also because of the discrepancy that the decoder works autoregressively but the encoder does non-autoregressively . To meet our expectations , the separation of encoder and decoder should be no more exist in the desired network . 2.2 The Architecture of REDER . To solve the above challenges , we include two corresponding solutions in REDER to address the reversibility and homogeneity issues respectively , i.e. , the Reversible Duplex Transformer layers , and the symmetric network architecture without the encoder-decoder framework . Layer 多 喝 热 ⽔lpad rpad Dr in k mo re ho t wa te r rpad 点 rpadlpad S e lf A tt e n tio n F F N Te fl- A tt e n tio n y y + + SAN FFN − − SAN FFN Duplex Communication so ur ce ta rg et X ↦ YX Y↦ x ( 1 ) ( 1 ) ( 2 ) x x y y ( 1 ) ( 2 ) ( 1 ) ( 2 ) Regular Form ( ) Reverse Form ( ) _____S AN _____ _____FFN _____ RevNet-style Transformer Layer : ( 1 ) x ( 2 ) _____S AN _____ _____FFN _____ _____S AN _____ _____FFN _____ __ __ _S AN __ __ _ __ __ _F FN __ __ _ __ __ _S AN __ __ _ __ __ _F FN __ __ _ __ __ _S AN __ __ _ __ __ _F FN __ __ _ x ( 2 ) [ , ] ↦ [ , ] x ( 1 ) x ( 2 ) y ( 1 ) y ( 2 ) 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 Duplex Sequence to Sequence Learning for Reversible Machine Translation Layer lpad rpad Dr in k mo re ho t wa te r rpad rpadlpad S e lf A tt e n tio n F F N Te fl- A tt e n tio n x x y y + + SAN FFN SAN FFN Duplex Communication so ur ce ta rg et X YX Y ( 1 ) ( 2 ) ( 1 ) ( 2 ) x x y y ( 1 ) ( 2 ) ( 1 ) ( 2 ) Regular Form Inverse Form _____S AN _____ _____FFN _____ Revnet Transformer Layer : [ x ( 1 ) , x ( 2 ) ] [ y ( 1 ) , y ( 2 ) ] ( 1 ) x ( 2 ) _____S AN _____ _____FFN _____ _____S AN _____ _____FFN _____ __ __ _S AN __ __ _ __ __ _F FN __ __ _ __ __ _S AN __ __ _ __ __ _F FN __ __ _ __ __ _S AN __ __ _ __ __ _F FN __ __ _ Figure 2 . The proposed REDER for duplex sequence-to-sequence generation . The bottom two diagrams show the computation of the regular and reverse forms of a reversible layer . Notice that , to make the whole model symmetric , we reverse the 1-th to L/2-th layers , such that the overall computational operations of forward and reverse of REDER are homogeneous . duplex sequence-to-sequence is distinct from a joint multilingual MT model ( e.g . multilingual Transformer ) . The multilingual Transformer takes input sentences in two ( or more ) languages only from the same end and outputs to the other end . Its output end can not be used as the input . 4 . REDER : REEVERSIBLE DUPLEX TRANSFORMER In this section , we design the architecture of a duplex sequence-to-sequence generation model for machine translation , REDER , that satisfies D finition 1 . F F 1 4.1 . Challenge of Reversible Translation Reversible machine translation is originally mentioned in van Noord ( 1990 ) . However , designing such reversible MT has the following challenges : ( i ) Reversibility . An inverse function of the network should be computed by reversing the connections in the network . Ordinary neural networks , such as convo- lutional layers , are not reversible , i.e . one can not just obtain its i verse function from the same convolutional nets . ( ii ) Symmetry . Both ends of the desired model can take input and produce output , and the forward and reverse computation flows move through the same model architecture . As a result , the head-side computation and the tail-side computation should resemble certain equivalence . The normal encoder-decoder architecture such as the Transformer certainly does not meet such input/output equivalence property due to the cross attention layer in Transformer which is only from the decoder to encoder . There should not be a division of encoder and decoder in the desired duplex model . ( iii ) Input-output length mismatch . The input and output sequences are not of the same length . Without encoderdecoder separation , an idea duplex model should match the sequences at different lengths . 4.2 . REDER Architecture To solve the above three challenges , we include three components in REDER , including reversible layers , equi-width duplex composition scheme , CTC loss . Figure 2 shows the overall architecture of the REDER model . For the following , ✓ is the model parameter . Reversible layers . We adopt the idea of reversible residual network ( revnet , Gomez et al. , 2017 ; Kitaev et al. , 2020 ) in our reversible layer design . Each REDER reversible layer is composed of a multi-head attention block and a feedforward block with a special reversible design to ensure duplex behavior . Formally , the regular form of the l-th layer Fl performs as follow : Hl = Fl ( Hl 1 ) , [ H ( 1 ) l ; H ( 2 ) l ] = Fl ( [ H ( 1 ) l 1 ; H ( 2 ) l 1 ] ) , where H ( 1 ) l = H ( 1 ) l 1 + SAN ( H ( 2 ) l 1 ) , H ( 2 ) l = H ( 2 ) l 1 + FFN ( H ( 1 ) l ) . Accordingly , the reverse form of F 1l can be computed by subtracting ( rather than adding ) the residuals : Hl 1 = F 1l ( Hl ) , [ H ( 1 ) l 1 ; H ( 2 ) l 1 ] = F 1l ( [ H ( 1 ) l ; H ( 2 ) l ] ) , where H ( 2 ) l 1 = H ( 2 ) l FFN ( H ( 1 ) l 1 ) , H ( 1 ) l 1 = H ( 1 ) l SAN ( H ( 2 ) l 1 ) . For better capability of the reordering between source and target languages , we employ relative self-attention ( Shaw et al. , 2018 ) instead of the original one in the original Transformer ( Vaswani et al. , 2017 ) . Equi-width duplex composition scheme . To make the model symmetric , we create an architecture with equal- width duplex and symmetric composition of the reversible layers . However , given the differences in the regular form and reverse form each layer is reversible but not symmet- ric , thus the model simply stacking L reversible layers is overall asymmetric for both directions . To make our model symmetric for both directions , we let the 1-th to L/2-th layers be the inverse form of the reversible Transformer layer , whereas the latter ( L/2 + 1 ) -th to L-th be the regular version . Thereby the forward and reverse computational operations of the whole model become homogeneous , which 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 Duplex Seque ce to Sequence Learning for Reversible Machine Translation Layer lpad rpad Dr in k mo re ho t wa te r rpad rpadlpad S e lf A tt e n tio n F F N Te fl- A tt e n tio n x x y y + + SAN FFN SAN FFN Duplex Communication so ur ce ta rg et X YX Y ( 1 ) ( 2 ) ( 1 ) ( 2 ) x x y y ( 1 ) ( 2 ) ( 1 ) ( 2 ) Regular Form Inverse Form _____S AN _____ _____FFN _____ Revnet Transformer Lay r : [ x ( 1 ) , x ( 2 ) ] [ y ( 1 ) , y ( 2 ) ] ( 1 ) x ( 2 ) _____S AN _____ _____FFN _____ _____S AN _____ _____FFN _____ __ __ _S AN __ __ _ __ __ _F FN __ __ _ __ __ _S AN __ __ _ __ __ _F FN __ __ _ __ __ _S AN __ __ _ __ __ _F FN __ __ _ Figure 2 . The proposed REDER for duplex sequence-to-sequence genera ion . The bottom two diagrams show the computation of the regular and reverse forms of a reversible layer . Notice that , to make the whole model symmetric , we reverse the 1-th to L/2-th layers , such that the overall computational operations of forward and reverse of REDER are homogeneous . duplex seq ence-to-sequence is distinct from a joint multilingual MT model ( e.g . multilingual Transformer ) . The multilingual Transformer takes input sentences in two ( or more ) languages only from the same end and outputs to the other end . Its output end can not be used as the input . 4 . REDER : REEVERSIBLE DUPLEX TRANSFORMER In this section , we design the architecture of a duplex seque ce-to-sequence generation model for machine translation , REDER , that satisfies Definition 1 . F F 1 4.1 . Challenge of Reversible Translation Reversible machine translation is originally mentioned in van Noord ( 1990 ) . However , d signing such reversible MT has the following challenges : ( i ) R versibility . An inverse function of the network sh uld be co puted by reversing the connections n the network . Or inary n ur l networks , such as c nvolutional layers , are not reversible , i.e . one can not just btain its inverse function ro the same convolutiona nets . ( ii ) Symmetry . Both ends of th desired mo el an take input and produce output , and the fo ward and r verse computation flows move through the sa e model architecture . As a result , the head-side computation and the tail-side computation s ould res mble cert in equivalence . The normal encoder-dec der architecture such a the Transformer certainly does not meet such input/output equivalence property due to the cross attention layer in Transformer which is only from the dec der to encoder . There should not be a division of encoder and decoder in the desired duplex model . ( iii ) Inp -output length mismatch . The input and output s quences are not of the same length . Without encoderdecoder separation , an idea duplex model should match the sequences at different lengths . 4.2 . REDER Architecture To solv the above three challenges , we include three components in REDER , including reversible layers , equi-width duplex composition scheme , CTC loss . Figure 2 shows the ov ral architecture of the REDER model . For the following , ✓ is the model parameter . Rev rsible layers . We adopt the idea of reversible residual n work ( revnet , Gomez et al. , 2017 ; Kitaev et al. , 2020 ) in our reversible layer design . Each REDER reversible layer is composed of a multi-head attention block and a feedforward block with a special reversible design to ensure duplex behavior . Formally , the regular form of the l-th layer Fl performs as follow : Hl = Fl ( Hl 1 ) , [ H ( 1 ) l ; H ( 2 ) l ] = Fl ( [ H ( 1 ) l 1 ; H ( 2 ) l 1 ] ) , where H ( 1 ) l = H ( 1 ) l 1 + SAN ( H ( 2 ) l 1 ) , H ( 2 ) l = H ( 2 ) l 1 + FFN ( H ( 1 ) l ) . Accordingly , the reverse form of F 1l can be computed by subtracting ( rather than adding ) the residuals : Hl 1 = F 1l ( Hl ) , [ H ( 1 ) l 1 ; H ( 2 ) l 1 ] = F 1l ( [ H ( 1 ) l ; H ( 2 ) l ] ) , where H ( 2 ) l 1 = H ( 2 ) l FFN ( H ( 1 ) l 1 ) , H ( 1 ) l 1 = H ( 1 ) l SAN ( H ( 2 ) l 1 ) . For better capability of the reordering between source and target languages , we employ relative self-attention ( Shaw et al. , 2018 ) instead of the original one in the original Transformer ( Vaswani et al. , 2017 ) . Equi-width duplex composition scheme . To make the model symmetric , we create an architecture with equal- width duplex and symmetric composition of the reversible layers . H wever , given the differences in the regular form a d reverse form each layer is reversible but not symmet- r c , thus the model simply stacking L reversible layers is overall asymmetric for both directions . To make our model symmetric for both directions , we let the 1-th to L/2-th layers be the inverse form of the reversible Transformer layer , where s the latter ( L/2 + 1 ) -th to L-th be the regular version . Thereby the forward and reverse computational operations of the whole model become homogeneous , which Figure 2 : The proposed REDER for duplex sequenceto-sequence gener ti n. The bottom wo diagrams show the computation of the regular and reverse forms of a reversible layer . Notice that , to make the whole model symmetric , we reverse the 1-th to L/2-th layers , such that th eral computation l operat ns of forward and reverse of REDER are homogeneous . Figure 2 shows the overall architecture of REDER . As illustrated , REDER has two ends : the source end ( left ) and the target end ( right ) . θ is the model parameter , shared by both directions . The architecture of REDER is composed of a series of identical Reversible Duplex Transformer layers . When performing the source-to-target mapping f→θ , a source sentence x ( blue circles ) 1 ) first transforms to its embedding e ( x ) and enters the source end ; 2 ) then goes through the entire stacked layers and evolves to final representations HL which predicts probabilities ; 3 ) finally its target translation ( orange circles ) will be generated from the target ends . The generation process is fully nonautoregressive . Likewise , the target-to-source mapping f←θ is achieved by reversely executing the architecture of REDER from target end to source end . We will dive into the details of the key components of REDER in the following parts . Reversibility : Reversible Duplex Transform r lay rs . We adopt the idea of the reversible residual network ( RevNet , Gomez et al. , 2017 , K ta v et l. , 2020 ) in the design of t e versible duplex Transformer layer . The bottom of Figure 2 illustrates the forward and reverse computations of a layer . Each layer is composed of a multi-head self-attention ( SAN ) and a feed-forward network ( SAN ) with a special reversible design to ensure duplex behavior , where the input and output representations of such a layer are split by 2 halves , i.e. , [ H ( 1 ) l−1 ; H ( 2 ) l ] and [ H ( 1 ) l−1 ; H ( 2 ) l ] . Formally , the regular form of the l-th layer Fl performs as follow : Hl = Fl ( Hl−1 ) ⇔ [ H ( 1 ) l ; H ( 2 ) l ] = Fl ( [ H ( 1 ) l−1 ; H ( 2 ) l−1 ] ) , where H ( 1 ) l = H ( 1 ) l−1 + SAN ( H ( 2 ) l−1 ) , H ( 2 ) l = H ( 2 ) l−1 + FFN ( H ( 1 ) l ) . The reverse form of F−1l can be computed by subtracting ( instead of adding ) the residuals : Hl−1 = F−1l ( Hl ) ⇔ [ H ( 1 ) l−1 ; H ( 2 ) l−1 ] = F−1l ( [ H ( 1 ) l ; H ( 2 ) l ] ) , where H ( 2 ) l−1 = H ( 2 ) l − FFN ( H ( 1 ) l ) , H ( 1 ) l−1 = H ( 1 ) l − SAN ( H ( 2 ) l−1 ) . For better modeling the reordering between source and target languages , we employ relative selfattention [ Shaw et al. , 2018 ] instead of the original one [ Vaswani et al. , 2017 ] . Homogeneity : Symmetric network architecture without encoder-decoder framework . To meet our need to ensure homogeneous network computations for forward and reverse directional tasks , we therefore choose to discard the encoder-decoder paradigm . Symmetric network . To achieve homogeneous computations , one solution is to make our network symmetric , as depicted at the top of Figure 2 . Specifically , we let the 1-th to L/2-th layers be the reverse form , whereas the latter ( L/2 + 1 ) -th to L-th be the regular form : f→θ ( x ) , F−11 ◦ · · · ◦ F−1L/2 ◦ FL/2+1 ◦ · · · ◦ FL ( x ) , f←θ ( y ) , FL ◦ · · · ◦ FL/2+1 ◦ F−1L/2 ◦ · · · ◦ F−11 ( y ) , where ◦ means a layer is connected to the next layer . Thereby the forward and reverse computations of REDER become homogeneous : the forward computational operation series reads as a palindrome string 〈fs · · · fssf · · · sf〉 and so does the reverse series , where s and f denote SAN and FFN . Fully non-autoregressive modeling . Note that without encoder-decoder division , REDER works in a fully non-autoregressive fashion in both reading and generating sequences . Specifically , given an input sequence x , H0 , i = [ H ( 1 ) 0 , i ; H ( 2 ) 0 , i ] = [ e ( xi ) ; e ( xi ) ] , is the i-th element of REDER ’ s input , which is the concatenation of two copies of the embedding of xi . Once a forward computation is done , the concatenation of the output of the model [ H ( 1 ) L , i ; H ( 2 ) L , i ] serves as the representations of target translation . And then , a softmax operation is performed to measure the similarity between the model output [ H ( 1 ) L , i ; H ( 2 ) L , i ] and the concatenated embedding of ground-truth reference [ e ( yi ) , e ( yi ) ] , to obtain the prediction probability : p ( yi|x ; θ ) = softmax ( [ e ( yi ) ; e ( yi ) ] > [ H ( 1 ) L , i ; H ( 2 ) L , i ] /2 ) . We can likewise derive the procedure of f←θ for the target-to-source direction . Due to the conditional independence assumption among target tokens introduced by non-autoregressive generation , the log-likelihood of a translation becomes : log p ( y|x ; θ ) = ∑ i log pθ ( yi|x ) Modeling variable-length input and output . Encoder-decoder models can easily model variablelength input and output of most seq2seq problems . However , discarding encoder-decoder separation imposes a new challenge : the width of all the layers of the network is depending on the length of the input , thus it is very difficult to allow variable-length input and output , especially when the input is shorter than the output . We resort to the Connectionist Temporal Classification ( CTC ) [ Graves et al. , 2006 ] to solve this problem , a latent alignment approach with superior performance and the flexibility of variable length prediction . Given the conditional independence assumption , CTC is capable of efficiently finding all valid alignments a which derives from the target y by allowing consecutive repetitions and inserting blank tokens , and marginalizes log-likelihood : log pctc ( y|x ; θ ) = log ∑ a∈Γ ( y ) pθ ( a|x ) , where Γ−1 ( a ) is the collapse function that recovers the target sequence by collapsing consecutive repeated tokens , and then removing all blank tokens . Note that CTC requires that the length of source input should not be smaller than the target output , which is not the case in machine translation . To deal with this , we follow previous useful practice by upsampling the source tokens by 2 times [ Saharia et al. , 2020 , Libovický and Helcl , 2018 ] , and filter those examples when the target lengths are still larger than the one of upsampled source sentences . Remark . Reversibility in REDER can be assured in the continuous representation level , where REDER can recover from output representations ( last layer ) to input embeddings ( first layer ) , which is also the motivation and basis of the auxiliary learning signal , i.e . Lfba , in the next section . Reversibility might not hold in the discrete token level , because of the existence of irreversible operations , e.g . the argmax operation discretizes probabilities to tokens and the CTC collapse process . But REDER still shows decent reconstruction capability in practice , as visually depicted in the experiment section .

Summary: The paper proposes a novel machine translation model that can be simultaneously applied to bidirectional translations. The model, named REDER, is symmetric at both input and output ends and consists of several identical reversiable duplex Transformer layers. REDER adopts the design of revnet for reversibility and reverses the first L/2 layers for homogeneity, which is a fully non-autoregressive NMT model. Extensive experiments on WMT14 En$\leftrightarrow$De and WMT16 En$\leftrightarrow$Ro datasets show that the proposed model consistently outperforms (non-)autoregressive baselines with improving decoding speed.