Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes duplex sequence-to-sequence learning where data streams can flow from both ends of a seq2seq model. More precisely, they design a parameter efficient Transformer (REDER) that can simultaneously input and output a distinct language from each end of the model. In this way, REDER enables reversible machine translation by simply flipping the input and output ends. The experimental results show that the proposed REDER improve the multitask trained baseline by over 1.3 BLEU while maintaining over 5x speed-up at inference time.
main_review: Overall, I think the idea of duplex learning is very interesting and novel. Different from standard multi-task learning, the proposed method shows evidence that it nicely combines the information from both directions, while avoiding hurting the performance due to the capacity issue of modeling two languages. Although the basic idea of reversible layers is borrowed from previous work, it is novel to me that the same idea can be applied to Transformers.

However, I still have the following questions:

(1) When applying upsampling to the source tokens, how can reversibility still be maintained? In Line 62, the definition means that you need to return to x when applying forward and backward functions to x. However, the CTC loss allows the model to predict any sentences with the same collapsed form. To rephrase my question, I think the collapse process used in CTC is not reversible, but I am not sure about that.

(2) For the cycle consistency loss (Line 189), is it differentiable to the forward pass? Since you need to generate target sentences first before sending it back to the backward pass, and generation is non-differentiable.

(3) How do you determine the coefficients of $\lambda_{cc}$ and $\lambda_{fba}$? 

(4) The proposed REDER has a similar number of parameters compared to standard NAT models (e.g. Gu and Kong, 2020). However, the vanilla model of REDER still works worse than it does despite it utilizes both directions. What do you think may be the cause? Note that the results of Gu and Kong, 2020 in table 1 is greedy without using any reranking and beam-search, while the results of REDER does. It would be better to denote that in the table caption. 

(5) Will the relative self-attention also benefit the standard simplex NAT models?

(6) The results without KD look quite low. Although it generally makes sense, what do you think how we can improve the performance without KD within the REDER framework?

limitations_and_societal_impact: The authors claimed to have addressed the limitations. However they did not show that in a separate section or discussion, which makes the reviewer difficult to find it.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes an architecture or a reversible NMT system. In contrast to an encoder-decoder network, the two components of the network for each language are identical. When a language is being analyzed or generated depends on the direction of data flow in the network. To enable this, the authors propose reversible components in each layer of the network (inspired from revnet). The model is trained with CTC loss along with auxiliary losses for cyclic consistency and layerwise agreement. The paper is an interesting proposal for a reversible NMT system, and it brings together work on revnet, CTC loss, NAT in a simple, elegant framework to enable reversible NMT. 
main_review: While the architecture is novel and interesting, it is not clear what implication does a reversible network have for sequence to sequence learning and MT. 
- The authors mention that reversible NMT can be useful for using supervised signals in both directions to improve translation quality. However, the results do not seem to substantiate this. The gains using REDER are none to minor over an MTL baseline. Further, the use of cyclic consistency loss for the joint training and the alignment losses also do not seem to help to any meaningful extent.  
- The other benefit pointed out by the authors is the decoding speedup on account of the non-autogressive nature of the solution. The speedup is much lower compared to the other NAT approaches, while the translation quality is comparable. REDER uses similar techniques as CTC-based NAT approaches - why is it then about 3x slower than these models? Can the authors elaborate on this discrepancy?
- Like other NAT approaches, the model still relies on KD data to be competitive with auto-regressive models. This seems to be one sore spot in an otherwise elegant model. 

Overall, I think the paper puts together many ideas and proposes an interesting idea of reversible NMT and an elegant solution. The results do not such much benefit in terms of benefits from bilingual signals or decoding speedup. However, this is an interesting proposal and framework to further explore. 
limitations_and_societal_impact: Nothing to comment
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a duplex reversible network based on Transformer and revnet for neural machine translation. With the proposed approach, one can use one single model to translate between a language pair from both directions. The model uses CTC loss and allows non-autoregressive decoding for faster inference time. Experiments are well executed on standard NAT benchmark datasets with close performance to that of the standard Transformer model.

**Contributions**: the proposed model is new in machine translation from the best of my knowledge and achieves good performance compared to non-autoregressive models and multi-task learning. The proposed method saves parameters in terms of translation from both directions and improves inference speed.
main_review: **Originality**: the proposed work resembles the coupling network in the literature of flow-based models; authors adapt the revnet to Transformer models by designing a fully symmetric architecture that accepts inputs and emits outputs from both ends.

**Clarity**: the paper is generally well written but contains many typos. I'll list a few here: L47 -> interference, L50 -> language, L62 -> (), L74 -> the sentence of multitask ..., L77 -> adopts, L79 -> bonus also influent sentence; please double-check the writing.

**Weakness and Questions**:
1. Authors mentioned one advantage of the proposed method in the introduction which is it works well for linguistically distant language pairs; I did find the results for ja-en in the appendix, but I think these are pretty interesting and important results and should be moved to the main paper.

2. It seems that the training cost of the duplex transformer could higher than standard training. Could you report approximately the training time for the proposed model versus. the Transformer model? Besides, **it heavily relies on distillation data**, which makes me worry about its broader application to other MT scenarios, e.g. multilingual MT.

3. The discussed future work seems to be interesting. Adding one such experiment for zero-shot or multilingual MT would make the paper stronger. There could be plenty of interesting concerns when applying this model to multilingual MT. I'd like to see if the duplex transformer would outperform standard multilingual training.

4. The other related approach which could potentially reduce the parameters is parameter sharing and separation for different language pairs, where you could share many parameters for both language pairs and have individual parameters (like adapters) for individual language pair to alleviate the parameter interference problem.
limitations_and_societal_impact: N/A.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a novel machine translation model that can be simultaneously applied to bidirectional translations. The model, named REDER, is symmetric at both input and output ends and consists of several identical reversiable duplex Transformer layers. REDER adopts the design of revnet for reversibility and reverses the first L/2 layers for homogeneity, which is a fully non-autoregressive NMT model. Extensive experiments on WMT14 En$\leftrightarrow$De and WMT16 En$\leftrightarrow$Ro datasets show that the proposed model consistently outperforms (non-)autoregressive baselines with improving decoding speed.
main_review: Strengths

This work to clearly identify shortcomings in the current literature. Experimental results seem to be solid. The identification of REDER as a possible approach to realize reversible machine translation.

Questions

* There is a definition of duplex sequence-to-sequence neural netwrok in line 57, and the proposed REDER is also optimized by the cycle consistency. However, there is no experiments to verify the resulted REDER model is satisfied with this pre-definition. Can you provide some quantitative results on the development sets, such as the BLEU or word prediction accuracy of the reconstructed outputs, e.g. $f_{\theta}^{\leftarrow}(f_{\theta}^{\rightarrow}(x))$ or $f_{\theta}^{\rightarrow}(f_{\theta}^{\leftarrow}(y))$.
* Existing multilingual machine translation models can also realize reversible machine translation that uses language embedding to avoid parameter interference problem. Can you provide more comparisons between REDER and multilingual NMT models (only involves the source and target languages.)
* A related work is missing, i.e. supervised dual learning [Xia et al., 2017: ICML].
* Why duplex neural networks can avoids the parameter interference problem? (Statement in line 51-52) It is not very clear for me.
* Writing could have been clearer. What the superscript (1) and (2) in Figure 2 mean? Why $y^{(1)}$ is dependent on $x^{(2)}$ and $y^{(2)}$ is dependent on $x^{(1)}$.
* I would to see the paragraph of $\textit{Modeling variable-length input and output}$ to be put behind $\textit{The Architecture of REDER}$, which is more clearer for me.

Minor comments

* In Definition 1, a closing bracket is missing in the last sentence, i.e. $f_{\theta}^{\leftarrow}(f_{\theta}^{\rightarrow}(x))=x$.
* In line 74, change "multitask learning with" to "With multitask learning".

limitations_and_societal_impact: There is no explicit presentation of limitations or potential negative societal impact in this paper.
review_title: null

Meta-review: The paper explores a "duplex" architecture for machine translation, by designing a network in which either end can be used as an input or output. To achieve this, the authors apply previous ideas about reversible networks to the transformer architecture, and use a non-autoregressive loss. The reviewers found this to be a novel, interesting and elegant approach. Like most non-autoregressive approaches, the need for knowledge distillation remains a limitation, and the authors reasonably point out that removing KD is not a goal of the paper. There are also questions about whether the approach really has large practical advantages over existing work - but this topic seems to be an interesting direction that is worth further exploration, so I recommend acceptance.
Decision: Accept (Poster)