Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper proposes an adversarial framework DINO to train translation models from source to target and target to source. The basic idea is to replace generator and discriminator in the energy based GAN with two source-to-target generation models. The discriminator(reverse generator) and the generator competes in a minimax game to reconstruct the data. The framework is further extended with duplicate output heads for both discriminator and generator to enhance the training robustness. 
The authors evaluated their framework on two tasks: image to image translation and silent-video to speech reconstruction. The DINO method impressive improvement in both tasks. 

Strong points:
1. The proposed DINO framework is well motivated. The objectives in DINO are reasonable and novel.
2. Experiments on both image to image translation and video to speech reconstruction verify the DINO method achieves significant improvement comparing with other translation methods. 

Weak points:
1. Important details are omitted in image-to-image translation and video to speech reconstruction. It is unclear about the backbone network as well as the parameter setup. Therefore, it is impossible to reproduce the method. 
2. DINO and DINO(bidirectional) are not consistent winners. It is not explained or analyzed why DINO sometimes wins while DINO(bidirectional) wins otherwise. There is no recommendation for practical use either. 
3. The adaptive balancing seems reasonable. But it is not studied in the experiment whether it improves the training. 
review_title: Energy based GAN with symmetric generators 

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a conditional energy-based GAN technique for translation between data domains.

First, let me preface this review by noting that this paper is far outside of my area of expertise. I have tried to do my best in reviewing it, but I'd appreciate any clarification of mistaken points from the authors.

Overall, the idea itself seems reasonable: in conditional GAN-based models, instead of using a discriminator that explicitly tries to predict whether the generated output is true or fake, the discriminator tries to maximize the reconstruction score of true outputs, and minimize it for fake outputs.

However, there were many questions I had based on just reading the paper. I'm not sure whether this is due to my lack of background knowledge in this field, or because the writing itself is unclear (perhaps a bit of both):

1. First big question: from my reading it seems that this is a *supervised* model, in that it needs $(x,y)$ pairs to calculate the objective in Equation (2). Is this correct? It doesn't seem to be explicitly stated anywhere.

2. Given this, I was not sure if the baselines in Table 1 and 2 actually represent the state-of-the-art in this domain. Pix2pix seems to be from 2017, which seems to be quite old given the huge progress this field has made in the past 3 years. BiCycleGAN, according to my understanding, is an unsupervised method, which presumably will do much worse than supervised methods.

3. Are the datasets in 4.1 standard and used in the literature? If so, what are some recent papers that evaluate on these datasets? If not, why use these datasets instead of others? While I'm not very familiar with the field, I do know that image style transfer is a big thing, and surely there are other datasets that people have evaluated on previously.

4. The description of Equation (1) was a bit hard to follow, as the role of the discriminator was not made explicit. The "margin loss" was also not explained concretely.

5. I was not able to understand the description of $\gamma$ in Equation (3), please elaborate if possible. The gain value $\lambda$ was also not easy to follow.

6. It was mentioned that MirrorGAN is the most similar method. While there was an explanation that DINO is simpler than MirrorGAN, it would be nice to explain the implications of this. Does this just mean that DINO is a bit easier to implement? Or does it mean that it is fundamentally applicable to a wider variety of tasks? Also, why is there no empirical comparison with MirrorGAN?
review_title: Seems reasonable, some clarity issues and not clear comparison is extensive (low confidence)

final_score: rating: 7: Good paper, accept
review_content: This paper presents a method for performing cross-domain or cross-modality translation models using a GAN-flavored framework where two models are trained to translate in both directions simultaneously.

[As a caveat: I am not well-versed in this area of the literature]

The paper is well-written for the most part and the experimental results are promising. My main concern are the ethical implications of some of the lip-reading experiments which go unaddressed.

**Pros**
- Clear presentation (mostly, see remarks for some exceptions)
- Good results as far as I can tell (although it is hard to interpret what all the various metrics mean, but it does seem like DINO is consistently better than the alternatives along most metrics)
- Experiments are not limited to image to image translation but also to "cross-modality translation" (image to text)

**Cons**
- If I understand correctly, the video->speech task is essentially a lip-reading task. State-of-the-art lip reading raises a number of privacy related concerns, and I think the potential impact of this research should at least be acknowledged in the paper.


**Remarks**
- The proposed method bears some conceptual similarity with recent work in unsupervised machine translation (see eg. Artetxe et al. https://arxiv.org/abs/1710.11041, Lample et al. https://arxiv.org/abs/1711.00043, also Lample et al. 2019 https://openreview.net/pdf?id=H1g2NhC5KQ which is particularly relevant as it tackles "style transfer" for text). These similarities are worth mentioning in the paper.
- I found the use of the term "discriminator" confusing, especially in the beginning in the paper. It makes sense in the usual GAN setup where the discriminator is an actual discriminative model, but it seems inappropriate in this case where the "discriminator" is a generative model.
- Eq. 6 (the main objective) is incredibly confusing. First, (and this relates to my last remark) the notation D_disc, D_gen, G_disc, G_gen unnecessarily confounding. Consider using "s -> t" and "t->s" for source to target and vice-versa instead of G and D. Also, perhaps color-coding Eq.6 would make it easier to parse (or maybe just separate the different terms a bit more).
review_title: Review

Meta-review: This paper presents a novel method for general-purpose supervised domain transfer that trains both generator and discriminator to compete in a minimax game in order to reconstruct data. This setup is meant to address a common issue in conditional GAN setups: they often ignore conditioning information. Results are positive and span two very different tasks: image-to-image translation and silent-video-to-speech reconstruction. Overall reviewers were quite positive about this paper: they found the method to be novel and well-motivated, and after rebuttal, found experimental results to be sufficiently convincing. Several concerns were brought up: (a) lack of emphasis that the approach is in fact supervised, (b) need for comparisons with stronger or task-specific baselines, (c) lack of description of experimental details for reproducibility, and (d) lack of discussion of ethical implications. All of these concerns were satisfactorily addressed by authors in rebuttal and reviewers unanimously vote for acceptance. I agree, and recommend this paper be accepted. 
Decision: Accept (Poster)