Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this work, the authors propose an approach for “truly” unsupervised image-to-image translation. Unlike the earlier image-to-image translation with image-level or set-level supervision, the proposed method introduces a contrastive learning method to simultaneously classify the image domain and transfer the domain without any supervision. Extensive experiments and ablation studies are conducted on labeled datasets (e.g. AnimalFaces and Food-101) and unlabeled datasets (e.g. FFHQ, LSUN car). 

[Paper strengths]
- The setting of the "truly” unsupervised image-to-image translation is new (although might not be practical). 
- The approach based on contrast learning to address this new problem is quite interesting and novel.
- In the experimentation, the claim about the automatic domain classification and translation is well demonstrated.

[Paper weakness]
1. Although the setting is new, it might not make sense in practice. For translation, no matter whether it is about language or image, we need to have target in mind. If no target is specified, translation does not make sense. Then, you are doing image generation, not translation. For example, for Fig. 3, if the source is a dog and the target is a dish, which is possible for the defined truly unsupervised scenario, then what will happen?
2. The main drawback of the experiments is lacking the fair comparisons to the SOTA. Although this work is an unsupervised method and the performance may have a gap to the SOTA (e.g. MUNIT, StarGAN, StarGAN v2), the authors should provide some examples to compare with the visual results or quantitative results of the SOTA. 
3. The authors only compare the proposed method TUNIT with FUNIT in Tables 1, 3 and 4. The results are quite confusing. In Table 1, in general, the proposed unsupervised one TUNIT outperforms the fully supervised one FUNIT. Shouldn't FUNIT be the upper bound? Comparing TUNIT across Tables 1, 3 and 4, it seems having more ground-truth set labels does improve the performance at all, which is not reasonable.
4. In Figure 4, the visual results of FFHQ cannot preserve the identity of the source person. Is this meaningful? Does this a common issue in all methods? As the authors did not provide a fair comparison (even the supplement only shows the results of the proposed method), it’s hard to judge. 
5. For Eq. (2), how do you choose positive and negative pairs? How to ensure the negative pairs hold different styles? I can't find the details.

----------
Update:
- Thanks the authors for the detailed reply. It addresses some of my concerns. Overall, I am fine if this paper gets accepted, given its novelty. My major concern is still with the experiment comparisons. The authors only compare the proposed method with FUNIT, which might not be fair since FUNIT is designed for few-shot translation, dealing with novel domains or classes with a few examples. The authors should compare with MUNIT or StarGANv2, with domain information obtained by clustering or any unsupervised methods.
- The authors should tune down the eye-catching statement of truly unsupervised translation since translating between dish and dog might not make sense, although this is a minor issue.


review_title: The truly unsupervised image-to-image translation is interesting, and the idea to address it seems reasonable. The paper is generally well written. However, there are still some major issues in the motivation and the experiments. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes an unsupervised I2I translation method TUNIT where there is no any supervision signal. Built upon the existing FUNIT work which disentangle style and content, the proposed method additionally adds a pseudo label prediction branch to separate domains based on maximizing the mutual information. Experimental results look pretty solid. Some of my concerns are listed below:

(1) Without any supervision, the selection of K sounds very important. I'm interested in a more crazy value for K in Table 2. If a collection of images are given and the estimation of K is unlikely to happen, how should K be selected?

(2) The experiment is based on randomly selecting 10 classes from the dataset. Not sure how much difference or similarity exist between those classes. It is better to show the average performance of several 10-class sets.

(3) I like the baseline comparison with K-means but it sounds like a very old clustering method. Will more advanced clustering method bring further improvement on performance?

(4) I suggest adding another line in Table 1 which is about TUNIT + using domain labels (with a classifier to predict real labels). This could serve as a bar on performance to let readers be clear about the gap between w/ and w/o labels. 

(5) I feel this work mostly focuses on unpaired data case. To my understanding, the claim "better than supervised models" should be about "better than set-level supervised models". If paired data is given, the proposed method might not beat its performance.

=======

Update:

Thanks for the feedback from authors. Mostly my concerns are addressed. I suggest adding the discussion on the selection of K in the draft as this is important for readers to know when facing just a collection of images.

review_title: Fully unsupervised I2I with extensive experimental results, some concerns about experiments

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper introduced a more challenging setup under the problem of unpaired image-to-image translation, where no domain labels or image sets are provided. Towards solving this problem, the paper proposed a TUNIT framework with a guiding network, which could encode style codes for the generator and predict the domain labels for the discriminator. The paper is well-written with nice figures. Abundant experiments are conducted and lots of generation results further verify its effectiveness. But there still exists some weaknesses:

1. A contradiction exists that the author claimed the number of domains "K" is an unknown property of the dataset (in the 2nd line of Sec 2), but using the guiding network to predict the domain labels under the assumption that "K" is known (Eq. (1)).

2. The description of "Style contrastive loss" (Eq. (4)) is confusing. The author claimed that the network may ignore the given style code $\tilde{s}$ and synthesize a random image from the same domain $\tilde{y}$ without Eq. (4), however, as we learned from Eq. (2), the "given style code $\tilde{s}$" and the "style codes of images from the same domain $\tilde{y}$" are similar to each other. A contradiction exists. In my point of view, Eq. (4) is used to make sure that the generated images show the same visual style features as the input reference images. More explanations are expected here. 

3. Since Eq. (2) and Eq. (4) have similar equations and maybe there are also some overlapping functions. I expect more discussions regarding the relations and differences between these two losses. Furthermore, how does the network perform if removing one of them?

4. The usage of contrastive loss and image queue is borrowed from MoCo [A], so it's better to acknowledge it and mention the relations.

[A] Momentum contrast for unsupervised visual representation learning. CVPR 2020.


Given the above weaknesses, I gave a rating of 5 in the initial comments and I will increase my score if the author could 1) provide better explanations of the contradictions, 2) discuss and add ablation studies on the functions of Eq. (2) and Eq. (4).



===========================================================================================================

Update:

Thanks to the author's response, solving my concerns on this work. I decide to increase my score to 6. Besides, I suggest supplementing the mentioned discussions and empirical comparisons between Eq. (2) and Eq. (4) in the manuscript, as least in the appendix.
review_title: Official Blind Review #1

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper tackles the problem of unsupervised image to image translation without making any assumptions about the existence of input pairs or input sets. To solve this task, the authors utilize a guiding network that generates a pseudo label (domain label in this case) as well as a style code which are then provided to a generator network. The generator given a source image produces an output that follows the structure of the source yet preserves the style and "domain" details of the reference image. These  networks are trained jointly with adversarial, contrastive and reconstruction losses. Results are reported on labeled and unlabeled datasets 

Pros: 
- What's quite neat is the incorporation of SSL to handle the lack of labels where augmentation is applied to the reference image with the goal of maximizing the mutual information between the output distribution of the input and the augmented image. 
- Very well documented results, Table 1 and Figure 3 are well designed to show how each of the components of this method (or alternatives that the authors could have used) affect the unsupervised translation results. 
- Results-wise what is provided is actually quite compelling, it's easy to understand the impact of style from the source and the "structure" of the reference images on the generated results at least for the animals and the cars. 

Cons:

- I am somewhat skeptical of the statement "our model is the first to succeed in this task in an end-to-end manner" as a selling point of this paper. For example, a prior work that comes to mind is CUT [1] where the authors also target the unsupervised image-to-image translation and also utilize contrastive losses in patches of images. One could argue that in CUT they still require a set of domains whereas this paper does not but one could first do some clustering and then apply CUT which is quite similar to what this paper is proposing. I think it would be fair to ask for a discussion/comparison with that paper.
- While the idea of estimating a pseudo-label from the encoder in a SSL fashion is well-designed the rest of the architecture is similar to other recent works that perform "exemplar-based image synthesis". In [2] for example the authors feed a reference image that serves as an exemplar from which they extract class-specific styles which are then fed to the generator that synthesizes the output image. This procedure is similar to what's happening with the style code that is fed to the generator in this paper. 
- Is joint training better compared to sequential for the guiding and generator networks? The results from Table 1 indicate that if one looks at mFID or Acc this could be dataset-specific. I would encourage the authors to potentially elaborate a little more on that since what's currently stated is that "we confirm that the joint training of style encoder and clustering improves overall performances"

Minor Comment:
- From what I understand from Figure 2 the estimated domain label from the encoder is fed to the discriminator but there is then a "detachment" so as we do not backprop through the encoder. I would encourage the authors to potentially expand this a little in the paper as it seems to be that besides noting it in that Figure there's no further description of this in the paper.

Overall:
This is an interesting paper, very well written, and easy to follow targeting a hard and under-researched problem. The guiding network is nicely designed and the results seem quite  strong in terms of texture transfer while maintaining the source shape although fidelity-wise the still have notable artifacts (easier to observe in faces). My main concerns are more related to how some statements related to novelty or training findings (joint vs sequential) are presented and how much new information does the reader get from this paper. 

References
[1] Contrastive Learning for Unpaired Image-to-Image Translation ECCV 2020
[2] SEAN: Image Synthesis with Semantic Region-Adaptive Normalization CVPR 2020

Update: 
I'd like to thank the authors for their detailed response to my comments. 

- I respect the argument that CUT requires a model per pair of domains hence it does not scale to 10 domains. However  presenting the problem in such an aspect ("we evaluate only on multi-domain datasets such as the AnimalFaces, Food") limits a little the prior work that one can evaluate against. In my humble opinion the authors constrain the definition of unsupervised image-to-image  translation such that their proposed approach "is the first to succeed in this task" whereas I also think CUT is an unsupervised image-to-image translation method that "succeeds in this task". If the  argument was presented in the multi-domain setup I would be willing to buy it but as it is presented currently I find the first contribution as an overstatement. 

- As for SEAN my point was not to compare against it but that the architectural design of this work is not far from what prior work is already doing. 

- Sounds good thanks for the explanation. 

Having said that I still think this is a solid submission with interesting findings in an under-researched problem hence I do recommend acceptance. 
review_title: Official review

Meta-review: This paper defines a truly unsupervised image translation scenario. Namely, there are no parallel images or domain labels. To achieve robust performance in this scenario, the authors use 1) clustering and 2) generator-discriminator structure to map images from different domains and generate images for target domains.  

In all, all the reviewers agree that this definition of unsupervised image translation is interesting. However, there are also several concerns for the real-world practical application and empirical results.  Unlike unsupervised text translation whose target language is known, the truly unsupervised image translation is difficult to make sense without identifying what is the target domain. This limits the contribution of this paper to some specific tasks instead of more general tasks. For the empirical results, the selection of data and the hyperparameter K do not convince the reviewers.


Decision: Reject