Title: Unsupervised Cross-Domain Image Generation
Authors: Yaniv Taigman
Adam Polyak
Lior Wolf
Venue: ICLR 2017
Keywords: Computer vision
Deep learning
Unsupervised Learning
Transfer Learning

Abstract: We study the problem of transferring a sample in one domain to an analog sample in another domain. Given two related domains, S and T, we would like to learn a generative function G that maps an input sample from S to the domain T, such that the output of a given representation function f, which accepts inputs in either domains, would remain unchanged. Other than f, the training data is unsupervised and consist of a set of samples from each domain, without any mapping between them. The Domain Transfer Network (DTN) we present employs a compound loss function that includes a multiclass GAN loss, an f preserving component, and a regularizing component that encourages G to map samples from T to themselves. We apply our method to visual domains including digits and face images and demonstrate its ability to generate convincing novel images of previously unseen entities, while preserving their identity.


Review discussion:
1. From the introduction and abstract it's not clear what do the authors mean by f. Why would anyone care about f and f-constancy? It would be nice if that could be explained in the very beginning.
2. In the emoji experiment, did you apply any tricks to make GAN training stable? What was the stopping criterion for the training procedure? Is superresolution applied during training too (to upscale model outputs to 152x152 and match the resolution of the real data)?
Thank you for your questions!
1. f-constancy is the mechanism that ensures that the analogy is valid, i.e., that the mapping of a sample from S to T maintains a similarity as defined by f.
For example, the DeepFace representation ensures that the created emoji maintains the identity information of the original image. We will make sure to clarify this point in the abstract and introduction.
2.  (i) As mentioned in the manuscript, our GAN follows DCGAN (Radford et al. 2015). No special tricks were applied.
(ii) Stopping was performed after 3 epochs.
(iii) As mentioned in Sec. 5.2, the upscaling to 152x152 was done in order to employ a pretrained $f$. We failed to mention the exact upscaling method — it was done using a simple bilinear upscaling. The superresolution method was not applied during training and is only employed to create print quality images. The appendix shows the results without it.
1. Could you please explain equation (3)? I assume R_GAN in equation (1) translate to L_GANG in equation (4), what is the motivation behind (3)?
2. Why do you want to enforce identify matrix on G as in equation (6)?
Thank you for your questions.
1.  Equation 1 is the GAN risk for a binary GAN that compares “fake” (i.e, generated) samples crated by mapping samples in S to “real” samples from T. It evolves to both Equation 3 and Equation 4. The differences between 1 and 3+4 are that we moved from the language of risk, i.e., using expectations, to the language of loss, using sums (the writing as expectations and not as sums is a typo).
Second, we moved from a binary D to a ternary D and distinguish between three populations, labeled respectively as classes 1,2,3: (1) G mapping of samples from s, (2) G mapping of samples from t, (3) the actual samples from t.
Equations 3 and 4 can no longer be specified as one equation, since we want to be specific that G maps the first two classes to class 3. If G were to simply maximize 3, there would be other options, e.g., confusing classes 1 and 2.
2.  Equations 3 and 4 make sure that the generated analogy, i.e., the output of G,  is in the target space T.
However, they do not enforce any similarity between the source sample x and the generated G(x). This is done by Equations 5 and 6. Equation 5 enforces f-constancy for x \in S. Equation 6 enforces that for samples x\in T, which are already in the target space, G is the identity mapping. This is the desirable behavior, e.g., given an emoji, one would like it to remain the same emoji under the mapping of G. It can also be seen as an autoencoder type of  loss, applied only to samples from T. This example and motivation will be added to the revised manuscript.
While Equation 5 is sufficient (see Tab. 1, DTN w/o L_{TID}), Equation 6 helps obtain better performance. Interestingly, Equation 6, even without Equation 5, is a strong enough constraint to ensure a meaningful mapping (see Tab 1, DTN w/o L_{CONST}). Removing both Equation 5 and Equation 6, (Tab1, DTN w/o L_{CONST} & L_{TID}) leads to a mapping that does not preserve identity, as expected. These observations appear in Sec. 5.1, and we will make sure to clarify the role of Equation 6 further.
Following the ongoing discussion with the reviewing team as well as with other readers, we have just uploaded a new version of our manuscript, which is aimed at improving clarity.
In order to provide a quick way to track the changes, we color all modifications in red.
In addition, as mentioned below, we will soon share our open implementation in Torch.
Thank you all for the extremely useful feedback.
Update: After reading the rebuttal comments and the revised paper, I'm leaving the rating as it was before.
This paper proposes an unsupervised algorithm for transferring samples from one domain to another (related) domain under the constraint that some predefined f returns same result for the input and the result.
Pros:
1. The paper presents an interesting idea of comparing samples from different domains using a fixed perceptual function f.
2. The proposed method produces visually appealing results on several datasets
3. The authors demonstrate how their approach can be used for domain adaptation and obtain improved results on the SVHN->MNIST task
4. The paper is well-written and easy to read
Cons:
1. The novelty of the method is relatively minor (I consider f-constancy term as the main contribution)
2. It feels like the proposed approach would break for more dissimilar domains. The method relies on a fixed f which is trained on the source domain. This f can potentially drop information important for obtaining 1) better reconstructions in the target domain  2) more tightly related x and g(f(x)). I think the authors should consider either training all the modules in the model end-to-end or incorporating target samples into the training of f.
3. A single domain adaptation experiment is definitely not enough to consider the proposed method as a universal alternative to the existing DA approaches.
I would also like to point out that using super-resolved outputs as opposed to the actual model’s outputs can produce a false impression of the visual quality of the transferred samples. I’d suggest moving original outputs from the appendix into the main part.
Thank you for your positive and supporting review.
Regarding the few observed disadvantages:
[AnonReviewer2:] 1. The novelty of the method is relatively minor (I consider f-constancy term as the main contribution)
[Authors:] There are a few very recent and concurrent publications that do images analogies. All require paired training samples. Our work is unique in its unsupervised nature, which we think is noteworthy. f-constancy is the means of obtaining this.
[AnonReviewer2:] 2. It feels like the proposed approach would break for more dissimilar domains. The method relies on a fixed f which is trained on the source domain. This f can potentially drop information important for obtaining 1) better reconstructions in the target domain 2) more tightly related x and g(f(x)). I think the authors should consider either training all the modules in the model end-to-end or incorporating target samples into the training of f.
[Authors:]  In our experiments, f is demonstratively more powerful in S than in T (Tab. 1 last row).
Still we observe what we call “the magic of analogies” — an f that is rather weak on T as a classifier is good enough in order to create analogies using nothing more than unlabeled examples from both domains. Based on these analogies we can transfer labels from S and create a good classifier in T (Tab. 2).
As the experiment in the second revision show, the f trained on photos can maintain not only identity information in T but also expression information. This further supports that even when f is trained separately on S, f-constancy assures that x and g(f(x)) are tightly related along many dimensions, including dimensions that f is supposedly invariant to.
[AnonReviewer2:] 3. A single domain adaptation experiment is definitely not enough to consider the proposed method as a universal alternative to the existing DA approaches.
[Authors:] We agree. The purpose of the DA experiment is to show that the reduction of the DA problem to the domain transfer problem (Sec. 2) is viable.
While the experiment is very successful, it is indeed only a single experiment and is not meant to shift the focus on the paper from the new cross domain transfer problem to the well explored problem of DA.
Update: thank you for running more experiments, and add more explanations in the manuscript. They addressed most of my concerns, so I updated the score accordingly.
The work aims at learning a generative function G that can maps input from source domain to the target domain, such that a given representation function f remain unchanged accepting inputs from either domain. The criteria is termed f-constancy. The proposed method is evaluated on two visual domain adaptation tasks.
The paper is relatively easy to follow, and the authors provided quite extensive experimental results on the two datasets. f-constancy is the main novelty of the work.
It seems counter-intuitive to force the function G to be of g o f, i.e., starting from a restricted function f which might have already lost information. As in the face dataset, f is learned to optimize the performance of certain task on some external dataset. It is not clear if an input from the source or target domain can be recovered from applying G as in equation (5) and (6).
Also, the f function is learned with a particular task in mind. As in the two experiments, the representation function f is learned to identify the digits in the source SVHN dataset or the identity of some face dataset.
As a result, the procedure has to be repeated if we were to perform domain adaptation for the same domains but for different tasks, such as recognizing expressions instead of identity.
Do the authors have insight on why the baseline method proposed in equation (1) and (2) perform so poorly?
Figure 5 shows some visual comparison between style transfer and the proposed method. It is not clear though which method is better. Will it be possible to apply style transfer to generate emojis from photos and repeat the experiments shown in table 4?
Thank you for your review.
**1.  AnonReviewer1: f-constancy is the main novelty of the work. **
Authors: Thank you for noting the novelty. A few prominent recent contributions present image to image mapping using supervised (input image,output image) pairs. We are unique in that we do not require such pairs. This is quite remarkable and is achieved using f-constancy among other contributions.
**2.  AnonReviewer1: It seems counter-intuitive to force the function G to be of g o f, i.e., starting from a restricted function f which might have already lost information. **
Authors:  Since L_CONST requires the preservation of identity, the information in f is exactly the relevant information. Experimentally, we show a very strong advantage over the baseline method "DTN G does not contain f" in Table 1.
**3.  AnonReviewer1: As in the face dataset, f is learned to optimize the performance of certain task on some external dataset. It is not clear if an input from the source or target domain can be recovered from applying G as in equation (5) and (6). **
Authors: We respectfully disagree. The fact that the generated caricatures are remarkably identifiable (median rank of 16 out of 100,000 for retrieving among real images) clearly shows that identity information is extremely well preserved.
**4.  AnonReviewer1: Also, the f function is learned with a particular task in mind. As in the two experiments, the representation function f is learned to identify the digits in the source SVHN dataset or the identity of some face dataset.
As a result, the procedure has to be repeated if we were to perform domain adaptation for the same domains but for different tasks, such as recognizing expressions instead of identity. **
Authors: Yes, but this is desirable.
For example, in the face experiments we wish to maintain identity and not other factors such as expression, illumination, head pose, etc.
**5.  AnonReviewer1: Do the authors have insight on why the baseline method proposed in equation (1) and (2) perform so poorly? **
Authors: As stated above (2), incorporating f in G allows the network to focus on the most relevant aspects of this mapping. In the unsupervised setting this seems crucial.
**6.  AnonReviewer1: Figure 5 shows some visual comparison between style transfer and the proposed method. It is not clear though which method is better. **
Authors: Figure 5 aims to clarify the major difference between the tasks of style-transfer and domain transfer:  "the output image [in 5(c)] is perhaps visually appealing;
However, it does not belong to the space t of emoji".
Therefore, style transfer does not perform cross-domain generation and solves a different problem.
**7.  AnonReviewer1: Will it be possible to apply style transfer to generate emojis from photos and repeat the experiments shown in table 4?**
Authors: Style transfer is not relevant to Table 4 since, as stated above, it does not create an emoji image. It is therefore not comparable to the other methods in the table.
Since AnonReviewer1 asked for additional explanations, we respectfully ask that the reviewer consider updating the review based on our clarifications or let us know if these are not satisfactory.
Dear reviewer,
In addition to information provided below ("More explanations") we would like to note that we just performed the expression transfer experiment you mentioned in your review. As it turns out, the same f can be used for both identity and expression!
We believe that the additional information we provide at openreview.net as well as the manuscript revision could justify a second view and would be very much interested in an open discussion in order to find out if there are any remaining unfavorable factors.
Thank you,
The authors
Dear reviewer,
Thank you for updating the review!
The authors.
This paper presents an unsupervised image transformation method that maps a sample from source domain to target domain. The major contribution lies in that it does not require aligned training pairs from two domains. The model is based on GANs. To make it work in the unsupervised setting, this paper decomposes the generation function into two modules: an encoder that identify a common feature space between two domains and an decoder that generates samples in the target domain. To avoid trivial solutions, this paper proposed two additional losses that penalize 1) the feature difference between a source sample and its transformed sample and 2) the pixel difference between a target sample and its re-generated sample. This paper presents extensive experiments on transferring SVHN digit images to MNIST style and transferring face images to emoji style.
+The proposed learning method enables unsupervised domain transfer that could be impactful in broad problem contexts.
+This paper presents careful ablation studies to analyze the effects of different components of the system, which is helpful for understanding the paper.
+The transferred images are visually impressive and quantitative results also show the image identities are preserved across domains to some degree.
-It will be more interesting to show results in other domains such as texts and images.
-In addition to the face identities, it is also of great interest to analyze how well the facial attributes are preserved when mapping to target domain.
We thank the reviewer for the comments.
Indeed, the main invention is learning to map without the supervision of matching pairs.
The reviewer pointed to two shortcomings of the experiments.
Reviewer: -It will be more interesting to show results in other domains such as texts and images.
Authors: Indeed, the options are endless. Analogies are a very powerful tool, and making these in an unsupervised manner has a lot of applications that are not explored in this manuscript.
However, we are afraid that this is beyond the scope of the current submission.
Reviewer: -In addition to the face identities, it is also of great interest to analyze how well the facial attributes are preserved when mapping to target domain.
Authors: Following the request of both AnonReviewer1 and AnonReviewer3, we added new experiments and revised our manuscript. As it turns out the face identification network does contain enough expression information to allow cross-domain transfer of expression. All that was needed is to provide unlabeled smiling emoji so that the discriminator would not identify smiling emoji as fake. Please see our subsequent comment regarding the revised version.
Following the request of the reviewing team, we have just uploaded a new version of our manuscript, which includes expression preserving experiments.
The new experiments are in Appendix B.
In order to provide a quick way to track the changes from the original submission, we color all modifications in red.
Thank you for the extremely useful feedback.
We thank the reviewers for their time and insights. All 3 reviewers seem to agree that the work is interesting, well-written and presents extensive experiments. R1 & R2 both note the f-constancy as a novelty of this work.
Also, the fact that the method does not require training pairs for the two domains is noted by R3 as a major contribution, which “could be impactful in broad problem context”. R2 & R3 both agree that the output generations are visually appealing. We have no factual dispute with the reviewers and replied to each individually below.
The open review discussion has been extremely beneficial to us and the paper has been revised in order to address all actionable items raised throughout the review period. We thank the reviewers for their thoughtful and constructive reviews and all other community members who shared their comments. During the review period, the work has already been cited several times, reimplemented on github, and drawn considerable attention.

Meta-review:
role: PC
data: comment: The authors propose a application of GANs to map images to new domains with no labels. E.g., an MNIST 3 is used to generate a SVHN 3. Ablation analysis is given to help understand the model. The results are (subjectively) impressive and the approach could be used for cross-domain transfer, an important problem. All in all, a strong paper.

Decision: Accept (Poster)