Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This method proposes a method for multi-domain image generation and unpaired image-to-image translation. The main challenge in these problems is learning a joint distribution from multiple marginal distributions is ill-posed, since there can be infinitely many joint distributions that can derive the same marginals.  Utilizing recent advances in nonlinear Independent Component Analysis (ICA) theory, the authors propose a new method to learn the joint distribution from the marginals. With the assumption that the influence of domain information is minimal in the data generation process, the method introduces the domain information through a component-wise monotonic transformation. Further, by assuming that the number of the underlying components affected by the domain information is minimal, the authors show that the true joint distribution can be recovered from the marginal distributions. Then the learned joint distribution can be used to sample meaningful tuples and translate input images to another domain without content distortion. First, the authors test the method on five multi-domain image generation datasets and evaluate using FID and domain-invariant perceptual distance (DIPD) metrics (for datasets with unpaired data.) On this dataset, the method obtains some improvement over baseline approaches, and especially shows good results on MNIST7, where other methods fail due to mode collapse. On the application of image-to-image translation, the method also obtains good results, outperforming baseline methods in both latent-based and reference-based settings. 

strength_and_weaknesses: Strengths:
1. The proposed approach shows pretty good improvement over the baseline methods, especially on the task of image-to-image translation. 

2. The ablation studies show the effect of different design choices, such as reducing the dimension of the style dimension or the effect of the number of style dimensions on generation quality. These are informative in understanding the effect of design choices on generation quality. 

Weaknesses/questions:
1. The quality of writing in the paper is uneven, with some parts not written clearly, which makes the paper difficult to understand in parts. 

2. As the authors also allude to in the last section, the proposed method is not as effective when the content across domains is not aligned, such as the case of the ArtPhoto dataset (where the method shows the least improvement in FID). Can the authors comment the general applicability of the method in the light of this limitation, beyond standard constrained facial image and digit datasets? 

clarity,_quality,_novelty_and_reproducibility: See above
summary_of_the_review: The paper proposes a method for multi-domain image generation and unpaired image-to-image translation, using an ICA-based method to learn the joint distribution from the marginals. The proposed method obtains decent improvement over baselines on standard datasets, however, performance gains are limited on complex images where the content across domains may not be aligned.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper provides an in-depth study of estimating the joint distribution of multi-domain image generation and unpaired image-to-image translation. To mitigate the highly ill-posed issues of mapping from marginal distributions to joint distribution, the authors provide an in-depth analysis for the distribution modeling, while their theory is based on some tight assumptions. Besides, the paper implements this theory in a cheap way. Specifically, the paper empirically demonstrates the disentangled content and style controllable generator through a simple framework and loss function. Then, based on this architecture and new regularization, the paper improves the performance of multi-domain image generation and unpaired image-to-image translation on multiple datasets. 
strength_and_weaknesses: #### **Strengths**
- There are many works using the StyleGAN or StarGAN architectures to perform the image generation and translation tasks but without in-depth analyses. This paper provides an in-depth study of the joint distribution for multi-domain generation and translation, which provdes some insights for the following researchers.
- The implementation way is easily achieved, and the loss function is interesting. The corresponding impressive results are good enough to support the contribution claimed by the authors.
- The paper is well organized and is easy to follow. Several observations are studied with quantitative and visual analyses, followed by the corresponding ablation study, making the paper easy to follow and concrete. 

#### **Weaknesses**
While the proposed method is interesting and the experimental analyses are comprehensive, I believe some parts need more clarification (even after considering the supplementary material and code).
- The joint distribution is the key motivation and novelty in the paper. However, the experiments just demonstrate the much better performance and the effectiveness of the provided framework. It will be better to visualize the distribution of the original dataset as well the generated results of different methods through the t-SNE, especially for MINIST7, which contains obvious domain and class category.
-  The domain $u$ is predefined. As claimed by the authors, "collecting corresponding data across domains can be prohibitively expensive". Then, how could the authors define the domain numbers in a dataset?
- More generally, for the CELEBA-HQ, the authors split it as female and male faces domains. Could we also add other domains, such as sad and happy, young and old and others? Does different domain numbers affect the performance?
clarity,_quality,_novelty_and_reproducibility: #### **Clarity**
- The paper is well written and easy to follow. 
- The key motivation and overall idea toward addressing the task is clearly explained.
- The implementation way is simple and reasonable.
- The code is well organized. 

#### **Quality**
- The model reaches better performance than the state-of-the-art methods.
- The effectiveness of the theory and model is clearly demonstrated in the experiments.

#### **Novelty**
- The cheap normalization is novel and interesting. 
- The implementation for the join distribution through a simple domain-specific component-wise monotonic transformation is reasonable. 

#### **Reproducibility**
- The code is well organized. I believe the experimental results can be reproduced using the provided code. 
summary_of_the_review: This paper investigates an important topic of unified multi-domain image generation and unpaired image-to-image translation task. Despite the experimental part could be further enriched to clearly demonstrated the better joint distribution learned by the proposed method, the easily achieved implement and thorough analyses on the properties provides many insights and might arouse following researchers. Therefore, my rating for this submission is positive.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on conditional image generation and image-to-image translation. Although current methods enable both tasks work well, they fail  to formulate suitable constraints for the joint distribution, since there can be infinitely many joint distributions that can derive the same marginals. Inspired by Independent Component Analysis (ICA) theory, authors propose a new regularization by enforcing a specific type of minimal changes across domains. The quantitative and qualitative results demonstrate the effectiveness of the proposed method.   
strength_and_weaknesses: Although there are a few outstanding methods proposed to explore conditional image generation, this paper explores a new view, which is effective to remove unwanted joint distributions. 

This looks solid which give strong theory support about how to guarantee the true joint distribution. 

The paper is well-written, and easy to follow.

Authors provides  conditional image generation and image-to-image translation results, which sounds interesting.
clarity,_quality,_novelty_and_reproducibility: I have a few concerns about this paper.

1) The motivation to use ICA is not really clear. Is it to build the true joint distribution? Also I am wondering whether the diversity are reduces with ICA regulation? Could authors show the precision/recall?, such as StyleGAN and the proposed method, and starganv2 and the proposed one?

2) As reported in Table 1, on mnist dataset the baseline (stylegan) has 95.9 FID, while the proposed method is 1.43. I am wondering how author perform the comparison experiment. 

3) The proposed method is easily influenced by hyper-parameters.

4)  In this introduction, authors mentions current methods fail to remove the unwanted joint distributions, which means this paper can achieve it. could authors give some examples? For example, the age or does not change when changing hair color. I fail to find some kinds of figures like this. 
summary_of_the_review: This paper sounds interesting, which first considers ICA into the training of both conditional image generation and image-to-image translation.  The main experiment  shows the effectiveness of the proposed method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper leverages recent theoretical developments in nonlinear ICA to introduce a novel model for multi-domain image generation and show that its joint distribution is identifiable from the marginal distributions. Motivated by this result, two practical algorithms are suggested, one for multi-domain image generation, which corresponds to sampling from the joint over domains, and one for image-to-image translation, which corresponds to sampling from a conditional. Both of these approaches are based on the assumptions that only a few latent variables are influenced by the domain label, which is implemented in the learned model via learned masks and regularization. Extensive experiments are performed, comparing multiple baselines on multiple datasets, which all point in the direction of improved matching between samples from the joint and better image-to-image generation.
strength_and_weaknesses: Strengths:
- I very much like the overall direction of this work. I believe leveraging recent identifiability results from the nonlinear ICA literature for the problem of multi-domain image generation and image-to-image translation is a very good idea, given how important the question of identifiability is to these problems. Bringing this more theory-driven mindset to this literature should benefit the community.
- The experimental section appears fairly complete, to the eye of a non-expert in multi-domain image generation.

Weaknesses:
- Overall, some theoretical statements lack clarity. One finds many undefined notation which makes some of the statements impossible to understand. Also some mathematical statements lack precision. I expand on this in the "Clarity" section later in my review.
- The connections between some sections could be improved, especially between the theoretical results and the suggested algorithms. For example, in Section 3.2, I believe there is a mismatch between equation (6) and how the ground-truth model (4) was specified: in the model (6) z_s = \epsilon + f_u (\epsilon), whereas in the ground-truth model, z_s = f_u(\epsilon). This could be fixed in the model by doing z_s = (1 - m) * \epsilon + m * f_u(\epsilon). Is there a specific reason for not doing that? 
- Section 3.3 is very hard to understand. A lot of new notation is introduced such as H, E, F and D without proper explanations. The function F is called the “shared generator” and then called the “mapping function”, this makes everything hard to follow. Also, it is not so clear to me exactly what approximates p_\theta(x^{(u_1)} | x^{(u_0)}). Overall, the connection with the theory from the previous section is not so clear to me.
- The motivation for using the masking mechanisms instead of just treating n_s as an hyperparameter is weak. An experiment attempts to show that the penalty coefficient associated to the mask is easier to tune than n_s. But I found this experiment unconvincing, since a priori, it’s unclear what the range of \lambda we should search over… The left top plot of Figure 4 shows a flat curve, but the curve would not look so flat had the range been different. How does one choose the proper range? All main experiments in the paper are performed with \lambda = 0.1 and yields very good results. What happens when these experiments are run without this masking mechanism and with n_s = 16 or 32 (which were the best dimensionality found in the ablation study)? Do we see similar performance? 
- I couldn’t find information in the main text about how the mask m is learned, nor a reference to appendix. Is m = sigmoid(parameter) ? Or is it a Gumbel-Sigmoid?



clarity,_quality,_novelty_and_reproducibility: Clarity:
- What is \mathcal{Z} ? Undefined?
- Lemma 3.1: 
    - Isn’t A1 a consequence of (4)?
    - A2 is also a consequence of (4). Indeed, if you apply an element-wise transformation to a vector of mutually independent random 
    variables (which in this case is a Normal(0,I)), the resulting vector remains mutually independent.
    - In A2, what is said in words corresponds to pairwise conditional independence while what is written mathematically corresponds to mutual conditional independence, which are not the same.
    - A3. q is not defined.
    - A4. What are B_{z_c}, B_{z^*_c} ? What is Z_s? Both are undefined, thus I can’t understand this assumptions. 
    - The conclusion of the theorem refers to “component-wise identifiability” and “block-wise identifiability”, but both terms were defined after the statement of the lemma, the author might want to consider introducing their definition before the lemma. 
- Section 3.2
    - I do not understand the following statement: “If we feed the domain label into the generator directly, the generator may over-utilize the information of domain label u in order to confuse the conditional discriminator D”. What does over-utilize mean? What does it mean to confuse the conditional discriminator?
    - If we set n_s = n and n_c = 0, is it equivalent to feeding the domain label u in f_u ?
- Section 4.1
    - Caption of figure 3: “We observe that there are unnecessary changes between the images (e.g., the added sun-glasses in the first row, the different poses of animals of StyleGAN2-ADA in second row) without regularization.” I agree that this is indeed observed in the samples provided in Figure 3, but is it an observation that generalizes across samples? 
    - The MNIST samples in Figure 3 for the baseline methods look a bit weak to me. Why is that? The baselines should give good sample quality, no? (Although we expect weaker pairing quality in the baselines)
    - I cannot really assess the relevance of the metrics used.

Novelty/originality: 
- As far as I know, this is the first work proposing an identifiability result for this setting, which I think is a significant contribution.

Minor:
- The model p(z_s | u) defined in equation (4) factorises over the components of z_s, i.e. this is the usual conditional independence assumptions of the latents made in previous ICA work, this should be mentioned explicitly when introduced.
- In first paragraph of second page: set of marginals from 1 to n, but should it be from 1 to d?
- Equation (1) also has n instead of d.
- Extra parenthesis in (4) 
summary_of_the_review: Even though I very much like the overall direction of this paper, (i) the overall lack of clarity, (ii) the lack of cohesion between the theory and the proposed methods and (iii) the lack of evidence for the usefulness of the masking mechanism prevent me from recommending acceptance. I am open to increase my  score if the authors address the main points I raised in my review (see Weaknesses section).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
This paper addresses multidomain image generation and unpaired image-to-image translation, treating them as two instances of estimating a joint distribution from marginals. Using recent theory from ICA, the paper shows certain conditions under which the joint distribution is identifiable, then implements a model that satisfies these conditions (by making some assumptions about the distributions). The results are strong on both multidomain generation and translation.

Strengths:
* One of the first results connecting unpaired image generation and translation to identifiability.
* Strong empirical results

Weaknesses:
* Clarity could be improved, especially regarding the connection between the theory and the practical method
justification_for_why_not_higher_score: After the discussion phase, all reviewers advocated acceptance, highlighting the novelty of the theory and the strong empirical results. This paper is unusual in how it connects a popular area of applicatoin -- unpaired image generation and translation -- to theoretical results on identifiability. The AC therefore feels it deserves a spotlight as it may be of broad interest to multiple communities. However, as it does not break entirely new ground on either applications or theory, the AC feels it falls short of the level of an oral.
justification_for_why_not_lower_score: See above.
Decision: Accept: notable-top-25%