Multi-domain image generation and translation with identifiability guarantees
Authors: Shaoan Xie
Lingjing Kong
Mingming Gong
Kun Zhang
Venue: ICLR 2023 notable top 25% (2023)
Keywords: multi-domain image generation
image translation
identifiability
Nonlinear ICA

Abstract: Multi-domain image generation and unpaired image-to-to-image translation are two important and related computer vision problems. The common technique for the two tasks is the learning of a joint distribution from multiple marginal distributions. However, it is well known that there can be infinitely many joint distributions that can derive the same marginals. Hence, it is necessary to formulate suitable constraints to address this highly ill-posed problem. Inspired by the recent advances in nonlinear Independent Component Analysis (ICA) theory, we propose a new method to learn the joint distribution from the marginals by enforcing a specific type of minimal change across domains. We report one of the first results connecting multi-domain generative models to identifiability and shows why identifiability is essential and how to achieve it theoretically and practically. We apply our method to five multi-domain image generation and six image-to-image translation tasks. The superior performance of our model supports our theory and demonstrates the effectiveness of our method. The training code are available at https://github.com/Mid-Push/i-stylegan.

Published as a conference paper at ICLR 2023
MULTI-DOMAIN IMAGE GENERATION AND TRANSLA- TION WITH IDENTIFIABILITY GUARANTEES
Shaoan Xie', Lingjing Kong', Mingming Gong 22, and Kun Zhang"
Carnegie Mellon University
Mohamed bin Zayed University of Artificial Intelligence
'The University of Melbourne
shaoan cmu edu, lingjingkonger cmu edu,, mingming, gong@ un 1b. edu kunz cmu edu
ABSTRACT
Multi-domain image generation and unpaired image-to-to-image translation are important and related computer vision problems. The common technique for the two tasks is the learning of joint distribution from multiple marginal distributions. However, it is well known that there can be infinitely many joint distributions that can derive the same marginals. Hence, it is necessary to formulate suitable constraints address this highly ill-posed problem. Inspired by the recent advances in nonlinear Independent Component Analysis (ICA) theory, we propose a new method to learn the joint distribution from the marginals by enforcing specific type of minimal change across domains. We report one of the first results connecting multi-domain generative models to identifiability and shows why identifiability is essential and how to achieve it theoretically and practically. We apply our method to five multi-domain image generation and six image-to- image translation tasks. The superior performance of model supports our theory and demonstrates the effectiveness of our method. The training code are available at attttps://gtthub com cwm/Miccc...ccocccooo Push/i-saaiie sttlissstyynnn.....
INTRODUCTION
Multi-domain image generation and unpaired image-to-iman translation are two important and closely related problems computer vision and machine learning. They have many promising applications such domain adaptation (Liu & Tuzel 2016 Hoffman et al. 2018 Murez et al. 2018 Wang & Jiang 2019) and medical analysis (Armanious et al. (019) 2020 Kong et al. 202 1). As shown in Fig. multi-domain image generation takes input the random noise and domain label u and the task aims to generate image tuples where the images in the tuple share the same content, e.g., different facial expressions of the same people. The second task takes input image in domain and target domain label u and aims to generate another image which is in the target domain but share the same content of input, e.g., the output image has the same identity but different facial expression from the input image.
Both tasks can be viewed instantiations of joint distribution learning problem. A joint distribution of multi-domain images probability density function that gives a density value to each joint occurrence of images in different domains such as images of the same people with different facial expressions. Once the joint distribution is learned, it can be used to generate meaningful tuples (the first task) and translate an input image into another domain without content distortion (the second task). If the correspondence across domains is given (e.g., the identity), apply supervised approaches to learn the joint distribution easily. However, collecting corresponding data across domains be prohibitively expensive. For example,
G
F
Generation
Trauslation
Figure 1: Two tasks.

---

Published as a conference paper at ICLR 2023
collecting different facial expressions of same person may need controlled experiments.
By contrast, collecting image domains without correspondence can be relatively cheap, e.g., facial expressions of different people can be easily accessed online (once permission is granted). Therefore, we consider the problem of unsupervised joint distribution learning P(x(1, x(2) ...., x()) where we are only given multiple marginal distributions in different yet related domains (P(x())) where d is the number of domains and x denotes the images in domain i. However, there can be infinitely many joint distributions that can produce the same marginals (Lindvall 2002). For example, we can apply conditional GANs (Mizza & Osindero 2014; Odena et al. 201 Miyato & Koyama 2018 Brock et al. (0188) match the marginal distributions each domain and they learn joint distributions implicitly. We can sample a tuple (zx(0) ...., x ( from its learned joint distribution using the same random noise and different domain labels. However, there is no guarantee that the learned joint distribution in conditional GANs are optimal and it may drop the correspondence in the generated tuples (e.g.. (b)). To tackle this problem, CoGAN (Liu & Tuzel (016) proposes to use different generators for each domain and share weights of high level layers in different generators. JointGAN (Pu et al. 2018 proposes to factorize the joint distribution (e.g.. P(x(1), x(²) ((x())) x x()))
and learn one marginal and one conditional distribution with cycle consistency (Zhu et al. 2017).
While existing approaches (e.g.. Liu et al. () Pu et al. (2018) add constraints for the purpose of removing unwanted joint distributions, they are not guaranteed to find the true joint distribution. If we are unsure about the learned joint distribution, we cannot guarantee that the generated tuples are meaningful. Learning the true joint distribution seems to be impossible since we only have access marginal distributions and domain label and we do not have access to the latent content and style variables. Fortunately, recent advances in nonlinear Independent Component Analysis (ICA) theory (Hyvarinen & Morioka 2016 2017 Hyvarinen et al., 2019: Khemakhem et al. 2020b bj Von Kügelgen et al. [021] Kong et al. 2022) show that deep nonlinear latent variable models are identihable with auxiliary variables (e.g., domain label in our case), meaning that we can recover the latent variables (e.g., content and style in our case) up to some component-wiss transformation. Inspired by these advances, we propose a new method to learn the true joint distribution from the marginals by enforcing a specific type of minimal changes across domains. Specifically, we assume that the influence of domain information (i.e., the underlying changes across domains) is minimal in the data generation process, e.g., only facial expression changes and no hair style changes is allowed. To achieve the minimal changes, we inject the domain information through a component-wi wise strictly increasing transformation instead arbitrary complex transformation. In addition, we assume the number of the underlying components affected by the domain information (and thus with changing distributions) minimal. Then we show that if the influence of domain information is minimal, the true joint distribution can be recovered from the marginal distributions. Afterwards, we can use the learned joint distribution to sample meaningful tuples and translate input images into another domain without content distortion. Our method can be applied to datasets where the content across domains are aligned and most existing datasets in multi-domain image generation and translation satisfy this requirement, e.g., the facial image dataset, animal face and digit images. Our method may not be as effective when the contents when some domains contain unaligned contents. We may need to pay more attention to the artworks-related dataset, since different painters focus on different objects in their painting (c.g., there are many fruit paintings by Cezanne and landscape paintings by Monet).
Our proposed method has several theoretical and practical contributions:
1. We provide the properties of an image generation process under which the true joint distri- bution can be recovered from marginal distributions.
2. In light of our theoretical results, we provide a practical method for multi-domain image generation. The proposed method can automatically determine of the latent variables with changing distributions. Our method achieves promising results across five image generation tasks.
the underlying dimension
3. We propose to encourage the mapping function in image translation task to preserve the cor- respondence learned by our image generation model. The significant gain over the baseline methods on six image translation tasks demonstrate the effectiveness of our technique.
2

---

Published as a conference paper at ICLR 2023
2 RELATED WORK
Multi-domain image generation and translation Generative Adversarial Network (GAN) (Goo fellow et al., 2014) performs adversarial training between the generator and the discriminator. At the end, the distribution of sample generated by the generator is matched to the real data distribu- tion. Conditional GAN (Mirza Osindero 2014) a variant of GAN that incorporates additional information and have been widely applied for class-conditional image generation (Brock et al. 2018). CoGAN (Liu & Tuzel 2016) learns the joint distribution by sharing the higher layers of two generators. JointGAN (Pu et al. 2018) proposes to factorize the joint distribution and learn marginal and conditionals and regularize the conditionals with cycle consistency. RegCGAN (Mao
& Li 2018) penalizes the distance between the features of the synthesized pairs. Methods by Liu & Tuzel (2016); Pu et al. (2118); Mao & Li| (2018) are shown to be effective on some tasks but they do not have identifiability guarantees that they are recovering the true joint distribution (we provide a formal definition of identifiability later). Unpaired image-to-image translation relies on additional assumptions to address the task and cycle consistency (Zhu et al.) 2017) is arguably most widely used. However, it has been argued that cycle consistency may not be enough to learn good mapping (Alami Mejjati et al. 2018 Xie et al. 2022). Therefore, we propose a new regularization with generated tuples to help find a good mapping. We provide more related works in appendix C
Nonlinear ICA Nonlinear ICA aims to recover independent latents from the data that are generated by nonlinear invertible transformations from the underlying independent variables (Hyvarinen &
Morioka
2016, 2017 Hyvarinen et al. 2019). Recent works have shown that the true latent variables
may be identifialle given additional information (Khemakhem et al. [2020a] Gresele et al. 2020
Locatello et al. 2020| (Shu et al.] [2019] Zimmermann et al. 2021 Hälvä vä & Hyvarinen| 2020 Klindt
al. 2) Khemakhem et al. (2020a) prove that the latent variable is identifiable if the prior
distribution is conditionally factorized. (oon Kügelgen et al.] (202T) show that the latent content
variable is block-identiffable given two views of same image, which is not applicable in our case since we do not have paired data. In the next section, we will show how our formulated problem is related some variant of nonlinear ICA and how we can establish the identifiability of the changes across domains.
UNSUPERVISED JOINT DISTRIBUTION LEARNING
In this section, we first provide the formulations of our conditional generative model and some additional conditioins, under which the true joint distribution is identifiable. Then we provide a practical implementation based on conditional GAN to achieve unsupervised multi-domain image generation. Finally, we propose a novel regularization technique to improve unpaired image-to-image translation.
3.1 IDENTIFIABILITY OF THE JOINT DISTRIBUTION
Given d marginal distributions (Px = derived from the true but unknown joint distribution Po (o())),.... x (d)) where A E θ is a vector of parameters, we need to recover the true joint distribution to generate meaningful image tuples or translating images without content distribution. To recover the true joint distribution, it needs to be identifiable. Formally, the true joint distribution is identifiable if following hold:
vO ()) Po Px + ()(
(1)) = Por
((d)).
(I)
That is, if another model with parameter 6' E θ matches the marginal distributions in each domain, then this would imply that the true joint distribution also matched perfectly by the model with parameter 0'. This is well-known ill-posed problem: the joint distribution is not identifiable from marginal distributions without further assumptions. Motivated by the multiple domain image generation and translation problems, we define the following data generation process using latent variable model (we provide the graphical model in appendix D):
d Po /
(x
(1))
Ze, (1) (z.)) II Po )dz,de)) ... ...dz!").
(2)
where z, is the common content and 22 is the style in the i-th domain. For instance, in expression generation, z, represents the human identity information, and z, represents the expression. Then
3

---

Published as a conference paper at ICLR 2023
data point sampled from the joint distribution of our model would be person with different expressions. In practice, we may not have sample from the joint distribution; instead, we only have samples from the marginal distributions of Po 7x(x1)),... x(d)): (...)
Po(x!")- = f Po(x z, Po ( Po (z. | de,
(3)
where u is the domain label and Pe (z,|u = 1) = Po! To recover the true joint distribution, we first recover the true content ze and true style Zg up some transformation and then we establish the identifiability of the true joint distribtion.
Recovering 2. and z, seems to be impossible since we only have the observations {x(") and domain label u. Fortunately, recent advances on nonlinear ICA theory have shown that deep non-linear latent variable models are identifiable given auxiliary variable (e.g.. domain label in our case) (Khemakhem)
ct al. 2020b
Von Kügelgen et al. 2021 Kong et al. 2022). Specifically, we define
Pz(z.lu) P(Z.)
2, fu(2.), ~ P P(žs).
(4)
where fu is domain-specifer component-wise strictly increasing transformation (i.e., in each domain, each dimension of 2, is strietly increasing function of the corresponding dimension of Z,) and J the absolute value of determinant of the Jacobian matrix of function fu. P(ž.) is a prior distribution and we define it as (o, I). Then we also define:
Po(ze) = P(ze);
z.) x = 8(x g(Ze. 2z)):
(5)
where is the true generation function, 8(.) J. is the Dirac delta function and P(ze) is a prior distribution and we also define it as NV I) our model.
We now have = (fu. g) and we can train a model with parameter 0' = (fa. 9) to match the marginal distributions Po (x())). As for the true variables (Ze, Z,, x()), we also have the learned ones (Z., 2, x Now we show that the true content z, and style z, are identifiable up to some transformations. We first define z is the domain of latent variable Ze C Th is the domain of latent variable Z., Z, C " is the domain of latent variable Z,, U1 is the support of the distribution u. We say Z. is component-wise identifiable if there exists component-wis invertible transformation h, s.t. 2, = h2 (z,) for the recovered 2.. The content ze is block-wise identifiable means that there exists an invertible function he s.t. i. = (z.) for the recovered i.
Z.]. n dim(z), n2 = dim(z,), n. = dim(z.). 2 C TR"
Lemma 3.1. If the underlying data generation process is consistent with and following assumptions hold:
Al (Smooth and Positive Density): The probability density function of latent variables is smooth (the second order derivative of the log density exists) and positive i.e. P(zlu) smooth and P(z]u) > 0 for all z E Z and E U.
A2 (Conditional independence): Conditioned on u, the components of mutually independent, which implies P(z = III," ((z.(u).
A3 (Linear independence): For any z, E 2, S R"=,, there exist 2n, + 1 values of u, i.e uj with j = 0, 1..., 2ng, such that the 2ng vectors w(z,, U,) w w(z,, Uo) with j = 1, 2ns are linearly independent, where vector w(s, u) is defined as follows:
[8qno+1 (Zn.ti,
Ogn (zn, u) dzn
8 p2 (Zn.
pag (zn u) dz
w(z,, u)
Ozn+t1
02 02,,11
where qi (z( u) log P(z:(u).
A4 (Domain Variability): For any set Ag C Z with the following two properties: (1) A
has nonzero probability measure, i.e. IP (fz A,]((u=u'] > for any u' E U. (2) A,
cannot be expressed as BN x Z, for any sel Br. C Z
un U2 U, such that SEE P(z )dz + SEE P(zluz)dz.
4

---

Published as a conference paper at ICLR 2023
by matching the marginal distributions (x each domain, the component-wise identifia- bility of the style Ze is ensured, Ze is block-wise identifiable.
We provide the proof and conditions in the supplementary. Intuitively, assumptions A3 and A4 require that the distribution P(z varies sufficiently across domains such that the have sufficient contrastive information to achieve the identifiability. A3 requires that the changes in the probability density functions are complex enough such that they don't always lie within (2n, + 1)-dimensional subspace. Kong et al. (2022) provide a similar result in the context of domain adaptation but it requires the dimensions of imput and output of the generating function g to be the same. If use GAN, the dimension of input noise and output image are obviously different, so our results are more general.
After recovering the true content Z. with &. and true style Z, with 2,, we now proceed to address the joint distribution identifiability problem. A main challenge the indeterminacy of the recovered content z. and style z, caused by the unknown transformation he and hg Fortunately, we show that this indeterminacy can be removed and the recovered joint distribution is the true one.
Theorem 3.2. If 1) the underlying data generation process is consistent with 455 2) assumptions Al A4-A4 hold, and 3) the marginal distributions are matched, i.e.. Pe (x Po )) for any domain i E [d]. then the true joint distribution is identical to that produced by the model with parameter 0',
Le., Po (x(1) ((2),...,x()) = Po (x(x))) x2() x..,),)(())))
The proof is provided in appendix [F] Unlike existing approaches, this theorem provides a guarantee that are able to recover the true joint distribution under some conditions. In other words, the tuples generated by the learned generative model with parameter O' can also be viewed being sampled from the true generative model. As a consequence, elements in the generated tuple (x.())..., have different styles but share the same content. We can apply this model to address the challenging multi-domain image generation task directly. As for unpaired image-to-image translation, we will show that we can also use the generated tuples help improve the performance of image translation.
,..., x (
3.2 A PRACTICAL IMPLEMENTATION OF MULTI-DOMAN IMAGE GENERATION
In this section, we provide a practical implementation of our conditional generative model and we can use achieve multi-domain image generation. Given images from d domains, we would like to train a conditional GAN model such that the generated tuples (G(e, 1), ..., G(e, d)) share the content where G is the generator in GAN and is the random Gaussian noise and u {1....d} d).
In order to generate meaningful tuples, we build our conditional GAN following 3545). Given random noise e ~ (N I) and domain label u, naive way would be splitting the noise into two parts, i.e., E = Ze: 2.] and apply component wise strictly increasing transformation fo on ž, But the problem is that the true latent variables Ze, Z, only identifiable when the dimensions match, ie., dim ( = n, = dim(z.) and dim(2.) = n, = dim(z.). We usually do not have access to the true dimension n. and n,. To address this problem, matched, i.e., n, + ne = dim(2.) + dim(z.). Then we only need to determine n. We may treat n, as a typerr-aramettr and sweep the possible values. But can be expensive since we have to determine the values for each dataset. To address this issue, we propose a simple way to allow the network to learn the optimal dimension automatically with an additional regularization term. The whole computation flow is shown in Fig. 2 Specifically, we apply component- wise strictly increasing transformation fu to all components of the input noise e. Then we multiply it with a trainable mask m and add it back to the input noise:
have assume the total dimension is
+ m fa(c), G(c, u) = g(Zc, 2.)
where the output à is the input to the generator G. The mask m is of the same dimension as noise 6 and the elements are in the range [0, +-0). For any element i, if m, 0, 2, only contains information from the shared and it belongs to é. Otherwise, é; contains the domain information u and it belongs to 2.. In order encourage the network determine the optimal dimension automatically, we apply L loss on the mask,
Espariit
|lmm]aa-
5

---

Published as a conference paper at ICLR 2023
0
7. 2.
x(n)
172
The computation flow of our model for multi-domain image generation. denotes the input noise, is domain index, fu the domain-specific component strictly increasing transformation, m is the trainable mask, 2e is the content which is shared across domains while ž, is the style which changes across domains and Ky the output image in domain u.
Figure 2:
Finally, proceed normal conditional GAN method. We introduce an conditional discrimina- tor and training objective is:
Lgan Ellog(D(x (x(x, u))] + El u))
(8)
Our full objective for multi-domain image generation is
Lgeneration
Lyan + XCsparityy
(9)
where Ả controls the influence domain u. We found that works well across datasets and is more stable than tuning the dimension in section 4.1.2
3.3 APPLICATION: UNPAIRED IMAGE- TO- IMAE TRANSLATION
Unpaired amage-to-image translation aims map input images target domain while preserving important content information. This task can also be formulated as a joint distribution learning problem (Liu et al. 2017). For example, given two domains (
o (uo)
and (MM)), image translation aims learn a reasonable condi- tional distribution Po. (x(*1) ( (*0)) through mapping function F. which should be close to the true conditional (x(x(")) (((()))))(()))))))) ((())), Therefore, we can use our multi-domain image generation model to help find proper conditional distribution. The differences between two tasks are visualized in Figure.
F
E
(u)
F(aito),55)
Capple
backbone method and other methods are also applicable. StarGAN-v2 mainly consists of: the mapping function F. style encoder E.
We employ StarGAN-v2 (Choi al., 2020
Figure 3: Our regularization.
The training loss is Staarann = Lado + Leye + Lety - Adiv where Ladv, Layer Caty and Late used for distribution matching, cycle consistency, style reconstruction and generation diversity, respectively. For more details, we refer readers to the original paper or our appendix J.1
After matching the marginal distributions, G(e, u) allows to sample meaningful tuples by changing the value of domain label u. c.g., (x( G(E, uo), (w1) G(e, uz)). Then we can use them to further regularize the mapping function F as
Luple (l(F((((0)) 31) x (w1))
(10)
where FF u1) ) the style code of domain #1- Through Luples we are encouraging the mapping function F to reconstruct the corresponding image G(e, "1) from the input image G(e. uo). means that the mapping function F is trained to preserve the correspondence between images of
the generated tuples. Since we are able recover the true joint distribution, Luple encourage the mapping function produce the true conditional distribution, i.e., Po (x(x)) (x(())))x(u)))
Our full objective for unpaired image translation is Liranslation = Lstargan + Ausple Luplee where Auple is control the influence of our propose tuple regularization.
the

---

Published as a conference paper at ICLR 2023
4 EXPERIMENTS
In this section, we first present results and analysis on multi-domain image generation task. Then we provide the results on unpaired image translation.
4.1 MULTI-DOMAIN IMAGE GENERATION
4.1.1 EXPERIMENT SETUP
Implementation We build method based the official pytorch implementation of StyleGAN2- ADA (Karras et al. 2020a) and the hyper-parameters are selected automatically by the code. We choose the deep sigmoid flow (DSF) (Huang et al. 8a) implement the domain transformation fu (Huang et al. 2018a) because DSF is designed to be component -wise strictly increasing, We use the embedding of domain label to generate
for the flow. We only introduce one hyper-paramelde À to control the sparsity of the mask. We set À = 0.1 for all experiments.
StyleGAN2-ADA
TGAN
Ours (A
Ours (A 0.1)
5
5
2
5
8
8
8
8
8
8
8
8 9 0
3
5
8
9
5
6
5
6
5 8
9
8
9
8
5
6
5
7
8
6
5
6
5 8
0
0
0
0
Figure Samples multi-domain image generation on the CELEBA-HO, AFHQ, ArtPhoto, CelebA5 and MNIST7. We provide more samples and methods in appendix G.2 method shares the same input noise e. We observe that there are unnecessary changes between the images (e.g., the added sun-glasses in the first row, the different poses of animals of StyleGAN2-ADA in second row) without regularization.
Each of the
Dataset
Metrics
StyleGAN2-ADA TGAN
CoGAN
Ours (A = 0)
Ours (A = 0.1)
FID
4.97
5.97
5.17
5.51
4.87
CELBEA-HQ
DIPD
0.61
0.60
0.58
0.60
0.58
FID
6.16
7.15
6.52
8,58
4.36
AFHQ
DIPD
1.12
1.01
1.07
1.13
1.02
FID
10.13
9.72
10.47
15.65
9.71
ArtPhoto
DIPD
1.56
1.63
1.57
1.65
1.49
FID
9.82
2.77
6.11
3.86
2.69
CelebA5
DIPD
0.40
0.25
0.38
0.26
0.23
FID
95.9
210.8
103.0
83.4
1.43
MNIST7
JoDn
160.70
277.00
152.22
144.90
9.73
Table 1: Results of multi-domain image generation on five datasets.
Datasets We use five datasets to evaluate our method: CELEBA-HQ (Choi et al. 2020 contains female and male faces domains; AFHQ (Choi et al. 2020) contains 3 domains: cat, dog and wild
7

---

Published as a conference paper at ICLR 2023
dian-,A
9
d6m 16,
0.0
6im 52.A
Senvrnoe
din 14.
0.11
<1m 200
5
Ouis. X
0.1
Senerood
(a) Ablation results
(b) Samples.
(c) I-SNE
Figure 5: Experiments on MNIST7 dataset. We observe that our proposed sparsity constraint enables network select the optimal dimension of changing components automatically (bottom left in (a)) and more stable than tuning the dimension manually (top left and right in the (a), examples in (b)). We visualize the real and generated samples with t-SNE in (c). Different colors denote different domains. We can find that the generated samples cover all classes each domain. We provide 1-SNE comparisons in the Fig. 9
life; ArtPhoto contains 4 domains: Cezanne, Monet, Photo and Ukiyoe; CelebA5 contains 5 domains: Black Hair, Blonde Hair, Eyeglasses, Mustache and Pale Skin; MNIST7 contains 7 domains: blue, cyan, green, purple, red, white and yellow MNIST digits. More information are in the appendix G.T
Evaluation Metrics. We evaluate our method using the Frechet inception distance (FID), which is widely used metric for distribution divergence between the generated images and the real images. lower FID is better. As for the first four datasets, there no pair data. So, we use the domain- invariant perceptual distance (DIPD) to measure the semantic correspondence (Liu et al., 2019). DIPD computes the distance between two instance-normalicc for MNIST7, we have ground truth tuples, so we first compute inception features of images in the tuple, concatenate the features and reduce the dimension by variance thresholding. Then we can compute the frechet distance between the features of ground truth and generated tuples. We name it Joint-FID as it computes the divergence between joint distributions.
Conv5 features of VGG network. As
Baselines. Since our method is built StyleGAN2-ADA (Karras et al. 2020a). also run StyleGAN2-ADA to verify the effectiveness of our introduced modules. We also run the most recent conditional GAN model TGAN (Shahbazi et al., (022). TGAN observes that the result of conditional generation of StyleGAN2-ADA unsatisfactory when the number of classes are large. Therefore starts as the unconditional StyeleGAN2-ADA and then gradually transits the conditional training. We reimplement CoGAN (Liu & Tuzel, 2016) based on the StyleGAN2-ADA architecture. We let the generators share all synthesis blocks except the last one.
4.1.2 RESULTS AND ANALYSIS
Comparison with Baselines We present the quantitative results in Table and show samples gener- ated by different methods in Fig. 4 Images of each row are generated by the same Gaussian noise with different values of domain label u. Our method achieves best FID on all datasets. The signifi- cant improvement over StyAeGAN2-ADA highlight the importance consider the correspondence information across domains. Our method also achieves lowest DIPD values on three datasets. The low values of DIPD demonstrates that images in our method share more content than other methods. In other words, our method is able to learn a proper joint distribution given only marginals.
The proposed sparsity loss reduces the influence domain stably and help us avoid expensive hyper-parameter sweeping. As we mentioned in section 3.2] naive way to further reduce the influence of domain label u is to reducing the dimension of z, directly. But the results can be sensitive to the dimension ni, To verify this point, we sweep the in [256. 64,32,16,8, 16,8,4 (and set À 0) and show the results in Fig. 5(a) a) (top two rows). We observe that our sparsity constraint works well if is in the range [0. J. which is common range for hyper- parametr.. However, tuning n, works well when n, 16, 32. We plot the valid number of 2, by checking the value of the trainable mask m in Fig. 5(a) (bottom left), learns to decrease n, to around 50, which is close to the optimal values by
8

---

Published as a conference paper at ICLR 2023
tuning ng. The samples in Fig. (5) also suggests that the generated tuples with sparsity loss share more content (when dim=32, it confuses between digit and 5, second row).
The identifiability when the number of domain is small The identifiability of the latent variables requires 2n, + domains as shown in section 3.1 Therefore, it would be interesting to see what happens when the number of domains is small. We have shown that method achieves the best result across different real world datasets in T Now, we choose MNIST7 dataset and decrease the number of domains used in the training. We show the results in Fig. 5 (a) (bottom right). We only compute the Joint-FID of the first two domains to ensure the results are comparable. We can observe a clear trend that the Joint-FID is improving as we increase the number of domains, which supports our theory. We observe that the StyleGAN2-ADA achieves Joint-FID of 54 and our method achieves 22 when there are only domains. We also show the generated examples in Fig 5 (b), last panel. StyleGAN2-ADA totally drops the correspondence between images in the generated tuples while our method is able to preserve the correspondence.
4.2 APPLICATION: IMAGE-TO-IMAGE TRANSLATION
Ioput
StarGAN
Ours
latent
reference
est- wildl
Method
FID4
LPIPS
FID 4 LPIPS
MUNIT (Huang al. 2018h)
41.5
0.511
223.9
0.199
DRIT [Le et al. 2018)
95.6
0.326
114.8
0.156
MSGAN (Ma
2018)
61.4
0.517
69.8
0.375
StarGAN-v2 (Chor et al
2020
16.2
0,450
19.8
0.432
SmoothGAN Liu al. 2021
51.9
0.418
52.1
0.342
LETIT Zhao & Chen
2021
15.9
StyleDis (Kim et al. Ours
2022)
14.7
11.1
0.518
13.4
0.481
Figure 6: Samples of image translation.
Table 2: Average results on 6 tasks
4.2. RESULTS
We present the experiment setup in the appendix J We present the average results of pairwise image-to-image translation tasks Table We observe significant improvements StarGAN-- V2 both latent and reference based tasks. The significant improvement of our method indicates that the generated tuples can help match the distribution well as improving the diversity of the outputs. We present more samples in the appendix
5 LIMITATION AND FUTURE WORK
Our model is trained to learn the true joint distri- bution. However, essential assumption is that the content 2, across domains are aligned. For example, the female and male domains share the same human content. For more complex data where the content sumption may be violated. For example, we find that Cezanne domain contains many fruit paint- ings while other domains contains landscape in the ArtPhoto dataset. Although our method has achieved best quality as well as semantic correspondence compared to other methods, we still observe some failure cases. As shown Figure. 7 to match the distribution Cezanne domain, the generator learns to generate the fruit painting (first column). Although can observe that our method still works on other three domains (the rest columns), we can observe content mismatch due to the unaligned training dataset. Our proposed architecture may over-constrain the influence of domain label in this misalignment case. We may also resort to some other quantitative measures of the influence help address this challenging problem and leave future work.
aligned, our as-
Figure 7: Failure case on ArtPhoto dataset, columns are Cezanne, Monet, Photo and Ukyioe

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
We thank the anonymous reviewers for their devoted time and constructive feedbacks, which are really helpful to improve the quality of this paper.
This project was partially supported by the National Institutes of Health (NIH) under Contract R01HL159805, by the JSF-Convergenne Accelerator Track-D award 2134901, by grant from Apple Inc., grant from KDDI Research Inc, and generous gifts from Salesforce Inc., Microsoft Research, and Amazon Research. MG was supported by ARC DF210101624.
REFERENCES
Youssef Alami Mejjati, Christian Richardt, James Tompkin, Darren Cosker, and Kwang In Kim. Unsu- pervised attention- -uideed image-to-image translation. Advances neural information processing systems, 31, 2018.
Karim Armanious, Chenming Jiang, Sherif Abdulatif, Thomas Küstnnr, Sergios Gatidis, and Bin Yang. Unsupervised medical image translation using cycle-medgan. In 2019 27th European Signal Processing Conference (EUSIPCO), pp. 1-5. IEEE, 2019.
Karim Armanious, Chenming Jiang, Mare Fischer, Thomas Küstner, Tobias Hepp, Konstantin Niko- laou, Sergios Gatidis, and Bin Yang. Medgan: Medical image translation using gans. Computerized medical imaging and graphics, 79:101684, 2020.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint 1rXi:: 1096, 2018.
Jiezhang Cao, Langyuan Mo, Yifan Zhang, Kui Jia, Chunhua Shen, and Mingkui Tan. Multi-marginal wasserstein gan. Advances in Neural Information Processing Systems, 32, 2019.
Yunjey Choi. Minje Choi, Munyoung Kim, Jung- Woo Ha, Sunghun Kim, and Jaegul Choo. Star- gan: Unified generative adversarial networks for multi-domain image-to-imas translation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8789-8797, 2018
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8188-8197, 2020.
Mingming Gong, Yanwu Xu, Chunyuan Li, Kun Zhang, and Kayhan Batmanghelich. Twin auxilary classifiers gan. Advances in neural information processing systems, 32, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
Luigi Gresele, Paul K Rubenstein, Arash Mehrjou, Francesco Locatello, and Bernhard Schölkopf. The incomplete rosetta stone problem: Identifiability results for multi-view nonlinear ica. In Uncertainty in Artificial Intelligence, pp. 217-227, PMLR, 2020.
Hermanni Hälvä and Aapo Hyvarinen. Hidden markov nonlinear ica: Unsupervised learning from nonstationary time series. In Conference on Uncertainty in Artificial Intelligence, pp. 939-948. PMLR, 2020.
Junlin Han, Mehrdad Shoeiby, Lars Petersson, and Mohammad Ali Armin. Dual contrastive learning for unsupervised image-to-image translation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 746-755, 2021.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun- Yan Zhu, Phillip Isola, Kate Saenko, Alexci Efros, and Trevor Darrell. Cycada: Cycle-consiseee adversarial domain adaptation. In International conference on machine learning, pp. 1989-1998. PMLR, 2018
10

---

Published as a conference paper at ICLR 2023
Chin-Wee Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. In International Conference on Machine Learning, pp. 2078-2087. PMLR, 2018a.
Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-image translation. In Proceedings of the European conference computer vision (ECCV), pp. 172-189, 2018b.
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in neural information processing systems, 29, 2016.
Aapo Hyvarinen and Hiroshi Morioka. Nonlinear ica of temporally dependent stationary sources, In Artificial Intelligence and Statistics. pp. 160-469, PMLR, 2017.
Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 859-868. PMLR, 2019.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1134, 2017.
Minguk Kang and Jaesik Park. Contragan: Contrastive learning for conditional image generation. Advances in Neural Information Processing Systems, 33:21357-21369, 2020.
Minguk Kang, Woohyeon Shim, Minsu Cho, and Jaesik Park. Rebooting acgan: Auxiliary classifier gans with stable training. Advances in Neural Information Processing Systems, 34, 2021.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33:12104- 121 14, 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8110-8119, 2020b.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. International Conference on Artificial Intelligence and Statistics, pp. 2207-2217. PMLR, 2020a.
Ilyes Khemakhem, Ricardo Monti, Diederik Kingma, and Aapo Hyvarinen. Ice-beem: Identifiable conditional energy-based deep models based on nonlinear ica. Advances in Neural Information Processing Systems, 33: 13:11788-12778 2020b.
Junho Kim, Minjae Kim, Hyconwoo Kang, and Kwanghee Lee. U-gat-it: Unsupervised generative attentional networks with adaptive layer-instance normalization for mmage-to-image translation. arXiv preprint arXiv: 10830, 2019.
Kunhee Kim, Sanghun Park, Eunyeong Jeon, Tachun Kim, and Daijin Kim. A style-aware discrimi- nator for controllable image translation. arXiv preprint arXiv.2203.15375 2022.
Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning discover cross-domain relations with generative adversarial networks. In International conference on machine learning. pp. 1857-1865. PMLR, 2017.
David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. arXiv preprint arXiv22007.10930 2020a.
David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. arXiv preprint arXiv:2007.1099 2020b.
Lingjing Kong, Shaoan Xie, Weiran Yao, Yujia Zheng, Guangyi Chen, Petar Stojanov, Victor Akinwande, and Kun Zhang. Partial disentanglement for domain adaptation. In International Conference Machine Learning, pp. 11455-1144 11472- PMLR, 2022.
11

---

Published as a conference paper at ICLR 2023
Lingke Kong, Chenyu Lian, Detian Huang, Yanle Hu, Qichao Zhou, al. Breaking the dilemma of medical image-to-image translation. Advances in Neural Information Processing Systems, 34: 1964-1978, 2021.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11):22782233) 1998.
Hsin- Ying Lee, Hung- Yu Tseng, Jia- Bin Huang, Maneesh Singh, and Ming- Hsuan Yang. Diverse image-to-image translation via disentangled representations. In Proceedings of the European conference on computer vision (ECCV), pp. 35-51, 2018.
Torgny Lindvall. Lectures on the coupling method. Courier Corporation, 2002.
Alexander H Liu, Yen-Cheng Liu, Yu- Ying Yeh, and Yu-Chiang Frank Wang. A unified feature disentangler for multi-domain image translation and manipulation. Advances neural information processing systems, 31, 2018.
Ming- Yu Liu and Oncel Tuzel. Coupled generative adversarial networks. Advances in neural information processing systems, 29, 2016.
Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to pongge-tt-immn to--iage translation networks. Advances in neural information processing systems, 30, 2017.
Ming- Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. Few-shot unsupervised image-to mmage-to-mmage translation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10551-10560, 2019.
Yahui Liu, Enver Sangineto, Yajing Chen, Linchao Bao, Haoxian Zhang, Nicu Sebe, Bruno Lepri, Wei Wang, and Marco De Nadai. Smoothing the disentangled latent style space for unsupervised image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10785-10794, 2021.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Francesco Locatello, Ben Poole, Gunnar Rätsch, Berhhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In International Conference on Machine Learning, pp. 6348-6359 PMLR, 2020.
Qi Mao, Hsin Hsin-Ying Lee, Hung- Yu Tseng, Siwei Ma, and Ming- Hsuan Yang. Mode seeking generative adversarial networks for diverse image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition, pp. 1429-1437, 2019.
Xudong Mao and Qing Li. Unpaired multi-domain image generation via regularized conditional gans. arXiv preprint arxiv: aete:::055005555 2018.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint
1rXiv:14111 1784,, 2014.
Takeru Miyato and Masanori Koyama. cgans with projection discriminator. arXiv preprint veXiv:1802.05:: 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05995, 2018.
Zak Murez, Soheil Kolouri, David Kriegman, Ravi Ramamoorthi, and Kyungnam Kim. Image to image translation for domain adaptation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4500-4509, 2018
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. In International conference machine learning, pp. 2642-2651. PMLR, 2017.
12

---

Published as a conference paper at ICLR 2023
Taesung Park, Alexei A Efros, Richard Zhang, and Jun-Yan Zhu. Contrastive learning for unpaired image-to-image translation. In European Conference on Computer Vision, pp. 319-345. Springer, 2020.
Yunchen Pu, Shuyang Dai, Zhe Gan, Weiyao Wang, Guoyin Wang, Yizhe Zhang, Ricardo Henao, and Lawrence Carin Duke. Jointgan: Multi-domain joint distribution learning with generative adversarial nets. In International Conference on Machine Learning, pp. 4151-4160. PMLR, 2018.
Mohamad Shahbazi, Martin Danelljan, Danda Pani Paudel, and Luc Van Gool. Collapse by con- ditioning: Training class-conditional gans with limited data. arXiv preprint arXiv:2201.06577 2022.
Rui Shu, Yining Chen, Abhishek Kumar, Stefano Ermon, and Ben Poole. disentanglement with guarantees. arXiv preprint 1rXiv:1910009772, 2019.
Weakly supervised
Hung- Yu Tseng, Lu Jiang, Ce Liu, Ming- -isuan Yang, and Weilong Yang. Regularizing generative adversarial networks under limited data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7921-7931, 2021.
Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34: 34:16451-16461 2021.
Jinghua Wang and Jianmin Jiang. Conditional coupled generative adversarial networks for zero-shot domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3375-3384, 2019.
Weilun Wang, Wengang Zhou, Jianmin Bao, Dong Chen, and Houqiang Li. Instance-wise hard negative example generation for contrastive learning in unpaired image-to-image translation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14020-14029, 2021.
Yan Wu, Jeff Donahue, David Balduzzi, Karen Simonyan, and Timothy Lillicrap. Logan: Latent 2019.
optimisation for generative adversarial networks. arXiv preprint
Shaoan Xie, Mingming Gong, Yanwu Xu, and Kun Zhang. Unaligned image-to-image translation by learning reweight. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14174-14184, 2021.
Shaoan Xie, Qirong Ho, and Kun Zhang. Unsupervised image-to-image translation with density changing regularization. In Advances in Neural Information Processing Systems, 2022.
Yanwu Xu, Shaoan Xie, Wenhao Wu, Kun Zhang, Mingming Gong, and Kayhan Batmanghelich. Maximum spatial perturbation consistency for unpaired image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1831 183220. 2022.
Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image- to-image translation. In Proceedings of the IEEE international conference on computer vision, pp. 2849-2857, 2017.
Xiaoming Yu, Yuanqi Chen, Shan Liu, Thomas Li, and Ge Li. Multi-mapping image-to-ing to-geet translation via learning disentanglement. Advances in Neural Information Processing Systems, 32, 2019.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International conference on machine learning, pp. 7354-7363. PMLR, 2019a.
Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lee. Consistency regularization for generative adversarial networks. arXiv preprint arXiv: /0027, 2019b.
13

---

Published as a conference paper at ICLR 2023
Yang Zhao and Changyou Chen. Unpaired image-to-image translation via latent energy transport. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
16418-16427, 2021. 2021.
Zhengli Zhao, Sameer Singh, Honglak Lee, Zizhao Zhang, Augustus Odena, and Han Zhang. Improved consistency regularization for gans, arXiv preprint arXiv:2002.04724, 2020.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference computer vision, pp. 2223-2232, 2017.
Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning, pp. 12979-12990. PMLR, 2021.
14

---

Published as a conference paper at ICLR 2023
A ETHICAL STATEMENT
Generative models such as GAN used in our paper enable various applications and our model facilitates the technique and may reduce the manual labeling cost. Unfortunately, as revealed by previous reports, it becomes easier to manipulate image data, such the deepfakes. In addition, the generative models may reveal the training data information. How to address these negative impacts still remains important problem. All datasets used are publicly available.
B DISCUSSION ABOUT OUR MODEL AND THE CONDITIONAL GANs
Conditional GANs usually feed the concatenation of the input noise e and domain label into the generator G. By contrast, we inject the domain influence to the noise e through component-wisg strictly increasing transformation fu. From theoretic perspective, our architecture allows us to recover the true joint distribution as shown in section 3.1] From an empirical perspective, we would like to reduce the influence of domain information in generative model. To generate images in each domain, the generator needs to utilize the input domain label u. However, the influence of can be very large if no constraint is applied. If the influence of domain variable is unnecessarily large, the generator focuses on the domain variable and pays little attention to the content variable ze- As consequence, the generated images in the tuple may lose correspondence (c.g., the generated animals have different poses on AFHQ dataset, Fig. 4. An extreme case would be that the content is totally ignored by the generator. So the generator still outputs images in different domains (because the input domain labels are different) ) but all images in the same domain look the same (because the content variable 2e ignored) (e.g., the generated digits look the same on MNIST7 dataset, Fig. 4.. By contrast, we first use the simple transformation fu to inject the domain information into rather than concatenating with e. Secondly, the sparsity loss Cspuriity reduces the number of components affected by the domain label Therefore, there are will only be necessary changes between the images the generated tuples.
C RELATED WORK
Multi-domain Image Generation Conditional GAN aims to maximize the influence conditioning
variable to improve the diversity of the generated samples (Gong et al. 2019 Brock et al. 2018
Miyato & Koyama 2018:
Odena et al. 2017 Kang et al., 2021 Kang & Park 2020 Tseng et al.
2021
Karras et al. 2020a Miyato et al. 2018 Zhang et al.) 201 9a; Brock et al. [2018] Wu et al.
2019
Zhang et al.] 2019b] Zhao ct al.] 2020j. Our task is also related to the conditional GAN problem
If we regard the domain label as the conditioning information. A major difference is that conditional
GAN mostly focusing on enlarging the influence of the condition class label and generate diverse images across classes (Kang al. I) while our method is trying reduce the influence of the condition domain label and generate meaningful tuples across domains.
Unpaired Image-to-Image Translation Image-to-image translation can also be viewed as joint distribution learning problem between the source and the target image domain. With pair data, pix2pix (Isola al.) 2017) applies conditional GAN match the distribution of the target domain and penalize the distance of generated image to the ground truth image. Unfortunately, paired data is usually difficult to collect. Therefore, additional assumptions are made to address the unsupervised task, such as the cycle consistency (Zhu et al. 2017 Kim et al. 2017 Yi et al.) 2017 Choi et al. 2018, 2020; Liu et al. 2021 Kim et al. 2022). ) shared latent space (Liu et al., 2017: Huang et al. 2018b [ee et al. 2018 Yu et al. 2019 Liu et al. 2018), relationship preservation(Part et al. 2020
Han et al.) 2021 Wang et al. |021] Cao et al. 2019: Xu et al.) (022). , density changing Xie et al.
(2022), importance reweighting Xie et al. (221)) We build our method based on StarGAN-v2 (Choi
et al. 2020), which relies on cycle consistency to preserve content.
D THE GRAPHICAL MODEL
To address the ill-posed problem, we assume the generation process follows following graphical model:
15

---

Published as a conference paper at ICLR 2023
d
ZC
Zs
Z
Z
x
Figure 8: The graphical model of our method. Ze is the shared content and 2(0) is the style from domain and x is the observation: data (images) in domain i.
PROOF OF THE IDENTIFIABILITY OF LATENT VARIABLES
We show that the true content Ze and style 2, are identifiable in Lemma. 3.1
E. THE IDENTIFIABILITY OF THE STYLE
We first provide the proof of the identifiability of the style variable z,,
Proof. As we have matched marginal distributions (x = (x(x) we have (()(z) ()) (u) = P[g(z)lu) since g(2), gz). Then ((2) and gz). which results in P(g o (()) = P(ğ-1 g(2) ()) = P(ε = P(ğ (u))
apply same transformation ğ to the variables
We define h g -1 9. then h و g. have
P(ž = P(h (z)lu.
(11)
which suggests that h is the indeterminacy between the recovered latent variable 2 and the true latent z. It worth noting that we don't t need to assume that the dimension of and x are same in order to compute the determinant of g
We further define of' = log P(z.(u) and q." = log P(2.uu). With the conditional independence assumption in A2, have
P(z|u)= = II P(z uJ + log P(z|u)
9
i=1
i=1
P(žlu) II P(z uu) + log P(žlu)
q
According to the change of variable rule, we can transform equation. into
P(ž J P(z|u)-
J
(12)
where J is the absolute value of the determinant of the Jacobian matrix of h. Since h invertible and the input and output share the same number of dimension, we have J
[Al A 0.
To simplify the notation, we define the following objects:
Oz,
Sz,
O2*
oz O2,
0q.¹
82 q."
(2,, u) =
T/ (Z,, u) = (8z;)²²
Oz.
Differentiating Equation 12 twice w.r.t. 24 and 2n where kg [ni] and k. + yields
16

---

Published as a conference paper at ICLR 2023
log [JA]
(Zi, u) Ma + rk(zi, u) ( +
0.
(13)
02
Therefore, there are 2n, + 1 equations corresponding to = uo Ugn, respectively. We subtract equations associated with u...... U2n, with the equation of ug, and we obtain the following 2n, equations:
Σ (Zi, - (z, uo)) (a + (7/(2i, uz) - 1/,(2, uo)) k( = 0, (14) i=
where j = ,,... 2n,- Due to the invariance of E.s we have P(z.) P(z.lu). Thus, we have 1// (2,, uz) = 112 (2; upr) and (2, U2) = (2, uy), Vj.j'. Hence only the style components i == ne + 1,..., n remain in the summation of each equation.
Under the linear independence condition in Assumption A3, the linear system is a 2n, x 2n, invertible. Therefore, the only solution is ( 0 and ((,(k.)) 0 for ne + 1...., n and
k, g E [n), #g.
As h(-) is smooth over 2, its Jacobian can be written as:
A
aze
B
Uze
Jn
C
Oz.
22
D
Oz.
Tz.
(15)
The
02.
Note that K K4,0e (()))) 0 implies that for each ne + l,..., n, M(,(k) * 0 for at most one element E [n]. Therefore, there is only at most one non-zero entry in each row indexed by = ne + 1,..., in the Jacobian matrix Jn Further, the invertibility of h(-) necessitates JA- - to be full-rank which implies that there is exactly one non-zero component in each row of matrices C and D.
Since for every i E {ne + 1, I....,n).. 2; has changing distributions over and all 2 's for E nee (i.e. 2e) have invariant distributions over u. we can deduce that only non-zero entry & must reside D with & E {n. + l...., n). Therefore, for each estimated variable in the changing part &,, i {ne - 1,..., n). there exists one true variable the changing part Zk, {n. + i,.... .......77 n such that 2, (zx)).. Further, because Jn,- is of full-rank (h(-) being invertible) and is a zero matrix, D must be full-rank, which implies that hç(-) invertible for each i {ne + 1,...., n.. Thus, the changing components 2, identified up to component-wiss invertible transformations.
n).
and the
zt.
E.2 THE IDENTIFIABILITY
CONTENT VARIABLE
Now we provide the proof for the block-identifiabiiity of the content variable Z.-
Proof. The proof is presented in four steps as follows.
Step 1. Due to assumption of the data generating process of the learned model, we can ob- tain the independence between the generating process &n and u. Thus, follows that for any Ag. C Ze,
1 (x) Az, luu un} f (x) E Av. Hu uz). Yu,, u2 u
{x (0 "
un} (x E -
(A (A, u2). Vui, Up (16)
where -- X + Z. denotes the estimated transformation from the observation to the content variable and ( - C X is the pre-image set of Az.-
17

---

Published as a conference paper at ICLR 2023
On account of the matching assumption, are able to extend Equation a6 follows:
(x E (fim.))
) ) "0 (Az.) 11) = {x (Di:n.) ) (Az. ) =
=
(i (x) Ag. buu = un to (x) E Ag. H( (u= = =2
(17)
Because and are smooth and injective, there exists smooth and injective h g-lo - Z. Expressing 9 - hog- and h.(.) ) == hi: (-) 2 - Zr in Equation 17 yields
(ho(z) Alu -un = (ho(2) E Az =u2h,
{z E no f = fz No (A (u =u2).
J "Anr))
Pz Paju(z|u1) (1) dz
Pzju Palu(Z|112) (2/112) dz.
(18)
JzEh, (Ane)
where h- (A2.) = {z E Z : hz(z) E Az. is the pre-image of Az, i.c. those latent variables containing content variables in Az, after the indeterminacy transformation h.
Based on the generating process, we can re-write Equation 18 as follows: VAz. S Ze,
Pz. (Ze) (Pe.)u ( Pa. (Zgl dzz 0.
(19)
TTTTE, (Aac)
Step 2. In this step, we prove that Z, = he _ 2. TT does not depend on 2,. To this end, we first develop an equivalent statement (i.e. Statement below) and prove subsequently. This enables us to leverage the full-supported density function assumption to avert technical issues.
Statement 1: he(lz] z " does not depend on Z..
Statement Vz. Ze. follows that h- (ze) = Be. x Z, where Bz, # O and Be, S Ze.
Statement Vz. Z r E RR follows that (B,, = B 2, where (zc) = {z', Ze: 1zz- zel1 < r]. Bz. + 0. and Bt C Zc-
Statement 2 is mathematical formulation of Statement 1. Statement 3 generalizes singletons in Statement 2 to open, non-empty balls (z.). Later, we show that under this generalization, B2, is necessarily of probability measure greater than 0. With this, we can proceed show its contraction to Equation 19
Leveraging the continuity of h.(.), we show the equivalence between Statement 2 and Statement 3 follows. We first show that Statement 2 implies Statement Vz, E Ze, R h. h- ((B,()))) = (x.)) (z)). Statement 2 indicates that every participating sets in the union satisfies h- (z)) = By, x Z,, thus the union h- ((B,(z))) also satisfies this property, which Statement 3.
Then, we show that Statement 3 implies Statement 2 by contradiction. Suppose that Statement 2 false, then 3ž, E Zo such that there exist - E {zi:. :z E (2c)) and 2 e Z, resulting As he(-) is continuous, there exists f E IR such that (2 E (2c)... That is, B @ h (B+(2.)) On the other hand, Statement 3 suggests that h- (BBx)z.) = Bz. x Zg By definition of 2, it is clear that 2 E Bz.. However, the fact that 2B & h- (Bx(2z)) contradicts Statement 3. Therefore, Statement 2 is true under the premise of Statement 3. We have shown that Statement 3 implies Statement 2. Consequently, State- ment 2 and Statement 3 are equivalent, and therefore proving Statement 3 suffices to show Statement 1.
in (2
B) of 2 where ² = (2 )T (2
Step 3. In this step, prove Statement 3 by contradiction. Intuitively, show that if h-(.) depended on 2,, the pre-image h- (B, (z.)) could be partitioned into two parts (i.e. B and h- (Az_) By defined below). The dependency between ha(+) and é, is characterized by Be which
18

---

Published as a conference paper at ICLR 2023
would not emerge otherwise. In contrast, (A B2 also exists when he(-) does not depend on
2.. We evaluate the invariance relation Equation 19 and show that the integral over h. 2 (A2, \ B
(i.e. T1) is always 0, however, the integral over 13, (i.e. T2) is necessarily non-zero, which leads to the contraction with Equation 19 and thus shows the (-)-) cannot depend on 2g-
First, note that because (ze) is open and h- is continuous, the pre-image (B, (ze))
is open. In addition, the continuity of h(-) and the matched observation distributions vu' E U, fx E Axllfu = ='} {x Ax u u'} lead to h(-) being bijection as shown in (Klindt et al. 20200),which implies that hn (z.))..) is non-empty. Hence, (1, is both non-empty and open. Suppose that 3A = (z,) where z Zc, r" E IR+,, such that 2g
B = {z E 2 partition of the pre-image h- (At.) that the style part Zm.+1mm cannot take on any value in Z,. Only certain values of the style part were able to produce specific outputs of indeterminacy he(-). Clearly,
( {Zimn.} } Z, - (A2,) + 0. Intuitively, B contains the
this would suggest that h2(.) depends on Ze-
To show contraction with Equation
19 we evaluate the LHS of Equation T9 with such A
[ [ 22] TeTTTh, Eho P(Age)
Pa. (Ze) (Pe, (2n| Pz, (2x (u2) dz
P2o (Ze) (Pe.ju(Zg/1ug Pa. Ju (22) dz
[a_ [T]) ENE (A2,)121
T1
Pa. (Ze) (Pe,)u( - Pa Pa (2uluz)) dz, dze.
T2
We first look at the value of Ti. When h (A B = 0, Ti evaluates to O. Otherwise, by definition can rewrite h- (Ar.) ) B Cū 2, where C+ + and C+ C Ze. With this expression, follows that
Pe. (ze) (px, (z)/uz) Pa, |u (zg (2)) dz
-I Pa. (Ze) (Pa.ju(zalup) Pa, (nljn(z (Zg (2)) dz
2.E2,
[
J2.Ec Pa. (ne) (1 doc 0.
Therefore, in both cases Tj evaluates to 0 for As..
Now, we address T2. As discussed above. h=' (At.) is open and non-empty. Because of the continuity of he(-), VzB E By, there exists r(ZB) E IR+ such that Br(zs) (ZB) S B As Pz u (z u) over (z, u), we have {z E B u'} 2 {z E Br(zn) ) {u u'}} 0 for any u' E U. Assumption A4 indicates that 3uj, uzz such that
T2 T2:=1
Pa. (ze) (Pa, (Zg) [uj Pz, (n (zg u3)) dz,dze + 0.
Therefore, for such A we would have Ti T2 o 0 which leads contradiction with Equation 19 We have proved by contradiction that Statement 3 is true and hence Statement holds, that is, h-(-) does not depend on the style variable z
Step 4. With the knowledge that h-(-) does not depend on the style variable z,, we now show that there exists an invertible mapping between the true content variable 2, and the estimated version 2
As h(-) is smooth over 2, its Jacobian can written as:
02g
B
02.
J h/
Dz.
Oz. :
(20)
C
Oz.
D
32.
az.
O2,
19

---

Published as a conference paper at ICLR 2023
where notation 2. 1.). and ã, ht(z)n++iin As we have shown that 2. does not depend on the style variable z. it follows = 0. On the other hand, as h(.) is invertible over 2, Jn. non-singular. Therefore, A must be non-singular due to B 0. Note that A is the Jacobian of the function (z,) = hz(2) 2. 2,. which takes only the content part 2, the input z into he- Also, note that the result of Theorem 3.1 implies that C = O. Together with the invertibility of h, we can conclude that I/ is invertible. Therefore, there exists an invertible function hyg between the estimated and the true content variables such that 2e hi (Ze),, which concludes the proof that ze bookkiidenifiable via 9 (-).
20

---

Published as a conference paper at ICLR 2023
F PROOF OF THE IDENTIFIABILITY OF TRUE JOINT DISTRIBUTION
In this section, we provide the proof of theorem 3.2
Proof. Two domains We consider the case of two variables first, i.e. we prove that Por (x())x(())) = Po (x(x x(2)). We factorize the joint distribution ,(x = ((x)(((x)))) (1)). Since we have matched the marginal distributions for each domain, we already have Pe (x(1) = (o(( So
we now only need to prove Pa. (x(2) (2)) = Pa (x (x1)).
Given x (Zz; 7.): we can identify the style and content through the inverse of the model, i.e., 2, = 9 (x). According to lemma. 3.1, we have he(z.) = 2e and ha(2e) = 2. where he is an invertible transformation and hg is component wise invertible transformation. Therefore, we have
glze: Zo) g(2.. 20) ((hc(ze), (z.)))
(21)
For the ease of notation, we denote c c Ze, = 7g. Given a pair of images (z r(0)) sampled from the true joint distribution, we have
z (1) g(c () (2) = g (
(22)
Since we assume that the domain specific transformation is component-wise monotonic fu. fu is component-win wise invertible. We denote f1 = fu=1, f2 = fu=2, then we have
22 = f2 fi (1()) s (1) = f2 (2)
(23)
Similarly, the learned model fu is also component-wise invertible, we also have
2 = f2 (((1))
(24)
Given (2) in the first domain, we have
e(1) = ho(c(")), 8(1) = (a((s(1)) x(1) = )((c()(())
(25)
Then we have
8(2) 120 t by equality in 24
(26)
1 o (h(())))))),, by equality in 25
fe j. t o T (s(22) by equality in 23
We now prove that g(2) is function of the true s(2), According to our lemma, the function between g(2) and s(2) can only be ha Therefore, have g(2) ha (2(2)) Then the output generated by our learned model with parameter 0' is
x(2) g(((1) ((2)) = )(ho(c()),, (s(2)))
(27)
g(c!!) ,(2)),
by equality in 21 21
(2))
The results show that given input 21),, our learned generative model with parameter 0' outputs the same result as the true generative model with parameter 0. In other words, Por (x)
Po (x(x())x(())
Generalization of the result with mathematical induction So far, we have proved that (x(())) x(22) (x(x (x))))))) Now we generalize the results into more domains with mathe- matical induction on the number of domains.
Base Case When the number of domains is 2, we have proved that Po (x(1), x(2)) (x(1)) x(2)).
21

---

Published as a conference paper at ICLR 2023
Inductive Step. Suppose the true joint distribution is identifiable when the num- ber of domains is - 1, i.e., Por (x((x()) .... (..,x((-11) = Po(x(1), ( x(d-1)), now we prove that the true joint distribution still identifiable when the number of domains is d, i.e., Por (x x(d) Px(x(1) x()). In fact, we only need to prove Pa (x)((x)) ( ( Pe (x) (()) ( according to the induction assump- tion.
P(x(0) (1(0))))) ((x)(x(((())) ,01)) ( ,())) ,,?)),..,,, we apply the invertible function g
(28)
P(x s(_,111111)
(29)
= -( ,si (1 ),.,, , Sa- fa-1(s((-1))), we apply the invertible transformations f-1
(30)
P(x(0)) , ",,,,..,))
(31)
= -(xx(0)))) P(x = 1( (1)
(32)
(33)
(34)
We can find that it becomes the two domain case, i.e., we just need to prove Por (x(d) x(1) =
Po (1)). which already holds according to our results in two domains.
Therefore, the true joint distribution identifiable.
G MULTI-DOMAIN IMAGE GENERATION
G.1 DATASET
We use five datasets to verify our model.
CELEBA-HQ (Choi et al., 2020) contains domain: female and male. We also use the training set to train our model. Female domain contains 17943 images and male domain contains 10057 images. We train the model at resolution x56x256.
AFHQ (Choi al. 2020) contains 3 domains: cat, dog and wild life (c.g., foxes, tigers and lions). We use the training set to train our conditional GAN model. Three domains contain 5153, 4739, 4738 images, respectively. We train the model resolution 256 x 256.
ArtPhoto contains 4 domains: Monet, Cezanne, Ukiyoe paintings and real photos (Zhu et al. 2017).
CelebA5 contains 5 domains: Black Hair, Blonde Hair, Eyeglasses, Mustache and Pale Skin. They are subsets of domain CelebA (Liu et al. 2015). We train them at 64 x 64 resolution.
MNIST7 contains 7 domains: blue, cyan, green, purple, red, white and yellow MNIST digits.
We generate these digits using the training MNIST dataset (LeCun et al. 1998). We train the model at resolution 32x32.
G.2 MORE RESULTS
We now provide more results of multi-domain image generation.
G.3 T-SNE OF REAL AND GENERATED SAMPLE
H TWo VERSIONS OF MASK MECHANISM
In our main paper, we propose to use 2, = + m @ (c) to encourage the network select the optimal dimension of n. automatically. Another possible version would be 2? (1 m2) ma fu (c).
22

---

Published as a conference paper at ICLR 2023
Dataset
Metrics
StyleGAN2-ADA
TGAN
CoGAN
Ours (1 =
Ours (1 0.1)
FID
4.97
5.97
5.17
5.51
4.87
CELBEA-HO
DIPD
0.61
0.60
0.58
0.60
0.58
Precison
0.716
0.703
0.700
0.739
0.721
Recall f
0.454
0.382
0.418
0.391
0.445
FID +
6.16
7.15
6.52
8,58
4.36
AFHQ
DIPD 4
1.12
1.01
1.07
1.13
1.02
Precision 1
0.757
0.710
0.745
0.769
0.706
Recall t
0.241
0,400
0,202
0.190
0.490
FID
10.13
9.72
10.47
15.65
9.71
ArtPhoto
DIPD
1.56
1.63
1.57
1.65
1.49
Precision t
0.696
0.651
0.690
0.715
0.651
Recall
0.281
0.373
0.298
0.106
0.359
FID 4
9.82
2.77
6.11
3.86
2.69
CelebA5
DIPD
0,40
0.25
0.38
0.26
0.23
Precision
0.751
0.661
0.718
0.689
0.662
Recall f
0.323
0.510
0.181
0.452
0.537
FID
95.9
210.8
103.0
83.4
1.43
MNIST7
Joint-FID4
160.70
277.00
152.22
144.90
9.73
Precision
0.046
0.000
0.009
0.022
0.453
Recall f
0.000
0.000
0.000
0.000
0.614
Table 3; Results of multi- dollii image generation on five datasets. We observe that the precisons of our method are slightly lower while the recalls are high across datasets, which is generally desirable since recall can be traded into precision via truncation, whereas the opposite is not true (Karras et al. 2020b Kynkäääniemi et al. 2019).
There is a main implementation reason why we choose the first version: if we choose the second version, have to wrap value inside [0,1]. A common way is sigmoid function. However, sigmoid known for gradient vanishing (the gradient is very small when input is far from 0). So, sparsity penalty and the GAN loss may have little effect the sigmoid mask, which leaves the influence domain very large. To testify the claim, we run experiments on AFHQ and CELEBA-HQ dataset with two versions in Table. We set
0.1.
AFHQ
MNIST7
Mask Version
FID
FID 4
Versionl: = c fu(e).
4.36
1.42
Version2: 2, = m)
Dc fu(e)
7.52
95.6
Table The results of two mask mechanisms.
I THE NECESSITY OF THE MASK MECHANISM
Our mask mechanism allows selecting the dimensions for injecting domain influences automatically. Another possible way seems to be manually define the dimension of style n,, and tune it. is worth noting that when one tries to find the optimal value of from 4,8, 16,32., in principle, one has to consider not only different values, but also which dimensions of the input of GAN should have changing distributions for each value of n., in order to achieve the best performance. Because of the complexity of the transformation implied by GAN, which dimensions of the input of GAN should have changing distributions may heavily depend on the initialization of GAN. That is, if we just allow first inputs of GAN to have changing distributions, is completely possible that other inputs, actually should learn be one of those with changing distributions. In this case, it will be hard for GAN to learn the optimal function, and hence the strategy forcing the first dimensions of the input of GAN to have changing distribution,
with the optimal value, may lead to high variability in the final performance. (On the other hand, is not computationally feasible to consider each subset the inputs of GAN of size, the procedure, and find the best one.) This phenomenon is analogous to
23

---

Published as a conference paper at ICLR 2023
Datset
fg =
71
16
n1
Ours
AFHQ
4.97
5.05
4.63
4.36
MNIST7
2.0
3.7
32.6
4.87
Table 5: FID values when tuning n2..
the relationship between traditional information criterion (like BIC)-based model selection or subset selection and parameter shrinkage (say, with the penalty) for variable selection.
J UNPAIRED IMAGE-TO-IMAGE TRANSLATION
J.1 DETAILS ABOUT STARGAN-V2
As mentioned in section 3.3 we build our method based on StarGAN-V2 et al. (2020). Now provide more details about the method. StarGAN-v2 consists of four modules: mapping network H(,,), style encoder E(.,)) .), the shared generator F and the discriminator D. The training loss is
Ssturgan
Lado + Loye + Luty Adiv
(35)
where Ladu. Leyes Lsty and Ldie are used for distribution matching, cycle consistency, style recon- struction and generation diversity, respectively. StarGAN-V2 supports two kinds of tasks: latent- based image translation and reference-based image translation. Given an input image, latent-based generates the style code from the random noise with the mapping network H, i.e., $ ((,, y). where is the target domain label. As for the reference based, the style is extracted from an image the target domain y. i.e.,& (xx2. y). The shared generator takes an input image and the style code ä and outputs an image from the domain y. StarGAN-V2 V2 concatenates multiple domain specific heads for each domain and we denote as Dy for domain y.
Ladv
Elog (x)] + [loog(1 - Dg (f,, 8)))
(36)
performs adversarial training between the generator and the discriminator.
Lsty | E(F(x, 8))
(37)
encourages the network F contains the style information of s.
Lav
E[ F(x, 51) (2)));
(38)
which encourages output images to be different by maximizing this loss.
The final one is the cycle consistency (Zhu et al. 2017).
Leye
E||| F(F(x, 5)]]]
(39)
which encourages the network F to reconstruct the input from the translated one. other words, encourages the mapping to be one-to-one.
J.2 OUR TUPLE LOSS
However, as shown in previous literature (Alami Mejjati et al. 2018 Kim et al. (019), the cycle consistency is not enough and can still leads to large content distortion. Hence, we introduce our loss
Luple (G((, wo), 52) G(e, ui)l
(40)
where G(e, uo), Gle, u1) are pair data generated by our multi-domain image generation model. As proved in section F the tuples can be viewed as sampling from the true joint distribution. Therefore, we are using Luple to encourage the mapping network F to recover the second image from the first image. In other words, by minimizing this tuple loss, the network F learns the correspondence relationship between our method. Therefore, our tuple loss helps further regularize the mapping network F and avoid large content distortion.
24

---

Published as a conference paper at ICLR 2023
J.2.1 SETUP
Implementation We build the official pytorch implementation StarGAN-V2 (Choi et 2018 We set Auppee to 0.1. We add our regularization loss in the every latent and reference iteration.
Datasets and Metries. We use the benchmark AFHQ dataset (Choi et al. 2020) for image-to-image translation. It consists of three domains stated before. Unlike the image generation task, we now have 6 pair-wise image-to-mmage translation tasks since we have 3 domains: cat cat->dog, cat cat->wild, dog->cat, dog-willd, wild-<cat and wild->oog. We use the benchmark metrics: FID and LPIPS.
Baselines. We compare the results with MUNIT (Huang et al. (01 DRIT (Lee et al. 2018). MSGAN (Mao et al. 2019) and recent methods StarGANNV2 (Choi et al. (020), SmoothGAN (Liu ct al. 2021), LETIT (Zhao & Chen (02I) and StyleDis (Kim et al. 2022). We cite the results of MUNIT, DRIT, MSGAN and StarGAN-v2 from (Choi et al. (020). StyleDis only reports the FID of reference based task. But doesn't 't support the latent-based generation defined in StarGAN-v2.
25

---

Published as a conference paper at ICLR 2023
StyleGAN
TGAN
CoGAN
Ours (Asparse=0)
Ours
Figure 9: The t-SNE of real andgenerated samples for MNIST7 dataset. We can find that the baseline methods fail to recover the joint distribution. By contrast, our method matches the joint distributions.
26

---

Published as a conference paper at ICLR 2023
StyleGAN2-ADA
TGAN
CoGAN
==
A=
2
Figure CelebA-HQ. Without the sparsity regularization, i.e., O, we observe some unnecessary changes between the image tuples in each row. For example, t.g.,the added sun un-glasses and skin color change in the first row. TGAN changes the background (first row of third panel). CoGAN changes the skin color (second row, second panel).
27

---

Published as a conference paper at ICLR 2023
StyleGANZ-ADA
TGAN
CoGAN
==
A =01.1
Figure 11: AFHQ. StyleGAN2-ADA changes animal poses in many examples, e.g. second and third row of first panel. Our base (A 0) also changes the poses, e.g., first and third row of second panel. CoGAN and TGAN are slightly better in preserving poses but we can observe that some generated images are unrealistic. For example, the wolf (first row, third panel of TGAN) and the dog (third row, third panel of CoGAN).
28

---

Published as a conference paper at ICLR 2023
StyleGAN2-ADA
TGAN
CoGAN
==0
Figure 12: ArtPhoto. This case is probably the most challenging since the content is not aligned. We can observe many changes between the images of the generated tuples of the basclines method, StyleGAN2-ADA, TGAN, CoGAN and our base
29

---

Published as a conference paper at ICLR 2023
StyleGAN2-ADA
TGAN
CoGAN
-=
A = 0.1
Figure 13: Celeba5. StyleGAN2-ADA suffers the mode collapse (samples in the columns identical). TGAN and our base (A = 0) often changes the background.
30

---

Published as a conference paper at ICLR 2023
StyleGAN2-ADA
TGAN
CoGAN
X = 4 8 9
0
A
=
0.1
9
5
3
5
C
2
P
2
B
9
9
6
6
6
6
6
5
9
3
5
&
&
&
7
9
9
3
3
8
9
3
9
3
5
6
6
8
8
9
9
9
9
9
9
9
3
5
4
4
8
F Y.
8
9
f
9
3
3
1
3
9
9
3
5
5
8
P
6
2 A.
9
9
9
f
f
9
4
4
4
4
4
9
5
Y
9
7
9
9
9
9
9
9
3
5
2
4
2
7
5
9
ç ç ç
7
7
7
7
7
9
9
3
5
8
2
5
9 9
9
4
4
4
4
4
4 6
9
9
3
5
7
6
6
9
9
3
5
9
7
7
7
7
5
9
3
5
5
6
B
9
9
9
1
1
7
/
5
9
3
5
0
9
8
7
3
3
3
3 9 2
9
5
B
9
4
4
4
9 7
9
9
3
5
5
L
S
B
9 9
Ç 9
2
2
9
9
3
5
5
4
9
9
9
9
3
5
4
9
8
8
8
9
9
3
5
8 9 4 8 9 8 9 Ç ç
f J
9
4
4
4
4
4
9
9
3
5
9
9
9
8
9
9
9
3
5
6
B B B
7
3
3
3
3
3
5
5
3
5
6
6
P
9
7
7 5
7
7
9
9
3
5
2
9 9
5
5
3
3
5
9
3
5
2
ç 5
9
3
3
3
9
9 1 6
9
9
3
5
4
9
9
9
3
5
S
B 9
F
9
6
4
6
6
6
5
5
3
5
0
8
9 9
ç
9
Y
4
8
4
8
8
g
5
9
3
5
5
B B 5
7
7
7
7
7
17
5
3
5
5
6
5
5
9
9
f 5
5
4
4
4
4
4
4 3
5
5
3
5
4
2
9 9
5
9
3
3
3
3
3
5
5
3
5
2
C 5
5
3
5
S
9
9
3
5
&
8
9
4
4
9
9
9
3
5
2
7
8 9
f
f
f
5
3
3
3
3
3
3
5
9
3
&
2
9 4
F
5
1
1 3 1
9
3
5
5
S
P
5
8
f
7
3
3
3
3
3
9
9
3
5
S
2
8 9
f
5
/
/
/
/
5
5
3
5
4
2
9
ç
3
5
5
3
Figure 14: MNIST7. All baselines seem to suffer the mode collapse issue while TGAN seems to be affected by the augmentation in StyleGAN2-AD4 negatively.
31

---

Published as a conference paper at ICLR 2023
293
Version1 Version2
Version1 Version2
221
Dimension of
200
175-
153
200
125
100
50
72
Training Progress
Training Progress
Figure 15: The trend of dimension of 2, training proceeds on MNIST7 dataset. We observe that the sigmoid version decreases much slower than our proposed version when we set 0.1. Even if we set À = 1.0, the speed of decreasing the dimension 2, is smaller than our version. However, decreasing the dimension of 2, very important at the beginning phase as we need reduce the influence of domain variable to avoid conditional collapse. The sigmoid version decreases faster than our proposed method the middle of training progress. The reason is that the GAN already collapse and GAN loss is close to 0 and it helps little in selecting the optimal dimension n.
225
Version1
Version1
250
Version2
Version2
Dimension Zs
200
13.5
100
15.0
50
Training Progress
Training Progress
Figure 16: The trend of dimension of 2, as training proceeds on AFHQ dataset.
32

---

Published as a conference paper at ICLR 2023
Input
Star G2
Ours
Figure 17: BeeentBBaeed:: cat cat->dog and cat cai-
Input
G2
Ours
Figure 18: BeenttBased:: dog and dwild
33

---

Published as a conference paper at ICLR 2023
Input
Ours
Figure 19: Latent-Bass Bastt- wild->cat and wild wild->dog
Input
Reference
StarGAN-V2
Ours
Figure 20: Reference- Seference-Based:: cat->dog and cat->wild
34

---

Published as a conference paper at ICLR 2023
Input
Reference
StarGAN-V2
Ours
Figure 21: Reference-Barte Baseed: dog->cat and dog-
Input
Reference
Star StarGAN-o V2
Ours
Figure 22: Reference- BBasee: wild->cat and wild->dog
35


Reviews:
[{"review_id": "yMrHZYVjyF", "reviewer": "Reviewer_1JgB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper leverages recent theoretical developments in nonlinear ICA to introduce a novel model for multi-domain image generation and show that its joint distribution is identifiable from the marginal distributions. Motivated by this result, two practical algorithms are suggested, one for multi-domain image generation, which corresponds to sampling from the joint over domains, and one for image-to-image translation, which corresponds to sampling from a conditional. Both of these approaches are based on the assumptions that only a few latent variables are influenced by the domain label, which is implemented in the learned model via learned masks and regularization. Extensive experiments are performed, comparing multiple baselines on multiple datasets, which all point in the direction of improved matching between samples from the joint and better image-to-image generation.", "summary_of_the_review": "Even though I very much like the overall direction of this paper, (i) the overall lack of clarity, (ii) the lack of cohesion between the theory and the proposed methods and (iii) the lack of evidence for the usefulness of the masking mechanism prevent me from recommending acceptance. I am open to increase my  score if the authors address the main points I raised in my review (see Weaknesses section).", "strength_and_weaknesses": "Strengths:\n- I very much like the overall direction of this work. I believe leveraging recent identifiability results from the nonlinear ICA literature for the problem of multi-domain image generation and image-to-image translation is a very good idea, given how important the question of identifiability is to these problems. Bringing this more theory-driven mindset to this literature should benefit the community.\n- The experimental section appears fairly complete, to the eye of a non-expert in multi-domain image generation.\n\nWeaknesses:\n- Overall, some theoretical statements lack clarity. One finds many undefined notation which makes some of the statements impossible to understand. Also some mathematical statements lack precision. I expand on this in the \"Clarity\" section later in my review.\n- The connections between some sections could be improved, especially between the theoretical results and the suggested algorithms. For example, in Section 3.2, I believe there is a mismatch between equation (6) and how the ground-truth model (4) was specified: in the model (6) z_s = \\epsilon + f_u (\\epsilon), whereas in the ground-truth model, z_s = f_u(\\epsilon). This could be fixed in the model by doing z_s = (1 - m) * \\epsilon + m * f_u(\\epsilon). Is there a specific reason for not doing that? \n- Section 3.3 is very hard to understand. A lot of new notation is introduced such as H, E, F and D without proper explanations. The function F is called the “shared generator” and then called the “mapping function”, this makes everything hard to follow. Also, it is not so clear to me exactly what approximates p_\\theta(x^{(u_1)} | x^{(u_0)}). Overall, the connection with the theory from the previous section is not so clear to me.\n- The motivation for using the masking mechanisms instead of just treating n_s as an hyperparameter is weak. An experiment attempts to show that the penalty coefficient associated to the mask is easier to tune than n_s. But I found this experiment unconvincing, since a priori, it’s unclear what the range of \\lambda we should search over… The left top plot of Figure 4 shows a flat curve, but the curve would not look so flat had the range been different. How does one choose the proper range? All main experiments in the paper are performed with \\lambda = 0.1 and yields very good results. What happens when these experiments are run without this masking mechanism and with n_s = 16 or 32 (which were the best dimensionality found in the ablation study)? Do we see similar performance? \n- I couldn’t find information in the main text about how the mask m is learned, nor a reference to appendix. Is m = sigmoid(parameter) ? Or is it a Gumbel-Sigmoid?\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "vTuxeeMRBk", "reviewer": "Reviewer_spME", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on conditional image generation and image-to-image translation. Although current methods enable both tasks work well, they fail  to formulate suitable constraints for the joint distribution, since there can be infinitely many joint distributions that can derive the same marginals. Inspired by Independent Component Analysis (ICA) theory, authors propose a new regularization by enforcing a specific type of minimal changes across domains. The quantitative and qualitative results demonstrate the effectiveness of the proposed method.   ", "summary_of_the_review": "This paper sounds interesting, which first considers ICA into the training of both conditional image generation and image-to-image translation.  The main experiment  shows the effectiveness of the proposed method. ", "strength_and_weaknesses": "Although there are a few outstanding methods proposed to explore conditional image generation, this paper explores a new view, which is effective to remove unwanted joint distributions. \n\nThis looks solid which give strong theory support about how to guarantee the true joint distribution. \n\nThe paper is well-written, and easy to follow.\n\nAuthors provides  conditional image generation and image-to-image translation results, which sounds interesting.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "g9efhQYbJMf", "reviewer": "Reviewer_6m6y", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This method proposes a method for multi-domain image generation and unpaired image-to-image translation. The main challenge in these problems is learning a joint distribution from multiple marginal distributions is ill-posed, since there can be infinitely many joint distributions that can derive the same marginals.  Utilizing recent advances in nonlinear Independent Component Analysis (ICA) theory, the authors propose a new method to learn the joint distribution from the marginals. With the assumption that the influence of domain information is minimal in the data generation process, the method introduces the domain information through a component-wise monotonic transformation. Further, by assuming that the number of the underlying components affected by the domain information is minimal, the authors show that the true joint distribution can be recovered from the marginal distributions. Then the learned joint distribution can be used to sample meaningful tuples and translate input images to another domain without content distortion. First, the authors test the method on five multi-domain image generation datasets and evaluate using FID and domain-invariant perceptual distance (DIPD) metrics (for datasets with unpaired data.) On this dataset, the method obtains some improvement over baseline approaches, and especially shows good results on MNIST7, where other methods fail due to mode collapse. On the application of image-to-image translation, the method also obtains good results, outperforming baseline methods in both latent-based and reference-based settings. \n", "summary_of_the_review": "The paper proposes a method for multi-domain image generation and unpaired image-to-image translation, using an ICA-based method to learn the joint distribution from the marginals. The proposed method obtains decent improvement over baselines on standard datasets, however, performance gains are limited on complex images where the content across domains may not be aligned. ", "strength_and_weaknesses": "Strengths:\n1. The proposed approach shows pretty good improvement over the baseline methods, especially on the task of image-to-image translation. \n\n2. The ablation studies show the effect of different design choices, such as reducing the dimension of the style dimension or the effect of the number of style dimensions on generation quality. These are informative in understanding the effect of design choices on generation quality. \n\nWeaknesses/questions:\n1. The quality of writing in the paper is uneven, with some parts not written clearly, which makes the paper difficult to understand in parts. \n\n2. As the authors also allude to in the last section, the proposed method is not as effective when the content across domains is not aligned, such as the case of the ArtPhoto dataset (where the method shows the least improvement in FID). Can the authors comment the general applicability of the method in the light of this limitation, beyond standard constrained facial image and digit datasets? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IZ_bIhpl-gW", "reviewer": "Reviewer_h5sT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides an in-depth study of estimating the joint distribution of multi-domain image generation and unpaired image-to-image translation. To mitigate the highly ill-posed issues of mapping from marginal distributions to joint distribution, the authors provide an in-depth analysis for the distribution modeling, while their theory is based on some tight assumptions. Besides, the paper implements this theory in a cheap way. Specifically, the paper empirically demonstrates the disentangled content and style controllable generator through a simple framework and loss function. Then, based on this architecture and new regularization, the paper improves the performance of multi-domain image generation and unpaired image-to-image translation on multiple datasets. ", "summary_of_the_review": "This paper investigates an important topic of unified multi-domain image generation and unpaired image-to-image translation task. Despite the experimental part could be further enriched to clearly demonstrated the better joint distribution learned by the proposed method, the easily achieved implement and thorough analyses on the properties provides many insights and might arouse following researchers. Therefore, my rating for this submission is positive.", "strength_and_weaknesses": "#### **Strengths**\n- There are many works using the StyleGAN or StarGAN architectures to perform the image generation and translation tasks but without in-depth analyses. This paper provides an in-depth study of the joint distribution for multi-domain generation and translation, which provdes some insights for the following researchers.\n- The implementation way is easily achieved, and the loss function is interesting. The corresponding impressive results are good enough to support the contribution claimed by the authors.\n- The paper is well organized and is easy to follow. Several observations are studied with quantitative and visual analyses, followed by the corresponding ablation study, making the paper easy to follow and concrete. \n\n#### **Weaknesses**\nWhile the proposed method is interesting and the experimental analyses are comprehensive, I believe some parts need more clarification (even after considering the supplementary material and code).\n- The joint distribution is the key motivation and novelty in the paper. However, the experiments just demonstrate the much better performance and the effectiveness of the provided framework. It will be better to visualize the distribution of the original dataset as well the generated results of different methods through the t-SNE, especially for MINIST7, which contains obvious domain and class category.\n-  The domain $u$ is predefined. As claimed by the authors, \"collecting corresponding data across domains can be prohibitively expensive\". Then, how could the authors define the domain numbers in a dataset?\n- More generally, for the CELEBA-HQ, the authors split it as female and male faces domains. Could we also add other domains, such as sad and happy, young and old and others? Does different domain numbers affect the performance?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%