Content-Style Learning from Unaligned Domains: Identifiability under Unknown Latent Dimensions

## Abstract

Abstract Understanding identifiability of latent content and style variables from unaligned multi-domain data is essential for tasks such as
domain translation and data generation.
Existing works on content-style identification were often developed under somewhat stringent conditions, e.g., that all latent components are mutually independent and that the dimensions of the content and style variables are known.
We introduce a new analytical framework via cross-domain latent distribution matching (LDM), which establishes content-style identifiability under substantially more relaxed conditions. Specifically, we show that restrictive assumptions such as component-wise independence of the latent variables can be removed. Most notably,
we prove that prior knowledge of the content and style dimensions is not necessary for ensuring identifiability, if sparsity constraints are properly imposed onto the learned latent representations. Bypassing the knowledge of the exact latent dimension has been a longstanding aspiration in unsupervised representation learning—our analysis is the first to underpin its theoretical and practical viability. On the implementation side, we recast the LDM formulation into a regularized multi-domain GAN loss with coupled latent variables. We show that the reformulation is equivalent to LDM under mild conditions—yet requiring considerably less computational resource. Experiments corroborate with our theoretical claims.

## 1 Introduction

In multi-domain learning, “domains” are typically characterized by a distinct “style" that sets their data apart from others Choi et al. (). Take handwritten digits as an example: writing styles of different persons can define different domains. Shared information across all domains, such as the identities of the digits in this case, is termed as “content".
Learning content and style representations from multi-domain data facilitates many important applications, e.g., domain translation (Huang et al., ), image synthesis (Choi et al., ), and self-supervised representation learning (Von Kügelgen et al., ; Lyu et al., ; Daunhawer et al., ); see more in Huang et al. (); Lee et al. (); Choi et al. (); Wang et al. (); Yang et al. (); Wu et al. ().

Recent advances showed that
understanding the identfiability of the latent content and style components from multi-domain data allows to design more reliable, predicable, and trustworthy learning systems
(Hyvarinen et al., ; Lyu et al., ; Xie et al., ; Kong et al., ; Shrestha & Fu, ; Gresele et al., ; Gulrajani & Hashimoto, ).
A number of works studied content/style identifiability when the multi-domain data have sample-to-sample cross-domain alignment according to shared contents.
Specifically, identifiability was established for sample-aligned multi-domain settings under the assumption that multi-domain data are linear and nonlinear mixtures of latent content and style components, in the context of canonical correlation analysis (CCA), multiview analysis and self-supervised learning (SSL); see Ibrahim et al. (); Sørensen et al. (); Wang & Isola (); Von Kügelgen et al. (); Lyu et al. (); Karakasis & Sidiropoulos (); Daunhawer et al. ()

When cross-domain samples are unaligned, it becomes significantly more challenging to establish identifiability of the content and style components.
The recent works in Xie et al. (); Sturma et al. (); Kong et al. (); Timilsina et al. () made meaningful progresses towards this goal. These works considered mixture models of content and style for each domain, similar to those in Lyu et al. (); Von Kügelgen et al. (); Ibrahim et al. (); Sørensen et al. (); Karakasis & Sidiropoulos (); Daunhawer et al. (), but without cross-domain alignment.
The new results in (Xie et al., ; Sturma et al., ; Kong et al., ; Timilsina et al., )
provide theory-backed solutions to a suite of timely and important applications, e.g., cross-language retrieval, multimodal single cell data alignment, causal representation learning, and image data translation and generation.

Challenges.
The content-style identifiability results in existing unaligned multi-domain learning works are intriguing and insightful, but some challenges remain.
First, the conditions used in their proofs have a number of restrictions, which limits the proof’s applicability in many cases.
For example, Sturma et al. (); Timilsina et al. () assume that the all data reside in a linear subspace, which is over-simplification of reality; Xie et al. (); Kong et al. () assume that the content and style variables are component-wise independent and that a large number of domains exist—both can be hard to fulfil.
Second, the existing identifiability analyses in unaligned multi-domain learning (Xie et al., ; Kong et al., ; Sturma et al., ; Timilsina et al., ) (as well as those in aligned multi-domain learning) all need to know the dimensions of the content and style variables, which are not available in practice. Selecting these dimensions often involves extensive trial and error.

Contributions.
In this work, we advance the analytical and computational aspects of content-style learning from unaligned multi-domain data. Our detailed contributions are as follows:

(i) Enhanced Identifiability of Content and Style: We propose a content-style identification criterion via constrained latent distribution matching (LDM). We show that the identifiability conditions under LDM are much more relaxed relative to those in existing works. Specifically,
our results hold for nonlinear mixture models, as opposed to the linear ones used in Sturma et al. (); Timilsina et al. (). Unlike Xie et al. (); Kong et al. (); Sturma et al. (),
no elementwise mutual independence assumption is needed in our proof. More importantly, our result holds for as few as two domains (whereas Xie et al. (); Kong et al. () needs the existence of a large number of domains).
The new results widens the applicability of content-style identifiable models in a substantial way.

(ii) Content-Style Identifiability under Unknown Latent Dimensions:
We consider the scenario where the latent content and style dimensions are unknown—which is the case in practical settings.
Note that existing works determine the content and style dimensions often by heuristics, e.g., trial-and-error. However, wrongly selected latent dimensions can largely degrade the performance of some tasks; e.g., an over-estimated style dimension hinders the diversity of data in generation tasks (see Sec. 6).
We show that, by imposing proper sparsity constraints onto the LDM formulation, the content-style identifiability is retained even without knowing the exact latent dimensions.
To our knowledge, this result is the first of the kind in the context of nonlinear mixture identification.

(iii) Efficient Implementation: We prove that the LDM formulation is equivalent to a sparsity-constrained, latent variable-coupled muti-domain GAN loss, under reasonable conditions. Directly realizing the LDM formulation would impose multiple complex modules, including the DM and content-style separation modules, in the learned latent domain.
Simultaneously learning the latent space and optimizing these modules can be computationally involved.
The GAN-based formulation circumvents such complicated operations and thus substantially simplifies the implementation.

For theory validation, we perform experiments over a series of image translation and generation tasks.

Notation. Please see Appendix A.1 for detailed notation designation. A particular remark is that $\mathbb{P}_{\boldsymbol{x}}$ and $p_{\boldsymbol{x}}(\cdot)$ represent the probability measure of $\boldsymbol{x}$ and the probability density function (PDF) of $\boldsymbol{x}$, respectively. The “push forward” notation $[\boldsymbol{f}]_{\#\mathbb{P}_{\boldsymbol{x}}}$ means the distribution of $\boldsymbol{f}(\boldsymbol{x})$.

## 2 Background

Content-Style Modeling in Multi-Domain Analysis.
Consider the case where the data are acquired over $N$ domains ${\cal X}^{(n)}\subseteq\mathbb{R}^{d}$, where $n=1,\ldots,N$. We assume that any sample from domain $n$ can be represented as a function (or, a nonlinear mixture) of content and style components, i.e.,

$$ $\displaystyle\boldsymbol{c}\sim\mathbb{P}_{\boldsymbol{c}},~{}\boldsymbol{s}^{ (n)}\sim\mathbb{P}_{\boldsymbol{s}^{(n)}},~{}\boldsymbol{x}^{(n)}=\boldsymbol{ g}(\boldsymbol{c},\boldsymbol{s}^{(n)}),$ (1) $$

where $\mathbb{P}_{\boldsymbol{s}^{(n)}}$ and $\mathbb{P}_{\boldsymbol{c}}$ are distributions of the style components in $n$th domain and the content components, respectively. Let $\mathcal{C}\subseteq\mathbb{R}^{d_{\rm C}}$ and $\mathcal{S}^{(n)}\subseteq\mathbb{R}^{d_{\rm S}}$ be the open set supports of $\mathbb{P}_{\boldsymbol{c}}$ and $\mathbb{P}_{\boldsymbol{s}^{(n)}}$.
Then, we define $\mathcal{X}^{(n)}=\{\boldsymbol{g}(\boldsymbol{c},\boldsymbol{s}^{(n)})|
\boldsymbol{c},\boldsymbol{s}^{(n)}\in\mathcal{C}\times\mathcal{S}^{(n)}\}
\subseteq\mathbb{R}^{d}$ as the support of $\boldsymbol{x}^{(n)}\sim\mathbb{P}_{\boldsymbol{x}^{(n)}}$.
Let $\mathcal{X}=\cup_{n=1}^{N}\mathcal{X}^{(n)}\subseteq\mathbb{R}^{d}$ and ${\cal S}=\cup_{n=1}^{N}\mathcal{S}^{(n)}\subseteq\mathbb{R}^{d_{\rm S}}$ represent the whole data space and the whole style space, respectively.
We assume that the nonlinear function $\boldsymbol{g}:\mathcal{C}\times\mathcal{S}\to\mathcal{X}$ is a differentiable bijective function.
Note that although $\mathcal{X}\subseteq\mathbb{R}^{d}$ and $d$ might be greater than $d_{\rm S}+d_{\rm C}$, the bijective property can hold as $\mathcal{X}$ resides within a low dimensional manifold (Von Kügelgen et al., ).

Figure: Figure 1: Cross-domain translation from source domain $s$ to target domain $t$.
Refer to caption: x1.png

The model in (1) is widely adopted (explicitly or implicitly) in multi-domain analysis; see examples from Huang et al. (); Lee et al. (); Choi et al. (); Wang et al. (); Yang et al. (); Wu et al. (). This model makes sense when the “domains” are participated using distinguishable semantic meaning; e.g., in Fig. 1, “style” includes the writing manners (handwritten/printed) and display background colors (black/gray).
Under the model in (1), learning $\boldsymbol{g}$ (and its inverse $\boldsymbol{f}$) as well as the latent components $\boldsymbol{c}$ and $\boldsymbol{s}^{(n)}$ is the key to facilitate a number of important applications.

Application: Cross-Domain Translation. Learning content and style components from a sample in the source domain $(\boldsymbol{c}_{i},\boldsymbol{s}_{i}^{(s)})=\boldsymbol{f}(\boldsymbol{x}_{i
}^{(s)})$ and a sample from the target domain $(\boldsymbol{c}_{j},\boldsymbol{s}_{j}^{(t)})=\boldsymbol{f}(\boldsymbol{x}_{j
}^{(t)})$ can assist translate $\boldsymbol{x}_{i}^{(s)}$ to its corresponding representation in the target domain. This can be realized by generating a new sample
$\boldsymbol{x}^{(s\rightarrow t)}_{i,j}=\boldsymbol{g}(\boldsymbol{c}_{i},
\boldsymbol{s}_{j}^{(t)});$
see Fig. 1 for illustration and Lyu et al. (); Huang et al. (); Wang et al. ().

Application: Data Generation.
If $\boldsymbol{c}$ and $\boldsymbol{s}^{(n)}$ can be learned from the samples, then one can also learn the distributions $\mathbb{P}_{\boldsymbol{c}}$ and $\mathbb{P}_{\boldsymbol{s}^{(n)}}$ using off-the-shelf distribution learning tools, e.g., GAN (Goodfellow et al., ).
This way, one can draw samples from the distributions, i.e.,
$\boldsymbol{c}_{\rm new}\sim\mathbb{P}_{\boldsymbol{c}},~{}\boldsymbol{s}^{(n)
}_{\rm new}\sim\mathbb{P}_{\boldsymbol{s}^{(n)}}$
and generate new samples $\boldsymbol{x}_{\rm new}^{(n)}=\boldsymbol{g}(\boldsymbol{c}_{\rm new},
\boldsymbol{s}^{(n)}_{\rm new})$ with intended styles.

Other Applications. We should mention that the content-style modeling is also a critical perspective for understanding representation learning paradigms, e.g., the SSL frameworks (Von Kügelgen et al., ; Lyu et al., ; Daunhawer et al., ; Wang & Isola, ).

Content-Style Identifiability.
In recent years, the identifiability of $\boldsymbol{f}$, $\boldsymbol{c}$ and $\boldsymbol{s}^{(n)}$ started drawing attention, due to its usefulness in building more reliable/predictable systems.

Aligned Domains: Results from Self-Supervised Learning (SSL).
The works (Von Kügelgen et al., ; Daunhawer et al., ; Lyu et al., ; Karakasis & Sidiropoulos, ) studied content identifiability in the context of representation learning, in particular, SSL and multiview learning.
It was shown that when $N=2$, if content-shared pairs $\{\boldsymbol{x}^{(1)},\boldsymbol{x}^{(2)}\}$ are available, then enforcing $\boldsymbol{f}(\boldsymbol{x}^{(1)})=\boldsymbol{f}(\boldsymbol{x}^{(2)}),~{}
\forall~{}\text{content-shared pairs}~{}(\boldsymbol{x}^{(1)},\boldsymbol{x}^{
(2)})$ can provably learn $\boldsymbol{c}$,
under reasonable conditions.
The learning criterion can be realized by various loss functions, e.g., Euclidean fitting-based (Lyu et al., ; Karakasis & Sidiropoulos, ) and contrastive loss-based (Von Kügelgen et al., ; Daunhawer et al., ) criteria.
The identifiability of the style components was also considered under similar aligned domain settings; see (Lyu et al., ; Eastwood et al., ).

Unaligned Domains: Progresses and Challenges.
Aligned samples are readily available in applications such as data-augmented SSL (Von Kügelgen et al., ; Daunhawer et al., ; Lyu et al., ). However, in other applications such as image style translation and image generation, aligned samples are hard to acquire (Zhu et al., ).
For unaligned multi-domain data, the identfiiability issue of content and style has also been recently addressed.
For example, the work of Sturma et al. () extended the linear ICA model to unaligned multi-domain settings, in the context of causal learning.
The work of Timilsina et al. () took a similar linear mixture model but showed content-style identifiability under more relaxed conditions.
The work of Xie et al. (); Kong et al. () proved content-style identifiability under a more realistic nonlinear mixture model similar to that in (1).
However, the main result there relies on a number of somewhat stringent conditions. That is, two notable assumptions in Xie et al. (); Kong et al. () boil down to (i) that all components in ${\boldsymbol{z}}=(\boldsymbol{c},\boldsymbol{s}^{(n)})$ are elementwise statistically independent given the domain index $n$; and (ii) that there exist at least $2d_{\rm S}+1$ domains. These conditions can be hard to fulfil. See more detailed discussions on existing results in Appendix B.

The Dimension Knowledge Challenge.
Notably, all the existing works in this domain (under both aligned and unaligned settings) assume that the dimensions of $\boldsymbol{c}$ and $\boldsymbol{s}^{(n)}$ are known.
However, in mixture model learning, such knowledge is hard to acquire (especially in the nonlinear mixture case).
As we will show, using wrongly selected $d_{\rm C}$ and $d_{\rm S}$ can be rather detrimental to content-style learning tasks—e.g., an over-estimated style dimension could lead to a serious lack of diversity in generated new samples.
Consequently, the dimensions are often selected by extensive trial and error in practice.

## 3 Main Result

In this work,
we revisit content-style learning problem from a latent distribution matching (LDM) viewpoint.
Consider the following assumption:

###### Assumption 3.1 (Block Independence) .

The block variables $\boldsymbol{c}$ and $\{\boldsymbol{s}^{(n)}\}_{n=1}^{N}$ are statistically independent, i.e., $p(\boldsymbol{c},\boldsymbol{s}^{(1)},\ldots,\boldsymbol{s}^{(N)})=p_{
\boldsymbol{c}}(\boldsymbol{c})\prod_{n=1}^{N}p_{\boldsymbol{s}^{(n)}}(
\boldsymbol{s}^{(n)})$.

The assumption was used in various multi-domain models (Lyu et al., ; Eastwood et al., ; Wang et al., ; Choi et al., ; Timilsina et al., ). It makes sense when the styles can be combined with contents in an “arbitrary” way without affecting the contents (e.g., the writing style of digits can change freely without affecting the identity of the digits). Next, we will use this assumption to build our learning criterion.

We propose the following learning criterion:

$$ find $\displaystyle\boldsymbol{f}:\mathcal{X}\rightarrow\mathbb{R}^{d_{\rm C}+d_{\rm S }}~{}\text{injective}$ s.t. $\displaystyle[\boldsymbol{f}_{\rm C}]_{\#\mathbb{P}_{\boldsymbol{x}^{(i)}}}=[ \boldsymbol{f}_{\rm C}]_{\#\mathbb{P}_{\boldsymbol{x}^{(j)}}},~{}i\neq j, \forall i,j\in[N],~{}\text{(distribution matching)}$ (2a) $\displaystyle[\boldsymbol{f}_{\rm S}]_{\#\mathbb{P}_{\boldsymbol{x}^{(n)}}} \rotatebox[origin={c}]{90.0}{$\models$}[\boldsymbol{f}_{\rm C}]_{\#\mathbb{P}_ {\boldsymbol{x}^{(n)}}},\forall n\in[N],~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }\text{(block-indep. enforcing)}$ (2b) $$

where $\boldsymbol{f}_{\rm C}(\boldsymbol{x}^{(n)})\in\mathbb{R}^{d_{\rm C}}$ represents the first $d_{\rm C}$ outputs of $\boldsymbol{f}$ that are designated to represent the content components, $\boldsymbol{f}_{\rm S}(\boldsymbol{x}^{(n)})\in\mathbb{R}^{d_{\rm S}}$ represents the learned style from domain $n$, Eq. (2a) matches the distributions of $\boldsymbol{f}_{\rm C}(\boldsymbol{x}^{(i)})$ and $\boldsymbol{f}_{\rm C}(\boldsymbol{x}^{(j)})$—i.e.,
the learned contents from domains $i$ and $j$, respectively—and Eq. (2b) imposes a block independence constraint on the learned content $\boldsymbol{f}_{\rm C}(\boldsymbol{x}^{(n)})$ and style $\boldsymbol{f}_{\rm S}(\boldsymbol{x}^{(n)})$ from each domain following Assumption (3.1).

###### Assumption 3.1 (Block Independence) .

### 3.1 Warm-up: Enhanced Identifiability with Known Latent Dimensions

We first show that the content-style identifiability under (1) and known $d_{\rm C}$ and $d_{\rm S}$ can be substantially enhanced relative to existing works. We will remove the need for the dimension knowledge in the next subsection.
To establish identifiability via solving (2), we make the following assumption:

###### Assumption 3.2 (Domain Variability) .

Let $\mathcal{A}\subseteq\mathcal{Z}:=\mathcal{C}\times\mathcal{S}$ be any measurable set that satisfies (i) $\mathbb{P}_{\boldsymbol{z}^{(n)}}[\mathcal{A}]>0$ for any $n\in[N]$ and (ii) ${\cal A}$ cannot be expressed as ${\cal B}\times{\cal S}$ for any set ${\cal B}\subset{\cal C}$. Then, there exists a pair of $i_{\mathcal{A}},j_{\mathcal{A}}\in[N]$ such that the following holds:

$$ $\displaystyle\mathbb{P}_{\boldsymbol{z}^{(i_{\mathcal{A}})}}[\mathcal{A}]\neq \mathbb{P}_{\boldsymbol{z}^{(j_{\mathcal{A}})}}[\mathcal{A}],.$ (3) $$

Note that for any ${\cal A}$, we only need one pair of $(i_{\cal A},j_{\cal A})$ to satisfy the condition, and the pair can change over different ${\cal A}$’s.
Essentially, Eq. (3) requires that the styles have sufficiently diverse distributions.
This assumption is a standard characterization for the distributional diversity of the domains in the literature; see Xie et al. (); Kong et al. () and its variant Timilsina et al. ().

Under Assumptions 3.1 and (3.2),
denote $\widehat{\boldsymbol{f}}$ as a solution to Problem (2).
Then, we have:

###### Theorem 3.3 (Content-Style Identifiability under Known Latent Dimensions) .

Under Eq. (1), suppose
that Assumptions 3.1 and 3.2 hold, and that the $\widehat{\boldsymbol{f}}$ is differentiable.
Then, we have $\widehat{\boldsymbol{f}}_{\rm C}(\boldsymbol{x}^{(n)})=\boldsymbol{\gamma}(
\boldsymbol{c})$ and $\widehat{\boldsymbol{f}}_{\rm S}(\boldsymbol{x}^{(n)})=\boldsymbol{\delta}(
\boldsymbol{s}^{(n)}),\forall n\in[N]$, where $\boldsymbol{\gamma}:\mathcal{C}\rightarrow\mathbb{R}^{d_{\rm C}}$ and $\boldsymbol{\delta}:\mathcal{S}\rightarrow\mathbb{R}^{d_{\rm S}}$ are injective functions.

Theorem 3.3 purports that the solution of (2) identifies the model (1)—including the content/style components and the inverse mapping of the generative function $\boldsymbol{g}$ (up to $\boldsymbol{\gamma}$ and $\boldsymbol{\delta}$). Theorem 3.3 uses conditions that are significantly more relaxed relative to those in existing works Xie et al. (); Sturma et al. (); Kong et al. (); Timilsina et al. ().
First, instead of assuming the elements of $\boldsymbol{z}^{(n)}=(\boldsymbol{c},\boldsymbol{s}^{(n)})$ are statistically independent as in Xie et al. (); Sturma et al. (); Kong et al. (), our proof is based on
the assumption that the content and styles are block independent (cf. Assumption 3.1).
This block-independence assumption is similar to those in Lyu et al. () and Timilsina et al. ()—but the former assumes aligned domains and the latter can only work under linear mixture models (see Theorem B.2 in Appendix B.2).
Second, Theorem 3.3 does not need the existence of $N=2d_{\rm S}+1$ domains as in Xie et al. (); Kong et al. () (see Theorem B.3 in Appendix B.3)—our result can hold over as few as $N=2$ domains.
As a result, our new content-style identifiability result in Theorem 3.3 applies to a considerably wider range of cases relative to those in existing works.

###### Assumption 3.2 (Domain Variability) .

###### Theorem 3.3 (Content-Style Identifiability under Known Latent Dimensions) .

### 3.2 Main Result: Identifiability without Dimension Knowledge

Theorem 3.3 still uses the knowledge of $d_{\rm C}$ and $d_{\rm S}$. In this subsection, we propose a modifed learning criterion that does not use the exact dimension information.
To proceed, let $\widehat{d}_{\rm C}$ and $\widehat{d}_{\rm S}$ denote the user-specified latent dimensions for $\boldsymbol{f}$, i.e., $\boldsymbol{f}:\mathcal{X}\to\mathbb{R}^{\widehat{d}_{\rm C}+\widehat{d}_{\rm S}}$, $\boldsymbol{f}_{\rm C}:\mathcal{X}\to\mathbb{R}^{\widehat{d}_{\rm C}}$ and $\boldsymbol{f}_{\rm S}:\mathcal{X}\to\mathbb{R}^{\widehat{d}_{\rm S}}$. Note that these dimensions need not to be exact. We consider the following learning criterion:

$$ $\displaystyle\operatorname*{\textrm{minimize}}_{\boldsymbol{f}:~{}\text{ injective}}$ $\displaystyle\quad\sum_{n=1}^{N}\mathbb{E}\left[\left\|\boldsymbol{f}_{\rm S} \left(\boldsymbol{x}^{(n)}\right)\right\|_{0}\right]$ (4a) subject to $\displaystyle[\boldsymbol{f}_{\rm C}]_{\#\mathbb{P}_{\boldsymbol{x}^{(i)}}}=[ \boldsymbol{f}_{\rm C}]_{\#\mathbb{P}_{\boldsymbol{x}^{(j)}}},\forall i,j\in[N],$ (4b) $\displaystyle[\boldsymbol{f}_{\rm S}]_{\#\mathbb{P}_{\boldsymbol{x}^{(n)}}} \rotatebox[origin={c}]{90.0}{$\models$}[\boldsymbol{f}_{\rm C}]_{\#\mathbb{P}_ {\boldsymbol{x}^{(n)}}},\forall n\in[N],$ (4c) $$

Problem (4) minimizes the “effective dimension” of the extracted style component, while satisfying the distribution matching and independence constraints.
The idea is to use excessive $\widehat{d}_{\rm C}$ and $\widehat{d}_{\rm S}$ so that one has enough dimensions to represent the content and style information.
Note that trivial solutions could occur when using over-estimated $\widehat{d}_{\rm C}$ and $\widehat{d}_{\rm S}$. For instance, when $\boldsymbol{f}_{\rm C}$ is a constant function, $\boldsymbol{f}_{\rm S}$ can still be an injective function of $\boldsymbol{x}^{(n)}$ given large enough $\widehat{d}_{\rm S}$. This pathological solution satisfies both constraints (4b) and (4c).
We use the sparsity objective in (4a) to “squeeze out” the redundant dimensions in $\boldsymbol{f}_{\rm S}$. This prevents the content information from “leaking” into the learned $\boldsymbol{f}_{\rm S}$. We formalize this intuition in the following theorem:

###### Theorem 3.4 (Identifiability without Dimension Knowledge) .

Assume that the conditions in Theorem 3.3 hold. Let $\widehat{\boldsymbol{f}}$ represent a solution of Problem (4) and $\widehat{\boldsymbol{f}}$ is differentiable.
Assume the following conditions hold: (a) $\widehat{d}_{\rm C}\geq d_{\rm C}$ and $\widehat{d}_{\rm S}\geq d_{\rm S}$.
(b) $0<p_{\boldsymbol{z}^{(n)}}(\boldsymbol{z})<\infty,\forall\boldsymbol{z}\in
\mathcal{Z}=\mathcal{C}\times\mathcal{S},\forall n\in[N]$.
Then, there exists injective functions $\boldsymbol{\gamma}:\mathcal{C}\to\mathbb{R}^{\widehat{d}_{\rm C}}$ and $\boldsymbol{\delta}:\mathcal{S}\to\mathbb{R}^{\widehat{d}_{\rm S}},\forall n
\in[N]$ such that $\widehat{\boldsymbol{f}}_{\rm C}(\boldsymbol{x}^{(n)})=\boldsymbol{\gamma}(
\boldsymbol{c})$ and $\widehat{\boldsymbol{f}}_{\rm S}(\boldsymbol{x}^{(n)})=\boldsymbol{\delta}(
\boldsymbol{s}^{(n)}),\forall n\in[N]$.

Theorem 3.4 means that using (4), there is no need to know $d_{\rm S}$ or $d_{\rm C}$ in advance.
The identifiability result has significant practical implications for content-style identification, where the latent dimension in practice is always hard to acquire.

###### Theorem 3.4 (Identifiability without Dimension Knowledge) .

## 4 Implementation: Sparsity-Regularized Multi-Domain GAN

At first glance, a conceptually straightforward realization of the learning criterion in (2) could take the following form:

$$ $\displaystyle\operatorname*{\textrm{minimize}}_{\boldsymbol{f}:~{}{\rm injective }}~{}\sum_{i=1}^{N}\sum_{j>i}^{N}{\cal L}_{\rm DM}(\boldsymbol{f}_{\rm C}( \boldsymbol{x}^{(i)}),\boldsymbol{f}_{\rm C}(\boldsymbol{x}^{(j)}))+\lambda \sum_{i=1}^{N}{\cal L}_{\rm indep}(\boldsymbol{f}_{\rm C}(\boldsymbol{x}^{(i)} ),\boldsymbol{f}_{\rm S}(\boldsymbol{x}^{(i)})),$ (5) $$

where the first term and the second term promotes the distribution matching (DM) constraint (2a) and the independence constraint (2b), respectively. This type of implementation is potentially viable but can be costly: Both the LDM modules and the block independence regularization on the learned components often needs rather nontrivial optimization (see (Lyu et al., )).
Enforcing $\boldsymbol{f}$ to be injective also needs extra regularization, e.g., autoencoder type regularization (Lyu et al., ; Zhu et al., ) and entropy-type regularization (Von Kügelgen et al., ; Daunhawer et al., ).

Instead of using the straightforward implementation as in (5), we propose a unified formulation for (2) and (4) using GAN learning:

$$ $\displaystyle\min_{\boldsymbol{q},\boldsymbol{e}_{\rm C},\boldsymbol{e}_{\rm S }}\max_{\boldsymbol{d}^{(n)}}$ $\displaystyle~{}\sum_{n=1}^{N}{\mathbb{E}}\left[\log\left(\boldsymbol{d}^{(n)} \left(\boldsymbol{x}^{(n)}\right)\right)+\log\left(1-\boldsymbol{d}^{(n)}\left (\boldsymbol{q}\left(\boldsymbol{e}_{\rm C}(\boldsymbol{r}_{\rm C}), \boldsymbol{e}_{\rm S}^{(n)}(\boldsymbol{r}^{(n)}_{\rm S})\right)\right)\right )\right]$ (6a) $\displaystyle{\rm subject~{}to}$ $\displaystyle~{}\boldsymbol{e}_{\rm S}^{(n)}(\boldsymbol{r}_{\rm S}^{(n)})~{} \text{has minimal}~{}\|\boldsymbol{e}_{\rm S}^{(n)}(\boldsymbol{r}_{\rm S}^{(n )})\|_{0},~{}\forall\boldsymbol{r}_{\rm S}^{(n)}.$ (6b) $$

The above approximates (2) and (4) when the constraint (6b) is absent and active, respectively. In practice, the sparsity constraint can be approximated using sparsity regularization terms (e.g., $\ell_{1}$ norm) easily.
Denote $\widehat{d}_{\rm C}$ and $\widehat{d}_{\rm S}$ are the estimates of $d_{\rm C}$ and $d_{\rm S}$, respectively.
The idea is to learn invertible nonlinear mappings $\boldsymbol{e}_{\rm C}$ and $\boldsymbol{e}_{\rm S}^{(n)}$ that transform independent Gaussian variables (i.e., $\boldsymbol{r}_{\rm C}$ and $\boldsymbol{r}_{\rm S}^{(n)}$) to represent content $\boldsymbol{c}$ and style $\boldsymbol{s}^{(n)}$, respectively.
Generate $\boldsymbol{r}_{\rm C}\sim{\cal N}(\boldsymbol{0},\boldsymbol{I}_{\widehat{d}_
{\rm C}})$ and
construct an invertible $\boldsymbol{e}_{\rm C}$ such that $\boldsymbol{e}_{\rm C}(\boldsymbol{r}_{\rm C})\in\mathbb{R}^{\widehat{d}_{\rm C}}$. Similarly, construct invertible $\boldsymbol{e}_{\rm S}^{(n)}$ such that $\boldsymbol{e}_{\rm S}^{(n)}(\boldsymbol{r}_{\rm S}^{(n)})\in\mathbb{R}^{
\widehat{d}_{\rm S}}$ with $\boldsymbol{r}_{\rm S}^{(n)}\sim{\cal N}(\boldsymbol{0},\boldsymbol{I}_{
\widehat{d}_{\rm S}})$.
Then, the content and style are mixed by $\boldsymbol{q}$ to match the distribution of $\boldsymbol{x}^{(n)}$ using a logistic loss (i.e., GAN-type DM).
In other words, the formulation looks for $\boldsymbol{e}_{\rm C}$, $\boldsymbol{e}_{\rm S}^{(n)}$ and $\boldsymbol{q}$
such that
$\mathbb{P}_{\boldsymbol{x}^{(n)}}=\mathbb{P}_{\boldsymbol{q}^{(n)}},$ $\boldsymbol{q}^{(n)}=\boldsymbol{q}(\boldsymbol{e}_{\rm C}(\boldsymbol{r}_{\rm
C
}),\boldsymbol{e}_{\rm S}^{(n)}(\boldsymbol{r}_{\rm S}^{(n)})),$ $\forall n\in[N].$
This way, instead of directly learning $\boldsymbol{f}$, we learn the generative process $\boldsymbol{g}$ using $\boldsymbol{q}$. Our next theorem shows that $\boldsymbol{q}$ is indeed the inverse of $\boldsymbol{f}$ (up to some ambiguities).

To proceed, denote $\widehat{\mathcal{C}}$ and $\widehat{\cal S}^{(n)}$ as the sets representing the range of $\widehat{\boldsymbol{e}}_{\rm C}$ and $\widehat{\boldsymbol{e}}_{\rm S}^{(n)}$, respectively. Then, the effective domain of $\widehat{\boldsymbol{q}}$ is $\widehat{\mathcal{C}}\times\widehat{\mathcal{S}}$ where $\widehat{\mathcal{S}}=\cup_{n}\widehat{\mathcal{S}}^{(n)}$. We show that:

###### Theorem 4.1 .

Let $(\widehat{\boldsymbol{q}},\widehat{\boldsymbol{e}}_{\rm C},\widehat{
\boldsymbol{e}}^{(n)}_{\rm S},\widehat{\boldsymbol{d}})$ be any differentiable optimal solution of (6).
Let $\mathcal{C}$ and $\mathcal{S}$ be simply connected open sets. Let $0<p_{\boldsymbol{z}^{(n)}}(\boldsymbol{z})<\infty,\forall\boldsymbol{z}\in
\mathcal{Z}=\mathcal{C}\times\mathcal{S}$.
Under the assumptions in Theorem 3.3, we have the following:

(a) If $\widehat{d}_{\rm C}=d_{\rm C}$ and $\widehat{d}_{\rm S}={d}_{\rm S}$ and (6b) is absent, then $\widehat{\boldsymbol{q}}:\widehat{\mathcal{C}}\times\widehat{\mathcal{S}}\to
\mathcal{X}$ is bijective and $\widehat{\boldsymbol{f}}=\widehat{\boldsymbol{q}}^{-1}$ is also a solution of (2).

(b) If $\widehat{d}_{\rm C}>d_{\rm C}$ and $\widehat{d}_{\rm S}>{d}_{\rm S}$ and $\widehat{\boldsymbol{q}}:\widehat{\mathcal{C}}\times\widehat{\mathcal{S}}\to
\mathcal{X}$ is bijective, then $\widehat{\boldsymbol{f}}=\widehat{\boldsymbol{q}}^{-1}$ is also a solution of (4).

The formulation in (6) has a number of practical advantages over the direct implementation in (5).
Particularly, it avoids complex operations in the latent domain.
In LDM, performing DM on $\boldsymbol{f}_{\rm C}(\boldsymbol{x}^{(i)})$ and $\boldsymbol{f}_{\rm C}(\boldsymbol{x}^{(j)})$ poses quite a nontrivial optimization process. This is because both of the inputs to the DM modules (i.e., $\boldsymbol{f}_{\rm C}(\boldsymbol{x}^{(i)})$ for all $i\in[N]$) change from iteration to iteration—yet the DM module (e.g., GAN and Wasserstein distance-based DM (Goodfellow et al., ; Arjovsky et al., )) itself often involves complex optimization with its own parameters updated on the fly.
The new formulation performs GAN-based DM in the data domain and keeps one input (the real data) to every GAN module fixed. This reduces a lot of agony in optimization parameter tuning.
Eq. (6) also does not need any explicit constraint/regularization to enforce the block independence of $\boldsymbol{f}_{\rm C}$ and $\boldsymbol{f}_{\rm S}$ (which could be resource consuming (Lyu et al., ; Gretton et al., )), as $\boldsymbol{e}_{\rm C}$ and $\boldsymbol{e}_{\rm S}^{(n)}$ are constructed to be block independent.

Another quite interesting observation is that, the proof of Theorem 4.1 (a) shows that the bijectivity constraint on $\boldsymbol{q}$ is automatically fulfilled when an additional condition (i.e., that ${\cal C}$ and ${\cal S}$ are simply connected) is met. This means that the LDM formulation would need extra modules, e.g., $\mathbb{E}\|\boldsymbol{r}\circ\boldsymbol{f}(\boldsymbol{x})-\boldsymbol{x}\|
^{2}$, to impose injectivity constraints, even when $d_{\rm S}$ and $d_{\rm C}$ are known.
When $d_{\rm C}$ and $d_{\rm S}$ are unknown, solving Problem (6) per se does not ensure $\boldsymbol{q}$ to be bijective. Nonetheless, we observed that not explicitly enforcing bijectivity in implementations does not affect the performance in practice. Similar phenomenon was observed in nICA implementations; see, e.g., (Hyvarinen & Morioka, ; Hyvarinen et al., ).

###### Theorem 4.1 .

## 5 Related Works

Nonlinear ICA. Learning content and style components from a nonlinear mixture model is reminiscent of nonlinear independent analysis (nICA) (Hyvärinen & Pajunen, ; Hyvarinen & Morioka, ; Hyvarinen et al., ).
Most nICA works were developed under single domain settings, with some recent generalizations to multiple views/domains (Gresele et al., ; Hyvarinen et al., ).
Nonetheless, nICA requires that all the latent variables are (conditionally) independent. This is considered a somewhat restrictive assumption in content-style learning.

Content-Style Models in Aligned Multi-Domain Learning.
Aligned multi-domain content-style learning is a key technique in data-augmented SSL and representation learning.
There, it was shown that elementwise (conditional) independence is not needed, if the goal is to isolate content from style (Von Kügelgen et al., ; Lyu et al., ; Karakasis & Sidiropoulos, ). It was further shown that block independence (similar to Assumption 3.1) is the key to identify the style (Lyu et al., ; Daunhawer et al., ). However, all these works require cross-domain data alignment.

Content-Style Identification in Unaligned Multi-Domain Learning.
Identifiability of unaligned multi-domain learning was studied in the context of various applications, e.g., image translation (Shrestha & Fu, ), data synthesis (Xie et al., ), cross-domain information retrieval Timilsina et al. (), and domain adaptation (Kong et al., ; Gulrajani & Hashimoto, ; Timilsina et al., ).
The work (Kong et al., ) postulated a similar content-style model as in (Xie et al., ) and came up with identifiability conditions similar to those in (Xie et al., ).
The mostly related work to ours is (Xie et al., ),
as both works are interested in content-style identification under (1).
Our implementation in (6) partially recovers the marginal distribution matching criterion in (Xie et al., ), despite the fact that our learning criteria started with an LDM perspective.
Nonetheless, our method enjoys much less restrictive model assumptions for content-style identifiability.
Our multi-domain GAN also admits more relaxed neural architecture (see Appendix G).

Content-Style Learning without Knowing Latent Dimensions.
The SSL work (Von Kügelgen et al., ) presented a proof that essentially established that the content can be learned without knowing the exact dimension $d_{\rm C}$. However, their result was under the assumption that the domains are aligned. In addition, the proof could not hold when style learning is also involved. Our proof solved these challenges. The work in (Xie et al., ) used a mask-based formulation to remove the requirement of knowing $d_{\rm C}$ and $d_{\rm S}$. The mask-based formulation has the flavor of sparsity promoting as in our proposed method. However, they still need to know $d_{\rm C}+d_{\rm S}$, which is unlikely available in practice. In addition, the mask-based method in (Xie et al., ) did not have theoretical supports.

## 6 Numerical Examples

Multi-Domain Data Generation.
For the data generation task, we validate our theoretical claims using three real world datasets: animal faces (AFHQ) (Choi et al., ), CelebA-HQ (Karras et al., ), and CelebA (Liu et al., ) with 3, 2, and 7 domains, respectively (see Appendix G.6).

**Table 1: Evaluation of the data generation task. Standard deviation reported using $\pm$ for style diversity**
| Method | FID ($\downarrow$) | Style Diversity ($\uparrow$) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| AFHQ | CelebA-HQ | CelebA-7 | AFHQ | CelebA-HQ | CelebA-7 |  |
| Transitional-cGAN | 38.00 | 8.12 | 70.45 | – | – | – |
| StyleGAN-ADA | 8.17 | 5.89 | 72.10 | – | – | – |
| I-GAN | 6.28 | 5.91 | 5.18 | 0.16 $\pm$ 0.02 | 0.07 $\pm$ 0.03 | 0.07 $\pm 0.03$ |
| Proposed | 6.19 | 5.70 | 5.27 | 0.50 $\pm$ 0.03 | 0.36 $\pm$ 0.04 | 0.26 $\pm$ 0.06 |

The baselines here are I-GAN (Xie et al., ), StyleGAN-ADA (Karras et al., ) and Transitional-cGAN (Shahbazi et al., ).

Following Xie et al. (), we use StyleGAN2-ADA (Karras et al., ) to represent our generative function $\boldsymbol{q}$ in (6a).
We set $\widehat{d}_{\rm C}=384$ and $\widehat{d}_{\rm S}=128$ in all the experiments. We use a regularization term $\lambda\|\boldsymbol{e}_{\rm S}^{(n)}(\boldsymbol{r}_{\rm S}^{(n)})\|_{1}$ to approximate the sparsity constraint (6b). We find that the algorithm is not very sensitive to the choice $\lambda$ as any positive $\lambda$ encourages sparsity of $\boldsymbol{e}_{\rm S}^{(n)}(\boldsymbol{r}_{\rm S}^{(n)})$. We use $\lambda=0.3$ for all the experiments.
More detailed experimental settings are in Appendix G.

Fig. LABEL:fig:md_generation shows the qualitative results for content-style identification using various methods for the cat domain ($n=1$) of AFHQ. For each row, we fix the content part $\boldsymbol{c}=\boldsymbol{e}_{\rm C}(\boldsymbol{r}_{\rm C})$ (i.e., pose of the cat) and randomly sample different styles $\boldsymbol{s}^{(1)}=\boldsymbol{e}_{\rm S}^{(1)}(\boldsymbol{r}_{\rm S}^{(1)})$ where $\boldsymbol{r}_{\rm S}^{(1)}\sim\mathcal{N}(\boldsymbol{0},\boldsymbol{I}_{
\widehat{d}_{\rm S}})$ to generate the images $\boldsymbol{x}^{(1)}=\boldsymbol{q}(\boldsymbol{c},\boldsymbol{s}^{(1)})$.
This way, the samples $\boldsymbol{s}^{(1)}_{i}$ for $i=1,2,\ldots$ correspond to various types of cats.
Fig.LABEL:fig:md_generation (a) shows that the I-GAN appears to generate the same type of cat even when repeatedly sampled from their learned distribution of $\boldsymbol{s}^{(1)}$. This suggests that the style components are not extracted properly.
Fig. LABEL:fig:md_generation (b) shows the result of using the proposed method without any sparsity regularization. As explained earlier, this can lead to learning constant content part with all information captured by the style part.

Fig. LABEL:fig:md_generation(b) corroborates with the intuition since we see little to no pose variation in the sampled contents (i.e., the three rows). Fig. LABEL:fig:md_generation (c) shows the result of proposed method, i.e., (6). One can see that both content and style parts demonstrate sufficient diversity, indicating well learned content and style distributions. Appendix H shows similar results for other domains and datasets.

Fig. LABEL:fig:md_generation_diff_domains shows the generated samples of $\boldsymbol{x}^{(n)}$ for different $n$’s using models learned from the AFHQ and CelebA-HQ datasets, which correspond to different species of animals (cat, dog, and tiger) and different genders of people, respectively.
The top three rows in each figure correspond to the three $n$’s (i.e., three domains) of the AFHQ dataset, whereas the bottom two rows correspond to two $n$’s of the CelebA-HQ dataset.
For the $j$th row associated with each dataset, we sample three different styles $\boldsymbol{s}_{i}^{(n_{j})},i\in\{1,2,3\}$ and combine it with a fixed content $\overline{\boldsymbol{c}}$ to generate the image $\boldsymbol{x}_{i}^{(n_{j})}=\boldsymbol{q}(\overline{\boldsymbol{c}},
\boldsymbol{s}_{i}^{(n_{j})})$ in the $j$th row and $i$th column.
Both the baseline I-GAN and our method can combine a fixed $\bar{\boldsymbol{c}}$ with $\boldsymbol{s}^{(n_{j})}_{i}$ for different $i$ to create content (pose)-consistent new data (see all the rows).
However, one can see that the baseline I-GAN was not able to sample different styles in each domain. It seems that every domain $n$ always repeatedly samples the same style components $\bar{\boldsymbol{s}}^{(n)}$ as the same images always appear in the same row.
The proposed method can generate quite diverse style samples in all the domains. Additional results are in the Appendix H.

Table 1 shows the FID (Heusel et al., ) and style diversity scores attained by different methods. We use LPIPS distance (Zhang et al., ) between pairs of images with the same $\bar{\boldsymbol{c}}$ and different style samples from $\boldsymbol{s}^{(n)}$ to measure the style diversity. The diversity scores are averaged over 6,000 images across all domains, where every 10 images contain the same content with different styles. Note that the baselines StyleGAN-ADA and Transitional-cGAN do not learn content-style models, and thus the style diversity scores of theirs are not reported. One can see that the FID scores of the methods are similar, meaning that all methods generate realistic looking images.
However, the style diversity of the proposed method is 3 to 5 times higher than the baseline over all datasets. The conditional generative models (Transitional-cGAN and StyleGAN-ADA) sometimes encountered convergence issues on specific datasets as reflected by their FID scores.

Multi-Domain Translation.
Existing methods use a dedicated system for multi-domain translation (Choi et al., , ; Yang et al., ). However, since a multi-domain generative model can already disentangle content and style (cf. Theorems-4.1 of this work), one can simply use the generative model for domain translation.

Given an image $\boldsymbol{x}^{(i)}$ in the source domain $i$, in order to extract the corresponding content $\boldsymbol{c}$ or style $\boldsymbol{s}^{(i)}$ , one can simply solve
$(\widehat{\boldsymbol{c}},\widehat{\boldsymbol{s}})=\arg\min_{\boldsymbol{c},
\boldsymbol{s}}~{}{\sf div}(\boldsymbol{q}(\boldsymbol{c},\boldsymbol{s}),
\boldsymbol{x}^{(i)}),$
where ${\sf div}$ is some distance metric/divergence measure. There exists many approaches to solving the problem, often referred to as GAN inversion (Xia et al., ). In our case, we simply use the Adam optimizer for this inversion step.
For ${\sf div}$, we use a pre-trained VGG16 (Simonyan & Zisserman, ) neural network. More details are in Appendix G.
To generate the desired translation, the GAN inversion-extracted content can be combined with a randomly sampled style $\boldsymbol{s}^{(t)}=\boldsymbol{e}_{\rm S}^{(t)}(\boldsymbol{r}_{\rm S}^{(t)}
),\boldsymbol{r}_{\rm S}^{(t)}\sim\mathcal{N}(\boldsymbol{0},\boldsymbol{I}_{
\widehat{d}_{\rm S}})$ from the target domain $t$. Additionally, one can also extract style from an image in target domain and combine it with extracted content from the source domain for guided translation.

The baselines used are the method in (Xie et al., ), StarGANv2 (Choi et al., ), and SmoothGAN (Liu et al., ). Note that (Xie et al., ) proposed a separate system for the domain translation that uses its pre-trained multi-domain generative model to train a separate translation model (see Appendix F). However, since the aforementioned GAN inversion procedure is also applicable to their generative model as it extracts content and style, we use two versions of their system, namely, I-GAN (Gen) for the method based on GAN inversion and I-GAN (Tr) for the separate translation system proposed in (Xie et al., ).

Figure: Figure 4: Guided translation by combining content (first column) with style (second column) of the images.
Refer to caption: extracted/5992817/images/translation/afhq_ref_translation.png

Fig. LABEL:fig:translation_samples shows the result of translation.
Fig. LABEL:fig:translation_samples (a) and (b) shows guided translation from wild domain ($n=3$) to cat ($n=1$). The first column contains two samples $\boldsymbol{x}_{i}^{(3)},i\in[2]$ in the wild domain, from which the content $\widehat{\boldsymbol{c}}_{i}^{(3)},i\in[2]$ is extracted using GAN inversion. The first row contains two samples $\boldsymbol{x}_{j}^{(1)},j\in[2]$ in the cat domain, from which the styles $\widehat{\boldsymbol{s}}_{j}^{(1)},j\in[2]$ are extracted. Finally, the translated image are synthesized using $\widehat{\boldsymbol{q}}(\widehat{\boldsymbol{c}}_{i}^{(3)},\widehat{
\boldsymbol{s}}_{j}^{(1)})$. One can see that the translated images appear consistent with their respective input contents (i.e., pose of wild) and styles (i.e., type of dog) for the proposed method, whereas I-GAN (Gen) seems to ignore the style information.

Fig. LABEL:fig:translation_samples (c) and (d) show the result of translation from wild domain ($n=3$) to dog domain ($n=2$) using randomly sampled style components. The content $\widehat{\boldsymbol{c}}_{i}^{(3)},i\in[3]$ extracted for samples in the wild domain is combined with randomly sampled styles $\boldsymbol{s}_{j}^{(2)},j\in[2]$ in the dog domain to synthesize the translated images. Our translations in each row contain the same content (i.e., pose of wild) as the input source image, but different styles (i.e., dog species). However, I-GAN (Gen) seems to produce unrealistic samples in some cases (first row).
Their style diversity also appears to be limited.

Fig. 4 shows results of guided-translation (similar to Fig. LABEL:fig:translation_samples (a, b)) for all methods for all pairs of domains in the AFHQ domain. Content extracted from the images in the first column is combined with the style from the second column. One can see that the proposed method preserves the style information better than the baselines.

**Table 2: Quantitative evaluation of all methods for the translated images.**
| Method | FID ($\downarrow$) | Style Diversity ($\uparrow$) | Training time ( hours) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| AFHQ | CelebA-HQ | AFHQ | CelebA-HQ | Generation | Translation | Total |  |
| StarGANv2 | 16.83 | 13.67 | 0.45 $\pm$ 0.03 | 0.45 $\pm$ 0.03 | – | 50.83 | 50.83 |
| SmoothGAN | 53.68 | 29.69 | 0.14 $\pm$ 0.04 | 0.09 $\pm$ 0.03 | – | 30.90 | 30.90 |
| I-GAN (Tr) | 19.57 | 15.26 | 0.46 $\pm$ 0.03 | 0.29 $\pm$ 0.05 | 29.53 | 67.46 | 96.99 |
| Proposed | 13.74 | 16.61 | 0.53 $\pm$ 0.03 | 0.41 $\pm$ 0.03 | 27.36 | – | 27.36 |

Table 2 shows that the image quality (see FID) and diversity (see style diversity) of the translated images are competitive or better than the baselines (see qualitative results in Fig. 5 and LABEL:fig:app_latent_trans of Appendix H.). One can also see that the training time (on a single Tesla V100 GPU) of proposed method is at least 22 and 69 hours shorter than the competitive baselines StarGANv2 and I-GAN (Tr), respectively.

## 7 Conclusion

We revisited the problem of content-style identification from unaligned multi-domain data, which is a key step for provable domain translation and data generation.
We offered a LDM perspective. This new viewpoint enabled us to prove identifiability results that enjoy considerably more relaxed conditions compared to those in previous research. Most importantly, we proved that content and style can be identified without knowing the exact dimension of the latent components. To our knowledge, this stands as the first dimension-agnostic identifiability result for content-style learning.
We showed that the LDM formulation is equivalent to a latent domain-coupled multi-domain GAN loss, and the latter features a simpler implementation in practice.
We validated our theorems using image translation and generation tasks.

Limitations. Our work focused on sufficient conditions for content-style identifiability, yet the necessary conditions were not fully understood—which is also of great interest. Additionally, our model considers that the domains are in the range of the same generating function. The applicability is limited to homogeneous multi-domain data, e.g., images with the same resolution. An interesting extension is to consider heterogeneous multi-domain models that can deal with very different types of data (e.g., text and audio).

## Appendix A Preliminaries

### A.1 Notation

- •
A set $\mathcal{A}$ is said to have strictly positive measure under $\mathbb{P}_{\boldsymbol{x}}$ if and only if $\mathbb{P}_{\boldsymbol{x}}[\mathcal{A}]>0$.
- •
For a (random) vector $\boldsymbol{x}$, $x(i)$ and $[\boldsymbol{x}]_{i}$ denote the $i$th element of $\boldsymbol{x}$, and $\boldsymbol{x}(i:j)$ and $[\boldsymbol{x}]_{i:j}$ denotes $[x(i),x(i+1),\dots,x(j)]$.
- •
The notation $[N]=\{1,2,\dots,N\}$.
- •
$\boldsymbol{I}_{\rm D}\in\mathbb{R}^{D\times D}$ denotes identity matrix of size $D\times D$.
- •
For a differentiable function $\boldsymbol{f}:\mathbb{R}^{M}\to\mathbb{R}^{N}$, the Jacobian of $\boldsymbol{f}$ with respect to its input $\boldsymbol{x}$, denoted by $\boldsymbol{J}_{\boldsymbol{f}}(\boldsymbol{x})$, is the matrix of partial derivatives as follows:
$\displaystyle\boldsymbol{J}_{\boldsymbol{f}}(\boldsymbol{x})=\begin{bmatrix}
\frac{\partial[\boldsymbol{f}(\boldsymbol{x})]_{1}}{\partial x_{1}}&\dots&
\frac{\partial[\boldsymbol{f}(\boldsymbol{x})]_{1}}{\partial x_{M}}\\
\vdots&\ddots&\vdots\\
\frac{\partial[\boldsymbol{f}(\boldsymbol{x})]_{N}}{\partial x_{1}}&\dots&
\frac{\partial[\boldsymbol{f}(\boldsymbol{x})]_{N}}{\partial x_{M}}\\
\end{bmatrix}$
- •
${\rm det}(\boldsymbol{X})$ denotes the determinant of matrix $\boldsymbol{X}$.
- •
$\boldsymbol{x}\rotatebox[origin={c}]{90.0}{$\models$}\boldsymbol{y}$ denotes that $\boldsymbol{x}$ is statistically independent of $\boldsymbol{y}$.
- •
$|x|$ denotes the absolute value of $x$.
- •
$\|\boldsymbol{x}\|_{0}$ denotes the number of non-zero elements in $\boldsymbol{x}$.
- •
$\epsilon$-ball: Given a metric space $\mathcal{X}\subseteq\mathbb{R}^{D}$ for some $D\in\mathbb{N}$, $\mathcal{N}_{\epsilon}(\boldsymbol{z})$ denotes the $\epsilon$-neighborhood of $\boldsymbol{z}\in\mathcal{X}$ defined as
$\mathcal{N}_{\epsilon}(\boldsymbol{z})=\{\widehat{\boldsymbol{z}}\in\mathcal{X
}~{}|~{}\|\boldsymbol{z}-\widehat{\boldsymbol{z}}\|_{2}<\epsilon\}.$
- •
Image set: For a function $\boldsymbol{m}:\mathcal{W}\to\mathcal{Z}$, and a set $\mathcal{A}\subseteq\mathcal{W}$, $\boldsymbol{m}(\mathcal{A})=\{\boldsymbol{m}(\boldsymbol{w})\in\mathcal{Z}~{}|
~{}\boldsymbol{w}\in\mathcal{A}\}$
- •
For an injective function $\boldsymbol{f}:\mathcal{X}\to\mathcal{Y}$, we denote $\boldsymbol{f}^{-1}:\boldsymbol{f}(\mathcal{X})\to\mathcal{Y}$ as the left inverse of $\boldsymbol{f}$, i.e., $\boldsymbol{f}^{-1}\circ\boldsymbol{f}(\boldsymbol{x})=\boldsymbol{x},\forall
\boldsymbol{x}\in\mathcal{X}$.
- •
Pre-Image set: For a general function $\boldsymbol{m}:\mathcal{W}\to\mathcal{Z}$, we overload the notation $\boldsymbol{m}^{-1}$ to represent the pre-image; i.e., for a set $\mathcal{A}\subseteq\mathcal{Z}$, $\boldsymbol{m}^{-1}(\mathcal{A})=\{\boldsymbol{w}\in\mathcal{W}~{}|~{}
\boldsymbol{m}(\boldsymbol{w})\in\mathcal{A}\}$. If $\boldsymbol{m}$ is injective, then $\boldsymbol{m}^{-1}$ is simply the left inverse of $\boldsymbol{m}$; i.e., $\boldsymbol{m}^{-1}\circ\boldsymbol{m}$ is identity.
- •
For a function $\boldsymbol{f}:\mathcal{X}\to\mathcal{Y}$ and a probability measure $\mathbb{P}_{\boldsymbol{x}}$ defined on $\mathcal{X}$, $[\boldsymbol{f}]_{\#\mathbb{P}_{\boldsymbol{x}}}$ denotes the push forward measure defined by $[\boldsymbol{f}]_{\#\mathbb{P}_{\boldsymbol{x}}}[\mathcal{A}]=\mathbb{P}_{
\boldsymbol{x}}[\boldsymbol{f}^{-1}(\mathcal{A})]$ for any measurable $\mathcal{A}\subseteq\mathcal{X}$.
- •
${\rm dim}(\mathcal{X})$ denotes the covering dimensio of the set $\mathcal{X}$ (see Appendix A.2).

### A.2 Definitions and Useful Facts

We employ standard definitions and facts from real analysis. We refer the readers to (Carothers, ; Rudin, ) for precise definition and more details.

Injection, bijection, and surjection.

Consider a function $\boldsymbol{f}:\mathcal{X}\to\mathcal{Y}$. Then

- •
$\boldsymbol{f}$ is injective, if for all $\boldsymbol{x}_{1},\boldsymbol{x}_{2}\in\mathcal{X}$, $\boldsymbol{x}_{1}\neq\boldsymbol{x}_{2}\implies\boldsymbol{f}(\boldsymbol{x}_
{1})\neq\boldsymbol{f}(\boldsymbol{x}_{2})$.
- •
$\boldsymbol{f}$ is surjective, if for all $\boldsymbol{y}\in\mathcal{Y}$, $\exists\boldsymbol{x}\in\mathcal{X}$ such that $\boldsymbol{f}(\boldsymbol{x})=\boldsymbol{y}$.
- •
$\boldsymbol{f}$ is bijective or invertible, if $\boldsymbol{f}$ is both injective and surjective.

Also consider the following useful facts:

- 1.
$\boldsymbol{f}$ is injective if and only if there exist a function $\boldsymbol{g}:\boldsymbol{f}(\mathcal{X})\to\mathcal{X}$ such that $\boldsymbol{g}\circ\boldsymbol{f}(\boldsymbol{x})=\boldsymbol{x},\forall
\boldsymbol{x}\in\mathcal{X}$, i.e., $\boldsymbol{g}\circ\boldsymbol{f}$ is an identity function.
- 2.
If $\boldsymbol{f}:\mathcal{X}\to\mathcal{Y}$ is injective, then $\boldsymbol{f}:\mathcal{X}\to\boldsymbol{f}(\mathcal{X})$ is bijective.
- 3.
If $\boldsymbol{f}\circ\boldsymbol{g}$ is injective, and $\boldsymbol{g}$ is surjective, then $\boldsymbol{f}$ is injective.
- 4.
If $\boldsymbol{f}\circ\boldsymbol{g}$ is bijective, then $\boldsymbol{g}$ is surjective and $\boldsymbol{f}$ is injective.

An important remark is that since injective functions can be “inverted” (fact 1 above), there is no loss of information of the input, i.e., the input information can “reconstructed” from the output. Hence, for the purpose of identifiability, it is sufficient to identify the latent variables upto injective transformations.

Simply Connected Sets.

First, a set $\mathcal{C}$ (in $\mathcal{X}$) is connected if and only if there does not exist any disjoint non-empty open sets $\mathcal{A},\mathcal{B}\subseteq\mathcal{X}$ such that $\mathcal{A}\cap\mathcal{C}\neq\phi,\mathcal{B}\cap\mathcal{C}\neq\phi$, and $\mathcal{C}\subset\mathcal{A}\cup\mathcal{B}$.

A connected set is called simply connected if any simple closed curve on the set can be shrunk to a point continuously while staying inside the set.

Homeomorphism.

A function $\boldsymbol{f}:\mathcal{X}\to\mathcal{Y}$ is a homeomorphism if it is a continuous bijection and has continuous inverse. If there exists a homeomorphism between $\mathcal{X}$ and $\mathcal{Y}$, then $\mathcal{X}$ and $\mathcal{Y}$ are called homeomorphic.

Some useful properties are as follows:

- 1.
$\mathbb{R}^{M}$ is not homeomorphic to $\mathbb{R}^{D}$ for $D\neq M$.
- 2.
Open balls in $\mathbb{R}^{D}$ are homeomorphic to $\mathbb{R}^{D}$.
- 3.
Invariance of Domain: If $\mathcal{X}$ is an open subset of $\mathbb{R}^{D}$ and $\boldsymbol{f}:\mathcal{X}\to\mathbb{R}^{D}$ is an injective continuous map, then $\mathcal{Y}=\boldsymbol{f}(\mathcal{X})$ is open in $\mathbb{R}^{D}$, and $\boldsymbol{f}$ is a homeomorphism between $\mathcal{X}$ and $\mathcal{Y}$.

Covering dimension (Engelking, ).

An open cover of a set $\mathcal{X}$ is a collection of open sets such that $\mathcal{X}$ lies inside their union.

A refinement of a cover $\mathcal{Y}$ of a set $\mathcal{X}$ is a new cover $\mathcal{Z}$ of $\mathcal{X}$ such that every set in $\mathcal{Z}$ is contained in some set in $\mathcal{Y}$.

The (Lebesgue) covering dimension, denoted by ${\rm dim}(\mathcal{X})$, of a set $\mathcal{X}$ is the smallest number $n$ such that for every open cover, there is a refinement in which every point in $\mathcal{X}$ lies in the intersection of no more than $n+1$ covering sets. For our purposes (subsets of Euclidean spaces), the covering dimension is the ordinary Euclidean deimension, e.g., one for lines and curves, two for planes, and so on.

Some propoerties of the covering dimension are as follows:

- 1.
Homeomorphic spaces have the same covering dimension.
- 2.
If $\mathcal{X}$ is an open subset in $\mathbb{R}^{D}$, then ${\rm dim}(\mathcal{X})=D$.
- 3.
For two open sets $\mathcal{X}\subseteq\mathbb{R}^{D},\mathcal{Y}\subseteq\mathbb{R}^{M}$, ${\rm dim}(\mathcal{X}\times\mathcal{Y})={\rm dim}(\mathcal{X})+{\rm dim}(
\mathcal{Y})$.
- 4.
For a continuous funtion $\boldsymbol{f}$ and open set $\mathcal{X}$, ${\rm dim}(\boldsymbol{f}(\mathcal{X}))\leq{\rm dim}(\mathcal{X})$.
- 5.
The set $\mathcal{A}_{q}\subset\mathbb{R}^{D}$ of $q$-sparse vectors, i.e., $\mathcal{A}_{q}=\{\boldsymbol{x}\in\mathbb{R}^{D}|\|\boldsymbol{x}\|_{0}\leq q\}$ has dimension ${\rm dim}(\mathcal{A}_{q})=q$. This is because it is the union of a finite number $D\choose q$ of $q$-dimensional hyperplanes (i.e., hyperplanes with all elements 0, except q elements at fixed positions).

## Appendix B Detailed Discussions on Existing Identifiability Results

In this section, we review the existing content-style identifiability results from Sturma et al. (); Timilsina et al. (); Kong et al. (); Xie et al. ().

### B.1 Identifiability result in Sturma et al. ( 2023 )

The works of (Timilsina et al., ; Sturma et al., ) considered a linear mixture-based generative model as follows:

$$ $\displaystyle\boldsymbol{x}^{(n)}=\boldsymbol{G}^{(n)}\boldsymbol{z}^{(n)}, \quad\boldsymbol{z}^{(n)}=[\boldsymbol{c}^{\top},(\boldsymbol{s}^{(n)})^{\top} ]^{\top},~{}n\in\{1,2\},$ (7) $$

where $\boldsymbol{c}\in\mathbb{R}^{d_{\rm C}}$, $\boldsymbol{s}^{(n)}\in\mathbb{R}^{d_{\rm S}^{(n)}}$, and $\boldsymbol{G}^{(n)}\in\mathbb{R}^{(d_{\rm C}+d_{\rm S}^{(n)})\times(d_{\rm C}
+d_{\rm S}^{(n)})}$ is an invertible mixing matrix for the $n$th domain.

[backgroundcolor=gray!10,topline=false,
rightline=false,
leftline=false,
bottomline=false]

###### Theorem B.1 (Identifiability from Sturma et al. ( 2023 ) ) .

Under (7), assume that the following are met: (i) The conditions for ICA identifiability (Comon, ) is met by each domain, including that the components of $\boldsymbol{z}^{(n)}=[\boldsymbol{c}^{\top},(\boldsymbol{s}^{(n)})^{\top}]^{\top}$ are mutually statistically independent and contain at most one Gaussian variable. In addition, each $\boldsymbol{z}^{(n)}_{i}$ has unit variance; (ii) $\mathbb{P}_{z^{(n)}_{i}}\neq\mathbb{P}_{z^{(n)}_{j}},\mathbb{P}_{z^{(n)}_{i}}
\neq\mathbb{P}_{-z^{(n)}_{j}}~{}\forall i,j\in[d_{\rm C}+d_{\rm S}^{(n)}],~{}i\neq
j$.
Then, assume that $(i_{m},j_{m})$ are obtained by ICA followed by cross domain matching distribution matching, i.e., enforcing $\widehat{c}_{m}^{(1)}\xlongequal{({\sf d})}\widehat{c}_{m}^{(2)}$ for $m=1,\ldots,d_{\rm C}$.
Denote $\widehat{c}^{(1)}_{m}=\boldsymbol{e}^{\!\top\!}_{i_{m}}\widehat{\boldsymbol{z}
}^{(1)}$ and $\widehat{c}^{(2)}_{m}=\boldsymbol{e}^{\!\top\!}_{j_{m}}\widehat{\boldsymbol{z}
}^{(2)}$.
We have the following:

$$ $\displaystyle\widehat{c}^{(n)}_{m}=kc^{(n)}_{\boldsymbol{\pi}(m)},~{}m\in[d_{ \rm C}],$ (8) $$

where $k\in\{+1,-1\}$ and $\boldsymbol{\pi}$ is a permutation of $\{1,\ldots,d_{\rm C}\}$.

Obviously, the result here relies on the element-wise independence among the entries of $\boldsymbol{z}^{(n)}$.

###### Theorem B.1 (Identifiability from Sturma et al. ( 2023 ) ) .

### B.2 Identifiability result in Timilsina et al. ( 2024 )

Timilsina et al. () proposed to solve the following problem in order to extract the content and style for model in (7):

$$ $\displaystyle{\rm find}\quad$ $\displaystyle\boldsymbol{Q}_{\rm C}^{(n)}\in\mathbb{R}^{d_{\rm C}\times(d_{\rm C }+d_{\rm S}^{(n)})},\boldsymbol{Q}_{\rm S}^{(n)}\in\mathbb{R}^{d_{\rm S}^{(n)} \times(d_{\rm C}+d_{\rm S}^{(n)})}~{}n=1,2,$ (9a) subject to $\displaystyle\boldsymbol{Q}_{\rm C}^{(1)}\boldsymbol{x}^{(1)}\xlongequal{({\sf d })}\boldsymbol{Q}_{\rm C}^{(2)}\boldsymbol{x}^{(2)},$ (9b) $\displaystyle\boldsymbol{Q}_{\rm C}^{(n)}\boldsymbol{x}^{(n)}\rotatebox[origin ={c}]{90.0}{$\models$}\boldsymbol{Q}_{\rm S}^{(n)}\boldsymbol{x}^{(n)}\quad n= 1,2,$ (9c) $\displaystyle\boldsymbol{Q}_{\rm C}^{(n)}\mathbb{E}\left[\boldsymbol{x}^{(n)}( \boldsymbol{x}^{(n)})^{\top}\right](\boldsymbol{Q}_{\rm C}^{(n)})^{\top}= \boldsymbol{I}\quad n=1,2,$ (9d) $\displaystyle\boldsymbol{Q}_{\rm S}^{(n)}\mathbb{E}\left[\boldsymbol{x}^{(n)}( \boldsymbol{x}^{(n)})^{\top}\right](\boldsymbol{Q}_{\rm S}^{(n)})^{\top}= \boldsymbol{I}\quad n=1,2,$ (9e) $$

where $a\xlongequal{({\sf d})}b$ denotes that the distribution of $a$ and $b$ are the same.
The following identifiability result was established of the solution of (9):
{mdframed}[backgroundcolor=gray!10,topline=false,
rightline=false,
leftline=false,
bottomline=false]

###### Theorem B.2 (Identifiability from Timilsina et al. ( 2024 ) ) .

Let $\widehat{\boldsymbol{Q}}_{\rm C}^{(n)}$ and $\widehat{\boldsymbol{Q}}_{\rm S}^{(n)}$ denote the solution of (9). Under (7), assume that the following are met:

- 1.
For any two linear subspaces $\mathcal{P}^{(n)}\subset\mathbb{R}^{d_{\rm C}+d_{\rm S}^{(n)}},~{}n=1,2$, with ${\rm dim}(\mathcal{P}^{(n)})=d^{(n)}_{\rm S}$, $\mathcal{P}^{(n)}\neq\boldsymbol{0}\times\mathbb{R}^{d_{\rm S}^{(n)}}$ and linearly independent vectors $\{\boldsymbol{y}^{(n)}_{i}\in\mathbb{R}^{d_{\rm C}+d_{\rm S}^{(n)}}\}_{i=1}^{d
_{\rm C}},~{}n=1,2$, the sets
$\mathcal{A}^{(n)}={\rm conv}\{\boldsymbol{0},\boldsymbol{y}^{(n)}_{1},\dots,
\boldsymbol{y}^{(n)}_{d_{\rm C}}\}+\mathcal{P}^{(n)},~{}n=1,2,$
are such that if $\mathbb{P}_{\boldsymbol{c},\boldsymbol{s}^{(n)}}[\mathcal{A}^{(n)}]>0$ for $n=1$ or $n=2$, then there exists a $k\in\mathbb{R}$ such that the joint distributions
$\mathbb{P}_{\boldsymbol{c},\boldsymbol{s}^{(1)}}[k\mathcal{A}^{(1)}]\neq
\mathbb{P}_{\boldsymbol{c},\boldsymbol{s}^{(2)}}[k\mathcal{A}^{(2)}]$, where $k\mathcal{A}^{(n)}=\{k\boldsymbol{a}~{}|~{}\boldsymbol{a}\in\mathcal{A}^{(n)}\}$.
- 2.
One of the following assumptions is satisfied:
(a)
The individual elements of the content components are statistically independent and non-Gaussian. In addition, $c_{i}\stackrel{{\scriptstyle({\sf d})}}{{\neq}}kc_{j},\forall i\neq j,\forall k
\in\mathbb{R}$ and $c_{i}\stackrel{{\scriptstyle({\sf d})}}{{\neq}}-c_{i},\forall i$, i.e., the marginal distributions of the content elements cannot be matched with each other by mere scaling.
(b)
The support $\mathcal{C}$, is a hyper-rectangle, i.e., $\mathcal{C}=[-a_{1},a_{1}]\times\dots\times[-a_{d_{\rm C}},a_{d_{\rm C}}]$. Further, suppose that $c_{i}\stackrel{{\scriptstyle({\sf d})}}{{\neq}}kc_{j},\forall i\neq j,\forall k
\in\mathbb{R}$ and $c_{i}\stackrel{{\scriptstyle({\sf d})}}{{\neq}}-c_{i},\forall i$.

Then, we have $\widehat{\boldsymbol{Q}}^{(n)}_{\rm C}\boldsymbol{x}^{(n)}=\boldsymbol{\Theta}
\boldsymbol{c}$ and $\widehat{\boldsymbol{Q}}^{(n)}_{\rm S}\boldsymbol{x}^{(n)}=\boldsymbol{\Xi}^{(
n)}\boldsymbol{s}^{(n)},~{}$ for some invertible $\boldsymbol{\Xi}^{(n)}$ for all $n=1,2.$

The first condition is similar to the domain variability condition considered in Kong et al. (); Xie et al. () and our Assumption 3.2,
but with a more specific form that is adjusted according to the linear mixture model in (7).

###### Theorem B.2 (Identifiability from Timilsina et al. ( 2024 ) ) .

### B.3 Identifiability result in Xie et al. ( 2023 ); Kong et al. ( 2022 )

Note that the results in Sec. B.1 and B.2 are only applicable in the linear mixing system case (see (7)). However, the linear model is hardly practical for complex data modalities like images.
In the nonlinear case, Xie et al. (); Kong et al. () showed the following:
{mdframed}[backgroundcolor=gray!10,topline=false,
rightline=false,
leftline=false,
bottomline=false]

###### Theorem B.3 (Identifiability from (Xie et al., 2023 ; Kong et al., 2022 ) ) .

Denote $\boldsymbol{z}=(\boldsymbol{c},\boldsymbol{s})\in{\cal C}\times{\cal S}$ and $\boldsymbol{u}$ as the latent variables and an auxiliary variable(^1^11The auxiliary variable $\boldsymbol{u}$ is a notion from the nICA literature (Hyvarinen & Morioka, ; Hyvarinen et al., ; Khemakhem et al., ). In the context of multi-domain learning, $\boldsymbol{u}$ often represents the domain index.), respectively. Assume the following assumptions hold:

(A1) (Smooth and Positive Density): $\log p(\boldsymbol{z}|\boldsymbol{u})$ is second-order differentiable and $p(\boldsymbol{z}|\boldsymbol{u})>0$.

(A2) (Conditional independence): $p(\boldsymbol{z}|\boldsymbol{u})=\prod_{i=1}^{d_{\rm S}+d_{\rm C}}p(z_{i}|
\boldsymbol{u})$.

(A3) (Linear independence): For any $\boldsymbol{s}\in{\cal S}\subseteq\mathbb{R}^{d_{\rm S}}$, there exists $2d_{\rm S}+1$ values of $\boldsymbol{u}$, i.e., $\boldsymbol{u}_{j}$ with $j=0,1,\ldots,2d_{\rm S}$ such that the $2d_{\rm S}$ vectors $\boldsymbol{w}(\boldsymbol{s},\boldsymbol{u}_{j})-\boldsymbol{w}(\boldsymbol{s
},\boldsymbol{u}_{0})$ are linearly independent, where
$\boldsymbol{w}(\boldsymbol{s},\boldsymbol{u})=\left(\nicefrac{{\partial q_{1}}
}{{\partial\boldsymbol{s}_{1}}},\ldots,\nicefrac{{\partial q_{d_{\rm S}}}}{{
\partial\boldsymbol{s}_{d_{\rm S}}}},\nicefrac{{\partial^{2}q_{1}}}{{\partial
\boldsymbol{s}^{2}_{1}}},\ldots,\nicefrac{{\partial^{2}q_{d_{\rm S}}}}{{
\partial\boldsymbol{s}^{2}_{d_{\rm S}}}}\right)$ and $q_{i}=\log p(s_{i}|\boldsymbol{u})$.

(A4) (Domain variability): For any set $\mathcal{A}\in{\cal Z}$ such that (i) $\mathbb{P}_{\boldsymbol{z}|u^{\prime}}[\mathcal{A}]>0$ for any $\boldsymbol{u}^{\prime}\in{\cal U}$ and (ii) ${\cal A}$ cannot be expressed as ${\cal B}\times{\cal S}$ for any set ${\cal B}\subset{\cal C}$, there exist $\boldsymbol{u}_{1},\boldsymbol{u}_{2}\in{\cal U}$ such that $\mathbb{P}_{\boldsymbol{z}|\boldsymbol{u}_{1}}[\mathcal{A}]\neq\mathbb{P}_{
\boldsymbol{z}|\boldsymbol{u}_{2}}[\mathcal{A}]$ where
$\mathbb{P}_{\boldsymbol{z}|\boldsymbol{u}_{i}}[\mathcal{A}]=\int_{\boldsymbol{
z}\in\mathcal{A}}P(\boldsymbol{z}|\boldsymbol{u}_{i})d\boldsymbol{z}$.

Then, by matching the model distribution in (1) with the marginals $\mathbb{P}_{\boldsymbol{x}^{(n)}}$ jointly, the component-wise and block-wise identifiability of $\boldsymbol{s}$ and $\boldsymbol{c}$ are guaranteed, respectively, both up to invertible nonlinear transformations.

The result is interesting and insightful—particularly, the domain variability condition (A4) is also used in our analysis. However, some major challenges still remain:
First, (A2) requires that all components of $\boldsymbol{z}^{(n)}=(\boldsymbol{c},\boldsymbol{s}^{(n)})$ are statistically independent given $n$ (corresponding to $\boldsymbol{u}$). The use of this condition is because the identifiability proof is reminiscent of nICA (Khemakhem et al., ; Hyvarinen et al., ).
But elementwise independence is a nontrivial assumption. As shown in our analysis, for block-wise identifiability of $\boldsymbol{c}$ and $\boldsymbol{s}^{(n)}$, elementwise independence among individual entries of $\boldsymbol{z}=(\boldsymbol{c},\boldsymbol{s})$ is not necessary.
In addition, (A3) needs that at least $N=2d_{\rm S}+1$ domains exist (as $\boldsymbol{u}$ represents the domain index in this context), which is hardly practical—many domain translation tasks were performed over $N=2$ domains (Choi et al., ).

###### Theorem B.3 (Identifiability from (Xie et al., 2023 ; Kong et al., 2022 ) ) .

## Appendix C Proof of Theorem 3.3

We restate the theorem here:

[backgroundcolor=gray!10,topline=false,
rightline=false,
leftline=false,
bottomline=false]
Theorem 3.3 Under Eq. (1), suppose
that Assumptions 3.1 and 3.2 hold.
Then, we have
$\widehat{\boldsymbol{f}}_{\rm C}(\boldsymbol{x}^{(n)})=\boldsymbol{\gamma}(
\boldsymbol{c})$ and $\widehat{\boldsymbol{f}}_{\rm S}(\boldsymbol{x}^{(n)})=\boldsymbol{\delta}(
\boldsymbol{s}^{(n)}),\forall n\in[N]$, where $\boldsymbol{\gamma}:\mathcal{C}\rightarrow\mathbb{R}^{d_{\rm C}}$ and $\boldsymbol{\delta}:\mathcal{S}\rightarrow\mathbb{R}^{d_{\rm S}}$ are injective functions.

Proof Outline.
The proof pipeline of Theorem 3.3 can be divided into two parts: content identification and style identification. The content identification part includes two steps: (i) showing that $\widehat{\boldsymbol{f}}_{\rm C}$ does not depend upon the style part, and thus is a function (denoted by $\boldsymbol{\gamma}$) of the content only and (ii) showing that $\boldsymbol{\gamma}$ is injective. For the first step, we use the domain variability assumption and matched content distribution to conclude that the preimage sets (under $\widehat{\boldsymbol{f}}\circ\boldsymbol{g}$) for any extracted content cover the entire style domain. This will imply that the extracted content is invariant of the style part.
In the second step, we use the property of the determinant of the jacobian of $\widehat{\boldsymbol{f}}\circ\boldsymbol{g}$ to derive that $\boldsymbol{\gamma}$ should be injective. For style identification, we use the independence constraint and the result of content identification part to conclude that the extracted style does not depend upon the content part. And finally we use the injectivity of $\widehat{\boldsymbol{f}}$ to show that $\boldsymbol{\delta}$ should also be injective.

###### Proof.

The proof is as follows:

###### Proof.

### C.1 Content Identification

Let $\boldsymbol{z}^{(n)}=(\boldsymbol{c},\boldsymbol{s}^{(n)})$ and $\mathcal{Z}=\mathcal{C}\times\mathcal{S}$.
Let $\widehat{\boldsymbol{c}}^{(n)}=\widehat{\boldsymbol{f}}_{\rm C}(\boldsymbol{x}
^{(n)}),\forall n\in[N]$, and $\widehat{\boldsymbol{s}}^{(n)}=\widehat{\boldsymbol{f}}_{\rm S}(\boldsymbol{x}
^{(n)}),\forall n\in[N]$.
The proof consists of the following two steps:

- Step 1.
First, we show that under the assumptions in Theorem 3.3, $\widehat{\boldsymbol{c}}^{(n)}$ does not depend upon $\boldsymbol{s}^{(n)}$.
- Step 2.
Next, we show that $\widehat{\boldsymbol{c}}^{(n)}$ is transformed from the true shared component $\boldsymbol{c}$ via an injective function $\boldsymbol{\gamma}:\mathcal{C}\to\mathbb{R}^{d_{\rm C}}$ for all $n\in[N]$.

Step 1: We want to establish the following:

$$ $\displaystyle\frac{\partial\widehat{c}_{i}^{(n)}}{\partial s_{j}^{(n)}}=0, \forall i\in[d_{\rm C}],j\in[d_{\rm S}].$ (10) $$

This can be shown using similar arguments in the domain adaptation work (Kong et al., ). To be specific, let $\boldsymbol{h}:\mathcal{Z}\to\mathbb{R}^{d_{\rm C}+d_{\rm S}}$
be defined as follows:

$$ $\boldsymbol{h}:=\widehat{\boldsymbol{f}}\circ\boldsymbol{g}.$ $$

Let $\widehat{\mathcal{C}}=\boldsymbol{h}_{\rm C}(\mathcal{Z})$. Due to the distribution matching constraint (2a), the following holds for any $\mathcal{A}_{\rm C}\subseteq\widehat{\mathcal{C}}$:

$$ $\displaystyle\mathbb{P}_{\widehat{\boldsymbol{c}}^{(i)}}[\mathcal{A}_{\rm C}]$ $\displaystyle=\mathbb{P}_{\widehat{\boldsymbol{c}}^{(j)}}[\mathcal{A}_{\rm C}] ,~{}\forall i,j\in[N]$ $\displaystyle\stackrel{{\scriptstyle(a)}}{{\iff}}\mathbb{P}_{\boldsymbol{z}^{( i)}}\left[\boldsymbol{h}_{\rm C}^{-1}(\mathcal{A}_{\rm C})\right]$ $\displaystyle=\mathbb{P}_{\boldsymbol{z}^{(j)}}\left[\boldsymbol{h}_{\rm C}^{- 1}(\mathcal{A}_{\rm C})\right]$ (11) $$

where $\boldsymbol{h}_{\rm C}^{-1}(\mathcal{A}_{\rm C}):=\{\boldsymbol{z}~{}|~{}
\boldsymbol{h}_{\rm C}(\boldsymbol{z})\in\mathcal{A}_{\rm C}\}$ is the preimage of $\boldsymbol{h}_{\rm C}(\cdot):=[\boldsymbol{h}(\cdot)]_{1:d_{\rm C}}$. The equivalence in (a) holds because $\mathbb{P}_{\widehat{\boldsymbol{c}}^{(n)}}[\mathcal{A}_{\rm C}]=\mathbb{P}_{
\boldsymbol{h}_{\rm C}(\boldsymbol{z}^{(n)})}[\mathcal{A}_{\rm C}]=\mathbb{P}_
{\boldsymbol{z}^{(n)}}[\boldsymbol{h}_{\rm C}^{-1}(\mathcal{A}_{\rm C})],
\forall n\in[N]$.

Sufficient Condition of (10):
In order to show (10) holds almost surely, it suffices to show that

$$ $\displaystyle\forall\overline{\boldsymbol{c}}\in\widehat{\mathcal{C}},~{} \exists~{}\mathcal{B}_{\rm C}\neq\phi\text{ and }\mathcal{B}_{\rm C}\subseteq \mathcal{C}$ $\displaystyle\text{s.t.}~{}\boldsymbol{h}_{\rm C}^{-1}(\overline{\boldsymbol{c }})=\mathcal{B}_{\rm C}\times\mathcal{S}.$ (12) $$

Eq. (C.1) implies that no matter what the value of the private variable $\boldsymbol{s}^{(n)}$ is, as long as the shared variable $\boldsymbol{c}\in\mathcal{B}_{\rm C}$, the extracted $\widehat{\boldsymbol{c}}=\overline{\boldsymbol{c}}$. This implies that small changes in $s_{j}\forall j\in[d_{\rm S}]$, while keeping all other variables fixed, does not result in any change in $\widehat{c}_{i},\forall i\in[d_{\rm C}]$—which means that (10) holds.

Further, the following condition is sufficient to ensure that (C.1) holds:

$$ $\displaystyle\forall\overline{\boldsymbol{c}}\in\widehat{\mathcal{C}},\forall \epsilon>0,~{}\exists~{}\mathcal{G}_{\rm C}\neq\phi\text{ and }\mathcal{G}_{ \rm C}\subseteq\mathcal{C},$ $\displaystyle\boldsymbol{h}_{\rm C}^{-1}(\mathcal{N}_{\epsilon}(\overline{ \boldsymbol{c}}))=\mathcal{G}_{\rm C}\times\mathcal{S},$ (13) $$

where $\mathcal{N}_{\epsilon}(\overline{\boldsymbol{c}})=\{\boldsymbol{c}^{\prime}\in
\widehat{\mathcal{C}}~{}|~{}\|\overline{\boldsymbol{c}}-\boldsymbol{c}^{\prime
}\|_{2}<\epsilon\}$ is an open set.
To see how (C.1) implies (C.1),
we use contradiction.
Suppose that (C.1) does not imply (C.1).
Then there exists an $\widetilde{n}\in[N]$ and $\widetilde{\boldsymbol{z}}=(\widetilde{\boldsymbol{c}},\widetilde{\boldsymbol{
s}}^{(\widetilde{n})})\in\mathcal{Z}$ with

$$ $\widetilde{\boldsymbol{c}}\in\mathcal{B}_{\rm C}:=\{\boldsymbol{z}_{1:d_{\rm C }}:\boldsymbol{z}\in\boldsymbol{h}_{\rm C}^{-1}(\overline{\boldsymbol{c}})\},$ $$

and $\widetilde{\boldsymbol{s}}^{(\widetilde{n})}\in\mathcal{S}$ such that $\boldsymbol{h}_{\rm C}(\widetilde{\boldsymbol{z}})\neq\overline{\boldsymbol{c}}$. Since $\boldsymbol{h}_{\rm C}(\cdot)$ is a continuous function there exists $\widehat{\epsilon}>0$ such that

$$ $\displaystyle\boldsymbol{h}_{\rm C}(\widetilde{\boldsymbol{z}})\not\in\mathcal {N}_{\widehat{\epsilon}}(\overline{\boldsymbol{c}})$ $\displaystyle\implies$ $\displaystyle\widetilde{\boldsymbol{z}}\not\in\boldsymbol{h}_{\rm C}^{-1}( \mathcal{N}_{\widehat{\epsilon}}(\overline{\boldsymbol{c}}))$ $$

However, (C.1) states that

$$ $\boldsymbol{h}_{\rm C}^{-1}(\mathcal{N}_{\widehat{\epsilon}}(\overline{ \boldsymbol{c}}))=\mathcal{G}_{\rm C}\times\mathcal{S},$ $$

for some $\mathcal{G}_{\rm C}\subseteq\mathcal{C}$. Since $\mathcal{B}_{\rm C}\subseteq\mathcal{G}_{\rm C}$ by their respective definitions, it implies that $\widetilde{\boldsymbol{c}}\in\mathcal{G}_{\rm C}$. And therefore,

$$ $(\widetilde{\boldsymbol{c}},\widetilde{\boldsymbol{s}}^{(\widetilde{n})})\in \boldsymbol{h}^{-1}_{\rm C}(\mathcal{N}_{\widehat{\epsilon}}(\overline{ \boldsymbol{c}})),$ $$

which is a contradiction.
Hence, (C.1) implies (C.1). Therefore, it is sufficient to show (C.1) in order to prove (10).

Proving (10):
Proving (C.1) (equivalently (10)) can be accomplished using contradition. From (C.1), it is sufficient to show that $\boldsymbol{h}_{\rm C}^{-1}(\mathcal{A}_{\rm C})=\mathcal{B}_{\rm C}\times
\mathcal{S}$ for a certain $\mathcal{B}_{\rm C}\subseteq\mathcal{C}$.

Suppose that $\boldsymbol{h}_{\rm C}^{-1}(\mathcal{A}_{\rm C}))\neq\mathcal{B}_{\rm C}\times
\mathcal{S}$ for any $\mathcal{B}_{\rm C}\subseteq\mathcal{C}$. Then, we can divide $\boldsymbol{h}_{\rm C}^{-1}(\mathcal{A}_{\rm C})$ into two disjoint subsets, namely, $\mathcal{G}$ and $\mathcal{F}$. The $\mathcal{G}$ part can be represented as $\mathcal{B}_{\rm C}\times\mathcal{S}$ for a certain set $\mathcal{B}_{\rm C}\subseteq\mathcal{C}$ (it is possible that $\mathcal{G}=\phi$). The $\mathcal{F}$ part cannot be written as $\mathcal{D}_{\rm C}\times\mathcal{S}$ for any $\mathcal{D}_{\rm C}\subseteq\mathcal{C}$. Note that $\mathcal{F}\neq\phi$ due to the fact that $\boldsymbol{h}_{\rm C}^{-1}(\mathcal{A}_{\rm C})\neq\mathcal{B}_{\rm C}\times
\mathcal{S}$, and has a strictly positive measure. Then, the following follows from (C.1):

$$ $\displaystyle\mathbb{P}_{\boldsymbol{z}^{(i)}}\left[\boldsymbol{h}_{\rm C}^{-1 }(\mathcal{A}_{\rm C})\right]=\mathbb{P}_{\boldsymbol{z}^{(j)}}\left[ \boldsymbol{h}_{\rm C}^{-1}(\mathcal{A}_{\rm C})\right]$ $\displaystyle\iff$ $\displaystyle\mathbb{P}_{\boldsymbol{z}^{(i)}}[\mathcal{G}]+\mathbb{P}_{ \boldsymbol{z}^{(i)}}[\mathcal{F}]=\mathbb{P}_{\boldsymbol{z}^{(j)}}[\mathcal{ G}]+\mathbb{P}_{\boldsymbol{z}^{(j)}}[\mathcal{F}]$ $\displaystyle\iff$ $\displaystyle\mathbb{P}_{\boldsymbol{c}}[\mathcal{B}_{\rm C}]\mathbb{P}_{ \boldsymbol{s}^{(i)}|\boldsymbol{c}}[\mathcal{S}|\mathcal{B}_{\rm C}]+\mathbb{ P}_{\boldsymbol{z}^{(i)}}[\mathcal{F}]=\mathbb{P}_{\boldsymbol{c}}[\mathcal{B} _{\rm C}]\mathbb{P}_{\boldsymbol{s}^{(j)}|\boldsymbol{c}}[\mathcal{S}|\mathcal {B}_{\rm C}]$ $\displaystyle+\mathbb{P}_{\boldsymbol{z}^{(j)}}[\mathcal{F}]$ $\displaystyle\stackrel{{\scriptstyle(a)}}{{\iff}}$ $\displaystyle\mathbb{P}_{\boldsymbol{z}^{(i)}}[\mathcal{F}]=\mathbb{P}_{ \boldsymbol{z}^{(j)}}[\mathcal{F}]$ (14) $$

where (a) holds because we have $\mathbb{P}_{\boldsymbol{s}^{(n)}|\boldsymbol{c}}[\mathcal{S}|\mathcal{B}_{\rm C
}]=1,\forall n\in[N]$.
However, (C.1) cannot hold due to Assumption 3.2. This is a contradication.

Step 2:

Consider the Jacobian of $\boldsymbol{h}$

$$ $\displaystyle\boldsymbol{J}_{\boldsymbol{h}}(\boldsymbol{c},\boldsymbol{s})= \begin{bmatrix}\boldsymbol{J}_{\boldsymbol{h}_{\rm C}}(\boldsymbol{c})& \boldsymbol{J}_{\boldsymbol{h}_{\rm C}}(\boldsymbol{s})\\ \boldsymbol{J}_{\boldsymbol{h}_{\rm S}}(\boldsymbol{c})&\boldsymbol{J}_{ \boldsymbol{h}_{\rm S}}(\boldsymbol{s})\end{bmatrix}.$ (15) $$

In step 1, we have shown that $\boldsymbol{J}_{\boldsymbol{h}_{\rm C}}(\boldsymbol{s})=\boldsymbol{0}$. This implies that

$$ $\displaystyle\widehat{\boldsymbol{c}}^{(n)}=\boldsymbol{\gamma}(\boldsymbol{c} ),\forall n\in[N]$ (16) $$

for some function $\boldsymbol{\gamma}:\mathcal{C}\to\mathbb{R}^{d_{\rm C}}$. Now, we want to show that $\boldsymbol{\gamma}$ is injective.

Note that $\boldsymbol{h}:\mathcal{Z}\to\mathbb{R}^{d_{\rm C}+d_{\rm S}}$ is a differentiable injective function because it is a composition of differentiable injection $\widehat{\boldsymbol{f}}$ and bijection $\boldsymbol{g}$.

This implies that $|{\rm det}(\boldsymbol{J}_{\boldsymbol{h}}(\boldsymbol{c},\boldsymbol{s}))|=|{
\rm det}(\boldsymbol{J}_{\boldsymbol{h}_{\rm C}}(\boldsymbol{c}))||{\rm det}(
\boldsymbol{J}_{\boldsymbol{h}_{\rm S}}(\boldsymbol{s}))|\neq 0,\forall(
\boldsymbol{c},\boldsymbol{s})\in\mathcal{C}\times\mathcal{S}$. Hence $|{\rm det}(\boldsymbol{J}_{\boldsymbol{h}_{\rm C}}(\boldsymbol{c}))|\neq 0,
\forall\boldsymbol{c}\in\mathcal{C}$, which in turn implies that $\boldsymbol{\gamma}$ is injective.

### C.2 Style Identification

Our goal is to show that $\boldsymbol{h}_{\rm S}(\boldsymbol{c},\boldsymbol{s}^{(n)})=\boldsymbol{\delta
}(\boldsymbol{s}^{(n)})$ for some injective function $\boldsymbol{\delta}$.
As before, we first show that $\boldsymbol{h}_{\rm S}(\boldsymbol{c},\boldsymbol{s}^{(n)})$ is only a function of $\boldsymbol{s}^{(n)}$, i.e., $\boldsymbol{h}_{\rm S}(\boldsymbol{c},\boldsymbol{s}^{(n)})=\boldsymbol{\delta
}(\boldsymbol{s}^{(n)})$ for some function $\boldsymbol{\delta}$. Then, we show that $\boldsymbol{\delta}$ has to be injective.

To proceed, note that the statistical independence constraint $\widehat{\boldsymbol{c}}^{(n)}\rotatebox[origin={c}]{90.0}{$\models$}\widehat{
\boldsymbol{s}}^{(n)}$ implies the following:

$$ $\displaystyle\widehat{\boldsymbol{c}}^{(n)}\rotatebox[origin={c}]{90.0}{$ \models$}\widehat{\boldsymbol{s}}^{(n)}$ $\displaystyle\implies$ $\displaystyle p(\widehat{\boldsymbol{c}}^{(n)},\widehat{\boldsymbol{s}}^{(n)}) =p(\widehat{\boldsymbol{c}}^{(n)})p(\widehat{\boldsymbol{s}}^{(n)})$ $\displaystyle\implies$ $\displaystyle I(\widehat{\boldsymbol{s}}^{(n)};\widehat{\boldsymbol{c}}^{(n)}) =0,$ $$

where $I(X;Y)$ denotes the mutual information between $X$ and $Y$.
Since we have already established $\widehat{\boldsymbol{c}}^{(n)}=\boldsymbol{\gamma}(\boldsymbol{c})$ for some injective function $\boldsymbol{\gamma}$, we express $\boldsymbol{c}=\boldsymbol{\gamma}^{-1}(\widehat{\boldsymbol{c}}^{(n)})$ (where $\boldsymbol{\gamma}^{-1}$ is the left inverse of $\boldsymbol{\gamma}$; see Appendix A.1).
Then, $\widehat{\boldsymbol{s}}^{(n)}\to\widehat{\boldsymbol{c}}^{(n)}\to\boldsymbol{
\gamma}^{-1}(\widehat{\boldsymbol{c}}^{(n)})=\boldsymbol{c}$ is a Markov chain. This is because, when conditioned on $\widehat{\boldsymbol{c}}^{(n)}$, $\boldsymbol{c}=\boldsymbol{\gamma}^{-1}(\widehat{\boldsymbol{c}}^{(n)})$ is a constant, which is independent with $\widehat{\boldsymbol{s}}^{(n)}$.
Next, the data processing inequality in information theory (Cover, , Theorem 2.8.1) (and corollary following the theorem) implies that

$$ $I(\widehat{\boldsymbol{s}}^{(n)};\widehat{\boldsymbol{c}}^{(n)})\geq I( \widehat{\boldsymbol{s}}^{(n)};\boldsymbol{\gamma}^{-1}(\widehat{\boldsymbol{c }}^{(n)}))=I(\widehat{\boldsymbol{s}}^{(n)};\boldsymbol{c})$ $$

As mutual information is always non-negative, we have $I(\widehat{\boldsymbol{s}}^{(n)};\boldsymbol{c})=0$. This implies that $\boldsymbol{h}_{\rm S}(\boldsymbol{c},\boldsymbol{s}^{(n)})$ does not depend upon $\boldsymbol{c}$.
In other words, as pointed out in (Eastwood et al., , Theorem 2, Step 3), we have
$\boldsymbol{h}_{\rm S}(\boldsymbol{c},\boldsymbol{s}^{(n)})=\boldsymbol{\delta
}(\boldsymbol{s}^{(n)}),\forall n$ for some function $\boldsymbol{\delta}$.

Finally, to see that $\boldsymbol{\delta}$ is injective, we use contradiction. Suppose that $\boldsymbol{\delta}$ is not injective. Then there exists $\boldsymbol{s}_{1},\boldsymbol{s}_{2}\in\mathcal{S}$ and $\boldsymbol{s}_{1}\neq\boldsymbol{s}_{2}$ such that

$$ $\displaystyle\boldsymbol{\delta}(\boldsymbol{s}_{1})=\boldsymbol{\delta}( \boldsymbol{s}_{2}).$ (17) $$

However,

$$ $\boldsymbol{h}(\boldsymbol{c},\boldsymbol{s}_{1})=(\boldsymbol{h}_{\rm C}( \boldsymbol{c},\boldsymbol{s}_{1}),\boldsymbol{h}_{\rm S}(\boldsymbol{c}, \boldsymbol{s}_{1}))=(\boldsymbol{\gamma}(\boldsymbol{c}),\boldsymbol{\delta}( \boldsymbol{s}_{1}))$ $$

Similarly,

$$ $\boldsymbol{h}(\boldsymbol{c},\boldsymbol{s}_{2})=(\boldsymbol{h}_{\rm C}( \boldsymbol{c},\boldsymbol{s}_{2}),\boldsymbol{h}_{\rm S}(\boldsymbol{c}, \boldsymbol{s}_{2}))=(\boldsymbol{\gamma}(\boldsymbol{c}),\boldsymbol{\delta}( \boldsymbol{s}_{2}))$ $$

However, Eq. (17) implies that

$$ $\boldsymbol{h}(\boldsymbol{c},\boldsymbol{s}_{1})=\boldsymbol{h}(\boldsymbol{c },\boldsymbol{s}_{2})$ $$

which is a contradiction to the injectivity of $\boldsymbol{h}$. Hence, $\boldsymbol{\delta}$ is an injective function.

∎

## Appendix D Proof of Theorem 3.4

The theorem is re-stated as follows:

[backgroundcolor=gray!10,topline=false,
rightline=false,
leftline=false,
bottomline=false]
Theorem 3.4
Assume that the conditions in Theorem 3.3 hold. Let $\widehat{\boldsymbol{f}}$ represent any solution of Problem (4).
Assume the following conditions hold: (a) $\widehat{d}_{\rm C}\geq d_{\rm C}$ and $\widehat{d}_{\rm S}\geq d_{\rm S}$.
(b) $0<p_{\boldsymbol{z}^{(n)}}(\boldsymbol{z})<\infty,\forall\boldsymbol{z}\in
\mathcal{Z}=\mathcal{C}\times\mathcal{S},\forall n\in[N]$.
Then, there exists injective functions $\boldsymbol{\gamma}:\mathcal{C}\to\mathbb{R}^{\widehat{d}_{\rm C}}$ and $\boldsymbol{\delta}:\mathcal{S}\to\mathbb{R}^{\widehat{d}_{\rm S}},\forall n
\in[N]$ such that $\widehat{\boldsymbol{c}}=\widehat{\boldsymbol{f}}_{\rm C}(\boldsymbol{x}^{(n)}
)=\boldsymbol{\gamma}(\boldsymbol{c})$ and $\widehat{\boldsymbol{f}}_{\rm S}(\boldsymbol{x}^{(n)})=\boldsymbol{\delta}(
\boldsymbol{s}^{(n)}),\forall n\in[N]$.

Proof Outline. The proof of Theorem 3.4 follows the following pipeline

- 1.
Content identification:
Step 1
Showing that the extracted content $\widehat{\boldsymbol{c}}^{(n)}$ does not depend upon the style part $\boldsymbol{s}^{(n)}$.
Step 2
Showing that the sparsity regularization forces the extracted style part to be free of any content information; and that, consequently, the extracted content has to be an injective function of the true content.
- 2.
Style identification: showing that the extracted style is an injective function of the true style.

We will fist show that unknown dimensionality does not affect Step 1 of content identification; i.e., when $\widehat{d}_{\rm C}\geq d_{\rm C}$, $\widehat{\boldsymbol{c}}^{(n)}$ will be independent of $\boldsymbol{s}^{(n)}$ using the arguments from the proof of Theorem 3.3.
The most important part of the proof is Step 2 of content identification, which requires completely different analytical approaches compared to that used for Theorem 3.3. For this, we first use contradiction to show that if $\boldsymbol{\gamma}$ were not injective, the injectivity of $\boldsymbol{h}$ will imply that $\boldsymbol{h}_{\rm S}$ depends upon $\boldsymbol{c}$ within a set of strictly positive measure. This will imply that the extracted style contains information from both the style as well as the content part, which will require $\mathbb{E}\|\widehat{\boldsymbol{s}}^{(n)}_{0}\|>d_{\rm S}$. However, a feasible solution to Problem (4) can be constructed using the inverse of $\boldsymbol{g}$ which has $\mathbb{E}\|\widehat{\boldsymbol{s}}^{(n)}_{0}\|=d_{\rm S}$.

This will contradict our assumption that $\widehat{\boldsymbol{f}}$ is an optimal solution to (4). Hence, $\boldsymbol{\gamma}$ should be injective.
Finally, style identification follows the same proof as that of Theorem 3.3.

###### Proof.

The proof is as follows:

Content Identification.

Step 1:
We want to show that

$$ $\displaystyle\frac{\partial\widehat{c}_{i}^{(n)}}{\partial s_{j}^{(n)}}=0, \forall i\in[\widehat{d}_{\rm C}],j\in[d_{\rm S}],$ (18) $$

which will imply that $\boldsymbol{h}_{\rm C}(\boldsymbol{c},\boldsymbol{s})=\boldsymbol{\gamma}(
\boldsymbol{c})$ for some function $\boldsymbol{\gamma}$.

For this, the proof of Step 1 of content identifiability of Theorem 3.3 holds without any modification since all arguments of the proof are dimension-agnostic.

Step 2:
We want to show that $\boldsymbol{\gamma}$ is an injective function. For the sake of contradiction, assume that $\boldsymbol{\gamma}$ is not injective. Then there exists $\overline{\boldsymbol{c}}\in\mathcal{C}$ and $\widetilde{\boldsymbol{c}}\in\mathcal{C}$, $\overline{\boldsymbol{c}}\neq\widetilde{\boldsymbol{c}}$, such that $\boldsymbol{\gamma}(\overline{\boldsymbol{c}})=\boldsymbol{\gamma}(\widetilde{
\boldsymbol{c}})$. This is equivalent to

$$ $\displaystyle\boldsymbol{h}_{\rm C}(\overline{\boldsymbol{c}},\boldsymbol{s})$ $\displaystyle=\boldsymbol{h}_{\rm C}(\widetilde{\boldsymbol{c}},\boldsymbol{s}).$ (19) $$

However, we know that $\boldsymbol{h}:\mathcal{Z}\to\mathbb{R}^{\widehat{d}_{\rm C}+\widehat{d}_{\rm S}}$ is an injective function, since it is a composition of injective and bijective function.
This implies that

$$ $\displaystyle\boldsymbol{h}(\overline{\boldsymbol{c}},\boldsymbol{s})\neq \boldsymbol{h}(\widetilde{\boldsymbol{c}},\boldsymbol{s})$ (20) $$

Eq. (19) and (20) imply that

$$ $\displaystyle\boldsymbol{h}_{\rm S}(\overline{\boldsymbol{c}},\boldsymbol{s})$ $\displaystyle\neq\boldsymbol{h}_{\rm S}(\widetilde{\boldsymbol{c}},\boldsymbol {s})$ (21) $$

Since $\boldsymbol{h}_{\rm S}$ is a differentiable function, (21) implies that there exists $\ell\in[\widehat{d}_{\rm S}],r\in[d_{\rm C}]$ and $\boldsymbol{c}^{\star}\in\mathcal{C},\boldsymbol{s}^{\star}\in\mathcal{S}$, such that

$$ $\displaystyle\frac{\partial[h_{\rm S}]_{\ell}}{\partial c_{r}}(\boldsymbol{c}^ {\star},\boldsymbol{s}^{\star})\neq 0.$ (22) $$

Also, $\boldsymbol{h}_{\rm S}$ being differentiable implies that the above partial derivative is continuous. Hence, there exists $\epsilon>0$, such that the above non-equality holds on $\mathcal{N}_{\epsilon}(\boldsymbol{z}^{\star})\subseteq\mathcal{Z}$, where $\boldsymbol{z}^{\star}=(\boldsymbol{c}^{\star},\boldsymbol{s}^{\star})$.
In addition, with a small enough $\epsilon$,
because of the continuity of $\frac{\partial[h_{\rm S}]_{\ell}}{\partial c_{r}}$, $[h_{\rm S}]_{\ell}$ is strictly monotonic (either increasing or decreasing) within the set $\mathcal{N}_{\epsilon}(\boldsymbol{z}^{\star})$, with respect to $c_{r}$.
This implies that $[h_{\rm S}]_{\ell}$ is a locally monotonic function w.r.t. its input $c_{r}$ at any $\overline{\boldsymbol{z}}\in\mathcal{N}_{\epsilon}(\boldsymbol{z}^{\star})$.

However, this will imply $[h_{\rm C}]_{i},\forall i\in[\widehat{d}_{\rm C}]$ cannot depend upon $c_{r}$ when restricted to $\mathcal{N}_{\epsilon}(\boldsymbol{z}^{\star})$, i.e., $\forall\overline{\boldsymbol{z}}\in\mathcal{N}_{\epsilon}(\boldsymbol{z}^{
\star})$ and all $i\in[\widehat{d}_{\rm C}]$,

$$ $\displaystyle\frac{\partial[h_{\rm C}]_{i}}{\partial c_{r}}(\overline{ \boldsymbol{z}})=0.$ (23) $$

This is because, if $\frac{\partial[h_{\rm C}]_{i}}{\partial c_{r}}(\overline{\boldsymbol{z}})\neq 0$ for some $\overline{\boldsymbol{z}}\in\mathcal{N}_{\epsilon}(\boldsymbol{z}^{\star})$ and $i\in[\widehat{d}_{\rm C}]$, then there exists a $\eta>0$ such that $[h_{\rm C}]_{i}$ is strictly monotonic within $\mathcal{N}_{\eta}(\overline{\boldsymbol{z}})$, with respect to $c_{r}$.

Consider the non-empty open set $\mathcal{R}=\mathcal{N}_{\epsilon}(\boldsymbol{z}^{\star})\cap\mathcal{N}_{
\eta}(\overline{\boldsymbol{z}})\cap\mathcal{Z}$. Due to the assumption that $0<p_{\boldsymbol{z}^{(n)}}(\boldsymbol{z})<\infty,\forall\boldsymbol{z}\in
\mathcal{Z}$, $\mathcal{R}$ has strictly positive measure. However, $[h_{\rm C}]_{i}$ and $[h_{\rm S}]_{\ell}$ are simultaneously changing monotonically everywhere in $\mathcal{R}$ with respect $c_{r}$, which implies that $[h_{\rm S}]_{\ell}\rotatebox[origin={c}]{90.0}{$\models$}[h_{\rm C}]_{i}$ does not hold. This contradicts the independence constraint (2b). Hence (23) has to hold.

Now, Eq. (23) implies that

$$ $\boldsymbol{h}_{\rm C}(\overline{\boldsymbol{c}}_{-r},\overline{c}_{r}, \overline{\boldsymbol{s}})=\boldsymbol{h}_{\rm C}(\overline{\boldsymbol{c}}_{- r},\widetilde{c}_{r},\widetilde{\boldsymbol{s}}),$ $$

for any two points $(\overline{\boldsymbol{c}},\overline{\boldsymbol{s}}),(\widetilde{\boldsymbol{
c}},\widetilde{\boldsymbol{s}})\in\mathcal{N}_{\epsilon}({\boldsymbol{z}^{
\star}})$. Here, $\overline{\boldsymbol{c}}_{-r}$ represents all components of $\overline{\boldsymbol{c}}$ excluding $\overline{c}_{r}$. Next, the injectivity of $\boldsymbol{h}$ indicates that for any $(\overline{\boldsymbol{c}},\overline{\boldsymbol{s}}),(\widetilde{\boldsymbol{
c}},\widetilde{\boldsymbol{s}})\in\mathcal{N}_{\epsilon}({\boldsymbol{z}^{
\star}})$, if $\overline{c}_{r}\neq\widetilde{c}_{r}$ or $\overline{\boldsymbol{s}}\neq\widetilde{\boldsymbol{s}}$, then

$$ $\displaystyle\boldsymbol{h}_{\rm S}(\overline{\boldsymbol{c}}_{-r},\overline{c }_{r},\overline{\boldsymbol{s}})\neq\boldsymbol{h}_{\rm S}(\overline{ \boldsymbol{c}}_{-r},\widetilde{c}_{r},\widetilde{\boldsymbol{s}}).$ (24) $$

Next, we show that Eq. (24) results in a contradiction to the fact that $\widehat{\boldsymbol{f}}$ is an optimal solution of (4). This contradiction will conclude that $\boldsymbol{\gamma}$ is injective, as Eq. (24) was obtained by the assumption that $\boldsymbol{\gamma}$ is not injective.

Note that one can construct a feasible solution $\widetilde{\boldsymbol{f}}$ of Problem (4) satisfying the following:

$$ $\widetilde{\boldsymbol{f}}_{\rm C}(\boldsymbol{x}^{(n)})=\begin{bmatrix}[ \boldsymbol{g}^{-1}]_{\rm C}(\boldsymbol{x}^{(n)})\\ 0\\ \vdots\\ 0\end{bmatrix}\in\mathbb{R}^{\widehat{d}_{\rm C}}\quad\text{and}\quad \widetilde{\boldsymbol{f}}_{\rm S}(\boldsymbol{x}^{(n)})=\begin{bmatrix}[ \boldsymbol{g}^{-1}]_{\rm S}(\boldsymbol{x}^{(n)})\\ 0\\ \vdots\\ 0\end{bmatrix}\in\mathbb{R}^{\widehat{d}_{\rm S}},\forall n\in[N],\boldsymbol{ x}^{(n)}\in\mathcal{X}^{(n)},$ $$

which implies that $\forall n$,

$$ $\displaystyle\mathbb{E}[\|\widetilde{\boldsymbol{f}}_{\rm S}(\boldsymbol{x}^{( n)})\|_{0}]=d_{\rm S}.$ (25) $$

This means that the optimal value of Problem (4b) is smaller than or equal to $d_{\rm S}$, as at least one solution can be constructed to attain this value.

Now, consider our learned $\widehat{\boldsymbol{s}}^{(n)}=[\widehat{\boldsymbol{f}}\circ\boldsymbol{g}(
\boldsymbol{x}^{(n)})]_{\rm S}=\boldsymbol{h}_{\rm S}(\boldsymbol{x}^{(n)})$. We will show the following chain of inequalities $\forall n\in[N]$:

$$ $\displaystyle\mathbb{E}_{\widehat{\boldsymbol{s}}\sim\mathbb{P}_{\widehat{ \boldsymbol{s}}^{(n)}}}[\|\widehat{\boldsymbol{s}}\|_{0}]$ $\displaystyle=\int_{\boldsymbol{h}_{\rm S}(\mathcal{Z})}\|\widehat{\boldsymbol {s}}\|_{0}~{}d\mathbb{P}_{\widehat{\boldsymbol{s}}^{(n)}}$ $\displaystyle=\int_{\boldsymbol{h}_{\rm S}(\mathcal{Z})\backslash\boldsymbol{h }_{\rm S}(\mathcal{N}_{\epsilon}(\boldsymbol{z}^{\star}))}\|\widehat{ \boldsymbol{s}}\|_{0}~{}d\mathbb{P}_{\widehat{\boldsymbol{s}}^{(n)}}+\int_{ \boldsymbol{h}_{\rm S}(\mathcal{N}_{\epsilon}(\boldsymbol{z}^{\star}))}\| \widehat{\boldsymbol{s}}\|_{0}~{}d\mathbb{P}_{\widehat{\boldsymbol{s}}^{(n)}}$ $\displaystyle\stackrel{{\scriptstyle(a)}}{{\geq}}\int_{\boldsymbol{h}_{\rm S}( \mathcal{Z})\backslash\boldsymbol{h}_{\rm S}(\mathcal{N}_{\epsilon}( \boldsymbol{z}^{\star}))}d_{\rm S}~{}d\mathbb{P}_{\widehat{\boldsymbol{s}}^{(n )}}+\int_{\boldsymbol{h}_{\rm S}(\mathcal{N}_{\epsilon}(\boldsymbol{z}^{\star} ))}(d_{\rm S}+1)~{}d\mathbb{P}_{\widehat{\boldsymbol{s}}^{(n)}}$ $\displaystyle=d_{\rm S}(\mathbb{P}_{\widehat{\boldsymbol{s}}^{(n)}}[ \boldsymbol{h}_{\rm S}(\mathcal{Z})]-\mathbb{P}_{\widehat{\boldsymbol{s}}^{(n) }}[\boldsymbol{h}_{\rm S}(\mathcal{N}_{\epsilon}(\boldsymbol{z}^{\star}))])+(d _{\rm S}+1)\mathbb{P}_{\widehat{\boldsymbol{s}}^{(n)}}[\boldsymbol{h}_{\rm S}( \mathcal{N}_{\epsilon}(\boldsymbol{z}^{\star}))]$ $\displaystyle=d_{\rm S}+\mathbb{P}_{\widehat{\boldsymbol{s}}^{(n)}}[ \boldsymbol{h}_{\rm S}(\mathcal{N}_{\epsilon}(\boldsymbol{z}^{\star}))]$ $\displaystyle>d_{\rm S}.$ (26) $$

where $(a)$ used the to-be-proven facts that $\|\widehat{\boldsymbol{s}}\|_{0}\geq d_{\rm S}$ almost everywhere on $\boldsymbol{h}_{\rm S}(\mathcal{Z})$ and that $\|\widehat{\boldsymbol{s}}\|_{0}\geq d_{\rm S}+1$ almost everywhere on $\boldsymbol{h}_{\rm S}(\mathcal{N}_{\epsilon}(\boldsymbol{z}^{\star}))$.
Note that if (26) holds, we reach a contradiction that $\widehat{\boldsymbol{f}}$ is an optimal solution of (4), as the solution in (25) can attain a smaller objective value.
We prove the facts used in (a) in the following:

First, to see that $\|\widehat{\boldsymbol{s}}\|_{0}\geq d_{\rm S},a.e.$ (on $\boldsymbol{h}_{\rm S}(\mathcal{Z})$), suppose for the sake of contradiction that $\|\widehat{\boldsymbol{s}}\|_{0}<d_{\rm S}$ on a set of strictly positive measure, say $\mathcal{Q}\subseteq\boldsymbol{h}_{\rm S}(\mathcal{Z})$, i.e.,

$$ $\displaystyle[\boldsymbol{h}_{\rm S}]_{\#\mathbb{P}_{\boldsymbol{z}^{(n)}}}( \mathcal{Q})>0$ $\displaystyle\implies\mathbb{P}_{\boldsymbol{z}^{(n)}}[\boldsymbol{h}_{\rm S}^ {-1}(\mathcal{Q})]>0.$ $$

Since $\mathbb{P}_{\boldsymbol{z}^{(n)}}$ admits a PDF with $0<p_{\boldsymbol{z}^{(n)}}(\boldsymbol{z})<\infty,\forall\boldsymbol{z}\in
\mathcal{Z}$, $\boldsymbol{h}_{\rm S}^{-1}(\mathcal{Q})$ should contain an open ball $\overline{\mathcal{Z}}\subseteq\mathcal{Z}$.
However ${\rm dim}(\overline{\mathcal{Z}})=d_{\rm C}+d_{\rm S}$ since $\overline{\mathcal{Z}}$ is homeomorphic to $\mathbb{R}^{d_{\rm C}+d_{\rm S}}$ (see Appendix A).
Let $\mathcal{A}_{q}=\{\boldsymbol{a}\in\mathbb{R}^{\widehat{d}_{\rm S}+\widehat{d}
_{\rm C}}~{}|~{}\|\boldsymbol{a}\|_{0}\leq q\}$, i.e., the set of all vectors with at most $q$ non-zero elements. Note that ${\rm dim}(\mathcal{A}_{q})=q$.
Then $\mathcal{Q}\subseteq\mathcal{A}_{d_{\rm S}-1}$. Hence ${\rm dim}(\mathcal{Q})\leq d_{\rm S}-1$.

Next, $\boldsymbol{h}$ is also an injective function from $\overline{\mathcal{Z}}$ to $\mathcal{Q}\times\boldsymbol{\gamma}(\mathcal{C})$. Note that ${\rm dim}(\boldsymbol{\gamma}(\mathcal{C}))\leq d_{\rm C}$ because $\boldsymbol{\gamma}$ is a continuous function and ${\rm dim}(\mathcal{C})=d_{\rm C}$.
But this implies that
${\rm dim}(\mathcal{Q}\times\boldsymbol{\gamma}(\mathcal{C}))={\rm dim}(
\mathcal{Q})+{\rm dim}(\gamma(\mathcal{C}))<d_{\rm C}+d_{\rm S}$.

However, this contradicts that $\boldsymbol{h}$ is an injective function from $\overline{\mathcal{Z}}$ to $\mathcal{Q}\times\boldsymbol{\gamma}(\mathcal{C})$ because the domain of $\boldsymbol{h}$ has larger dimension than the co-domain $\mathcal{Q}\times\boldsymbol{\gamma}(\mathcal{C})$. Hence $[\boldsymbol{h}_{\rm S}]_{\#\mathbb{P}_{\boldsymbol{z}^{(n)}}}(\mathcal{Q})=0$ and $\|\widehat{\boldsymbol{s}}\|_{0}\geq d_{\rm S},a.e.$

Second, to show that $\|\widehat{\boldsymbol{s}}\|_{0}\geq d_{\rm S}+1$ almost everywhere on $\boldsymbol{h}_{\rm S}(\mathcal{N}_{\epsilon}(\boldsymbol{z}^{\star}))$, consider the set $\mathcal{S}\times\mathcal{C}_{r}$, where $\mathcal{C}_{r}$ is the support of $c_{r}$. Eq. (24) implies that $\mathcal{M}=(\mathcal{S}\times\mathcal{C}_{r})\cap\mathcal{N}_{\epsilon}(
\boldsymbol{z}^{\star})$ is homeomorphic to a subset of $\boldsymbol{h}_{\rm S}(\mathcal{N}_{\epsilon}(\boldsymbol{z}^{\star}))$. To see this, consider a fixed $(\overline{\boldsymbol{c}},\overline{\boldsymbol{s}})\in\mathcal{N}_{\epsilon}
(\boldsymbol{z}^{\star})$. Then, define a continuous function

$$ $\boldsymbol{t}(c_{r},\boldsymbol{s})=\boldsymbol{h}_{\rm S}(\overline{ \boldsymbol{c}}_{-r},c_{r},\boldsymbol{s}),$ $$

i.e., $\boldsymbol{h}_{\rm S}$ with fixed partial input $\overline{\boldsymbol{c}}_{-r}$. Eq. (24) implies that $\boldsymbol{t}$ is an injective function over $\mathcal{M}$. Now, note that we have $\boldsymbol{t}(\mathcal{M})\subseteq\boldsymbol{h}_{\rm S}(\mathcal{N}_{
\epsilon}(\boldsymbol{z}^{\star}))$, and that $\boldsymbol{t}$ defines a homeomorphism between $\mathcal{M}$ and $\boldsymbol{t}(\mathcal{M})$. Hence $\mathcal{M}$ is homeomorphic to a subset of $\boldsymbol{h}_{\rm S}(\mathcal{N}_{\epsilon}(\boldsymbol{z}^{\star}))$. This implies that ${\rm dim}(\boldsymbol{h}_{\rm S}(\mathcal{N}_{\epsilon}(\boldsymbol{z}^{\star}
))\geq{\rm dim}(\mathcal{M})$. However, ${\rm dim}(\mathcal{M})=d_{\rm S}+1$ because $\mathcal{M}$ is an open set in $\mathbb{R}^{d_{\rm S}+1}$ as $\mathcal{S}$ is an open set in $\mathbb{R}^{d_{\rm S}}$, $\mathcal{C}_{r}$ is an open set in $\mathbb{R}$, and thus $(\mathcal{S}\times\mathcal{C}_{r})\cap\mathcal{N}_{\epsilon}(\boldsymbol{z}^{
\star})$ is an open set in $\mathbb{R}^{d_{\rm S}+1}$. Hence, ${\rm dim}(\boldsymbol{h}_{\rm S}(\mathcal{N}_{\epsilon}(\boldsymbol{z}^{\star}
))\geq{\rm dim}(\mathcal{M})=d_{\rm S}+1$. This concludes the proof of the inequalities in (26).

Finally $\mathbb{E}_{\widehat{\boldsymbol{s}}^{(n)}}[\|\widehat{\boldsymbol{s}}\|_{0}]>
d_{\rm S},\forall n\in[N]$ contradicts that $\widehat{\boldsymbol{f}}$ is an optimal solution to Problem (4). Hence, $\boldsymbol{\gamma}$ is an injective function.

Style Identification.

Finally, $\boldsymbol{h}_{\rm S}$ cannot depend upon $\boldsymbol{c}$ by the same reason as outlined in Sec. C.2. This implies that $\boldsymbol{h}_{\rm S}(\boldsymbol{c},\boldsymbol{s})=\boldsymbol{\delta}(
\boldsymbol{s})$, for some function $\boldsymbol{\delta}:\mathcal{S}\to\mathbb{R}^{\widehat{d}_{\rm S}}$. Similarly, $\boldsymbol{\delta}$ is injective by the same reason outlined in Sec. C.2.

This concludes the proof.

∎

###### Proof.

## Appendix E Proof of Theorem 4.1

The theorem is restated as follows:
{mdframed}[backgroundcolor=gray!10,topline=false,
rightline=false,
leftline=false,
bottomline=false]
Theorem 4.1
Let $(\widehat{\boldsymbol{q}},\widehat{\boldsymbol{e}}_{\rm C},\widehat{
\boldsymbol{e}}^{(n)}_{\rm S},\widehat{\boldsymbol{d}})$ be any differentiable optimal solution of (6).
Let $\mathcal{C}$ and $\mathcal{S}$ be simply connected open sets. Let $0<p_{\boldsymbol{z}^{(n)}}(\boldsymbol{z})<\infty,\forall\boldsymbol{z}\in
\mathcal{Z}=\mathcal{C}\times\mathcal{S}$.
Under the assumptions in Theorem 3.3, we have the following:

(a) If $\widehat{d}_{\rm C}=d_{\rm C}$ and $\widehat{d}_{\rm S}={d}_{\rm S}$ and (6b) is absent, then $\widehat{\boldsymbol{q}}:\widehat{\mathcal{C}}\times\widehat{\mathcal{S}}\to
\mathcal{X}$ is bijective and $\widehat{\boldsymbol{f}}=\widehat{\boldsymbol{q}}^{-1}$ is also a solution of (2).

(b) If $\widehat{d}_{\rm C}>d_{\rm C}$ and $\widehat{d}_{\rm S}>{d}_{\rm S}$ and $\widehat{\boldsymbol{q}}:\widehat{\mathcal{C}}\times\widehat{\mathcal{S}}\to
\mathcal{X}$ is bijective, then $\widehat{\boldsymbol{f}}=\widehat{\boldsymbol{q}}^{-1}$ is also a solution of (4).

### E.1 Proof of Part (a).

One can see that problem (6a) is equivalent to the following optimization problem:

$$ find $\displaystyle\boldsymbol{q},\boldsymbol{e}_{\rm C},\{\boldsymbol{e}_{\rm S}^{( n)}\}_{n=1}^{N}$ (27a) subject to $\displaystyle[\boldsymbol{q}]_{\#\mathbb{P}_{\boldsymbol{e}_{\rm C}\left( \boldsymbol{r}_{\rm C}\right),\boldsymbol{e}_{\rm S}^{(n)}\left(\boldsymbol{r} _{\rm S}^{(n)}\right)}}=\mathbb{P}_{\boldsymbol{x}^{(n)}},$ (27b) $\displaystyle\boldsymbol{r}_{\rm C}\sim\mathcal{N}(\boldsymbol{0},\boldsymbol{ I}_{d_{\rm C}}),\boldsymbol{r}_{\rm S}^{(n)}\sim\mathcal{N}(\boldsymbol{0}, \boldsymbol{I}_{d_{\rm S}}).$ (27c) $$

First, the solution of (6a) satisfies the distribution matching constraint (27b) because of (Goodfellow et al., ) [Theorem 1]. Next, define:

$$ $\boldsymbol{\alpha}^{(n)}(\boldsymbol{r}_{\rm C},\boldsymbol{r}_{\rm S}^{(n)}) =\widehat{\boldsymbol{q}}(\widehat{\boldsymbol{e}}_{\rm C}(\boldsymbol{r}_{\rm C }),\widehat{\boldsymbol{e}}_{\rm S}^{(n)}(\boldsymbol{r}_{\rm S}^{(n)})).$ $$

Consider the following theorem from (Zimmermann et al., ).

###### Proposition E.1 (Proposition 5 (Zimmermann et al., 2021 ) ) .

Let $\mathcal{M}$ and $\mathcal{N}$ be simply connected and oriented $\mathcal{C}^{1}$ manifolds without boundaries and $\boldsymbol{h}:\mathcal{M}\to\mathcal{N}$ be a differentiable map. Further, let the random variable $\boldsymbol{z}\in\mathcal{M}$ be distributed according to $\boldsymbol{z}\in p(\boldsymbol{z})$ for a regular function $p$, i.e.,$0<p<\infty$. If the push forward $p_{\#\boldsymbol{h}}(\boldsymbol{z})$ of $p$ through $\boldsymbol{h}$ is also a regular density, i.e., $0<p_{\#h}<\infty$, then $\boldsymbol{h}$ is a bijection.

In our case, $\mathcal{Z}$ and $\mathcal{X}$ is a simply connected and oriented $\mathcal{C}^{1}$ manifold without boundaries. In addition, $\boldsymbol{\alpha}^{(n)}:\mathbb{R}^{d_{\rm S}+d_{\rm C}}\to\mathcal{X}$ is the differentiable map that we hope to use the above proposition to characterize. Every element of the random vector $\boldsymbol{r}=(\boldsymbol{r}_{\rm C},\boldsymbol{r}_{\rm S}^{(n)})$ follows the standard normal distribution. Hence, it is readily seen that $0<p_{\boldsymbol{r}^{(n)}}(\boldsymbol{r})<\infty,\forall\boldsymbol{r}\in
\mathbb{R}^{d_{\rm C}+d_{\rm S}}$.
In addition, the assumptions that $0<p_{\boldsymbol{z}^{(n)}}(\boldsymbol{z})<\infty,\forall\boldsymbol{z}\in
\mathcal{Z}$ with simply-connected $\mathcal{Z}$ and that $\boldsymbol{g}$ is a differentiable bijection imply that $0<p_{\boldsymbol{x}^{(n)}}(\boldsymbol{x})<\infty,\forall\boldsymbol{x}\in
\mathcal{X},\forall n$ and that $\mathcal{X}$ is simply connected.

Following the above arguments, we can apply Proposition E.1 to conclude that $\boldsymbol{\alpha}^{(n)},\forall n\in[N]$ are bijections. This, in turn, implies that $\widehat{\boldsymbol{q}}$ is a surjection and $\widehat{\boldsymbol{e}}_{\rm C}$ and $\widehat{\boldsymbol{e}}_{\rm S}$ are injections. However, since $\widehat{\boldsymbol{q}}$ is defined on the range of $\widehat{\boldsymbol{e}}_{\rm C}$ and $\widehat{\boldsymbol{e}}_{\rm S}$, $\widehat{\boldsymbol{q}}$ is a bijection. Consequently, $\widehat{\boldsymbol{e}}_{\rm C}$ and $\widehat{\boldsymbol{e}}_{\rm S}$ are also bijections.

To proceed, notice that (27b) implies that $\widehat{\boldsymbol{f}}=\widehat{\boldsymbol{q}}^{-1}$ satisfies the following:

$$ $\displaystyle[\widehat{\boldsymbol{q}}^{-1}_{\rm C}]_{\#\mathbb{P}_{ \boldsymbol{x}^{(i)}}}=[\widehat{\boldsymbol{q}}^{-1}_{\rm C}]_{\#\mathbb{P}_{ \boldsymbol{x}^{(j)}}},\forall i,j\in[N],$ $$

where $\widehat{\boldsymbol{q}}^{-1}_{\rm C}$ is the output dimensions of $\widehat{\boldsymbol{q}}^{-1}$ that correspond to the content part.
To see the above equality, let $\widehat{\boldsymbol{x}}^{(n)}=\widehat{\boldsymbol{q}}(\widehat{\boldsymbol{e
}}_{c}(\boldsymbol{r}_{\rm C})),\widehat{\boldsymbol{e}}_{s}^{(n)}(\boldsymbol
{r}_{\rm S})$. Then,

$$ $\displaystyle\widehat{\boldsymbol{q}}^{-1}_{\rm C}(\widehat{\boldsymbol{x}}^{( n)})=\widehat{\boldsymbol{q}}^{-1}_{\rm C}\circ\widehat{\boldsymbol{q}}( \widehat{\boldsymbol{e}}_{c}(\boldsymbol{r}_{\rm C}),\widehat{\boldsymbol{e}}_ {s}^{(n)}(\boldsymbol{r}_{\rm S}))=\widehat{\boldsymbol{e}}_{c}(\boldsymbol{r} _{\rm C}).$ $$

Since $\widehat{\boldsymbol{x}}^{(n)}\sim\mathbb{P}_{\boldsymbol{x}^{(n)}}$, this implies that

$$ $\displaystyle[\widehat{\boldsymbol{q}}^{-1}_{\rm C}]_{\#\mathbb{P}_{ \boldsymbol{x}^{(n)}}}=[\widehat{\boldsymbol{e}}_{c}]_{\#\mathbb{P}_{ \boldsymbol{r}_{\rm C}}},\forall n\in[N]$ $$

Since the distribution $[\widehat{\boldsymbol{e}}_{c}]_{\#\mathbb{P}_{\boldsymbol{r}_{\rm C}}}$ is independent of $n$, $[\widehat{\boldsymbol{q}}^{-1}_{\rm C}]_{\#\mathbb{P}_{\boldsymbol{x}^{(n)}}},
\forall n\in[N]$ are matched. This means that $\widehat{\boldsymbol{q}}^{-1}$ satisfies the constraint (2a). Similarly, since $\widehat{\boldsymbol{e}}_{c}(\boldsymbol{r}_{\rm C})\rotatebox[origin={c}]{90.
0}{$\models$}\widehat{\boldsymbol{e}}_{s}^{(n)}(\boldsymbol{r}_{\rm S}^{(n)}),
\forall n\in[N]$,

$$ $\displaystyle[\widehat{\boldsymbol{q}}^{-1}_{\rm C}]_{\#\mathbb{P}_{ \boldsymbol{x}^{(n)}}}\rotatebox[origin={c}]{90.0}{$\models$}[\widehat{ \boldsymbol{q}}^{-1}_{\rm S}]_{\#\mathbb{P}_{\boldsymbol{x}^{(n)}}},\forall n \in[N].$ $$

Hence $\widehat{\boldsymbol{q}}^{-1}$ also satisfies the constraint (2b).
Therefore, $\widehat{\boldsymbol{q}}^{-1}$ is an optimal solution of Problem (2)

It remains to show that a solution to Problem (27) under our data generative model in (1) exists.
When $\mathcal{C}\subseteq\mathbb{R}^{d_{\rm C}}$ and $\mathcal{S}\subseteq\mathbb{R}^{d_{\rm S}}$ are open sets and simply connected, and $p_{\boldsymbol{c}}(\boldsymbol{c})>0$ and $p_{\boldsymbol{s}^{(n)}}(\boldsymbol{s})>0,\forall\boldsymbol{s}\in\mathcal{S}
,\boldsymbol{c}\in\mathcal{C},n\in[N]$, there exists a smooth bijective function that transforms standard normal distributions, i.e., $\mathcal{N}(\boldsymbol{0},\boldsymbol{I}_{d_{\rm C}})$ and $\mathcal{N}(\boldsymbol{0},\boldsymbol{I}_{d_{\rm S}})$, to the content and style distributions, $\mathbb{P}_{\boldsymbol{c}}$ and $\mathbb{P}_{\boldsymbol{s}^{(n)}}$, respectively. Such functions can be constructed by using Darmois construction (Darmois, ) to first convert the normal distribution to uniform distribution, and then using inverse Darmois construction to convert the uniform distribution to $\mathbb{P}_{\boldsymbol{c}}$ (and $\mathbb{P}_{\boldsymbol{s}^{(n)}}$) (see (Hyvärinen & Pajunen, ) for more details). Let $\boldsymbol{\kappa}_{\rm C}:\mathcal{N}(\boldsymbol{0},\boldsymbol{I}_{d_{\rm C
}})\to\mathcal{C}$ and $\boldsymbol{\kappa}_{\rm S}:\mathcal{N}(\boldsymbol{0},\boldsymbol{I}_{d_{\rm S
}})\to\mathcal{S}$ denote the aforementioned functions. Then, a solution to Problem (27) is $\boldsymbol{g}$, $\boldsymbol{\kappa}_{\rm C}$, and $\boldsymbol{\kappa}_{\rm S}^{(n)},\forall n\in[N]$ for the optimization functions $\boldsymbol{q}$, $\boldsymbol{e}_{\rm C}$, and ${\boldsymbol{e}}_{\rm S}^{(n)}\forall n\in[N]$, respectively. Hence, problem (27) is feasible.
This concludes the proof.

###### Proposition E.1 (Proposition 5 (Zimmermann et al., 2021 ) ) .

### E.2 Proof of Part (b).

Problem (6) is equivalent to the following problem:

$$ $\displaystyle\min_{\boldsymbol{q},\boldsymbol{e}_{\rm C},\{\boldsymbol{e}_{\rm S }^{(n)}\}_{n=1}^{N}}\quad$ $\displaystyle\frac{1}{N}\sum_{n=1}^{N}\mathbb{E}\left[\left\|\boldsymbol{e}_{ \rm S}^{(n)}(\boldsymbol{r}_{\rm S}^{(n)})\right\|_{0}\right]$ (28a) subject to $\displaystyle[\boldsymbol{q}]_{\#\mathbb{P}_{\boldsymbol{e}_{\rm C}( \boldsymbol{r}_{\rm C}),\boldsymbol{e}_{\rm S}^{(n)}(\boldsymbol{r}_{\rm S}^{( n)})}}=\mathbb{P}_{\boldsymbol{x}^{(n)}},$ (28b) $\displaystyle\boldsymbol{r}_{\rm C}\sim\mathcal{N}(\boldsymbol{0},\boldsymbol{ I}_{d_{\rm C}}),\boldsymbol{r}_{\rm S}^{(n)}\sim\mathcal{N}(\boldsymbol{0}, \boldsymbol{I}_{d_{\rm S}})$ (28c) $$

Following the same arguments as in the proof of Theorem Part (a), Eq. (28) implies that

$$ $\displaystyle[\widehat{\boldsymbol{q}}^{-1}_{\rm C}]_{\#\mathbb{P}_{ \boldsymbol{x}^{(i)}}}=[\widehat{\boldsymbol{q}}^{-1}_{\rm C}]_{\#\mathbb{P}_{ \boldsymbol{x}^{(j)}}},\forall i,j\in[N]$ $$

The above implies that

$$ $\displaystyle[\widehat{\boldsymbol{q}}^{-1}_{\rm C}]_{\#\mathbb{P}_{ \boldsymbol{x}^{(n)}}}=[\widehat{\boldsymbol{e}}_{c}]_{\#\mathbb{P}_{ \boldsymbol{r}_{\rm C}}},\forall n\in[N],$ $$

which means that $\widehat{\boldsymbol{q}}_{\rm C}^{-1}$ satisfies the constraint (4b).

Similarly, since $\widehat{\boldsymbol{e}}_{c}(\boldsymbol{r}_{\rm C})\rotatebox[origin={c}]{90.
0}{$\models$}\widehat{\boldsymbol{e}}_{s}^{(n)}(\boldsymbol{r}_{\rm S}^{(n)}),
\forall n\in[N]$,

$$ $\displaystyle[\widehat{\boldsymbol{q}}^{-1}_{\rm C}]_{\#\mathbb{P}_{ \boldsymbol{x}^{(n)}}}\rotatebox[origin={c}]{90.0}{$\models$}[\widehat{ \boldsymbol{q}}^{-1}_{\rm S}]_{\#\mathbb{P}_{\boldsymbol{x}^{(n)}}},\forall n \in[N].$ $$

Hence $\widehat{\boldsymbol{q}}^{-1}$ also satisfies the constraint (4c). Therefore, $\widehat{\boldsymbol{q}}^{-1}$ is also a solution to (4).

## Appendix F Benchmarked Multi-Domain Translation Schemes

StarGAN and StarGAN v2.
The representative multi-domain translation frameworks, namely, StarGAN (Choi et al., ) and StarGAN v2 (Choi et al., ), learn a common translation function that matches distribution between all pairs of domains. To be precise, let $\boldsymbol{t}:\mathcal{X}\times\mathcal{S}\to\mathcal{X}$ represent the translation function that takes in a sample $\boldsymbol{x}^{(s)}$ in the source domain $s$, and a style component $\boldsymbol{s}^{(t)}$ from the target domain $t$, and outputs the translated sample $\boldsymbol{x}^{(s\to t)}$ from the target domain $t$ as

$$ $\boldsymbol{x}^{(s\to t)}=\boldsymbol{t}(\boldsymbol{x}^{(s)},\boldsymbol{s}^{ (t)}).$ $$

In (Choi et al., ) $\boldsymbol{t}$ is learnt by optimizing the following criteria:

$$ $\displaystyle\mathcal{L}_{\rm StarGAN}=\mathcal{L}_{\rm match}+\mathcal{L}_{ \rm inv},$ $$

where $\mathcal{L}_{\rm inv}$ promotes invertibility of the translation function, and is composed of cycle-consistency, style diversity, and style invertibility terms, whereas $\mathcal{L}_{\rm match}$ promotes distribution matching between all pairs of domains after translation by $\boldsymbol{t}$. Specifically,

$$ $\displaystyle\mathcal{L}_{\rm match}=$ (29) $\displaystyle\sum_{s,t\in[N]\times[N]}{\mathbb{E}}\left[\log\boldsymbol{d}_{s} (\boldsymbol{x}^{(s)})+\log\left(1-\boldsymbol{d}_{t}(\boldsymbol{\tau}( \boldsymbol{x}^{(s)},\boldsymbol{s}^{(t)}))\right)\right],$ (30) $$

where $\boldsymbol{d}_{i}$ is the discriminator for the $i$th domain.
Here, all pairs of domains, $(s,t)\in[N]\times[N]$, are considered for distribution matching. This can be potentially expensive for large $N$.
The StarGAN loss matches $N(N-1)$ pairs of distributions for $N$ domains, which may not be affordable for large $N$.

Learning Pipeline of (Xie et al., ).
The work (Xie et al., ) proposed the following pipeline. First, an multi-domain generative model is trained, which is similar to our system.
Second, instead of directly using the trained system, the work proposed
to train a separate StarGAN with new regularization.
To be specific,
the work proposed to
use synthetic paired samples $(\boldsymbol{x}^{(s)},\boldsymbol{x}^{(t)})$ generated by their generative model to further regularize $\mathcal{L}_{\rm StarGAN}$, which leads to the following loss:

$$ $\mathcal{L}_{\rm StarGAN}+\mathcal{L}_{\rm sup}$ $$

where the second term is defined as

$$ $\mathcal{L}_{\rm sup}=\mathbb{E}_{\boldsymbol{x}^{(s)},\boldsymbol{x}^{(t)} \sim\mathbb{P}_{\widehat{\boldsymbol{\theta}}}}\|\boldsymbol{t}(\boldsymbol{x} ^{(s)},\boldsymbol{s}^{(t)})-\boldsymbol{x}^{(t)}\|,$ $$

in which $\mathbb{P}_{\widehat{\boldsymbol{\theta}}}$ represents distribution of the learned multi-domain generative model, and $\boldsymbol{s}^{(t)}$ is obtained from $\boldsymbol{x}^{(t)}$ by using a separate style encoder.

As we mentioned, as one can show that the generators are in fact invertible in the learned generative model (see Theorem 4.1), the above procedure can be circumvented. Using an off-the-shelf GAN inversion solver assists translation without retraining a regularization StarGAN.

## Appendix G Experimental Details

### G.1 Neural Network

We adopt the neural architecture of StyleGAN-ADA to represent the generator $\boldsymbol{q}$ as well as the discriminator $\boldsymbol{d}$ in (6). Specifically, StyleGAN-ADA generator consists of a mapping network and a synthesis network (Karras et al., ). We use the synthesis network to represent $\boldsymbol{q}$, whereas the mapping network is discarded. For each of $\boldsymbol{e}_{\rm C}$ and $\{\boldsymbol{e}_{\rm S}^{(n)}\}_{n=1}^{N}$, we use 3 layer MLP with 512 hidden units in each hidden layer. The input and output size of the MLPs for $\boldsymbol{e}_{\rm C}$ and $\{\boldsymbol{e}_{\rm S}^{(n)}\}_{n=1}^{N}$ are $\widehat{d}_{\rm C}=384$ and $\widehat{d}_{\rm S}=128$, respectively. We feed the output of $\boldsymbol{e}_{\rm C}$ to the first 5 layers of the synthesis network (i.e., $\boldsymbol{q}$), and the output of $\boldsymbol{e}_{\rm S}^{(n)},\forall n\in[N]$ are fed to the remaining layers of $\boldsymbol{q}$. Compared to (Xie et al., ), $\boldsymbol{e}_{\rm S}^{(n)},\forall n$ are much more simplified since they are no longer restricted to component-wise invertible functions, which require special neural network design.

### G.2 Training Hyperparameter Setting

The hyperparameters used for GAN training (6) is similar to those used in StyleGAN-ADA. Mainly, we use Adam (Kingma & Ba, ) with an initial learning rate of $0.0025$ with a batch size of 16. The hyperparameters in Adam that control the exponential decay rates of first and second order moments are set to $\beta_{1}=0$ and $\beta_{2}=0.99$, respectively. For all datasets, we train the networks for 300,000 iterations. Sparsity regularization weight in (6) is set to 0.3 for all experiments. We also use the data augmentation techniques proposed in (Karras et al., ) for improved GAN training on limited data.

### G.3 GAN Inversion Setting

We follow the optimization procedure in (Karras et al., )(^2^22https://github.com/NVlabs/stylegan2-ada-pytorch.git) for GAN Inversion. To summarize, recall the GAN inversion problem for the multi-domain translation task:

$$ $\displaystyle(\widehat{\boldsymbol{c}},\widehat{\boldsymbol{s}})=\arg\min_{ \boldsymbol{c},\boldsymbol{s}}~{}{\sf div}(\boldsymbol{q}(\boldsymbol{c}, \boldsymbol{s}),\boldsymbol{x}^{(i)}),$ (31) $$

Eq. 31 is solved using gradient based optimization of $\boldsymbol{c}$ and $\boldsymbol{s}$ using Adam (Kingma & Ba, ) with an initial learning rate of $0.1$. The hyperparameters of Adam are set to $\beta_{1}=0.9,\beta_{2}=0.999$. The optimization is carried out for 400 steps. The ${\sf div}$ function is realized by using mean squared error in the feature space of pre-trained VGG16 (Simonyan & Zisserman, ) neural network, i.e.,

$$ ${\sf div}(\boldsymbol{a},\boldsymbol{b})=\frac{1}{M}\|{\rm VGG}(\boldsymbol{a} )-{\rm VGG}(\boldsymbol{b})\|_{2}^{2},$ $$

where $M$ is the output dimension of ${\rm VGG}(\cdot)$.

### G.4 Baselines for Multi-domain Generation

We use I-GAN(^3^33https://github.com/Mid-Push/i-stylegan.git) (Xie et al., ), StyleGAN-ADA2(Karras et al., ), and Transitional-cGAN(^4^44https://github.com/mshahbazi72/transitional-cGAN.git) (Shahbazi et al., ) as baselines for multi-domain generation task. Note that I-GAN is chosen because it is the most relevant to our work, whereas StyleGAN-ADA is a very representative work among conditional GANs. Finally, Transitional-cGAN is a recent and representative conditional GAN for limited data regime (note that the datasets used in our work fall under limited data regime (Karras et al., )). For all the baselines, training is done with their default setting for all datasets. All the baselines are also trained for 300,000 iterations with a batch size of 16 for a fair comparison, as well as to control the training time.

### G.5 Baselines for Multi-domain Translation

We use StarGANv2(^5^55https://github.com/clovaai/stargan-v2.git) (Choi et al., ), SmoothGAN(^6^66https://github.com/yhlleo/SmoothingLatentSpace.git) (Liu et al., ), I-GAN (Gen)3 (Xie et al., ) and I-GAN (Tr)3 (Xie et al., ) as baselines for the multi-domain translation task. Note that I-GAN (Gen) refers to the method that uses the multidomain generative model proposed by (Xie et al., ) to carry out the translation procedure via GAN inversion as described in Section 6. Whereas, I-GAN (Tr) is the separate domain translation system proposed by (Xie et al., ) as described in Appendix F. Note that I-GAN (Tr) is trained using the paired samples generated by I-GAN (see Sec. G.4).

### G.6 Datasets Details

AFHQ. We use the AFHQ (Choi et al., ) dataset for both multi-domain generation and translation tasks.
The AFHQ dataset contains images of animal faces in three domains: cat, dog, and wild with 5066, 4679, and 4594 training images, and 494, 492, and 484 testing images. We resize all images to $256\times 256$ for training and testing.

CelebA-HQ. The CelebA-HQ dataset (Karras et al., ) for both multi-domain generation and translation tasks. It contains high-resolution images of celebrity faces along with attributes such as gender, hair color, etc. We split the dataset into two domains based on gender. The male domain contains 18,875 images, whereas the female domain contains 11,025 images. We hold out 1000 images from each domain for testing and use the rest for training. Similar to AFHQ, we resize all images $256\times 256$ for training and testing.

CelebA. The CelebA dataset (Liu et al., ) is used for multi-domain generation task. It contains 202,599 images of celebrity faces along with different attributes. We split the dataset into 7 domains based on the following attributes: “Black hair”, “Blonde hair”, “Brown hair”, “Female”, “Male”, “Old”, and “Young”. We resize all images to $64\times 64$ for training and testing.

## Appendix H Additional Experimental Results

### H.1 Multi-domain Translation.

In this section, we present qualitative results for multi-domain translation tasks by the proposed method and the baselines.

Fig. 5 shows the result of target domain image-guided translation between Male and Female of the CelebA-HQ datasets. Here, the first and second columns show the images in the source domain $\boldsymbol{x}^{(s)}$ and target domain $\boldsymbol{x}^{(t)}$, respectively. We, first, extract the content $\widehat{\boldsymbol{c}}^{(s)}$ from $\boldsymbol{x}^{(s)}$ and style from $\boldsymbol{s}^{(t)}$ using GAN inversion. Then, the translated image $\boldsymbol{x}^{(s\to t)}$ is generated using $\boldsymbol{q}(\boldsymbol{c}^{(s)},\boldsymbol{s}^{(t)})$. Fig. 5 shows that the proposed method generates translations that are more consistent with the style (appearance) of the target domain images compared to the baselines (e.g., see first row).

Figure: Figure 5: Target image-guided translation between different pairs of domains. Content (pose) from the image in the first column is combined with the style (appearance) of the image in the second to generate the translated images.
Refer to caption: extracted/5992817/images/translation/celebahq_ref_translation.png

Fig. LABEL:fig:app_latent_trans shows the result of randomly sampled translations between different pairs of domains in the AFHQ and CelebA-HQ datasets. Here, the first column shows the images in the source domain $\boldsymbol{x}^{(s)}$. We extract the content $\widehat{\boldsymbol{c}}^{(s)}$ from $\boldsymbol{x}^{(s)}$ using GAN inversion. Then, a style vector $\boldsymbol{s}^{(t)}$ is randomly sampled in the target domain $t$ to generate the translated image $\boldsymbol{x}^{(s\to t)}=\boldsymbol{q}(\widehat{\boldsymbol{c}}^{(s)},
\boldsymbol{s}^{(t)})$. One can see that the translation quality is competitive with the baselines. Moreover, in some cases, the translations produced by the proposed method appear more natural compared to the baselines (e.g., see ears of wild animals in the “dog to wild” translation task).

### H.2 Multi-domain Generation.

In this section, we present additional qualitative results for multi-domain data generation by the proposed method and the baselines.

Fig. LABEL:fig:app_c_s_ident shows the generated images by varying content and style components in various domains for the proposed method and the baseline. Image in the $i$th row and $j$th column is generated by combining content $\boldsymbol{c}_{i}$ and $\boldsymbol{s}_{j}^{(n)}$ for a given domain $n$. Hence each row contains the same content and each column contains the same style. One can see that the proposed method has successfully learnt to disentangle content (pose of the object) and style (appearance of the object) for all domains. Whereas, for the baseline I-GAN, it seems that the style component is not properly learnt.

Fig. LABEL:fig:app_celeba shows the qualitative results for the CelebA dataset. As in Fig. LABEL:fig:md_generation_diff_domains, for the $j$th column associated with domain $n_{j}$, we generate three different styles $\boldsymbol{s}_{i}^{(n_{j})},i\in[3]$. Then, the image $\boldsymbol{x}_{i,j}$ in the $i$th row and $j$th column is generated by $\boldsymbol{q}(\overline{\boldsymbol{c}},\boldsymbol{s}_{i}^{(n_{j})})$, using fixed content $\overline{\boldsymbol{c}}$. One can see that the proposed method generates content (pose) aligned samples in different domains with varying style. However, the baseline I-GAN seems to generate the same image for a given domain, as in other experiments.

### H.3 Conditional Image generation.

Fig. LABEL:fig:app_cls_cond_afhq and LABEL:fig:app_cls_cond_male2female shows random samples from each domain. That is, each image is generated using randomly sampled content and styles for the proposed method and I-GAN. Whereas, for StyleGAN-ADA and Transitional-cGAN, each image in a given domain is generated using randomly sampled latent vector. One can see that the compared to the class conditional generative models StyleGAN-ADA and Transitional-cGAN, the proposed multi-domain generative model does not incur any loss in the visual quality. However, due to disentangled latent representations, the proposed method enables content/style controlled generation and domain translation.

### H.4 Underestimated Latent Dimension

In this section, we provide some discussions on the implications of underestimated latent dimensions $d_{\rm C}$ and $d_{\rm S}$. Generally, when $\hat{d}_{C}<d_{C}$ or $\hat{d}_{S}<d_{S}$, solving (4) cannot identify the content and the style parts. In addition, a feasible solution to Problem (4) may not even exist. To clarify, we provide some intuitive explanation for the following three cases:

- 1.
$\hat{d}_{C}+\hat{d}_{S}<d_{C}+d_{S}$ : In this case, a differentiable injective map $\boldsymbol{f}$ does not exist from $d_{C}+d_{S}$-dimensional manifold, $\mathcal{X}$, to $R^{\widehat{d}_{\rm C}+\widehat{d}_{\rm S}}$. Considering our implementation in problem (6), the generated data may not be able to match the data distribution $\mathbb{P}_{{\boldsymbol{x}}^{(n)}}$ well since the generated data ${\boldsymbol{q}}\left({\boldsymbol{e}}_{\rm C}({\boldsymbol{r}}_{\rm C}),{
\boldsymbol{e}}_{\rm S}^{(n)}({\boldsymbol{r}}^{(n)}_{\rm S})\right)$ will have to live on a lower dimensional manifold than $\mathcal{X}$.
- 2.
$\hat{d}_{\rm C}<d_{\rm C}$ but $\hat{d}_{\rm C}+\hat{d}_{\rm S}\geq d_{\rm C}+d_{\rm S}$ : The content dimension will be insufficient to capture all the content information. As such the style part might be a mixture of content and style.
- 3.
$\widehat{d}_{\rm S}<d_{\rm S}$ but $\widehat{d}_{\rm C}+\widehat{d}_{S}\geq d_{C}+d_{S}$ : The style dimension will be insufficient to capture all the style information. But Assumption 3.2 still ensures that the style does not leak into the extracted content part. Hence, the style variability may not be captured by the generated images.

Fig. LABEL:fig:underestimated_latent shows generated samples by varying the content and style dimensions for the cat domain of the AFHQ dataset. We show results corresponding to each of the above cases. Empirical tests showed that $d_{\rm C}>8$ and $d_{\rm S}>8$ are reasonable settings for the AFHQ dataset, which we will treat as the “ground truth”. For each row, we fix the content part ${\boldsymbol{c}}={\boldsymbol{e}}_{C}({\boldsymbol{r}}_{C})$ (i.e., pose of the cat) and randomly sample different styles $\boldsymbol{s}^{(1)}=\boldsymbol{e}_{\rm S}^{(1)}({\boldsymbol{r}}_{\rm S}^{(1
)})$ where ${\boldsymbol{r}}_{\rm S}^{(1)}\sim\mathcal{N}({\boldsymbol{0}},{\boldsymbol{I}
}_{\widehat{d}_{\rm S}})$ to generate the images ${\boldsymbol{x}}^{(1)}={\boldsymbol{q}}({\boldsymbol{c}},{\boldsymbol{s}}^{(1)})$. We set ${d}_{\rm C}=8$ and/or $\widehat{d}_{\rm S}=8$ according to the specific case that we tested.

Fig. LABEL:fig:underestimated_latent(a) corresponds to Case 1. One can see that although content-style disentanglement appears satisfactory, the FID attained is slightly worse than the other two cases, which could be due to insufficient latent dimension.

Fig. LABEL:fig:underestimated_latent(b) corresponds to Case 2. One can see that changing the style component seems to change both pose and appearance, which means that extracted style is a mixture of the content and style.

Fig. LABEL:fig:underestimated_latent(c) corresponds to Case 3. Here, changing the style component has very little effect on the appearance of the cat which corroborates our intuition that the style information is not fully captured.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - Do you believe that L1​ regularization provides a strong enough sparsity constraint to justify the Domain Variability assumption? Could you elaborate on why you chose L1 ​over potentially stricter sparsity-inducing methods, particularly given its central role in ensuring the injective property of the mapping function.
- The model relies on some broad assumptions. Could you discuss the practical conditions that need to be met to ensure invertibility, and how sensitive the method might be to deviations from these conditions?
- Have you observed scenarios where the L1 constraint may not be strong enough to prevent information leakage between content and style representations?
- It appears from the results that the content representation often corresponds to pose, while the style representation captures semantic attributes. Is this always the case, or have you observed instances where content and style encode different aspects of the data? For example, in Figure 3(b), it seems that changing the domain also changes hair color.
- It would be helpful to include experiments on simpler datasets, such as rotated MNIST, to provide a clearer demonstration of the model’s ability to separate content and style. For example, with rotated MNIST, the content could represent class information, while rotation and other stylistic aspects could be designated as style. Such examples could offer readers a more intuitive understanding of how content and style representations are disentangled and controlled within the model.
soundness: 3
strengths: - The paper is well-structured and clearly written.
- It introduces a unique and interesting distribution matching perspective for content and style transfer in image translation
- By relaxing restrictive assumptions often required in previous work—such as the need for component-wise independence and pre-defined latent dimensions—the approach enhances practicality and scalability.
- The paper provides both a theoretical foundation for content-style identifiability and promising qualitative results.
summary: This paper enhances the identifiability of latent content and style representations in unaligned multi-domain data by framing the problem within a distribution matching framework. The authors establish that, under relaxed conditions, their approach does not require restrictive assumptions like component-wise independence or known latent dimensions. By applying sparsity constraints, they ensure identifiable content-style separation, and they ultimately reformulate the distribution matching objective into a simpler GAN training framework. The authors empirically demonstrate improvement in image translation.
weaknesses: - The paper’s dimension-agnostic approach relies heavily on sparsity constraints to prevent information leakage between content and style representations, which may not be effective in all cases.
- The paper relies on several broad assumptions, such as the invertibility of the function up to certain ambiguities, without fully detailing the specific conditions under which this assumption holds.
- The L1 regularization appears to me to be too "soft" of a sparsity constraint to adequately justify the injective property of the mapping function $f$, particularly given that the model relies heavily on this constraint to ensure generative model is a valid invertible function.
- The paper, in general, lacks a thorough ablation study to validate the author's claims.

$\textbf{Minor Remarks}$
- Line 741, I think $\mathcal{A}\in\mathcal{Y}$ rather than $\mathcal{A}\in\mathcal{X}$.
- This is a minor suggestion, but it may be worth considering alternative abbreviations for "DM" and "LDM" due to the common use of these terms in diffusion models and latent diffusion models.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. In Theorem 4.1,  $q$ is a bijective and $f=q^{-1}$, thus trivially $f$ becomes an bijective, which somehow is inconsistent with the fact    that $f$ is a injective from $\mathcal{X}$ to $\mathbb{R}^{\hat{d}_{c}+\hat{d}_s}$ , 
   can the authors clarification? 

   Particularly on the relation between $\mathcal{C}$, $\mathcal{S}$ and $\mathbb{R}^{\hat{d}_{c}+\hat{d}_s}$ as they are actually different spaces. 
2. In Assumption 3.1, $N$ is the dimension of style variables, but somehow in later part of the paper it has different meaning, can the authors clarify? Similarly, the notation $n$ also has different meanings in different part of the paper.
3. Can you provide the correct citation for Proposition E.1?
soundness: 3
strengths: 1. The writing of the paper is good. Most parts of the paper are easy to follow and understand. 
2. The experimental results are good.
summary: The authors try to attach the problem of identifying latent content and style variables from observations from un-aligned domains. The authors claim their main contribution as follows: (1) t restrictive assumptions such as component-wise independence of the latent variables can be removed; (2)  knowledge of the content and style dimensions is not necessary for ensuring identifiability. However, there are some inconsistencies between the main results and Theorem 4.1.
weaknesses: There may be some inconsistency between  the main results and Theorem 4.1. In the main results, it claims that $f$, $\gamma$, and $\delta$ are injective. However, in theorem, the $q$ is a bijective and $f=q^-1$, which implies that $f$ will be an bijective. As a result, $\gamma$ and $\delta$ becomes bijective and it implies the dimension of content and style is actually $\hat{d}_c$ and $\hat{d}_s$. This causes a contradiction in the theory.

The proof of Theorem 4.1 uses  Proposition E.1 from  (Zimmermann et al., 2021) https://arxiv.org/pdf/2102.08850. However, I did not find the Proposition in the reference. Can the authors fix the problem?

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Can the authors please add some remarks regarding the potential weaknesses mentioned above, regarding the suitability of their approach for text, and regarding the potential instability of adversarial GAN training used in their approach?
soundness: 3
strengths: - The paper provides a good overview of background and related work on multi-domain content-style modeling, content-style identifiability, and the challenge of knowing the content and style dimensions apriori. 
- The contributions in this paper appear to be novel and technically sound from both a theoretical and experimental standpoint.
- The proposed approach in this paper appears to be reproducible, given the provided source code and experimental details in the paper.
- The experimental results show that the proposed approach performs well on multi-domain image translation and generation tasks.
summary: This paper tackles the problem of identifiability of latent content and style variables from unaligned multi-domain data. A new approach based on cross-domain latent distribution matching (LDM) is presented, which removes restrictive assumptions from prior work regarding component-wise independence of latent variables and knowledge of content and style dimensions. The LDM formulation is implemented as a regularized multi-domain GAN loss with coupled latent variables. In addition to theoretical contributions on these topics, the authors present experimental results on image translation and generation tasks.
weaknesses: - Because the proposed approach is implemented as a GAN, it seems likely that this approach will be most suitable for images (and possibly audio), and may be problematic for discrete data modalities, particularly text. 
- The authors don’t directly mention or address potential instabilities that may result from adversarial training of their GAN-based approach.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see the weaknesses part.
soundness: 3
strengths: Strengths:
- This paper is well-written, well-motivated, and easy-to-follow. Both the experiments and theoretical justification are intuitive and convincing.
- The contribution of easing independent assumption is helpful for developing simple and efficient causal models.
- A sparsity-regularized implementation is proposed to solve cross-modal learning efficiently.
summary: Summary:
This paper studies causal identification of latent factors content and style in the cross-domain learning problems. Specifically, existing works assume that all latent components are independent to each other. Such an assumption is strict and could not be fulfilled in realistic scenarios. In this paper, the authors propose a cross-domain latent distribution matching framework which does not require strict independence between different latents. Moreover, the proposed framework can ensure content style identification can be free from the specific latent dimensions which further ease the assumption for tackling the problem. Through both theoretical and empirical justifications, the authors carefully testified the effectiveness of the proposed method.
weaknesses: Weaknesses:
- The proposed target in Equation 4 aims to minimize the norm of style latents by enforcing distribution matching across content and block-independence between content and style. However, it could lead to a degenerate solution when there is misidentification between content and style, and some content features already achieves small norm. Could you explain how can the proposed identifiability result can avoid the problem from happening?
- The computational efficiency of the framework is unknown. Only numerical result of generation is provided, there is no quantitative comparison to show the efficiency compared to other baseline methods. Moreover, why sparsity regularization is needed for training GAN needs a justification.
- Missing related references:  
Hong et al., Improving Non-Transferable Representation Learning by Harnessing Content and Style, in ICLR 2024.  
Huang et al., Harnessing Out-Of-Distribution Examples via Augmenting Content and Style, in ICLR 2023.  
Dai et al., Disentangling writer and character styles for handwriting generation, in CVPR 2023.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 4
questions: 1. Can you further explain the reasonability using Assumption 3.1? Can you relax it? If not, what are the difficulties of relaxing this assumption? Furthermore, in what real-world scenarios might this assumption be violated?
2. Can you explain the gap between your method in 4(a) and the way you do implementation? What is the motivation for not using (4) directly? Any trade-offs or limitations of their implementation approach compared to directly using formulation (4).
soundness: 3
strengths: The paper is well written and the contribution is clear.
summary: This paper introduces an analytical framework via cross-domain latent distribution matching(LDM), which establishes content-style identifiability under substantially more relaxed conditions.
weaknesses: The author seems to slip the main result part (3.2) too quickly (with no more than half a page). I am unsure what the key difference between your approach and other low-rank techniques is. In addition, there seems to exist a gap between the optimization problem (4a)(4b)(4c) and the implementation. (I mean: why not to use the formulation in implementation section as the theoretical modeling approach in the methodology part?) There are some writing issues in the paper too, e.g. you can not directly quote (4) , you should use 'according to formulation (4)'.(e.g. the writing in Theorem 4.1)

Meta-review: Summary:
This paper studies contrastive learning through the lens of understanding how it inverts the data generating process. The authors prove that feed-forward models trained with InfoNCE-type objectives learn to implicitly invert the underlying generative model of the observed data, under certain statistical assumptions. The paper establishes theoretical connections between contrastive learning, generative modeling, and nonlinear independent component analysis (ICA). The authors validate their theoretical findings through controlled experiments and show that the conclusions hold even when some theoretical assumptions are violated.

Main strengths:

- Novel theoretical insights connecting contrastive learning to generative modeling and nonlinear ICA
- Clear mathematical exposition and proofs of the key theoretical results
- Strong empirical validation across multiple experimental settings, including complex visual data
- Introduction of a new benchmark dataset (3DIdent) for evaluating representation learning
- Good balance between theoretical contributions and practical implications

Main weaknesses:

- Some notational inconsistencies and ambiguities in the mathematical formulation
- Could benefit from more intuitive explanations alongside the formal proofs
- Some citations and references need more precise specification

Decision: Accept (Poster)