Towards Identifiable Unsupervised Domain Translation: A Diversified Distribution Matching Approach
Authors: Sagar Shrestha
Xiao Fu
Venue: ICLR 2024 poster (2024)
Keywords: unsupervised domain translation
translation identifiability
distribution matching
unpaired image to image translation
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Unsupervised domain translation (UDT) aims to find functions that convert samples from one domain (e.g., sketches) to another domain (e.g., photos) without changing the high-level semantic meaning (also referred to as "content"). The translation functions are often sought by probability distribution matching of the transformed source domain and target domain. CycleGAN stands as arguably the most representative approach among this line of work. However, it was noticed in the literature that CycleGAN and variants could fail to identify the desired translation functions and produce content-misaligned translations.
This limitation arises due to the presence of multiple translation functions---referred to as ``measure-preserving automorphism" (MPA)---in the solution space of the learning criteria. Despite awareness of such identifiability issues, solutions have remained elusive. This study delves into the core identifiability inquiry and introduces an MPA elimination theory. Our analysis shows that MPA is unlikely to exist, if multiple pairs of diverse cross-domain conditional distributions are matched by the learning function.
Our theory leads to a UDT learner using distribution matching over auxiliary variable-induced subsets of the domains---other than over the entire data domains as in the classical approaches.  The proposed framework is the first to rigorously establish translation identifiability under reasonable UDT settings, to our best knowledge.
Experiments corroborate with our theoretical claims.

Published as a conference paper at ICLR 2024
TOWARDS IDENTIFIABLE UNSUPERVISED DOMAIN TRANSLATION: A DIVERSIFIED DISTRIBUTION
MATCHING APPROACH
Sagar Shrestha Xiao Fu"
School of Electrical Engineering and Computer Science
Oregon State University Corvallis, OR 97331, USA
(shressag, xiao. fu/@oreggontate..ee
ABSTRACT
Unsupervised domain translation (UDT) aims to find functions that convert sam- ples from one domain (e.g., sketches) to another domain (e.g., photos) without changing the high-level semantic meaning (also referred to as "content").. The translation functions are often sought by probability distribution matching of the transformed source domain and target domain. CycleGAN stands arguably the most representative approach among this line of work. However, it was noticed the literature that CycleGAN and variants could fail to identify the desired transla- translations. This limitation arises as "measure-
tion functions and produce content-misaligee due to the presence of multiple translation funetions--reeerred preserving automorphism" (MPA)-in the solution space of the learning crite- ria. Despite awareness of such identifiability issues, solutions have remained elu- sive. This study delves into the core identifiability inquiry and introduces an MPA elimination theory. Our analysis shows that MPA unlikely to exist, if multiple pairs of diverse cross-domain conditional distributions are matched by the learn- ing function. Our theory leads a UDT learner using distribution matching over auxiliary variable-induced subsets of the domains-other than over the entire data domains as in the classical approaches. The proposed framework is the first to rigorously establish translation identifiability under reasonable UDT settings, our best knowledge. Experiments corroborate with our theoretical claims.
INTRODUCTION
Domain translation (DT) aims to convert data samples from one feature domain to another, while keeping the key content information. DT naturally arises in many applications, e.g., transfer learning (Zhuang et al. 2020), domain adaptation (Ganin et al. [016 Courty et al.) (017), and cross- retrieval (Huang et al. 2015). Among them, premier application (e.g., profile photo to cartonized emoji and satellite images to street map plots (Isola et al., (017 Supervised domain translation (SDT) relies on paired data from the source and target domains. There, the translation functions are learned via matching the sample pairs.
limage-to-image (121) translation
Nonetheless, paired data are not always available. In unsupervised domain translation (UDT), the arguably most widely adopted idea is to find neural transformation functions that perform probability distribution matching of the domains. The idea emerged in the literature in early works, e.g., (Liu & Tuzel 2016 Taigman et al. 2017] Kim et al. (017). High-resolution image translation using distribution matching was later realized by the seminal work, namely, CycleGAN (Zhu al. 2017). CycleGAN learns pair of transformations that are inverse of each other. One of transformations maps the source domain to match the distribution of the target domain, and the other transformation does the opposite. The distribution matching part is realized by the generative adversarial network (GAN) (Goodfellow et al. 2014). Using GAN-based distribution matching for UDT has attracted much attention -any follow-up works emerged; the survey (Pang et al. 2021)
'Surce code available http://gtttaa..om/naaaaaaaaaaaat

---

Published as a conference paper at ICLR 2024
Challenge- Lack of Translation Identifiability. While UDT approaches have demonstrated signif- icant empirical success, the theoretical question of translation identifiability has received relatively limited attention. Recent works (Galanti et al., 2018b Moriakov et al.. 2020 Galanti et al.) 2021 pointed out failure cases of CycleGAN (e.g., content-misaligned translations like those in Fig. largely attribute the lack of translation identifiability. That is, translation functions in the solu- tion space of CycleGAN (or any distribution matching-basel learners) is non-unique, due to the existence of meeasure-preservvn was called density- preserving mappings in (Galanti et al. [018b] a)). MPA can "swap" the cross- domain sample correspondences without changing the data cerrbbuttoonwhiiie source of producing content misaligned samples after translation as seen in Fig. [ Many efforts were made to empirically enhance the performance of UDT, via implicitly or explicitly promoting solution uniqueness of their loss functions (Liu et al. 2017 Courty et al.) 2017: Xu et al. 2022] Yang et al., 2023). A number of notable works approached the identfaabiliy''niquenes by assuming that the desired translation functions have simple (c.g.. linear (Gulrajani & Hashimoto/ ) or specific structures (de Bézenac et al., (021). However, translation identifiability without using such restrictive structural assumptions have remained elusive.
automorphism (MPA) (Moriakov et al. (020) (the same concept
likely the main
challenge
Contributions. In this work, we revisit distribution matching baased UDT. Our contribution lies in both identifiability theory and implementation:
Theory Development: Establishing Translation Identifiability We delve into the core theoretical chal- lenge regarding identifiability of the translation functions. As mentioned, the solution space of existing distribution matching criteria could be easily affected by MPA. How- ever, our analysis shows that the chance of having MPA decreases quickly when the translation function aligns more than one pair of diverse distributions. This insight allows us to come up sufficient condition, namely, the sufficiently diverse condition (SDC), to establish transla- tion identifiability of UDT. To our best knowledge, our result stands as the first UDT identifiability theory with- out using simplified structural assumptions.
Source
Tanget
GeduGAN
UNIT
MUNIT
7
Figure 1: Lack of translation identifi- ability often leads to content misalign- ment in distribution matching based UDT methods, e.g., CycleGAN (Zhu et al.) 2017). MUNIT(Huang et al. 2018). and UNIT (Liu et al.) [207]) Source domain: MNIST Digits. Target Domain: Rotated Display of MNIST.
Simple Implementation via Auxiliary Variables. Our theoretical revelation naturally gives rise novel UDT learning criterion. This criterion aligns multiple pairs of conditional distributions across the source and target domains. We define these conditional distributions over (overlapping) sub-domaions of the source/target domains using auxiliary variables. We demonstrate that in prac- tical applications such as unpaired I2I translation, obtaining these sub-domains can be a straight- forward task, e.g., through available side information querying the foundation models like CLIP Radford et al. 2021). Consequently, our identification theory can be readily put into practice.
Notation. The full list of notations is in the supplementary material. Notably, we use Pa and Parju to denote the probability measures of æ and x conditioned on respectively. We denote the corresponding probability density function (PDF) of x by p(z). For a measurable function f: X + and distribution Irx defined over space X, the notation f#F_ denotes the push-forward measure; that is, for any measurable set A V. JWF = Pr | frreims (A)). where foreing (A) {æ E X f(x) ) E A}. Simply speaking, JHP., denotes the distribution of f(x) where x ~ P.. The notation f+P= IP means that the PDFs of off(c fxx) and y are identical almost everywhere (a.c.).
PRELIMINARIES
Considers two data domains (e.g., photos and sketches). The samples from the two domains are represented by x E X C RD= and y E ) C RR,, We make the following assumption:
Assumption 1. For every x E X, has a corresponding y Y, and vice versa. addition, there exist deterministic continuous functions f y + X and g" X + y that link the corresponding pairs; i.e.,
f² x, g'(x)) y, A corresponding pair (æ. y)
(1)
2

---

Published as a conference paper at ICLR 2024
In the context of domain translation, linked (æ. y) pair can be regarded as cross-domain data samples that represent the same "content", and the translation functions (f*.g") are responsible for changing their "appearances/stllee", The term "content" refers to the semantic information to be kept across domains after translation. In Fig. [ the content is the identity of the digit ( other than writing style the rotation); in Fig. of See. []] the content can be understood as the shared characteristics of the person in both the cartoon and the photo domains, which can collectively identify the person.
Note that in the above setting, the goal is to find Iwo ground-truth translation functions where one function's source is the other's target. Hence, both X and y can serve as the source/target domains. In addition, the above also implies f = (g*) invertible. Under this setting, if one can identify g* and f", then the samples in one domain can be translated to the other domain-while not changing the content. Note that Assumption means that there is one-to-one correspondence between samples in the two domains, which can be a somewhat stringent condition in some cases. However, as we will explain in detail later, many UDT works, e.g., CycleGAN (Zuu et 2017) and variants (Liu et al. 2017: Kim et al. 2017 Choi et al. 2018 Park et al. 2020), essentially used the model in Assumption I to attain quite interesting empirical results. This makes it a useful model and intrigues us to understand its underlying properties.
i.e., the ground-truth translation functions are
Supervised Domain Translation (SDT). In SDT, the corresponding pairs (x, y) assumed to be aligned a priori. Then, learning a translation function is essentially a regression problem-e-s via finding g (or f) such that D(g(2)) az) y) (or (f(())( is minimized over all given pairs, where D(-1I-) is a certain "distance" measure; see, e.g., (Isola et al., 2017 Wang et al. 2018).
Unsupervised Domaín Translation (UDT). In UDT, samples from the two domains are acquired separately without alignment. Hence, sample-level matching as often done in SDT is not viable. Instead, UDT is often formulated as probability distribution matching problem (see, e.g., (Zhu itt al.) 2017, Taigman et al. 2017 Kim et al.) 2020; Park et al., (020)) as distribution matching can be attained without using sample-level correspondences. Assume that æ and y are the random vectors that represent the data from the X'-domain and the Y-domain, respectively. Then, the desired f and g" are sought via finding f and g such that
py 9 # and Pa. - SWP. f
The hope is that distribution matching can work as a surrogate of sample-level matching as in SDT. The arguably most representative work in UDT is CycleGAN (Zhu et al. 2017). The CycleGAN loss function is as follows:
min max LGAN (g, dyy v) + CGAN(f, dz, x. v) + XCeyo (g, f.g da
(3)
where di and dy represent two discriminators in domains X' and V, respectively.
CGAN dy, x. y) = E [log (y)) + Egse, [ (g(1))))) LGAN(S, du, v) )) is defined in the same way, and the cycle-consistency term is defined as Leye (g. f)= = Ea-P_ [llf(((=)) - x x111] + Ey-,,, [lg(f())) - -yll]-
(4)
(5)
The minimax optimization of the LGAN terms enforces I#P, Py and f#r, Pa.. The Leye term encourages f = و 9-1 CycleGAN showed the power of distribution matching in UDT and has triggered a lot of interests in 121 translation. Many variants of CycleGAN were also proposed to improve the performance; see the survey (Pang et al. 2021).
Lack of Translation Identifiability, MPA and Content Misalignment. Many works have noticed that distribution matching-type learning criterion may suffer from the lack of translation identifia- bility (Liu et al. 2017| Moriakov et al. 2020; Galanti et al., 20188) 202T Xu et al. 2022); i.c., the solution space these criteria could have multiple solutions, and thus lack the ability recover the ground-trui -tuth g" and f*. The lack of identifiability often leads to issues such as content mis- alignment as we saw in Fig. [ To understand the identifiability challenge, let us formally define identifiability of any bi-directional UDT learning criterion:
Definition 1. (Identifiabilityy) Under the setting of Assumption assume that (f g) is any optimal solution of a UDT learning criterion. Then, identifiability of (f .g'") holds under the UDT learning criterion if and only if f = f and 9 g* a.e.
3

---

Published as a conference paper at ICLR 2024
Notice that used the optimal solution in the definition. This is because identifiability is char- acterization of the "kernel space" (which contains all the zero-loss solutions) of a learning criterion (Moriakov et al. 2020; Fu et al., 2019). In other words, when a UDT criterion admits translation identifiability, it indicates that the criterion provides a valid objective for the learning task-bur identifiability is not related to the optimization procedure. We will also use the following:
Definition 2. (MPA) A meusure-preserrvng aulomorphism (MPA) of Pr is a continuous function h: X + X such that P-= = P.
Simply speaking, MPA defined in this work is the continuous transformation h(x)) whose output has the same PDF as p(æ). Take the one-dimensional Gaussian distribution x ~ NN,,, o²) as an example. The MPA of [i,, o²) is h(x) -r 2/.. A recent work (Moriakov al. 2020)
suggested that non-identifiability of the desired translation functions by CycleGAN Is caused by the existence of MPA. Their finding can be summarized the following Fact:
Fact If MPA of = or Py exists, then CycleGAN and any criterion using distribution matching in 2 do not have identifiability of f and g"
Proof: is straightforward see that Py Thp and x are invertible. Hence, the ground truth gg) is an optimal solution of CycleGAN that makes the loss in equal to zero. However, due to the existence MPA, one can see that f h f also attain Pæ f# This is because have f#r, h f hwp, Pa Plus, as h f is still invertible, ff still makes the cycle-consistency loss zero. Hence, the solution of CycleGAN is not unique and this loses identifiabil- ity of the ground truth translation functions.
f In addition, f and g"
pla)
O
The existence of MPA in the solution space of the UDT learning losses may be detrimental in terms of avoiding content misalignment. To see this, con- sider the example in Fig.
y (x)
[ There, Pz (M,, o²) and h(x))= = x + 2/1 an MPA of P. as men- tioned. Note that f = h o f can be an opti- mal solution found by CycleGAN. However, such an can cause misalignment. To explain, assume a and y b are associated with the same entity, which means that a = (b) represents the ground-truth alignment and translation. How- ever, as p( (a-a a + 2/)) = p(h(a)) p(h o f (b)) = p( (b)), the learned function ff wrongly translates y to x a + 2p.
('(y)
MPA transformed
h f'
desired
Figure 2: Illustration of of the lack of iden-
tifiability and MPA-induced content misalign-
ment;" (d) means distribution matching.
p(x)
p(an
p(22)
Our Gaussian example seems to be special as it has symmetry about its mean. However, the existence of MPA is not unusual. To see this, we show the following result:
1
#
Proposition 1. Suppose that Pa admits a contin- uous PDF, p(z) and p(z) > V. E X. Assume that X' is simply connected. Then, there exists a continuous non-trivial (non-identity) h(.) such that h = P..-
h((s) - -
No unified MPA
Figure: 3: A unified MPA is harder to exist for group of distributions.
Note that there are similar results in (Moriakov et al. 2020 regarding the existence of MPA. but more assumptions were made their proof. The universal existence of MPA attests to the challeng- ing nature of establishing translation identfiability in UDT.
3 IDENTIFIABLE UDT VIA DIVERSIFIED DISTRIBUTION MATCHING
Intuition - Exploiting Diversity of Distributions. Our idea starts with the following observation: If two distributions have different PDFs, shared MPA is unlikely exist. Fig. 3 illustrates the
4

---

Published as a conference paper at ICLR 2024
intuition. Consider two Gaussian distributions 21 Mur, 1) and T2 (u22 1) with μr # A2- For each of them, h(z) = -I + 2/1; for i = 1,2 is an MPA. However, there is not a function that can serve unified MPA to attain h
PI2 simultancossly. Intuitively, the diversity of the PDFs of 21 and z2 has made finding unified MPA h(-) difficult. This suggests that instead of matching the distributions a and f(y) and those of and ((z), it may be beneficial to match the distributions of more variable pairs whose probability measures diverse.
Pz,, & h
Auxiliary Variable-Assisted Distribution Diversification. In applications. the corresponding sam- ples æ; y often share some aspects/traits. For example, in Fig. 4 the corresponding æ: and y both have dark hair or the same gender. If we model collection of such traits as different realizations of discrete random variable u, the alphabet of u, denoted {ui,... { 27 represents these traits. We should emphasize that the traits is a result of the desired content invariance across domains, but need not to represent the whole content.
To proceed, we observe that the conditional distributions P Palumw; and P yylmm, satisfy Pælu=u, = f F Fyle=,, = - Vi. The above holds since x and y have de- terministic relation and because the trait u; is shared by the content-aligned pairs (z.
In practice, u can take various forms. In I2I translation, one may use image categories or labels, Note that knowing the image categories does not mean the samples
if available, serve from the two domains are aligned, as each category could contain a large amount samples. In addition, can sample attributes (such hair color, gender as in Fig. 4 these attributes are not meant to be changed in the considered translation tasks. If not immediately available, these attributes can be annotated by open-sourced AI models, e.g., CLIP (Radford al.) 2021]; see detailed implementation in the supplementary material. A similar idea of using CLIP to acquire auxiliary information was explored in (Gabbay et al. 2021
as u, if
By Proposition]] T. it is almost certain that IP Pælu=u, has an MPA h, for all & E [7]. However, it is likely that h; + h if IP p and IP ae u are sufficiently different. As a consequence, similar to what we saw in Fig. 3 if one looks for f that does simultaneous matching of
Black Hair
P Pælu=u: P Vi E [4];
(6)
it is more possible that f = fr instead of having other solutions-tii leads to identfiaaility of f*
Female
Proposed Loss Function. We propose to match multiple distribution pairs (P Pu/uu f# (as well as (IP ull P for I,..., .....,,. I. For each pair, use discriminator d!!)) : X [0, 1] (and d!) y + [0, 1] in reverse direction). Then, our loss
Figure 4: Examples of u,
function is as follows:
min max fa tal" "
Σ
(GGAN(g, (g , LGAN(S. d .....))) + (g,
where we have
LGAN g, d n.
Pr(u = uz) (Fy-e, [oo (()
F Fr-Pa
[Log (1-d(((()))
Note that Paju, represents samples that share the same characteristic defined by (e.g., hair color, eye color, gender). This means that the loss function matches a suite of distributions defined over (potentially overlapping) subdomains over the entire domain X' and V. We should emphasize that the auxiliary variable is only needed in the training stage, but not the testing stage.
We call the proposed method diversified distribution matching for unsupervised domain translation (D (DIMENS ION) The following lemma shows that DIMENS ION exactly realizes our idea in (6):
Lemma 1. Assume that an optimal solution of 7 is (f.g. {d.)) a.")). Then, under Assumption have P Palu=u, f f P p 9 j/umm Vi E [7], and g- a.e.
Note that we still use the term "unsupervised" despite the need of auxiliary information as no paired samples are required. We avoided using "semi-supervised"" or "weakly supervised" as these are often reserved for methods using some paired samples; see, e.g., (Wang al. 2020 Mustafa & Mantiuk 2020)
5

---

Published as a conference paper at ICLR 2024
Identfiability Characterization. Lemma means that solving the DIMEENSION loss leads to ditional distribution matching as we hoped for in (6). Hower, it does not guarantee that (f g) found by DIMENSION ION satisfies f = f and g g" Towards establishing identifiability of the ground-truth translation functions via DIMENSION,
will use the following definition:
Definition 3 (Admissible MPA). Given auxiliary variable u, the function h(()) is said to be an admissible MPA of {IP (///um V and only if u=u, ho# umu, Vi [T]-
Now, due to the deterministic relationship between the pair and y, we have the following fact:
Fact Suppose that Assumption 7 holds. Then, there exists an admissible MPA of (allumn if and only if there exists an admissible MPA of { (Pylu=u,) )
The above means that if we establish that there is no admissible MPA the {I (Pa/umu: H. it suffices to conclude that there is admissible MPA of {P ([n,,,,,KK)
As described before, to ensure identifiability the transla- tion functions via solving the DIMENSION loss, we hope the conditional distributions Pæ]/um,, and Pyjuuu,, to be suf- ficiently different. We formalize this requirement in the fol- lowing definition:
p(a Wi)
p(a|u w2)
Definition 4 (Sufficiently Diverse Condition (SDC)). For any two disjoint sets A,B B C X, where A and B are connected, open, and non-empty, there exists ((4.8)
Conditional PDFs p(r/u ui) and p(z|u u2) that
Figure 5:
{ui....,u1} such that P. Pain=MAA,,) [A] # P Patu [ Then, the set of conditional distributions {Palu=u)) called sufficiently diverse.
satisfy the SDC.
p( mz)
p(r|u un)
Definition puts the desired "diversity" into context. is
important to note that the SDC only requires the existence of
a certain u(A.B) E {uI...., út} for a given disjoint set pair (A, B). does not require a unified u for pairs; i.e., U(A. needs not to be the same as "(AA'BB)) for B) * B'). Fig. shows& simple example where the two conditional distributions satisfy the SDC. In more general cases, this im- plies that if the PDFs of the conditional distributions exhibit different "shapes" " over their supports, SDC is likely to hold. translation identifiability result:
A
B
Figure 6: SDC (r-SDC).
Illustration of relaxed
Using SDC, we show the following
Theorem 1 (Identifiability). Suppose that Assumption 7 holds. Let Eu, denote the event that the pair (P aejums PPz) (all==,)) does not satisfy the SDC. Assume that Pr[E.j] p for any i + j. where i.j E [T. Let (f.g) be from an optimal solution of the DIMENSION loss (7). Then, there is admissible MPA of (P 1e of the solution, i.e., f f", a.c. and g g", with probability of at least 1 p1))))
Theorem shows that if the conditional distributions are sufficiently diverse, solving 7 can cor- rectly identify the ground-truth translation functions. Theorem D also spells the importance of having more u,'s 's (which means more auxiliary information). The increase of improves the probability of success quickly.
Towards More Robust Identifiability. Theorem uses the fact that the SDC holds with high probability for every pair of (P (Pn) Paju,) (cf. Es,i I p). also of interest to see the method is robust to violation of the SDC. To this end, consider the following condition:
Definition (Relaxed Condition: r-SDC). Let dia(AA
2||2 and V..j {(A, = = u, [B] & Palu, [A] = P [B], A Ann B = Φ), where A, B are non-empty, open and connected. Denote Mr, = max max(dia(AA), dia(B)). Then, (Palu,y Paluy) satisfies the r-SDC if Mr, for 0.
supw,zeAller

---

Published as a conference paper at ICLR 2024
Note that the r-SDC becomes the SDC when = 0. Unlike SDC in Definition 4 the relaxed SDC condition allows the violation of SDC over regions Vi.j. Our next theorem shows that the translation identifiability still approximately holds, long the largest region in V.,j is not substantial:
Theorem 2 (Robust Identifiability. Suppose that Assumption 4 holds with g" being L-Lipschita continuous, and that any pair of (IP 2), Palus) satisfiex the r-SDC (cf. Definition 5) with probability
at least 7, i.e., Pr[Ms.j 2 I Y for any + j. where (i )) E [T] x [J]. Let g be from any
optimal solution of the D DIMENSION loss in 7 Then, we have lg g" (a))l2 11222 2rL, Vx E N,,
with a probability of at least 1-11--4 x()), The same holds for f.
Theorem 2 asserts that the estimation error of g scales linearly with the "degree" of violation of the SDC (measured by r). The result is encouraging; It shows that even if the SDC is violated, the performance of DIMENS ION will not decline drastically. The Lipschitz continuity assumption in Theorem i mild. Note that translation functions are often represented by neural networks in practice, and neural networks with bounded weights are Lipschitz continuous functions (Bartlet) et al. ). Hence, the numerical successes of many neural UDT models (e.g., CycleGAN) suggest that assuming that Lipschitz continuous ground wround-truth translation functions exist is reasonable.
4 RELATED WORKS
Prior to CycleGAN (Zhu et al. 2017). J. the early works (Liu & Tuzel, 2016; Taigman et al. 2017
Kim et al. (017) started using GAN-based neural structures for distribution matching in the context of 12I translation. Similar ideas appeared in UDT problems in NLP (c.g., machine translation) (Conneau et al. 2017 Lample et al. 2017). In the literature, it was noticed that distribution matching modules lack so lution uniqueness, and many works proposed remedies (see, e.g. (Liu et al. 2017 Xu jet al.] [222] Xie al. 2022; Park et al., (2200). These approaches have worked to various extents empirically, but the translation identifiability question was unanswered. The term "content" was used in the vision literature (in the context of 121 translation) to refer domain-invariant attributes (e.g.. pose and orientation (Kim et al.) 2020 Amodio & Krishnaswamy 2019] Wu et al., 2019] Yang et al. ).23). This narrower interpretation of content relative ours ours-as as content in our case can be high-level or latent semantic meaning that is not represented by specific attributes. Our definition of content is closer to that in multimodal and self-supervised learning (Von Kügelgenn et al., 2021: Lyu et al., 2022, Daunhawer et al., (023). Before our work, auxiliary information UDT. For example, semi-supervised UDT (see, e.g., (Wang et al., 2020 was also considered
Mustafa & Mantiuk 2020g) uses small set of paired data samples, but our method does not use any sample-level pairing information. Attribute- guided 121 translation (see, e.g., (Li et al.) 2019 Choi et al. 2018 2020)) specifies the desired attributes in the target domain to "guide" the translation. These are different from our auxiliary variables that can be both sample attributes or high-level concepts (which is closer to the "auxiliary variables" in nonlinear independent component analysis works, e.g.. (Hyvarinen et al.] 2019)). Again, translation identifiability was not considered for semi- supervised or attribute-guided UDT. There has been efforts towards understanding the translation identifiability of CycleGAN. The works of Galanti et al.) (2018b a recognized that the success of UDT may attribute to the existence of small number of MPAS. Moriakov al.] (2020) showed that MPA exists in the solution space of CycleGAN, and used it to explain the ill ellpddsesssss of CycleGAN. Chakrabarty Das (2022) studied the finite sample complexity of CycleGAN in terms of distribution matching and cycle consistency. Gulraaani & Hashimoto (2022) andide Bézenac et al. (20T)) argued that the target translation functions have known structures (e-g., linear or optimal transport structures), then translation identifiability can be established. However, these conditions can be restrictive. Translation identifiability without using such structural assumptions had remained unclear before our work.
5 NUMERICAL. VALIDATION
Constructing Challenging Translation Tasks. We construct challenging translation tasks to val- idate our theorems and to illustrate the importance of translation identifiability. To this end, we make three datasets. The first two are "MNIST v.s. Rotated MNIST" (MrM) and "Edges v.s. Ro- tated Shoes" (ErS). In both datasets, the rotated domains consist of samples from the "MNIST" and "Shoes" with 90 degree rotation, respectively. We intentionally make this rotation, as rotation is
7

---

Published as a conference paper at ICLR 2024
Cydlesely-
Cydee44
Source
Target
>ropesed
UNIT
NUNIT
U-GAI-IT
Star6AN-2 Hine-SRR
GP-UNIT
DownCCRD
ZonoDOM
CycleGAte Lass
CycaGAN
Proposed
U-GATHIT
StarGAN-V2
GP-UNT
OverlORD
ZeroOtN
Figure 8: Qualitative results (a) Edges Rotated Shoes, (b) Bitmoji Faces to CelebA-HQ. and (c) CelebA-HQ to Bitmoji Faces tasks. More comprehensive illustrations are in the appendix.
large geometric change across domains. This type of large geometric change poses challenging translation task (Kim et al., 2020 Wu et al. 2019, Amodio & Krishnaswamy (2099 Yang et al, 2023). In addition, 2020 (CB), In this task, profile photos of celebrities are translated to cartoonized bitmoji figures, and vice versa. We intentionally choose these two domains make the translation challenging: The profile photos have rich details and are diverse in terms of face orientation, expression, hair style, etc., but the Bitmoji pictures have a relatively small set of choices of these attributes (e.g., they are always front-facing). More details of the datasets are in Sec. F.4 in the supplementary material.
construct task "CelebA-HQ (Karras al. 2017) v.s. Bitmoji (Mozafari
Baselines. The baselines include some representative UDT methods and some recent developments, ie., GP-UNIT Yang et al. (023). Hneg- SRC (Jung et al. 2022), OverLoRD (Gabbay & Hoshen
2021.
ZeroDIM (Gabbay et al. (022T) St arrAN-v v2 (Choi et al. 2020). U-GAT-I7 (Kim al.
2020 MUNIT (Huang al. 2018), UNIT (Liu et al. 2017). and CycleGAN (Zhu et al. 2017 In particular, two versions of CycleGAN are used. "Cyc
"CycleGAN eGAN Loss" refers to the plan-vam vanilla CycleGAN objective in B and CycleGAN+ Id refers to the "identity
version in (Zhu 2017). ZeroDIM uses the same auxiliary information as that used by the proposed method.
ct al.
MNIST to Rotated MNIST. Fig. 7 shows the results. In this case, we use E {1,.... 10), i.e., the labels of the identity of digits, as the alphabet of the auxiliary variable. Note that knowing such labels does not mean that the cross- domain pairs (x, y) are known. Alter- natively, one can also use digit shapes as the alphabets (see Sec. F.6).. One can see that DIMENSION learns to translate the digits to their corresponding rotated versions. But the baselines sometimes misalign the samples. The results are
CydeGAN
UNIT
MUNIT U-GAT-IT Stargoo--
9
8
M
9
Figure 7: Translation from MNIST to rotated MNIST.
consistent with our analysis (see Sec, [66 for more results).
Edges to Rotated Shoes. From Fig. g (a), one can see that the baselines all misalign the edges with wrong shoes. Instead, the proposed DIMENST ION, using the shoe types (shoes, boots, sandals, and slippers) the alphabet of u. does not encounter this issue. More experiments including the reverse translation (i.e., shoes to edges) are in Sec. F6 in the supplementary material.
CelebA-HQ and Bitmoji. Figs. g (b)-(c) show the results. The proposed method uses ('male', "femalee' "bblack hair :non-black hair'). To obtain the auxilliary information CLIP to automatically annotate the images. A remark is that translating from the Bitmoji domain to the CelebA-HQ domain [see. Fig, (b)) is particularly hard. This is be- cause the learned translation function needs "fill in" lot of details to make the generated profiles
for each sample,
8

---

Published as a conference paper at ICLR 2024
Table I: LPIPS scores for the ErS and MrM tasks and FID scores for all tasks. E: Edges, rS: rotated Shoes, M: MNIST, rM: rotated MNIST, C: CelebA-HQ, B: Bitmoji faces.
LPIPa(1)
Method
18
XM
rh1
Progosed
0.29 0.06
11.5
0.11 0.08
0.04
21.47
40.14
13.05
16.07
32.03
20.50
CydeGANN-Lss
043++0.00
55.42
16.11
36.
CydedAN
0.00
0.54
0.00
250.31
150.84
34.01
U-GAT-IT
0.50:::0.00
0.00
0.25
258.01
11.67
UNIT
0.49 0.00
0.58 # 0.113
è: 0.06
è 0.08
33.95
96.28
20.44
19:15
53.63
MUNIT
0.50
0.58 0.04
0.09
0.09
43.83
14.89
15.96
62.49
27.59
StaGAN-12
0.30 0.05
0.52 +
0.28 0.09
0.29 + 0.10
75.46
158.34
30.07
32.20
35
282.98
HRe
0:45+
0.50 0.007
210.27
198.77
GI-NNIT
0:49 1:00:00.00 0.08
0.44 + 0.05
231.31
96,
32:40
30.30
OveLLORD
0/40 = 0.00
11:22 a 0.00
101.14
124,12
76.
31.08
ZcroDIM
0.38 = 0.06
0141 + 0.07
85.56
187.45
88.
36.21
means that mettod is not applicable to the dataset due to small resolution.
photorealistic. Our method clearly outperforms the baselines in both directions of translation; see more in Sec. F.6 in the supplemenary material.
Metrics and Quantative Evaluation. We employ two widely adopted metrics in UDT. The first the learned perceptual image patch similarity (LPIPS) (Zhang et al., 2018), which leverages the known ground-truth correspondence between (æ, y) LPIPS measures the "perceptual distance" between the translated images and the ground-truth target images. In addition, also use the Fréchet inception distance (F ID) score (Heusel et al. 2017) in all tasks. FID measures the visual quality of the learned translation using a distribution divergence between the translated images and the target domain. In short, LPIPS and FID correspond to the content alignment performance and the target Nooainn-atainninn ability, respectively; see details of the metrics Sec. [F44]
Table[[shows the LPIPS scores over the first two datasets where the ground-truth pairs are known. One can see that DIMENSION significantly outperforms the baselines content alignment. The FTD scores in the same table show that our method produces translated images that have similar characteristics of the target domains. The F ID scores output by our method either the lowest the second lowest.
a result of good
Detailed Settings and More Experiments. See Sec. E for settings and more results.
CONCLUSION
In this work, we revisited the UDT and took a deep look at a core theoretical challenge, namely, the translation identifiability issue. Existing UDT approaches (such as CycleGAN) often lack trans- lation identifiability and may produce content-misaligned translations. This issue largely attributes to the presence of MPA in the solution space of their distribution matching modules. Our approach leverages the existence of domain-invariant auxiliary variables to establish translation identifiability, using novel diversified distribution matching criterion. To our best knowledge, the identifiability result stands as the first of its kind, without using restrictive conditions on the structure of the desired translation functions. We also analyzed the robustness of proposed method when the key sufficient condition for identifiability is violated. Our identifiability theory leads to an easy-to-implement UDT system. Synthetic and real-data experiments corroborated with our theoretical findings.
Limitations. Our work considers a model where the ground-truth translation functions are deter- ministic and bijective. This setting has been (implicitly or explicitly) adopted by a large number of existing works, with the most notable representative being CycleGAN. However, there can be multiple "correct" translation functions in UDT, as the same "content" can be combined with vari- ous "styles". Such cases may be modeled using probabilistic translation mechanisms (Huang et al.) 2018 Choi et al., 2020 Yang et al. 2023), yet the current analytical framework needs a significant revision to accommodate the probabilistic setting. In addition, our method makes use of auxiliary variables that may be nontrivial to acquire in certain cases. We have shown that open-sources foun- dation models such as CLIP can help acquire such auxiliary variables and that the method is robust to noisy/wrong auxiliary variables (see Sec. However, is still of great interest to develop provable UDT translation schemes without using auxiliary variables.

---

Published as a conference paper at ICLR 2024
This work is supported in part by the Army Research Office (ARO) under Project ARO 1911222211-0 and part by the National Science Foundation (NSF) CAREER Award ECCS-214488
Acknowledgement.
REFERENCES
Matthew Amodio and Smita Krishnaswamy. TravelGAN: Image-to-image translation by transfor- mation vector learning. In Proceedings of IEEE/CVF Computer Vision and Pattern Recognition (CVPR), pp. 9983-8992, 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
Peter L Bartlett, Dylan Foster, and Matus Telgarsky. Spectrally-normalized margin bounds for neural networks. Advances in Neural Information Processing Systems (Neur/PS), 30, 2017.
Neal L Carothers. Real analysis. Cambridge University Press, 2000.
Anish Chakrabarty and Swagatam Das. On translation and reconstruction guarantees of the cycle- consistent generative adversarial networks. Advances in Neural Information Processing Systems (NeurIPS), 35:23607- 23620, 2022.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. Star- GAN: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of IEEE/CVF Computer Vision and Pattern Recognition (CVPR), pp. 8789-8797, 2018.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. StarGAN v2: Diverse image synthesis for multiple domains. Proceedings of IEEF/CVF Computer Vision and Puttern Recognition (CVPR), pp. 8188-8197, 2020.
Alexis Conneau, Guillaume Lample, Marc' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. arXiv preprint arXiv:1710.04087, 2017.
Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamonjy. Joint distribution op- timal transportation for domain adaptation. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
George Darmois. Analyse des liaisons de probabilité. In Proceedings of International Statistic Conferences, pp. 231, 1951.
Imant Daunhawer, Alice Bizeul, Emanuele Palumbo, Alexander Marx, and Julia Vogt. Identifia- bility results for multimodal contrastive learning. arXiv preprint arXiv:2303.09) 0966,,,,,, 2023.
Emmanuel de Bézenac, Ibrahim Ayed, and Patrick Gallinari. CycleGAN through the lens of (dy- namical) optimal transport. In Proceedings of Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML), pp. 132-147. Springer, 2021.
Xiao Fu, Kejun Huang, Nicholas D Sidiropoulos, and Wing-Kin Ma. Nonnegative matrix factor- ization for signal and data analytics: Identifiability, algorithms, and applications. IEEE Signal Processing Magazine, 36(2):59-80 80, 2019.
Aviv Gabbay and Yedid Hoshen. Scaling-up disentanglement for image translation. In Proceedings of the IEEF/CVF International Conference on Computer Vision (CVPR), pp. 6783-6792, 2021.
Aviv Gabbay, Niv Cohen, and Yedid Hoshen. An image is worth more than a thousand words: Towards disentanglement in the wild. Advances in Neural Information Processing Systems (NeurIPS), 34:9216-9229 92288, 2021.
Tomer Galanti, Sagie Benaim, and Lior Wolf. Generalization bounds for unsupervised cross-domain mapping with WGANs. arXiv preprint arXiv: (807,08501 2018a.
10

---

Published as a conference paper at ICLR 2024
Tomer Galanti, Lior Wolf, and Sagie Benaim. The role of minimal complexity functions in unsuper- vised learning of semantic mappings. In Proceedings of International Conference on Learning Representations (ICLR), 2018b.
Tomer Galanti, Sagie Benaim, and Lior Wolf. Risk bounds for unsupervised cross-domain mapping with ipms. The Journal of Machine Learning Research, 22(1))4019-400 2021.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. Journal of Machine Learning Research (JMLR), 772009--2030, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In Advances in Neural Information Processing Systems (NeurIPS), 2014.
Ishaan Gulrajani and Tatsunori Hashimoto. Identifiability conditions for domain adaptation. In Proceedings of International Conference Machine Learning (ICML), pp. 7982-7997, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
Junshi Huang, Rogerio S Feris, Qiang Chen, and Shuicheng Yan. Cross-domain image retrieval with dual attribute-awas aware ranking network. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 1062-1070, 2015.
Xun Huang, Ming-Yi Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to- image translation. In Proceedings of European Conference on Computer Vision (ECCV), pp. 172-189, 2018.
Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429 439, 1999.
Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In Proceedings of International Conference on Artificial Intelli- gence and Statistics (AISTATS), pp. 859-868. PMLR, 2019.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of IEEE/CVF Computer Vision and Pattern Recognition (CVPR), pp. 1125-1134, 2017.
Chanyong Jung, Gihyun Kwon, and Jong Chul Ye. Exploring patch-wise semantic relation for con- trastive learning in image-to-image translation tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18260-18269, 2022.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. arXiv preprint arXiv: 1019, 2017.
Junho Kim, Minjae Kim, Hyeonwoo Kang, and Kwanghee Lee. U-GAT-IT:
Unsupervised genera- tive attentional networks with adaptive layer-instance normalization for image-to-image transla- tion. In Proceedings of International Conference on Learning Representations (ICLR), 2020.
Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning to discover cross-domain relations with generative adversarial networks. In Proceedings of International Conference on Machine Learning (ICML), pp. 1857-1865, 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of International Conference on Learning Representations (ICLR), 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep volutional neural networks. Advances in Neural Information Processing Systems (NeurIPS), 25, 2012.
11

---

Published as a conference paper at ICLR 2024
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc' Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only, arXiv preprint arXiv:1711.00443, 2017.
Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. ATT Labs [On- l.ne Available: http://yann.lecme accun.com//xxb//mmis... 2, 2010.
Xinyang Li, Jie Hu, Shengchuan Zhang, Xiaopeng Hong, Qixiang Ye, Chenglin Wu, and Rongrong Ji. Attribute guided unpaired image-to-image translation with semi-supervised learning. arXiv preprint arXiv:1904, 12428, 2019.
Ming- Yu Liu and Oncel Tuzel. Coupled generative adversarial networks. Advances in Neural Information Processing Systems (NeurIPS), 29, 2016.
Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, 2017.
Ming- Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. Few-shot unsupervised image-to-image translation. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision (CVPR), pp. 10551-10560 2019.
Qi Lyu, Xiao Fu, Weiran Wang, and Songtao Lu. Understanding latent correlation- multiview learning and self-supervision: An identifiability perspective. In Proceedings of International Conference on Learning Representations (ICLR), 2022.
Andrew L Maas, Awni Y Hanmun, and Andrew Y Ng. Rectifier nonlinearities improve neural net- work acoustic models. In Proceedings of International Conference on Machine Learning (ICML), volume 30, pp. 2013.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of International Conference on Computer Vision (ICCV), pp. 2794-2802, 2017.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for GANs do actually converge? In Proceedings of International Conference on Machine Learning (ICML), pp. -488-3490. PMLR, 2018.
Nikita Moriakov, Jonas Adler, and Jonas Teuwen. Kernel of CycleGAN as a principle homogeneous space. In Proceedings of International Conference on Learming Representations (ICLR), 2020.
Mostafa Mozafari.
Bitmoji
faces.
https://wwe www. https::/www.kagll1.
datasttsss
mosttafamozafarr//iiiiii faces 2020. Accessed September 20th, 2023.
Aamir Mustafa and Rafal Mantiuk. Transformation consistency regularization: semi-supervised paradigm for image-to-image translation. In Proceedings of the IEEE/CVF Conference on Com-
puter Vision and Pattern Recognition (CVPR), pp. 599-6199999 615, 2020.
Yingxue Pang, Jianxin Lin, Tao Qin, and Zhibo Chen. Image-to-image translation: Methods and applications. IEEE Transactions on Multimedia, 24:3859-3881 2021.
Taesung Park, Alexei A Efros, Richard Zhang, and Jun-Yan Zhu. Contrastive learning for unpaired image-to-ing -magge translation. In Proceedings of European Conference on Computer Vision (ECCV), pp. 319-345, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of International Conference on Ma- chine Learning (ICML), pp. 8748-8763, 2021.
Walter Rudin. Principles of mathematical analysis, volume 3. McGraw-hill New York. 1976.
Christian Szegedy, Vincent Vanhoucke. Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE/CVÉ Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818-28266 2016.
12

---

Published as a conference paper at ICLR 2024
Yaniv Taigman, Adam Polyak, and Lior Wolf. Unsupervised cross-domain image generation. In Proceedings of International Conference on Learning Representations (ICLR), 2017.
Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pp. 16451-16467, 2021.
Ting-Chum Chun Wang, Ming- Yu Liu, Jun Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional GANs. In Proceedings of IEEE/CVF Computer Vision and Pattern Recognition (CVPR), pp. 7798-8807, 2018.
Yaxing Wang, Salman Khan, Abel Gonzalez-Garciaa Joost van de Weijer, and Fahad Shahbaz Khan. Semi-supervised learning for few-shot image-to- to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4453-4462, 2020.
Wayne Wu, Kaidi Cao, Cheng Li, Chen Qian, and Chen Change Loy. TransGaGa: Geometry-ave unsupervised image-to-image translation. In Proceedings of IEEE/CVF Computer Vision and Pattern Recognition (CVPR), pp. 8012-8021, 2019.
Shaoan Xie, Qirong Ho, and Kun Zhang. Unsupervised image-to-image translation with density changing regularization. Advances in Neural Information Processing Systems (NeurlPS), 35:
2022.
Yanwu Xu, Shaoan Xie, Wenhao Wu, Kun Zhang, Mingming Gong, and Kayhan Batmanghelich. Maximum spatial perturbation consistency for unpaired image-to-image translation. In Proceed- ings of /EEE/CVF Computer Vision and Pattern Recognition (CVPR), pp. 18311-18320, 2022.
Shuai Yang, Liming Jiang, Ziwei Liu, and Chen Change Loy. Gp-unit: Generative prior for versatile unsupervised image-to-image translation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Yasin Yaz, Chuan-Sheng Foo, Stefan Winkler, Kim-Hui Yap, Georgios Piliouras, Vijay Chan- drasekhar, et al. The unusual effectiveness of averaging in gan training. In International Confer- ence on Learning Representations, 2018.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 586-595, 2018
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of IEEE/CVF Computer Vision and Pattern Recognition (CVPR), pp.
2017.
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1): 43-76, 2020.
Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In Proceedings of International Confer- ence on Machine Learning (ICML), pp. 12979-12990, 2021.
13

---

Published as a conference paper at ICLR 2024
Supplementary Material of "Towards Identifiable Unsupervised Domain Translationn Translation: A Diversified Distribution Matching Approach"
A PRELIMINARIES
A.1 NOTATION
2, & denote scalar, vector, and set, respectively.
p(æ)) and p(æ/u) denote the marginal probability density function (PDF) of and condi- tional PDF of conditioned on u, respectively.
| denotes the f2-mrrm of
(Aa BE la bl|2-
I: X' + X denotes the identity function such that (xx) = x, Va X.
A cl(AA), bd(A) and (.) denote the complement, closure, boundary, and the interior
of set A.
A set is is said to have strictly positive measure under p(x) and only P >
For a (random) vector a. sr1) and [az], denote the ith element of æ, and æ(i j) denotes [xt(t), z(i + xx(])
Distance between two sets is defined as
dist (A, B)
inf aEA.bEB
||a - bll2-
Distance between set and point is defined as
dist (a, B) = inf |a - bll2- BEB
(z) denotes the e-neighborhood of E R. defined as
N.(2) {2 R |R 2||2 < c).
cA) denotes the set of connected components of A (see definition of connected com- ponents in Appendix (.2))
For any function m W + Z, and set A W. m m(A) (m(w) Z w E A}
A.2 DEFINITIONS
We will employ standard notions from real analysis. We refer the readers to (Carothers) 2000 Rudin 1976) for precise definitions and more details. Here we provide working definition with illustration.
Connected set. A set C is connected (in X'), and only if there does not exist any disjoint non-empty open sets A, X' such that A A + Φ. BCC # o, and C A U B (see Fig. 9).
Not Simply Connected
Not Simply Connected
Simply Connected
Connected
Not Connected
Connected
Figure 9: Illustration of connected and simply connected sets
14

---

Published as a conference paper at ICLR 2024
Simply connected set. A simply connected set is connected set such that any simple closed curve can be shrunk to a point continuously in the set (see Fig.
Connected components. Given set A, the maximal connected subsets of A. such that the subsets are not themselves contained any other connected subsets of A, are called connected components of A. Specifically, connected set C A is connected component of A if there does not exist any other connected set D [ A, such that C D. In Fig. [O] denotes the entire shaded regions, and has three connected components Ci. C2, and Cg. Note that any set can be uniquely written as a disjoint union of its connected components. In Fig. 10 C₁ U C2 U C3 is a unique disjoint union representing A.
C
C3
3 Connected
Components
C2
Figure 10: A set A = C1 UC2 UC with 3 connected components: C₁, C2, and C3.
Continuous function. A function m + 2, with W C RRW 2 C R² is said to be continuous if for any E W and >>> 0, there exists 5 > 0 such that
m (WN((w) n W)) C C((((w)) n Z.
Continuous and invertible Functions. If function m W + Z is continuous and invertible, then its inverse m -1 > W also continuous. Some useful properties continuous and invertible function are as follows:
If A C W is closed, then m(.) also closed.
If A C is open, then m(A) also open.
If A C W is connected, then m (AA) is also connected.
B PROOF OF LEMMAS AND FACTS
Note that for the ease of reading, the lemmas, facts, and theorems from the main paper re-stated and highlighted using shaded boxes.
Proposition 1. Suppose that Paz admits continuous PDF, p(æ) and p(x:) > 0, Va E X. Assume that X is simply connected. Then, there exists a continuous non-trivial (non-identity) h(-) such that h = Pa-
Proof. We want show that there exists continuous h. X' - X' such that
h#F = Par-
To this end, we will construct such MPA by reducing the problem of finding MPA of p(æ) to finding an MPA of the uniform distribution. Note that one can always construct a continuous invertible function d X' + (0, 1x such that the function maps any continuous distribution with a simply connected support to the uniform distribution. This mapping can be found via the so-called Darmois construction (Darmois, 1951 Hyvärinen Pajunen 1999). Specifically, under the Darmois construction, the ith output of d(x)) Va E X given by
[d(z)), := F (I(1) x(1 i 1) - w( : ::l:i-1) 1)), i= I=..,,,
where F' (z(i) ()) denotes the conditional CDF of (z))) i.c.
F (x(i)) ax1 s01 - (00) [(x(1) : =((1) x()))
15

---

Published as a conference paper at ICLR 2024
more detailed introduction the Darmois construction in (Hyvärinen & Pajunen 1999)
With the constructed d, one can form a continuous mapping : X + as follows
h d hy d,
where by 1), + (0, is a continuous MPA on the uniform distribution over (0, P.. Since is continuous for continuous distribution, h is continuous because the composition of continuous functions.
Now, it remains show that hy exists. simple example by is reflection around the mean of d(x), i.e.
(2(2) 2 + 2j1,
where 14/2..... 1/2l" E RD= This concludes the proof.
Lemma 1 Assume that optimal solution of is (f.g. (d 1) Then, under Assump-
tion] we have P Pælu=u,
f #P ==ue
/ u=u; 9#P #//=mu,? Vi E [7], and f 9 a.e.
Proof.
Fis direct consequence of (Goodfellow et al. 2014 Theorem 1).
First all, recall the objective in
Σ
(CDSCAn (g. d xx,y) + LDSGAN , v,,)) + Ag. f).
min
22,,,,11
(8)
f.9 [d
The global minimum LDSGAN d x. y) achieved when (Goodfellow al. 2014 Theorem 1)
g alumu y/u=
Similarly, the global minimum of LDSCAN (f, (()) y) is achieved when
f#P p
P Pm|u=u-
Finally, the global minimum of Leye (g. f), which is zero, is achieved when
g ==f a.ee
know that g" and f* can achieve global minimums of all loss terms simultancously. Hence the solution of (7). f and y. should satisfy
Pat u= = -f#P w Palu=u, = P#Papumuy Vi E [7], and f f= 9-1 a.e.
Fact 2 Suppose that Assumption I
holds. Then, there exists an admissible MPA of (P Telu=,, // if and only if there exists an admissible MPA of (P Me
Proof. Let h be admissible MPA of Peluzu,HEn Then
hyPoluan = P Palu=uy Vi E [4].
g" h4PPluan, IxPolumm, Vi E [[].
g" a h fip P Pylu=ur E [/].
This implies that 9* oho f* is an admissible MPA of (P y H if and only if h is an admissible MPA of (Paj=sw))/
Hence, there exists an admissible MPA of {P (Palu=u,He Has if and only if there exists an admissible
MPA of (P (Pyl=m,,NE1
16

---

Published as a conference paper at ICLR 2024
C PROOF OF THEOREMS
C.1 PROOF OF THEOREM 1
Theorem Suppose that Assumption 7 holds. Let Ev, denote the event that the {P (Pnlumuy (ll does not satisfy the SDC. Assume that Pr[E.,] S for any i i j. where
i E [.. Let (f.g) be from an optimal solution of the DIMENSION loss 7.. Then, there is no admissible MPA of {P Paauumu Ji of the solution, i.e., f = f", a.e., 9 = g" a.e., with a probability of at least 1 p(á).
Theorem is a direct consequence of following lemma:
Lemma A.1. Suppose that Assumption / holds. Assume that (Palu=u, are sufficiently diverse. Then, g" and =
Proof of Lemma A. First, we show that no non-trivial continuous admissible MPA exists for {IP m|jmm 1/ 1= Le., if continuous h satisfies
h h# = Pa|u=a, Vi E [7],
(9)
then h = a.e.
Eq. 9.. by the definition of push-forward measure, implies that
Palu. [b) P Palu, [A], Vi [7].
(10)
For the sake of contradiction assume that h satisfies 9 however, h + I on set of strictly positive measure. This means that there exists a z E X' such that
h 2 T.
Now, let define open set around denoted by D such that
D = ==a((>>) n.K.
Because of the continuity and invertibility of h. h(D) C X' is also an open set and
h(z) E h(D).
Now, one can select d be small enough (because of the continuity of h) such that D n h(D) and D is a connected set. D being connected set implies that h(D) is also connected.
The above a contradiction to A Assumption since D and h(D) are two disjoint, open and connected sets which satisfy
IP Pa|ju [b()) PaIM DD. Vi E [4].
Hence, any h that satisfy h P
PP F m such that h I, a.c.
Finally, We want show that و g" a.e. Lemma implies that
I#P Pajumuu
y/u=o Vi [7]
9 aalumus
g Paluany Vi E [T]
9 Paluann
Pa P Vi [7]
(11)
where (a) obtained by applying g 1 on both sides, which is allowed because applying the same function preserves the equivalence of the distributions.
Eq. (T)) implies that g" gg continuous admissible MPA of (Paalu=u,) K= which means that the following has to hold:
9'1 a.a.e
Therefore, we always have ğ =9", a.c. By role symmetry of f and g" (also see Fact 2). also have f a.e.
17

---

Published as a conference paper at ICLR 2024
Proof of Theorem !
Using the assumption that Pr(E,,y p. the probability that (Pajumui M/ not sufficiently diverse can be bounded as follows:
Pr[[P al V are not sufficiently diverse)
(a)
Pr
E
(8)
Pr[En,]
<(())
where the (a) holds since { uuuu M not being sufficiently diverse implies the existence of open connected sets A and B such that
P Palu, [A]= P Pz (0l,,(8 [8] E [T]
Pa Paju. [A] Pz|u [B, Vi E {i,j) C [1].
Finally, (b) is due the independence of the events E and Eggy' for j $ j'.
Hence, (Paju=a, are sufficiently diverse with probability at least - pla), which implies that f f and g" with probability at least p(1).
C.2 PROOF OF THEOREM 2
Theorem (Robust Identifiabilityy. Suppose that Assumption 7 holds with g" being Lipsehitz continuous, and that any pair of (Po]ug: P Pa(u,) satisfies the r-SDC (cf. Defini-
tion 5 with probability at least 1 - 7,, i.e., Pr[Ms,j 2 r] S 7 for any + j. where (i,j) [1] [J]. Let g be from any optimal solution of the DIMENSION loss in 77. Then, we
have (g) g" (x))l2 s 2rL, Vx X, with probability of at least 1 - 7(1), The same holds for f.
First, consider the following lemma. Lemma A.2. Given any continuous admissible MPA h of ( (Palu=nHHE let En be set defined as
E/- = {x h(x)) + x, Vo E X).
Then, any connected component C C cl(E)) satisfies
x = h(z) C.
Lemma A.2 states an interesting property of a subset of X' (namely, εr) that is "modified" by the continuous MPA h. Here, "modification" means that any point in the subset will land on a dif- ferent point after the h--transformation. The lemma shows that the source point from &n and its h-transformation both reside in the same connected component, namely, This will useful in proving Theorem
Proof Idea: The main idea behind the proof of Lemma A.2 is to first note that any point outside of cl(En) is stationary under the transformation h (i.c., h(()) = x). Next, if there was point / from connected component C1 E conn(cl(Ex)) was such that h ( was not in C1, then it should be either in X (lE,)(E,, or in cl(E,)) (C.. However, since h is invertible, h cannot map point from Cn X' (.),). Therefore hxx) should lie in &, C,, But this will make the function h discontinuous. Hence, h( should be in CI.
Proof of Lemma A.2 Let conn(cl(Ex)) denote the set of connected components of cl(Ex). Suppose that there exists 2 C₁ and C₁ E (cln(c)(Enn) such that h(x) & C, First,
(a)
h ( = σ, Ya E X((c((E)))
h o ž vz E X (lE,) h x
C2 for some C2 conn(cl(Ex))CC.
18

---

Published as a conference paper at ICLR 2024
C1
C2
h (
22gg
Figure 11: Illustration of Lemma A.2 C1. C2 are the two connected components of cl In this case, cl(En) C2U,2. Points 2 and 2 inside C₁ and C2 stay inside the same connected component after transformation by h.
where (a) is due to the invertibility of h.
Because the continuity of h, the h(C1) is closed connected set containing hz)) However, h(C1) n (IE)) = ø. This means that
h(G,) S C2.
(12)
otherwise h(1)) would be disconnected.
Note that C₁ and C2 are closed, connected and disjoint sets (by the property of connected compo- nents). Therefore, one can define as follows:
== dist(C1, C2) > 0.
(13)
Now, take any point bd(C)), where bd(Ci) denotes the boundary of C,. Due to the continuity of h. there exists d > 0 such that
(NN(()))) S (Neea(h(ao))
(14)
However, take any point z E (aa))))ee(EE with || æb112 < c/4. Such point exists because any neighborhood of a point on the boundary closed set has non-empty intersection with the complement of the closed set. Therefore, we have
h(z) because E..
(15)
Since 12
implies that h(x))) C2.
3c - 2
[1((x))
2bble 2 e and dist((),.N(/((hxxz)))
Therefore
< | =112 dist(z,
3E
K
+ dist(zz.Nc/u(((zz)))
4
dist(2,Ne./ (1a
2V
dist(h(=).NN, (h
IV
(16)
where 16 is by 15
Note that (66) is contradiction
14p. Hence, have
C for any C conn(cl(E.)) h(x) E C.
This concludes the proof.
Lemma A.3. Let g" be L-Lipschitz continuous. Suppose that any pair of (P u.: P Paluy) satisfies
the DDC (ef. Definition 5j. Then
[g(()) g" (a:))12 & 2r
(17)
19

---

Published as a conference paper at ICLR 2024
Proof Idea: The proof is by contradiction. Suppose that under the conditions of Lemma A.3
Eq. (17) does not hold for some a E X. Then, there would exist a continuous non-trivial admissible MPA h of (Paju=u, ) such that f((x) - æ||2 > 2r. However, this would imply, using Lemma
A.2 that one can construct an open, connected, disjoint set pair (A, B) whose diameters are large,
which is a contradiction to r-SDC that M S
Proof of Lemma A.3
Suppose that there exists E X such that
(T))
( 2rL.
(18)
Eq.
18
means that g * g". By Lemma
we have that
g g#P
IP yluu=
I#Falumm
9 Pa al
If MEN
9
I
Pala-as
g
g MEN
P
20) P
As g # g", the function h = 9" # is continuous admissible MPA of (Paju=e,Jes Men. This implies that
g" o h.
(19)
Using
19
Eq.
18
implies that
llg" h((x)) 9" (2) 2rL
+L
h(( T 2r
hT()) . 2r.
(20)
Note that one can re-express (20
h i
ø|2 2r c,
(21)
using a certain e 0. By Lemma
A.2
we know that
ZEč, π(x)) EC,
where Č connected component of clE)))
Now, let d >> 0 and
Ta N2
Let Ra denote the connected component of Ta that contains Note that we need to consider the connected component as Ta can a disconnected set. An illustration of these sets can be seen in
Fig. 12
From 21 easy to see that dia(C) 2 2r + e. Then, when S 2r + e, since connected, bd(Aa n # db Note that the connected property necessary for bd(Na(>:)) C + e to hold for any d < + c. One can further select w E bd(Na(z)) nC such that w cl(R.). i.e., ur lies in the same connected component of Ta as E.
Note that such wo to exist. Suppose that such a w does not exist. Then, cl(R.) bd(Na) Φ. which means that Ra would be disconnected from Cy, By the definition of Ra. Ra then disconnected from Tay Ra. which implies that C that is a union of Ra, Ra, and C/a2 is disconnected. This is contradiction. Hence cR(..) bd((2) & o holds.
One can see that
l 22 = d, and w. E cl(Ra) = dia(Ra) 2 d, for d < 2r +c.
Hence, there exists large enough d < 2r c such that
0 dist(RR, h(Raa) < e/3.
20

---

Published as a conference paper at ICLR 2024
C
Na
R.,(z) (x)
h (R
h
Figure 12: Illustration of the idea in the proof of Lemma A.3 The green shaded region denote Rd- Note that Ta = (a) nCC is disconnected in this case.
This implics that
(ix) 2
(22)
Indeed, suppose that max(dia(Raa) T(Ra),,
< r + e/3. Then, since x Ra and h(x)))
h()))l 2max (ia) dist k(R.a)
2r +e < 2r + 2 + e/3
2r + 2r c,,
which is a contradiction. Hence,
dia max (ia(h))))) 2r e/3.
(23)
Fig. 12 provides simple illustration of the sets. It follow from the continuity and invertibility of h that (Ra) (aa(iit(Rz)) and dia (R((a) dia(int(((Ra)))
By the same argument of reaching disjoint sets such that
(10). {int (R.), hiint hRn,,, (Ra))) forms pair of open, connected,
Palu, [int P Pij, (Tiin((R.))))
(24)
Note that 24 and
233 constitute a contradiction to the assumption that max(dia(A)) dia(B))
M
(A,B)E
for any open, connected, and disjoint sets A and + Hence, we must have
(g(x) g" (2=)ll2 & 2rL, X.
This concludes the proof.
Proof of Theorem 2
We can bound the probability with which 17 does not hold as follows:
Pr[
(7) does hold]
Pr[M
r]
(a)
Pr Mrj
Pr Pr[M
< (1)
"Note that such requirements are used ensure that the sets have nonzero measure. The statements can be simplified by replacing the "open and non-empty" sets in Definition Assumption 5 with "measurable sets with positive measures".
21

---

Published as a conference paper at ICLR 2024
where (a) follows because M >> implies that Mij r, Vi J. and i.j E [2] holds, and (b) follows from the independence the events Ma. > r.
Hence, with probability at least 1-7())
[ī(x) g (x))l2 S Vr EX.
The same result follows for if f L-Lipschitz continuous, following the same procedure as above.
D ADDITIONAL REMARK: RELATION TO SUPERVISED DOMAIN TRANSLATION
remark on objective (7) is that supervised domain translation can be seen as a special case of 7.
When paired samples {æ,, yi are available, one can view the auxiliary information u as the sample identity. Specifically, P >=lw=wu and P u.m. are Dirac delta distributions peaked æ, and y: respectively. Matching distributions between IP m//u=u; and P will be equivalent to enforcing a, f (y:)) Therefore, the sample loss will be equivalent to minimizing the following objective:
minimize fgg
Σ ax
f(v))l + 11: g
which is exactly the supervised learning loss. This makes the distribution matching problem boil down sample matching problem.
E SYNTHETIC DATA EXPERIMENTS
In this section, we use controlled generation validate our identifiability theorems.
Data Generation. We generate æ from a Gaussian mixture with Q components. Let (P (p(0) denote the Q component distributions of the Gaussian mixture, i.e.,
sq=1
p(a) NWpg E). = 1.....Q Q.
Here, each My is sampled randomly from the uniform distribution R², i.e. Unif ([-1, We set covariance to be = 0.3² P. To represent g* we use three-layer multi-layer perceptron (MLP) with smoothed leaky ReLU, which is defined as s(x) ar + (1 ax) log(1 + exp(x)), where we set a to 0.2. To make 9" invertible, we generate the neural network weights using the same process in (Hyvärinen & Pajunen| [999] Zimmermann et al. 202I). Specifically,
use two-hidden units in each layer. We first generate 10. 000 2 x matrices, whose elements are sampled randomly from uniform distribution Unif([-1, 1, 1)). The matrices' columns normalized by their respective lo norms. In addition, only the top 25% well-conditioned matrices in terms of the condition number are used. This way, all the layers of the g" are relatively well-conditioned invertible matrices. Combining with the fact that the activation functions are invertible, such constructed g" in each trial also invertible.
We use N = 20, 000 samples in both domains, denoted as {x, J. {vn } to be the training samples. In addition, we have 1,000 testing samples. The data generation process is as follows:
ag Uni,([-1,1]}) 1]3), Va E [Q]. Fgg- Nm P(g), Vn [N₂] Aq [Q],
y( 1)N = g" (I),,),,++
where N, = [20000/Q], indicating that the mixture components have equal probability. our experiments we use (I.,, Vn''s association with one of the Q mixture components our auxiliary variable. Therefore, we have = Q. addition, u is uniformly distributed, i.e., Pr(u = u,) = 1/Q, Vg E [Q), and
P u=
pP())
22

---

Published as a conference paper at ICLR 2024
i fgy)
(g)
111
60
-05
-6
-10
-20
((x)
@ (ar)
-12
0.5
-122
-04
-0.4
-05
0.7
-05 -10 -35
110
455
-101
60
Proposed
CycleGAN Loss
Figure 13: Scatter plots of the source and translated samples. The proposed method uses / = Q = 3.
Evaluation Metric. In the synthetic data, have access to the ground-truth pairs (ær, yn)) Hence, we measure the translation error (TE) using
N 1/2N (ll (xn) yn 11 + 11 (y)) x2.113)
TE
Implementation Details. To represent g R2 -> R² and f RR - R2, we use three-layer MLPs, where 256 hidden units are used in each of the 2 hidden layers. We also use leaky ReLU activations with slope of The discriminator is five-layer MLP with 128 hidden units in each of the hidden layers. Each layer, except for the last, is followed by layer normalization (Ba et al. 2016) and leaky ReLU activations (Maas et al. 2013) with a slope of 0.2. We use the same architecture for all discriminators in DIMENSION In the synthetic-date data experiments, matching module using the least-square GAN loss (Mao et al. (017).
implement the distribution
Baseline. In the sythetic experiments, of naive distribution matching. Hence, we use the CycleGAN loss (3) as a benchmark.
purpose is to show the lack of translation identifiability
Hyperparameter Settings. We the Adam optimizer with initial learning rate of 0.0001 with hyperparameters 31 0.5 and 32 = 0.999 (Kingma & Ba 2015). Note that |3| and 33 are hyperparameters of Adam that control the exponential decay rates of first and second order moments, respectively. We use a batch size of 1000 and train the models for 2000 iterations, where one iteration refers to one step of gradient descent of the translation and discriminator neural networks. We use = 10 for (7.
Results. Fig. 13 shows the scatter plots of the original and translated samples for the 1000 testing samples. Here, we set I = Q = 3. The original data {x, J.V and [ are plotted on the left- most column. The result of translation using DIMENSION and CycleGAN Loss are presented in the middle and right columas, respectively. In order to qualitatively evaluate the translation performance, we use the same color to plot the paired data points (æn,, yn) and their translations (f(y)) (().)). The color is determined by the angle of wn in polar coordinates.
As see, the supports of and fly) (as well those of y and g(x)) are well matched by both methods. This implies that both methods can match the distributions fairly well. How- ever, CycleGAN Loss misaligns the samples (by observing the color). The results given by DIMENSION does not have this misalignment issue.
Fig. 14 shows the average TE (over 10 random trials) and the standard deviation attained by DIMENS ION and the bascline under different Q's. Here, we also set Q as before. One can that the average TE decreases with the increase in J. Notably, the variance of TE also becomes
23

---

Published as a conference paper at ICLR 2024
0.6
CycleGAN Loss
Proposed
0.4
0.2
0.0
2
3
Figure 14: TE under various l's.
much smaller when I grows from 5-this shows more stable translation performance when increases. The result is consistent with our theorems, which shows that having a larger / has better chance avoid MPA.
F REAL.-DATA EXPERIMENT SETTING AND ADDITIONAL RESULTS
In this section we provide details of the real-data experiment settings.
F.1 OBTAINING (u1....,
MrM and ErS Datasets. For these two datasets, we use the available category labels as the alphabet of u. Specifically, for MrM dataset, we use fI..... 10), i.e., the labels of the identity of digits. For ErS datasets, we use u (shoes, sandals, slippers, boots), which indicates the types of the shoes/edges.
CB Dataset. In this dataset, we designate the alphabet of u to be =" black hair =" non-black hair". Ug =" male", u4 =" female". This information not fully available in the original CB dataset (to be specific, Bitmoji (Mozafari 2020) has the gender attributes available the hair color not available). We use the loundation model, namely, CLIP (Radford et al. 2021]. acquire the hair color information of each Bitmoji face. Specifically, prompts "a cartoon a person whose hair color mostly black" and "a cartoon of a person whose hair color is not black". The presence of black hair for each image is decided based cosine distance of the image embedding with the text embeddings of the two prompts.
use the text
F.2 NEURAL NETWORK DETAILS
We use the nomenclature in Table to describe the neural network architecture. For example,
Conv-(C-Ni.. Nout, K-N S-N,, ZP-Np). LN, LeakyReLU
refers to convolutional layer with Ni input channels and Nout output channels: K-N, means that the size of kernel is NR: S-N, means that the stride is N₁; and ZP-N, means that the zero padding has size of Np. The convolutional layer is followed by layer normalization (LN) and then LeakyReLU activations.
The translation neural networks, and f. for images of size 256 x 256 follow the architecture outlined Table 3 For images of size 128 x 128, a modified architecture used, where one down-sampling layer (see Layer #6) and one up-sampling layer (Layer # #11) in Table 3 are not included. For images of size 32 x 32, three down-sampling layers (indices from #4 4 to #6) and three up-sampling layers (indices from #11 to #13) not included.
ResBlock refers to block of convolutional layers with shortcut connection and optional downsam- pling. Specifically, MABBlock-CC-M-N Operation) is composed of two smaller blocks, namely, Process-(C-M-3) Operation) and Shortcut-CC-M-NN Operation). The Pro-ess-CC-M-N. Opera- tion) block has the following layers:
24

---

Published as a conference paper at ICLR 2024
Table 2: Nomenclature for neural network components
Abbreviation
Definition
Conv
Convolutional Layer
IN
Instance normalization
ReLU
ReLU activation
LeakyReLU
Leaky-ReLU activation with 0.2 slope
Tanh
tanh activation function
UpSample
Upsample using nearest neighbor with scale factor of Downsample using average pooling with a scale factor of 2 Kernel (filter) size
DownSample
K-N
S-N
Stride of size
ZP-N
Zero Padding of size
C-M-N
M input and output channels
Table 3: Translation neural network architecture for f and g-
Layer Number
Layer Details
1
Conv-(C-3-64, K-1, S-1, ZP-0)
2
ResBlock-(C-64-128 DownSample)
3
ResBlock-(CC-128-556 DownSample)
4
ResBlock-(CC2556-51 DownSample)
5
ResBlock-(C-512-551 DownSample)
6
RcsBBock-(C-512-512, Dann
7
ResBlock-(C-512-512..
8
ResBBock-(C--112111.
9
ResBlock-(C--512-512.
10
ResBBock-(CC-51--512.
11
ResBlock-(C-512-512 UpSample)
12
RcsBlock-(C-512-112, UpSample)
13
ResBBoak-(CC-1122256 UpSample)
14
ResBloak--C--55-112,, UpSample)
15
ResBlock-(C-112-644, UpSample)
16
Conv-(C-64-3, K-1, S-1,
1. IN, LeakyReLU, Conv-(C-M-M, K-3, S-1, ZP-1)
2. Operation
3. IN, LeakyReLU, Conv-(C-M-N, K-3, S-1, ZP-1)
The SC-M-, Operation) block consists of the following layers:
Conv-(C-M-N, K-1,5-1,7P-0)
2. Operation
Let denote the input to the ResBlock and w the output of the ResBlock. Then the forward pass of ResBlock is expressed follows:
w = ResBlock(z)) Process(2 (z) + Shortcut(z).
2019 with output dimension of I represent ( d Vi E [4]. Specifically, each of the multi-task discriminators d, and d, has / output dimensions. The ith outputs of d,, and d,, correspond d!!)) and df respectively.
We multi-task discriminators (Liu
25

---

Published as a conference paper at ICLR 2024
Table 4: Discriminator architecture for dæ [2256x256x3 R!, dy IR256X : + +R.
Layer Number
Layer Details
Conv-(C-3-64, S-1,
2
ResBlock-(C-64-188, DownSample),
3
ResBlock-(C-128-556, DownSample),
4
ResBlock-(C2256-522 DownSample),
ResBlock-(C-512-512, DownSample),
6
RexBlock-(C-51225122 DownSample),
7
ResBlock-(C-511251 DownSamplel, LeakyReLU
Conv-(C-512,512, K-4, S=1, ZPP0), LeakyRel.U
9
Reshape-512 Linear-(512.I)
10
F.3 HYPERPARAMETER SETTING
We use the Adam optimizer with an initial learning rate of 0.0001 with hyperparameters B1 = 0.0 and Bg 0.999 Kingma & Ba 2015). Note that $1 and 32 are hyperparameters of Adam that con- trol the exponential decay rates of first and second order moments, respectively. We set our regular- ization parameter À = 10. We use batch size of 16. We train the networks for 100,000 iterations. Following standard practice, we add squared f--oomm regularization on the network parameters and weight decay of 0.00001. For the translation tasks with 256 x 256 images (CelebA-HQ to Bitmoji Faces), the runtime using a single Tesla V100 GPU is approximately 55 hours. For the translation tasks with 128 128 images (Edges to Rotated Shoes), the runtime using single Tesla V100 GPU is approximately 35 hours. In order to stabilize the GAN training dynamics, we add gradient penalty term. This term penalizes discriminators' large gradients which is known to help the convergence of the GAN objective (Mescheder et al.) 2018). We modified the rgeularizer to accommodate our diversified DT loss function. The modified regularization term as follows:
R
7 Pr(uu = uz) (EzorR u-m
( Ey V (
where Vdl denotes the gradient of d()). set the value of y to be 1.0. We take exponential moving average (EMA) of the parameters during training as the final estimate of the parameters of the trained neural networks. We use a weighting factor of 0.999. This has been observed to improve the performance of GANs (Karras et al. 2017 Yaz et al. 2018).
F.4 DATASET DETAILS
MNIST to Rotated MNIST (MrM), We use 60, 000 training samples of the MNIST digits (LeCun al. 2010) that have dimension 28 28 as the X'ddomai.. For the f-domain, each of the 60, 000 digits is rotated by 90 degrees. The orders of samples are shuffled in both domains to "break" the content correspondence. Under this setting, each æ has ground correspondence y.
Edges to Rotated Shoes (ErS). Edges2Shoes dataset (Isola et al. (017) consists of 49, 825 training samples. We resize the all images to have 128 x 128 pixels. The X -domain corresponds to edges of the shoes, and the y-domain corresponds to the shoes that are rotated by 90 degrees. Like in the MrM dataset, the ground- woound-truth correspondence known to us, which can assist evaluation.
CelebA-HQ to Bitmoji Faces (CB) We use 200 training samples from CelebA-HQ (Krras cet al., 2017) as the æ-domain, and 3, 984 training samples from Bitmoji faces (Mozafari 2020) as the y domain.. Note that Bitmoji Faces consists of only 4, 084 samples in total, of which 100 samples are held out as the test samples. We resize all images both domains to have 256 x 256 pixels. Unlike the previous two datasets, the ground-trult correspondence is not known us in this dataset.
Evaluation Details. The LPIPS score computed using 100 test samples. Pre-trained Arezhe((Kr et al. 2012: 7hang et al. 2018) is used in order compute the I.PTPS scores.
26

---

For StarGAN-v2 and GP-UNIT, training is done with their default settings (specifically, the con- figurations for the 'AFHQ' dataset in their papers are used). For CycleGAN+ Id, UNIT, MUNIT, I-GAT-IT, and Hneg-Siiie SRC, we train the models for 200,000 iterations. We use a batch size for these methods except for U-GAT-ID IT, which uses 4 in order to control the computational load and runtime. These parameters are carefully set for the baselines to our best extent. For OverLORD (Gabbay & Hoshen 2021), use the setting used for male to female translation task CelebA -HQ dataset in their paper. For ZeroDIM (Gabbay et al. 2021), we use the setting used for experiments on FFHQ dataset, which has similar size the datasets used in paper. Note that ZeroDIM Published as a conference paper at ICLR 2024
The FID score is computed using 1000 translated and real samples for each domain. Pre-trained Inception-v3 Szegedy et al. 2016 is used order to compute the F IDD scores.
F.3 BASELINES
We use CycleGAN+ Id Zhu et al.
2017 UNIT
(Liu et al. 2017 MUNIT (Huang et 2018
g
D-CAT-IT IT Kim et al. 20000) p
StarGAN CAN-v2 Choi et al. 2020)
ZeroDIM Gabbay et al. Hneg-SRC (ung et al., 2022) GP-UNIT
2021
Ove LORD (Gabbay & Hoshen 2021
Yang et al.
2023 10 and the plain-vanilla Cyc eGAN Loss in (3) as the baselines.
also uses the same auxiliary variables as those used the proposed method.
F.6 ADDITIONAL RESULTS
In this subsection, we present additional qualitative and quantitative results.
Fig. 16 shows the result of translating Bitmoji faces (B) to celebrity proflie photos (C). As men- tioned in the main text, translating from the B domain to the C domain is hard task as the learned translation function needs to "fill in" lot of details to make the generated profiles photorealistic. Visually, one can see that the proposed method (with alignment relative the baselines. In addition, the proposed method using 2 (only using 'male' and 'female" as the auxiliary variable alphabet) also provides more satisfactory results relative to the baselines. This echos theoretical claim that the chance of attaining translation identifiability grows quickly when 7 increases. It also shows that diversifying the distributions to be matched, even if just one more distribution pair is included, helps improve the final performance.
4) exhibits much more intuitive content
Fig. 15 shows the result translating edges (E) to rotated shoes (rS). Visually, our method sig- interesting notice that,
nificantly outperforms the baselines terms of content alignment. although "edges to shoes" (no rotation) is a well studied dataset, our experiments show that simple rotation makes most of the existing methods struggle to produce reasonable results. However, our method is insensitive to this kind of geometric changes. In the literature, the baselines U-GAT-IT (Kim et al. 2020) and GP-UNIT (Yang et al. 2023) were shown to be good handling certain ge- ometric variations. However, one can see that their performance over the ErS dataset is still far from ideal. The result shows the importance of taking transaltion identifiabiltyy into account, especially when drastic geometric changes happen across domains.
Fig. and Fig. 17 show similar results for the translation of CelebA-HO -HQ (C) to Bitmoji Faces (B) and Rotated Shoes (rS) to Edges (E), respectively. Fig. 19 shows the translations between MNIST (M) and rotated MNIST digits (rM).
huttss//gthhbbcom//aaii///eeeree
"attps//gihuub.om//yyyy12211111 SRC.git
27

---

Published as a conference paper at ICLR 2024
Source
Target
Proposed CydeGANiLow CydeGAN
UNIT
MUNIT
U-GAT-IT
Strgasa2
Heeg SRC
GP-UNIT
Over ORD
ZerDDIM
Figure 15: Translation of edges to rotated shoes. All images rotated by anit-clockwise 90 degrees for visualization.
28

---

Published as a conference paper at ICLR 2024
Sourcee
Propored
Proporel (1-2)
OOKSAN Lots
CydlesAN-id
UNIT
MUNT
U-GAT-IT
Strgan-12
Hins SRC
GRUNNTT
OvnLORD
ZeroDii
Figure 16: Translation of Bitmoji CelebA-HQ.
29

---

Published as a conference paper at ICLR 2024
Source
Torget
Propased CydeGAN Loss CydeGAN
UNIT
MUNIT
U-GATIIT
Starponv? v2
Heng SRC
GP-UNIT
Overl ORD
ZeroDiM
8
Figure 17: Translation of rotated Shoes Edges.
Sce
Pepaed
Eapaad 2-2)
GeNG
UNIT
WAVI
Sagees2
HANSRC
GPAVIT
DeelO
ZANO
Figure 18: Translation of CelebA-HQ to Bitmoji.
30

---

Published as a conference paper at ICLR 2024
GyKGMWyyGGG
7
7
7
4
4
8
4
2
0
0
6
27
2 2
7
D
6
0
0
6
5
5
5
9
4
7
4
7
3
O
9
O
200
1
1
7
0
2
8
0
9
99
20
0
0
0
2
5
6
5
9
9
1
1
7
6
7
9 7 2
9
4
4
4
6
1
7
9
9
4
7
7
9
7
1
3
7
O
2
O
9
1
7
2
6
9
4
Figure 19: Translation between MNIST digits and rotated MNIST digits.
There are multiple ways to define auxiliary variables for a given UDT task. However, different choices of auxiliary information can result in different level of translation performance. For exam- ple, the MNIST example, one can alternatively use digit shape as the alphabets of the auxiliary variable. Figure 20 shows the result of using different shapes of digits. Here ing attributes (with corresponding digits that has those attributes): "line" [1,2,4,5.7], "circle". [0,6,8,9], "curve" [0,2.3.5,6,8,9), "vertical line" [1, 4, 5]. "horizontal line" [2.5,4.7], "curve without loops": (2.3,5,6,9). "only vertical line" : [1]. "only horizontal line" [2]). One can see that using digit identity auxiliary information results in slightly enhanced performance compared to using the digit shape as auxiliary information.
the follow-
G IMPROVING EXISTING METHODS USING DIVERSIFIED DISTRIBUTION MATCHING
We hope emphasize that the diversified distribution matching (DDM) principle be combined with many other existing UDT approaches to avoid failure cases. In this section, we use our di- versified distribution matching module to replace the their original ones in existing paradigms and observe the performance. For the datasets "Edges vs. Rotated Shoes" (ErS) and "CelebA-HO vs. Bitmoji" (CB), we select the baselines that are able to generate faithful samples in the target domain based on their F ID scores (see Table To be specific, for the ErS dataset, we integrate DDM with UNIT (Liu et al. 2017). For the CB dataset, we combine DDM with GP-UNIT (Yang et al. 2023). In both cases, we keep their method-defined regularization terms and other settings unchanged. We refer to the modified methods as UNIT-DDK and GP -UNIT-NIIDDI respectively.
Our way of combining DDM with these existing approaches is to replace their discriminators. To obtain UNIT-DDM and GP-UNIT DDM, we modify the discriminator neural networks of UNIT and GP-UNIT into multi-task discriminators. Specifically, for UNIT-DDM, the multi-scale discriminator of UNIT which has one output channel for each scale, is modified to produce 1 output channels for each scale. Similarly, obtain -E--NII---EE have / output channels instead of one output channel at the output layer. The ith output channel is interpreted the ith discriminator associated with u,
SINIT-DDY DDM, the discriminator of IN-TIIT is modified to
Fig. 21 shows the qualitative results attained by the original versions of UNIT and GP-UNIT as well as their DDM-modified versions. One can see that there is significant improvement in terms of content alignment, without compromising the visual quality-sec the F FID and LPIPS scores in Table 5 This attests the hypothesis that iestriuttionmmtchiin based domain translation frameworks can benefit from the proposed MPA eliminating idea.
31

---

Published as a conference paper at ICLR 2024
Prepocal Proposal CydeGAN CydcGAN
Sourse
Torpet (Dug Hest Tahapeo)
Lox
-18
MUNIT STGT--ISSmngss
7
7
7
4
8
4
2
3
3
3
3
4
/
7
8
O 9
3
9
0
9
9
9
8
9
7
9 9 3
3
4
4
4
6
0
0
3 9
1
Figure 20: Result of using different auxiliary variable for MNIST digits rotated MNIST digits
task. Using shape attributes incur LPIPS=0.19 + 0.08, compared to LPIPS=0.11 + 0.08 for the digit identity as auxiliary variable.
Table 5: The FID and LP IPS scores attained by UNIT and GP-UNIT as well their DDM-modii modified versions.
FID (4)
APIPS (L)
Method
Edges
Shoes
Edges
Rot. Shoes
Rot. Shoes - Edges
UNIT
33.95
96.28
0.49 0.035
0.58 0.038
UNIT-DDK
43.95
88.58
0.30 +: 0.075
0.35 + 0.092
Method
CelebA-HQ
Bitmoji
GP-UNIT
32.40
30.30
GP-UNIT-DDM
37.79
30.33
32

---

Published as a conference paper at ICLR 2024
GP-UNIT- DDM
Source
GP-UNIT
Source
Target
UNIT-DDM
UNIT
Figure 21: [Left] Result of auxiliary based diverse distribution matching GP-UNIT
Bitmoji -> CelebaA-HQ translation task. [Right] Result Conditional distribution matching on UNIT Edges > Rotated Shoes translation task.
33

---

Published as a conference paper at ICLR 2024
Table 6: LPIPS score attained by DIMENSION using random u, assignments.
random iy proportion
MNIST
MNIST
Rot. MNIST MNIST
0%
0.11 0.082
0.047
20%
0.09 = 0.050
0.08 + 0.040
40%
0,10 = 0.049
0.13 1 0.064
50%
0.19 0.080
0.19 0.083
60%
0.25 = 0.124
0.21 + 0.086
Source
Target
0%
205
40%
50%
60%
Source
Target
0%
20%
40%
50%
60%
2
2
2
2
m
3
3
3
3
3
3
7
7
7
2
7
0
0
9
9
9
9
4
9
9
9
9
9
9
7
7
57
5
5
5
5
5 6
5 6
5
5
5
5
5
5
9
6
6
6
6
6
6
6
6
6
6
6
Figure 22: Result of DIMENSION under random assignments to various fractions of training data.
H ROBUSTNESS TO NOISY AUXILIARY VARIABLES.
It is of interest to know whether using noisy wrong auxiliary variables would heavily affect the performance of DIMENSION, To this end, we assign random u:'s to a fraction of the training samples in the "MNIST vs. Rotated MNIST" dataset.
Tnd Fig. 22 show the LPIP LPISS scores and qualitative results attained by DIMENSSON, respec- tively, under different fractions of random (and highly possibly wrong) auxiliary variables. Notably, there is almost no performance degradation of D IMEESTTTN ION even when 40% of the assigned u, random. This shows the method's robustness to wrong/noisy auxiliary variables.
34


Reviews:
[{"review_id": "qVNEvRAwLl", "reviewer": "Reviewer_4LhS", "summary": "The paper aims to propose an unsupervised image translation framework that ensures identifiability of underlying generator maps. It achieves the same by relying on auxiliary variables. Promising empirical results showcase the potential of the method on benchmark image datasets.", "questions": "1. My first concern is regarding the strong assumption that continuous functions $f^*$  and  $g^*$ exist under arbitrary input data on both domains. It is quite challenging to ensure that the optimal transport map between distributions has any regularity (e.g. Lipschitz continuity). Also, in most cases proving so requires the support of the base distribution to have restrictions in terms of convexity. In real data, the same hardly follows. Perhaps sacrificing generality for the sake of accuracy would be better for the theory.\n\n2. The discussion on the notion of \"content\" seems vague. Is the content of an image and its rotated counterpart the same? Is there any generalization of it for general group actions?\n\nIs there any way of justifying Assumption 1, even if with examples? \n\nIt seems to me that homeomorphic spaces will have the same \"content\", whereas the assumption claims the converse. Am I right in saying that?\n\n3. Given there exist non-unique members in the kernel (i.e. multiple solutions bringing about zero loss), is Definition 1 even meaningful in a non-parametric setup, where there is no inherent identifier?\n\n4. The entire theory revolves around the CycleGAN loss in particular. This does not complement the initial impression of unsupervised domain translation in general. The proposed loss function ($7$) is also a modified CycleGAN setup. Also, what are the \"any criterion\" in Fact 1?\n\nDoes the discriminator play any role in ensuring identifiability? This seems crucial as the resultant translation map would be a result of a stable discriminator.\n\n5. [Section 3] Can this at all be called unsupervised given that pseudo or weak labels ($u$) are used? Also, what is meant by \"sufficiently different\" $P_{x|u_i}$ and $P_{x|u_j}$?\n\nShouldn't the difference between distributions $P_{x|u(A,B)}[A]$ and $P_{x|u(A,B)}[B]$ be based on a divergence measure and not inequality ($\\neq$)? \n\nThere remain some typographical/grammatical errors in the manuscript (e.g. see the Section Identifiability Characterization).", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The writing of the paper is good with detailed exposition of the problem. It also includes detailed notes on related literature. The paper produces promising qualitative and quantitative results based on experiments. It also gives ample ablation and suggestions on the architecture and parameters involved. The brief declaration of limitations is appreciated.", "weaknesses": "The theory under quite strong assumptions tends to be straightforward and does not fully complement what the paper set out to achieve. It revolves around a particular model and due to certain vague notions becomes somewhat vacuous. The empirical results give the paper strength which the theory fails to support. In my opinion, the experiments should be prioritized.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GwxeHlvI75", "reviewer": "Reviewer_bENZ", "summary": "This paper seeks to address the issue of content misalignment in unsupervised domain translation. The authors pinpoint the presence of \"measure preserving automorphism\" (MPA) as the primary culprit and present a theoretically-founded method to neutralize it. Their method was validated on datasets like Edges to Rotated Shoes, yielding high-quality samples that maintained content integrity.", "questions": "In this study, it appears that only one auxiliary variable is used to diversify the distribution. What would happen if we used several variables? For example, we're considering not just the distributions $p(x|u=u_i)$ and $p(y|u=u_i)$, but also $p(x|v=v_j)$ and $p(y|v=v_j)$. Intuitively, utilizing one auxiliary variable is akin to \"slicing\" the original distribution in one way, while employing multiple variables is like trying different ways to make the \"cuts\".", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It is innovative to introduce auxiliary variables for tackling the MPA issue. I'd like to offer more insights on this approach. Essentially, **supervised domain translation can be seen as a specific instance of their method.** By choosing a specific auxiliary variable, we can tailor each conditional distribution $p(x|u=u_i)$ to hold precisely one sample, $x_i$, with a probability of 1, whereas all other samples in the space have a zero probability. Similarly, we manipulate each conditional distribution $p(y|u=u_i)$ to include only one sample, $y_i$, also with a probability of 1. These corresponding sample pairs, $(x_i, y_i)$, are essentially the supervised pairs for domain translation. By adjusting loss function 7 (i.e., the distance metric of cycle loss and the balance parameter $\\lambda$), this approach could replicate any supervised domain translation methods. The paper's impact would be significantly enhanced if the authors included this observation.\n\n2. Overall, the structure of the proof is clear and easy to follow.\n3. The experiments verified that this method could generate content-preserved samples with high quality, which corroborates their theory.", "weaknesses": "Overall, the structure of the theory is clear. However, there are several mistakes that should be corrected:\n1. The MPA of the PDF of a gaussian distribution $N(\\mu, \\sigma)$ should be $h(x) = 2\\mu - x$, rather than $h(x) = \\mu - x$.\n2. Within the \"Notation\" section of the introduction, \"A\" ought to be a subset of \"Y\", not \"X\".\n\nHonestly, it is impractical to check every detail of the proof. The author should ensure the proof's rigor and review it meticulously.\n\nAdditional suggestion: Assumption 1 is confusing. I think it refers to the existence of the content-preserved mapping $f^*$ and $y^*$. Please make it clearer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A0W8cwdIMw", "reviewer": "Reviewer_siTh", "summary": "The paper studies Unsupervised domain translation (UDT),\ne.g. to learn to generate cartoon sketches from ID photos\nwithout supervision. The authors study why CycleGANs fail to\nlearn the desired UDT function; previous work has suggested that\nthe reason is the existence of automorphisms of the generative\ndistributions; they corroborate this suggestion with a theoretical\nargument and then propose a second theoretical argument that\nprevents existence of such automorphisms if one introduces conditioning\non auxiliary variables. They then show the effectiveness of the proposed\nautomorphisms elimination approach on a few benchmarks.", "questions": "My initial rating inclines towards acceptance. A limitation of my review is that I have not a direct experience with the baselines, so I cannot assess if the chosen baselines were too easy to beat.\n\n**Questions**:\n1. In assumption 1 how realistic are the invertibility assumptions?\n  Are there weakened versions, e.g. in a probabilistic sense?\n2. Regarding Proposition 1 and the MNIST example in Figure 1, it seems\n  that for MNIST the support of $P_x$ would not be path-connected, with one path-component\n  for each digit. Then Proposition 1 would not apply directly. Can you formulate a case of Proposition 1 that would apply to this case?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well written and easy to read.\n\n2. The suggested elimination idea is well-motivated and simple\n  to implement.\n\n3. While *Theorem 1* operates under idealized assumptions, *Theorem 2*\n  makes an attempt to show that the author's proposal is robust\n  under more realistic circumstances.\n\n4. The UDT tasks they experiment on seem challenging enough to be interesting.", "weaknesses": "The abstract sounds very specialistic to me. I think the paper might be of interest to a broader audience, but some readers unfamiliar with the jargon might be put off by the abstract.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MBEbyaRJmW", "reviewer": "Reviewer_rKLC", "summary": "The paper tackles common failures in CycleGAN and variants where the desired translation functions are not successfully identified and the methods produce content-misaligned translations. \n\nThis limitation is claimed to be related to the presence of multiple translation functions (MPA). The authors introduce an MPA elimination theory and suggest a modified learning approach in which the cross-domain distributions are matched over auxiliary variable-induced subsets of the domains (e.g. translation between real human faces to cartoonized figures is conditioned on hair color and gender). \n\nQuantitative and qualitative evaluation on several geometrically-unaligned pairs of datasets (Rotated MNIST, Rotated Edges-2-Shoes and CelebA to Bitmoji) are presented to support the theoretical claims.", "questions": "1. I find the qualitative results quite limited. For example, In Fig. 8, the translation from human faces to bitmoji does not preserve the facial expression. Considering that the gender and hair color is provided to the model, and the facial expression is not preserved, what other properties should the reader focus on to verify the validity of the translation?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The studied problem of matching the distributions of unaligned image domains is of great interest in the image-to-image translation line of work. The development of theoretical frameworks as the one introduced in this paper can shed light on the limitations of such unsupervised approaches and lead to more robust translation methods.", "weaknesses": "1. The experimental study is not conducted in the most relevant setting in my opinion. As the proposed method relies on several auxiliary variables (e.g. hair color in human faces or the digit class in the MNIST experiment), I believe the baselines should represent methods in weakly-supervised image-to-image translation [1]. Comparison against unsupervised methods is unfair.\n\n2. The authors claim the auxiliary variables can be queried from available foundation models as CLIP. This idea is already explored in [1], could the authors please provide any experimental benchmark including CLIP-based annotations against [1]?\n\n3. There are some other works relating the failures in geometrically-unaligned image domains to architectural inductive biases [2]. Moreover, methods as [2] present translations between domains with some degree of geometry variation without access to additional labels in the form proposed in this paper. Could the authors provide a comparison to [2] on the celebA-to-bitmoji?\n\n[1] Gabbay et al. “An Image is Worth More Than a Thousand Words: Towards Disentanglement in the Wild”. In NeurIPS, 2021.\n\n[2] Gabbay et al. “Scaling-up Disentanglement for Image Translation”. In ICCV, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)