Content-Style Learning from Unaligned Domains: Identifiability under Unknown Latent Dimensions

Authors: Sagar Shrestha, Xiao Fu
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: unsupervised learning, identifiability, unknown latent dimension

Abstract:
Understanding identifiability of latent content and style variables from unaligned multi-domain data is essential for tasks such as
domain translation and data generation. Existing works on content-style identification were often developed under somewhat stringent conditions, e.g., that all latent components are mutually independent and that the dimensions of the content and style variables are known. We introduce a new analytical framework via cross-domain *latent distribution matching* (LDM), which establishes content-style identifiability under substantially more relaxed conditions. Specifically, we show that restrictive assumptions such as component-wise independence of the latent variables can be removed. Most notably, we prove that prior knowledge of the content and style dimensions is not necessary for ensuring identifiability, if sparsity constraints are properly imposed onto the learned latent representations. Bypassing the knowledge of the exact latent dimension has been a longstanding aspiration in unsupervised representation learning---our analysis is the first to underpin its theoretical and practical viability. On the implementation side, we recast the LDM formulation into a regularized multi-domain GAN loss with coupled latent variables. We show that the reformulation is equivalent to LDM under mild conditions---yet requiring considerably less computational resource. Experiments corroborate with our theoretical claims.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 8, 5

Reviews and discussion:
<thread>
creation date:
1729848986284

thread_type:
official_review

<thread_object>
creator: reviewer_ezhh

<content>
summary:
This paper introduces an analytical framework via cross-domain latent distribution matching(LDM), which establishes content-style identifiability under substantially more relaxed conditions.

strengths:
The paper is well written and the contribution is clear.

weaknesses:
The author seems to slip the main result part (3.2) too quickly (with no more than half a page). I am unsure what the key difference between your approach and other low-rank techniques is. In addition, there seems to exist a gap between the optimization problem (4a)(4b)(4c) and the implementation. (I mean: why not to use the formulation in implementation section as the theoretical modeling approach in the methodology part?) There are some writing issues in the paper too, e.g. you can not directly quote (4) , you should use 'according to formulation (4)'.(e.g. the writing in Theorem 4.1)

questions:
1. Can you further explain the reasonability using Assumption 3.1? Can you relax it? If not, what are the difficulties of relaxing this assumption? Furthermore, in what real-world scenarios might this assumption be violated?
2. Can you explain the gap between your method in 4(a) and the way you do implementation? What is the motivation for not using (4) directly? Any trade-offs or limitations of their implementation approach compared to directly using formulation (4).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730502836760

thread_type:
official_review

<thread_object>
creator: reviewer_dub2

<content>
summary:
Summary:
This paper studies causal identification of latent factors content and style in the cross-domain learning problems. Specifically, existing works assume that all latent components are independent to each other. Such an assumption is strict and could not be fulfilled in realistic scenarios. In this paper, the authors propose a cross-domain latent distribution matching framework which does not require strict independence between different latents. Moreover, the proposed framework can ensure content style identification can be free from the specific latent dimensions which further ease the assumption for tackling the problem. Through both theoretical and empirical justifications, the authors carefully testified the effectiveness of the proposed method.

strengths:
Strengths:
- This paper is well-written, well-motivated, and easy-to-follow. Both the experiments and theoretical justification are intuitive and convincing.
- The contribution of easing independent assumption is helpful for developing simple and efficient causal models.
- A sparsity-regularized implementation is proposed to solve cross-modal learning efficiently.

weaknesses:
Weaknesses:
- The proposed target in Equation 4 aims to minimize the norm of style latents by enforcing distribution matching across content and block-independence between content and style. However, it could lead to a degenerate solution when there is misidentification between content and style, and some content features already achieves small norm. Could you explain how can the proposed identifiability result can avoid the problem from happening?
- The computational efficiency of the framework is unknown. Only numerical result of generation is provided, there is no quantitative comparison to show the efficiency compared to other baseline methods. Moreover, why sparsity regularization is needed for training GAN needs a justification.
- Missing related references:  
Hong et al., Improving Non-Transferable Representation Learning by Harnessing Content and Style, in ICLR 2024.  
Huang et al., Harnessing Out-Of-Distribution Examples via Augmenting Content and Style, in ICLR 2023.  
Dai et al., Disentangling writer and character styles for handwriting generation, in CVPR 2023.

questions:
Please see the weaknesses part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730665380589

thread_type:
official_review

<thread_object>
creator: reviewer_tekb

<content>
summary:
This paper tackles the problem of identifiability of latent content and style variables from unaligned multi-domain data. A new approach based on cross-domain latent distribution matching (LDM) is presented, which removes restrictive assumptions from prior work regarding component-wise independence of latent variables and knowledge of content and style dimensions. The LDM formulation is implemented as a regularized multi-domain GAN loss with coupled latent variables. In addition to theoretical contributions on these topics, the authors present experimental results on image translation and generation tasks.

strengths:
- The paper provides a good overview of background and related work on multi-domain content-style modeling, content-style identifiability, and the challenge of knowing the content and style dimensions apriori. 
- The contributions in this paper appear to be novel and technically sound from both a theoretical and experimental standpoint.
- The proposed approach in this paper appears to be reproducible, given the provided source code and experimental details in the paper.
- The experimental results show that the proposed approach performs well on multi-domain image translation and generation tasks.

weaknesses:
- Because the proposed approach is implemented as a GAN, it seems likely that this approach will be most suitable for images (and possibly audio), and may be problematic for discrete data modalities, particularly text. 
- The authors don’t directly mention or address potential instabilities that may result from adversarial training of their GAN-based approach.

questions:
Can the authors please add some remarks regarding the potential weaknesses mentioned above, regarding the suitability of their approach for text, and regarding the potential instability of adversarial GAN training used in their approach?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721152117

thread_type:
official_review

<thread_object>
creator: reviewer_r56a

<content>
summary:
The authors try to attach the problem of identifying latent content and style variables from observations from un-aligned domains. The authors claim their main contribution as follows: (1) t restrictive assumptions such as component-wise independence of the latent variables can be removed; (2)  knowledge of the content and style dimensions is not necessary for ensuring identifiability. However, there are some inconsistencies between the main results and Theorem 4.1.

strengths:
1. The writing of the paper is good. Most parts of the paper are easy to follow and understand. 
2. The experimental results are good.

weaknesses:
There may be some inconsistency between  the main results and Theorem 4.1. In the main results, it claims that $f$, $\gamma$, and $\delta$ are injective. However, in theorem, the $q$ is a bijective and $f=q^-1$, which implies that $f$ will be an bijective. As a result, $\gamma$ and $\delta$ becomes bijective and it implies the dimension of content and style is actually $\hat{d}_c$ and $\hat{d}_s$. This causes a contradiction in the theory.

The proof of Theorem 4.1 uses  Proposition E.1 from  (Zimmermann et al., 2021) https://arxiv.org/pdf/2102.08850. However, I did not find the Proposition in the reference. Can the authors fix the problem?

questions:
1. In Theorem 4.1,  $q$ is a bijective and $f=q^{-1}$, thus trivially $f$ becomes an bijective, which somehow is inconsistent with the fact    that $f$ is a injective from $\mathcal{X}$ to $\mathbb{R}^{\hat{d}_{c}+\hat{d}_s}$ , 
   can the authors clarification? 

   Particularly on the relation between $\mathcal{C}$, $\mathcal{S}$ and $\mathbb{R}^{\hat{d}_{c}+\hat{d}_s}$ as they are actually different spaces. 
2. In Assumption 3.1, $N$ is the dimension of style variables, but somehow in later part of the paper it has different meaning, can the authors clarify? Similarly, the notation $n$ also has different meanings in different part of the paper.
3. Can you provide the correct citation for Proposition E.1?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731392472460

thread_type:
official_review

<thread_object>
creator: reviewer_wgdt

<content>
summary:
This paper enhances the identifiability of latent content and style representations in unaligned multi-domain data by framing the problem within a distribution matching framework. The authors establish that, under relaxed conditions, their approach does not require restrictive assumptions like component-wise independence or known latent dimensions. By applying sparsity constraints, they ensure identifiable content-style separation, and they ultimately reformulate the distribution matching objective into a simpler GAN training framework. The authors empirically demonstrate improvement in image translation.

strengths:
- The paper is well-structured and clearly written.
- It introduces a unique and interesting distribution matching perspective for content and style transfer in image translation
- By relaxing restrictive assumptions often required in previous work—such as the need for component-wise independence and pre-defined latent dimensions—the approach enhances practicality and scalability.
- The paper provides both a theoretical foundation for content-style identifiability and promising qualitative results.

weaknesses:
- The paper’s dimension-agnostic approach relies heavily on sparsity constraints to prevent information leakage between content and style representations, which may not be effective in all cases.
- The paper relies on several broad assumptions, such as the invertibility of the function up to certain ambiguities, without fully detailing the specific conditions under which this assumption holds.
- The L1 regularization appears to me to be too "soft" of a sparsity constraint to adequately justify the injective property of the mapping function $f$, particularly given that the model relies heavily on this constraint to ensure generative model is a valid invertible function.
- The paper, in general, lacks a thorough ablation study to validate the author's claims.

$\textbf{Minor Remarks}$
- Line 741, I think $\mathcal{A}\in\mathcal{Y}$ rather than $\mathcal{A}\in\mathcal{X}$.
- This is a minor suggestion, but it may be worth considering alternative abbreviations for "DM" and "LDM" due to the common use of these terms in diffusion models and latent diffusion models.

questions:
- Do you believe that L1​ regularization provides a strong enough sparsity constraint to justify the Domain Variability assumption? Could you elaborate on why you chose L1 ​over potentially stricter sparsity-inducing methods, particularly given its central role in ensuring the injective property of the mapping function.
- The model relies on some broad assumptions. Could you discuss the practical conditions that need to be met to ensure invertibility, and how sensitive the method might be to deviations from these conditions?
- Have you observed scenarios where the L1 constraint may not be strong enough to prevent information leakage between content and style representations?
- It appears from the results that the content representation often corresponds to pose, while the style representation captures semantic attributes. Is this always the case, or have you observed instances where content and style encode different aspects of the data? For example, in Figure 3(b), it seems that changing the domain also changes hair color.
- It would be helpful to include experiments on simpler datasets, such as rotated MNIST, to provide a clearer demonstration of the model’s ability to separate content and style. For example, with rotated MNIST, the content could represent class information, while rotation and other stylistic aspects could be designated as style. Such examples could offer readers a more intuitive understanding of how content and style representations are disentangled and controlled within the model.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732208960845

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We would like to thank the reviewers for their time and constructive comments. We believe that the comments and questions have helped to greatly improve the manuscript. Based on the reviewers’ comments, we have revised our manuscript. The new changes are **marked in blue**. 

A summary of our responses and revisions are as follows:

1. **Reviewer wgDt**: The comments focused on $\ell_1$ regularization for sparsity, the invertibility assumption, and additional experiments with MNIST digits where content does not indicate pose. In response, we conducted new experiments with $\ell_p$ regularization (p = 0.5) and MNIST digits where content represents digit identity. We also clarified that the independence assumption is widely used in generative modeling and latent component identification literature.


2. **Reviewer r56a**: This review raised points on the consistency of bijectivity and injectivity in the mapping function, notation consistency, and the accuracy of a citation. In response, we clarified our use of terminology and notation, provided the exact citation location, and revised our manuscript to address these issues.


3. **Reviewer teKB**: The comments focused on potential extensions to discrete data and GAN training instabilities. We discussed the requirements for extending to the discrete case and explained the scenarios where training instabilities occurred in our experiments. These points were added to the manuscript.


4. **Reviewer dub2**: The comments pertained to possible misidentification when content has a small norm, and the computational efficiency of our method. We clarified why misidentification would not occur in our framework and included a table of training times to demonstrate the method’s efficiency.


5. **Reviewer EZhh**: The feedback requested clarification on presentation, the block independence assumption, and the motivation behind the reformulation in implementation. We revised the subsection title for clarity, clarified block independence, and elaborated on the motivation for the reformulation. The manuscript was updated accordingly.

Kindly check our responses and revisions below. Please feel free to ask if you have any further questions or need additional clarification.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary:
This paper studies contrastive learning through the lens of understanding how it inverts the data generating process. The authors prove that feed-forward models trained with InfoNCE-type objectives learn to implicitly invert the underlying generative model of the observed data, under certain statistical assumptions. The paper establishes theoretical connections between contrastive learning, generative modeling, and nonlinear independent component analysis (ICA). The authors validate their theoretical findings through controlled experiments and show that the conclusions hold even when some theoretical assumptions are violated.

Main strengths:

- Novel theoretical insights connecting contrastive learning to generative modeling and nonlinear ICA
- Clear mathematical exposition and proofs of the key theoretical results
- Strong empirical validation across multiple experimental settings, including complex visual data
- Introduction of a new benchmark dataset (3DIdent) for evaluating representation learning
- Good balance between theoretical contributions and practical implications

Main weaknesses:

- Some notational inconsistencies and ambiguities in the mathematical formulation
- Could benefit from more intuitive explanations alongside the formal proofs
- Some citations and references need more precise specification