Title: Identifiable Shared Component Analysis of Unpaired Multimodal Mixtures
Venue: NeurIPS 2024
Accepted: yes
Keywords: Identifiability

Multi-modal Learning

Canonical Correlation Analysis

Shared Component Analysis

Independent Component Analysis
Abstract: A core task in multi-modal learning is to integrate information from multiple feature spaces (e.g., text and audio), offering modality-invariant essential representations of data. Recent research showed that, classical tools such as canonical correlation analysis (CCA) provably identify the shared components up to minor ambiguities, when samples in each modality are generated from a linear mixture of shared and private components.  Such identifiability results were obtained under the condition that the cross-modality samples are aligned/paired according to their shared information. This work takes a step further, investigating shared component identifiability from multi-modal linear mixtures where cross-modality samples are unaligned.  A distribution divergence minimization-based loss is proposed, under which a suite of sufficient conditions ensuring identifiability of the shared components are derived. Our conditions are based on cross-modality distribution discrepancy characterization and density-preserving transform removal, which are much milder than existing studies relying on independent component analysis. More relaxed conditions are also provided via adding reasonable structural constraints, motivated by available side information in various applications. The identifiability claims are thoroughly validated using synthetic and real-world data.

Identifiable Shared Component Analysis of Unpaired Multimodal Mixtures

Abstract

A core task in multi-modal learning is to integrate information from multiple feature spaces (e.g., text and audio), offering modality-invariant essential representations of data. Recent research showed that, classical tools such as canonical correlation analysis (CCA) provably identify the shared components up to minor ambiguities, when samples in each modality are generated from a linear mixture of shared and private components. Such identifiability results were obtained under the condition that the cross-modality samples are aligned/paired according to their shared information. This work takes a step further, investigating shared component identifiability from multi-modal linear mixtures where cross-modality samples are unaligned. A distribution divergence minimization-based loss is proposed, under which a suite of sufficient conditions ensuring identifiability of the shared components are derived. Our conditions are based on cross-modality distribution discrepancy characterization and density-preserving transform removal, which are much milder than existing studies relying on independent component analysis. More relaxed conditions are also provided via adding reasonable structural constraints, motivated by available side information in various applications. The identifiability claims are thoroughly validated using synthetic and real-world data.

1 Introduction

The same data entities can often be represented in different feature spaces (e.g., audio, text and image), due to the variety of sensing modalities or domains. Learning common latent components of data from multiple modalities is well-motivated in representation learning. The shared components are considered modality-invariant essential representations of data, which can often enhance performance of downstream tasks by shedding modality-specific noise [1-4] and avoiding over-fitting [5-7].

A prominent theoretical aspect of shared component learning lies in identifiability of the components of interest. The literature posed an intriguing theoretical question [1, 2, 8]: If every modality of data is represented by a linear mixture of shared and private components with an unknown mixing system, are the shared components identifiable (up to acceptable ambiguities)? Such component identification problems are often nontrivial due to the ill-posed nature of any linear mixture model (see, e.g., [9-14]). Interestingly, the work [1] showed that using the classical canonical correlation

analysis (CCA) provably find the shared components up to rotation and scaling. In fact, shared component identification from multimodal/multiview linear mixtures were considered in various contexts (see, e.g., [15-18]), although some of these works did not model private components. The identifiability results in [1, 2] were generalized to nonlinear mixture models as well [4, 19]. The shared component identification perspective was also related to the success of representation learning in self-supervised learning (SSL) [5-7].

Nonetheless, the treatment in [1, 2] and the related works [15-17] all assumed that the cross-modality data are aligned (i.e., paired) according to their shared components. In many applications, such as cross-language information retrieval [20-22], domain adaptation [23-25], and biological data translation [26, 27], aligned cross-modality data are hard to acquire, if not outright unavailable. A natural question is: When the multimodal linear mixtures are unaligned, can the shared latent components still be provably identified under reasonably mild conditions?

Existing Studies. Theoretical characteristics of unaligned multimodal learning were studied under various settings. The work [28] considered a case where one modality is a linear transform of another modality, and showed that the linear transformation is potentially identifiable. The recent work [29] extended this model to a nonlinear transform setting. However, these works did not consider latent component models-yet the latter are more versatile in many ways, e.g., facilitating one-to-many cross-domain translations [30, 31]. The work [32] considered unaligned mixtures of shared and private components, but the assumptions (e.g., the availability of a large amount of modalities) to ensure identifiability may not be easy to satisfy. The most related work is perhaps [8]. But their approach also relied on somewhat stringent assumptions, e.g., that all the latent components are element-wise statistically independent with at most one component being Gaussian. This is because their procedure had to invoke the classical independent component analysis (ICA) [33].

Contributions. In this work, we provide a suite of sufficient conditions under which the shared components can be provably identified from unaligned multimodal linear mixtures up to reasonable ambiguities. The model and identification problem are referred to as unaligned shared component analysis (unaligned SCA) in the sequel.

(i) An Identifiable Learning Loss for Unaligned SCA. We propose to tackle the unaligned SCA problem by matching the probability distributions of linearly embedded multi-modal data. We show that under reasonable conditions, the linear transformations identifies the shared components up to the same ambiguities as those in the aligned case [1, 2]. The conditions are considerably milder compared to the existing unaligned SCA work [8].

(ii) Enhanced Identifiability via Structural Constraints. We come up with two types of structural constraints, motivated by available side information in applications, to further relax the identifiability conditions. Specifically, we look into cases where the multi-modal data have similar linear mixing systems and cases where a few cross-domain aligned samples available. We show that by adding constraints accordingly, unaligned SCA are identifiable under much milder conditions.

Our contributions primarily lie in identifiability analysis. Nonetheless, we also show the usefulness of our results in real-world applications, namely, cross-lingual word retrieval , genetic information alignment and image data domain adaptation . Particularly, it shows that our succinct multimodal linear mixture model can effectively post-process outputs of pre-trained encoders, e.g., those in [34, 35], to improve data representations and enhance downstream task performance.

Notation. Notation definitions can be found in Appendix A.

2 Background

Generative Model of Interest. Following the classical settings in [1, 2, 15, 16, 18], we consider modeling the multi-modal data as linear mixtures. More specifically, we adopt the model in [1, 2]that splits the latent representation of data into shared components and private components:

where x ( q ) ∈ R d ( q ) represents the data from the q th modality, z ( q ) ∈ R d C + d ( q ) P represents the corresponding latent code, c ∈ R d C and p ( q ) ∈ R d ( q ) P stand for the shared components and the private components, respectively. The data x ( q ) 's are assumed to be zero-mean, which can be enforced by

centering. Note that the positions of c and p q are not necessarily arranged as [ c ⊤ , ( p ( q ) ) ⊤ ] ⊤ (more generally, z ( q ) = Π ( q ) [ c ⊤ , ( p ( q ) ) ⊤ ] ⊤ with an unknown permutation matrix Π ( q ) ). However, the representation in (1) is without loss of generality as one can define A ( q ) := A ( q ) ( Π ( q ) ) ⊤ to reach the representation in (1). For all the domains, we have

where P c and P p ( q ) represent the distributions of the shared components and the domain-private components, respectively. Under (1), the two different range spaces range( A ( q ) ) for q = 1 , 2 represent two feature spaces. Then latent p ( q ) further distinguishes the modalities and often has interesting physical interpretation. For example, some vision literature use c to model 'content' and p ( q ) 'style' of the images [31, 36]. In cross-lingual word embedding retrieval [2], c represents the semantic meaning of the words, while p ( q ) represents the language-specific components. The goal of SCA boils down to finding linear operators to recover c to a reasonable extent.

Aligned SCA: Identifiability of CCA and Extensions. Learning c without knowing A ( q ) is a typical component analysis problem. Learning latent components from linear mixture models (LMMs) like x = Az lacks identifiability in general, due to the bilinear nature of the models. This is because one can find an infinite number of invertible matrices B such that x = ABB -1 z . Then, both ( A , z ) and ( AB , B -1 z ) can fit to the data x , making the problem ill-posed in terms of solution uniqueness; see, e.g., [9, 37] and more discussions in Sec. 5. Nonetheless, the works [1, 2] studied the identifiability of c under the model (1), using the assumption that the cross-modality samples share the same c are aligned. In particular, [1] formulated the c -identification problem as a CCA problem:

where Q ( q ) ∈ R d C × d ( q ) . The expectation in (3a) is taken from the joint distribution of the aligned pairs P x (1) , x (2) , where every pair ( x (1) , x (2) ) shares the same c . The formulation aims to find Q ( q ) such that the transformed representations of the aligned pairs Q (1) x (1) and Q (2) x (2) are equal. In [1], it was shown that

̂ Q ( q ) x ( q ) = Θ c (4) under mild conditions (see Appendix E.1 for details), where ( ̂ Q (1) , ̂ Q (2) ) is an optimal solution of the CCA formulation and Θ is a certain non-singular matrix. Eq. (4) means that ̂ Q ( q ) finds the range space where c lives in, i.e., range( A ( q ) 1: d C ) under our notation.

̂ c ( q ) m = kc ( q ) π ( m ) , m ∈ [ d C ] , (5) where k ∈ { +1 , -1 } and π is a permutation of { 1 , . . . , d C } (see details in Appendix E.2 summarized from [8]). This method effectively applies ICA to each modality, and thus the ICA identifiability conditions [33] have to met by x (1) and x (2) individually. However, if one only aims to extract Θ c as in CCA, these assumptions appear to be overly stringent.

Unaligned SCA: Existing Result and Theoretical Gap. The work in [8] studied the identifiability of c under (1) when x (1) and x (2) are unaligned . Their approach works under the condition that the elements of z ( q ) = [ c ⊤ , ( p ( q ) ) ⊤ ] ⊤ are mutually statistically independent . There, ̂ z ( q ) = Π ( q ) Σ ( q ) z ( q ) is assumed to have been estimated by ICA, where Π ( q ) and Σ ( q ) represent the scaling and permutation ambiguities, respectively, which cannot be removed by ICA. The work [8] assumed Σ ( q ) = I by imposing a unit-variance assumption on all the z ( q ) i 's. Then, a cross-domain matching algorithm is used to match the shared elements in ̂ z (1) and ̂ z (2) . The formulation can be summarized as finding d C pairs of non-repetitive ( i, j ) such that e ⊤ i ̂ z (1) and e ⊤ j ̂ z (2) have identical distributions, where e i is the i th unit vector. Denote ̂ c (1) m = e ⊤ i m ̂ z (1) and ̂ c (2) m = e ⊤ j m ̂ z (2) for m ∈ [ d C ] . It can be shown that

3

2

1

0

1

2

3

4

-

4

-

3

-

2

-

1

0

1

2

3

4

Θ

(1)

c

3

2

1

0

1

2

3

4

-

4

-

3

-

2

-

1

0

1

2

3

4

Θ

(2)

c

-

-

-

-

-

-

3 Proposed Approach

Unaligned SCA: Problem Formulation We assume that x ( q ) 's are zero-mean. We use the notation from CCA in (3a). However, since no aligned samples are available, we replace the sample-level matching objective with a distribution matching (DM) module, as DM can be carried out without sample level alignment:

where ' u ( d ) == = v ' means the distributions of u and v are the same.

The formulation in (6) can be realized using various distribution matching tools, e.g., maximum mean discrepancy (MMD) [38] and Wasserstein distance [39]. We use the adversarial loss:

The first and second terms comprise the adversarial loss from GAN [40]. It finds Q ( q ) to confuse the best-possible discriminator f : R d C → R , where f is represented by a neural network in practice. It is well known that the minimax optimal point of the first two terms is attained when (6b) is met [40]. We use R ( Q ( q ) ) = ∥ Q ( q ) E [ x ( q ) ( x ( q ) ) ⊤ ]( Q ( q ) ) ⊤ -I ∥ 2 F to 'lift' the constraints. This way, the learning criterion in (7) can be readily handled by any off-the-shelf adverserial learning tools.

Identifiability of Unaligned SCA As we saw in Theorem 4, CCA identifies ̂ Q ( q ) x ( q ) = Θ c where Θ ∈ R d C × d C under the settings of aligned SCA. Establishing a similar result for unaligned SCA is much more challenging. First, it is unclear if (6b) could disentangle c from p ( q ) . In general, Q ( q ) x ( q ) could still be a mixture of c and p ( q ) yet (6b) still holds (e.g., when both c and p ( q ) are Gaussian.)

̸

Second, even when the disentanglement is attained via enforcing (6b) and we have Q ( q ) x ( q ) = Θ ( q ) c , in general it does not hold that Θ (1) = Θ (2) . This is because Θ (1) c ( d ) == = Θ (2) c where Θ (1) = Θ (2) can still be perfectly met (e.g., when P Θ ( q ) c is symmetric Gaussian in Fig. 1 ). However, Θ (1) = Θ (2) means that the extracted representations from the two modalities are not matched. This creates challenges for applications like cross-domain information retrieval, language translation, or domain adaptation.

̸

Our intuition is as follows: If the two distributions P c , p (1) and P c , p (2) are very different, then Q (1) x (1) ( d ) == = Q (2) x (2) cannot hold unless Q ( q ) A ( q ) = [ Θ ( q ) , 0 ] . We use the following to characterize such difference between the joint distributions:

̸

̸

Assumption 1 (Modality Variability) . For any two linear subspaces P ( q ) ⊂ R d C + d ( q ) P , q = 1 , 2 , with dim( P ( q ) ) = d ( q ) P , P ( q ) = 0 × R d ( q ) P and linearly independent vectors { y ( q ) i ∈ R d C + d ( q ) P } d C i =1 , q = 1 , 2 , the sets A ( q ) = conv { 0 , y ( q ) 1 , . . . , y ( q ) d C } + P ( q ) , q = 1 , 2 , are such that if P c , p ( q ) [ A ( q ) ] > 0 for q = 1 or q = 2 , then there exists a k ∈ R such that the joint distributions P c , p (1) [ k A (1) ] = P c , p (2) [ k A (2) ] , where k A ( q ) = { k a | a ∈ A ( q ) } .

The condition in Assumption 1 is a geometric way to characterize the difference between P c , p (1) and P c , p (2) -if the joint distributions have different measures for all possible 'stripes', each being a direct sum of a subspace and a convex hull (see Fig. 2), then P c , p (1) and P c , p (2) must be very different. Note that the difference is contributed by the modality-specific term p ( q ) , and thus we call this condition 'modality variability'. Modality variability is similar to the 'domain variablity' used in [32, 41]-both characterize the discrepancy of the joint probabilities P c , p (1) and P c , p (2) . However, there are key differences: The domain variability was defined in a unified latent domain over arbitrary sets A , which could be stringent. Instead, we use the fact that (6) relies on linear operations to construct A ( q ) , which makes the condition defined over a much smaller class of sets-thereby largely relaxing the requirements. Restricting A ( q ) to be stripes also makes the modality variability condition much more relaxed compared to the domain variability condition.

We show the following:

Theorem 1. Under Assumption 1 and the generative model in (1) , denote any solution of (6) as ̂ Q ( q ) q = 1 , 2 . Then, if the mixing matrices A ( q ) are full column ranks and E [ cc ⊤ ]) is full rank, we have ̂ Q ( q ) x ( q ) = Θ ( q ) c . In addition, assume that either of the following is satisfied:

̸

(a) The individual elements of the content components are statistically independent and nonGaussian. In addition, c i ( d ) = kc j , ∀ i = j, ∀ k ∈ R and c i ( d ) = -c i , ∀ i , i.e., the marginal distributions of the content elements cannot be matched with each other by mere scaling.

̸

̸

(b) The support of P c , denoted by C , is a hyper-rectangle, i.e., C = [ -a 1 , a 1 ] ×···× [ -a d C , a d C ] . Further, suppose that c i ( d ) = kc j , ∀ i = j, ∀ k ∈ R and c i ( d ) = -c i , ∀ i .

̸

̸

̸

Then, we have ̂ Q ( q ) x ( q ) = Θ c , i.e., Θ ( q ) = Θ for all q = 1 , 2 , where Θ ( q ) .

In Theorem 1, Assumption 1 is used to guarantee ̂ Q ( q ) x ( q ) = Θ ( q ) c and either of conditions (a) or (b) is used to make sure Θ (1) = Θ (2) . Note that both (a) and (b) are milder than those in [8] (cf. Theorem 5), where the element-wise statistical independence of z ( q ) was relied on to find shared representation of x (1) and x (2) . The proof is in Appendix B.

Numerical Validation. In Fig. 3, the top and bottom rows validate Theorem 1 under the assumptions in (a) and (b), respectively. In the top row, we set c ∈ R 2 , where c 1 is sampled from Gaussian mixtures with three components and c 2 is sampled from a Gamma distribution (and c 1 ⊥ ⊥ c 2 ). We set p (1) and p (2) as one-dimensional Laplacian and uniform distributions. In the bottom row, the dimensions of c and p ( q ) for q = 1 , 2 are unchanged, but their distributions are replaced in order to satisfy conditions in (b) (see details in Appendix F). One can see that clearly ̂ c ( q ) = Θ c ; i.e., the learned ̂ c ( q ) for q = 1 , 2 are identically rotated and scaled versions of c . A remark is that our framework still allows to identify individual c i 's as in [8].

̂ The corollary means that to identify individual c i , using our formulation still enjoys much milder conditions relative to [8]. Specifically, our condition only specifies the independence among elements of c , but the condition in [8] needs that all the elements in z ( q ) = [ c ⊤ , ( p ( q ) ) ⊤ ] ⊤ are independent.

Corollary 1. Under the conditions in Theorem 1 (a), Assume that at most one c i for i ∈ [ d C ] is Gaussian. Then, the components of c are identifiable up to permutation and scaling ambiguities by applying ICA to c ( q ) = ̂ Q ( q ) x ( q ) for either q = 1 or q = 2 .

-

2

-

1

0

1

2

3

4

5

0

.

0

2

.

5

5

.

0

7

.

5

10

.

0

12

.

5

15

.

0

17

.

5

20

.

0

(a)

c

-

75

-

50

-

25

0

25

50

75

100

-

60

-

40

-

20

0

20

40

60

x

(1)

-

75

-

50

-

25

0

25

50

75

-

75

-

50

-

25

0

25

50

75

x

(2)

-

6

-

4

-

2

0

-

2

0

2

4

6

̂

c

(1)

-

8

-

6

-

4

-

2

0

-

2

0

2

4

6

̂

c

(2)

3

2

1

0

1

2

3

-

3

-

2

-

1

0

1

2

3

(b)

c

100

75

50

25

0

25

50

75

100

-

60

-

40

-

20

0

20

40

60

80

100

x

(1)

60

40

20

0

20

40

60

80 100

-

100

-

75

-

50

-

25

0

25

50

75

100

x

(2)

5

4

3

2

1

0

1

-

2

-

1

0

1

2

3

4

̂

c

(1)

5

4

3

2

1

0

1

-

2

-

1

0

1

2

3

4

̂

c

(2)

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

4 Enhanced Identifiability via Structural Constraints

Theorem 1 was well-supported by the synthetic data experiments. However, our experiments found that the learning criterion (6) often struggles to produce sensible results in some applications. Our conjecture is that the Assumptions in Theorem 1 (a) and (b) might not have been satisfied by the real data under our tests. Although they are not necessary conditions for identifiability, these conditions do indicate that the requirements to guarantee identifiability of unaligned SCA using (6) are nontrivial to meet. In this section, we explore a couple of structural constraints arising from side information in applications to remove the need for the relatively stringent assumptions on c .

Homogeneous Domains. The first structural constraint that we consider is A ( q ) = A for q = 1 , 2 . This model is motivated by the fact that advanced representation learning tools, e.g., selfsupervised learning tools (e.g., SimCLR [42]) and foundation models (e.g., CLIP [35]), are already capable of mapping the data clusters to a shared linearly separable space-which indicates that the representations share a subspace, i.e., x ( q ) ≈ Az ( q ) . Under such circumstances, the proposed model and method can be used to further process the data by discarding the private components in the latent representation.

Here, we consider the special case of generative process in (1) where,

Under this model, we look for the shared components by solving (6) with a single Q = Q (1) = Q (2) . We use the following version of the modality variability condition:

Assumption 2. For any linear subspace P ⊂ R d C + d P , d P = d (1) P = d (2) P , with dim( P ) = d P , P ̸ = 0 × R d P and linearly independent vectors { y i ∈ R d C + d P } d C i =1 , q = 1 , 2 , the sets A = conv { 0 , y 1 , . . . , y d C } + P , q = 1 , 2 . are such that if P c , p ( q ) [ A ] > 0 for q = 1 or q = 2 , then the joint distributions P c , p (1) [ k A ] = P c , p (2) [ k A ] for some k ∈ R .

Theorem 2. Consider the mixture model in (8) . Assume that rank( A ) = d C + d P and rank( E [ cc ⊤ ]) = d C , and that Assumption 2 holds. Denote ̂ Q as any solution of (6) by constraining Q = Q (1) = Q (2) . Then, we have Qx ( q ) = Θ c .

̸

̸

̂ One can see that the conditions (a) and (b) in Theorem 1 are completely removed, if the structure A (1) = A (2) is imposed. In fact, the result in Theorem 2 is expected and readily seen from the proof of Theorem 1, as the cause for Θ (1) = Θ (2) is the use of two different Q ( q ) 's. Nonetheless, this simple variation will prove useful in a series of real-data experiments.

The Weakly Supervised Case. Another way to add structural constraints is to use available auxiliary information. For example, some datasets have weak annotations and selected pairs; see, e.g., [43, 44].

Assumption 3 (Weak Supervision) . There exist a set of available aligned samples ( x (1) ℓ , x (2) ℓ ) for ℓ ∈ L such that x ( q ) ℓ = A ( q ) z ( q ) ℓ , z ( q ) ℓ = [ c ⊤ ℓ , ( p ( q ) ℓ ) ⊤ ] ⊤ ; i.e., ( x (1) ℓ , x (2) ℓ ) share the same c ℓ .

The condition can be added into our formulation in (6) as a constraint, i.e.,

In the next theorem, we show that the incorporation of aligned samples helps relax conditions (a) and (b) in Theorem 1:

Theorem 3. Assume that Assumption 1 is satisfied, that |L| ≥ d C paired samples ( x (1) ℓ , x (2) ℓ ) are available, that A ( q ) for q = 1 , 2 have full column rank, and that P c is absolutely continuous. Denote ( ̂ Q (1) , ̂ Q (2) ) as any optimal solution of (6) under the constraint (9) . Then, we have ̂ Q ( q ) x ( q ) = Θ c .

The proof and synthetic data validation can be found in Appendices D and F, respectively. Note that to realize (9), one only needs to add a regularization term β ∑ ℓ ∈L ∥ Q (1) x (1) ℓ -Q (2) x (2) ℓ ∥ 2 2 to the loss in (7), where β ≥ 0 is a tunable parameter. The overall loss is still differentiable and thus can be easily handled by gradient based approaches.

A remark is that our weakly supervised formulation can use as few as d C pairs of ( x (1) ℓ , x (2) ℓ ) to establish identifiability of shared component. In contrast, CCA requires at least d C + d (1) P + d (2) P pairs to attain the same identifiability (cf. Appendix. E.1).

Private Component Identifiability. Although our focus is shared component identification, we show that private components are also identifiable with additional assumptions; see Appendix H.

5 Related Works

Identifiability of Component Analysis under Linear Mixture Models. Various component analysis models were studied in the past several decades, e.g., principal component analysis [45], independent component analysis [33], sparse component analysis [10, 12], bounded component analysis [13], simplex component analysis [46, 47], and polytopic component analysis [14]-motivated by their applications in dimensionality reduction, representation learning, and latent variable identification (see, e.g., topic mining [48, 49], hyperspectral unmixing [46, 47], audio/speech separation [33] and community detection [50]). The classical component analysis tools mostly study a single modality. The identifiability results under these models are well developed and documented.

Identifiability of Shared Components from Aligned Modalities. Modeling multimodal data as two or more linear/nonlinear mixtures of latent components was considered in CCA-related works [1, 2, 15, 19], independent vector analysis (IVA) works [17, 18], multiview ICA works [16, 51], and SSL works [5-7, 52]. Partitioning the latent components into shared and private blocks was considered in [1, 2, 4, 5, 7, 52]. Shared component identifiability was established at the block level (see, e.g., [1, 2, 5]) and the individual component level (e.g., [51]) in these works. Nonetheless, they all rely on completely paired/aligned cross-modality samples, which we do not use in this work.

Distribution Matching and Unaligned Multimodal Analysis. Using distribution matching in unaligned multimodal data analytics for different purpose also has a long history; see applications in imageto-image translation [53], domain adaptation [54], cross-platform image super-resolution [55], and cross-domain information retrieval [21]. The recent works [56] and [57] pointed out the identifiability challenge and the existence of density-preserving transforms. The works in [28, 29] started studying the uniqueness issues in distribution matching. However, the latent mixture models were not studied in this line of work.

Identifiability of Unaligned SCA. The works in [32, 41] investigated the shared component identifiability when the multimodal data are nonlinear mixtures of content and style (which are shared and private components, respectively) under the same mixing system. Hence, our identical linear mixing case in Theorem 2 can be understood as a special case of theirs. But their analysis relies on the assumption that all the latent components are statistically independent, which is much stronger than our conditions in Theorem 2. Their results also require that there are a large amount of modalities available. But our proof works for just two modalities. The most related work is [8], which uses the model in (1) in the context of multi-view causal graph learning. As discussed before, their assumptions on the latent components are much stronger than ours (see Corollary 1 and Appendix E.2).

±

6 Numerical Validation

More Synthetic-Data Validation. We first validate our proposed method on synthetic data that follows our model; see Appendix F for details.

Application (i) - Domain Adaptation. We first test the proposed methods over a number of domain adaptation (DA) tasks. In DA, we have the source domain data { x (1) } and the target domain { x (2) } , respectively. Only the source domain data have labels and the two domains are unaligned. We hope to use our method to find shared representations of source and target, and thus the classifier trained using source data can also work well on the target data.

Dataset : We use two standard benchmarks of DA, i.e., Office-31 [58] and Office-Home [59]. The Office-31 dataset has 4652 images and 31 categories from three domains, namely, Amazon images ( A ), Webcam images ( W ) and DSLR images ( D ). The Office-Home dataset contains 15,500 images with 65 object classes from four domains, i.e., Artistic images ( Ar ), Clip art images ( Cl ), Product images ( Pr ), and Real-world images ( Rw ).

Setup : We first test the homogeneous domain model in Sec. 4. The images are pre-processed using a ResNet50-based image encoder pre-trained over ImageNet1k [42]. As mentioned, it was observed that self-supervised representation encoders find embeddings that are linearly separable [42], which justifies the use of the model x ( q ) ≈ Az ( q ) in the embedding domain. After pre-processing, each image is represented by d ( q ) = 2048 features for q = 1 , 2 . We set d C = 256 for Office-31 and d C = 512 for Office-Home . More detailed settings are in Appendix G.

Baselines and Training Setup : The baselines are representative DA methods, namely, DANN [25], MDD [60], MCC [61], SDAT [62], and ELS [63]. All the baselines use the same encoder-produced embeddings as inputs; see Appendix G.1 for their configurations. We also use ResNet encoder's outputs as an extra baseline as it learns informative and transferable features from the ImageNet1K dataset. We follow the training strategies adopted by the baselines [25, 60, 62] to learn a classifier jointly with the shared latent components. This strategy arguably regularizes towards more classification-friendly geometry of the shared features. Therefore we append a cross-entropy (CE) based classifier training module to our loss in (7) that learns our feature extractor Q . More details are in Appendix G.1.

Metric : The evaluation metric is the classification accuracy in the target domain { x (2) } . The classifier is trained with the projected source domain Qx (1) and the associated labels.

̂ Result : Table 1 and Table 2 show the classification accuracy (mean ± std) on Office-31 and OfficeHome , respectively. The results are averaged over 5 runs. One can observe that the proposed method offers the best and second best performance in most of the cases. In some tasks (e.g.,' A → W ', ' Ar → Cl ', ' Ar → Pr ' and ' Rw → Cl '), the proposed method outperforms the best-performing baselines by at least 2% in accuracy.

More results on the DA task can be found in Appendix G.1.

Application (ii) - Single Cell Sequence Analysis. In biomedical research, it is desired to fuse measurements from multiple sensorial modalities of the same cells, in order to have better characterizations of the cells. However, obtaining multimodal data of the same cells simultaneously is almost impossible, due to the sensing limitations. Therefore, many methods are proposed in the literature for aligning unpaired multi-modal single cell data [27, 64, 65]. We focus on the following two modalities of single-cell data [66]: (1) the RNA sequences { x (1) } and (2) the ATAC sequences { x (2) } .

±

±

±

±

±

±

±

±

5

8

10

12

14

16

18

20

K of the K-NN evaluation metric

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

K-NN accuracy

Proposed: No paired

Proposed: 1 paired

Proposed: 50 paired

Proposed: 256 paired

CM-AE: 256 paired + class labels

CM-AE: 770 paired + class labels

Dataset : We use human lung adenocarcinoma A549 cells data from [66]. The dataset contains 1,874 samples of RNA sequences { x (1) } and ATAC sequences { x (2) } . Each data set is split into 1534 training samples and 340 testing samples as in [27]. The data have labeled associations between the two domains-part of which will be used to test our weakly supervised formulation. For this experiment, features of RNA sequence and the ATAC sequence have dimensions of d (1) = 815 and d (2) = 2613 , respectively. We set d C = 256 . We use our weakly supervised formulation as shown in (9). We uniformly sampled a set of indices from the training set to serve as L .

Baseline and Metric : We use weakly supervised algorithm, namely, cross-modal autoencoder (CMAE) work in [27], as a baseline, which also learns the shared representation between unaligned RNA and ATAC sequences. We use the K -nearest neighbor ( k -NN) accuracy to evaluate the performance as suggested in [27].

Result : The plot in Fig. 4 shows the k -NN accuracy of the methods on the test set. Results show the mean and standard deviation over 10 runs, each having a different random initialization. For the proposed method, we vary the number of available paired samples from 0 (cf. Theorem 1) to d C = 256 (cf. Theorem 3). Note that the baseline uses more (i.e., 256 and 770 ) paired samples. It also needs additional class labels, i.e., y ( q ) i for the i th sample x ( q ) i . Here, y ( q ) i represents the number of hours ( 0 , 1 or 3 ) of cell treatment [27, 66]. The proposed method without any supervision (i.e., 0 paired samples) already exhibits around 3 times greater k -NN accuracy compared to the baseline for all k . Moreover, including just one paired sample boosts the k -NN accuracy of the proposed method to around 5 times higher than the baseline for all k . Finally, one can observe a steadily increasing k -NN accuracy with respect to the number of available paired samples. This corroborates with our Theorem 3.

±

±

±

±

±

Application (iii) - Multi-lingual Information Retrieval. We also evaluated our method on a word embedding association problem from the natural language processing literature [20, 21]. This task aims to associate high-dimensional word embeddings across different languages according to their semantic meaning. The word embeddings in two languages are represented using two sets of vectors, i.e., { x (1) i } I i =1 and { x (2) j } J j =1 . The postulate is that if x (1) i and x (2) j have the same meaning (e.g., both representing 'cat') in two languages (e.g., English and German), they should share a latent components c .

Dataset : We use the word embeddings from the MUSE dataset ( https://github.com/ facebookresearch/MUSE ) [21]. These monolingual word embedding are generated using fastText [67] and has dimensions of d ( q ) = 300 for q = 1 , 2 . The training dataset include 200,000 word embeddings in each language. In our experiment we set d C = 256 . We follow the generative model under (8) and run the formulation in (7) to learn the linear transformation Q .

Baseline : We use Adv [21] as the baseline which also uses distribution matching between two language domains. Unlike our method, Adv does not use linear mixture models.

Metric : We follow [21] to use the average precision score calculated based on nearest neighbor (NN) and cross domain similarity local scaling (CSLS). Precision at k (' k precision') is computed by the number of times that one of the correct translations of source word is retrieved at topk results ( k ={1, 5, 10}). The final score is normalized to be in the range of 0 to 100, with 100 being the highest score indicating the best performance. To evaluate the performance, we use the same test data as in [21]. For each source and target language pair, this dataset includes 1,500 source word embeddings. The source embeddings are used to retrieve corresponding embeddings from a pool of 200,000 target word embeddings.

Result : Table 3 reports the P@1 scores over the test data calculated for each source and target language pair. The languages are denoted as as en - English, es - Spanish, it - Italian, fr - French, de - Germany, ru - Russian, ar - Arabic and vi - Vietnamese. One can observe that the proposed method exhibits a better precision performance than that of Adv in most of the translation tasks. In particular, the proposed method significantly outperforms the baseline on the tasks en → ar , ar → en , en → vi and vi → en , showing at least 10% precision gains. Similarly, our method shows at least 5% improvements in both NN and CSLS based precision metrics in en → es and es → en tasks.

More details and additional experiments can be found in Appendix G.3.

7 Conclusion

In this work, we considered the problem of identifying shared components from unaligned multidomain mixtures. We proposed a learning loss that matches the distributions of linearly transformed data. Based on this loss, we came up with a suite of sufficient conditions to ensure the identifiability of shared components. Furthermore, we proposed modified models and losses that enjoy more relaxed conditions for shared component identifiability. This was achieved via introducing structural constraints, namely, the homogeneity of the mixing systems and the existence of weak supervision. Our theoretical claims were validated with both synthetic and real-world data, demonstrating soundness of the theorems and usefulness of the models/algorithms.

Limitations. First, our conditions for shared component identification are sufficient. The necessary conditions are not underpinned, but necessary conditions assist understanding the limitations of the models and algorithms. Second, our methods were developed under the linear mixture model, which has limited expressiveness, and thus often requires pre-processing to approximately meet the model specification. We expect that results with similar flavors to be derived for nonlinear models in the future. Third, the results were derived under an unlimited data assumption. It would be interesting have a finite sample analysis. Finally, optimizing GAN-based losses is sensitive to hyperparameter settings. Back-propagation based minimax optimization occasionally fails to converge. More optimization-friendly losses and more stable algorithms are desirable in the context of distribution matching.

Acknowledgment

This work is supported in part by the Army Research Office (ARO) under Project ARO W911NF-211-0227, and in part by the National Science Foundation (NSF) CAREER Award ECCS-2144889.

References

M. S. Ibrahim, A. S. Zamzam, A. Konar, and N. D. Sidiropoulos, 'Cell-edge detection via selective cooperation and generalized canonical correlation,' IEEE Transactions on Wireless Communications , vol. 20, no. 11, pp. 7431-7444, 2021.

M. Sørensen, C. I. Kanatsoulis, and N. D. Sidiropoulos, 'Generalized canonical correlation analysis: A subspace intersection approach,' IEEE Transactions on Signal Processing , vol. 69, pp. 2452-2467, 2021.

P. Rastogi, B. Van Durme, and R. Arora, 'Multiview LSA: Representation learning via generalized CCA,' in Proc. Conference of the North American chapter of the Association for Computational Linguistics: human language technologies , 2015, pp. 556-566.

Q. Lyu and X. Fu, 'Nonlinear multiview analysis: Identifiability and neural network-assisted implementation,' IEEE Transactions on Signal Processing , vol. 68, pp. 2697-2712, 2020.

J. Von Kügelgen, Y. Sharma, L. Gresele, W. Brendel, B. Schölkopf, M. Besserve, and F. Locatello, 'Self-supervised learning with data augmentations provably isolates content from style,' in Advances in Neural Information Processing Systems (NeurIPS) , vol. 34, 2021, pp. 16 451-16 467.

Q. Lyu, X. Fu, W. Wang, and S. Lu, 'Understanding latent correlation-based multiview learning and self-supervision: An identifiability perspective,' in Proc. International Conference on Learning Representations (ICLR) , 2022.

I. Daunhawer, A. Bizeul, E. Palumbo, A. Marx, and J. E. Vogt, 'Identifiability results for multimodal contrastive learning,' in Proc. International Conference on Learning Representations (ICLR) , 2022.

N. Sturma, C. Squires, M. Drton, and C. Uhler, 'Unpaired multi-domain causal representation learning,' in Advances in Neural Information Processing Systems (NeurIPS) , vol. 34, 2021.

P. Comon and C. Jutten, Handbook of Blind Source Separation: Independent component analysis and applications . Academic press, 2010.

J. Hu and K. Huang, 'Global identifiability of ℓ 1 -based dictionary learning via matrix volume optimization,' in Advances in Neural Information Processing Systems (NeurIPS) , vol. 36, 2024.

N. Gillis, 'The why and how of nonnegative matrix factorization,' Regularization, optimization, kernels, and support vector machines , vol. 12, no. 257, pp. 257-291, 2014.

J. Sun, Q. Qu, and J. Wright, 'Complete dictionary recovery over the sphere i: Overview and the geometric picture,' IEEE Transactions on Information Theory , vol. 63, no. 2, pp. 853-884, 2016.

A. T. Erdogan, 'A class of bounded component analysis algorithms for the separation of both independent and dependent sources,' IEEE Transactions on Signal Processing , vol. 61, no. 22, pp. 5730-5743, 2013.

G. Tatli and A. T. Erdogan, 'Polytopic matrix factorization: Determinant maximization based criterion and identifiability,' IEEE Transactions on Signal Processing , vol. 69, pp. 5431-5447, 2021.

F. R. Bach and M. I. Jordan, 'A probabilistic interpretation of canonical correlation analysis,' 2005.

H. Richard, P. Ablin, B. Thirion, A. Gramfort, and A. Hyvarinen, 'Shared independent component analysis for multi-subject neuroimaging,' in Advances in Neural Information Processing Systems (NeurIPS) , vol. 34, 2021, pp. 29 962-29 971.

Y.-O. Li, T. Adali, W. Wang, and V. D. Calhoun, 'Joint blind source separation by multiset canonical correlation analysis,' IEEE Transactions on Signal Processing , vol. 57, no. 10, pp. 3918-3929, 2009.

M. Anderson, G.-S. Fu, R. Phlypo, and T. Adalı, 'Independent vector analysis: Identification conditions and performance bounds,' IEEE Transactions on Signal Processing , vol. 62, no. 17, pp. 4399-4410, 2014.

P. A. Karakasis and N. D. Sidiropoulos, 'Revisiting deep generalized canonical correlation analysis,' IEEE Transactions on Signal Processing , vol. 71, pp. 4392-4406, 2023.

G. Lample, A. Conneau, L. Denoyer, and M. Ranzato, 'Unsupervised machine translation using monolingual corpora only,' in Proc. International Conference on Learning Representations (ICLR) , 2018.

G. Lample, A. Conneau, M. Ranzato, L. Denoyer, and H. Jégou, 'Word translation without parallel data,' in Proc. International Conference on Learning Representations (ICLR) , 2018.

E. Grave, A. Joulin, and Q. Berthet, 'Unsupervised alignment of embeddings with wasserstein procrustes,' in Proc. International Conference on Artificial Intelligence and Statistics (AISTATS) . PMLR, 2019, pp. 1880-1890.

S. Ben-David, J. Blitzer, K. Crammer, and F. Pereira, 'Analysis of representations for domain adaptation,' in Advances in Neural Information Processing Systems (NeurIPS) , vol. 19, 2006.

M. Long, Y. Cao, J. Wang, and M. Jordan, 'Learning transferable features with deep adaptation networks,' in Proc. International Conference on Machine Learning (ICML) . PMLR, 2015, pp. 97-105.

Y. Ganin and V. Lempitsky, 'Unsupervised domain adaptation by backpropagation,' in Proc. International Conference on Machine Learning (ICML) . PMLR, 2015, pp. 1180-1189.

S. Waaijenborg, P. C. Verselewel de Witt Hamer, and A. H. Zwinderman, 'Quantifying the association between gene expressions and dna-markers by penalized canonical correlation analysis,' Statistical applications in genetics and molecular biology , vol. 7, no. 1, 2008.

K. D. Yang, A. Belyaeva, S. Venkatachalapathy, K. Damodaran, A. Katcoff, A. Radhakrishnan, G. Shivashankar, and C. Uhler, 'Multi-domain translation between single-cell imaging and sequencing data using autoencoders,' Nature communications , vol. 12, no. 1, p. 31, 2021.

I. Gulrajani and T. Hashimoto, 'Identifiability conditions for domain adaptation,' in Proc. International Conference on Machine Learning (ICML) , 2022, pp. 7982-7997.

S. Shrestha and X. Fu, 'Towards identifiable unsupervised domain translation: A diversified distribution matching approach,' in Proc. International Conference on Learning Representations (ICLR) , 2024.

Y. Choi, Y. Uh, J. Yoo, and J.-W. Ha, 'StarGAN v2: Diverse image synthesis for multiple domains,' in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2020, pp. 8188-8197.

X. Huang, M.-Y. Liu, S. Belongie, and J. Kautz, 'Multimodal unsupervised image-to-image translation,' in Proc. European Conference on Computer Vision (ECCV) , 2018, pp. 172-189.

S. Xie, L. Kong, M. Gong, and K. Zhang, 'Multi-domain image generation and translation with identifiability guarantees,' in Proc. International Conference on Learning Representations (ICLR) , 2023.

P. Comon, 'Independent component analysis, a new concept?' Signal processing , vol. 36, no. 3, pp. 287-314, 1994.

K. He, X. Zhang, S. Ren, and J. Sun, 'Deep residual learning for image recognition,' in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016, pp. 770-778.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , 'Learning transferable visual models from natural language supervision,' in Proc. International Conference on Machine Learning (ICML) . PMLR, 2021, pp. 8748-8763.

Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo, 'StarGAN: Unified generative adversarial networks for multi-domain image-to-image translation,' in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018, pp. 8789-8797.

X. Fu, K. Huang, N. D. Sidiropoulos, and W.-K. Ma, 'Nonnegative matrix factorization for signal and data analytics: Identifiability, algorithms, and applications,' IEEE Signal Processing Magazine , vol. 36, no. 2, pp. 59-80, 2019.

A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola, 'A kernel two-sample test,' The Journal of Machine Learning Research , vol. 13, no. 1, pp. 723-773, 2012.

C. Villani et al. , Optimal transport: old and new . Springer, 2009, vol. 338.

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, 'Generative adversarial networks,' in Advances in Neural Information Processing Systems (NeurIPS) , 2014.

L. Kong, S. Xie, W. Yao, Y. Zheng, G. Chen, P. Stojanov, V. Akinwande, and K. Zhang, 'Partial disentanglement for domain adaptation,' in Proc. International Conference on Machine Learning (ICML) , 2022, pp. 11 455-11 472.

T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, 'A simple framework for contrastive learning of visual representations,' in Proc. International Conference on Machine Learning (ICML) . PMLR, 2020, pp. 1597-1607.

O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra et al. , 'Matching networks for one shot learning,' in Advances in Neural Information Processing Systems (NeurIPS) , vol. 29, 2016.

E. Triantafillou, T. Zhu, V. Dumoulin, P. Lamblin, U. Evci, K. Xu, R. Goroshin, C. Gelada, K. Swersky, P.-A. Manzagol et al. , 'Meta-dataset: A dataset of datasets for learning to learn from few examples,' in Proc. International Conference on Learning Representations (ICLR) , 2019.

S. Wold, K. Esbensen, and P. Geladi, 'Principal component analysis,' Chemometrics and intelligent laboratory systems , vol. 2, no. 1-3, pp. 37-52, 1987.

J. Li and J. M. Bioucas-Dias, 'Minimum volume simplex analysis: A fast algorithm to unmix hyperspectral data,' in IEEE International Geoscience and Remote Sensing Symposium , vol. 3, 2008, pp. III-250.

X. Fu, K. Huang, B. Yang, W.-K. Ma, and N. D. Sidiropoulos, 'Robust volume minimizationbased matrix factorization for remote sensing and document clustering,' IEEE Transactions on Signal Processing , vol. 64, no. 23, pp. 6254-6268, 2016.

S. Arora, R. Ge, Y. Halpern, D. Mimno, A. Moitra, D. Sontag, Y. Wu, and M. Zhu, 'A practical algorithm for topic modeling with provable guarantees,' in Proc. International Conference on Machine Learning (ICML) . PMLR, 2013, pp. 280-288.

K. Huang, X. Fu, and N. D. Sidiropoulos, 'Anchor-free correlated topic modeling: Identifiability and algorithm,' Advances in Neural Information Processing Systems , vol. 29, 2016.

X. Mao, P. Sarkar, and D. Chakrabarti, 'Estimating mixed memberships with sharp eigenvector deviations,' Journal of the American Statistical Association , vol. 116, no. 536, pp. 1928-1940, 2021.

L. Gresele, P. K. Rubenstein, A. Mehrjou, F. Locatello, and B. Schölkopf, 'The incomplete Rosetta stone problem: Identifiability results for multi-view nonlinear ICA,' in Proc. Uncertainty in Artificial Intelligence , 2020, pp. 217-227.

C. Eastwood, J. von Kügelgen, L. Ericsson, D. Bouchacourt, P. Vincent, M. Ibrahim, and B. Schölkopf, 'Self-supervised disentanglement by leveraging structure in data augmentations,' in Causal Representation Learning Workshop at NeurIPS 2023 , 2023.

J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, 'Unpaired image-to-image translation using cycleconsistent adversarial networks,' in Proc. the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017, pp. 2223-2232.

M. Long, H. Zhu, J. Wang, and M. I. Jordan, 'Deep transfer learning with joint adaptation networks,' in Proc. International Conference on Machine Learning (ICML) . PMLR, 2017, pp. 2208-2217.

W. Wang, H. Zhang, Z. Yuan, and C. Wang, 'Unsupervised real-world super-resolution: A domain adaptation perspective,' in Proc. IEEE International Conference on Computer Vision (ICCV) , 2021, pp. 4318-4327.

T. Galanti, L. Wolf, and S. Benaim, 'The role of minimal complexity functions in unsupervised learning of semantic mappings,' in Proc. International Conference on Learning Representations (ICLR) , 2018.

N. Moriakov, J. Adler, and J. Teuwen, 'Kernel of CycleGAN as a principle homogeneous space,' in Proc. International Conference on Learning Representations (ICLR) , 2020.

K. Saenko, B. Kulis, M. Fritz, and T. Darrell, 'Adapting visual category models to new domains,' in Proc. European Conference on Computer Vision (ECCV) . Springer, 2010, pp. 213-226.

H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan, 'Deep hashing network for unsupervised domain adaptation,' in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017, pp. 5018-5027.

Y. Zhang, T. Liu, M. Long, and M. Jordan, 'Bridging theory and algorithm for domain adaptation,' in Proc. International Conference on Machine Learning (ICML) , 2019.

Y. Jin, X. Wang, M. Long, and J. Wang, 'Minimum class confusion for versatile domain adaptation,' in Proc. IEEE International Conference on Computer Vision (ICCV) . Springer, 2020, pp. 464-480.

H. Rangwani, S. K. Aithal, M. Mishra, A. Jain, and V. B. Radhakrishnan, 'A closer look at smoothness in domain adversarial training,' in Proc. International Conference on Machine Learning (ICML) . PMLR, 2022, pp. 18 378-18 399.

Y. Zhang, J. Liang, Z. Zhang, L. Wang, R. Jin, T. Tan et al. , 'Free lunch for domain adversarial training: Environment label smoothing,' in Proc. International Conference on Learning Representations (ICLR) , 2023.

L. Eyring, D. Klein, T. Uscidda, G. Palla, N. Kilbertus, Z. Akata, and F. Theis, 'Unbalancedness in neural monge maps improves unpaired domain translation,' arXiv preprint arXiv:2311.15100 , 2023.

M. Amodio and S. Krishnaswamy, 'MAGAN: Aligning biological manifolds,' in Proc. International Conference on Machine Learning (ICML) . PMLR, 2018, pp. 215-223.

J. Cao, D. A. Cusanovich, V. Ramani, D. Aghamirzaie, H. A. Pliner, A. J. Hill, R. M. Daza, J. L. McFaline-Figueroa, J. S. Packer, L. Christiansen et al. , 'Joint profiling of chromatin accessibility and gene expression in thousands of single cells,' Science , vol. 361, no. 6409, pp. 1380-1385, 2018.

A. Joulin, E. Grave, P. Bojanowski, M. Douze, H. Jégou, and T. Mikolov, 'Fasttext.zip: Compressing text classification models,' arXiv preprint arXiv:1612.03651 , 2016.

K. B. Athreya and S. N. Lahiri, Measure theory and probability theory . Springer, 2006, vol. 19.

J.-F. Cardoso, 'Super-symmetric decomposition of the fourth-order cumulant tensor. blind identification of more sources than sensors.' in Proc. International Conference on Acoustics, Speech, and Signal Processing (ICASSP) , vol. 91, 1991, pp. 3109-3112.

D. P. Kingma and J. Ba, 'Adam: A method for stochastic optimization,' in Proc. ICLR , 2015.

A. L. Maas, A. Y. Hannun, and A. Y. Ng, 'Rectifier nonlinearities improve neural network acoustic models,' in Proc. International Conference on Machine Learning (ICML) , vol. 30, no. 1, 2013, p. 3.

K. Ghasedi Dizaji, A. Herandi, C. Deng, W. Cai, and H. Huang, 'Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization,' in Proc. the IEEE International Conference on Computer Vision (ICCV) , 2017, pp. 5736-5745.

J. Xie, R. Girshick, and A. Farhadi, 'Unsupervised deep embedding for clustering analysis,' in Proc. International Conference on Machine Learning (ICML) . PMLR, 2016, pp. 478-487.

W. Wang, X. Yan, H. Lee, and K. Livescu, 'Deep variational canonical correlation analysis,' arXiv preprint arXiv:1610.03454 , 2016.

A. Gretton, K. Fukumizu, C. Teo, L. Song, B. Schölkopf, and A. Smola, 'A kernel statistical test of independence,' in Advances in Neural Information Processing Systems (NeurIPS) , vol. 20, 2007.

T. M. Cover, Elements of information theory . John Wiley & Sons, 1999.

Supplementary Material of 'Identifiable Shared Component Analysis of Unpaired Multimodal Mixtures'

A Notation

The notations used throughout the paper are summarized in the Table 4.:

̸

B Proof of Theorem 1

We restate the theorem here:

Theorem 1 Under Assumption 1 and the generative model in (1), denote any solution of (6) as ̂ Q ( q ) q = 1 , 2 . Then, if the mixing matrices A ( q ) are full column ranks and E [ cc ⊤ ]) is full rank, we have Q ( q ) x ( q ) = Θ ( q ) c . In addition, assume that either of the following is satisfied:

̸

̂ (a) The individual elements of the content components are statistically independent and non-Gaussian. In addition, c i ( d ) = kc j , ∀ i = j, ∀ k ∈ R and c i ( d ) = -c i , ∀ i , i.e., the marginal distributions of the content elements cannot be matched with each other by mere scaling.

̸

̸

(b) The support C is a hyper-rectangle, i.e., C = [ -a 1 , a 1 ] ×··· × [ -a d C , a d C ] . Further, suppose that c i ( d ) = kc j , ∀ i = j, ∀ k ∈ R and c i ( d ) = -c i , ∀ i .

̸

̸

̸

Then, we have ̂ Q ( q ) x ( q ) = Θ c , i.e., Θ ( q ) = Θ for all q = 1 , 2 , where Θ ( q ) .

We will prove the theorem in following two steps. For the first step we will prove ̂ Q ( q ) x ( q ) = Θ ( q ) c and for second step we will employ either assumption (a) or (b) to prove that Θ ( q ) = Θ , ∀ q = 1 , 2 .

B.1 Linearly transformed content identification

Let us define

We want to show that

since this will imply that that H ( q ) does not depend upon the style component. Combined with the fact that rank( H ( q ) ) = d C , this will imply that H ( q ) is an invertible function of the content component. To that end, consider the following line of arguments.

Since the objective in (6) matches the distribution for latent random variables ̂ c (1) = Q (1) x (1) and ̂ c (2) = Q (2) x (2) , the following holds for any R c ⊆ R d C , ∀ k ∈ R ,

where, H ( q ) PreImg ( R c ) := { z ( q ) | H ( q ) z ( q ) ∈ R c } is the pre-image of H ( q ) . ( a ) follows because P ̂ c ( q ) [ k R c ] = P H ( q ) z ( q ) [ k R c ] = P z ( q ) [ H ( q ) PreImg ( k R c )] [68, Section 2.2]. ( b ) follows because H ( q ) is a linear operator.

Although (11) holds for any R c , we will see that it is sufficient to consider a special R c to prove (10). To that end, take R c = conv { 0 , a 1 , . . . , a d C } , where a i ∈ R d C such that P ̂ c ( q ) [ R c ] > 0 . Let us take y ( q ) i ∈ R d C + d ( q ) P , such that H ( q ) y ( q ) i = a i . For reasons that will be clear later, we hope to show that

To that end, observe that for any r ∈ R c , we can represent r as,

For both view q = 1 , 2 , we get,

We can write,

Wehave that Null( H ( q ) ) ⊂ R d C + d ( q ) P is a linear subspace with dim(Null( H ( q ) )) = d ( q ) P . Let A ( q ) = H ( q ) PreImg ( R c ) . Note that P z (1) [ k A (1) ] = P z (2) [ k A (2) ] , ∀ k ∈ R (from (11), and P z ( q ) [ A ( q ) ] > 0 (by the construction of R c ). Further, the set A ( q ) is of the form

because Null( H ( q ) ) is a subspace of dimension d ( q ) P , hence it satisfies the definition of P ( q ) . Hence, Assumption 1 implies that

Denoting the N th to M th columns of H ( q ) by H ( q ) ( N : M ) , the above is equivalent to saying

Next, we use Assumption (a) or (b) to show that Θ (1) = Θ (2) = Θ . To that end, note that the distribution matching constraint implies that

Hence M = ( Θ (1) ) -1 Θ (2) is an invertible matrix such that c ( d ) == = Mc . However, in the following, we will show that if either Assumption (a) or (b) is satisfied, then M = I , and thus Θ (1) = Θ (2) .

B.2 Considering Assumption (a)

We want to show that when Assumption (a) is satisfied, if Mc ( d ) == = c for any invertible M , then M = I .

̸

are statistically independent c i ⊥ ⊥ c j , i = j , non-Gaussian, and has non-zero kurtosis. Then, according to cumulant multilinearity and additivity properties, the fourth order cumulant tensor Cum( Mc ) of Mc has the following unique decomposition [69],

Denote,

Then, we can write,

where ◦ is the outer product, κ i is the kurtosis of component c i , and m i , i ∈ [ d C ] are the columns of M .

Since Mc ( d ) == = c , the following should hold

e i is the i th column of identity matrix I .

Because of statistical independence of components of c , the CP-decomposition of Cum( Mc ) = Cum( Ic ) is unique [69] upto permutation and scaling ambiguities, i.e., M should be a permutation scaling matrix.

Let M = ΠΣ where, Π ∈ R d C × d C is a permutation matrix and Σ = Diag( r 1 , . . . r d C ) ∈ R d C × d C is a diagonal scaling matrix.

̸

Finally, since c i ( d ) == = kc j , ∀ i = j, ∀ k ∈ R , M has to to be identity matrix. To see the reason, for the sake of contradiction, suppose that either (i) there exist i, j ∈ [ d C ] × [ d C ] and k ∈ R , with i = j such that [ Mc ] i = kc j , or (ii) ∃ i ∈ [ d C ] such that [ Mc ] i = kc i for some k ∈ R , k = 1 .

̸

For case (i), since Mc ( d ) == = c , [ Mc ] i ( d ) == = c i , ∀ i . Hence,

which is a contradiction to the assumption c i ( d ) = kc j .

̸

̸

For case (ii), [ Mc ] i = kc j implies that c i ( d ) == = kc j . First, k = ± 1 , cannot hold because it will mean that var( c i ) = k 2 var( c i ) which cannot hold for k = ± 1 since var( c i ) > 0 . Hence, the only possible option is k = -1 , which is already ruled out by the assumption that c i ( d ) = -c i . Hence M is an identity matrix. This concludes the proof.

B.3 Considering Assumption (b)

Let

denote the standard basis vector in R d C . Let vertex of hyper-rectangle v i = a i e i = Λ e i , where

where Diag( · ) represents the diagonal matrix formed by the given vector.

If Mc ( d ) == = c , then the supports of Mc and c should match, i.e., where M ( C ) = { Mc | c ∈ C} .

Note that ∀ c ∈ C , the set of points v i satisfy the following property

̸

̸

̸

̸

Since the support of Mc is C , this implies that ∀ c ∈ C ,

The last equation implies that the set of points Mv i , ∀ i ∈ [ d C ] also satisfy property (19). Hence, for each i ∈ [ d C ] , Mv i = ± v j for some unique j ∈ [ d C ] . Note that j should be unique for each i because M is invertible, hence M cannot map two orthogonal vectors v i and v k , i = k , to the same vector ± v j with same or different signs.

Let V = [ v 1 , . . . , v d C ] T . Then one can write

where Σ is some diagonal matrix with diagonal entries from { +1 , -1 } and Π is a permutation matrix. Then the above implies

Hence M is a permutation and scaling matrix.

Finally, by the same argument presented in last paragraph of Sec. B.2 (i.e., proof with Assumption (a)), we conclude that M is an identity matrix.

C Proof of Theorem 2

We restate the theorem here:

Theorem 2 Consider the mixture model in (8). Assume that rank( A ) = d C + d P and rank( E [ cc ⊤ ]) = d C , and that Assumption 2 holds. Denote ̂ Q as any solution of (6) by constraining Q = Q (1) = Q (2) . Then, we have ̂ Qx ( q ) = Θ c . One can follow the same argument as in the step 1 of proof in B.

Let us define

We want to show that

since this will imply that that H does not depend upon the style component. Combined with the fact that rank( H ) = d C , this will imply that H is an invertible function of the content component. To that end, consider the following line of arguments.

Since the objective in (6) matches the distribution for latent random variables ̂ c (1) = Qx (1) and ̂ c (2) = Qx (2) , the following holds for any R c ⊆ R d C , , ∃ k ∈ R P ̂ c (1) [ k R c ] = P ̂ c (2) [ k R c ] , ( a ) ⇐⇒ P z (1) [ H PreImg ( k R c )] = P z (2) [ H PreImg ( k R c )] (21) ( b ) ⇐⇒ P z (1) [ k H PreImg ( R c )] = P z (2) [ k H PreImg ( R c )] , (22)

where, H PreImg ( R c ) := { z | Hz ∈ R c } is the pre-image of H . ( a ) follows because P ̂ c ( q ) [ k R c ] = P Hz ( q ) [ k R c ] = P z ( q ) [ H PreImg ( k R c )] [68, Section 2.2]. ( b ) follows because H is a linear operation.

Although (21) holds for any R c , we will see that it is sufficient to consider a special R c to prove (20). To that end, take R c = conv { 0 , a 1 , . . . , a d C } , where a i ∈ R d C such that P ̂ c ( q ) [ R c ] > 0 . Let us take y i ∈ R d C + d P , such that Hy i = a i . For reasons that will be clear later, we hope to show that

̸

To that end, observe that for any r ∈ R c , we can represent r as,

For both view q = 1 , 2 , we get,

We can write,

We have that Null( H ) ⊂ R d C + d P is a linear subspace with dim(Null( H )) = d P . Let A = H PreImg ( R c ) . Note that P z (1) [ k A ] = P z (2) [ k A ] , ∀ k ∈ R (from (21), and P z ( q ) [ A ] > 0 (by the construction of R c ). Further, the set A is of the form

because Null( H ) is a subspace of dimension d P , hence it satisfies the definition of P . Hence, Assumption 2 implies that

Denoting the N th to M th columns of H by H ( N : M ) , the above is equivalent to saying

Denote,

Then, we can write,

This concludes the proof.

D Proof of Theorem 3

We restate the theorem here:

Theorem 3 Assume that Assumption 1 is satisfied, that |L| ≥ d C paired samples ( x (1) ℓ , x (2) ℓ ) are available, that A ( q ) , q = 1 , 2 have full column rank, and that P c is absolutely continuous. Denote ( ̂ Q (1) , ̂ Q (2) ) as any optimal solution of (6) under the constraint (9). Then, we have ̂ Q ( q ) x ( q ) = Θ c . From our objective in (6), we obtain

Using Assumption 1 and following the proof of step 1 in Theorem B, we can obtain:

for some invertible matrices Θ ( q ) , ∀ q . Hence,

Hence we can have linear transformation M := ( Θ (1) ) -1 Θ (2) which has same probability density as P c . However, the sample matching constraint (9), for ℓ -th sample implies that

Let C = [ c 1 . . . c N p ] . Then the above implies:

Now we show that C is a full row rank matrix, which implies that M -I = 0 = ⇒ M = I . To that end, note that random variables x (1) i and x (2) i being i.i.d implies that c ( i ) are i.i.d from P c . This implies that for any 1 ≤ i ≤ |L| ,

This is because span( { c n 1 , . . . , c n d C -1 } ) for n j ∈ [ |L| ] , is a lower dimensional subspace in R d C , which has zero probability under absolutely continuous distribution P c . Hence any d C out of |L| column vectors in C are linearly independent with probability 1.

This concludes the proof.

E Detailed Identifiability Conditions of Existing Results

E.1 Identifiability of CCA

Theorem 4 (Identifiability of Aligned SCA via CCA [1]) . Under (1) , assume that every aligned pair ( x (1) , x (2) ) share the same c , and that A ( q ) has full column rank. Also assume that there exists an N -sample set { ℓ 1 , . . . , ℓ N } such that [ C ⊤ , ( P (1) ) ⊤ , ( P (2) ) ⊤ ] ∈ R N × ( d C + d (1) P + d (2) P ) has full column rank, where C = [ c ℓ 1 , . . . c ℓ N ] ∈ R d C × N and P ( q ) = [ p ( q ) ℓ 1 . . . p ( q ) ℓ N ] ∈ R d ( q ) P × N for q = 1 , 2 . Denote ( ̂ Q (1) , ̂ Q (2) ) as an optimal solution of the CCA formulation. Then, we we have where Θ is nonsingular.

In the above theorem, one can see that N ≥ ( d C + d (1) P + d (2) P ) is a necessary condition for the identifiability of Θ c . Hence, CCA needs at least d C + d (1) P + d (2) P paired samples for identifiability.

E.2 Identifiability of Unaligned SCA in [8]

We summarize the result in [8] in the following

Theorem 5 (Identifiability of Unaligned SCA via ICA [8]) . Under (1) , assume that the following are met: (i) The conditions for ICA identifiability [33] is met by each modality, including that the components of z ( q ) = [ c ⊤ , ( p ( q ) ) ⊤ ] ⊤ are mutually statistically independent and contain at most one Gaussian variable. In addition, each z ( q ) i has unit variance; (ii) P z ( q ) i = P z ( q ) j , P z ( q ) i =

̸

P -z ( q ) j ∀ i, j ∈ [ d C + d ( q ) P ] , i = j . Then, assume that ( i m , j m ) are obtained by ICA followed by cross domain matching (see the part on Unaligned SCA in Section 2 ) for m = 1 , . . . , d C .

̸

̸

F Additional Synthetic Data Experiments

Hyperparameter Settings: We use Adam optimizer [70] to solve (7) and learn matrices Q ( q ) , q = 1 , 2 and the discriminator f . We set the initial learning rate of matrix and discriminator to be 0 . 009 and 0 . 00008 respectively. We set the λ = 0 . 1 in (7) to enforce (6c). For weak supervision experiment in F, we set β = 0 . 01 in (9). We generate total of 100,000 samples in each domain. For our experiment we set the batch size to be 1,000 and run (7) for 50 epochs. Our discriminator is a 6-layer multilayer perceptron (MLP) with hidden units { 1024, 521, 512, 256, 128, 64 } in each layer. All the layers use leaky ReLU activation functions [71] with a slope of 0.2 except for the last layer which has sigmoid activations. We include a label smoothing coefficient of 0.2 in the discriminator predictions as suggested in [40].

Additional Details for Validation of Theorem 1 in Sec. 3: Here we explain the data generation details of the result shown in Fig. 3. For the result in top row, we sample c 1 from a Gaussian mixture with three Gaussian components. Each component follows a normal distribution N ( µ, 2) where µ ∼ N (0 , 10) . The second component, i.e., c 2 , is independently sampled from the gamma distribution Gamma (1 , 3) . The private components are sampled from p (1) ∼ Laplace (1 . 0 , 6 . 5) and p (2) ∼ Uniform [ -10 , 10] , both only having one dimension. In the bottom row, we sample c ∈ R 2 ∼ VonMises (2 . 5 , 2 . 0) distribution. The private components satisfy p (1) ∼ Laplace (1 . 0 , 6 . 5) and p (2) ∼ Gamma (0 . 5 , 3 . 0) . Each element of mixing matrices are sampled from A ( q ) ij ∼ N (0 , 1) , q = 1 , 2 . The readers are referred to Table 4 for the definition of notations used for distributions.

Validation of Theorem 1 under different sample sizes and imbalanced data: Here we observe the shared component identification performance of the proposed method numerically. We conducted two experiments in different settings. First, we vary the sample sizes in both modalities, but the two modalities have the same sample size. Second, we only vary the sample size of modality 2 while keeping the sample size of modality 1 fixed. This way, we create the data imbalance between modalities. Note that the shared components are identified if the following two conditions are met:

Therefore, we use the above as our performance evaluation metrics. For the following experiments (Table 5 and 6), we generate the data for the two modalities by sampling a two-dimensional content c ∼ VonMises (2 . 5 , 2 . 0) and private components from p (1) ∼ Laplace (1 . 0 , 6 . 5) and p (2) ∼ Gamma (0 . 5 , 3 . 0) . The elements of the mixing matrices are sampled as A ( q ) ij ∼ N (0 , 1) , q = 1 , 2 . We report the mean and standard deviation of ∥ ̂ Θ (1) (1 : d C ) -̂ Θ (2) (1 : d C ) ∥ F and 1 / 2 ∑ 2 q =1 ∥ ̂ Θ ( q ) ( d C : d C + d ( q ) P ) ∥ F obtained from 5 different runs.

Table 5 shows the performance of SCA and CCA under different sample sizes (i.e., N ). One can see that the proposed method (SCA) clearly identifies the shared components even when only 100 samples are available. The performance starts to deteriorate when N ≤ 50 , probably because the min-max optimization problem is difficult to solve with very few samples. CCA does not really work under this setting as it needs aligned cross-domain samples.

Table 6 shows the performance of SCA in the cases where two modalities have unbalanced data sizes. The number of samples in the first modality is fixed to 100,000 while the second modality's data size varies from 10,000 to 10 samples. The data generation process remains the same as in the previous experiment. One can see that even under obvious cross-domain data size imbalance (e.g., 100,000 to 1,000), the proposed method performs reasonably well in terms of shared component identification.

±

±

±

±

±

±

Validation of Theorem 3. Fig. 5 presents numerical validation for Theorem 3.

Data Generation: We set d C = 3 and d ( q ) P = 1 for q = 1 , 2 . We sample each component of shared component c i ∼ Laplace (0 . 0 , 6 . 5) i = 1 , 2 , 3 , p (1) ∈ R 1 ∼ Uniform [ -10 , 10] and p (2) ∼ Gamma (0 . 5 , 3 . 0) . Although c satisfies component-wise independence assumption, it does not

̸

satisfy the condition that c i ( d ) = kc j , ∀ i = j because c i ( d ) == = c j , ∀ i, j ∈ [3] . Therefore, Theorem 1 does not cover this case. Nonetheless, this case falls under the jurisdiction of Theorem 3.

̸

Result: Fig. 5 corroborates with our Theorem 3. That is, one needs at least |L| ≥ d C = 3 pairs of 'anchors' (i.e., aligned cross domain pairs) to ensure identifiability of ̂ c ( q ) = Θ c for q = 1 , 2 . Ground truth

-

100

-

50

0

50

100

c

-

100

-

50

0

50

100

-

100

-

50

0

50

100

-

50

0

50

x

(1)

-

50

0

50

-

50

0

50

x

(2)

-

100

-

50

0

50

100

̂

c

(1)

-

100

-

50

0

50

100

No anchors

-

100

-

50

0

50

100

-

100

-

50

0

50

100

-

100

-

50

0

50

100

̂

c

(1)

-

100

-

50

0

50

100

1 anchors

-

100

-

50

0

50

100

c

(2)

-

100

-

50

0

50

100

-

100

-

50

0

50

100

̂

c

(1)

-

100

-

50

0

50

100

3 anchors

-

100

-

50

0

50

100

c

(2)

-

100

-

50

0

50

100

G Real Data Experiment Settings and Additional Results

G.1 Domain Adaptation

Hyperparameter Settings: The domain adaptation task follows the hyperparameter settings described in Table. 7.

̂

Baselines and Training Setup : The baselines are representative DA methods, namely, DANN [25], MDD [60], MCC [61], SDAT [62], and ELS [63]. We use the implementations of DANN , MDD , and MCC from the https://github.com/thuml/Transfer-Learning-Library , while SDAT and ELS are taken from https://github.com/yfzhang114/Environment-Label-Smoothing . In all the baselines, the classifier is jointly optimized with the feature extractor Q which arguably regularizes towards more classification-friendly geometry of the shared features; see [72, 73]. Following their training strategy, we also append a cross-entropy (CE) based classifier training module to our loss in (7) (which learns our feature extractor Q ). The CE part uses Qx (1) and the labels of the sources as inputs to learn the classifier, i.e.,

where r θ ( · ) : R d C → R K is the classifier that aims to map the learned feature vector Qx (1) ℓ to a K -dimensional probability mass function (i.e., the distribution of the ground-truth label over K classes), y ℓ ∈ [ K ] represents the label of the ℓ th sample in source domain, and the indicator function I [ y ℓ = k ] = 1 only when the event y ℓ = k happens (other wise I [ y ℓ = k ] = 0 . The γ ≥ 0 is the tunable parameter. The joint loss is still differentiable, and thus we still use the Adam optimizer to jointly optimize Q and θ .

Additional domain adaptation experiment using CLIP features: In this experiment, we use CLIP as an image encoder as it learns informative and transferable features from very large datasets [35]. Table 8 and Table 9 show the results on Office-31 and Office-Home datasets, respectively, using CLIP embeddings. Compared to the results on ResNet50 embeddings in Table 1 and Table 2, one can observe that all the methods, including proposed method, gains an advantage. This is likely because CLIP was trained on a large and diverse dataset [35], which may have include similar content to the Office-31 and Office-Home datasets.

The results show that, as a foundation model, CLIP can already unify the embeddings of the source and target domains to a reasonable extent. In addition, our model and algorithm when combined with regularization techniques like MCC, can still further enhance performance, even with simple post-processing of CLIP embeddings.

Visualization Result: Fig. 6 shows the 2-dimensional visualization of the CLIP-learned features ( d = 256 ) from two domains, namely, DSLR and Amazon images ( Office-31 ), using t-SNE. One can see that CLIP could roughly group the same classes from the two domains together. But the proposed method can further pull the circles and the triangle markers together-meaning that the Q really learns shared representations of the same data in the DSLR and Amazon domains.

40

30

20

10

0

10

20

30

40

50

40

30

20

10

0

10

20

30

Features from CLIP

(a) Clip features

d

(

q

)

= 768

.

40

20

0

20

40

40

30

20

10

0

10

20

30

40

Low dimensional features from proposed method

(b) Features from proposed method

d

(

q

)

= 256

.

C

G.2 Single-cell Sequence Analysis

Hyperparameter Settings: The hyperparameter settings for single-cell sequence analysis is presented in Table. 10.

Baseline: For more details on baseline refer to the implementation in https://github.com/ uhlerlab/cross-modal-autoencoders .

G.3 Multi-lingual Information Retrieval

Hyperparameter Settings: The hyperparameter settings for multi-lingual information retrieval is described in the Table. 11.

Additional Results: Table 12 reports the P@5 and P@10 scores over the test data, calculated for different source and target language pairs. It can be observed that the proposed method achieves higher precision than Adv in most of the translation tasks (e.g., by at least 1% in the en → es and es → en tasks) when considering both P@5 and P@10 scores.

G.4 Computation resources

All the experiments were run on Nvidia H100 GPU. The approximate runtime for a single run of the algorithm is 20 minutes for multi-lingual information retrieval, 15 minutes for domain adaptation, and 3 minutes for single-cell sequence analysis.

Complexity Analysis:

Since the proposed objective is tackled using stochastic gradient (SG)-based first-order iterative method, the computational complexity of the proposed algorithm depends upon the per-iteration complexity.

For each sample, the per-iteration complexity is composed of a forward pass and a backward pass.

Note that the problem size depends upon d ( q ) (the data dimension), d C , and the batch size B . We assume that the network architecture of f (the number of layers and hidden units in each layer) is fixed, represented by f = σ ◦ F L ◦ · · · ◦ σ ◦ F 1 , where F ℓ and σ are the linear layer (matrix) and activation function corresponding to the ℓ th layer. Only the input dimension d C of first matrix F 1 , varies with the problem size.

The forward pass involves computing ̂ c ( q ) = Q ( q ) x ( q ) and f ( q ) ( ̂ c ( q ) ) , both of which scale linearly with d C , d ( q ) and the batch size B . Hence, the forward pass time complexity is O ( Bd C ( d (1) + d (2) )) .

Similarly, the backward pass requires computing of ∂L ∂ c ( q ) , i [ d C ] and ∂ ̂ c ( q ) i ∂ Q ( q ) jk , i [ d C ] , j, k

̂ i ∀ ∈ ∀ ∈ ∈ [ d C × d ( q ) ] , where L is the loss function. The first gradient computation is linear in Bd C , while the second gradient computation has a complexity of O ( Bd C ( d (1) + d (2) )) . Hence the computational complexity of our method is O ( Bd C ( d (1) + d (2) )) .

→

The memory complexity involves storing the network parameters and the aforementioned gradients. Hence, only the size of Q ( q ) , F 1 , and c ( q ) changes with the problem dimension. The size of Q ( q ) , F 1 , and c ( q ) are d C d ( q ) , O ( d C ) and d C , respectively. Therefore, the space complexity is O ( Bd C ( d (1) + d (2) )) .

In summary, both the memory and computational complexities of the proposed method scales linearly with d C .

H Extension: Private Component Identification

Theorems 1-3 are concerned with learning the shared component c . The goal, there, was to ensure that Q ( q ) C x ( q ) Θ c , ∀ q . In some cases, the private components p ( q ) is also of interest [6, 31, 74]. To learn p ( q ) , we propose to solve the following learning criterion:

where u ⊥ ⊥ v means that the random vectors u and v are independent with each other.

For implementation we use following criterion,

where, first two term are adversarial loss for distribution matching. The constraint on (33d) and (33e) are enforced as R ( Q C ( q ) ) and R ( Q C ( q ) ) respectively, where R ( Q ( q ) ) = ∥ Q ( q ) E [ x ( q ) ( x ( q ) ) ⊤ ]( Q ( q ) ) ⊤ -I ∥ 2 F . The constraint on (33c) is realized with Hilbert-Schmidt Independence Criterion (HSIC) [75]. HSIC measures the independence between two distribution. So, we minimize HSIC between estimated shared component and estimated private component to promote independence between shared and private components.

We show that under some reasonable conditions the block p ( q ) can also be learned up to a matrix multiplication:

Theorem 6. Assume that the blocks c , p (1) and p (2) are statistically independent, i.e., p ( c , p (1) , p (2) ) = p ( c ) p ( p (1) ) p ( p (2) ) . Then, if one of the following holds:

(i) Assumption 1 and assumptions in Theorem 1 are satisfied, and (33) is solved yielding solutions Q ( q ) C and Q ( q ) P

̂ ̂ (iii) Assumption 1 is satisfied and d C paired samples ( x (1) ℓ , x (2) ℓ ) are available (weak supervision), and denote Q ( q ) C and Q ( q ) P as the solutions after solving (33) .

̂ ̂ (ii) Assumption 2 is satisfied and has same mixing matrix A ( q ) = A and (33) with Q ( q ) P = Q P and Q ( q ) C = Q C is solved yielding Q ( q ) C and Q ( q ) P as the solutions.

̂ ̂ Then, we have ̂ Q ( q ) C x ( q ) = Θ c and ̂ Q ( q ) P x ( q ) = Ξ ( q ) p ( q ) , for some invertible Ξ ( q ) for all q = 1 , 2 .

Proof. For each case in Theorem. 6 (i) - (iii), we can prove

̂ ̂ using Theorems 1-3. The proofs are referred to Appendix B-D. Let us denote where H ( q ) = ̂ Q ( q ) P A ( q ) ∈ R d ( q ) P × ( d C + d ( q ) P ) . Note that the constraint (33c) implies that the mutual information between ̂ p ( q ) and ̂ c ( q ) is zero, i.e., I ( ̂ p ( q ) ; ̂ c ( q ) ) = 0 . Note that ̂ p ( q ) → ̂ c ( q ) → Θ -1 ̂ c ( q ) = c is a Markov chain. This is because when conditioned on ̂ c ( q ) , Θ -1 ̂ c ( q ) becomes constant, making it independent of ̂ p ( q ) . This allows us to use the data processing inequality [76, Theorem 2.8.1], which results in the following:

Therefore ̂ p ( q ) = H ( q ) [ d C + 1 : d C + d ( q ) P ] p ( q ) = Ξ ( q ) p ( q ) , ∀ q , where Ξ ( q ) = H ( q ) [ d C + 1 : d C + d ( q ) P ] . Note that H ( q ) is full row-rank because of constraint (33e). This implies that Ξ ( q ) , q = 1 , 2 are invertible matrices.

I ( ̂ p ( q ) ; ̂ c ( q ) ) ≥ I ( ̂ p ( q ) ; Θ -1 ̂ c ( q ) ) = I ( ̂ p ( q ) ; c )) . Since mutual information is always non-negative, the above implies that I ( ̂ p ( q ) ; c ) = 0 . This implies that ̂ p ( q ) = H ( q ) [ c p ( q ) ] is independent of c . Hence, H ( q ) [1 : d C ] = 0 , ∀ q .

This concludes the proof.

H.1 Validation of Theorem 6

Fig. 7 presents numerical validation for Theorem 6.

Hyperparameter Setting The hyperparameter setting is the same as mentioned in Appendix. F. We solve (34) to obtain ̂ Q ( q ) C and ̂ Q ( q ) P to recover the shared and private components, respectively. For learning Q ( q ) P , we use Adam optimizer and set initial learning rates to be 0 . 001 . Also we set the regularization parameter ω = 10 . 0 and ρ = 50 . 0 .

Data Generation: We set d C = 2 and d ( q ) P = 1 as in the previous synthetic experiments. We sample c ∼ VonMises (2 . 5 , 2 . 0) The private components are sampled from p (1) ∼ Beta (1 . 0 , 3 . 0) and p (2) ∼ Gamma (0 . 5 , 3 . 0) distributions. Each element of mixing matrices are sampled from A ( q ) ij ∼ N (0 , 1) , q = 1 , 2 .

Result: Fig. 7 shows the result for proposed method for private component identification. The first column shows the data domain, the second column shows the true and extracted shared component, and the third and fourth columns shows the true and extracted private components. Especially, the last row of the third and fourth columns shows the plot of ground truth p ( q ) on x -axis and ̂ p ( q ) on the y-axis. The plot is approximately a straight line which indicates that the estimated private components ̂ p ( q ) are scaled version (i.e., invertible linear transformations) of ground truth private components. This verifies our Theorem 6.

-

-

3

-

2

-

1

0

1

2

3

-

3

-

2

-

1

0

1

2

3

c

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

0

50

100

150

200

250

300

p

(1)

0

10

20

30

40

50

60

70

0

1000

2000

3000

4000

p

(2)

-

40

-

20

0

20

40

60

80

-

75

-

50

-

25

0

25

50

75

x

(1)

-

4

-

3

-

2

-

1

0

1

2

-

1

0

1

2

3

4

5

̂

c

(1)

-

0

.

5

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

0

50

100

150

200

250

̂

p

(1)

0

10

20

30

40

0

1000

2000

3000

4000

̂

p

(2)

-

100

-

50

0

50

100

-

80

-

60

-

40

-

20

0

20

40

60

80

x

(2)

-

4

-

3

-

2

-

1

0

1

2

-

1

0

1

2

3

4

5

̂

c

(2)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

p

(1)

-

0

.

5

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

-

̂

p

(1)

→

0

10

20

30

40

50

60

70

p

(2)

0

10

20

30

40

-

̂

p

(2)

→

→

-

→

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See Section 3, 4 and 6.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Section 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: See Appendix B, C, D and H.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Appendix F and G.

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Yes the code is provided in the supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Appendix F and G.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We have provided the error bars for the experiment that is computationally less demanding.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have followed the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The contribution of this paper is on theoretical aspects of machine learning. We don't foresee any immediate societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Theoretical paper. So not applicable in our case.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We donot release any new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Not applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Not Applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a method for performing a shared component analysis (SCA) in the case of multi-modal unpaired data drawn from a linear mixture. This problem (and method) will be referred to as Unaligned SCA. Unaligned SCA is tackled by matching the probability distributions of the embedded (features) multi-modal data. More specifically, the authors draw inspiration from the traditional adversarial loss used in GANs, and formulate the component analysis algorithm as a min-max optimization problem where a discriminator (a neural network) is trained to maximize the confusion between the alignment of two representations (typically from two different modalities), and then the best alignment is sought in order to “fool” the current discriminator. The alignment matrices are structured so that the algorithm can distinguish between components shared across modalities and private components specific to each modality. The authors claim that the shared components can be identified up to the same ambiguities as those identifiable in the aligned case. Furthermore, the authors explain that while there are other methods that attempt to solve the Unaligned SCA, the conditions of the proposed algorithm are considerably milder.\n\nThe algorithm is then extended to cases where additional knowledge is present. First the algorithm is modified to accommodate the scenario where the data is generated by a single modality (uniform modality). Then, the algorithm is modified to accommodate the case where some data pairing is available (similar to a weakly supervised case). The authors show that by adding appropriate constraints  the shared component can be identified under milder conditions.\n\nThe author provide first a theoretical analysis with numerical simulations, and then some concrete applications of Unaligned SCA for the problem of domain adaptation (same modality), Multi-lingual Information Retrieval (only in the appendix, same modality) and Single Cell Sequence Analysis (multi modality with and without pairing).\n\nSTRENGTHS:\n- The work tackles an important area of research with potential applications that range from explainability, to SSL, or multi-modal problems.\n- The method seems very flexible: \n    - The method proposed can work for completely unpaired data but if some paring is available it can take advantage of such additional knowledge. \n    - The method proposed is meant for multi-modal scenarios but it can also work in homogenous use cases.\n- All parameters are well documented in the appendix and code was part of the additional material.\n\nWEAKNESSES:\nI would divide the weaknesses into two groups: the empirical evaluation, of which I am fairly confident about, and the theoretical analysis, of which due to my limited knowledge in this field I am less confident of. I will share here my concerns as solving them might also be helpful for other readers in my position\n\n*Empirical Analysis.* \n\nI find the empirical analysis weak. The multi-modal and unpaired scenario results are underwhelming, while I find the domain adaptation (same modality) potentially problematic due to the use of CLIP as pre-processing step, and the lack of a strong recent baseline (less problematic than the CLIP reason).\n- The only practical results where the data are multi-modal and unpaired are the ones shown in Figure 4 first blue dot where the accuracy is about 10%. Since this is the only case presented it is unclear if the method, in practice, cannot cope in this scenario or if the specific problem chosen is particularly challenging (in which case other evidence would be maybe better).\n- In the domain adaptation experiments the paper reads: “The images are pre-processed by the pretrained CLIP model [34] that uses ViT-L/14 transformer architecture.” It is not clear if ALL baselines used the CLIP embeddings as pre-processing, or if this was only done for the proposed algorithm. For fair comparison the same pre-processing should be applied to all algorithms. \n- Additionally, CLIP is known to have been pre-trained on a large and diverse dataset and there is a good chance it has been trained on Home-Office and Office-31 too, so it is difficult to appreciate the ability of the proposed method when using such a powerful pre-processing step (which as I mentioned might have been trained on theses datasets, including the their test sets). So making sure the pre-processing is equal is necessary, I would also encourage to present results with a less powerful pre-processing step (e.g., something pre-trained on ImageNet either supervised or SSL style like SimCLR) in order to better distinguish the contribution of CLIP vs every baseline and the proposed model.\n- The authors use a lot of baselines as comparison however all these baselines seem to be fairly old (all before 2020?). I would suggest comparing with a stronger baseline (either check the leaderboard or here are some suggestions [1-4], note that not all might be immediately applicable). \n\n[1] D3GM (https://arxiv.org/pdf/2401.05465)\n\n[2] CLUE (https://arxiv.org/abs/2010.08666)\n\n[3] LAMDA (https://arxiv.org/abs/2208.06604)\n\n[4] SDM (https://arxiv.org/abs/2203.05738)\n\n\n*Theoretical analysis.*\nI have struggled to follow the theoretical explanation. Specifically, I understand the rationale behind formulation in eq (7) but I would fully agree with it if the samples were paired. I do not understand how 6(b) holds for unpaired samples. I suppose this is the explanation currently presented before assumption 1 but even after reading it I was left with the same question.\n\nQUESTIONS:\nA satisfactory answers to these points could improve the \"contribution\" (and partially the \"soundness\") of the work.\n- Was CLIP used as a pre-process step for all the baselines in the Domain Adaptation? If not, could the authors present those results by using the same pre-processing step? As I mentioned above ideally without using CLIP due to the potential contamination of the test sets.\n- Could the author explain why 6(b) holds for unpaired samples?\n\nFurther addressing these less critical aspects could improve the \"Presentation\" score (and partially the contribution see first point)\n- I believe the paper would be stronger with more recent baselines as suggested above.\n- I find unclear how Fig.1 was created. More explanation would help the understanding. \n- I find Fig 2 not clear: why are there c1 and c2 and only p1, whereas I was expecting p1 and p2 and a common (shared) c? Reading if further it might be that c1 and c2 are the two axes of the common space $c$. If this is the case I’d make sure to clarify it.\n- Assumption 1 is not clear to me. Is this saying that both points y1 and y2 leave in the same sub-space within the span of $(c, p^{(1)}, p^{(2)})$?  If so in which way is this useful?\n- The authors say the results are an average of 5 runs, which is great, but  the standard deviation should also be reported.\n- I find this sentence confusing: \"First, it is unclear if (6b) could disentangle c from p(q). In general, Q(q)x(q)\ncould still be a mixture of c and p(q) yet (6b) still holds (e.g., when both c and p(q) are Gaussian.)\" first it says it is unclear if they can be disentangled, but the whole identifiably relies on the ability to disentangle them no?\n- In a couple of places in the manuscript the authors refer to an experiments where all the results are in the appendix. I would add a brief summary of the results so the main paper is self-standing (this happens in More Synthetic-Data Validation and Application (iii)).\n- In a few places $p^{(2)}$ appears without a closed bracket, i.e., $p^{(2}$.\n- Sometimes the authors used the comma (,) to separate thousands but other time they didn’t. I would use a uniform notation.\n\nLIMITATIONS:\nYes, the authors have identified and clearly stated three main limitations:\n- The fact that the conditions presented are sufficient while the necessary are not yet known.\n- The fact that the method works only for linear mixtures (which limits the expressivity).\n- Lastly, the fact that the theoretical derivation assume infinite data.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

comments: SUMMARY:
The paper considers the identifiability of shared components from a linear mixture. The theory requires multiple domains. However, compared to previous works, the required domains do not need to be aligned in this work. A practical estimation model has been proposed according to the theory.

STRENGTHS:
1. The discussion on the related work is comprehensive.

2. The experiments have been conducted on both synthetic and real-world datasets.

3. The proposed algorithm looks pretty neat.

4. Limitations have been discussed in detail together with potential next steps.

WEAKNESSES:
1. Since there are already many works in learning shared components in the nonlinear setting, and some of them can even handle unpaired mixtures, the linear setting appears less appealing in comparison.

2. Assumption 1 is similar to the one used in the previous works, which should be highlighted earlier in the paper.

3. The discussion of the assumption of hyper-rectangle support is missing. Is it restrictive? Maybe some real-world examples could be helpful.

QUESTIONS:
1. I didn't fully understand the proof in Line 780--does the usage of data processing inequality require a Markov chain? If so, has it been shown in the proof?

2. Could you please elaborate more on the connection between the proposed theory and previous work focusing on identifying content and style variables? It seems like they share a similar goal.

LIMITATIONS:
The authors have discussed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work considers a problem similar to classical Canonical Correlation Analysis (CCS), which assumes a linear generative model for data $(x_1, x_2)$: $x_1=W_1z$, $x_2=W_2z$ and aims to identify the underlying components. \n\nThis problem has been extended previously to include \"private information\": $x_1=W_1z_1, z_1=[c,p_1]$, $x_2=W_2z_2, z_2 = [c,p_2]$ for common $c$ and independent $p_j$. \n\nThe current work further assumes that data is *unpaired* and rather than mapping each pair ($x_1$, $x_2$) such that $z_1$, $z_2$ are close together (in some metric), it is proposed that all $x_1$ are mapped to be similar *in distribution* to the mapped $x_2$s.\n\nSTRENGTHS:\nThe paper aims to provide rigorous criteria in which underlying generative factors are identifiable in the extended CCA problem it tackles (unpaired CCA with private information).\n\nThe results show improvement over benchmarks indicating promise to the approach.\n\nWEAKNESSES:\nHigh level:\n* While I understand the basics I am not an expert in the area of CCA, but I find the paper fairly difficult to follow. More explanation would be helpful, e.g. \n    - [28] why is any linear mixture model ill-posed (is that strictly true in *every* linear mixture case?)\n    - [47] what is meant by \"facilitating one-to-many translations\", the context/meaning is unclear.\n* The theoretical part of the paper relates to a simple linear model, but none of the experiments follow this model\n    - e.g. the algorithm is applied to CLIP embeddings, which are not \"the data\", so to make claims about a simple linear model z=Ax and then apply it to CLIP seems incongruous. This experiments seem to relate more to a CCA-based \"loss function\" that takes representations and looks to align them/encourage independent factors etc.\n    - other experiments appear to be on discrete data, which the methods doesn't apply to, presumably these are also represented as some intermediate step?\n    - it seems strange to propose a simple linear model, present theoretical results about identifiability that rely on that simplicity but make a dramatic departure in the experiments where the assumptions clearly do not hold and the notion of identifiability is unclear.\n*  if the work does achieve an improvement in a CCA type setting, it seems appropriate to compare with other CCA methods on suitable data. Adding results on more complex data/representations may be of additional interest.\n\nAssumption 1 \n - hard to parse and could be made more clear. \n - unclear if correctly defined, don't vectors y need to be orthogonal to subspace P? It seems extremely loose to the point of simply saying $P_{c,p_1} \\ne P_{c,p_2}$ (specifying where any difference lies to this might add clarity).\n\nTheorem 1 \n - is this saying that if all dims of c are distributed differently, p(z)'s can only match (e.g. under GAN loss) by correctly aligning each dim? If so, that is pretty intuitive and it could be made clearer that you are putting that mathematically and proving it for the sake of rigour.\n - I have not been through the proof, 5+ pages of proof without a sketch in the paper might be more suitable for a journal as appendices are not typically expected to be reviewed in detail.\n\nOverall, there may be useful results in the paper, but in my view it should be re-written to make more clear what it is doing. It seems a confusing mix of simple linear generative model and related CCA methodology mixed with much more complex representations (e.g. CLIP) passed through a GAN + linear layer.\n\nQUESTIONS:\nsee weaknesses\n\nLIMITATIONS:\nsee weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work considers the identifiability of linear latent representations that are shared (i.e., identical) across data modalities, in the special case that they are unaligned/unpaired.\nThe approach leverages GAN-style training to achieve divergence minimization between the latent distribution of each modality.\nThe approach appears to be restricted to two modalities.\nUnder the assumption of shared latents, sufficient (not necessary) conditions for identifiability are presented, which are milder and, thus, more general than existing studies.\nFurther, structural constraints based on side information are introduced to further relax identifiability conditions.\nSeveral experiments on simulation and real-world data are provided.\n\nSTRENGTHS:\nOriginality :\n- The work introduces a combination of novel and known ideas in a clever formulation that yields new, less restrictive conditions for identifiability of shared signals from two modalities.\n- The work differs from and extends previous contributions, dealing two unaligned/unpaired modalities.\n- The work further relaxes the identifiability conditions via structural constraints based on additional side information that may be available in certain problems.\n- The manuscript cites related work on identifiability conditions for aligned/paired data, as well as unaligned/unpaired results using stricter ICA conditions, also linking the work to nonlinear studies, adequately indicating the sources of inspiration.\nQuality :\n - The work is technically sound, including proofs for identifiability claims.\n - The theoretical claims are well supported by the experiments.\nClarity :\n - The work is well written and organized, focusing on the key points and contributions.\nSignificance :\n - The results appear to be quite meaningful, with a potentially wide range of application.\n\nWEAKNESSES:\nQuality :\n - The code is not too friendly to readers, lacking higher 1-to-1 correspondence with the notation in the paper. Suggest improving documentation and tidying up the codebase for readability.\n - Figure 5 does not seem to replicate well in `synthetic_train.ipynb` --> Clarify\n - The numerical validations (simulation) are limited to 100,000 samples. Could you illustrate performance at 10,000, at 1,000, and at 100 samples? Many multimodal applications are limited to sample-poor regimes (N < 100), where classical CCA is one of the few performant methods, so it would be useful to assess the performance of unaligned SCA at varying sample sizes, and perhaps include comparable CCA results. Is there a summary measure (like Amari distance) that could be reported alongside figures?\nClarity :\n - The paper contains some typos that limit the clarity. Besides fixing these typos, consider improving the readability of the proofs by being a bit more explicit with \"obvious\" steps that may be currently omitted.\n - I think the set L of paired samples was not defined?\n - Line 59: \"transformations identifies\" --> \"transformations identify\"\n - Line 65: \"samples available\" --> \"samples are available\"\n - Lines 95-96: \"the cross-modality samples share the same c are aligned\" --> unclear meaning... maybe drop \"share the same c\"?\n - Line 117: \"to met\" --> \"to be met\"\n - Line 183: Sentence ends abruptly at: \"where \\Theta^(q).\"\n\nQUESTIONS:\n1. What do you mean by \"lift\" the constraints in line 131?\n2. Although theorem 1(a) does not \"require\" independence between c and p, isn't that implied/necessary? Otherwise, could you show that Dependence between p and c still yields identifiability of c? Specifically, say, if c1 and c2 are conditionally independent p(c1,c2,p) = p(c1|p)p(c2|p)p(p).\n\t2.a. Does this have to do with Line 149: Q^(q)A^(q) = [Θ^(q), 0] ?\n3. Are there obvious limitations wrt differences in the sample size for x^(1) and x^(2)? Are there any provable biases if the data is highly unbalanced? How does unbalanced data affect the identifiability?\n4. Is the methodology and identifiability theory limited to the two-dimensional case?\n\nLIMITATIONS:\n- A discussion of the asymptotic computational performance is amiss? Specifically, with respect to d_C and the total sample size.\n- The paper does not appear to discuss the applicability of the theorems to the case of q > 2 (i.e., more than 2 modalities).\n- Lacks a discussion of the stability of GAN training, especially with several additional loss augmentations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-23"}