Identifiable Shared Component Analysis of Unpaired Multimodal Mixtures
Authors: Subash Timilsina
Sagar Shrestha
Xiao Fu
Venue: NeurIPS 2024 poster (2024)
Keywords: Identifiability
Multi-modal Learning
Canonical Correlation Analysis
Shared Component Analysis
Independent Component Analysis
Primary area: other

Abstract: A core task in multi-modal learning is to integrate information from multiple feature spaces (e.g., text and audio), offering modality-invariant essential representations of data. Recent research showed that, classical tools such as canonical correlation analysis (CCA) provably identify the shared components up to minor ambiguities, when samples in each modality are generated from a linear mixture of shared and private components.  Such identifiability results were obtained under the condition that the cross-modality samples are aligned/paired according to their shared information. This work takes a step further, investigating shared component identifiability from multi-modal linear mixtures where cross-modality samples are unaligned.  A distribution divergence minimization-based loss is proposed, under which a suite of sufficient conditions ensuring identifiability of the shared components are derived. Our conditions are based on cross-modality distribution discrepancy characterization and density-preserving transform removal, which are much milder than existing studies relying on independent component analysis. More relaxed conditions are also provided via adding reasonable structural constraints, motivated by available side information in various applications. The identifiability claims are thoroughly validated using synthetic and real-world data.

Identifiable Shared Component Analysis of Unpaired Multimodal Mixtures
Subash Timilsina School of EECS
Sagar Shrestha School of EECS
Oregon State University
Oregon State University
Corvallis, OR 97331
Corvallis, OR 97331
timilasis@oregonstate...cl
shressagoorggonsatte.co edu
Xiao Fu School of EECS
Oregon State University Corvallis, OR 97331
xiao wfuorreoonsttte.... edu
Abstract
A core task in multi-modal learning is to integrate information from multiple feature spaces (e.g., text and audio). offering medaalty-invvriian essential repre- sentations of data. Recent research showed that, classical tools such as canonical correlation analysis (CCA) provably identifyy the shared components up to minor ambiguities, when samples in each modality are generated from a linear mixture shared and private components. Such identifiability results were obtained under the condition that the cross- modos-y samples are aligned/paired according to their shared information. This work takes step further, investigating shared component identifiability from multi-modal linear mixtures where cross-modality samples are unaligned. A distribution divergence minimization-basee loss is proposed, under which a suite of sufficient conditions ensuring identifiability of the shared components derived. Our conditions are based on cross modalityyyy distribution discrepancy characterization and are much milder than existing studies relying on independent component analy- sis. More relaxed conditions are also provided via adding reasonable structural constraints, motivated by available side information various applications. The identifiability claims are thoroughly validated using synthetic and real-world data.
transform removal, which
Introduction
The same data entities can often be represented in different feature spaces (e.g., audio, text and image), due to the variety of sensing modalities or domains. Learning common latent components of data from multiple modalities is well-motivated representation learning. The shared components are considered poddality-ivvarial essential representations of data, which can often enhance performance of downstream tasks by shedding modality- speciiii noise [T-4 and avoiding over-fitting (5
prominent theoretical aspect of shared component learning lies in identifiability of the components of interest. The literature posed intriguing theoretical question 2 8 If every modality of data is represented by a linear mixture shared and private components with an unknown mixing system, are the shared components identifiable (up to acceptable ambiguities)? Such component identification problems are often nontrivial due to the ill-posed nature of any linear mixture model (see, e.g., (4)) Interestingly, the work showed that using the classical canonical correlation
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

analysis (CCA) provably find the shared components up to rotation and scaling, In fact, shared component identification from multimoda/mmultviiew linear mixtures were considered in various contexts (see, e.g., [15 18] although some of these works did not model private components. The identifiability results in [T 2] were generalized to nonlinear mixture models as well (4 19). The shared component identification perspective was also related to the success of representation learning in self-supervised learning (SSL) [5
Nonetheless, the treatment in and the related works [5 all assumed that the cross-modality data are aligned (i.e., paired) according to their shared components. In many applications. such as cross- Pross-language information retrieval (20 domain adaptation [23] 25), and biological data translation 26) 27]. aligned gross-modalle data are hard to acquire, if not outright unavailable. A natural question is: When the multimodal linear mixtures are unaligned, can the shared latent components still be provably identified under reasonably mild conditions?
Existing Studies. Theoretical characteristics of unaligned multimodal learning were studied under various settings. The work (28) considered a case where one modality a linear transform of another modality, and showed that the linear transformation is potentially identifiable. The recent work (29) extended this model nonlinear transform setting. However, these works did not consider latent component models-yet the latter are more versatile in many ways, e.g., facilitating one-to-many cross-domain translations [30, 3T.. The work (32) considered unaligned mixtures of shared and private components, but the assumptions (e.g., the availability of a large amount of modalities) to ensure identifiability may not be easy satisfy. The most related work perhaps (8]. But their approach also relied on somewhat stringent assumptions, e.g., that all the latent components are element-wise statistically independent with at most one component being Gaussian. This is because their procedure had to invoke the classical independent component analysis (ICA) [33]
Contributions. In this work, we provide a suite of sufficient conditions under which the shared components can be provably identified from unaligned multimodal linear mixtures up to reasonable ambiguities. The model and identification problem are referred to unaligned shared component analysis (unaligned SCA) in the sequel.
(i) An Identifiable Learning Loss for Unaligned SCA. We propose tackle the unaligned SCA problem by matching the probability distributions of linearly embedded multi-modal data. We show that under reasonable conditions, the linear transformations identifies the shared components up to the same ambiguities as those in the aligned case [12). The conditions are considerably milder compared to the existing unaligned SCA work
(ii) Enhanced Identifiaiiiity via Structural Constraints. We come up with two types of structural constraints, motivated by available side information in applications. to further relax the identifiability conditions. Specifically, we look into cases where the multi-mode data have similar linear mixing systems and cases where a few cross-domain aligned samples available. We show that by adding constraints accordingly, unaligned SCA are identifiable under much milder conditions.
Our contributions primarily lie in identifiability analysis. Nonetheless, we also show the usefulness of our results in real-world applications, namely, cross-lingual word retrieval, genetic information alignment and image data domain adaptation. Particularly, it shows that our succinet multimodal linear mixture model can effectively post-process outputs of pre-trained encoders, e.g., those in to improve data representations and enhance downstream task performance.
34
35
Notation. Notation definitions can be found in Appendix A
2
Background
Generative Model of Interest. Following the classical settings in I modeling the multi-modal data as linear mixtures. More specifically, we adopt the model in [ 2hhat splits the latent representation of data into shared components and private components:
15 16 18
we consider
(4)
(1))())
( = (c) TI 9= 1,2,
(1)
where (44)) E Rd represents the data from the 9th modality, (()) e ad represents the
corresponding latent code, E Rd and p(.) e R stand for the shared components and the private components, respectively. The data x (9))s are assumed to be zero-mean, which can be enforced by

---

centering. Note that the positions of с and Py are not necessarily arranged as 'c (p()) TT (more
generally, z(0) = II(( TTT with an unknown permutation matrix II)) However, the
representation in (I) is without loss of generality as one define A (2) A (9) (T) to reach the representation in For all the domains, we have
c~ Par p ~P p(o))
where Pec and I plui represent the distributions of the shared components and the domain-privane components, respectively. Under ( the two different range spaces range(A (1) for = 2 represent two feature spaces. Then latent p further distinguishes the modalities and often has interesting physical interpretation. For example, some vision literature use c to model "comtent" and p(9) "style" of the images [31 36). In cross-lingual word embedding retrieval (2), c. represents the semantic meaning of the words, while p represents the language-specifs components. The goal of SCA boils down to finding linear operators to recover e to a reasonable extent.
A Aignned SCA: Identifiability of CCA and Extensions. Learning e without knowing (9) is typical component analysis problem. Learning latent components from linear mixture models (LMMs) like a = Az lacks identifiability in general, due to the bilinear nature of the models. This is because one can find an infinite number of invertible matrices B such that AB B z. Then, both (A, x) and (AB. B 2) can fit to the data æ making the problem ill-posed in terms of solution uniqueness; see, e.g.,
(9] (7) and more discussions in Sec. 3 Nonetheless, the works [T 2 studied the identifiability of c under the model ( using the assumption that the cross- cross-modality samples share the same c are aligned. In particular,
formulated the c-identification problem CCA problem:
minimize (Q
E
Q
20)
2()) 2
(3a)
subject to
Q (x(1)) (Q(0))== =I q q= 1,2,
(3b)
where Q(9)) Rdexxd" The expectation in 33a is taken from the joint distribution of the aligned
pairs IP (2(")), where every pair (x()), (2) shares the same c. The formulation aims find Q(9)
such that the transformed representations of the aligned pairs Q ())x((1) and ((2)x((2) are equal. In 1
it was shown that
(1) (q) Oc
under mild conditions (see Appendix E. for details), where (Q is an optimal solution of
the CCA formulation and θ is a certain non-singular matrix. Eq. means that Q (g) space where e lives in, i.c., range( (A
finds the range
Iale) under our notation.
The work in (8] studied the identifiability of under ① when x and (2) are unaligned. Their approach works under the condition that the elements of (n) |c (T are mutually statistically independent. There, 2(9)
Unaligned SCA: Existing Result and Theoretical Gap.
II(9) E(4) 2(1) is assumed to have been estimated by ICA, where (a) and Σ(9) represent the scaling and permutation ambiguities, respectively, which cannot be removed by ICA. The work (8) assumed (4)) I by imposing a unit-variance assumption on all the 2
's. Then, cross-domain matching algorithm is used to match the shared elements in 2()) and ((2), The formulation can be summarized
_!4)'s.
as finding de pairs of non-repetitive (i,j) j) such that e, and e] have identical distributions,
where e, is the ith unit vector. Denote em eT_ 2(1) and Gm TT for m E [dc]. It can be shown that
ke
(q) (())' E [dc],
where & {+1, -1} and is permutation of {1. f1...., dc} (see details in Appendix E.2 summarized
from 8 )) This method effectively applies ICA to each modality, and thus the ICA identifiability
conditions 33) have to met by (1) and (2) individually. However, if one only aims to extract Oc as in CCA, these assumptions appear to be overly stringent.
3

---

c
(00)
pa
Figure I: Scatter plots of matched distribution ()) (left) and ( (right) when follows the Gaussian distribution. Colors in the scatter plot represent align- ment; same color represent the data are aligned.
Figure 2: Illustration of AA() in
Assumption [ in case where de = 2 and dp =
3 Proposed Approach
Unaligned SCA: Problem Formulation We assume that (())s are zero-mean. We use the notation from CCA in (3a). However, since no aligned samples are available, we replace the sample-level matching objective with distribution matching (DM) module, DM can be carried without sample level alignment:
find Q(9) E Rdoxd
= 1,2,2
(6a)
subject
(1)
() x
(6b)
Q (1) E ( (Q()) = 9 1,2.
(6c)
where "u (d)
0" means the distributions u and v are the same.
The formulation in 6 can be realized using various distribution matching tools, e.g., maximum mean discrepancy (MMD) 38) and Wasserstein distance [39]. We use the adversarial loss:
min max Egc log (I)) + Eaten log (1 ((-)(Q))) + Σ R (Q()) (7) (()
The first and second terms comprise the adversarial loss from GAN 40) It finds ((9) confuse the best-possible discriminator f RR + R, where ff is represented by neural network in practice. It is well known that the minimax optimal point of the first two terms is attained when (6b) is met (40). We
I| to "lift" the constraints. This way, the learning criterion in can be readily handled by any off-the-shelf adverserial learning tools.
use RRQQ (4) [[Q(9) (9) ([x(<) (ax()) (Q(())
Identifiability of Unaligned SCA As we saw in Theorem CCA identifies Q(1) ()))) = Oc where @ e dde under the settings of aligned SCA. Establishing similar result for unaligned SCA is much more challenging. First, it is unclear if (6b) could disentangle e from p(9) In general, (1)))(()
could still mixture of and ((9) yet (bb) still holds (e.g., when both and p(9) are Gaussian.)
Second, even when the disentanglement is attained via enforcing (6b) and we have (1)) ( c,
in general it does not hold that (1) (2), This is because θ (1)
( where (1) * (2)
can still be perfectly met (e.g., when P Pe is symmetric Gaussian in Fig. 1.. However, ( A (2)
means that the extracted representations from the two modalities are not matched. This creates challenges for applications like cross-domain information retrieval, language translation, domain adaptation.
Our intuition is as follows: If the two distributions P c.pm) and P c.p()) are very different, then ((9) (9) (0()), 0). We use the following to charac-
(1) (d) Q (2) cannot hold unless Q
terize such difference between the joint distributions:

---

Assumption 1 (Modality Variability). For any two linear subspaces pla) C RRoc+ap
-9 9=112 with
dim(P(a) d p(s) o 0 x Rd4 and linearly independent vectors {y:") (9) E (do+ );m =
1,2, the sets A cOnv(O. y1 (0) ..dc! + +p((), q = 1,2, are such that if p.p(l)) [A >
......ya
for q or q 2, then there exists a k E IR such that the joint distributions P Pep( [HAA * P Pe.p(a) [kAA(2) where kA(1) {ka a E A
The condition in Assumption T is geometric way to characterize the difference between I' c.ppl)) and P -c.p(2)-if the joint distributions have different measures for all possible "stripes", each being a direct sum of subspace and convex hull (see Fig. ( then P Pepp(()) and Pe Pe,p(:: must be very different. Note that the difference is contributed by the modality-specific term p(9), and thus we call this condition "modality variability". Modality variability similar to the "domain variablity" used in [32 both characterize the discrepancy of the joint probabilities IP P.op(1>) and IP p...t.). However, there are key differences: The domain variability was defined in a unified latent domain over arbitrary sets A, which could be stringent. Instead, we use the fact that (6) relies on linear operations to construct A(9), which makes the condition defined over a much smaller class of sets-thereby largely relaxing the requirements. Restricting ( 3 to be stripes also makes the modality variability condition much more relaxed compared to the domain variability condition.
We show the following:
Theorem 1. Under Assumption and the generative model in (T), denote any solution of (6) as Q
q = 2, Then, if the mixing matrices A are full column ranks and E[ce")) is full rank, we have (q) ((a)) =())). addition, assume that either of the following is satisfied:
(a) The individual elements of the content components are statistically independent and non- (d) Gaussian. In addition, G f kc,, Vi * j. v. R and c + -C,, Vi, i.e., the marginal
(d)
distributions of the content elements cannot be matched with each other by mere scaling.
(b) The support of Pe. denoted by C, is a hyper-rectangle, i.e., == - [==-aa,,0 x...x x xI-ade Adel (d)
(d) Further, suppose that F kej, Vi + j,yk E R and cs + -c, Vi.
Then, we have Q (9) x0(0) Oc, i.e., @(4) A for all q 1,2, where f0))
(1) (4)) and either of conditions (a) or (b) is used to make sure θ (2). Note that both (a) and (b) are milder than those in [8] (cf. Theorem 5) where the clement-wise statistical independence of (9) was relied on to find shared representation of (1) and x (2), The proof is in Appendix
In Theorem A Assumption | is used guarantee Q
Numerical Validation. In Fig. 3 the top and bottom rows validate Theorem I [under the assumptions in (a) and (b), respectively. In the top row, we set e E IR², where e₁ is sampled from Gaussian mixtures with three components and C2 is sampled from a Gamma distribution (and JL c2). We set p and p(2) as one-dimensional Laplacian and uniform distributions. In the bottom row, the dimensions of and p(9) for q 1,2 2 are unchanged, but their distributions are replaced in order to satisfy conditions in (b) (see details Appendix F). One can see that clearly @(4) Oc; i.e., the learned el for q = 1,2 2 are identically rotated and scaled versions of c.
remark is that our framework still allows to identify individual c²'s as in
Corollary 1. Under the conditions Theorem 7 (a), Assume that at most one ca for i [de] is Gaussian. Then, the components of c are identifiable up to permutation and scaling ambiguities by applying ICA to e( ((4)x(1 for either 1
2.
The corollary means that to identify individual c,. using our formulation still enjoys much milder conditions relative to [8]. Specifically, condition only specifies the independence among elements of c, but the condition in 8 needs that all the elements in 2 =cT,, (T are independent.
5

---

x(2)
201
X0
-5-30-2 0 S0
-5 -5-50-1
x(1
20
e
-1
10----0--55 8
16-3a-2 0
di Arile
Figure 3: Validation of Theorem
Top row: results under assumption (a). Bottom results under
assumption (b).
Enhanced Identifiability via Structural Constraints
Theorem was well-supported by the synthetic data experiments. However, experiments found that the learning criterion (6) often struggles to produce sensible results in some applications. Our conjecture that the Assumptions in Theorem (a) and (b) might have been satisfied the real data under our tests. Although they are not necessary conditions for identifiability. these conditions do indicate that the requirements guarantee identifiability of unaligned SCA using (6) are nontrivial to meet. In this section, we explore couple of structural constraints arising from side information in applications to remove the need for the relatively stringent assumptions on c.
Homogeneous Domains. The first structural constraint that we consider is ( A for q 1, 2. This model is motivated by the fact that advanced representation learning tools, e.g., self- supervised learning tools (e.g., SimCLR (22 and foundation models (e.g.. CLIP (35)), are already capable of mapping the data clusters to shared linearly separable space space-which which indicates that the representations share a subspace, i.e., (9) ze (9). Under such circumstances, the proposed model and method can be used to further process the data by discarding the private components in the latent representation.
Here, we consider the special case of generative process in where,
Alc (p(9))T]
(8)
Under this model, we look for the shared components by solving (6) with a single Q = Q(1) = (Q) We the following version of the modality variability condition:
Assumption For any linear subspace P C Rdc+dp dr dl) d(2) with dim
dr.
1,2. are such that if P..plal [A f q 1 or q then the P * 0 x Rdv and linearly independent vectors {y, E RRde+dr joc = 1,2, the sets A = conv (0. V1....,
P, joint distributions P Pe.p(1)| [R-A] # P Pe.ptao (k:A) for some k E R.
Theorem 2. Consider the mixture model in (8). Assume that rank( (A) dc + dp and rank(E[cc]) de, and that Assumption holds. Denote Q as any solution of 6 by constraining Q Q (1) = Q (2) Then, we have (x Oc.
One can see that the conditions (a) and (b) in Theorem are completely removed, if the structure A (1) = (2) is imposed. In fact, the result in Theorem 2 is expected and readily seen from the proof of Theorem 1 as the cause for (1) * (2) is the use of two different Q (4) Nonetheless, this simple variation will prove useful in series of real-data experiments.
The Weakly Supervised Case. Another way to add structural constraints is to use available auxiliary information. For example, some datasets have weak annotations and selected pairs; see, e.g., 43 44
Assumption (Weak Supervision). There exist set of available aligned samples (x) x )
(1) (2)
i C such that (g) A (4)
(4)
(4) |cT, ( i.e., (xf ,x!2) share the same Ca.
z z

---

The condition can be added into our formulation in
6 as constraint, i.e.,
Q x
Q
(1) Q x E
(2)
(9)
In the next theorem, we show that the incorporation of aligned samples helps relax conditions (a) and (b) in Theorem DT
Theorem 3. Assume that Assumption is sctisfied, that [C] 2 de paired samples (x) 2 are
available, that A44) for 1, 2 have full column rank, and that Pe is absolutely continuous. Denote (9)
(2) as any optimal solution of (6 under the constraint 9 Then, we have Q x Oc
The proof and synthetic data validation can be found in Appendices D and respectively. Note that ((2) x 2 to the loss in 77 where 3 2 0 is tunable parameter. The overall loss is still differentiable and thus can be easily handled by gradient based approaches.
to realize ( one only needs to add a regularization term Erec. |!Q
remark is that our weakly supervised formulation can use as few as de pairs of (xx xx))))))) to establish identifiability of shared component. In contrast, CCA requires at least de + de + de pairs to attain the same identifiability (cf. Appendix. E.I).
Private Component Identifiability. Although our focus is shared component identification, we show that private components are also identifiable with additional assumptions; see Appendix H
Related Works
Identifiability of Component Analysis under Linear Mixture Models. Various component analysis
models were studied the past several decades, e.g., principal component analysis (45), independent component analysis [33], sparse component analysis (TO] 12]. bounded component analysis (13), simplex component analysis (46] 47). and polytopic component analysis [14]- motivated by their applications in dimensionality reduction, representation learning, and latent variable identification (see, c.g., topic mining [48] 49]. hyperspectral unmixing [46] 47). audio/speech separation (33) and
community detection [50]). The classical component analysis tools mostly study single modality. The identifiability results under these models are well developed and documented.
Identifiability of Shared Components from Aligned Modalities. Modeling multimodal data as two or more linear/nonlinear mixtures latent components was considered in CCA-related works 19]. independent vector analysis (IVA) works [17 18). multiview ICA works [16] 51). and SSL works 15-7.52 Partitioning the latent components into shared and private blocks was considered in [ 4 52 Shared component identifiability was established the block level (see, c.g., 2 and the individual component level (c.g., (5I) in these works. Nonetheless, they all rely completely paired/aligned cross-modality samples, which we do not use in this work.
22,1 15
Distribution Matching and Unaligned Multimodal Analysis. Using distribution matching in unaligned multimodal data analytics for different purpose also has long history; see applications in image- to-image translation (53), domain adaptation [54), cross-plan cross-pllaffom image super-resolution [55). and cross-domain information retrieval (21). The recent works (56) and (57) pointed out the identifiability
challenge and the existence of density-preserviee transforms. The works in [28 29 started studying
the uniqueness issues in distribution matching. However, the latent mixture models were not studied
in this line of work.
Identifiability of Unaligned SCA. The works in 32 investigated the shared component identifiabil- ity when the multimodal data are nonlinear mixtures of content and style (which are shared and private components, respectively) under the same mixing system. Hence, our identical linear mixing case in Theorem
2 can be understood as a special case of theirs. But their analysis relies on the assumption that all the latent components are statistically independent, which is much stronger than our conditions Theorem 2 Their results also require that there are large amount of modalities available. But our proof works for just two modalities. The most related work is (8). which uses the model in (T) in the context of multi-view causal graph learning. As discussed before, their assumptions on the latent components are much stronger than ours (see Corollary I and Appendix E.2
7

---

Table 1: Classification accuracy on the target domain of office-31 dataset (ResNet50 embedding).
source + target
ReaNet
DANN
MDD
MCC
SDAT
ELS
Proposed
A
W
85,2 1 0.2
86,3
1 0.3
86,4 1 0,4
88,3 = 0.3
88.6 = 0,4
87.2 + 0.3
90.4 1 0.4
D
W
97.5 0.1
97.4
+ 0.3
97.7
0.1
96,9 = 0.1
97.6 = 0.1
97.7 + 0.1
97.8 + 0.2
W + D
99.5 +.3
98.7 + 0.2
99.7
+ 0.1
97.4 + 0.2
99.1 + 0.2
99.3 + 0.2
99.5 + 0.3
A +
D
89.4 + 0.2
84.3
+ 0.4
89.9
0.2
87.4 = 0.5
86.3 = 0,4
87.1
+ 0.2
90.1 + 0.3
D 7 A
71.4 0.3
71.7 0.4
70.6 0.3
74,9 = 0.4
72.3=04
71.6
0.3
71.9 + 0.1
W + A
73.1 + 0.2
73.5
+ 0.2
72.3
+ 0.4
73.0 = 0.4
73.6 = 0.3
73.7
+ 0.3
74,6
+ 0.1
Average
86.0
1 0.2
85.3
1 0.3
86.1
1 0.2
86.3 = 0.3
86.2 = 0.3
86.
1 0.2
87.3 1 0.2
6 Numerical Validation
More Synthetic-Data Validation. We first validate our proposed method on synthetic data that follows our model; see Appendix details.
Application (i) Domain Adaptation. We first test the proposed methods over number of domain adaptation (DA) tasks. In DA, we have the source domain data (x(1) } and the target domain (x(2)) J. respectively. Only the source domain data have labels and the two domains are unaligned. We hope to use our method to find shared representations of source and target, and thus the classifier trained using source data can also work well on the target data.
Datasel: We use two standard benchmarks of DA, i.e., Office-3/ (58) and Office-Home (59] The Office-31 dataset has 4652 images and categories from three domains, namely, Amazon images (A), Webcam images (W) and DSLR images (D). The Office-Home dataset contains 15,500 images with 65 object classes from four domains, i.c., Artistic images (Ar), Clip art images (CI), Product images (Pr), and Real-world images (Rw).
Setup: We first test the homogeneous domain model in Sec. 4 The images are pre-processed using ResNet50-baa based image encoder pre- pre-trained over ImageNetIk 442). As mentioned, it was observed that sufe-superviee representation encoders find embeddings that are linearly separable [42), which justifies the of the model (44) Rs Az(9) in the embedding domain. After pre-processing. each image is represented by d(a) 2048 features for q 1.2. We set de 256 for Office-31 and de = 512 for Office-Home. Home. More detailed settings are in Appendix G
Baselines and Training Setup: The baselines are representative DA methods, namely, DANN [25]. MDD (60). MCC [61], SDAT (62), and ELS (33 All the baselines use the same encoder-produced
embeddings inputs; see Appendix G. for their configurations. We also use ResNet encoder's outputs as an extra baseline as learns informative and transferable features from the ImageNet-
IK dataset. We follow the training strategies adopted by the baselines [25] 60 62) to learn classifier jointly with the shared latent components. This strategy arguably regularizes towards more classification- -friiel geometry of the shared features. Therefore append a cross-entropy (CE) based classifier training module to our loss in that learns our feature extractor Q. More details are in Appendix G.
Metric: The evaluation metric is the classification accuracy in the target domain (2)(2). The classifier is trained with the projected source domain (x( and the associated labels.
Resulr: Table and Table 2 show the classification accuracy (mean on Office-31 and Office- Home, respectively. The results are averaged over 5 runs. One can observe that the proposed method offers the best and second best performance in most of the cases. In some tasks (e.g., W"" CI", "Ar and "Rw->CI"), the proposed method outperforms the best-performing base- lines by least 2% accuracy.
More results on the DA task can be found in Appendix
G.1
Application (ii) Single Cell Sequence Analysis. In biomedical research, it is desired fuse measurements from multiple sensorial modalities of the same cells, in order to have better characteri- zations of the cells. However, obtaining multimodal data of the same cells simultaneously is almost impossible, due to the sensing limitations. Therefore, many methods are proposed in the literature for
aligning unpaired multi-modal single cell data (27 64 65] We focus on the following two modalities of single-cell data (66) (1) the RNA sequences o and (2) the ATAC sequences (x))
8

---

Table Classification accuracy on the target domain of office- Home dataset (ResNet55 embedding).
source + target
Realiet
DANK
MDD
MCC
SDAT
ELS
Proposed
Ar C1
42.0 = 0.2
46.7 1 0.2
47.4
= 0.3
44.4
0.3
47.3 1 0.4
48.5 i 0.2
51.0 1 0.3
Ar + Pr
69.2 = 0.1
70.2 + 0.4
72.8 = 0.4
72.4 0.2
71.1
0.3
71.0 + 0.3
75,8 0.1
Ar + Rw
80.2 + 0.3
81.2 0.4
81.2 + 0.1
80.3
0.3
80.5 +.1
80.8 + 0.4
82.5 + +
CI + Ar
60.7 = 0.4
40.
6.14=011
0.4
5.2
59.8 0.1
62.7 0.4
CI + Pr
71.0 0.1
69.8 0.3
7..0
7.41
6..5
72.5 6..5
CI + Rw
74.8 = 0.2
7.1
7..1
0.2
70.7
7.17
75.8 0.1
Pr - Ar
60.6 = 0.2
62.2 0.1
6.1
0.1
62.5
60.9
0.3
Pr - CI
44.8 = 0.1
48.8 0.1
0.3
4..
0.3
0.3
50.4 0.1
Pr + Rw
79.6 - 0.1
80.3 + 0.4
79.6
8.2
8..0
79.2 0.1
81.7 022
Rw + Ar
70.1 + 0.2
71.5
714
0.2
71.3 7..6
726
Rw CI
45.8 = 0.2
50.9 0.2
50.3 0.1
5.2
5..4
50.7 +
53.2 0.1
Rw Pr
80.7 + O.1
80.6 +
81.1 0.1
8..2
80.7
0.1
0.3 79.8
82.9
Average
64.9 O.1
66.3 + 0.2
66.8
0.2
65.7 6.22
0.2
65.9 0.2
68.7
Table 3: Average precision P@] of cross-language in- formation retrieval.
0.80
Propoee Ro paitd
Prozeend palind parto
Pspooet puared
source Adv
NN
proposed
NN
Adv CSLS
propozed CSLS
0.60
CB 0n--05 61.3
66.4
70.2
74.9
K-NN accuracy
es-ten
0.50
en -1
48.2
54.4
601.8
67.7
it-
51.9
63.8
66.0
ff
63.6
60.2
72.6
73.7
Ir->en
55.4
58.4
64.1
71.4
-dde
51.4
36.7
59.3
67.6
de--en
42.5
$7.0
51.0
59.3
32.7
34,9
38.6
41.4
ru->e
27.6
41.6
35.0
50.8
CD CB-Ar
12.6
22.7
16.7
29.1
0.0
15.7
26.9
20.1
35.6
cm -v
10.4
7.7
22.8
2.7
17.3
4.4
33,0
12
Average
37.6
44.3
43.1
54,9
K the KNN evaluation metric
Figure 4: k-NN accuracy for single-cell sequence alignment.
Dataser: We use human lung adenocarcinoma A549 cells data from [66] The dataset contains 1,874 samples RNA sequences (ax()) and ATAC sequences (a(2)). Each data set is split into 1534 training samples and 340 testing samples as in [27). 'he data have labeled associations between the two domains-part of which will be used to test our weakly supervised formulation. For this experiment, features of RNA sequence and the ATAC sequence have dimensions of (11) 815 and
(22) 2613, respectively. We set de 256. We our weakly supervised formulation shown in We uniformly sampled a set of indices from the training set to serve as L.
Baseline and Metric: We use weakly supervised algorithm, namely, cross-modal autoencoder (CM- AE) work in (27). as a baseline, which also learns the shared representation between unaligned RNA and ATAC sequences. We use the K K-nearest neighbor (k-NN) accuracy to evaluate the performance as suggested in [27].
Resulr:
The plot in Fig. shows the h-NN accuracy of the methods on the test set. Results show the mean and standard deviation over 10 runs, each having different random initialization. For the proposed method, we vary the number of available paired samples from 0 (cf. Theorem I to de 256 (cf. Theorem 3 Note that the baseline uses more (i.e., 256 and 770) paired samples.
also needs additional class labels, i.e., y/ for the ith sample (q) Here, represents the number of hours (0, 1 or of cell treatment (27] The proposed method without any supervision (i.e., paired samples) already exhibits around 3 times greater I--N accuracy compared to the baseline for all k. Moreover, including just one paired sample boosts the k-NN accuracy the proposed method to around 5 times higher than the baseline for all h. Finally, one can observe steadily increasing k-NN accuracy with respect to the number of available paired samples. This corroborates with our Theorem

---

Application (iii) MMlti-lingur Information Retrieval. We also evaluated our method on a word embedding association problem from the natural language processing literature [20, This task aims to associate high-dimensional word embeddings across different languages according to their semantic meaning. The word embeddings two languages are represented using two sets of vectors, i.e. {-; H/ and { ( 1==1- The postulate is that if z!"" and 2
(2) have the same meaning (e.g.. both representing "cat") two languages (e.g., English and German), they should share latent components c.
Dataset: We use the word embeddings from the MUSE dataset (https://gi com/ facebookreseercch/BBSS (21). These monolingual word embedding are generated using fast- Text (67) and has dimensions of d = 300 for = 1, The training dataset include 200,000 word set de 256. We follow the generative model under ( and run the formulation in 7 to learn the linear transformation Q.
embeddings in each language. In our experiment
Baseline: We use Adv 21 the baseline which also uses distribution matching between two language domains. Unlike our method, Adv does not use linear mixture models.
Metric: We follow 21) to use the average precision score calculated based on nearest neighbor (NN) and cross domain similarity local scaling (CSLS). Precision k ("k precision") is computed by the number of times that one of the correct translations of source word is retrieved at top-k results (k =={[ 5, 10)). The final score is normalized to be in the range of 0 to 100, with 100 being the highest score indicating the best performance. To evaluate the performance, we use the same test data as in (21). For each source and target language pair, this dataset includes 1,500 source word embeddings. The source embeddings are used to retrieve corresponding embeddings from pool of 200,000 target word embeddings.
Result: Table 3 reports the P@1 1 scores over the test data calculated for each source and target language pair. The languages are denoted as en English, Spanish, it Italian, fr French, de -Germnny, ru- Russian, ar - Arabic and vi - Vietnamese, One can observe that the proposed method exhibits a better precision performance than that of Adv in most of the translation tasks. In particular, the proposed method significantly outperforms the baseline on the tasks en->ar, ar- en en-vvi and vi vi- showing at least 10% precision gains. Similarly, our method shows least 5% improvements in both NN and CSLS based precision metrics in en >s and es es->en tasks.
More details and additional experiments can be found in Appendix G.3
7 Conclusion
In this work, we considered the problem of identifying shared components from unaligned multi- domain mixtures. We proposed a learning loss that matches the distributions of linearly transformed data. Based on this loss, we came up with suite of sufficient conditions to ensure the identifiability of shared components. Furthermore, we proposed modified models and losses that enjoy more relaxed conditions for shared component identifiability. This was achieved via introducing structural con- straints, namely, the homogeneity of the mixing systems and the existence of weak supervision. Our theoretical claims were validated with both synthetic and real-world data, demonstrating soundness of the theorems and usefulness of the models/algorithms.
Limitations. First, our conditions for shared component identification are sufficient. The necessary conditions are not underpinned, but necessary conditions assist understanding the limitations of the models and algorithms. Second, our methods were developed under the linear mixture model, which has limited expressiveness, and thus often requires pre-processing to approximately meet the model specification. We expect that results with similar flavors to be derived for nonlinear models in the future. Third, the results were derived under an unlimited data assumption. would be interesting have finite sample analysis. Finally, optimizing GAN-based losses is sensitive to hyperparameter settings. Back-propagation based minimax optimization occasionally fails to converge. More optimization-friendly losses and more stable algorithms are desirable in the context of distribution matching.
10

---

Acknowledgment
This work is supported part by the Army Research Office (ARO) under Project ARO W91 INF-1- 1-0227, and in part by the National Science Foundation (NSF) CAREER Award ECCS-2144882
References
[1] M. S. Ibrahim, A. Zamzam, A. Konar, and D. Sidiropoulos, "Cell-edge detection via selective cooperation and generalized canonical correlation," JEEE Transuctions on Wireless Communications, vol. 20, no. 11, pp. 7431-7444, 2021.
[2] M. Sørensen, C. Kanatsoulis, and N. D. Sidiropoulos, "Generalized canonical correlation analysis: A subspace intersection approach," IEEE Transactions on Signal Processing, vol. 69, pp. 2452-2467, 2021.
[3] Rastogi, B. Van Durme, and R. Arora, "Multiview LSA: Representation learning via gen- eralized CCA," in Proc. Conference of the North American chapter of the Association for Computational Linguistics: human language technologies, 2015, pp. 556-566.
[4] Q. Lyu and X. Fu, "Nonlinear multiview analysis: Identifiability and neural network-assisted implementation." " IEEE Transactions on Signal Processing, vol. 68, pp. 2697-2712, 2020.
[5] Von Kügelgen, Sharma, I., Gresele, W. Brendel, B. Schölkopf, M. Besserve, and E. Lo- catello, "Self-supervised learning with data augmentations provably isolates content from style," " in Advances in Neural Information Processing Systems (NeurIPS), vol. 34, 2021, pp. 16451-1648
[6] Q. Lyu, X. Fu, W. Wang, and S. Lu, "Understanding latent correlation-based multiview learning and supervuup: An identifiability perspective," in Proc. International Conference on Learning Representations (ICLR), 2022.
[7] I. Daunhawer, A. Bizeul, E. Palumbo, A. Marx, and J. E. Vogt, "Identifiability results for multi- modal contrastive learning," in Proc. International Conference on Learning Representations (ICLR), 2022.
[8] N. Sturma, C. Squires, M. Drton, and C. Uhler, "Unpaired multi-domain causal representation learning," in Advances in Neural Information Processing Systems (NeurIPS), vol. 34, 2021.
[9] P. Comon and C. Jutten, Handbook of Blind Source Separation: Independent component analysis and applications. Academic press, 2010.
[10] Hu and K. Huang, "Global identifiability fı-based dictionary learning via matrix volume optimization," in Advances in Neural Information Processing Systems (NeurIPS), vol. 36, 2024.
[11] N. Gillis, "The why and how of nonnegative matrix factorization," Regularization, optimization, kernels, and support vector machines, vol. 12, no. 257, pp. 257-291, 2014.
[12] J. Sun, Q. Qu, and J. Wright, "Complete dictionary recovery over the sphere i: Overview and the geometric picture," IEEE Transactions on Information Theory, vol. 63, no. 2, pp. 853-884, 2016.
[13] A. T. Erdogan, "A class of bounded component analysis algorithms for the separation of both independent and dependent sources," IEEE Transactions on Signal Processing, vol. 61, no. 22, pp. pp30--55 2013.
[14] G. Tatli and A. T. Erdogan, "Polytopic matrix factorization: Determinant maximization based criterion and identifiability." IEEE Transactions on Signal Processing, vol. 69, pp. 5431-5447, 2021.
[15] F. R. Bach and M. I. Jordan, "A probabilistic interpretation of canonical correlation analysis," 2005.
[16] H. Richard, P. Ablin, B. Thirion, A. Gramfort, and A. Hyvarinen, "Shared independent compo- nent analysis for multi-subject neuroimaging." in Advances in Neural Information Processing Systems (NeurIPS), vol. 34, 2021, pp. 29 972
[17] Y.-O. Li, T. Adali, W. Wang, and V. D. Calhoun, "Joint blind source separation by multiset canonical correlation analysis," IEEE Trunsactions on Signal Processing, vol. 57, no. 10, pp. 3918-3929, 2009.
11

---

[18] M. Anderson, G.-S. Fu, R. Phlypo. and T. Adal, "Independent vector analysis: Identification conditions and performance bounds," IEEE Transactions on Signal Processing, vol. 62, no. 17. pp. 4310- 2014.
[19] P. A. Karakasis and N. D. Sidiropoulos, "Revisiting deep generalized canonical correlation analysis," IEFE Transactions on Signal Processing, vol. pp.
pp. 4392-4406, 2023.
[20] G. Lample, A. Conneau, L. Denoyer, and M. Ranzato, "Unsupervised machine translation using monolingual corpora only," in Proc. International Conference on Learning Representations
(ICLRR), 2018.
[21] G. Lample, A. Conneau, M. Ranzato, L. Denoyer, and H. Jégou, "Word translation without parallel data," in Proc. International Conference on Learning Representations (ICLR), 2018.
[22] E. Grave, A. Joulin, and Q. Berthet, "Unsupervised alignment of embeddings with wasserstein procrustes," in Proc. International Conference on Artificial Intelligence and Statisties (AISTATS). PMLR, 2019, pp. 1880-1890.
[23] S. Ben--avvd, J. Blitzer, K. Crammer, and F. Pereira, "Analysis of representations for domain adaptation," " in Advances in Neural Information Processing Systems (NeurIPS), vol. 19, 2006.
[24] M. Long, Y. Cao, J. Wang, and M. Jordan, "Learning transferable features with deep adaptation networks," in Proc. International Conference on Machine Learning (ICML). PMLR, 2015, pp. 97-105.
[25] Y. Ganin and V. Lempitsky, "Unsupervised domain adaptation by backpropagation." in Proc. International Conference on Machine Learning (ICML). PMLR, 2015, pp. 1180-1189.
[26] S. Waaijenborg, P. C. Verselewel de Witt Hamer, and A. H. Zwinderman, "Quantifying the association between gene expressions and dna-markers by penalized canonical correlation analysis." Statistical applications in genetics and molecular biology, vol. 7, no. 1, 2008.
[27] K. D. Yang, A. Belyaeva, S. Venkatachalapathy, K. Damodaran, A. Katcoff, A. Radhakrishnan, G. Shivashankar, and C. Uhler, "Multi-domain translation between single-cell imaging and sequencing data using autoencoders," Nature communications,, vol. 12, no. 1, 31, 2021.
[28] I. Gulrajani and T. Hashimoto, "Identifiability conditions for domain adaptation," in Proc. International Conference on Machine Learning (ICML), 2022, pp. 7982-7997.
[29] S. Shrestha and X. Fu, "Towards identifiable unsupervised domain translation: A diversified distribution matching approach." " in Proc. International Conference on Learning Representations (ICLR), 2024.
[30] Y. Choi, Y. Uh, J. Yoo, and J.-W. Ha, "StarGAN v2: Diverse image synthesis for multiple domains," in Proc. IEEE Conference Computer Vision and Pattern Recognition (CVPR). 2020, pp. 8188-8197.
[31] X. Huang. M.-Y. Liu, S. Belongie, and J. Kautz, "Multimodal unsupervised mmage-to-image translation," in Proc. European Conference on Computer Vision (ECCV), 2018, pp. 172-189.
[32] S. Xie, L. Kong, M. Gong, and K. Zhang, "Multi-domain image generation and translation with identifiability guarantees," in Proc. International Conference on Learning Representations (ICLR), 2023.
[33] Comon, "Independent component analysis, new concept?" Signal processing, vol. 36, no. 3, pp. 287-314, 1994.
[34] K. He, x. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," " in Proc. IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770-778.
[35] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark et al., "Learning transferable visual models from natural language super- vision," in Proc. International Conference on Machine Learning (ICMI.. PMIR, 2021, pp. 9748-8763.
[36] Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo, "StarGAN: Unified generative adversarial networks for multi-domain image-to-image translation." " in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 8789-8797.
[37] X. Fu, K. Huang, N. D. Sidiropoulos, and W.-K. Ma, "Nonnegative matrix factorization for signal and data analytics: Identifiability. algorithms, and applications," IEEE Signal Processing Magazine, vol. 36, no. 2. pp. 59-80- 2019.
12

---

[38] A. Gretton, K. M. Borgwardt, M. Rasch, B. Schölkopf, and A. Smola, "A kemne two-se test," The Journal of Machine Learning Research, vol. 13, no. 1, pp. 723-773, 2012.
[39] C. Villani et al., Optimal transport: old and new. Springer, 2009, vol. 338.
[40] I. Goodfellow, J. Gouget-Abadie, M. Mirza, Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial networks," in Advances in Neural Information Processing Systems (NeurIPS), 2014.
[41] L.. Kong, S. Xie, W. Yao, Y. Zheng, G. Chen, Stojanov, Akinwande, and K. Zhang, "Partial disentanglement for domain adaptation." in Proc. International Conference on Machine Learning (ICML), 2022, pp. 11 455-11472 472.
[42] Chen, S. Kornllith, M. Norouzi, and G. Hinton, "A simple framework for contrastive learning of visual representations," in Proc. International Conference on Machine Learning (ICML). PMLR, 2020, pp. 1597-1607.
[43] O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra et al., "Matching networks for one shot learning," in Advances in Neural Information Processing Systems (NeurIPS), vol. 29, 2016.
[44] E. Triantafillou, T. Zhu, V. Dumoulin, P. Lamblin, U. Evci, K. Xu, R. Goroshin, C. Gelada, K. Swersky, P.-A. Manzagol et al., "Meta-dataset: dataset of datasets for learning to learn from few examples." in Proc. International Conference on Learning Representations (ICLR), 2019.
[45] S. Wold, K. Esbensen, and P. Geladi, "Principal component analysis," Chemometrics and intelligent laboratory systems, vol. 2, no. 1-3, pp. 37-52, 1987.
[46] J. Li and J. M. Bioucas- Dias, "Minimum volume simplex analysis: A fast algorithm to unmix hyperspectral data," in IEEE International Geoscience and Remote Sensing Symposium, vol. 3, 2008, pp. I11-
[47] X. Fu, K. Huang, B. Yang, W.-K. Ma, and N. D. Sidiropoulos, "Robust volume minimization- based matrix factorization for remote sensing and document clustering," IEEE Transactions on Signal Processing, vol. 64, no. 23, pp. 6254 6268, 2016.
[48] S. Arora, R. Ge, Y. Halpern, D. Mimno, A. Moitra, D. Sontag, Y. Wu, and M. Zhu, "A practical algorithm for topic modeling with provable guarantees," in Proc. International Conference on Machine Learning (ICML). PMLR, 2013, pp. 280-288.
[49] K. Huang, X. Fu, and N. D. Sidiropoulos, "Anchor-free correlated topic modeling: Identifiability and algorithm," Advances in Neural Information Processing Systems, vol. 29, 2016.
[50] X. Mao, P. Sarkar, and D. Chakrabarti, "Estimating mixed memberships with sharp eigenvector deviations," Journal of the American Statistical Association, vol. 1 116, no. 536, pp. 1928-1940, 2021.
[51] L. Gresele, P. K. Rubenstein, A. Mehrjou, F. Locatello, and B. Schölkopf, "The incomplete Rosetta stone problem: Identifiability results for multi-view nonlinear ICA," in Proc. Uncer- tainty in Artificial Intelligence, 2020, pp. 217-227.
[52] C. Eastwood, J. von Kügelgen, L. Ericsson, D. Bouchacourt, P. Vincent, M. Ibrahim, and B. Schölkopf, "Self-supervised disentanglement by leveraging structure in data augmentations." in Causal Representation Learning Workshop at NeurIPS 2023, 2023,
[53] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, "Unpaired mnage-to-image translation using cycle- consistent adversarial networks." " in Proc. the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 2223-22-222222 22322.
[54] M. Long, H. Zhu, J. Wang, and M. I. Jordan, "Deep transfer learning with joint adaptation networks." " in Proc. International Conference on Machine Learning (ICML). PMLR, 2017, pp-
2208 2217.
[55] W. Wang, H. Zhang, Z. Yuan, and C. Wang, "Unsupervised real-world super-resolution: A domain adaptation perspective," in Proc. IEEE International Conference on Computer Vision (ICCV), 2021, pp. 4318-4327.
[56] T. Galanti, L. Wolf, and S. Benaim, "The role of minimal complexity functions in unsupervised learning of semantic mappings," in Proc. International Conference on Learning Representations (ICLR), 2018
13

---

[57] N. Moriakov, J. Adler, J.d Teuwen, "Kernel of CycleGAN as principle homogeneous space." in Proc. International Conference on Learning Representations (ICLR). 2020.
[58] K. Saenko, B. Kulis, M. Fritz, and T. Darrell, "Adapting visual category models new domains," in Proc. European Conference on Computer Vision (ECCV). Springer, 2010, pp. 213-226.
[59] H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan, "Deep hashing network for unsupervised domain adaptation," in Proc. IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2017, pp. 5018-5022
[60] Y. Zhang, T. Liu, M. Long, and M. Jordan, "Bridging theory and algorithm for domain adapta- tion," in Proc. International Conference on Machine Learning (ICML), 2019.
[61] Y. Jin, X. Wang, M. Long, and J. Wang, "Minimum class confusion for versatile domain adaptation," in Proc. IEEE International Conference on Computer Vision (ICCV). Springer, 2020, pp. 464-480.
[62] H. Rangwani, S. K. Aithal, M. Mishra, A. Jain, and V. B. Radhakrishnan, "A closer look at smoothness in domain adversarial training." in Proc. International Conference Machine Learning (ICML). PMLR, 2022, pp. 18 378-18 399.
[63] Y. Zhang. J. Liang, Z.. Zhang, L.. Wang, R. Jin, T. Tan et al., "Free lunch for domain adversar- ial training: Environment label smoothing," in Proc. International Conference on Learning Representations (ICLR), 2023.
[64] L. Eyring, D. Klein, Uscidda, G. Palla. N. Kilbertus, Z. Akata, and F. Theis, "Unbalancedness in neural monge maps improves unpaired domain translation," " arXiv preprint arXiv:2311.15100 2023.
[65] M. Amodio and S. Krishnaswamy, "MAGAN: Aligning biological manifolds," in Proc. Interna- tional Conference on Machine Learning (ICML). PMLR, 2018, pp. 215-223.
[66] J. Cao, D. A. Cusanovich, V. Ramani, D. Aghamirzaie, H. A. Pliner, A. J. Hill, R. M. Daza, J. L. McFaline-Figueree J. S. Packer, L. Christiansen et al., "Joint profiling of chromatin accessibility and gene expression in thousands of single cells," " Science, vol. 361, no. 6409, pp. 1380-1388 2018.
[67] A. Joulin, E. Grave, P. Bojanowski, M. Douze, H. Jégou, and T. Mikolov, "Fasttext.zip: Com- pressing text classification models," arXiv preprint arXiv:1612.03661. 2016.
[68] K. B. Athreya and S. N. Lahiri, Measure theory and probability theory. Springer, 2006, vol. 19,
[69] J.-F. Cardoso, "Super-symmetric decomposition of the fourth-order cumulant tensor. blind identification of more sources than sensors." in Proc. International Conference on Acoustics, Speech, and Signal Processing (ICASSP), vol. 91, 1991, pp. 3109-3112.
[70] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," in Proc. ICLR, 2015. [71] A. L. Maas, A. Y. Hannun, and A. Y. Ng, "Rectifier nonlinearities improve neural network acoustic models," in Proc. International Conference on Machine Learning (ICML), vol. 30, no. 1, 2013, 3.
[72] K. Ghasedi Dizaji, A. Herandi, C. Deng, W. Cai, and H. Huang, "Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization." in Proc. the IEEE International Conference on Computer Vision (ICCV), 2017, pp. 5736-5745.
[73] Xie, R. Girshick, and A. Farhadi, "Unsupervised deep embedding for clustering amalysis," in Proc. International Conference on Machine Learning (ICML). PMLR, 2016, pp. 478-487.
[74] W. Wang, X. Yan, H. Lee, and K. Livescu, "Deep variational canonical correlation analysis," arXiv preprint arXiv: 1e.000000..... 03454, 2016
[75] A. Gretton, K. Fukumizu, C. Teo, L. Song, B. Schölkopf, and A. Smola, "A kernel statistical test of independence," in Advances in Neural Information Processing Systems (NeurIPS), vol. 20, 2007.
[76] T. M. Cover, Elements of information theory. John Wiley & Sons, 1999.
14

---

Supplementary Material of "Identifiable Shared Component Analysis of Unpaired Multimodal Mixtures"
A
Notation
The notations used throughout the paper are summarized in the Table 4
Table 4: Definition of notations.
Notation
Definition
T, σ,
scalar, vector and matrix
x(1)
variable from q-th domain
T3 3y
both represents i-th element of vector a
X kg
represents the element of i-th row and j-th column of matrix X transpose of and X
X
X'
represents the cardinality of set X'
Null(X*
represents the null space matrix returns the convex hull of the given set
conv()
di)
denotes the dimension of subspace X
KA
(ka a E A, k R { x x } { x E X. y y y preimage of X:: {2r Ar E X)
+ X
X' + Y
Aprelmg(d))
[N]
set of whole numbers up N; NJ.
identity matrix
Px
probability distribution of random variable joint probability distribution of random variable and y expectation
P a
(d)
y
x and y random vectors have the same distribution
(d)
x * y
x and y random vectors have different distributions x and y random vectors are statistically independent represents continuous interval between a and normal distribution with mean l and variance o² uniform distribution with interval and gamma distribution with the shape parameter a and scale parameter 0
x IL y
(a, b]
N N(,,o"
Uniform aa,
Gamma a, 0)
Laplace (1,, b)
Laplace distribution with location μ and diversity or scale parameter
VonMisea(( k)
von Mises distribution with location y and concentration parameter.
Beta (1, 3)
beta distribution with the shape parameters a and B
15

---

B Proof of Theorem I
We restate the theorem here:
Theorem
[ Unnder Assumption | and the generative model in denote any solution of ( as
(4)
1.2. Then, if the mixing matrices are full column ranks and E(ec"]) ) is full rank,
we have Q ()) x (0) = ()) c. In addition, assume that either of the following is satisfied:
(a) The individual elements of the content components statistically independent and
(d)
(d)
Gaussian.. In addition, 6 * kej, + j. vk E and c * Ca: Vi, i.e., the marginal distributions of the content elements cannot be matched with each other by mere scaling.
(b) The support C is hyper-rectangle, i...
[-a1, a1] x... ade adel Further, (d)
(d) suppose that C * kej, Vi + v. E R and & # -q, Vi.
Then, we have Q (9) x = θc, i.e., e + for all 1, 2, where (())
We will prove the theorem in following two steps. For the first step we will prove x<(1) @ and for second step we will employ either assumption (a) or (b) to prove that (
O Aq 1.2.
B.1
Linearly transformed content identification
Let us define
(9 Q(1) (()) Rox((do
We want to show that
Null(H
R4
(10)
since this will imply that that H does not depend upon the style component. Combined with the fact that rank( (1) de, this will imply that H an invertible function the content component. To that end, consider the following line of arguments.
Since the objective in 6 matches the distribution for latent random variables z(1) Q (1) and (22) ((2) the following holds for any R. C Rdc, Vk R.
Pari [R a [RR,
Preiig (R)))) = P 20 (2) Prelmg (kR))
(11)
(1) [K (1) PreImg (Re) (e) K (2)
Pme (R.)),
where, (9) PreIing (R.) = (2(4)
(9)) (44) E Rof is the pre-image of (4) (a) follows because
Pace) [RR]] = P 2 [kR[] = P Po(a H (g) a linear operator.
Prelng (kR.)) [68 Section 2.2]. (b) follows because H (g)
Although II] holds for any Re, we will that is sufficient to consider special R2 to prove (0)
To that end, take R2 = conv {0, a ai...., J. where a; E (Rde such that Pate [Ro] >> 0. Let us take
y E ta such that H(1) y,
(g)
(q) a, For reasons that will be clear later, we hope to show that
H (g) Protmg (Re) conv v (g)
yde Nu)l(H
(
To that end, observe that for any r E Re, we can represent r as,
1 de
dc Wiag for some {wi s.t.
Wi < 1, Vi.
16

---

For both view q = 1, we get,
dc
HiHh 91) y.
W
dc +
de
H(4)
wiy (4)
r
dc +
==1
de
(0) H
Proimg
de +
w:
dc
ways (1)
+ Null ()))(H())
(12)
iml
We can write,
H
9 Prelmg (Re) cO.v (4)
dd
(4)
Null(H(
(13)
We have that Null(H ()) C. Rdotd is linear subspace with (im(Nul((HH)) = (1)) Let Aet = H H(9) Prelin (Re). Note that Pacs) [k = PP Paco) (1)) AA: (from (1 and P P2(0) (1)) [A(i)] (by the construction of R.). Further, the set (l is of the form
(4) conv y ..... . de + p(.),
because Null(HH)) a subspace of dimension de hence satisfies the definition of p((), Hence, Assumption immlies that
Null(H (1) Rd
Denoting the Nth to Mth columns of 4 by 4) (N M), the above equivalent to saying
(9) (de + 1: dc + (3) = 0.
(14)
Denote,
( = H(9 dc) 4 = 1,2.
Then, can write,
Q g (9) e, 44= 1,2.
(15)
Next, use Assumption (a) or (b) to show that ( ())) O To that end, note that the distribution matching constraint implies that
e
(d) ( (1)
(2)
Hence M (0)1))-1((2) is an invertible matrix such that c (d) Mc. However, in the following,
we will show that if either Assumption (a) or (b) is satisfied, then M = I, and thus ( = (2))
B.2 Considering Assumption (a)
We want to show that when Assumption (a) is satisfied, Mc (d) c for any invertible M, then
M = I.
C1
Note that Me- = [m] Mdel
By Assumption (a), we have that the components of content
Các_
statistically independent G; IIL Cj. j non-Gaussian, and has non-zero kurtosis. Then, according to cumulant multilinearity and additivity properties, the fourth order cumulant tensor Cun(Me) of Me has the following unique decomposition (69).
dc
Cum(Mo Σ
k;m; m. o m, m,
(16)
17

---

where is the outer product, Kę is the kurtosis of component C₁ and m,, i E [de] are the columns of M.
Since Mc (d)
C, the following should hold
Cum(Me) Cum(c) Cum
(17)
de
de
Ka Mi m. m. m.
Rd ed o 6d ea ed:
(18)
e; is the ith column of identity matrix I.
Because of statistical independence of components of c, the CP-dccomposition Cum Cum(Me) = Cum(Ic) is unique [69] upto permutation and scaling ambiguities, i.e., M should be permutation scaling matrix.
Let M = IE where, II E Rdexde is permutation matrix and Σ = Diag(ri..... .Tde) E Rdexde a diagonal scaling matrix.
Finally, since 6 _ key, Vi V R. M has to be identity matrix. To the reason, for the sake of contradiction, suppose that either (i) there exist i.j E [dc] x [dc] and & E R, with o j such that [Mc] kej. or (ii) di E [dc] such that [Mc] kc; for some & R, k 1. (d) c, [Mc],
(d)
For case (i), since Mc
C Vi. Hence,
[Mc]: kcj
(d)
[Mc]:
kej
(d)
koj,
(d)
which is a contradiction to the assumption 6s + kej.
(d) kcj. First, & # =1, cannot hold because it will mean
For case (ii). [Mc]: = ke, implies that c2
that var (c;) 222 var(c;) which cannot hold for k * +1 since var(e,) O. Hence, the only possible (d)
option is k = -1 1, which is already ruled out by the assumption that * -cs- Hence M is an identity matrix. This concludes the proof.
B.3 Considering Assumption (b)
Let
e, [0, 0,
0, Rdc
denote the standard basis vector in Rdd, Let vertex of hyper-rectangle v; = a,e; = Ae;, where
= Diag((at,..... Ide)
where Diag(-) represents the diagonal matrix formed by the given vector.
If Mc (d) c, then the supports of Mc and should match, i.e..
M(C) = C,
where (C) {Mcl (ieccee) CE
Note that Vc C, the set points v, satisfy the following property
de
0
for some 1 a a 51
(19)
- Mc= Σ (Mv))
18

---

Since the support of Mc is C, this implies that Vc E C,
0 (Mu));
for some 1 0; 3
The last equation implies that the set of points Mv, Vi E [dc] also satisfy property (9). Hence, for each E [dc]. Mv. łu; for some unique j E [de]. Note that j should be unique for each because M is invertible, hence M cannot map two orthogonal vectors and Ug. i * k, to the same vector Łe, with same or different signs.
Let V U 01.... Udol". Then one can write
MV = VEN.
where Σ is some diagonal matrix with diagonal entries from (+1,-1] and IT is permutation matrix. Then the above implies
MAI AIEI
M ΑΣПA!
Hence M is a permutation and scaling matrix.
Finally, by the argument presented in last paragraph of Sec. B.2 (i.e., proof with Assumption (a)), we conclude that M is an identity matrix.
C Proof of Theorem
We restate the theorem here:
Theorem Consider the mixture model (8) Assume that rank(A) dc + dp and
rank(E[cc' ]) = dc, and that Assumption 2 holds. Denote Q as any solution of (6) by constrain-
ing Q Q(1) Q Then, we have Qx!4) Oc.
One can follow the same argument as in the step of proof in
Let us define
-QAE (dex
We want show that
Null 0 Rdr
(20)
since this will imply that that H does not depend upon the style component. Combined with the fact that rank(H) = de, this will imply that is an invertible function of the content component. To that end, consider the following line of arguments.
Since the objective in (6 matches the distribution for latent random variables ((1) = Qx((1) and
2 = (2) the following holds for any R, C [Rde,. & E R
Paus [kR]]] Parss [kRR,,
(1 H Proting (RR))) 22(0)] H ProTin (kR.))
(21)
(1) 1k Pretme [R,) =P P2(e)) [KH Proing (Ro)],
(22)
where, H Prellmg (Re) (z) Hz R. is the pre-image of H. (a) follows because ia [kRR,]
F F [R] P Pain [H Preling (kR))) 688, Section 2.2]. (b) follows because H a linear operation.
Although (2 holds for any Re, we will see that is sufficient to consider a special R, to prove (20).
To that end, take R. conv (O, a1...., ade).
). where a, Rdo such that Pat [R]] > 0. Let us take y: Rdo+dr, such that Hy; a. For reasons that will be clear later, hope show that
H Prelnn (R.) conv (0, y
yao } + Null(H).
19

---

To that end, observe that for any Re, we can represent as,
de w for some (w;) s.t. Σ
wi S Vi.
de
For both view -= 1, we get,
dc
W;
y
dc +
de
r
W:Y,
dc
+1 2=1
de dc +1
dc
H PreImg
Wia;
=
w,y; + Null(H)
(23)
dc +
1=1
1=1
We can write,
Preimg (R.)) = conv conv(0, vo....... yack + Null(H)
(24)
We have that Null(H) с +do+ H Preimg (Re). Note that 2200 [kA] = Pa [kA]], Ak E IR (from (1.). and (2 [A] > 0 (by the construction of R.). Further, the set A is of the form
linear subspace with dim(Null(H)) dp. Let A
conv y1 V....... Vdc) +P,
because Null H) is subspace of dimension dr, hence it satisfies the definition of P. Hence, Assumption 2 implies that
Null H) = x Rd
Denoting the Nth to Mth columns of H by H(N M), the above equivalent to saying
Hide + 1: dc + dp) = 0.
(25)
Denote,
θ H(:: de) x t 1,2.
Then, write,
Qx Oc, Ve 1,2.
(26)
This concludes the proof.
D Proof of Theorem
We restate the theorem here:
Theorem Assume that Assumption is satisfied, that |C| 2 de paired samples (x x 2²)) are available, that A(a), q 1,2 have full column rank, and that IP. is absolutely continuous. as any optimal solution of (6) under the constraint 9 Then, we have Oc.
Denote (g)
From objective in (6) obtain
Q (1) (1) (d)
2)
(27)
Using Assumption I and following the proof of step in Theorem we can obtain:
(4) (9) (1) c, Vq 1,2,
20

---

for some invertible matrices ( Vg. Hence,
11) (d)
(2)
(28)
(d)
( (00))
(29)
Hence we can have linear transformation M = e (+ which has same probability density
as P. However, the sample matching constraint 9 for f-th sample implies that
Q oz Q x( (2)
(1)
1 C7 ( Cf
= (0
C Mct.
Let C [c1... ....,,l Then the above implies:
C
(M-
Now we show that C' a full row rank matrix, which implies that M - M To that end, note that random variables xt and æ; being i.idd implies that ( are i.i.d from ce.. This implies that for any 1 S [4], Pr(c, span((e.)..... Coao
- -l 0.
(30)
This is because span( ((Cm)) Cnac )) for nj E [[]], is lower dimensional subspace in Rdo, which has zero probability under absolutely continuous distribution Pe. Hence any de out of [c]
column vectors in C' are linearly independent with probability 1.
This concludes the proof.
E Detailed Identifiability Conditions of Existing Results
E.1 Identifiability of CCA
Theorem (Identifiability of Aligned SCA via CCA [[). Under L assume that every aligned pair (x(1), (?) share the same c, and that A ( has full column rank. Also assume that there exists an N N-sample set (E,....,EN) such that [CT, (P(I),, (P(2))7 E N +d)) + has (g) full column rank, where C= [CEL Cixl E RRdexN and p(4) pp ... ...PP E Rd xN for
(1)
(2). as an optimal solution of the CCA formulation. Then, we we
= 1,2. Denote
have
Q (9) Oc,
where is nonsingular.
In the above theorem, one see that N (dc + df) + d(2) is necessary condition for the identifiability of @c Hence, CCA needs at least de + d, + df paired samples for identifiability.
E.2 Identifiability of Unaligned SCA in S
We summarize the result in 8 in the following
Theorem 5 (Identifiability of Unaligned SCA via ICA (8). Under TT. assume that the following are met: The conditions for ICA identifiability (33) is met by each modality, including that the components of 2 = _c,, T[ are mutually statistically independent and contain at most
(4) has unit variance; (ii) (u) P P..o, 2,00 + one Gaussian variable. In addition, each z
P -2000 V E Idc + + j. Then, assume that (fm:jm) are obtained by ICA followed by cross domain matching (see the part on Unaligned SCA in Section 2 for 1,..., dc.
21

---

Denote ((1) e, 2(1) and c(2)
e 2 We have the following:
S ke T(m)² E [dc],
(31)
where & E (+1,-1) and π is permutation of dc).
F
Additional Synthetic Data Experiments
Hyperparameter Settings: We use Adam optimizer [70] to solve 7 and learn matrices Q (9) 9 = 1, 2 and the discriminator f. We set the initial learning rate of matrix and discriminator to be 0.009 and 0.00008 respectively. We set the in F we set B 0.01 in ().. We generate total of 100,000 samples in each domain. For our experiment we set the batch size to he 1,000 and run 7 for 50 epochs. Our discriminator is a 6-layer multilayer perceptron (MLP) with hidden units 1024, 521, 512, 256, 128, 64 in each layer. All the layers use leaky ReLU activation functions (71) with slope of 0.2 except for the last layer which has sigmoid activations. We include a label smoothing coefficient of 0.2 in the discriminator predictions as suggested in [40].
0.1 in 7 to enforce (6c). For weak supervision experiment
Additional Details for Validation of Theorem in Sec. 3- Here we explain the data gen- eration details of the result shown in Fig. 3 For the result in top row, we sample ej from Gaussian mixture with three Gaussian components. Each component follows normal dis- tribution (N,, 2) where μ ~ (O 10). The second component, i.c., C2, is independently sampled from the gamma distribution Gamma(1, 3). The private components are sampled from p()) Laplace(1.0, 6.5) and p(2) Uniform|- foorrm10...00 10, 10). both only having one dimension. In the bottom row, we sample e E R² ~ VonMises 2.5, 2.0) distribution. The private components satisfy p()) ~ Laplace(1.0, 6.5) and p!²) ~ Gamma(0.5, 3.0). Each element of mixing matrices
are sampled from A(4) NO, 1), q 1. The readers are referred Table 4 for the definition of notations used for distributions.
Validation of Theorem [ Junder different sample sizes and imbalanced data: Here we observe the shared component identification performance of the proposed method numerically. We conducted two experiments different settings. First, we vary the sample sizes in both modalities. but the two modalities have the same sample size. Second, we only vary the sample size of modality 2 while keeping the sample size of modality fixed. This way, we create the data imbalance between modalities. Note that the shared components are identified if the following two conditions are met:
1. (1) A (o) ④ 0). i.e., (1) e (2) θ and
2. (Q) A (4) (dc de + de ]|F 0.
Therefore, we use the above as our performance evaluation metrics. For the following experiments
(Table and 6 we generate the data for the two modalities by sampling a two-dimensional content
~ SonMises(22) (2.5, 2.0) and private components from p(1) Laplace(1.0, 6.5) and p(2) ~
Gamma (0.5, 3.0). The elements the mixing matrices are sampled as A N 1), q= 1,2. We
report the mean and standard deviation of || (1()) dc) - -2))) de)lle and 1/2 1/2 @
de + ((4))|[= obtained from 5 different runs.
Table 5 shows the performance of SCA and CCA under different sample sizes (i.c., N). One can that the proposed method (SCA) clearly identifies the shared components even when only 100 samples are available. The performance starts to deteriorate when < 50, probably because the min-max optimization problem is difficult to solve with very few samples. CCA does not really work under this setting as it needs aligned cross-domain samples.
Table shows the performance of SCA in the cases where two modalities have unbalanced data sizes. The number of samples in the first modality is fixed to 100,000 while the second modality's 's data size varies from 10,000 to 10 samples. The data generation process remains the same as in the previous experiment. One can see that even under obvious cross-domain data size imbalance (e.g., 100,000 to 1,000), the proposed method performs reasonably well in terms of shared component identification.
22

---

Table 5: Shared component identification performance over different N.
[1:
()) dF
1/2 | (de de dp)) )|p
N
SCA
CCA
SCA
CCA
100,000
0.015
0.020
1.623 0.273
0.031 0.009
0.232 + 0.033
10,000
0.021 + 0.006
1.667 + 0.240
0.030 + 0.002
0.267 + 0.031
1,000
0.018 0.011
1,572 0.474
0.042 + 0.059
0.280 + 0.070
100
0.053 0.014
2.224 + 0.525
0.083 0.096
0.364 0.153
50
0.132 0.118
1.469 0.299
0.142 + 0.132
1.470 = 1.520
20
1.373 + 0.626
2.084 + 0.661
0.490 + 0.321
0.546 = 0,269
Table 6: Shared component identification performance under imbalanced multi-modal data sizes.
# samples in modality 2
[Q de) ( dc)llr
Σ
( (dc de + d )ll
10,000
0.020 0.018
0.018 0.005
1,000
0.065 0.029
0.026 0.015
100
0.145 = 0.051
0.081 0.049
10
1.290 + 0.239
0.293 + 0.064
Validation of Theorem Fig. presents numerical validation for Theorem
Data Generation: We de 3 and dea) 1 for g 2, We sample each component of shared component c Laplace(O.0, 6.5) i 1,2,3, p(1) R Uniform[-10, 10] and
((2) Gamma(0.5, 3.0). Although e satisfies component wise independence assumption, it does not (d)
Cj. Vi,j E [3]. Therefore, Theorem does not cover this case. Nonetheless, this case falls under the jurisdiction of Theorem 3
satisfy the condition that G, F kez, Vi & j because C3
(d)
Result: Fig. 5 corroborates with our Theorem 3 That is, one needs least [4] 2 de = pairs of
"anchors" (i.e., aligned cross domain pairs) to ensure identifiability of 2(4) Oc for g = 1,2. 2.
Ground truth
No anchors
anchors
anchors
300
-00
-101
-100
100
-50
100
100
-30
100
-100
-30
2(1)
100
-100
-50
(12)
22)
Figure Validation of Theorem dc 3 and dp
G Real Data Experiment Settings and Additional Results
G.1 Domain Adaptation
Hyperparameter Settings:
The domain adaptation task follows the hyperparameter settings de-
scribed in Table. 7
23

---

Table 7: Hyperparameter settings for domain adaptation.
Parameter
Value
Optimizer
Adam
Learning rate of Q
0.0002
Learning rate of f
0,00002
Learning rate of classifier
0.02
Learning rate decay of classifier
0.75
À (see Eq. (7))
1.0
y (see Eq. (32)
0.1
Batch size
64
Number of epochs
20
Discriminator; / architecture
6 layers, hidden units (1024, 512, 128, 64)
Activation functions of f
Leaky ReL.U (slope 0.2), Sigmoid (final layer) 0.2
Label smoothing coefficient in
Table 8: Classification accuracy on the target domain of office-31 31 dataset using CLIP embeddings.
source
target
CLIP
DANN
MDD
MCC
SDAT
SDAT+MCC
ELS
ELS+XCC
Proposed
Proposed-NO3
93.7
94.1
95.9
95.0
96.8
98.3
D
W
99.1
100.0
99.3
100.0
100.0
100.0
100.0
100.0
99.7
100.0
W
+
D
100.0
99.5
99.5
98.4
99.5
99.5
99.5
100.0
100.0
99.9
91.9
92.1
94.2
97.7
95.7
95.0
97.7
93.7
99,1
A
81.9
79.2
85.7
81.2
84.6
81.3
83.0
83.9
85.9
W + A
81.7
83.0
82.2
84.7
84.7
86.7
82.6
86.3
85.8
87.1
Average
91.2
91.6
91.4
93.7
92.6
94.4
92.5
94.2
93.0
95.0
Baselines and Training Setup: The basclines are representative DA methods, namely, DANN (25) MDD [60], MCC [61]. SDAT [62]. and ELS [63]. We use the implementations of DANN, MDD, and MCC from the htttttt///gtthhut
taken from https: /ttpttttttuuut com wow.l/yyhhhaaaglhheeeeeeoooe Library while SDAT and ELS are Lawel-Smoothingg.gggggggoooo Label Smoothing, In all the
baselines, the classifier is jointly optimized with the feature extractor Q which arguably regularizes towards more classification-fries friendly geometry of the shared features; see (72) 73 Following their training strategy, we also append a cross-entropy (CE) based classifier training module to our loss in (7) (which learns our feature extractor Q). The CE part uses Qx(1) and the labels of the sources as inputs to learn the classifier, i.e.,
LCE
Uly k) log (e([
(32)
E11=
(1) to a where re(.) Rdc + R the classifier that aims to map the learned feature vector QØ
K -diimensional probability mass function (i.c.. the distribution of the ground-truth label over K classes), y/ E [K] represents the label of the εth sample in source domain, and the indicator function I[ye k| = only when the event y/ = & happens (other wise Byye k] 0. The 7 2 is the tunable parameter. The joint loss is still differentiable, and thus we still use the Adam optimizer to jointly optimize Q and O.
Additional domain adaptation experiment using CLIP features: In this experiment, we use CLJP as an image encoder as it Iearns informative and transferable features from very large datasets [35] and Table 9 show the results Office-31 31 and Office-Home datasets, respectively, using CLIP embeddings. Compared to the results on ResNet50 embeddings in Table D and Table 2 one can observe that all the methods, including proposed method, gains advantage. This is likely because CLIP was trained on large and diverse dataset (35] which may have include similar content to the Office-31 31 and Offfce-Honee datasets.
Table and
The results show that, as foundation model, CLIP can already unify the embeddings of the source and target domains to reasonable extent. In addition, our model and algorithm when combined with regularization techniques like MCC, can still further enhance performance, even with simple prot-proc of CLIP embeddings.

---

Table Classification aceuracy on the target domain office office-Home dataset using CLIP embeddings.
source > targel
CLIP
DANN
MDD
NCC
SDAT
SDAT-NCC
ELS
ELS+N0C
Proposed
Proposed+MCC
Ar
78.0
80.2
80.9
79.6
80.7
80.0
81.3
82.0
83.2
Ar Pr
88.7
88.9
93.3
89.4
94.3
91.2
93.9
91.4
95.2
Ar Rw
90.6
90.2
91.0
92.8
90.1
92.1
89.4
92.1
91.9
93,8
CI + Ar
85.2
83.2
85.1
87.4
83.1
86.1
84.4
87.2
85.4
87.7
CI + Pr
89.0
89.7
90.1
93.4
90.2
93.5
89.7
93.5
91.1
94,9
CI
89.8
87.9
88.3
Pr
78.2
80.4
81.8
83.7
81.0
85.0
81.8
86.1
83.0
86.6
Pr CI
72.7
75.8
75.8
78.4
75.4
78.5
75.7
78.3
77.7
81.3
Pr + Rw
89.0
90.4
90.3
92.6
90.8
92.1
90.0
92.3
91.0
93.6
Rw
86.6
84.9
85.9
85.3
86.3
85.4
87.0
87.7
88.2
78.1
79.8
78.6
79.8
79.1
79.8
81.3
81.8
Rw Pr
94.3
94.6
93.9
95.9
94.8
95.4
94.0
95.2
94.7
96.0
Average
85.0
85.7
86.0
87.6
85.5
87,8
85.7
88.1
87.3
89.5
Visualization Result: Fig- shows the 2-dimensional visualization of the CIIIP-learne features (d = 256) from two domains, namely, DSLR and Amazon images (Office-31), using t-SNE. One can that CLIP could roughly group the same classes from the domains together. But the proposed method can further pull the circles and the triangle markers logether- that the Q really learns shared representations of the same data the DSLR and Amazon domains.
Features from CLIP
30
Low dimensional features from proposed method
40
30
10
20
10
-10
-20
-10-
200-
-00
30-
-50
40-
-40
-30
-20
-10
20
30
40
(a) Clip features d(e)
768.
(b) Features from proposed method d(r) 256.
Figure 6: Office-31 dataset: DSLR images features represented as circle markers, Amazon images features represented as triangle markers. Different color represent different classes.
G.2 Single-cell Sequence Analysis
Hyperparameter Settings: The hyperparameter settings for single-cell sequence analysis is pre- sented in Table. [0
Baseline: For more details on baseline refer to the implementation in https://github..r com/ uhlerlab/ caarlab/cross.ao modal- autoencoerss
G.3 Multi-lingual Information Retrieval
Hyperparameter Settings: described in the Table. [I]
The hyperparameter settings for multi-lingual information retrieval
25

---

Table 10: Hyperparameter settings for single-cell sequence analysis.
Parameter
Value
Optimizer
Adam
Learning rate of Q
0.001
Learning rate of f
0.0001
À (see Eq.
1.0
3 (see Eq.
9
10.0
Batch size
32
Number of epochs
75
Discriminator; f architecture
6 layers, hidden units (1024, 512, 256, 128, 64)
Activation functions of f
Leaky ReLU (slope 0.2), Sigmoid (final layer)
Label smoothing coefficient in
0.2
Table 11: Hyperparameter settings for multi-lingus information retrieval.
Parameter
Value
Optimizer
Adam
Learning rate of Q
0.0001
Learning of f (see Eq. (7)
0.00001
1.0
Batch size
32
Number of epochs
Discriminator; f (similar as in (21)
layers, 2048 hidden units each
Activation functions of f
Leaky ReLU (slope 0.2). Sigmoid (final layer) 0.1
Dropout rate (Input) in ff
Label smoothing coefficient in
0.2
Table reports the P@5 and P@ 10 scores over the test data, calculated for different source and target language pairs. can be observed that the proposed method achieves higher precision than Adv in most of the translation tasks (e.g., by at least 1% in the en-ee and es es->e tasks) when considering both P@5 and P@ 10 scores.
Additional Results:
G.4 Computation resources
All the experiments were run on Nvidia H100 GPU. The approximate runtime for a single run of the algorithm is 20 minutes for multi-lingual information retrieval, 15 minutes for domain adaptation, and 3 minutes for single-cell sequence analysis.
Complexity Analysis:
Since the proposed objective is tackled using stochastic gradient (SG)-based first-order iterative method, the computational complexity of the proposed algorithm depends upon the per-iteration complexity.
For each sample, the per-iteration complexity is composed of forward pass and backward pass.
Note that the problem size depends upon d( (the data dimension), de, and the batch size B. We assume that the network architecture of ff (the number of layers and hidden units in each layer) is fixed, represented by f = o o F,, o .... activation function corresponding the 9th layer. Only the input dimension de of first matrix Fi, varies with the problem size.
Fi, where F, and o are the linear layer (matrix) and
The forward pass involves computing ((4) (()) and ((4) (2(*)), both of which scale linearly with de, d(() and the batch size B. Hence, the forward pass time complexity O(Boc(d")) + )())))
aale)
Similarly, the backward pass requires computing of OL Vi [dc] and aq 94
Vi E [dc].j,
[dc x d where L is the loss function. The first gradient computation is linear in Bdc, while the second gradient computation has a complexity of O(Bdo(d()) + d(2))), Hence the computational complexity of our method is O (( + d(2))).
26

---

Table 12: Average precision P@k of cross-language information retrieval
P@k
source -> target
Adv NN
proposed NN
Adv CSLS
proposed CSLS
en-ces
77.9
78.8
83.6
85.2
es->en
73.2
79.0
83.2
86.0
en >it it->en
68.0
70.8
76.8
82.4
P@5
79.0
77.6
71.2
66.9
en mn-ff
79.2
75.2
85.7
85.2
fr-
69.8
73.5
77.2
83.5
en->es
82.4
82.6
87.0
87.8
es->en
79.0
82.3
87.2
88.8
en en->it
74.1
75.6
81.7
85.6
P@ 10
it it->en
71.5
75.8
82.0
82.9
en
83.4
79.1
88.0
88.4
fr-
73.8
77.4
80.6
86.6
The memory complexity involves storing the network parameters and the aforementioned gradients. Hence, only the size of Q(9) Fi, and c(1) changes with the problem dimension. The size of Q(9), Fi, and cl9) are ded()), O(dc) and de, respectively. Therefore, the space complexity is O(Bdc(d()) + ))
In summary, both the memory and computational complexities of the proposed method scales linearly with de.
H Extension: Private Component Identification
Theorems are concerned with learning the shared component c. The goal, there, was to ensure
that Q C learn (9), we propose to solve the following learning criterion:
Oc, Vq.. In some cases, the private components p(1) is also of interest [6 31
74 To
find (g) E Rddexd
E
Rd xd
= 1.2
(33a)
subject
(33b)
Q (1)
= 1.2,
(33c)
d
E
( = 9= 1,2,
(33d)
E
(1)
TT =1 1,2,
(33e)
where _ means that the random vectors and independent with each other. For implementation we use following criterion,
E EaCts log (1100am!)) + Fa log (1 1(Q0 - (1)
(9) min
max f
M0 ())
R Q ((0)) + Σ R @pp ((0))) 4=1
+ HSTC @ (1), Q (1)).
(34)
9=1
9=1
where, first
adversarial loss for distribution matching. The constraint
33d) and B3ch are enforced as (t) and R(Q, (9)) respectively, where R()(9)) I| The constraint on (33c) is realized with Hilbert-Schmile Independence Criterion (HSIC) (75). HSIC measures the independence between two distribution. So. we minimize HSIC between estimated shared component and estimated private component to promote independence between shared and private components.
QQ TQ(" (9) ((Q
We show that under some reasonable conditions the block ( can also be learned up to a matrix multiplication:
27

---

Theorem 6. Assume that the blocks c, p and pl)) are statistically independent, i.e., p(c, pl)) p p(c)p(p p(p!))) Then, if one of the following holds:
(i) Assumption and assumptions in Theorem 2 are satisfied, and 33 is solved yielding
solutions Q and QT
(ii Assumption is satisfied and has mixing matrix A A and 33 with
9 Qp and Q Qc is solved yielding Qc and (g) the solutions.
(iii) Assumption satisfied and de paired samples (a) 3 are available (weak (4)
supervision), and denote (9) and Qp as the solutions after solving (33
(1) x(4) = Oc and QP xl9) = =(0)p(s), for some invertible E(4) for all
Then, we have Qe 1,2.
Proof. For each case Theorem, (iii), we can prove
(4) x
Oc, q= 1,2
(35)
using Theorems 13
The proofs are referred to Appendix D
Let us denote
Qp x(1)
(4)
=2p
A
c
H(1)
p(9)
((9))
(36)
where ( Q
(g) A E R: x(do+ Note that the constraint
33c
implies that the mutual
information between p( and @(1) is zero, i.e..
(6
Note that (( + ق - 1-1) = e a Markov chain. This is because when conditioned on 2(9)
@ becomes constant, making it independent of p(9), This allows to use the data processing inequality (76] Theorem 2.8.1], I] which results in the following:
T(p
;))
Since mutual information always non-negative, the above implies that T(p(o); c)) = 0. This implies HH69 (2)
that ((9)
independent c. Hence, H(() (9) [ dcl = 0, Vq.
p
Therefore (0)
H (de + 1 de + ( ( (0)) Vg. where 3(O) ( dc 1
dc+ Note that H (9) is full row-rank because of constraint (33e). This implies that E(0) 1,2 invertible matrices.
This concludes the proof.
H.1 Validation of Theorem
6
Fig. presents numerical validation for Theorem 6
Hyperparameter Setting
The hyperparameter setting is the same mentioned in Appendix.
We solve (34 to obtain Qc and Q to recover the shared and private components, respectively.
(g)
(q)
For learning (4) we use Adam optimizer and set initial learning rates be 0.001. Also we set the regularization parameter 10.0 and p 50.0.
28

---

Data Generation: We set de 2 and de) = in the previous synthetic experiments. We sample VonMisess2.5, 2.0) The private components sampled from p()) Beta(1.0, 3.0) and p(2) Gamma(0.5, 3.0) distributions. Each element of mixing matrices sampled from (4)) (0, 1), q 1,2.
Result: Fig- 7 shows the result for proposed method for private component identification. The first column shows the data domain, the second column shows the true and extracted shared component, and the third and fourth columns shows the true and extracted private components. Especially, the last row of the third and fourth columns shows the plot of ground truth p(1) on x- and plo) on the y-axis. The plot is approximately straight line which indicates that the estimated private components p(9) are scaled version (i.e., invertible linear transformations) of ground truth private components. This verifies our Theorem
D(1)
(2)
300
250
4200
200
150
100
30
1300
-5
22 -1
0.0
0.2
0.4
0.6
0.8
20
20
20
00
70
(1)
1
F(2)
25
3200
150
2000
100
-75
205
(2)
1.5
1.0
0-
10-
-0.5
000
50
100
80
0/1
66
10
(1)
(2)
Figure Validation of Theorem dc 2 and d
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See Section 3 and
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section 7
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
30

---

Justification:
See Appendix B C D and H
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Appendix and
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open tions to faithfully reproduce the main experimental results, as described in supplemental material?
the data and code, with sufficient instruc-
Answer: [Yes]
31

---

Justification:
Yes the code is provided in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ tiaaps.cc/ubblic/uuideess /CoeeS/mmddebbeee ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
See Appendix and
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: We have provided the error bars for the experiment that is computationally less demanding.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
32

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See Appendix G.4
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:p:s///orrpp..ccppbbbbbeeee http:/rwg....ooooaocc/cccccaaaaa
Question:
Answer: [Yes]
Justification:
We have followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The contribution of this paper is on theoretical aspects of machine learning. We don't 't foresee any immediate societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
33

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Theoretical paper. So not applicable in our case.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
34

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We donot release any assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not Applicable.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "UJr01wWu9R", "reviewer": "Reviewer_DxMx", "summary": "This work considers the identifiability of linear latent representations that are shared (i.e., identical) across data modalities, in the special case that they are unaligned/unpaired.\nThe approach leverages GAN-style training to achieve divergence minimization between the latent distribution of each modality.\nThe approach appears to be restricted to two modalities.\nUnder the assumption of shared latents, sufficient (not necessary) conditions for identifiability are presented, which are milder and, thus, more general than existing studies.\nFurther, structural constraints based on side information are introduced to further relax identifiability conditions.\nSeveral experiments on simulation and real-world data are provided.", "questions": "1. What do you mean by \"lift\" the constraints in line 131?\n2. Although theorem 1(a) does not \"require\" independence between c and p, isn't that implied/necessary? Otherwise, could you show that Dependence between p and c still yields identifiability of c? Specifically, say, if c1 and c2 are conditionally independent p(c1,c2,p) = p(c1|p)p(c2|p)p(p).\n\t2.a. Does this have to do with Line 149: Q^(q)A^(q) = [Θ^(q), 0] ?\n3. Are there obvious limitations wrt differences in the sample size for x^(1) and x^(2)? Are there any provable biases if the data is highly unbalanced? How does unbalanced data affect the identifiability?\n4. Is the methodology and identifiability theory limited to the two-dimensional case?", "limitations": "- A discussion of the asymptotic computational performance is amiss? Specifically, with respect to d_C and the total sample size.\n- The paper does not appear to discuss the applicability of the theorems to the case of q > 2 (i.e., more than 2 modalities).\n- Lacks a discussion of the stability of GAN training, especially with several additional loss augmentations.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Originality :\n- The work introduces a combination of novel and known ideas in a clever formulation that yields new, less restrictive conditions for identifiability of shared signals from two modalities.\n- The work differs from and extends previous contributions, dealing two unaligned/unpaired modalities.\n- The work further relaxes the identifiability conditions via structural constraints based on additional side information that may be available in certain problems.\n- The manuscript cites related work on identifiability conditions for aligned/paired data, as well as unaligned/unpaired results using stricter ICA conditions, also linking the work to nonlinear studies, adequately indicating the sources of inspiration.\nQuality :\n - The work is technically sound, including proofs for identifiability claims.\n - The theoretical claims are well supported by the experiments.\nClarity :\n - The work is well written and organized, focusing on the key points and contributions.\nSignificance :\n - The results appear to be quite meaningful, with a potentially wide range of application.", "weaknesses": "Quality :\n - The code is not too friendly to readers, lacking higher 1-to-1 correspondence with the notation in the paper. Suggest improving documentation and tidying up the codebase for readability.\n - Figure 5 does not seem to replicate well in `synthetic_train.ipynb` --> Clarify\n - The numerical validations (simulation) are limited to 100,000 samples. Could you illustrate performance at 10,000, at 1,000, and at 100 samples? Many multimodal applications are limited to sample-poor regimes (N < 100), where classical CCA is one of the few performant methods, so it would be useful to assess the performance of unaligned SCA at varying sample sizes, and perhaps include comparable CCA results. Is there a summary measure (like Amari distance) that could be reported alongside figures?\nClarity :\n - The paper contains some typos that limit the clarity. Besides fixing these typos, consider improving the readability of the proofs by being a bit more explicit with \"obvious\" steps that may be currently omitted.\n - I think the set L of paired samples was not defined?\n - Line 59: \"transformations identifies\" --> \"transformations identify\"\n - Line 65: \"samples available\" --> \"samples are available\"\n - Lines 95-96: \"the cross-modality samples share the same c are aligned\" --> unclear meaning... maybe drop \"share the same c\"?\n - Line 117: \"to met\" --> \"to be met\"\n - Line 183: Sentence ends abruptly at: \"where \\Theta^(q).\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Code Clarity]** We will clean the code and change the variable names according to the notation used in the paper.  We found that Fig. 5 sometimes could not be replicated due to occasional failure of GAN convergence. We fixed the issue by increasing the regularization parameter $\\beta$ from 0.001 to 0.005.\n\nTo ensure reproducibility, we have fixed the random seed in the code. If the AC allows, we could share an anonymous link of the code for experiment in Fig. 5 (the rebuttal policy does not allow any link in the replies) through the AC. Please let us know.\n\n\n**[Results under Various Sample Sizes]** Following the reviewer’s suggestion, we have calculated the Amari distance [R1] between $\\widehat{\\bf{\\Theta}}^{(1)}$ and $\\widehat{ \\bf{\\Theta}}^{(2)}$ for different number of samples of unpaired data for CCA and SCA. The data for two views were generated, by sampling the shared component of dimension ($D=2$) from VonMises distribution and private components from Gamma and Laplace distributions. \n\nAs the sample size decreases, the Amari distance increases for unaligned SCA because the distribution matching is difficult only using a few samples. CCA does not really work under this setting as it needs aligned cross-domain samples. However, our unaligned SCA does not need aligned samples.\n\n**Table 1**: Amari distance between $\\widehat{\\bf{\\Theta}}^{(1)}$ and $\\widehat{ \\bf{\\Theta}}^{(2)}$.\n\n---\nN &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | SCA &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | CCA\n\n---\n\n100,000 | 6.5 × 10^-3  | 0.677\n\n10,000 &nbsp; | 5.5 × 10^-3 | 0.533\n\n1,000 &nbsp;&nbsp;&nbsp;&nbsp;| 8.4 × 10^-3 | 0.352\n\n100 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;| 4.2 × 10^-3  | 0.364\n\n50 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;| 0.071&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;| 0.313\n\n20 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;| 0.298 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;| 0.402\n***\n**[Typos, Definitions]** Thank you for your careful reading. We will fix the grammar and typos, and include more detailed definitions.\n\n**[Questions]**\n**Q1.** We meant that our reformulation (7) uses a regularization term $R(\\bf{Q}^{(q)})$ to promote the constraint in Problem (6c). This operation ``lifts'' up the constraint that was below the objective function into the new objective function. We will make this clearer. \n\n**Q2:** No, the independence between $\\bf{c}$ and $\\bf{p}$ is not necessary. In fact, Theorem 1 did not use the independence between $\\bf{c}$ and $\\bf{p}$. However, Theorem 1(a) did use marginal independence of the components of $\\bf{c}$, i.e., for $\\bf{c}$ $=[c_1, …, c_{d_C}]^T$, $p(c_1, …, c_{d_C}) = \\prod_{i=1}^{d_C} p(c_i)$.\n\nNote that conditional independence does not imply marginal independence, i.e., $p(c_1, c_2, \\bf{p})$ $= p(c_1 | p) p(c_2 | \\bf{p}) p(\\bf{p})$ does not imply $p(c_1, c_2) = p(c_1) p(c_2)$. Hence, assumption in Theorem 1(a) may not be satisfied, and identifiability cannot be guaranteed.  However, it is still possible for $\\bf{c}$ to be dependent upon $\\bf{p}$ and satisfy Theorem 1(a), e.g., $p(c_1, c_2, \\bf{p} )$ $= p(c_1) p(c_2 |\\bf{p}) $$p(\\bf{p})$, then $p(c_1, c_2) = p(c_1) p(c_2)$.\n\nThe statement in Line 149, $\\bf{Q}^{(q)}\\bf{A}^{(q)} = [\\bf{\\Theta}^{(q)}, 0]$, is the goal of Theorem 1. In our proof, this goal is achieved without the assumption that $\\bf{c}$ and $\\bf{p}$ are independent.\n\n**Q3.** Our conjecture to both of the first two questions is “yes”, but we have not had analytical underpinning yet. Empirically, unbalanced data is not friendly to such unaligned multi-domain learning problems. Our identifiability does not consider sample sizes of $\\bf{x}^{(1)}$ and $\\bf{x}^{(2)}$, or differences thereof. The analysis is carried out in the limit of infinite data, i.e., assuming an exact solution to the proposed optimization problem---which is already a challenging analysis problem. Therefore, data imbalance conditions are not covered by the current analysis. For latent component analysis problems, e.g., ICA and nonlinear ICA, the vast majority of the literature have to consider the population case to simplify analysis. Nonetheless, finite sample analysis does exist [R2], but such analysis itself is highly nontrivial and might deserve a standalone study.\n\n**Q4.** The methodology and identifiability theory are not limited to the two-dimensional case. Note that the latent dimensions $d_C$ and $d_P$ can be much larger positive integers.\n\n**[Limitations - Q>2 Case]**  The algorithm can be easily extended to Q>2 cases. Nonetheless, it does require additional work to understand what benefits can be brought upon in terms of identifiability by extra domains.\n\n\n**[Limitations - Discussion on GAN stability]** In our experience, GAN training is sensitive to hyperparameter setting, and can fail occasionally for different random initializations. However, adding regularization (e.g., homogenous mixing case, weakly supervised case, and classification loss in domain adaptation) does seem to improve the training stability. We will add this discussion.\n\n## References\n\n[R1] Amari et al., 1995. A New Learning Algorithm for Blind Signal Separation.\n\n[R2] Lyu et al., 2022. Understanding Latent Correlation-Based Multiview Learning and Self-Supervision: An Identifiability Perspective.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "80dQDZRnTY", "reviewer": "Reviewer_1Nhs", "summary": "This work considers a problem similar to classical Canonical Correlation Analysis (CCS), which assumes a linear generative model for data $(x_1, x_2)$: $x_1=W_1z$, $x_2=W_2z$ and aims to identify the underlying components. \n\nThis problem has been extended previously to include \"private information\": $x_1=W_1z_1, z_1=[c,p_1]$, $x_2=W_2z_2, z_2 = [c,p_2]$ for common $c$ and independent $p_j$. \n\nThe current work further assumes that data is *unpaired* and rather than mapping each pair ($x_1$, $x_2$) such that $z_1$, $z_2$ are close together (in some metric), it is proposed that all $x_1$ are mapped to be similar *in distribution* to the mapped $x_2$s.", "questions": "see weaknesses", "limitations": "see weaknesses", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper aims to provide rigorous criteria in which underlying generative factors are identifiable in the extended CCA problem it tackles (unpaired CCA with private information).\n\nThe results show improvement over benchmarks indicating promise to the approach.", "weaknesses": "High level:\n* While I understand the basics I am not an expert in the area of CCA, but I find the paper fairly difficult to follow. More explanation would be helpful, e.g. \n    - [28] why is any linear mixture model ill-posed (is that strictly true in *every* linear mixture case?)\n    - [47] what is meant by \"facilitating one-to-many translations\", the context/meaning is unclear.\n* The theoretical part of the paper relates to a simple linear model, but none of the experiments follow this model\n    - e.g. the algorithm is applied to CLIP embeddings, which are not \"the data\", so to make claims about a simple linear model z=Ax and then apply it to CLIP seems incongruous. This experiments seem to relate more to a CCA-based \"loss function\" that takes representations and looks to align them/encourage independent factors etc.\n    - other experiments appear to be on discrete data, which the methods doesn't apply to, presumably these are also represented as some intermediate step?\n    - it seems strange to propose a simple linear model, present theoretical results about identifiability that rely on that simplicity but make a dramatic departure in the experiments where the assumptions clearly do not hold and the notion of identifiability is unclear.\n*  if the work does achieve an improvement in a CCA type setting, it seems appropriate to compare with other CCA methods on suitable data. Adding results on more complex data/representations may be of additional interest.\n\nAssumption 1 \n - hard to parse and could be made more clear. \n - unclear if correctly defined, don't vectors y need to be orthogonal to subspace P? It seems extremely loose to the point of simply saying $P_{c,p_1} \\ne P_{c,p_2}$ (specifying where any difference lies to this might add clarity).\n\nTheorem 1 \n - is this saying that if all dims of c are distributed differently, p(z)'s can only match (e.g. under GAN loss) by correctly aligning each dim? If so, that is pretty intuitive and it could be made clearer that you are putting that mathematically and proving it for the sake of rigour.\n - I have not been through the proof, 5+ pages of proof without a sketch in the paper might be more suitable for a journal as appendices are not typically expected to be reviewed in detail.\n\nOverall, there may be useful results in the paper, but in my view it should be re-written to make more clear what it is doing. It seems a confusing mix of simple linear generative model and related CCA methodology mixed with much more complex representations (e.g. CLIP) passed through a GAN + linear layer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Linear mixture models (LMMs) are Ill-posed]**\n\nIn general, LMMs are not identifiable. Because for any $\\bf{y}=\\bf{A}\\bf{x}$, where both $\\bf{A}$ and $\\bf{x}$ are unknown, one can find an infinite number of invertible $\\bf{Q}$ such that $\\bf{y}=\\bf{AQQ}^{-1}\\bf{x}$. Then, both $(\\bf{A},\\bf{x})$ and $(\\bf{AQ},\\bf{Q}^{-1}\\bf{x})$ are equally fit to the data $\\bf{y}$, making the problem ill-posed in terms of identifiability (or, solution uniqueness) [R3-R5]. In our case, we aim to identify two blocks in $\\bf{x}$, i.e., $\\bf{x}=[\\bf{c},\\bf{p}]$. The same ill-posedness remains.\n\n**[One-to-many translations]**\n\nTranslation means changing the appearance of a sample in $\\bf{x}^{(1)}=\\bf{A}^{(1)}[\\bf{c},\\bf{p}^{(1)}]^T$ to its corresponding samples in the other domain. Note that the content is given by $\\bf{c}$, and the appearances are controlled by $\\bf{A}^{(q)}$ and $\\bf{p}^{(q)}$. One-to-many translation means that one can combine the $\\bf{c}$ extracted from $\\bf{x}^{(1)}$ with many different $\\bf{p}^{(2)}$; see examples in [R6].\n\n**[Using CLIP as Pre-processing]**\n\nPlease note that we only used CLIP as pre-processing (analogous to using PCA in the old days). Powerful pre-processing tools like CLIP and pretrained vision models (eg. ResNet) can map images to approximately linear subspaces [R7] (also see linear probing experiments in [R8-R9]).\n\n**[Theory-Experiment Consistency]**\n\nWe respectfully disagree with the comment that our experiments are \"dramatic departure\" from our theory. Please note that our experiments in Fig. 3, 5, and 7 **exactly follow the LMM**. The experiments on images, single-cell data, and language data used proper pre-processing in order to map data to linear subspaces, approximating our model and supporting the * **usefulness** * of the model in practice.\n\n**[“Discrete Data”]**\n\nNo. We did not run experiments with discrete data. We applied our methods on the continuous feature representation space of the data. Those features were obtained from gene expression counts [R10] and fast-Text embeddings [R11] in single cell and multi-lingual experiments, respectively. Modeling such features as continuous random variables is a common practice [R12,R13].\n\n**[The Significance of Understanding the LMMs]**\n\nUnderstanding the LMM-based unaligned SCA is the first step towards more complex models. This is how studies evolved from ICA [R3], PCA, and NMF [R4] (all LMMs) to provable nonlinear mixture models. However, there has been no existing theoretical support for the identifiability of the clearly important unaligned SCA model considered in this work.\n\n**[Comparing with CCA]**\n\nLet us clarify: CCA cannot work under our setting. The limitation of CCA is that CCA needs to know the one-to-one correspondence between multi-domain data. For example, for multilingual word alignment, it needs paired data $(\\bf{x}^{(1)}\\_\\ell,\\bf{x}^{(2)}\\_\\ell )\\_{\\ell=1}^L$ where $\\bf{x}^{(q)}\\_\\ell,$ $q=1,2$ represent the same entity (e.g., \"cat\") in two languages. In our case, the data is not paired, i.e., $\\bf{x}^{(q)}\\_\\ell,$ $q=1,2$ need not correspond to the same word. Hence, our method is called \"unaligned SCA\", but CCA is essentially \"aligned SCA\". Their settings are fundamentally different.\n\n**[Clarifying Assumption 1 (A1) ]**\n\n(A1) aims to provide a way to characterize how different the latent distributions $P_{\\bf{c},\\bf{p}^{(1)} }$ and $P_{\\bf{c},\\bf{p}^{(2)}}$ are.\n\nLet $S^{(q)}$ denote the set of all possible \"stripes\" $\\cal{A}^{(q)}$ described in (A1) for domain $q=1,2$. Then, (A1) simply says that there should not exist any two stripes $\\cal{A}^{(q)}$$\\in S^{(q)},~q=1,2$ such that the two latent distributions assign the same probability to all scaled versions of stripes $\\cal{A}^{(q)}$. This makes the two distributions sufficiently different from each other.\n\n**(i) Vectors $y$ orthogonal to subspace $\\cal{P}$**\n\nNo, $\\bf{y} \\perp \\cal{P}$ is not needed since this orthogonality constraint leaves $S^{(q)}$ unchanged, i.e., $\\hat{S}^{(q)}= \\\\{\\cal{A}^{(q)}\\in S^{(q)}|~\\hat{\\bf{y}}_i^{(q)}\\perp\\cal{P}^{(q)}\\\\}=S^{(q)}.$ To see this, first it is clear that $\\hat{S}^{(q)}\\subseteq S^{(q)}$. Second, any $\\cal{A}^{(q)}\\in S^{(q)}$ is equal to some $\\hat{A}^{(q)}\\in\\hat{S}^{(q)}$ constructed using $ \\bf{\\Pi}\\_{\\cal{P}^{(q)}}\\{y}\\_i^{(q)}$ instead of  $\\bf{y}\\_i^{(q)}$, i.e., the projection of $\\bf{y}\\_i^{(q)}$ onto the orthogonal subspace of $\\cal{P}^{(q)}$. Hence $S^{(q)}\\subseteq \\hat{S}^{(q)}$, and thus $S^{(q)}=\\hat{S}^{(q)}$.\n\n**(ii) Difference between (A1) and $P_{\\bf{c},\\bf{p}^{(1)}}\\neq P_{\\bf{c},\\bf{p}^{(2)}}$** \n\nWe are a little confused by the reviewer’s comment that \"It seems extremely loose … saying $P_{\\bf{c},\\bf{p}^{(1)}}\\neq P_{\\bf{c},\\bf{p}^{(2)}}$\". **We hope to clarify that we never used $P_{\\bf{c},\\bf{p}^{(1)}}\\neq P_{\\bf{c},\\bf{p}^{(2)}}$ but only (A1).** For $P_{\\bf{c},\\bf{p}^{(1)}}\\neq P_{\\bf{c},\\bf{p}^{(2)}}$ to hold, the two joint PDFs can be exactly the same everywhere except for an arbitrarily small subset of their domain. However, Assumption 1 only holds if the two joint PDFs are sufficiently different, by comparing the measures over the specifically defined “stripe” regions.\n\n**[Theorem 1 clarification]**\n\n**(i)  Meaning of theorem 1**, the reviewer’s description is not entirely accurate, but partially correct. Theorem 1 (a) states that if all dimensions of $\\bf{c}$ are distributed differently (as in Line 179) and independent, then $p(\\bf{z}^{(1)})$ can only match $p(\\bf{z}^{(2)})$, where $\\bf{z}^{(q)}=\\hat{\\bf{Q}}^{(q)}\\bf{x}^{(q)}$, only when $\\bf{z}^{(1)}$ and $\\bf{z}^{(2)}$ are aligned, i.e., $\\bf{z}^{(1)}=\\bf{z}^{(2)}=\\bf{\\Theta}\\bf{c}$. This is called block-identifiability [R14,R15].\n\n**(ii) Regarding the proof length**, We will include a proof sketch in the main paper when it fits; otherwise, we will include it in the appendix with clear pointers in the main paper.\n\n&nbsp;\n\nFor references, please refer to the **References** section in the **Overall Response.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8fU0VrEZFj", "reviewer": "Reviewer_tDhK", "summary": "The paper considers the identifiability of shared components from a linear mixture. The theory requires multiple domains. However, compared to previous works, the required domains do not need to be aligned in this work. A practical estimation model has been proposed according to the theory.", "questions": "1. I didn't fully understand the proof in Line 780--does the usage of data processing inequality require a Markov chain? If so, has it been shown in the proof?\n\n2. Could you please elaborate more on the connection between the proposed theory and previous work focusing on identifying content and style variables? It seems like they share a similar goal.", "limitations": "The authors have discussed the limitations.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The discussion on the related work is comprehensive.\n\n2. The experiments have been conducted on both synthetic and real-world datasets.\n\n3. The proposed algorithm looks pretty neat.\n\n4. Limitations have been discussed in detail together with potential next steps.", "weaknesses": "1. Since there are already many works in learning shared components in the nonlinear setting, and some of them can even handle unpaired mixtures, the linear setting appears less appealing in comparison.\n\n2. Assumption 1 is similar to the one used in the previous works, which should be highlighted earlier in the paper.\n\n3. The discussion of the assumption of hyper-rectangle support is missing. Is it restrictive? Maybe some real-world examples could be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Regarding \"Many Shared Component Learning Methods Exist\"]** We would like to note that the existing **identifiability research** on shared component learning (including nonlinear mixture based ones) from **unaligned** multi-domain data is in fact rather limited (although empirical studies are abundant). In addition, the limited existing identifiability results cannot cover our settings, and thus study under our settings is still of its own significance.  We reviewed these identifiability studies in the manuscript. We reiterate with a bit more details as below:\n\nThe work in [R16] considers identifiability of unaligned data learning using linear transformation. But they did not model shared and private latent components. Their identifiability conditions are also much more restrictive relative to ours (e.g., they require the 3rd-order moments of data are high-rank tensors, which is hard to satisfy and a little counter-intuitive).\n\nThe work of [R17,R20] considered a non-linear mixing function for the homogeneous data case. Compared to our work, [R17,R20] required at least a large number (i.e., $ 2 d_P + 1$) of domains, component-wise independence of latent variables, and much more stringent domain variability assumptions as discussed in Line 167-173 of the manuscript. Our method can work with as few as only 2 domains and the domain variability condition (A1) that we use is much more relaxed.\n\nAs detailed from Line 105-118, the work [R19] considered the same generative model as ours, but operated under much more stringent assumptions on the latent variables, such as all variables being component-wise independent and having unit variance.  Both conditions are not needed under our framework.\n\nThe works [R2,R14,R18] all consider identifiability of the shared components under linear or nonlinear mixture settings. However, they all assume that the cross-domain data are aligned, which is significantly simpler than our setting.\n\n**[Regarding the Rectangle Assumption]** We consider the assumption of hyper-rectangle support to be not very restrictive. Any collection of real-valued features could form a hyper-rectangle. For example, in image-to-image translation between animal faces (e.g., dog and cat images be the two domains) [R21], the position and orientation of the animal is generally the shared information whereas the appearance of the animal is the private information. If these two aspects are represented by two real-valued components in the latent space, then their supports could easily form a rectangle.\n\n**[Questions]**\n\n**Q1.** Yes, the data processing inequality requires a Markov chain. And we do have a Markov chain: $\\hat{\\bf{p}}^{(q)} \\to \\hat{\\bf{c}}^{(q)} \\to \\bf{\\Theta}^{-1} \\hat{\\bf{c}}^{(q)} = \\bf{c} $. It is a Markov chain because conditioned on $\\hat{\\bf{c}}^{(q)}$, $\\bf{\\Theta}^{-1} \\hat{\\bf{c}}^{(q)}$ is a constant, and thus independent of $\\hat{\\bf{p}}^{(q)}$. \nIt was not explicitly written in the manuscript. We will add this clarification.\n\n**Q2.** Please refer to **[Regarding “Many Shared Component Learning Methods Exist”]**\n\n&nbsp;\n\n## References\n\n[R2] Lyu et al., 2022. Understanding Latent Correlation-Based Multiview Learning and Self-Supervision: An Identifiability Perspective.\n\n[R14] Kugelgen et al., 2021. Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style.\n\n[R16] Gulrajani et al., 2022. Identifiability Conditions for Domain Adaptation.\n\n[R17] Xie et al., 2023. Multi-Domain Image Generation And Translation With Identifiability Guarantees.\n\n[R18] Sorensen et al., 2021. Generalized Canonical Correlation Analysis: A Subspace Intersection Approach.\n\n[R19] Sturma et al., 2023. Unpaired Multi-Domain Causal Representation Learning.\n\n[R20] Kong et al., 2022. Partial disentanglement for domain adaptation.\n\n[R21] Choi et al., 2019. StarGAN v2: Diverse Image Synthesis for Multiple Domains.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ct5suMF3Ut", "reviewer": "Reviewer_YjhB", "summary": "This work presents a method for performing a shared component analysis (SCA) in the case of multi-modal unpaired data drawn from a linear mixture. This problem (and method) will be referred to as Unaligned SCA. Unaligned SCA is tackled by matching the probability distributions of the embedded (features) multi-modal data. More specifically, the authors draw inspiration from the traditional adversarial loss used in GANs, and formulate the component analysis algorithm as a min-max optimization problem where a discriminator (a neural network) is trained to maximize the confusion between the alignment of two representations (typically from two different modalities), and then the best alignment is sought in order to “fool” the current discriminator. The alignment matrices are structured so that the algorithm can distinguish between components shared across modalities and private components specific to each modality. The authors claim that the shared components can be identified up to the same ambiguities as those identifiable in the aligned case. Furthermore, the authors explain that while there are other methods that attempt to solve the Unaligned SCA, the conditions of the proposed algorithm are considerably milder.\n\nThe algorithm is then extended to cases where additional knowledge is present. First the algorithm is modified to accommodate the scenario where the data is generated by a single modality (uniform modality). Then, the algorithm is modified to accommodate the case where some data pairing is available (similar to a weakly supervised case). The authors show that by adding appropriate constraints  the shared component can be identified under milder conditions.\n\nThe author provide first a theoretical analysis with numerical simulations, and then some concrete applications of Unaligned SCA for the problem of domain adaptation (same modality), Multi-lingual Information Retrieval (only in the appendix, same modality) and Single Cell Sequence Analysis (multi modality with and without pairing).", "questions": "A satisfactory answers to these points could improve the \"contribution\" (and partially the \"soundness\") of the work.\n- Was CLIP used as a pre-process step for all the baselines in the Domain Adaptation? If not, could the authors present those results by using the same pre-processing step? As I mentioned above ideally without using CLIP due to the potential contamination of the test sets.\n- Could the author explain why 6(b) holds for unpaired samples?\n\nFurther addressing these less critical aspects could improve the \"Presentation\" score (and partially the contribution see first point)\n- I believe the paper would be stronger with more recent baselines as suggested above.\n- I find unclear how Fig.1 was created. More explanation would help the understanding. \n- I find Fig 2 not clear: why are there c1 and c2 and only p1, whereas I was expecting p1 and p2 and a common (shared) c? Reading if further it might be that c1 and c2 are the two axes of the common space $c$. If this is the case I’d make sure to clarify it.\n- Assumption 1 is not clear to me. Is this saying that both points y1 and y2 leave in the same sub-space within the span of $(c, p^{(1)}, p^{(2)})$?  If so in which way is this useful?\n- The authors say the results are an average of 5 runs, which is great, but  the standard deviation should also be reported.\n- I find this sentence confusing: \"First, it is unclear if (6b) could disentangle c from p(q). In general, Q(q)x(q)\ncould still be a mixture of c and p(q) yet (6b) still holds (e.g., when both c and p(q) are Gaussian.)\" first it says it is unclear if they can be disentangled, but the whole identifiably relies on the ability to disentangle them no?\n- In a couple of places in the manuscript the authors refer to an experiments where all the results are in the appendix. I would add a brief summary of the results so the main paper is self-standing (this happens in More Synthetic-Data Validation and Application (iii)).\n- In a few places $p^{(2)}$ appears without a closed bracket, i.e., $p^{(2}$.\n- Sometimes the authors used the comma (,) to separate thousands but other time they didn’t. I would use a uniform notation.", "limitations": "Yes, the authors have identified and clearly stated three main limitations:\n- The fact that the conditions presented are sufficient while the necessary are not yet known.\n- The fact that the method works only for linear mixtures (which limits the expressivity).\n- Lastly, the fact that the theoretical derivation assume infinite data.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The work tackles an important area of research with potential applications that range from explainability, to SSL, or multi-modal problems.\n- The method seems very flexible: \n    - The method proposed can work for completely unpaired data but if some paring is available it can take advantage of such additional knowledge. \n    - The method proposed is meant for multi-modal scenarios but it can also work in homogenous use cases.\n- All parameters are well documented in the appendix and code was part of the additional material.", "weaknesses": "I would divide the weaknesses into two groups: the empirical evaluation, of which I am fairly confident about, and the theoretical analysis, of which due to my limited knowledge in this field I am less confident of. I will share here my concerns as solving them might also be helpful for other readers in my position\n\n*Empirical Analysis.* \n\nI find the empirical analysis weak. The multi-modal and unpaired scenario results are underwhelming, while I find the domain adaptation (same modality) potentially problematic due to the use of CLIP as pre-processing step, and the lack of a strong recent baseline (less problematic than the CLIP reason).\n- The only practical results where the data are multi-modal and unpaired are the ones shown in Figure 4 first blue dot where the accuracy is about 10%. Since this is the only case presented it is unclear if the method, in practice, cannot cope in this scenario or if the specific problem chosen is particularly challenging (in which case other evidence would be maybe better).\n- In the domain adaptation experiments the paper reads: “The images are pre-processed by the pretrained CLIP model [34] that uses ViT-L/14 transformer architecture.” It is not clear if ALL baselines used the CLIP embeddings as pre-processing, or if this was only done for the proposed algorithm. For fair comparison the same pre-processing should be applied to all algorithms. \n- Additionally, CLIP is known to have been pre-trained on a large and diverse dataset and there is a good chance it has been trained on Home-Office and Office-31 too, so it is difficult to appreciate the ability of the proposed method when using such a powerful pre-processing step (which as I mentioned might have been trained on theses datasets, including the their test sets). So making sure the pre-processing is equal is necessary, I would also encourage to present results with a less powerful pre-processing step (e.g., something pre-trained on ImageNet either supervised or SSL style like SimCLR) in order to better distinguish the contribution of CLIP vs every baseline and the proposed model.\n- The authors use a lot of baselines as comparison however all these baselines seem to be fairly old (all before 2020?). I would suggest comparing with a stronger baseline (either check the leaderboard or here are some suggestions [1-4], note that not all might be immediately applicable). \n\n[1] D3GM (https://arxiv.org/pdf/2401.05465)\n\n[2] CLUE (https://arxiv.org/abs/2010.08666)\n\n[3] LAMDA (https://arxiv.org/abs/2208.06604)\n\n[4] SDM (https://arxiv.org/abs/2203.05738)\n\n\n*Theoretical analysis.*\nI have struggled to follow the theoretical explanation. Specifically, I understand the rationale behind formulation in eq (7) but I would fully agree with it if the samples were paired. I do not understand how 6(b) holds for unpaired samples. I suppose this is the explanation currently presented before assumption 1 but even after reading it I was left with the same question.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Empirical Results, CLIP, and Recent Baselines]**\n\n**(i) “Only Practical Result is in Fig 4.”** We believe that there might have been some misunderstanding. Fig. 4 is used to validate Theorem 3. The blue dot markers suggest that conditions (a-b) in Theorem 1 might not be satisfied.\n\nTo clarify, the paper studied three settings, corresponding to Theorems 1, 2 and 3. The motivation for studying Theorems 2-3 is that condition (a-b) in Theorem 1 seemed to be restrictive for many applications. Theorems 2 and 3 also work under **practical settings**, but with more structural info. relative to Theorem 1. Theorem 2 uses homogeneity and Theorem 3 uses weak supervision; see similar settings in [R17,R20,R22]. Theorem 1 is validated in Fig. 3; Theorem 2 is validated in Sec. 6, applications (i) and (iii), and Theorem 3 is validated in Fig. 5 and Sec. 6 application (ii).\n\n**(ii) Avoid Using CLIP and More Recent Baselines.** Thanks for this comment. The reviewer has a good point. \n\nWe agree with the reviewer that the CLIP was trained on a large and diverse dataset and could have included Office31 and Office Home as well. Hence, it may not be fair to run the methods over the CLIP-learned space.  To address this issue, we follow the reviewer’s suggestion and use features from Resnet50 pre-trained over the ImageNet1k dataset. All methods use the same pre-trained features for a fair comparison. \n\nAs per reviewer’s suggestion we also added two recent baselines:\n\n**ELS [R24]**  (ICLR 2023)\n\n**SDAT [R23]**  (ICML 2022)\n\nNote that we didn't use the reviewer-suggested active learning baselines as they applied to different settings.\n\nPlease see Table 1 and 2 in the **attached PDF** for the new experiments.\n\n**[Regarding Distribution Matching (Eq(7) and Eq(6b))]**\nThe first two terms (GAN loss) in Eq. (7) is used to enforce (6b), i.e., matching the distributions of random variables $\\bf{Q}^{(1)}\\bf{x}^{(1)}$ with $\\bf{Q}^{(2)}\\bf{x}^{(2)}$ [R25]. \nWe do not need paired samples to match the distributions. Intuitively, (7) “learns” $\\bf{Q}^{(q)}, q=1,2$, in such a way that the samples of $\\bf{Q}^{(1)}\\bf{x}^{(1)}$ cannot be distinguished from the samples of $\\bf{Q}^{(2)}\\bf{x}^{(2)}$ in terms of distribution.\n\n**[Questions Regarding Contribution and Soundness]**\n\n**a.** CLIP: The reviewer has a great comment. No, the CLIP was not used for other methods. We agree with the reviewer that this should be rectified. We followed the reviewer’s suggestion and ran all methods using CLIP features (by throwing away the baselines’ original encoders). It turned out that all methods perform similarly well (**see Table 3 in the attached PDF**). As the reviewer pointed out, this is perhaps because CLIP memorizes too much information.\n\n**Tables 1 and 2** in the attached PDF show new results where every baseline uses features from ImageNet1k-pretrained ResNet50. We hope these new results would alleviate this concern.\n\n**b.** Please refer to **[Regarding Distribution Matching (Eq(7) and Eq(6b))].**\n\n**[Questions Regarding Presentation]**\n\n**a.** Please refer to **[Empirical Results, CLIP, and Recent Baselines]**\n\n**b.** To create Fig. 1, we first sample $\\bf{c} \\sim \\cal{N}(\\bf{0}, \\bf{I}), \\bf{c} \\in R^2$, and set $\\bf{\\Theta}^{(q)}, q=1,2$ to two different rotation matrices. A unique color is picked for each sample of $\\bf{c}$ and used to color both $\\bf{\\Theta}^{(q)} \\bf{c}, q=1,2$. The purpose of Fig. 1 is to illustrate that $ \\bf{\\Theta}^{(1)} \\bf{c} \\stackrel{(d)}{=} \\bf{\\Theta}^{(2)} \\bf{c}$ even when $\\bf{\\Theta}^{(1)}  \\neq \\bf{\\Theta}^{(2)} $, which is clearly the case. We will make Fig.1 caption clearer.\n\n**c. [Clarification of Assumption 1]**\n\n(A1) aims to provide a way to characterize how different the latent distributions $P_{\\bf{c},\\bf{p}^{(1)}}$ and $P_{\\bf{c},\\bf{p}^{(2)}}$ are.\n\nLet $S^{(q)}$ denote the set of all possible ``stripes'' $\\cal{A}^{(q)}$ described in (A1) for domain $q=1,2$. Then, (A1) simply says that there should not exist any two stripes $\\cal{A}^{(q)} \\in S^{(q)}, q=1,2$ such that the two latent distributions assign the same probability to all scaled versions of stripes $\\cal{A}^{(q)}$. This makes the two distributions sufficiently different from each other, needed to identify common and private information in the two domains.\n\nRegarding $\\bf{y}_1, \\bf{y}\\_2$: No, it is not saying that $\\bf{y}\\_1$ and $\\bf{y}\\_2$ live in the same subspace within the span of $(\\bf{c},\\bf{p}^{(1)},\\bf{p}^{(2)})$. The vectors $\\bf{y}_i^{(q)}$ are any linearly independent vectors in $\\bf{R}^{d\\_C+d\\_P^{(q)}}$, whereas $\\bf{c}$ and $\\bf{p}^{(q)}$ are random vectors in $\\bf{R}^{d\\_C}$ and $\\bf{R}^{d\\_P^{(q)}}$, respectively.\n\nThe **use** of the vectors $\\bf{y}\\_1, \\bf{y}\\_2, \\dots$ is to characterize the stripes $\\cal{A}^{(q)}$, which are ultimately used to characterize the difference between the two latent distributions,  $P\\_{\\bf{c},\\bf{p}^{(1)}}$ and $P\\_{\\bf{c},\\bf{p}^{(2)}}$.\n\n**d.** Due to time constraints in the rebuttal process, we could only produce the standard deviation for a couple of transfer tasks in the new experiments. Table 2 (first 2 rows) shows the corresponding **mean and standard deviations** for (Ar $\\to$ Cl, and Ar $\\to$ Pr) using 5 trials. We will include the new experiments with multiple runs and standard deviation in the revised version.\n\n**e.** Yes, identifiability implies disentanglement of $\\bf{c}$ and $\\bf{p}^{(q)}$. The two sentences highlight the challenge of ensuring this disentanglement by solving Problem (6), and provide an example where disentanglement fails, such as when $\\bf{c}$ and $\\bf{p}^{(q)}$ are Gaussian. This indicates that disentanglement is not always possible. \nThe purpose of our work is to derive precise conditions under which disentanglement can be ensured. \n\n**f,g,h.** Thanks. We will fix the typos, and add a brief summary of the results in the main paper.\n\nFor references, please refer to the **References** section in the **Overall Response**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
### [ **Overall Response** ]

We sincerely thank all the reviewers for their effort in reviewing our manuscript.  Our responses are summarized as follows:

**Reviewer DxMx** suggested improving code clarity and observing the effect of sample size with a new evaluation metric. Following the comments, we ran an additional experiment and evaluated the Amari distance between $\widehat{\bf{\Theta}}^{(1)}$ and $\widehat{ \bf{\Theta}}^{(2)}$. We also improved our code readability and clarified the questions of the reviewer regarding the methodologies and identifiability.

**Reviewer 1Nhs’s** major comments were regarding the use of pre-processed data, theory-experiment consistency, significance of linear mixture models, and clarifications on our assumption and theorem. We provided clarifications. In particular, we pointed out that features obtained after common pre-processing tools (e.g., pre-trained models and word embeddings) are more likely to follow the linear mixture model by empirical evidence from the literature. Hence, such settings are arguably suitable for applying our methods. 

**Reviewer tDhk** suggested discussing differences from existing content-style approaches. There are also comments regarding the practicality of the hyper-rectangle support assumption for the content and the use of the data processing inequality in our proof step. Following the suggestions, we explained use cases where the content could be seen as hyper-rectangle support. Further we clarified our proof step where we use data processing inequality and added the discussion regarding the connection with the previous related works.

**Reviewer YjhB** suggested using the same pre-processing for all methods in experiments. The reviewer also made a good point that using CLIP might not be appropriate as CLIP may have seen all the data under test. To address this concern, all methods now use ImageNet1k pretrained ResNet features instead of the CLIP features. The results are attached in the **enclosed PDF**. We have also added more recent baselines following reviewer suggestions.

&nbsp;

Due to space constraints for the rebuttal, we have the references used in the rebuttal are as follows:

&nbsp;

### **References**

[R1] Amari et al., 1995. A New Learning Algorithm for Blind Signal Separation.

[R2] Lyu et al., 2022. Understanding Latent Correlation-Based Multiview Learning and Self-Supervision: An Identifiability Perspective.

[R3] Common et al., 1994. Independent Component Analysis, A New Concept?

[R4] Lee et al., 1999. Learning the parts of objects by non-negative matrix factorization.

[R5] Erdogan et al., 2013. A Class of Bounded Component Analysis Algorithms for the Separation of Both Independent and Dependent Sources.

[R6] Huang et al., 2018. Multimodal Unsupervised Image-to-Image Translation.

[R7] Bhalla et al., 2024. Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE).

[R8] Radford et al., 2021. Learning Transferable Visual Models From Natural Language Supervision.

[R9] Chen et al., 2020. A Simple Framework for Contrastive Learning of Visual Representations.

[R10] Cao et al., 2018. Joint profiling of chromatin accessibility and gene expression in thousands of single cells.

[R11] Joulin et al., 2016. FastText.zip: Compressing Text Classification Models.

[R12] Lample et al., 2018. Word Translation without using Parallel Data.

[R13] Yang et al., 2021. Multi-domain translation between single-cell imaging and sequencing data using autoencoders.

[R14] Kugelgen et al., 2021. Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style.

[R15] Lyu et al., 2022. Understanding Latent Correlation-Based Multiview Learning And Self-Supervision: An Identifiability Perspective.

[R16] Gulrajani et al., 2022. Identifiability Conditions for Domain Adaptation.

[R17] Xie et al., 2023. Multi-Domain Image Generation And Translation With Identifiability Guarantees.

[R18] Sorensen et al., 2021. Generalized Canonical Correlation Analysis: A Subspace Intersection Approach.

[R19] Sturma et al., 2023. Unpaired Multi-Domain Causal Representation Learning.

[R20] Kong et al., 2022. Partial disentanglement for domain adaptation.

[R21] Choi et al., 2019. StarGAN v2: Diverse Image Synthesis for Multiple Domains.

[R22] Wu et al., 2018. Multimodal Generative Models for Scalable Weakly-Supervised Learning.

[R23] Rangwani et al., 2022. A Closer Look at Smoothness in Domain Adversarial Training.

[R24] Zhang et al., 2023. Free Lunch For Domain Adversarial Training: Environment Label Smoothing.

[R25] Goodfellow et al., 2014. Generative Adversarial Nets.

&nbsp;

&nbsp;

Meta-review: The submission proposes a new approach for unaligned shared component analysis, where the multiple data modalities are drawn from a common Bayesian linear model with some latent shared components but are not aligned (or registered) to coincide sample by sample.

The contribution consists both in a novel algorithm minimizing a min-max distribution matching criterion somewhat inspired by GAN ideas and in sets of conditions under which such unaligned factor models can still be identifiable (up to fundamental permutation and scale indeterminacy). The significance of the problem that is addressed as well as the value and originality of both of these contributions have been recognized by the reviewers. The reviewers however made many relevant comments about weaknesses of the original submission that need to be taken into account by the authors. This is all the more important that several reviewers have raised their score following the discussions, taking into account the responses of the authors, including some additional experimental results.

Given the target selectivity set for the conference, the submission is close to the decision threshold but in light of priority given to the novelty of the content and of the clear path for improving the submission based on the exchanges with the reviewers, I am leaning toward acceptance at the conference.
Decision: Accept (poster)