Identifiable Shared Component Analysis of Unpaired Multimodal Mixtures

Authors: subash  timilsina
sagar  shrestha
xiao  fu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 01:12:47

Abstract: A core task in multi-modal learning is to integrate information from multiple feature spaces (e.g., text and audio), offering modality-invariant essential representations of data. Recent research showed that, classical tools such as canonical correlation analysis (CCA) provably identify the shared components up to minor ambiguities, when samples in each modality are generated from a linear mixture of shared and private components.  Such identifiability results were obtained under the condition that the cross-modality samples are aligned/paired according to their shared information. This work takes a step further, investigating shared component identifiability from multi-modal linear mixtures where cross-modality samples are unaligned.  A distribution divergence minimization-based loss is proposed, under which a suite of sufficient conditions ensuring identifiability of the shared components are derived. Our conditions are based on cross-modality distribution discrepancy characterization and density-preserving transform removal, which are much milder than existing studies relying on independent component analysis. More relaxed conditions are also provided via adding reasonable structural constraints, motivated by available side information in various applications. The identifiability claims are thoroughly validated using synthetic and real-world data.

Main paper: 
 introduction: 
 The same data entities can often be represented in different feature spaces (e.g., audio, text and image), due to the variety of sensing modalities or domains. Learning common latent components of data from multiple modalities is well-motivated in representation learning. The shared components are considered modality-invariant essential representations of data, which can often enhance performance of downstream tasks by shedding modality-specific noise [1] [2] [3] [4] and avoiding over-fitting [5] [6] [7] .A prominent theoretical aspect of shared component learning lies in identifiability of the components of interest. The literature posed an intriguing theoretical question [1, 2, 8] : If every modality of data is represented by a linear mixture of shared and private components with an unknown mixing system, are the shared components identifiable (up to acceptable ambiguities)? Such component identification problems are often nontrivial due to the ill-posed nature of any linear mixture model (see, e.g., [9] [10] [11] [12] [13] [14] ). Interestingly, the work [1] showed that using the classical canonical correlation analysis (CCA) provably find the shared components up to rotation and scaling. In fact, shared component identification from multimodal/multiview linear mixtures were considered in various contexts (see, e.g., [15] [16] [17] [18] ), although some of these works did not model private components. The identifiability results in [1, 2] were generalized to nonlinear mixture models as well [4, 19] . The shared component identification perspective was also related to the success of representation learning in self-supervised learning (SSL) [5] [6] [7] .Nonetheless, the treatment in [1, 2] and the related works [15] [16] [17] all assumed that the cross-modality data are aligned (i.e., paired) according to their shared components. In many applications, such as cross-language information retrieval [20] [21] [22] , domain adaptation [23] [24] [25] , and biological data translation [26, 27] , aligned cross-modality data are hard to acquire, if not outright unavailable. A natural question is: When the multimodal linear mixtures are unaligned, can the shared latent components still be provably identified under reasonably mild conditions?Existing Studies. Theoretical characteristics of unaligned multimodal learning were studied under various settings. The work [28] considered a case where one modality is a linear transform of another modality, and showed that the linear transformation is potentially identifiable. The recent work [29] extended this model to a nonlinear transform setting. However, these works did not consider latent component models-yet the latter are more versatile in many ways, e.g., facilitating one-to-many cross-domain translations [30, 31] . The work [32] considered unaligned mixtures of shared and private components, but the assumptions (e.g., the availability of a large amount of modalities) to ensure identifiability may not be easy to satisfy. The most related work is perhaps [8] . But their approach also relied on somewhat stringent assumptions, e.g., that all the latent components are element-wise statistically independent with at most one component being Gaussian. This is because their procedure had to invoke the classical independent component analysis (ICA) [33] .Contributions. In this work, we provide a suite of sufficient conditions under which the shared components can be provably identified from unaligned multimodal linear mixtures up to reasonable ambiguities. The model and identification problem are referred to as unaligned shared component analysis (unaligned SCA) in the sequel.(i) An Identifiable Learning Loss for Unaligned SCA. We propose to tackle the unaligned SCA problem by matching the probability distributions of linearly embedded multi-modal data. We show that under reasonable conditions, the linear transformations identifies the shared components up to the same ambiguities as those in the aligned case [1, 2] . The conditions are considerably milder compared to the existing unaligned SCA work [8] .(ii) Enhanced Identifiability via Structural Constraints. We come up with two types of structural constraints, motivated by available side information in applications, to further relax the identifiability conditions. Specifically, we look into cases where the multi-modal data have similar linear mixing systems and cases where a few cross-domain aligned samples available. We show that by adding constraints accordingly, unaligned SCA are identifiable under much milder conditions.Our contributions primarily lie in identifiability analysis. Nonetheless, we also show the usefulness of our results in real-world applications, namely, cross-lingual word retrieval, genetic information alignment and image data domain adaptation. Particularly, it shows that our succinct multimodal linear mixture model can effectively post-process outputs of pre-trained encoders, e.g., those in [34, 35] , to improve data representations and enhance downstream task performance.Notation. Notation definitions can be found in Appendix A. 
background: 
 Generative Model of Interest. Following the classical settings in [1, 2, 15, 16, 18] , we consider modeling the multi-modal data as linear mixtures. More specifically, we adopt the model in [1, 2] that splits the latent representation of data into shared components and private components:x (q) = A (q) z (q) , z (q) = [c ⊤ , (p (q) ) ⊤ ] ⊤ , q = 1, 2,where x (q) ∈ R d (q) represents the data from the qth modality, z (q) ∈ R dC+d (q) centering. Note that the positions of c and p q are not necessarily arranged as [c ⊤ , (p (q) ) ⊤ ] ⊤ (more generally, z (q) = Π (q) [c ⊤ , (p (q) ) ⊤ ] ⊤ with an unknown permutation matrix Π (q) ). However, the representation in (1) is without loss of generality as one can define A (q) := A (q) (Π (q) ) ⊤ to reach the representation in (1) . For all the domains, we have c ∼ P c , p (q) ∼ P p (q) ,where P c and P p (q) represent the distributions of the shared components and the domain-private components, respectively. Under (1) , the two different range spaces range(A (q) ) for q = 1, 2 represent two feature spaces. Then latent p (q) further distinguishes the modalities and often has interesting physical interpretation. For example, some vision literature use c to model "content" and p (q) "style" of the images [31, 36] . In cross-lingual word embedding retrieval [2] , c represents the semantic meaning of the words, while p (q) represents the language-specific components. The goal of SCA boils down to finding linear operators to recover c to a reasonable extent.Aligned SCA: Identifiability of CCA and Extensions. Learning c without knowing A (q) is a typical component analysis problem. Learning latent components from linear mixture models (LMMs) like x = Az lacks identifiability in general, due to the bilinear nature of the models. This is because one can find an infinite number of invertible matrices B such that x = ABB -1 z. Then, both (A, z) and (AB, B -1 z) can fit to the data x, making the problem ill-posed in terms of solution uniqueness; see, e.g., [9, 37] and more discussions in Sec. 5. Nonetheless, the works [1, 2] studied the identifiability of c under the model (1), using the assumption that the cross-modality samples share the same c are aligned. In particular, [1] formulated the c-identification problem as a CCA problem: minimizeEQUATIONwhere Q (q) ∈ R dC×d (q) . The expectation in (3a) is taken from the joint distribution of the aligned pairs P x (1) ,x (2) , where every pair (x (1) , x (2) ) shares the same c. The formulation aims to find Q (q) such that the transformed representations of the aligned pairs Q (1) x (1) and Q (2) x (2) are equal. In [1] , it was shown thatEQUATIONunder mild conditions (see Appendix E.1 for details), where ( Q, Q) is an optimal solution of the CCA formulation and Θ is a certain non-singular matrix. Eq. ( 4) means thatQ (q)finds the range space where c lives in, i.e., range(A (q) 1:dC ) under our notation. Unaligned SCA: Existing Result and Theoretical Gap. The work in [8] studied the identifiability of c under (1) when x (1) and x (2) are unaligned. Their approach works under the condition that the elements of z (q) = [c ⊤ , (p (q) ) ⊤ ] ⊤ are mutually statistically independent. There, z (q) = Π (q) Σ (q) z (q) is assumed to have been estimated by ICA, where Π (q) and Σ (q) represent the scaling and permutation ambiguities, respectively, which cannot be removed by ICA. The work [8] assumed Σ (q) = I by imposing a unit-variance assumption on all the z (q) i 's. Then, a cross-domain matching algorithm is used to match the shared elements in z (1) and z (2) . The formulation can be summarized as finding d C pairs of non-repetitive (i, j) such that e ⊤ i z (1) and e ⊤ j z (2) have identical distributions, where e i is the ith unit vector. Denote c(1) m = e ⊤ im z (1) and c(2)EQUATIONwhere k ∈ {+1, -1} and π is a permutation of {1, . . . , d C } (see details in Appendix E.2 summarized from [8] ). This method effectively applies ICA to each modality, and thus the ICA identifiability conditions [33] have to met by x (1) and x (2) individually. However, if one only aims to extract Θc as in CCA, these assumptions appear to be overly stringent. -3 -2 -1 0 1 2 3 4 -4 -3 -2 -1 0 1 2 3 4 Θ (1) c -3 -2 -1 0 1 2 3 4 -4 -3 -2 -1 0 1 2 3 4 Θ (2) c 
proposed approach: 
 Unaligned SCA: Problem Formulation We assume that x (q) 's are zero-mean. We use the notation from CCA in (3a). However, since no aligned samples are available, we replace the sample-level matching objective with a distribution matching (DM) module, as DM can be carried out without sample level alignment:EQUATIONEQUATIONEQUATIONwhere "u(d)=== v" means the distributions of u and v are the same.The formulation in (6) can be realized using various distribution matching tools, e.g., maximum mean discrepancy (MMD) [38] and Wasserstein distance [39] . We use the adversarial loss:minEQUATIONThe first and second terms comprise the adversarial loss from GAN [40] . It finds Q (q) to confuse the best-possible discriminator f : R dC → R, where f is represented by a neural network in practice. It is well known that the minimax optimal point of the first two terms is attained when (6b) is met [40] . We use R(Q (q) ) = ∥Q (q) E[x (q) (x (q) ) ⊤ ](Q (q) ) ⊤ -I∥ 2 F to "lift" the constraints. This way, the learning criterion in (7) can be readily handled by any off-the-shelf adverserial learning tools.Identifiability of Unaligned SCA As we saw in Theorem 4, CCA identifies Q (q)x (q) = Θc where Θ ∈ R dC×dC under the settings of aligned SCA. Establishing a similar result for unaligned SCA is much more challenging. First, it is unclear if (6b) could disentangle c from p (q) . In general, Q (q) x (q) could still be a mixture of c and p (q) yet (6b) still holds (e.g., when both c and p (q) are Gaussian.) Second, even when the disentanglement is attained via enforcing (6b) and we have Q (q) x (q) = Θ (q) c, in general it does not hold that Θ (1) = Θ (2) . This is because Θ (1) c (d)=== Θ (2) c where Θ (1) ̸ = Θ (2) can still be perfectly met (e.g., when P Θ (q) c is symmetric Gaussian in Fig. 1 ). However, Θ (1) ̸ = Θ (2) means that the extracted representations from the two modalities are not matched. This creates challenges for applications like cross-domain information retrieval, language translation, or domain adaptation.Our intuition is as follows: If the two distributions P c,p (1) and P c,p (2) are very different, thenQ (1) x (1) (d)=== Q (2) x (2) cannot hold unless Q (q) A (q) = [Θ (q) , 0]. We use the following to characterize such difference between the joint distributions: Assumption 1 (Modality Variability). For any two linear subspaces P (q) ⊂ R dC+d (q) P , q = 1, 2, with dim(P (q) ) = d (q) P , P (q) ̸ = 0 × R d (q) P and linearly independent vectors {y (q) i ∈ R dC+d (q) P } dC i=1 , q = 1, 2, the sets A (q) = conv{0, y (q) 1 , . . . , y (q) dC } + P (q) , q = 1, 2, are such that if P c,p (q) [A (q) ] > 0 for q = 1 or q = 2, then there exists a k ∈ R such that the joint distributions P c,p (1) [kA (1) ] ̸ = P c,p (2) [kA (2) ], where kA (q) = {ka | a ∈ A (q) }.The condition in Assumption 1 is a geometric way to characterize the difference between P c,p (1) and P c,p (2) -if the joint distributions have different measures for all possible "stripes", each being a direct sum of a subspace and a convex hull (see Fig. 2 ), then P c,p (1) and P c,p (2) must be very different. Note that the difference is contributed by the modality-specific term p (q) , and thus we call this condition "modality variability". Modality variability is similar to the "domain variablity" used in [32, 41] -both characterize the discrepancy of the joint probabilities P c,p (1) and P c,p (2) . However, there are key differences: The domain variability was defined in a unified latent domain over arbitrary sets A, which could be stringent. Instead, we use the fact that (6) relies on linear operations to construct A (q) , which makes the condition defined over a much smaller class of sets-thereby largely relaxing the requirements. Restricting A (q) to be stripes also makes the modality variability condition much more relaxed compared to the domain variability condition.We show the following: Theorem 1. Under Assumption 1 and the generative model in (1), denote any solution of (6) as Q (q) q = 1, 2. Then, if the mixing matrices A (q) are full column ranks and E[cc ⊤ ]) is full rank, we haveQ (q)x (q) = Θ (q) c. In addition, assume that either of the following is satisfied: Then, we have Q (q)x (q) = Θc, i.e., Θ (q) = Θ for all q = 1, 2, where Θ (q) . In Theorem 1, Assumption 1 is used to guarantee Q (q)x (q) = Θ (q) c and either of conditions (a) or (b) is used to make sure Θ (1) = Θ (2) . Note that both (a) and (b) are milder than those in [8] (cf. Theorem 5) , where the element-wise statistical independence of z (q) was relied on to find shared representation of x (1) and x (2) . The proof is in Appendix B.Numerical Validation. In Fig. 3 , the top and bottom rows validate Theorem 1 under the assumptions in (a) and (b), respectively. In the top row, we set c ∈ R 2 , where c 1 is sampled from Gaussian mixtures with three components and c 2 is sampled from a Gamma distribution (and c 1 ⊥ ⊥ c 2 ). We set p (1) and p (2) as one-dimensional Laplacian and uniform distributions. In the bottom row, the dimensions of c and p (q) for q = 1, 2 are unchanged, but their distributions are replaced in order to satisfy conditions in (b) (see details in Appendix F). One can see that clearly c (q) = Θc; i.e., the learned c (q) for q = 1, 2 are identically rotated and scaled versions of c.A remark is that our framework still allows to identify individual c i 's as in [8] .Corollary 1. Under the conditions in Theorem 1 (a), Assume that at most one c i for i ∈ [d C ] is Gaussian. Then, the components of c are identifiable up to permutation and scaling ambiguities by applying ICA to c (q) = Q (q) x (q) for either q = 1 or q = 2.The corollary means that to identify individual c i , using our formulation still enjoys much milder conditions relative to [8] . Specifically, our condition only specifies the independence among elements of c, but the condition in [8] needs that all the elements in z (q) = [c ⊤ , (p (q) ) ⊤ ] ⊤ are independent.  
enhanced identifiability via structural constraints: 
 Theorem 1 was well-supported by the synthetic data experiments. However, our experiments found that the learning criterion (6) often struggles to produce sensible results in some applications. Our conjecture is that the Assumptions in Theorem 1 (a) and (b) might not have been satisfied by the real data under our tests. Although they are not necessary conditions for identifiability, these conditions do indicate that the requirements to guarantee identifiability of unaligned SCA using (6) are nontrivial to meet. In this section, we explore a couple of structural constraints arising from side information in applications to remove the need for the relatively stringent assumptions on c.Homogeneous Domains. The first structural constraint that we consider is A (q) = A for q = 1, 2. This model is motivated by the fact that advanced representation learning tools, e.g., selfsupervised learning tools (e.g., SimCLR [42] ) and foundation models (e.g., CLIP [35] ), are already capable of mapping the data clusters to a shared linearly separable space-which indicates that the representations share a subspace, i.e., x (q) ≈ Az (q) . Under such circumstances, the proposed model and method can be used to further process the data by discarding the private components in the latent representation.Here, we consider the special case of generative process in (1) where,EQUATIONUnder this model, we look for the shared components by solving (6) with a single Q = Q (1) = Q (2) . We use the following version of the modality variability condition:Assumption 2. For any linear subspace P ⊂ R dC+dP , d P = dEQUATIONP , with dim(P) = d P , P ̸ = 0 × R dP and linearly independent vectors {y i ∈ R dC+dP } dC i=1 , q = 1, 2, the sets A = conv{0, y 1 , . . . , y dC } + P, q = 1, 2. are such that if P c,p (q) [A] > 0 for q = 1 or q = 2, then the joint distributions P c,p (1) [kA] ̸ = P c,p (2) [kA] for some k ∈ R. Theorem 2. Consider the mixture model in (8) . Assume that rank(A) = d C + d P and rank(E[cc ⊤ ]) = d C , and that Assumption 2 holds. Denote Q as any solution of (6) by constraining Q = Q (1) = Q (2) . Then, we have Qx (q) = Θc.One can see that the conditions (a) and (b) in Theorem 1 are completely removed, if the structure A (1) = A (2) is imposed. In fact, the result in Theorem 2 is expected and readily seen from the proof of Theorem 1, as the cause for Θ (1) ̸ = Θ (2) is the use of two different Q (q) 's. Nonetheless, this simple variation will prove useful in a series of real-data experiments.The Weakly Supervised Case. Another way to add structural constraints is to use available auxiliary information. For example, some datasets have weak annotations and selected pairs; see, e.g., [43, 44] .). There exist a set of available aligned samples (xEQUATIONEQUATIONℓ ) share the same c ℓ .The condition can be added into our formulation in (6) as a constraint, i.e., Q (1) x(1)EQUATIONIn the next theorem, we show that the incorporation of aligned samples helps relax conditions (a) and (b) in Theorem 1:EQUATIONℓ ) are available, that A (q) for q = 1, 2 have full column rank, and that P c is absolutely continuous. DenoteEQUATION) as any optimal solution of (6) under the constraint (9) . Then, we have Q (q)x (q) = Θc.The proof and synthetic data validation can be found in Appendices D and F, respectively. Note that to realize (9) , one only needs to add a regularization term β ℓ∈L ∥Q (1) x(1) ℓ -Q (2) x (2) ℓ ∥ 22 to the loss in (7) , where β ≥ 0 is a tunable parameter. The overall loss is still differentiable and thus can be easily handled by gradient based approaches.A remark is that our weakly supervised formulation can use as few as d C pairs of (xEQUATIONℓ ) to establish identifiability of shared component. In contrast, CCA requires at least d C + d Private Component Identifiability. Although our focus is shared component identification, we show that private components are also identifiable with additional assumptions; see Appendix H. 
related works: 
 Identifiability of Component Analysis under Linear Mixture Models. Various component analysis models were studied in the past several decades, e.g., principal component analysis [45] , independent component analysis [33] , sparse component analysis [10, 12] , bounded component analysis [13] , simplex component analysis [46, 47] , and polytopic component analysis [14] -motivated by their applications in dimensionality reduction, representation learning, and latent variable identification (see, e.g., topic mining [48, 49] , hyperspectral unmixing [46, 47] , audio/speech separation [33] and community detection [50] ). The classical component analysis tools mostly study a single modality. The identifiability results under these models are well developed and documented.Identifiability of Shared Components from Aligned Modalities. Modeling multimodal data as two or more linear/nonlinear mixtures of latent components was considered in CCA-related works [1, 2, 15, 19] , independent vector analysis (IVA) works [17, 18] , multiview ICA works [16, 51] , and SSL works [5] [6] [7] 52] . Partitioning the latent components into shared and private blocks was considered in [1, 2, 4, 5, 7, 52] . Shared component identifiability was established at the block level (see, e.g., [1, 2, 5] ) and the individual component level (e.g., [51] ) in these works. Nonetheless, they all rely on completely paired/aligned cross-modality samples, which we do not use in this work.Distribution Matching and Unaligned Multimodal Analysis. Using distribution matching in unaligned multimodal data analytics for different purpose also has a long history; see applications in imageto-image translation [53] , domain adaptation [54] , cross-platform image super-resolution [55] , and cross-domain information retrieval [21] . The recent works [56] and [57] pointed out the identifiability challenge and the existence of density-preserving transforms. The works in [28, 29] started studying the uniqueness issues in distribution matching. However, the latent mixture models were not studied in this line of work.Identifiability of Unaligned SCA. The works in [32, 41] investigated the shared component identifiability when the multimodal data are nonlinear mixtures of content and style (which are shared and private components, respectively) under the same mixing system. Hence, our identical linear mixing case in Theorem 2 can be understood as a special case of theirs. But their analysis relies on the assumption that all the latent components are statistically independent, which is much stronger than our conditions in Theorem 2. Their results also require that there are a large amount of modalities available. But our proof works for just two modalities. The most related work is [8] , which uses the model in (1) in the context of multi-view causal graph learning. As discussed before, their assumptions on the latent components are much stronger than ours (see Corollary 1 and Appendix E.2). Application (i) -Domain Adaptation. We first test the proposed methods over a number of domain adaptation (DA) tasks. In DA, we have the source domain data {x (1) } and the target domain {x (2) }, respectively. Only the source domain data have labels and the two domains are unaligned. We hope to use our method to find shared representations of source and target, and thus the classifier trained using source data can also work well on the target data.Dataset: We use two standard benchmarks of DA, i.e., Office-31 [58] and Office-Home [59] . The Office-31 dataset has 4652 images and 31 categories from three domains, namely, Amazon images (A), Webcam images (W) and DSLR images (D). The Office-Home dataset contains 15,500 images with 65 object classes from four domains, i.e., Artistic images (Ar), Clip art images (Cl), Product images (Pr), and Real-world images (Rw).Setup: We first test the homogeneous domain model in Sec. 4. The images are pre-processed using a ResNet50-based image encoder pre-trained over ImageNet1k [42] . As mentioned, it was observed that self-supervised representation encoders find embeddings that are linearly separable [42] , which justifies the use of the model x (q) ≈ Az (q) in the embedding domain. After pre-processing, each image is represented by d (q) = 2048 features for q = 1, 2. We set d C = 256 for Office-31 and d C = 512 for Office-Home. More detailed settings are in Appendix G.The baselines are representative DA methods, namely, DANN [25] , MDD [60] , MCC [61] , SDAT [62] , and ELS [63] . All the baselines use the same encoder-produced embeddings as inputs; see Appendix G.1 for their configurations. We also use ResNet encoder's outputs as an extra baseline as it learns informative and transferable features from the ImageNet-1K dataset. We follow the training strategies adopted by the baselines [25, 60, 62] to learn a classifier jointly with the shared latent components. This strategy arguably regularizes towards more classification-friendly geometry of the shared features. Therefore we append a cross-entropy (CE) based classifier training module to our loss in (7) that learns our feature extractor Q. More details are in Appendix G.1.The evaluation metric is the classification accuracy in the target domain {x (2) }. The classifier is trained with the projected source domain Qx (1) and the associated labels.Result: Table 1 and Table 2 show the classification accuracy (mean±std) on Office-31 and Office-Home, respectively. The results are averaged over 5 runs. One can observe that the proposed method offers the best and second best performance in most of the cases. In some tasks (e.g.,"A→W", "Ar→Cl", "Ar→Pr" and "Rw→Cl"), the proposed method outperforms the best-performing baselines by at least 2% in accuracy.More results on the DA task can be found in Appendix G.1.Application (ii) -Single Cell Sequence Analysis. In biomedical research, it is desired to fuse measurements from multiple sensorial modalities of the same cells, in order to have better characterizations of the cells. However, obtaining multimodal data of the same cells simultaneously is almost impossible, due to the sensing limitations. Therefore, many methods are proposed in the literature for aligning unpaired multi-modal single cell data [27, 64, 65] . We focus on the following two modalities of single-cell data [66] : (1) the RNA sequences {x (1) } and (2) the ATAC sequences {x (2) }. Dataset: We use human lung adenocarcinoma A549 cells data from [66] . The dataset contains 1,874 samples of RNA sequences {x (1) } and ATAC sequences {x (2) }. Each data set is split into 1534 training samples and 340 testing samples as in [27] . The data have labeled associations between the two domains-part of which will be used to test our weakly supervised formulation. For this experiment, features of RNA sequence and the ATAC sequence have dimensions of d (1) = 815 and d (2) = 2613, respectively. We set d C = 256. We use our weakly supervised formulation as shown in (9) . We uniformly sampled a set of indices from the training set to serve as L.Baseline and Metric: We use weakly supervised algorithm, namely, cross-modal autoencoder (CM-AE) work in [27] , as a baseline, which also learns the shared representation between unaligned RNA and ATAC sequences. We use the K-nearest neighbor (k-NN) accuracy to evaluate the performance as suggested in [27] .Result: The plot in Fig. 4 shows the k-NN accuracy of the methods on the test set. Results show the mean and standard deviation over 10 runs, each having a different random initialization. For the proposed method, we vary the number of available paired samples from 0 (cf. Theorem 1) to d C = 256 (cf. Theorem 3). Note that the baseline uses more (i.e., 256 and 770) paired samples. It also needs additional class labels, i.e., y (q) i for the ith sample x (q)i . Here, y (q) i represents the number of hours (0, 1 or 3) of cell treatment [27, 66] . The proposed method without any supervision (i.e., 0 paired samples) already exhibits around 3 times greater k-NN accuracy compared to the baseline for all k. Moreover, including just one paired sample boosts the k-NN accuracy of the proposed method to around 5 times higher than the baseline for all k. Finally, one can observe a steadily increasing k-NN accuracy with respect to the number of available paired samples. This corroborates with our Theorem 3.Application (iii) -Multi-lingual Information Retrieval. We also evaluated our method on a word embedding association problem from the natural language processing literature [20, 21] . This task aims to associate high-dimensional word embeddings across different languages according to their semantic meaning. The word embeddings in two languages are represented using two sets of vectors, i.e., {xi } I i=1 and {x(2) j } J j=1 . The postulate is that if x(1) i and x(2) j have the same meaning (e.g., both representing "cat") in two languages (e.g., English and German), they should share a latent components c.Dataset: We use the word embeddings from the MUSE dataset (https://github.com/ facebookresearch/MUSE) [21] . These monolingual word embedding are generated using fast-Text [67] and has dimensions of d (q) = 300 for q = 1, 2. The training dataset include 200,000 word embeddings in each language. In our experiment we set d C = 256. We follow the generative model under (8) and run the formulation in (7) to learn the linear transformation Q.Baseline: We use Adv [21] as the baseline which also uses distribution matching between two language domains. Unlike our method, Adv does not use linear mixture models.We follow [21] to use the average precision score calculated based on nearest neighbor (NN) and cross domain similarity local scaling (CSLS). Precision at k ("k precision") is computed by the number of times that one of the correct translations of source word is retrieved at top-k results (k ={1, 5, 10}). The final score is normalized to be in the range of 0 to 100, with 100 being the highest score indicating the best performance. To evaluate the performance, we use the same test data as in [21] . For each source and target language pair, this dataset includes 1,500 source word embeddings. The source embeddings are used to retrieve corresponding embeddings from a pool of 200,000 target word embeddings.Result: Table 3 reports the P@1 scores over the test data calculated for each source and target language pair. The languages are denoted as as en -English, es -Spanish, it -Italian, fr -French, de -Germany, ru -Russian, ar -Arabic and vi -Vietnamese. One can observe that the proposed method exhibits a better precision performance than that of Adv in most of the translation tasks. In particular, the proposed method significantly outperforms the baseline on the tasks en→ar, ar→en, en→vi and vi→en, showing at least 10% precision gains. Similarly, our method shows at least 5% improvements in both NN and CSLS based precision metrics in en→es and es→en tasks.More details and additional experiments can be found in Appendix G.3. 
conclusion: 
 In this work, we considered the problem of identifying shared components from unaligned multidomain mixtures. We proposed a learning loss that matches the distributions of linearly transformed data. Based on this loss, we came up with a suite of sufficient conditions to ensure the identifiability of shared components. Furthermore, we proposed modified models and losses that enjoy more relaxed conditions for shared component identifiability. This was achieved via introducing structural constraints, namely, the homogeneity of the mixing systems and the existence of weak supervision. Our theoretical claims were validated with both synthetic and real-world data, demonstrating soundness of the theorems and usefulness of the models/algorithms.Limitations. First, our conditions for shared component identification are sufficient. The necessary conditions are not underpinned, but necessary conditions assist understanding the limitations of the models and algorithms. Second, our methods were developed under the linear mixture model, which has limited expressiveness, and thus often requires pre-processing to approximately meet the model specification. We expect that results with similar flavors to be derived for nonlinear models in the future. Third, the results were derived under an unlimited data assumption. It would be interesting have a finite sample analysis. Finally, optimizing GAN-based losses is sensitive to hyperparameter settings. Back-propagation based minimax optimization occasionally fails to converge. More optimization-friendly losses and more stable algorithms are desirable in the context of distribution matching.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: We have provided the error bars for the experiment that is computationally less demanding. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: Theoretical paper. So not applicable in our case.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: See Appendix G.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: The notations used throughout the paper are summarized in the Table 4 .: Table 4 : Definition of notations. Notation Definition x, x, X scalar, vector and matrix x (q) variable from q-th domain x i , x i both represents i-th element of vector x X ij represents the element of i-th row and j-th column of matrix X x ⊤ , X ⊤ transpose of x and X |X | represents the cardinality of set X Null(X)represents the null space of matrix X conv(•)returns the convex hull of the given set dim(X ) denotes the dimension of subspace We restate the theorem here:X kA {ka | a ∈ A, k ∈ R} x + X {x + z | z ∈ X } X + Y {x + y | x ∈ X , y ∈ Y} A PreImg (X ) preimage of X ; {x | Ax ∈ X } [N ]Theorem 1 Under Assumption 1 and the generative model in (1) , denote any solution of (6) asQ (q)q = 1, 2. Then, if the mixing matrices A (q) are full column ranks andE[cc ⊤ ]) is full rank, we have Q (q)x (q) = Θ (q) c. In addition, assume that either of the following is satisfied: [-a 1 , a 1 ] × • • • × [-a dC , a dC ]. Further, suppose that c i (d) ̸ = kc j , ∀i ̸ = j, ∀k ∈ R and c i (d) ̸ = -c i , ∀i.Then, we have Q (q)x (q) = Θc, i.e., Θ (q) = Θ for all q = 1, 2, where Θ (q) . We will prove the theorem in following two steps. For the first step we will prove Q (q)x (q) = Θ (q) c and for second step we will employ either assumption (a) or (b) to prove that Θ (q) = Θ, ∀q = 1, 2.Let us define H (q) = Q (q) A (q) ∈ R dC×(dC+d (q) P ) . We want to show thatEQUATIONsince this will imply that that H (q) does not depend upon the style component. Combined with the fact that rank(H (q) ) = d C , this will imply that H (q) is an invertible function of the content component. To that end, consider the following line of arguments.Since the objective in (6) matches the distribution for latent random variables c (1) = Q (1) x (1) and c (2) = Q (2) x (2) , the following holds for any R c ⊆ R dC , ∀k ∈ R,EQUATION⇐⇒ P z (1) [HEQUATIONEQUATION(b)EQUATIONEQUATIONPreImg (R c )],where, HPreImg (R c ) := {z (q) | H (q) z (q) ∈ R c } is the pre-image of H (q) . (a) follows becauseP c (q) [kR c ] = P H (q) z (q) [kR c ] = P z (q) [H (q) PreImg (kR c )] [68, Section 2.2]. (b) follows because H (q)is a linear operator.Although (11) holds for any R c , we will see that it is sufficient to consider a special R c to prove (10) . To that end, take R c = conv{0, a 1 , . . . , a dC }, where a i ∈ R dC such that P c (q) [R c ] > 0. Let us take y (q) i ∈ R dC+d (q) P , such that H (q) y (q) i = a i . For reasons that will be clear later, we hope to show thatEQUATION1 , . . . , ydC } + Null(H (q) ). To that end, observe that for any r ∈ R c , we can represent r as,r = 1 d C + 1 dC i=1 w i a i , for some {w i } dC i=1 s.t. dC i=1 w i ≤ 1, ∀i.For both view q = 1, 2, we get,EQUATIONWe can write,EQUATION1 , . . . , yEQUATIONWe have that Null(H (q) ) ⊂ R dC+d (q) P is a linear subspace with dim(Null(H (q) )) = d (q) P . Let A (q) = H (q) PreImg (R c ). Note that P z (1) [kA (1) ] = P z (2) [kA (2) ], ∀k ∈ R (from (11) , and P z (q) [A (q) ] > 0 (by the construction of R c ). Further, the set A (q) is of the form conv{0, y (q) 1 , . . . , y (q) dC } + P (q) , because Null(H (q) ) is a subspace of dimension d (q) P , hence it satisfies the definition of P (q) . Hence, Assumption 1 implies that Null(H (q) ) = 0 × R d (q) P . Denoting the N th to M th columns of H (q) by H (q) (N : M ), the above is equivalent to sayingEQUATIONDenote, Θ (q) = H (q) (1 : d C ) ∀ q = 1, 2. Then, we can write,EQUATIONNext, we use Assumption (a) or (b) to show that Θ (1) = Θ (2) = Θ. To that end, note that the distribution matching constraint implies thatΘ (1) c (d) === Θ (2) c =⇒ c (d) === (Θ (1) ) -1 Θ (2) c. Hence M = (Θ (1) ) -1 Θ (2) is an invertible matrix such that c (d)=== M c. However, in the following, we will show that if either Assumption (a) or (b) is satisfied, then M = I, and thus Θ (1) = Θ (2) .We want to show that when Assumption (a) is satisfied, if M c(d) === c for any invertible M , then M = I. Note that M c = [m 1 . . . m dC ]    c 1 . . . c dC   .By Assumption (a), we have that the components of content are statistically independent c i ⊥ ⊥ c j , i ̸ = j , non-Gaussian, and has non-zero kurtosis. Then, according to cumulant multilinearity and additivity properties, the fourth order cumulant tensor Cum(M c) of M c has the following unique decomposition [69] ,EQUATION)where • is the outer product, κ i is the kurtosis of component c i , andm i , i ∈ [d C ] are the columns of M . Since M c (d)=== c, the following should holdEQUATIONEQUATIONe i is the ith column of identity matrix I.Because of statistical independence of components of c, the CP-decomposition of Cum(M c) = Cum(Ic) is unique [69] upto permutation and scaling ambiguities, i.e., M should be a permutation scaling matrix.Let M = ΠΣ where, Π ∈ R dC×dC is a permutation matrix and Σ = Diag(r 1 , . . . r dC ) ∈ R dC×dC is a diagonal scaling matrix.Finally, since c i ̸(d)=== kc j , ∀i ̸ = j, ∀k ∈ R, M has to to be identity matrix. To see the reason, for the sake of contradiction, suppose that either (i) there exist i, j∈ [d C ] × [d C ] and k ∈ R, with i ̸ = j such that [M c] i = kc j , or (ii) ∃i ∈ [d C ] such that [M c] i = kc i for some k ∈ R, k ̸ = 1. For case (i), since M c (d) === c, [M c] i (d) === c i , ∀i. Hence, [M c] i = kc j =⇒ [M c] i (d) === kc j =⇒ c i (d) === kc j ,which is a contradiction to the assumption c i Note that ∀c ∈ C, the set of points v i satisfy the following propertyEQUATION) =⇒ M c = dC i=1 α i (M v i ).Since the support of M c is C, this implies that ∀c ∈ C,c = dC i=1 α i (M v i ), for some -1 ≤ α i ≤ 1The last equation implies that the set of points M v i , ∀i ∈ [d C ] also satisfy property (19) . Hence,for each i ∈ [d C ], M v i = ±v j for some unique j ∈ [d C ].Note that j should be unique for each i because M is invertible, hence M cannot map two orthogonal vectors v i and v k , i ̸ = k, to the same vector ±v j with same or different signs.Let V = [v 1 , . . . , v dC ] T . Then one can write M V = V ΣΠ,where Σ is some diagonal matrix with diagonal entries from {+1, -1} and Π is a permutation matrix.Then the above impliesM ΛI = ΛIΣΠ =⇒ M = ΛΣΠΛ -1 .Hence M is a permutation and scaling matrix.Finally, by the same argument presented in last paragraph of Sec. B.2 (i.e., proof with Assumption (a)), we conclude that M is an identity matrix.We restate the theorem here:Theorem 2 Consider the mixture model in (8) . Assume that rank(A) = d C + d P and rank(E[cc ⊤ ]) = d C , and that Assumption 2 holds. Denote Q as any solution of ( 6) by constraining Q = Q (1) = Q (2) . Then, we have Qx (q) = Θc.One can follow the same argument as in the step 1 of proof in B.H = QA ∈ R dC×(dC+dP) .We want to show thatEQUATIONsince this will imply that that H does not depend upon the style component. Combined with the fact that rank(H) = d C , this will imply that H is an invertible function of the content component. To that end, consider the following line of arguments.Since the objective in ( 6) matches the distribution for latent random variables c (1) = Qx (1) and c (2) = Qx (2) , the following holds for anyR c ⊆ R dC ,, ∃ k ∈ R P c (1) [kR c ] = P c (2) [kR c ],(a)EQUATIONEQUATIONwhere,H PreImg (R c ) := {z | Hz ∈ R c } is the pre-image of H. (a) follows because P c (q) [kR c ] = P Hz (q) [kR c ] = P z (q) [H PreImg (kR c )] [68, Section 2.2]. (b) follows because H is a linear operation.Although (21) holds for any R c , we will see that it is sufficient to consider a special R c to prove (20) .To that end, take R c = conv{0, a 1 , . . . , a dC }, where a i ∈ R dC such that P c (q) [R c ] > 0. Let us take y i ∈ R dC+dP , such that Hy i = a i . For reasons that will be clear later, we hope to show thatH PreImg (R c ) = conv{0, y 1 , . . . , y dC } + Null(H).To that end, observe that for any r ∈ R c , we can represent r as,r = 1 d C + 1 dC i=1 w i a i , for some {w i } dC i=1 s.t. dC i=1 w i ≤ 1, ∀i.For both view q = 1, 2, we get,EQUATIONWe can write,EQUATIONWe have that Null(H) ⊂ R dC+dP is a linear subspace with dim(Null(H)) 21), and P z (q) [A] > 0 (by the construction of R c ). Further, the set A is of the form conv{0, y 1 , . . . , y dC } + P, because Null(H) is a subspace of dimension d P , hence it satisfies the definition of P. Hence, Assumption 2 implies that Null(H) = 0 × R dP .= d P . Let A = H PreImg (R c ). Note that P z (1) [kA] = P z (2) [kA], ∀k ∈ R (from (Denoting the N th to M th columns of H by H(N : M ), the above is equivalent to sayingEQUATION)Denote,Θ = H(1 : d C ) ∀ v = 1, 2.Then, we can write,EQUATIONThis concludes the proof.We restate the theorem here:EQUATIONℓ ) are available, that A (q) , q = 1, 2 have full column rank, and that P c is absolutely continuous.EQUATION) as any optimal solution of (6) under the constraint (9) . Then, we haveQ (q)x (q) = Θc.From our objective in (6), we obtainEQUATION)Using Assumption 1 and following the proof of step 1 in Theorem B, we can obtain:Q (q) x (q) = Θ (q) c, ∀q = 1, 2,for some invertible matrices Θ (q) , ∀q. Hence,EQUATION)Hence we can have linear transformation M := (Θ (1) ) -1 Θ (2) which has same probability density as P c . However, the sample matching constraint ( 9), for ℓ-th sample implies that Q (1) x(1)ℓ = Q (2) x (2) ℓ =⇒ Θ (1) c ℓ = Θ (2) c ℓ =⇒ c ℓ = (Θ (1) ) -1 Θ (2) c ℓ =⇒ c ℓ = M c ℓ . Let C = [c 1 . . . c Np ].Then the above implies:C = M C =⇒ (M -I)C = 0.Now we show that C is a full row rank matrix, which implies that M -I = 0 =⇒ M = I. To that end, note that random variables x (1) i and x(2) i being i.i.d implies that c (i) are i.i.d from P c . This implies that for anyEQUATION)This is because span({c n1 , . . . ,c n d C -1 }) for n j ∈ [|L|], is a lower dimensional subspace in R dC , which has zero probability under absolutely continuous distribution P c . Hence any d C out of |L| column vectors in C are linearly independent with probability 1.This concludes the proof.E.1 Identifiability of CCA Theorem 4 (Identifiability of Aligned SCA via CCA [1] ). Under (1), assume that every aligned pair (x (1) , x (2) ) share the same c, and that A (q) has full column rank. Also assume that there exists an N -sample set {ℓ 1 , . . . , ℓ N } such that [C ⊤ , (P (1) ) ⊤ , (P (2) ) ⊤ ] ∈ R N ×(dC+d (1) P +d(2) P ) has full column rank, where C = [c ℓ1 , . . . c ℓ N ] ∈ R dC×N and PEQUATION, Q) as an optimal solution of the CCA formulation. Then, we we haveQ (q) x (q) = Θc,where Θ is nonsingular.In the above theorem, one can see that N ≥ (d C + d We summarize the result in [8] in the following Theorem 5 (Identifiability of Unaligned SCA via ICA [8] ). Under (1), assume that the following are met: (i) The conditions for ICA identifiability [33] is met by each modality, including that the components of z (q) = [c ⊤ , (p (q) ) ⊤ ] ⊤ are mutually statistically independent and contain at most one Gaussian variable. In addition, each zEQUATIONP ], i ̸ = j. Then, assume that (i m , j m ) are obtained by ICA followed by cross domain matching (see the part on Unaligned SCA in Section 2 ) for m = 1, . . . , d C .(1) m = e ⊤ im z (1) and cm = e ⊤ jm z (2) . We have the following:EQUATIONwhere k ∈ {+1, -1} and π is a permutation of {1, . . . , d C }.Hyperparameter Settings: We use Adam optimizer [70] to solve (7) and learn matrices Q (q) , q = 1, 2 and the discriminator f . We set the initial learning rate of matrix and discriminator to be 0.009 and 0.00008 respectively. We set the λ = 0.1 in (7) to enforce (6c). For weak supervision experiment in F, we set β = 0.01 in (9) . We generate total of 100,000 samples in each domain. For our experiment we set the batch size to be 1,000 and run (7) for 50 epochs. Our discriminator is a 6-layer multilayer perceptron (MLP) with hidden units { 1024, 521, 512, 256, 128, 64 } in each layer. All the layers use leaky ReLU activation functions [71] with a slope of 0.2 except for the last layer which has sigmoid activations. We include a label smoothing coefficient of 0.2 in the discriminator predictions as suggested in [40] .Additional Details for Validation of Theorem 1 in Sec. 3: Here we explain the data generation details of the result shown in Fig. 3 . For the result in top row, we sample c 1 from a Gaussian mixture with three Gaussian components. Each component follows a normal distribution N (µ, 2) where µ ∼ N (0, 10). The second component, i.e., c 2 , is independently sampled from the gamma distribution Gamma(1, 3). The private components are sampled from p (1) ∼ Laplace(1.0, 6.5) and p (2) ∼ Uniform[-10, 10], both only having one dimension. In the bottom row, we sample c ∈ R 2 ∼ VonMises(2.5, 2.0) distribution. The private components satisfy p (1) ∼ Laplace(1.0, 6.5) and p (2) ∼ Gamma(0.5, 3.0). Each element of mixing matrices are sampled from A (q) ij ∼ N (0, 1), q = 1, 2. The readers are referred to Table 4 for the definition of notations used for distributions.Validation of Theorem 1 under different sample sizes and imbalanced data: Here we observe the shared component identification performance of the proposed method numerically. We conducted two experiments in different settings. First, we vary the sample sizes in both modalities, but the two modalities have the same sample size. Second, we only vary the sample size of modality 2 while keeping the sample size of modality 1 fixed. This way, we create the data imbalance between modalities. Note that the shared components are identified if the following two conditions are met:1. Q (q) A (q) = [Θ, 0], i.e., Θ (1) = Θ (2) = Θ and 2. ∥Q (q) A (q) (d C : d C + d (q) P )∥ F = 0.Therefore, we use the above as our performance evaluation metrics. For the following experiments (Table 5 and 6 ), we generate the data for the two modalities by sampling a two-dimensional content c ∼ VonMises(2.5, 2.0) and private components from p (1) ∼ Laplace(1.0, 6.5) and p (2) ∼ Gamma(0.5, 3.0). The elements of the mixing matrices are sampled as A (q) ij ∼ N (0, 1), q = 1, 2. We report the mean and standard deviation of ∥ Θ (1) (1 : d C )-Θ (2) (1 : d C )∥ F and 1/2 2 q=1 ∥ Θ (q) (d C : d C + d (q) P )∥ F obtained from 5 different runs.Table 5 shows the performance of SCA and CCA under different sample sizes (i.e., N ). One can see that the proposed method (SCA) clearly identifies the shared components even when only 100 samples are available. The performance starts to deteriorate when N ≤ 50, probably because the min-max optimization problem is difficult to solve with very few samples. CCA does not really work under this setting as it needs aligned cross-domain samples.Table 6 shows the performance of SCA in the cases where two modalities have unbalanced data sizes. The number of samples in the first modality is fixed to 100,000 while the second modality's data size varies from 10,000 to 10 samples. The data generation process remains the same as in the previous experiment. One can see that even under obvious cross-domain data size imbalance (e.g., 100,000 to 1,000), the proposed method performs reasonably well in terms of shared component identification. # samples in modality 2 ∥ Θ (1) (1 :N ∥ Θ (1) (1 : d C ) -Θ (2) (1 : d C )∥ F 1/2 2 q=1 ∥ Θ (q) (d C : d C + d (q) P )∥ F SCA CCA SCAEQUATIONP )∥ F 10,000 0.020 ± 0.018 0.018 ± 0.005 1,000 0.065 ± 0.029 0.026 ± 0.015 100 0.145 ± 0.051 0.081 ± 0.049 10 1.290 ± 0.239 0.293 ± 0.064Validation of Theorem 3. Fig. 5 presents numerical validation for Theorem 3.We set d C = 3 and d (q) P = 1 for q = 1, 2. We sample each component of shared component c i ∼ Laplace(0.0, 6.5) i = 1, 2, 3, p (1) ∈ R 1 ∼ Uniform[-10, 10] and p (2) ∼ Gamma(0.5, 3.0). Although c satisfies component-wise independence assumption, it does not satisfy the condition that c i(d) ̸ = kc j , ∀i ̸ = j because c i (d) === c j , ∀i, j ∈ [3]. Therefore, Theorem 1 does not cover this case. Nonetheless, this case falls under the jurisdiction of Theorem 3.Result: Fig. 5 corroborates with our Theorem 3. That is, one needs at least |L| ≥ d C = 3 pairs of "anchors" (i.e., aligned cross domain pairs) to ensure identifiability of c (q) = Θc for q = 1, 2. x-50 0 50EQUATION-100 -50 0 50 100 c Hyperparameter Settings: The domain adaptation task follows the hyperparameter settings described in Table . 7. Baselines and Training Setup: The baselines are representative DA methods, namely, DANN [25] , MDD [60] , MCC [61] , SDAT [62] , and ELS [63] . We use the implementations of DANN, MDD, and MCC from the https://github.com/thuml/Transfer-Learning-Library, while SDAT and ELS are taken from https://github.com/yfzhang114/Environment-Label-Smoothing. In all the baselines, the classifier is jointly optimized with the feature extractor Q which arguably regularizes towards more classification-friendly geometry of the shared features; see [72, 73] . Following their training strategy, we also append a cross-entropy (CE) based classifier training module to our loss in (7) (which learns our feature extractor Q). The CE part uses Qx (1) and the labels of the sources as inputs to learn the classifier, i.e.,EQUATIONEQUATIONwhere r θ (•) : R dC → R K is the classifier that aims to map the learned feature vector Qx Additional domain adaptation experiment using CLIP features: In this experiment, we use CLIP as an image encoder as it learns informative and transferable features from very large datasets [35] . Table 8 and Table 9 show the results on Office-31 and Office-Home datasets, respectively, using CLIP embeddings. Compared to the results on ResNet50 embeddings in Table 1 and Table 2 , one can observe that all the methods, including proposed method, gains an advantage. This is likely because CLIP was trained on a large and diverse dataset [35] , which may have include similar content to the Office-31 and Office-Home datasets.The results show that, as a foundation model, CLIP can already unify the embeddings of the source and target domains to a reasonable extent. In addition, our model and algorithm when combined with regularization techniques like MCC, can still further enhance performance, even with simple post-processing of CLIP embeddings. Hyperparameter Settings: The hyperparameter settings for single-cell sequence analysis is presented in Table . 10.Baseline: For more details on baseline refer to the implementation in https://github.com/ uhlerlab/cross-modal-autoencoders.Hyperparameter Settings: The hyperparameter settings for multi-lingual information retrieval is described in the Table. 11. The memory complexity involves storing the network parameters and the aforementioned gradients. Hence, only the size of Q (q) , F 1 , and c (q) changes with the problem dimension. The size of Q (q) , F 1 , and c (q) are d C d (q) , O(d C ) and d C , respectively. Therefore, the space complexity is O(Bd C (d (1) + d (2) )).In summary, both the memory and computational complexities of the proposed method scales linearly with d C .Theorems 1-3 are concerned with learning the shared component c. The goal, there, was to ensure that QC x (q) Θc, ∀q. In some cases, the private components p (q) is also of interest [6, 31, 74] . To learn p (q) , we propose to solve the following learning criterion:find Q (q) C ∈ R dC×d (q), QP ∈ R d (q) P ×d (q) q = 1, 2, (33a)EQUATIONEQUATIONEQUATIONEQUATIONwhere u ⊥ ⊥ v means that the random vectors u and v are independent with each other.For implementation we use following criterion,EQUATIONwhere, first two term are adversarial loss for distribution matching. The constraint on (33d) and (33e) are enforced as R(Q C (q) ) and R(Q C (q) ) respectively, where R(Q (q) ) = ∥Q (q) E[x (q) (x (q) ) ⊤ ](Q (q) ) ⊤ -I∥ 2 F . The constraint on (33c) is realized with Hilbert-Schmidt Independence Criterion (HSIC) [75] . HSIC measures the independence between two distribution. So, we minimize HSIC between estimated shared component and estimated private component to promote independence between shared and private components.We show that under some reasonable conditions the block p (q) can also be learned up to a matrix multiplication: Theorem 6. Assume that the blocks c, p (1) and p (2) are statistically independent, i.e., p(c, p (1) , p (2) ) = p(c)p(p (1) )p(p (2) ). Then, if one of the following holds:(i) Assumption 1 and assumptions in Theorem 1 are satisfied, and (33) is solved yielding solutions Q (q) C and Q (q) P (ii) Assumption 2 is satisfied and has same mixing matrix A (q) = A and (33) withQ (q) P = Q P and Q (q) C = Q C is solved yielding Q (q)C and Q (q) P as the solutions.(iii) Assumption 1 is satisfied and d C paired samples (xEQUATIONℓ ) are available (weak supervision), and denote Q (q) C and Q (q) P as the solutions after solving (33) .Then, we have Q (q) C x (q) = Θc and Q (q) P x (q) = Ξ (q) p (q) , for some invertible Ξ (q) for all q = 1, 2.Proof. For each case in Theorem. 6 (i) -(iii), we can proveEQUATIONusing Theorems 1-3. The proofs are referred to Appendix B-D.Let us denoteEQUATIONwhereEQUATIONP ) . Note that the constraint (33c) implies that the mutual information between p (q) and c (q) is zero, i.e., I( p (q) ; c (q) ) = 0.Note that p (q) → c (q) → Θ -1 c (q) = c is a Markov chain. This is because when conditioned on c (q) , Θ -1 c (q) becomes constant, making it independent of p (q) . This allows us to use the data processing inequality [76, Theorem 2.8.1], which results in the following: I( p (q) ; c (q) ) ≥ I( p (q) ; Θ -1 c (q) ) = I( p (q) ; c)).Since mutual information is always non-negative, the above implies that I( p (q) ; c) = 0. This implies that p (q) = H (q) c p (q) is independent of c. Hence, H (q) [1 :EQUATIONP ]p (q) = Ξ (q) p (q) , ∀q, whereΞ (q) = H (q) [d C + 1 : d C +d (q) P ].Note that H (q) is full row-rank because of constraint (33e). This implies that Ξ (q) , q = 1, 2 are invertible matrices. This concludes the proof. Hyperparameter Setting The hyperparameter setting is the same as mentioned in Appendix. F.We solve (34) to obtain Q (q) C and Q (q) P to recover the shared and private components, respectively. For learning Q (q) P , we use Adam optimizer and set initial learning rates to be 0.001. Also we set the regularization parameter ω = 10.0 and ρ = 50.0. Data Generation: We set d C = 2 and d (q) P = 1 as in the previous synthetic experiments. We sample c ∼ VonMises(2.5, 2.0) The private components are sampled from p (1) ∼ Beta(1.0, 3.0) and p (2) ∼ Gamma(0.5, 3.0) distributions. Each element of mixing matrices are sampled from A (q) ij ∼ N (0, 1), q = 1, 2.Result: Fig. 7 shows the result for proposed method for private component identification. The first column shows the data domain, the second column shows the true and extracted shared component, and the third and fourth columns shows the true and extracted private components. Especially, the last row of the third and fourth columns shows the plot of ground truth p (q) on x-axis and p (q) on the y-axis. The plot is approximately a straight line which indicates that the estimated private components p (q) are scaled version (i.e., invertible linear transformations) of ground truth private components. This verifies our Theorem 6. Justification: Yes the code is provided in the supplemental material. Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e. • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 


Decision: The submission proposes a new approach for unaligned shared component analysis, where the multiple data modalities are drawn from a common Bayesian linear model with some latent shared components but are not aligned (or registered) to coincide sample by sample.

The contribution consists both in a novel algorithm minimizing a min-max distribution matching criterion somewhat inspired by GAN ideas and in sets of conditions under which such unaligned factor models can still be identifiable (up to fundamental permutation and scale indeterminacy). The significance of the problem that is addressed as well as the value and originality of both of these contributions have been recognized by the reviewers. The reviewers however made many relevant comments about weaknesses of the original submission that need to be taken into account by the authors. This is all the more important that several reviewers have raised their score following the discussions, taking into account the responses of the authors, including some additional experimental results.

Given the target selectivity set for the conference, the submission is close to the decision threshold but in light of priority given to the novelty of the content and of the clear path for improving the submission based on the exchanges with the reviewers, I am leaning toward acceptance at the conference.

Reviews: review: paper_summary: paper_summary: This work considers the identifiability of linear latent representations that are shared (i.e., identical) across data modalities, in the special case that they are unaligned/unpaired.
The approach leverages GAN-style training to achieve divergence minimization between the latent distribution of each modality.
The approach appears to be restricted to two modalities.
Under the assumption of shared latents, sufficient (not necessary) conditions for identifiability are presented, which are milder and, thus, more general than existing studies.
Further, structural constraints based on side information are introduced to further relax identifiability conditions.
Several experiments on simulation and real-world data are provided.
strength_weakness: strength_weakness: Quality :
 - The code is not too friendly to readers, lacking higher 1-to-1 correspondence with the notation in the paper. Suggest improving documentation and tidying up the codebase for readability.
 - Figure 5 does not seem to replicate well in `synthetic_train.ipynb` --> Clarify
 - The numerical validations (simulation) are limited to 100,000 samples. Could you illustrate performance at 10,000, at 1,000, and at 100 samples? Many multimodal applications are limited to sample-poor regimes (N < 100), where classical CCA is one of the few performant methods, so it would be useful to assess the performance of unaligned SCA at varying sample sizes, and perhaps include comparable CCA results. Is there a summary measure (like Amari distance) that could be reported alongside figures?
Clarity :
 - The paper contains some typos that limit the clarity. Besides fixing these typos, consider improving the readability of the proofs by being a bit more explicit with "obvious" steps that may be currently omitted.
 - I think the set L of paired samples was not defined?
 - Line 59: "transformations identifies" --> "transformations identify"
 - Line 65: "samples available" --> "samples are available"
 - Lines 95-96: "the cross-modality samples share the same c are aligned" --> unclear meaning... maybe drop "share the same c"?
 - Line 117: "to met" --> "to be met"
 - Line 183: Sentence ends abruptly at: "where \Theta^(q)."
questions: questions: 1. What do you mean by "lift" the constraints in line 131?
2. Although theorem 1(a) does not "require" independence between c and p, isn't that implied/necessary? Otherwise, could you show that Dependence between p and c still yields identifiability of c? Specifically, say, if c1 and c2 are conditionally independent p(c1,c2,p) = p(c1|p)p(c2|p)p(p).
	2.a. Does this have to do with Line 149: Q^(q)A^(q) = [Θ^(q), 0] ?
3. Are there obvious limitations wrt differences in the sample size for x^(1) and x^(2)? Are there any provable biases if the data is highly unbalanced? How does unbalanced data affect the identifiability?
4. Is the methodology and identifiability theory limited to the two-dimensional case?
limitations: limitations: - A discussion of the asymptotic computational performance is amiss? Specifically, with respect to d_C and the total sample size.
- The paper does not appear to discuss the applicability of the theorems to the case of q > 2 (i.e., more than 2 modalities).
- Lacks a discussion of the stability of GAN training, especially with several additional loss augmentations.
paper_summary: paper_summary: This work considers a problem similar to classical Canonical Correlation Analysis (CCS), which assumes a linear generative model for data $(x_1, x_2)$: $x_1=W_1z$, $x_2=W_2z$ and aims to identify the underlying components. 

This problem has been extended previously to include "private information": $x_1=W_1z_1, z_1=[c,p_1]$, $x_2=W_2z_2, z_2 = [c,p_2]$ for common $c$ and independent $p_j$. 

The current work further assumes that data is *unpaired* and rather than mapping each pair ($x_1$, $x_2$) such that $z_1$, $z_2$ are close together (in some metric), it is proposed that all $x_1$ are mapped to be similar *in distribution* to the mapped $x_2$s.
strength_weakness: strength_weakness: High level:
* While I understand the basics I am not an expert in the area of CCA, but I find the paper fairly difficult to follow. More explanation would be helpful, e.g. 
    - [28] why is any linear mixture model ill-posed (is that strictly true in *every* linear mixture case?)
    - [47] what is meant by "facilitating one-to-many translations", the context/meaning is unclear.
* The theoretical part of the paper relates to a simple linear model, but none of the experiments follow this model
    - e.g. the algorithm is applied to CLIP embeddings, which are not "the data", so to make claims about a simple linear model z=Ax and then apply it to CLIP seems incongruous. This experiments seem to relate more to a CCA-based "loss function" that takes representations and looks to align them/encourage independent factors etc.
    - other experiments appear to be on discrete data, which the methods doesn't apply to, presumably these are also represented as some intermediate step?
    - it seems strange to propose a simple linear model, present theoretical results about identifiability that rely on that simplicity but make a dramatic departure in the experiments where the assumptions clearly do not hold and the notion of identifiability is unclear.
*  if the work does achieve an improvement in a CCA type setting, it seems appropriate to compare with other CCA methods on suitable data. Adding results on more complex data/representations may be of additional interest.

Assumption 1 
 - hard to parse and could be made more clear. 
 - unclear if correctly defined, don't vectors y need to be orthogonal to subspace P? It seems extremely loose to the point of simply saying $P_{c,p_1} \ne P_{c,p_2}$ (specifying where any difference lies to this might add clarity).

Theorem 1 
 - is this saying that if all dims of c are distributed differently, p(z)'s can only match (e.g. under GAN loss) by correctly aligning each dim? If so, that is pretty intuitive and it could be made clearer that you are putting that mathematically and proving it for the sake of rigour.
 - I have not been through the proof, 5+ pages of proof without a sketch in the paper might be more suitable for a journal as appendices are not typically expected to be reviewed in detail.

Overall, there may be useful results in the paper, but in my view it should be re-written to make more clear what it is doing. It seems a confusing mix of simple linear generative model and related CCA methodology mixed with much more complex representations (e.g. CLIP) passed through a GAN + linear layer.
questions: questions: see weaknesses
limitations: limitations: see weaknesses
paper_summary: paper_summary: The paper considers the identifiability of shared components from a linear mixture. The theory requires multiple domains. However, compared to previous works, the required domains do not need to be aligned in this work. A practical estimation model has been proposed according to the theory.
strength_weakness: strength_weakness: 1. Since there are already many works in learning shared components in the nonlinear setting, and some of them can even handle unpaired mixtures, the linear setting appears less appealing in comparison.

2. Assumption 1 is similar to the one used in the previous works, which should be highlighted earlier in the paper.

3. The discussion of the assumption of hyper-rectangle support is missing. Is it restrictive? Maybe some real-world examples could be helpful.
questions: questions: 1. I didn't fully understand the proof in Line 780--does the usage of data processing inequality require a Markov chain? If so, has it been shown in the proof?

2. Could you please elaborate more on the connection between the proposed theory and previous work focusing on identifying content and style variables? It seems like they share a similar goal.
limitations: limitations: The authors have discussed the limitations.
paper_summary: paper_summary: This work presents a method for performing a shared component analysis (SCA) in the case of multi-modal unpaired data drawn from a linear mixture. This problem (and method) will be referred to as Unaligned SCA. Unaligned SCA is tackled by matching the probability distributions of the embedded (features) multi-modal data. More specifically, the authors draw inspiration from the traditional adversarial loss used in GANs, and formulate the component analysis algorithm as a min-max optimization problem where a discriminator (a neural network) is trained to maximize the confusion between the alignment of two representations (typically from two different modalities), and then the best alignment is sought in order to “fool” the current discriminator. The alignment matrices are structured so that the algorithm can distinguish between components shared across modalities and private components specific to each modality. The authors claim that the shared components can be identified up to the same ambiguities as those identifiable in the aligned case. Furthermore, the authors explain that while there are other methods that attempt to solve the Unaligned SCA, the conditions of the proposed algorithm are considerably milder.

The algorithm is then extended to cases where additional knowledge is present. First the algorithm is modified to accommodate the scenario where the data is generated by a single modality (uniform modality). Then, the algorithm is modified to accommodate the case where some data pairing is available (similar to a weakly supervised case). The authors show that by adding appropriate constraints  the shared component can be identified under milder conditions.

The author provide first a theoretical analysis with numerical simulations, and then some concrete applications of Unaligned SCA for the problem of domain adaptation (same modality), Multi-lingual Information Retrieval (only in the appendix, same modality) and Single Cell Sequence Analysis (multi modality with and without pairing).
strength_weakness: strength_weakness: I would divide the weaknesses into two groups: the empirical evaluation, of which I am fairly confident about, and the theoretical analysis, of which due to my limited knowledge in this field I am less confident of. I will share here my concerns as solving them might also be helpful for other readers in my position

*Empirical Analysis.* 

I find the empirical analysis weak. The multi-modal and unpaired scenario results are underwhelming, while I find the domain adaptation (same modality) potentially problematic due to the use of CLIP as pre-processing step, and the lack of a strong recent baseline (less problematic than the CLIP reason).
- The only practical results where the data are multi-modal and unpaired are the ones shown in Figure 4 first blue dot where the accuracy is about 10%. Since this is the only case presented it is unclear if the method, in practice, cannot cope in this scenario or if the specific problem chosen is particularly challenging (in which case other evidence would be maybe better).
- In the domain adaptation experiments the paper reads: “The images are pre-processed by the pretrained CLIP model [34] that uses ViT-L/14 transformer architecture.” It is not clear if ALL baselines used the CLIP embeddings as pre-processing, or if this was only done for the proposed algorithm. For fair comparison the same pre-processing should be applied to all algorithms. 
- Additionally, CLIP is known to have been pre-trained on a large and diverse dataset and there is a good chance it has been trained on Home-Office and Office-31 too, so it is difficult to appreciate the ability of the proposed method when using such a powerful pre-processing step (which as I mentioned might have been trained on theses datasets, including the their test sets). So making sure the pre-processing is equal is necessary, I would also encourage to present results with a less powerful pre-processing step (e.g., something pre-trained on ImageNet either supervised or SSL style like SimCLR) in order to better distinguish the contribution of CLIP vs every baseline and the proposed model.
- The authors use a lot of baselines as comparison however all these baselines seem to be fairly old (all before 2020?). I would suggest comparing with a stronger baseline (either check the leaderboard or here are some suggestions [1-4], note that not all might be immediately applicable). 

[1] D3GM (https://arxiv.org/pdf/2401.05465)

[2] CLUE (https://arxiv.org/abs/2010.08666)

[3] LAMDA (https://arxiv.org/abs/2208.06604)

[4] SDM (https://arxiv.org/abs/2203.05738)


*Theoretical analysis.*
I have struggled to follow the theoretical explanation. Specifically, I understand the rationale behind formulation in eq (7) but I would fully agree with it if the samples were paired. I do not understand how 6(b) holds for unpaired samples. I suppose this is the explanation currently presented before assumption 1 but even after reading it I was left with the same question.
questions: questions: A satisfactory answers to these points could improve the "contribution" (and partially the "soundness") of the work.
- Was CLIP used as a pre-process step for all the baselines in the Domain Adaptation? If not, could the authors present those results by using the same pre-processing step? As I mentioned above ideally without using CLIP due to the potential contamination of the test sets.
- Could the author explain why 6(b) holds for unpaired samples?

Further addressing these less critical aspects could improve the "Presentation" score (and partially the contribution see first point)
- I believe the paper would be stronger with more recent baselines as suggested above.
- I find unclear how Fig.1 was created. More explanation would help the understanding. 
- I find Fig 2 not clear: why are there c1 and c2 and only p1, whereas I was expecting p1 and p2 and a common (shared) c? Reading if further it might be that c1 and c2 are the two axes of the common space $c$. If this is the case I’d make sure to clarify it.
- Assumption 1 is not clear to me. Is this saying that both points y1 and y2 leave in the same sub-space within the span of $(c, p^{(1)}, p^{(2)})$?  If so in which way is this useful?
- The authors say the results are an average of 5 runs, which is great, but  the standard deviation should also be reported.
- I find this sentence confusing: "First, it is unclear if (6b) could disentangle c from p(q). In general, Q(q)x(q)
could still be a mixture of c and p(q) yet (6b) still holds (e.g., when both c and p(q) are Gaussian.)" first it says it is unclear if they can be disentangled, but the whole identifiably relies on the ability to disentangle them no?
- In a couple of places in the manuscript the authors refer to an experiments where all the results are in the appendix. I would add a brief summary of the results so the main paper is self-standing (this happens in More Synthetic-Data Validation and Application (iii)).
- In a few places $p^{(2)}$ appears without a closed bracket, i.e., $p^{(2}$.
- Sometimes the authors used the comma (,) to separate thousands but other time they didn’t. I would use a uniform notation.
limitations: limitations: Yes, the authors have identified and clearly stated three main limitations:
- The fact that the conditions presented are sufficient while the necessary are not yet known.
- The fact that the method works only for linear mixtures (which limits the expressivity).
- Lastly, the fact that the theoretical derivation assume infinite data.

Comments: title: thanks for the response
Reviewer response
comment: We thank the reviewer for the constructive comments and discussion. We will make the promised changes as discussed.
Thank you for your responses.
Conditioned on the inclusion of all adjustments, edits, results, and discussions the authors have committed to, I've updated my rating on Soundness from Good --> Excellent, largely due to the additional experiments and evidence provided.
Keeping my current score rating of 7 (Accept) as my overall impression of the work remains unchanged.
Thank you for your response. I would like to maintain my positive score.
We would like to thank the reviewer for helping us improve the clarity and for the constructive communication.
Thanks, if the proposed changes are made I think the readability and understanding of the paper should improve. I have not been through the proof in detail, but the approach makes sense and the empirical results suggest the method works. Assuming the proposed changes are made, which are not substantive, I recommend the work being published.

Score 4 --> 6
**[Amari Distance Computation]**

In our evaluation, we used $\bf{\hat{\Theta}^{(q)}} = \bf{Q}^{(q)} \bf{A}^{(q)}$$(1:d\_C)$, where $\bf{A}^{(q)}$$(1:d\_C)$ represents the first $d\_C$ columns of $\bf{A}^{(q)}$. Note that $\bf{Q}^{(q)}$ is our estimated linear operator, and  $\bf{A}^{(q)}$ is the ground-truth mixing system that is available for synthetic data experiments (we only evaluated Amari distance for the synthetic data on our previous reply). 

Another note is that (thanks to the above discussion) we realized that general matrix distances (such as Euclidean distance) could be a better fit for our case than the Amari distance. To see, recall that we have content identifiability if and only if 
$  \bf{Q}^{(q)} \bf{A}^{(q)}  = [\bf{\Theta}, \bf{0}]$. Hence, we need $\hat{\bf{\Theta}}^{(1)} = \hat{\bf{\Theta}}^{(2)} = \bf{\Theta}$. However, Amari distance is insensitive (invariant) to permutation and scaling, i.e., $\hat{\bf{\Theta} }^{(1)} = \bf{P} \Lambda \hat{\Theta}^{(2)}$ incurs zero Amari distance, where $\bf{P}$ and $\Lambda$ are any permutation and scaling matrices respectively. Hence, we present the Euclidean distance instead of the Amari distance. Additionally, we also report the $\\| \bf{Q}^{(q)} \bf{A}^{(q)}$$(d\_C+1 : d\_C + d^{(q)}\_P ) \\|_F$ which needs to be close to $\bf{0}$ for identifiability. 

Table 1: Numerical evaluation of identifiability $\\| \widehat{\bf{\Theta} }^{(1)} (1: d\_C) - \widehat{ \bf{\Theta} }^{(2)}(1:d\_C) \\|_{F} .$

|N            | SCA   | CCA |
| :- | :- | :- |
|100,000 | 0.009  | 1.368|
|10,000   | 0.007  | 1.544|
|1,000     | 0.003  | 2.206|
|100        | 0.032  | 1.755|
|50          | 0.133  | 1.667|
|20          | 1.462  | 1.522|

&nbsp;

Table 2: &nbsp; $1/2  \sum\_{q=1}^2 \\| \widehat{\bf{\Theta}}^{(q)} ( d\_C+1 : d\_C+d^{(q)}\_P) \\|\_{F}$.

|N           | SCA    | CCA |
| :- | :- | :- |
|100,000 | 0.021  | 0.284|
|10,000   | 0.034  | 0.279|
|1,000     | 0.002  | 0.329|
|100        | 0.043  | 0.368|
|50          | 0.131  | 4.092|
|20          | 0.747  | 0.755|


We will follow the reviewer’s suggestion and add the new experiment (with mean and standard deviation) for the synthetic data experiments in the revised version.

**Q2.** Yes. We will add a footnote about conditional independence in the main paper.

**Q3.** Thanks for the suggestion. We have run the following experiment with unbalanced data. 
For the following experiment, the data for two modalities were generated, by sampling the shared component of dimension($D=2$) from VonMises distribution and private components from Gamma and Laplace distributions.  The number of samples in the first modality is fixed to 100,000 and the second view ranges from 10,000 to 10 samples. 

Table 3:  Performance of SCA on imbalance data based on following two metrics,

**metric1** = &nbsp; $\\| \widehat{\bf{\Theta}}^{(1)} (1: d\_C) - \widehat{ \bf{\Theta}}^{(2)}(1: d\_C) \\|\_{F} $,

 **metric2** = &nbsp; $1/2  \sum\_{q=1}^2 \\| \widehat{\bf{\Theta}}^{(q)} ( d\_C+1: d\_C+d^{(q)}\_P) \\|\_{F}$.

| \# samples in modality 2 | metric1 | metric2 |
| :- | :- | :- |
|10,000         | 0.008  | 0.025 |
|1,000           | 0.025  | 0.015 |
|100              | 0.091  | 0.087 |
|10                | 1.375  | 0.209 |

We will include the above result (with mean and standard deviation) in the revised version.

**Q4. [Possible Benefit of $Q \geq 2$ domains]**

One foreseeable benefit of having more than two domains is that Assumption 1, when modified for $Q \geq 2$ domains, could be more relaxed. This is because modality variability in general is satisfied if at least two of the total number of domains satisfy the current Assumption 1. Having more modalities can make the chance of Assumption 1 increased. On the other hand, enforcing the distribution matching constraint Eq. (6b) could be more challenging for more than two domains. We will add this discussion in the revised version.

**[Complexity Analysis]**

The short answer is that **both the memory and computational complexities of the proposed method scales linearly with** $d_C$. The per-iteration computational complexity is $O(B d\_C (d^{(1)} + d^{(2)}) )$, where $B$ is the mini-batch size. The per-iteration memory complexity is $O(B d\_C (d^{(1)} + d^{(2)}) )$ as well. The complexities are based on the fact that we use mini-batch based stochastic gradient-type optimizer. We will add a section in the appendix to detail the complexity calculation.
We would like to stress that we absolutely found the reviewer’s comments valuable for improving the paper’s clarity. It was our fault that we missed adding sentences that commit changes (while concentrating too much on the 6000 character limitation), which was not our intention. We do agree with the reviewer: any suggestion that may help the general readers to better understand the paper is appreciated. We thank the reviewer for the help on clarity and will definitely make revisions accordingly.

**[LMM and 1-many translations]**

We agree with the reviewer that explanations to these points could make our paper more self-contained. Hence, we will add the explanations (provided in the rebuttal) in a separate "Preliminaries" section in the Appendix with clear pointers in the main paper.

**[Preprocessing]**

By "linear subspaces", we mean representation spaces where the representations are likely to be linear mixtures of semantic information. As mentioned in our original rebuttal (**[Using CLIP as Pre-processing]**), this has been observed to be the case for embedding spaces of neural networks such as CLIP [R7], word embeddings [R30] etc.

We would like to clarify that our method is always applicable wherever CCA is applicable, since CCA shares the same generative model [R18, R29] (also see section 2 Aligned SCA in the manuscript) as ours (the only difference is that CCA further requires the cross-modality samples to be aligned according to their content). Note that CCA also uses pre-processed features representations for complex real-world data (e.g., image, text) [R28, R18]. This is because these complex real-world data might not follow the linear mixture model in Eq. (1) in the manuscript, however the pre-processed representations might. Note that it is common for identifiability works to use pre-processed features for real-data validation of their Theorems [R18]. However, we understand the reviewer’s comment on the applicability to complex data directly. We will explain in more detail in the beginning of the experiment section regarding why preprocessing is involved.

We also hope to remark that the sentence in our abstract "*This work takes a step further, investigating shared component identifiability from multi-modal linear mixtures where cross-modality samples are unaligned*" is an accurate claim. Note that our claim is for multi-modal **linear** mixtures. Therefore, for complex datasets, it is necessary to find appropriate linear representation spaces. We will add more clarifications/reminders when it comes to the experiment section.

**[Data]**

We agree with the suggestion of first using simpler raw data and then representations of more complex data to run experiments. The presented experiments in fact may have implicitly reflected this comment.

To explain, note that the single-cell data is not pre-processed using any encoder but a normalized (zero mean and unit std) version of the raw-data, which is a simpler dataset as the reviewer mentioned.  The more complex image and language data were preprocessed by existing encoders.

Following the reviewer’s suggestion on “simpler datasets ---> harder datasets” comment, we will change the order of presenting the single-cell experiment and the other experiments, to make this more explicit.

**[Thm 1]**

Thank you for your suggestion. We will include a simpler, intuitive explanation of Theorem 1 in the revised version.

**References**

[R28] Shi et al., 2019. Image Retrieval via Canonical Correlation Analysis.

[R29] Ibrahim et al., 2020. Reliable Detection of Unknown Cell-Edge Users via Canonical Correlation Analysis.

[R30] Mikolov et al., 2013. Efficient Estimation of Word Representations in Vector Space.
Thank you for your clarifications. I have some follow up questions below:

Sample size assessment: The Amari distances are quite remarkable (very low) considering the sample sizes. Could you clarify how you estimate \Theta? I recommend reporting this result (e.g., median +/- std Amari values) for every experiment (assuming you can estimate \Theta). CCA result does not need to be reported.

Q2: Can you add the note about conditional independence in a footnote?

Q3: Can you provide any empirical evidence about how unbalanced data affects identifiability in the different scenarios investigated here?

Q4: So the current theory is limited to Q = 2 domains, but can be extended. Could you elaborate further on the point about "benefits" from extra domains? Do you anticipate diminished returns from including extra domains?

Complexity analysis: Could you add a discussion about the computational complexity of the proposed model? Do the memory/computation requirements grow linearly/quadratically/other w.r.t. d_C?
I realise (as an author) that reviews can sounds attacking. I would like to stress, since your response doesn't seem to acknowledge any change, that I appreciate this line of work and my comments are to improve the paper if possible.

* **LMMs & 1-many translations**: these are points of clarity to "the general reader", not just me, I think the paper could be more clear and standalone than it currently is
* **Preprocessing**: you mention "linear subspaces", I'm not sure I follow, for sure various representation models give representations that already untangle much complexity in the data (e.g. so that semantically similar items are clustered). CCA acts on the raw data, you are acting on representations. It does not make the approach invalid, but it should be more clearly stated that is what you are doing. You are in effect heavily relying on what other models achieve, which is completely arbitrary with respect to your contribution. In effect you are providing a loss function to wrap around pre-trained representations along the principles of CCA. This is not what is in the abstract for example: "This work takes a step further, investigating shared component
identifiability from multi-modal linear mixtures where cross-modality samples are unaligned". Given you don't know what the "representation model" has done, it detracts from identifiability claims, which typically refer to the data itself and should at least be caveated.
* **Data**: you do run experiments on discrete data: text is discrete. As above you rely on representations that have already done a lot of work in re-representing it. It would be better, in my view, to demonstrate that the linear CCA-type workings actually work as expected on multiple appropriate (simpler) datasets and then show that that still holds for more complex scenarios where non-linear encoders (or similar) have effectively taken the non-linearity into account. Identifiability should relate to factors that those underlying models have identified.
* **Thm 1**: I think an intuitive explanation in the paper would improve readability/understandability.
We thank the reviewer for their detailed and constructive discussion during the rebuttal.

The comment about multi-modality seems to be a terminology issue. We will follow the reviewer's suggestion and change the term to "multi-domain".

We will also add the summary of results of application (iii) in the main paper.

Nonetheless, the terminology issue doesn’t seem to affect our contributions. We believe that our contribution lies in providing rigorous understanding of the proposed unaligned multi-domain problem structure. Our synthetic and real-data experiments were designed to validate that understanding. The unaligned SCA problem is of great interest as a latent component analysis problem, like ICA, PCA and CCA. Its identifiability has been elusive and our work filled this gap.
We wonder if the reviewer could re-assess the contribution from the identifiability research viewpoint, rather than the “multi-modality” vs “mono-modality” viewpoint.

In any case, we sincerely thank the reviewer for the comments and discussion, and for pushing us to improve our experiments and presentation.
Thank you for your explanation. I now think I understand where the misunderstanding comes from: by multi-modal I was expecting two completely different data modality (e.g., image and sound). By this definition I thought the only true multi-modal setting was application (ii) with one type of data being RNA sequences and one being ATAC sequences. After a bit of further investigation I think even this setting is not quite multi-modal (as in two separate and different modality) as I believe (but I am not a biologist) both sequences are actually made by the 4 DNA bases. Similarly, the other two applications are mono-modal: application (i) is images (taken from different type of cameras) and application (iii) is text (coming from different languages).

I appreciate the in all three applications the data have different distributions, but I think it should be made clear that the focus is not multi-modal settings (as in multi-modality type of data) but rather in same modality with a distribution shift, also known as domain adaptation. I believe changing the title and the explanation to domain adaptation rather than multi-modal would help setting the right expectation for the reader. I would also encourage the authors to provide a short summary of the results of application (iii) rather than delegating everything to the appendix.

With all the additional experiments and clarifications provided during this rebuttal I believe the authors have increased the quality of their work to the following:

Soundness: 3: Fair -> Good

Presentation: 3: Fair -> Good

Contribution: 2: Fair


The reason for keeping the contribution as Fair is that:
- The method was tested only on domain adaptation experiments rather than what I thought was a multi-modality setting. 
- It was shown in the additional experiments that using a powerful feature extractor (CLIP in this case) is arguably more beneficial than any sophisticated algorithm.

I am ok to increase my overall recommendation from 4 to 5.
(1) We will follow the reviewer’s suggestion and use ImageNet features as the main result in the paper, while moving updated CLIP experiments to the Appendix.

(2) All 3 applications (i.e., (i) image domain adaptation, (ii) single-cell sequence alignment, and (iii) multilingual embedding retrieval) are multimodal and unpaired. But their detailed settings vary. We proposed our unaligned SCA approach under three different settings, depending on how much structural information can be exploited.

**[Setting 1] Multimodal and unpaired:** The setting uses $\bf{x}^{(i)} = \bf{A}^{(i)} \bf{z}^{(i)}$ where $\bf{z}^{(i)}=(\bf{c},\bf{p}^{(i)})$ for modality $i$. The different $\bf{A}^{(i)}$’s and $\bf{p}^{(i)}$’s both represent the modality discrepancies. Synthetic data was used to validate the setting. We argued the condition needed here for identifiability was too strong for many applications. This was the motivation for us to consider Settings 2-3.  (see Sec. 4 Line 203-208). 

**[Setting 2] Multimodal and unpaired; the modalities share a homogeneous feature space (sometimes called multi-domain setting):** The setting uses $\bf{x}^{(i)} = \bf{A} \bf{z}^{(i)}$ where $\bf{z}^{(i)}=(\bf{c},\bf{p}^{(i)})$ for modality $i$. The modality/domain differences are captured by $\bf{p}^{(i)}$’s. Unlike Setting 1 where $\bf{A}^{(i)}$ varies across $i$, here the mixing systems $\bf{A}^{(i)}=\bf{A}$ for all $i$. This is often called a homogeneous multi-domain setting, which is a more special case of multimodal learning. This setting makes sense when the data $\bf{x}^{(i)}$ for all domains share the same feature space, often used in applications like image-to-image domain adaptation [R17] and image-to-image style translation [R21]. **Applications (i) and (iii) were used to validate the method under this setting**. Application (i) is on domain adaptation of images. The images from different domains are unpaired. Application (iii) is Multilingual retrieval problem. The modalities correspond to unpaired words in different languages.


**[Setting 3] Multimodal and largely unpaired (with a small number of paired data):** The setting uses $\bf{x}^{(i)} = \bf{A}^{(i)} \bf{z}^{(i)}$ where $\bf{z}^{(i)}=(\bf{c},\bf{p}^{(i)})$ for modality $i$. The vast majority of data are unpaired. But there exists a small number of paired data (for example, in our experiment of Fig 4, the total number of data in each domain is 1,874. We considered cases where 0 to 256 paired data exist, i.e., 13% at its maximum). This setting is considered realistic in applications such as [R22, R26]. **Application (ii) was tested under this setting**. It corresponds to the single-cell experiment. Modalities correspond to unpaired RNA sequences and ATAC sequences.

[R26] *Wang et. al., 2020. Semi-supervised Learning for Few-shot Image-to-Image Translation.*
Thanks to these authors for their answers. I appreciate the explanations and the effort in running the domain adaptation experiments using my suggestions. I suggest using the ImageNet features as the main result in the paper and provide the CLIP ones in the Appendix (as it shows that a powerful feature extractor such as CLIP reduces most of the differences among all the algorithms).

There is still one aspect I do not fully understand. Could the authors state explicitly which of the three experiment settings (i), (ii) or (iii) shown in the paper are at the same time multi-modal and unpaired?