Propensity Score Alignment of Unpaired Multimodal Data
Authors: Johnny Xi
Jana Osea
Zuheng Xu
Jason Hartford
Venue: NeurIPS 2024 poster (2024)
Keywords: unpaired data
multimodal
causal representations
propensity score
Primary area: machine_learning_for_healthcare
TL;DR: We propose learning propensity scores as a common space to align unpaired multimodal data.

Abstract: Multimodal representation learning techniques typically require paired samples to learn shared representations, but collecting paired samples can be challenging in fields like biology, where measurement devices often destroy the samples. This paper presents an approach to address the challenge of aligning unpaired samples across disparate modalities in multimodal representation learning. We draw an analogy between potential outcomes in causal inference and potential views in multimodal observations, allowing us to leverage Rubin's framework to estimate a common space for matching samples. Our approach assumes experimentally perturbed samples by treatments, and uses this to estimate a propensity score from each modality. We show that the propensity score encapsulates all shared information between a latent state and treatment, and can be used to define a distance between samples. We experiment with two alignment techniques that leverage this distance---shared nearest neighbours (SNN) and optimal transport (OT) matching---and find that OT matching results in significant improvements over state-of-the-art alignment approaches in on synthetic multi-modal tasks, in real-world data from NeurIPS Multimodal Single-Cell Integration Challenge, and on a single cell microscopy to expression prediction task.

Propensity Score Alignment of Unpaired Multimodal Data
Johnny Xi
Jana Osea Valence Labs Montreal, Canada
Zuheng (David) Xu *
Department of Statistics University of British Columbia Vancouver, Canada johnny. xi@stat......... ca
Department of Statistics University of British Columbia
jana@valencellass.... com
Vancouver, Canada zuheng xuêstat, ubc. ca
Jason Hartford
Valence Labs London, UK jason@valeencelabs.com
Abstract
Multimodal representation learning techniques typically require paired samples learn shared representations, but collecting paired samples can be challenging fields like biology, where measurement devices often destroy the samples. This paper presents an approach to address the challenge of aligning unpaired samples across disparate modalities in multimodal representation learning. We draw analogy between potential outcomes in causal inference and potential views in multimodal observations, allowing to leverage Rubin's 's framework to estimate a common space for matching samples. Our approach assumes experimentally perturbed samples by treatments, and uses this estimate a propensity score from each modality. We show that the propensity score encapsulates all shared information between latent state and treatment, and can be used define a distance between samples. We experiment with two alignment techniques that leverage this distance- shared nearest neighbours (SNN) and optimal transport (OT) matching and find that OT matching results in significant improvements over state-of-the-art alignment approaches in on synthetic multi-modal tasks, real-world data from NeurlPS Multimodal Single-Cell Integration Challenge, and on a single cell microscopy to expression prediction task.
1 Introduction
Large-scale multimodal representation learning techniques such as CLIP [Radford et al. 2021 have lead remarkable improvements zero-shot classification performance and have enabled the recent success in conditional generative models. However, the effectiveness of multimodal methods hinges on the availability paired samples-stt modalities. This reliance on paired samples is most obvious in the InfoNCE loss [Gutmann and Hyvärinen] 2010, /van den Oord et al. 2018) used in CLIP [Radford et al. 2021] which explicitly leams representations to maximize the true matching between images and their captions.
as images and their associated captions-acron across data
While paired image captioning data is abundant the internet, paired multimodal data often challenging to collect in scientific experiments. For instance, unpaired data are the norm in biology for technical reasons: RNA sequencing, protein expression assays, and the collection of microscopy
Work done during intersship at Valence Labs
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Progensity modalitt
Propensity moddalit 2
Mirroscopy imagee
P
Matching P
Expression
Figure 1: Visualization of propensity score matching for two modalities (e.g., Microscopy images and RNA expression data). We first train classifiers to estimate the propensity score for samples from each modalities; the propensity score reveals the shared information p(t[z;), which allows to re-pair the observed disconnected modalities. The matching procedure then performed within each perturbation class based on the similarity bewteen the propensity scores.
images for cell painting assays are all destructive processes. As such, we cannot collect multiple different measurements from the same cell, and can only explicitly group cells by their experimental condition. If we could accurately match unpaired samples across modalities, we could use the aligned samples as proxies for paired samples and apply existing multimodal learning techniques.
In this paper, we formalize this setting by viewing each modality as a potential measurement, (Z) E x2)), (Z) E x2(2,, of the same underlying latent state E 2, where we are only able to make single measurement for each sample unit (e.g. an individual cell). The task is to reconcile (match) unpaired observations (1))) and (2)) with the same (or maximally similar) z. Estimating the latent, Z. is hopelessly underspecified without making unverifiable assumptions on the system, and furthermore, Z may still be sparse and high- -igh-iimensional, leading to inefficient matching. This motivates the need for approaches that use only the observable data.
We identify two major challenges for this problem. First, measurements are often made in very different spaces (1) and x22) (e.g., pixel space and gene expression counts), which make defining notion of similarity across modalities challenging. Second, the measurement process inevitably introduces modality-specifit variation that can be impossible to disentangle from the relevant infor- mation (Z). For example in cell imaging, we would not want the matching depend irrelevant appearance features such as the orientation of the cell or the lighting of the plate.
In this paper, we address these challenges by appealing to classical ideas from causal inference [Rubin] 1974). in the case where we additionally observe some label for each unit, e.g., indexing an experiment. By making the assumption that perturbs the observations via their shared latent state, we identify an observable link between modalities with the same underlying z. Under conditions which we discuss in Section 2 the propensity score, defined ptt|Z), is transformation of the latent that satisfies three remarkable properties (Proposition 3. (1) provides a common space for matching, (2) is fully identifiable via classification individual modalities, and (3) it maximally reduces the dimension Z, retaining only the information revealed by the perturbations.
The practical implementation of the methodology (as illustrated in Fig. ① is then straightforward: we train two separate classifiers, one for each modality, to predict the treatment t applied to (9) We then match across modalities based on the similarity between the predicted probabilities (the propensity score) within each treatment group. This matching procedure is highly versatile and can be applied to match labeled observations between any modalities for which classifier can efficiently trained. However, since the same sample unit does not appear in both modalities, we cannot use naive bipartite matching. To address this, we use soft matching techniques to estimate the missing modality for each sample unit by allowing matching to multiple observations. We experiment with two recent
matching approaches: shared nearest neighbours (SNN) matching [Lance al. 2022 2022) and optimal transport (OT) matching Villani 2009
Cao and Gao

---

In our experiments, we find that OT matching with distances defined on the proposenity score leads to significant improvement on matching and a downstream cross-modality prediction task on both synthetic and real-world biological data. Notably, our prediction method, which leverages the soft matching to optimize an OT projected loss, outperforms supervised learning on the true pairs on CITE-seq data from the NeurlPS Multimodal Single-Cell Integration Challenge [[ance ct al. 2022]. Finally, we applied our method to match single-cell expression data (from a PeturbSeq assay [ixxi et al. 2016)) with single cell crops of image data [Fay et al. 2023). We find improved generalization in predicting the distribution of gene expression from the cell imaging data with unseen perturbations.
1.1 Related Work
Unpaired and Multimodal Data
Learning from unpaired data has long been considered for image
Allahairi et al., 20018) and more recently for biological translation (Liu et al. 2017 Zhu et al. 201////aalmaai
modality translation Amoodi and Krishnaswamyl [2018] Yang et al. (02T]. particular, Yang et al.)
[2021] also takes the perspective of shared latent variable for biological modalities. This setting
has been studied more generally for multi-view representation learning [Gresele et al.
2020 Sturma
et al. 2023 for its identifiability benefits.
Perturbations and Heterogeneity Many methods in biology treat observation-level heterogeneity as a nuisance dimension to globally integrate, even when cluster labels are observed [Butler et al.)
2018 Korsunsky et al. 2019 Foster al., 2022]. This is sensible when clusters correspond to noise rather than the signal of interest. However, it is well known in causal representation learning that heterogeneity- -aareenntttt-rttuullrryyyy
particularly heterogeneity arising from perturbations has theoretical benefits
in constraining the solution set [Khemakhem et al.) 2020 [Squires et al.) [0233 Ahuja et al., 2023
Buchholz et al. [0233] von Kügelgen et al., 2023]. There, the benefits (weakly) increase with the number of perturbations, which is also true of our setting (Proposition 3.2). In the context of unpaired data, only Yang et al. [202T] explicitly leverage this heterogeneity in their method, while Ryu et al. [2024] treat it a constraint in solving OT. Specifically, Yang et al. [202]] require their VAE representations to classify experimental labels in addition to reconstructing modalities, while our method is simpler, only requiring the classification objective. Notably, Yang et al. [2021] treat our objective as a regularizer, but our theory suggests that it is actually primarily responsible for the matching performance. Our experiment results coincide with the theoretical insights; requiring reconstruction, as in a VAE, led to worse matching performance with identical model architectures.
Optimal Transport Matching OT is common tool in single-cell biology. In cell trajectory inference, the unpaired samples are gene expression values measured at different time points in a shared (metric) space. OT matching minimizes this shared metric between time points [Schiebinger] ct al. 2019 Tong et al., 20220. Recent work [Demetci et al. 2022] extends this to our setting where each modality is observed in separate metric spaces by using the Gromov- Wasserstenn distance, which computes the difference between the metric evaluated within pairs of points from each modality (Demetci et al. 2022]. In concurrent work, this approach was recently extended to ensure matching within experimental labels [Ryu et al.] 2024]. In addition to these "pure" OT approaches, Gossi et al. (0233 use OT contrastive learning representations, though this approach requires matched pairs for training, while Who et al. (022) use OT in the latent space of a multi-mode VAE.
2 Setting
We consider the setting where there exist two potential views, ( E xe) from two different modalities indexed by e E (1,2), and experiment / that perturbs shared latent state of these observations. This defines jointly distributed random variable (X(1), ( e, f), from which
observe only single modality, its index, and label, (x((4) es. 4 2 We aim to match or estimate the samples from the missing modality, which corresponds to the realization of the missing random variable. Since t is observed, in practice we match observations within the same label class t.
Formally, assume each modality is generated by common latent random variable Z as follows:
t~ Pr, 200 |t~ Pl) U(t) ~ p(c) ( LZ, ( (()())) (e) I t= ((() (())), (I)
We will denote random variables by upper-case letters, and samples by their corresponding lower-case letter.
3

---

where t indexes the experimental perturbations, and we take t = 0 to represent a base environ- ment. (((e) represents the modality-specific measurement noise that is unperturbed by t. and also independent across samples. The structural equations f are deterministic after accounting for the randomness in Z and U: it represents the measurement process that captures the latent state. For example, in a microscopy image, this would be the microscope and camera that maps cell pixels.
Comparison to Multimodal Generative Models Our setting is technically that of a multimodal generative model with latent perturbations. However, by focusing on matching rather than generation. we are able to make significantly weaker and more meaningful assumptions while still ensuring the theoretical validity of our method. Without the effects of the perturbation, our Eq. [ is essentially the same as (Yang et al. 2021 Equation I] in an abstract sense. However, in order to fit the generative model, it is required to formulate explicit models over f(e) and P&)) which requires specifying the function class (e.g., continuous) and the space of Z (e.g., IRd) as assumptions, even in universal approximation settings, In contrast, since we will not directly fit the model Eq. ① we do not make any technical assumptions about the generative model. Instead, we will make the following assumptions on the underlying data generating process itself.
Key Assumptions Our theory makes the following assumptions about the data generating process.
(A1) L J Z, and L.I. (L))) In words, / has non-trivial effect on Z, but does not affect U(t), implying that interventions target the common underlying process without affecting modality- specific properties. For example, an intervention that affects the underlying cell state, but not the measurement noise of the individual modalities.
(A2) Injectivity of f(e); (z, u) (2') (2', =
(z.u) (z', ,'). In words, each modality captures enough information to distinguish changes in the underlying state
(A1) ensures that the conditional distribution (+) is identical for = 1, (A2) then ensures that t ( E t x & t Z, which allows us to estimate the conditional distribution t with observed data alone. Though sharp assumptions are required for the theory, versions replaced with approximate distributional equalities intuitively also allow for effective matchings when combined with our soft matching procedures in practice. A particular relaxation of (A1) when combined with OT matching is described in Appendix A
3 Multimodal Propensity Scores
Under ([.. if Z were observable, an optimal matching can be constructed by simply matching the samples with the most similar 24. However, the prerequisite of inverting the model and disentangling Z arguably more difficult than the matching problem itself. In particular, Z is unidentifiable without strong assumptions on Eq. (1) [Ii and Bloem- Reddy 2023]. and even formulating the identifiability problem requires well-speeification
the model as prerequisite. We take an alternative approach that is robust to these problems, by using the perturbations ł as an observable link to reveal information about Z. Specifically, we show that the propensity score
T (2) == P(1]Z 2) E 1]
is identifiable as a proxy for the latent Z under our assumptions of the data generating process. This is a consequence of the injectivity of f(e), since it will be that x(2) = π T(X()), = 1, indicating that we can compute from either modality. Not only does the propensity score reveal shared information, classical causal inference theory [Rubin| 1974] states that it captures all information about Z that contained in f,, and does so minimally, in terms of having minimum dimension and entropy. Since i contains the only observable information that is useful for matching, the propensity score is hence an optimal compression of the observed information. We collect these observations into the following proposition.
Proposition 3.1. In the model described by Eq. T). further assume that f(e) are injective for e == 1, Then, the propensity scores in either modality is equal to the propensity score given by Z.
*Note that the injectivity is in the sense of f as a function of both u and z, which allows observations that have shared z but differ by their value in u. and the function remains injective. For example, rotated images with the exact same content can have shared but remain injective due to the rotation being captured in u.

---

i.e, T(x()) = = = π (Z) as random variables. This implies
[t, Z m(2) = [tt, Z m(X(=))) =0
(3)
for each 2, where I is the mutual information. Furthermore, any other fincction b(Z) satisfying [[t, 2 b(2) = such that π (Z) = f(b(2)).
The proof can be found in Appendix C Practically, Proposition 3.1 shows that computing the propensity score on either modality is equivalent computing on the unobserved shared latent, which means that it is identifiable, and thus estimable, from the observations alone. Furthermore, the estimation does not require modified objectives or architectures for joint multimodal processing, instead they are simple and separate classification problems for each modality. Finally, / does not affect (7(e) by assumption, and thus the propensity score, being a representation of the information in t, discards the modality-spectee information that may be counterproductive matching. Therefore. even if Z were observed, it may be sensible to match on its propensity score instead.
Number of Perturbations Note that point-wiss equality of the propensity score T(21) = π (22) does not necessarily imply equality of the latents Z2, due to potential non-injectivity of ग. For example, consider / {0, J. then <>)) is compression
z). single dimension p(1
Intuitively, collecting data from more perturbations improves the amount of information contained in the label I. If the latent space is Rd, the propensity score necessarily compresses information about 7 if the latent dimension exceeds the number of perturbations, echoing impossibility results from the causal representation learning literature [Squires et al. 2023).
Proposition 3.2. Let Z E B. Suppose that P(l) has smooth density p(z)t) for each t ...... T. Then, Tf < d, the propensity score n, restricted to its strictly positive part, is non- ton-injective.
The proof can be found in Appendix C
Note the above only states impossibility result when < More generally, can be seen from the proof of Proposition 3.2 that the injectivity of the propensity score depends the injectivity of the following expression
log(p(a(t 1)) - log(p(z|t 0))
g(2)
log(p(=|( = 7)) log(p(=|( 0))
which then depends on the latent process itself. If the above mapping is non- coj-innective, this represents a fundamental indeterminacy that cannot be resolved without making strong assumptions point- latent variable recovery. As we have already established in Proposition [3.]] the propensity score contains the maximal shared information across modalities. Nonetheless, collecting data form larger number perturbations is clearly beneficial for matching, since g in Eq. (4) is injective if any of the subset of its entries are.
4 Estimation and Matching
For the remainder of the paper, we drop the notation e and use (x,, to denote observations from modality 1, and (zy, t2) denote observations from modality 2. Given multimodal dataset with observations {2,,, t.) (==1 and ((z,,t))172 we wish to compute a matching matrix (or coupling) between the two modalities. We define a ni X 122 matching matrix M where M,J represents the always perform matching only within observations with the same value of f, so that in practice we obtain a matrix M, for each t.
likelihood of r; being matched to zy. Since t is observed,
Our method approximates the propensity scores by training separate classifiers that predicts given for each modality. We denote the estimated propensity score by n; and π. respectively, where
क; ~ क (a;) )= = P(T = f x = =.))-
(5)
This yields the transformed datasets {m, ) and {T, { where π, ny are in the T dimensional simplex. We use this correspondence to compute a cross-modality distance function:
d(z,, 23) == d' (m: n,)-
In practice, we typically compute the Fuclidean distance in T of the logit-transformed classification scores, but any metric over bijective transformation of the propensity scores also theoretically
5

---

valid. Given this distance function, we use existing matching techniques to constructing a matching matrix. In our experiments, we found that OT matching gave the best performance, but we also evaluated Shared Nearest Neighbour matching; details of the latter can be found in Appendix B
Optimal Transport Matching The propensity score distance allows easily compute cost function associated with transporting mass between modalities, c(t;, = d'(m;, Ty). Let pl, P2 denote the uniform distribution over { ) and { respectively. Discrete OT aims solve the problem of optimally redistributing mass from to P2 in terms of incurring the lowest cost. Let Cjj c( xy) denote the nı X n2 cost matrix. The Kantorovich formulation of optimal transport aims to solve the following constrained optimization problem:
n2
min Σ
CiyMy
My 2 0, =1= PL M 1=111 p2-
This is a linear program, and for mh 12, it can be shown that the optimal solution is bipartite matching between { } and { 182 We refer to this as exact OT; in practice we add an entropic regularization term, resulting in a soft matching, that ensures smoothness and uniqueness, and can be solved efficiently using Sinkhorn's 's algorithm. Entropic OT takes the following form:
min M
My (M),
My 20, 0, M1=P; = PL, M = P2.
(8)
where (M) = E., My (yg( the entropy of the joint distribution implied by M. This approach regularizes towards higher entropy solution, which has been shown to have statistical benefits [Genevay et al.] 2018), but nonetheless for small enough serves as a computationally appealing approximation to exact OT.
5 Downstream Tasks
The matching matrix M can be seen as defining an empirical joint distribution over the samples in each modality. The OT approach in particular makes this explicit. Each row is proportional to the probability that each sample from modality (1) is matched to sample in modality (2), i.e., M., P( (r,). We can thus use M to obtain pseudosamples for any learning task that uses paired samples by (Iz, 22), where ž is obtained by sampling from the conditional distribution defined by M, or by a suitable conditional expectation, e.g., the barycentric projection (conditional mean) as Ear [X; X; = zi] = E, Mi,gar.. In what follows, we describe cross-modality prediction method based on both barycentric projection and stochastic gradients according to Mi.--
Cross-modalit prediction We can use the matching matrix to design method for cross- ross-modality prediction/translation The following MSE loss corresponds to constructing a prediction function fo such that the barycentric projection Eas (Xo(X,) X; = æz). under M minimizes the squared error for predicting ny:
((0) Σ(x (xi Σ MazJo(zz))
However, this requires evaluating So for all n2 examples from modality (2) for each of the examples in modality (1), In practice, we can avoid this cost with stochastic gradient descent by sampling from modality (2) via M, for each training example (1). To obtain an unbiased estimate of VoL, we need two independent samples from modality (2) for each sample from modality (1),
C(())
(x. fo(t;)) (o,)
ig. P(aylas (m;)
(10)
By taking two samples as in Eq. 11).. we get an unbiased estimator of VL()), whereas a single sample would have resulted in optimizing
upper-bound on equation 9 : for details, see Hartford
al. [2077] where a similar issue arises the gradient of their causal effect estimator, We thus refer to prediction models trained via Eq. (10) as unbiased.

---

6 Experiments
We present a comprehensive evaluation of our proposed methodology on three distinct datasets: (1) synthetic paired images, (2) single-cell CITE-seq dataset (simultaneous measurement of single-cell RNA-seq and surface protein measurements) [Stoeckius et al., 2017]. and (3) Perturb-seq and single- cell image data. In the first two cases, there is ground-trum truth matching that we use for evaluation, but samples are randomly permuted during training. This allows us to exactly compute the quality of the matching in comparison to the ground truth. The final dataset is more realistic setting where ground truth paired samples do not exist, and matching becomes necessary in practice. In this case, we compute distributional metrics to compare our proposed methodology against other baselines.
Experimental Details All models for the experiments are implemented using Torch 2.. PPaszke et al.) 2017] and Pytorch Lightning v2.2.4 [Falcon and PyTorch Lightning Team 2023]. The classifier used to estimate the propensity score is always a linear head top of an encoder E, which is specific to each modality and dataset. All models are saved at the optimal validation loss to perform subsequent matching. Shared nearest neighbours (SNN) is implemented using scikit-learn v1.4. 4. .. [Pedregosa et al., [011]] using a single neighbour, and OT is implemented using the Sinkhorn algorithm implemented in the pot v0.9.3 package (Flamary al. (02T]. Both SNN and OT use the Euclidean distance as the metric. Whenever random variation can affect the results of the experiments, we report quantiles corresponding to variation from different random seeds. Additional experimental details are provided in Appendix D
Description of Baselines Our main baseline, which we evaluate against on all three datasets, is matching using representations learned by the multimodal VAE Yang et al. [2021), which is the only published method that is able to leverage perturbation labels for unpaired multimodal data (they refer to the labels as "prior information").. The standard multimodal VAE loss is a reconstruction loss based on encoder and decoders E,, D; for each modality, plus latent invariance loss that aims to align the modalities in the latent space. In our setting, the multimodal VAE loss further includes an additional label classification loss from the latent space of each modality, i.c., encouraging the encoder simultaneously learn P(t] | (x:)). This additional objective, which acts regularizer for the multimodal VAE, is exactly the loss for our proposed method. To ensure a fair comparison, we always use the same architecture in the encoders E, of multimodal VAE and in our propensity score classifier. The performance differences between propensity score matching and multimodal VAE then represent the effects of the VAE reconstruction objective and latent invariance objectives. For additional baselines, we also compare against a random matching, where the samples are matched with equal weight within each perturbation compare against Gromov- Wasserstein OT (SCOT) [Demetci et al. [022] computed separately within each perturbation. SCOT uses OT directly by computing cost function derived based on pairwise distances within each modality, thus learning a local description of the geometry which can be compared between modalities. For the CITE-seq dataset, we also compare against matching using a graph-linked VAE, seGLUE (Cao and Gao 2022 where the graph is constructed from linking genes with the associated proteins.
sanity check. For datasets (1) and (2), we also
Evaluation Metrics We use the known ground truth matching to compute performance metrics on datasets (1) and (2). The trace and FOSCTTM (Liu et al. (019) measure how much weight M places on the true pairing. However, this not necessarily indicative of downstream performance as similar, but not exact matches are penalized equally to wildly incorrect matches. For this reason, we also measure the latent MSE for dataset (1) and the performance of a CITE-seq gene-to-protein predictive model based on the learned matching for dataset (2). For more details, see Appendix D.I
6.1 Experiment 1: Synthetic Interventional Images
Data We followed the data generating process Eq. (T) with a latent variable Z encoding the coordi- nates of two objects. Perturbations represent different do- inteereeeeeeinss the different dimensons of
Z. The difference between modalities corresponds to whether the objects are circular or square, and fixed transformation of Z, while the modality-specife noise U controls background distortions.
Model and Evaluation We used convolutional neural network adapted from Yang et al. 2021]
as the encoder. We report two evaluation metrics: (I) the trace metric, and (2) the MSE between
7

---

Synthetic Image Data
CITE-seq Data
MSE (L)
Trace (1)
FOSCTTM (L)
Trace (1)
Method
Med (Q1, Q3)
Med (Q1, Q3) x00
Med (Q1, Q3)
Med (QI, Q3)
PS+OT
0.0316
18.329
0.3049
0.1163
(0.0300, 0.0330)
(17.068, 18,987)
(0.3008, 0.3078) (0.1093, 0.1250)
VAE+OT
0.0324
7.733
0.3953
0.0814
(0.0316, 0.0350)
(7 .73, 7.794)
(0.3912, 0,4045) (0.0777, 0.8895)
PS+SNN
0.0552
7.924
0.3
0.0941
(0.0530, 0.0558)
(7.569, 9,504)
(0.3121, 0.3 160) (0.0880, 0.0989)
VAE+SNN
0.0622
3,116
0,3816
0.0612
(0.0571, 1, 0.0676)
(2.818, 3,213)
(0.3760, 0.3822) (0.0588, 0.0634)
SCOT GLUE+SNN GLUE+OT Random
0.0354
0.5964
0.4596
0.0200
0,4412
0.0362
0,5309
0.0323
0.0709
(0.0707,0.0) 0.0714)
Table 1: Alignment metrics results using synthetic interventional image dataset and CITE-seq data.
the matched and the true latents. The latent MSE metric does not penalize close neighbours of the true match (i.e. examples for which || z is small) heavily the trace metric. These "near matches" will typically still be useful on downstream multimodal tasks.
Results In Table I metrics are computed on held out test set over 12 groups corresponding to interventions on the latent position, with approximately 1700 observations per group. A random matching, with weight 1/n, will hence have a trace metric of of 1 1/1700 & 0.588 x 10-3, This implies, for example, that the median performance of PS+OT is approximately 31 times that of random matching. On both metrics, we found that propensity scores matched with OT (PS OT) consistently outperformed other matching methods on both metrics.
6.2 Experiment 2: CITE-Seq Data
Data We used the CITE-scu dataset from the NeurIPS 2021 Multimodal single-cell data integration competition [Lance al. 2022 consisting of paired RNA-seq and surface level protein measurements over 45 cell types. In the absence of perturbations, we used the cell type as the observed label to classify and match within. Note the cell types determined by consensus by pooling annotations from marker genes/proteins. In most cells, the annotations from each modality agreed, suggesting that the label is independent from the modality-specifii noise. We used the first 200 principal components as the gene expression modality, and normalized (but otherwise raw) protein measurements as input.
Model and Evaluation We used fully-connected MLPs as encoders. To assess matching, report (1) the trace, and (2) the Fraction Of Samples Closer Than the True Match (FOSCTTM) ([Demetei et al. 2022]. [Liu et al. 2019) (lower is better, 0.5 corresponds to random guessing). To evaluate against a downstream task, we also compared the performance of random and VAE matching procedures, as well as directly using the ground truth (Mą = 1), on predicting protein levels from gene expression. We trained a 2-layer MLP (the same architecture for all matchings) with both MSE loss and the unbiased procedure as described in Section S Jsing pseudosamples sampled according to the matching matrix. We evaluated the predictive models against ground truth pairs by computing the prediction R. (higher is better) on a held-out, unpermuted, test set.
Results In Table T metrics are computed on a held-out test set averaged over 45 cell types with varying observation counts per group. While interpreting the average trace can be challenging due to group size variations, OT matching on PS consistently outperformed other methods both within and across groups. In these experiments, OT matching on PS was consistently the top performer, often followed by SNN matching on PS or OT matching on VAE embeddings.
We present downstream task performance in Table 2 Note that R² is computed using the sample average across possibly multiple cell types, which explains why random matching within each cell type results in non-zero R2 (see Appendix D.I.. We found that PS OT matching outperforms other methods on this task. Surprisingly, the PS + OT prediction model performed even better on average than training with the standard MSE loss ground truth pairings (though confidence intervals
8

---

CITE-seq Data
PerturbSex/Single Cell Image Data
Method
MSE Loss
Unbiased Loss
In Distribution
Out of Distribution
R Med (Q1, Q3) (t) R Med (Q1, Q3) (t)
KL Med (Q1, Q3) (
KL (L)
Random
0.138 (0.137, 0.140)
0.173
58.806
51.310
(0.170. 0.173)
(58.771, 60.531)
VAB+OT
0.149
0.114
55.483
47.910
(0.1 18,, 0.172)
(0.079, 0.159)
(55.410, 56.994)
PS+OT
0.217
0.233
50.967
43.554
(0.206, 0.223)
(0.207, 0.250)
(50.898, 52.457)
True Pairs
0.224
00.223, 0.226)
Table 2: Cross-modal prediction results using CITE-seq data and PerturbSeq/single cell image data including an out of distribution distance evaluation for PerturbSeq/single cell images.
overlap). This highlights the potential benefit of soft (OT) matching as a regularizer, beyond that of simply reconciling most likely pairs: the soft matching effectively averages over modality specific variation from samples with similar latent states in manner analogous to data augmentation (with an unknown group action).
6.3
Experiment 3: PerturbSeq and Single Cell Images
Data We collected PerturbSeq data (200 genes) and single-cell images of HUVEC cells with 24 gene perturbations and control perturbation, resulting in 25 total labels across both modalities. As preprocessing, we embed the raw PerturbSeq counts into I 128-dimensional space using scVI [Lopez al., 2018) and the cell images into a 1024-dimensional space using pre-trained Masked Autoencoder He et al. 2022 Kraus et al., 2023) to train our gene expression and image classifiers.
Model and Evaluation We used fully connected 2-layer MLP the encoder for both PerturbSeq and cell image classifiers. Similarly to the CITE-seq dataset, we evaluated the matchings based on downstream prediction of gene expression from (embeddings of) images. We used the unbiased procedure to minimize the projected loss Eq. (9) and evaluated on two held-out sets, one consisting of in-distribution samples from the 25 perturbations the classifier was trained on, and an out-of- distribution set consisting of an extra perturbation not seen in training. In the absence of ground truth matching. we assessed three distributional metries between the actual and predicted gene expression values within each perturbation: the L2 norm of the difference in means, the Kullback-Leibhe (KL) divergence, and I-Wasserttein distance (lower indicates better alignment). We report inverse cell-count weighted averages over each perturbation group. Each metric measures a slightly different aspect of fit-the I.2 norm reports a first-order deviation, while the KL divergence is an empirical estimate of the deviation of the underlying predicted distribution, while the 1- Wasserstein distance measures deviations in terms of the empirical samples themselves.
Note that matching is performed using classifiers trained on scVI embeddings, but the cross-modal predictions are generated in the original log transformed gene expression space (i.e. we predicted actual observations, not embeddings). We also evaluated distance measures on an out-of-distribution gene perturbation that was not used in either the matching or training of the translation model.
Results We present KL divergence values for in-distribution and Additional metries show similar patterns and can be found in Appendix D.4 OT PS matching consistently outperforms its VAE counterpart both on in-distribution and duirof-disstibution metrics, supporting our findings the CITE-seq data to the case where ground truth pairs not available.
in Table
6.4 Validation Monitor
As in our Perturb-seq and cell imaging example, the ground truth matching is typically unknown in real problems. It is hence desirable to have an observable proxy of the matching performance
WWe computation of in distribution metrics using random subsamples from the test set. The of disttibbutoonn metric computed small dataset with single perturbation and subsamples not needed.

---

0.50
0.50
P8 + OT
VAE + OT
better) FOSCTTM
0.48
PS + SNN
0.48
VAE +
0.46
0.16
0.42
0.42
0.40
427
0.40
0.38
0.38
0.36
0.36
0.75
1.00
1.25
1.50
1.75
2,00
2.25
10
Decreasing Validation
Decreusing Validation
Figure 2: VAE and classifier validation metries on the CITE-seq dataset. Notice that validation cross-entropy inversely tracks the ground truth matching metrics, and thus can be used as a proxy in practical settings where the ground truth is unknown. The same pattern does not hold for the VAE Yang ct al. 2021], which we suspect is because reconstruction is largely irrelevant for matching.
as a validation during hyperparameter tuning. Figure demonstrates that the propensity score validation loss (cross-entropy) empirically satisfies this role our CITE-seq experiments, where lower validation loss corresponds to better matching performance, as if it were computed with the ground truth. By contrast, we found that the optimal VAE, in terms of matching, had higher validation loss. This empirically supports our intuition that the reconstruction loss minimization requires the VAE to capture modality specific information, i.c., the (ee) variables, which hinders its matching performance.
7 Limitations
Our methods limited to settings where have some signal play the role of an experiment label, but we believe this is where these methods are most needed. Matching is impossible in general-e.3 if you tried match modalities that have shared information, it would clearly fail- but theory formally articulates both where we expect this method to succeed and its limitations. Both (A1) and (A2) are strong assumptions, but the empirical results suggest the method is fairly robust failures.
8 Conclusion
This work presents a simple algorithm for aligning unpaired data from different modalities using propensity scores. The method is very general, requiring only classifier to be trained on each modality, and demonstrates excellent matching performance, which we validate both theoretically and empirically. We also showcase the effectiveness of the matching algorithm a downstream cross-modality prediction task, achieving better generalization compared to random matching, VAE- based matching, and even the ground truth matching on the evaluated dataset. This improved generalization over the ground truth may be attributed to implicitly enforcing invariance modality- specific information; a rigorous investigation of this phenomenon would be interesting for further investigation.
Acknowledgements
We are extremely grateful for the discussions with many external collaborators and colleagues at Recursion that lead to this work. The original ideas for this work stemed from conversations with Alex Tong with feedback from Yoshua Bengio at Mila. We received a lot of helpful feedback from all of our colleagues at Valence Labs, especially Berton Farnshaw and Ali Denton. The single cell image experiments are built code originally written by Oren Kraus and his team.
10

---

References
K. Ahuja, D. Mahajan, Y. Wang, and Y. Bengio. Interventional causal representation learning. In ICML, 2023.
A. Almahairi, S. Rajeshwar, A. Sordoni, P. Bachman, and A. Courville. Augmented eyclegan: Learning many-to-many mappings from unpaired data. In JCML, 2018.
M. Amodio and S. Krishnaswamy. Magan: Aligning biological manifolds. In ICML, 2018.
S. Buchholz, G. Rajendran, E. Rosenfeld, B. Aragam, B. Schölkopf, and P. Ravikumar. Learning linear causal representations from interventions under general nonlinear mixing. In NeurlPS, 2023.
A. Butler, P. Hoffman, P. Smibert, E. Papalexi, and R. Satija. Integrating single-cell transcriptomic data across different conditions, technologies, and species. Nature Biotechmology. 3665):411-42)) 2018.
K. Cao, Q. Gong, Y. Hong, and L. Wan. unified computational framework for single-cell data integration with optimal transport. Nature Communications, 13(1):7419, 2022.
Z.-.. Cao and G. Gao. Multi-omics single-cell data integration and regulatory inference with graph- linked embedding. Nature Biotechnology, 40(10):1458 1466, 2022.
Demetei, R. Santorella, Sandstede, W. Noble, and Singh. Scot: single-cell multi-omics alignment with optimal transport. Journal of Computational Biology, 29(1):3-18, 2022.
A. Dixit, O. Parnas, B. Li, J. Chen, C. P. Fulco, L. Jrony,Amon, N. D. Marjanovic, D. Dionne, T. Burks, R. Raychowdhury, et Perturb-seq: dissecting molecular eircuits with scalable single- cell ra profiling of pooled genetic screens. cell, (07(7))1853-1888 2016.
W. Falcon and PyTorch Lightningg Team. pytorchlightning..a
Pytorch lightning, 2023. URL https: // www
M. M. Fay, O. Kraus, M. Victors, L. Arumugam, K. Vuggumudi, Urbanik, K. Hansen, Celik, N. Cernek, G. Jagamnathan, J. Christensen, B. A. Earnshaw, I. S. Haque, and B. Mabey. Rxrx3:
Phenomics map of biology. bioRxiv, 2023. doi: 10. 10.1100/222....000 22..70000..02007572355.... 5273500. URL https
//www. www.wwiirvv...om org/ weteeeettt/arll/////////2/0
.2.2223..2..00. .2.7355
R. Flamary, N. Courty, A. Gramfort, M. Z. Alaya, A. Boisbunon, S. Chambon, L. Chapel, A. Corenflos, K. Fatras, N. Fournier, L. Gautheron, N. T. Gayraud, H. Janati, A. Rakotomamoniy. Redko, A. Rolet, A. Schutz, V. Seguy, D. J. Sutherland, R. Tavenard, A. Tong, and T. Vayer. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78)11-8, 2021.
A. Foster, Á. Vezér, C. A. Glastonbury, P. Creed, S. Abujudeh, and A. Sim. Contrastive mixture of posteriors for counterfactual inference, data integration and faimesss. In ICML, 2022.
A. Genevay, G. Peyré, and M. Cuturi. Learning generative models with sinkhorn divergences. In AISTATS, 2018.
F. Gossi, P. Pati, P. Chouvardas, A. L. Martinelli, M. Kruithof-de Julio, and M. A. Rapsomaniki. Matching single cells across modalities with contrastive learning and optimal transport. Briefings in Bioinformatics, 24(3), 2023.
L. Gresele, P. Rubenstein, A. Mehrjou, Locatello, and Schölkopf. The incomplete rosetta stone problem: Identifiability results for multi-view nonlinear ica. In UAI, 2020.
M. Gutmann and A. Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In AISTATS, 2010.
Hartford, G. Lewis, Leyton-Brown, and M. Taddy. Deep iv: flexible approach for counterfac- tual prediction. In ICML, 2017.
K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEFF/CVE conference on computer vision and pattern recognition, pages 16000-16009, 2022.
11

---

I. Khemakhem, D. P. Kingma, R. P. Monti, and A. Hyvärinen. Variational autoencoders and nonlinear ICA: A unifying framework. In AISTATS, 2020.
I. Korsunsky, N. Millard, Fan, K. Slowikowski, F. Zhang, K. Wei, Y. Baglaenko, M. Brenner, P.-r. Loh, and S. Raychaudhuri. Fast, sensitive and accurate integration of single-cell data with harmony. Nature Methods, (2812)):2899111)
2019.
O. Kraus, K. Kenyon-Dean, S. Saberian, M. Fallah, P. McLean, J. Leung, V. Sharma, A. Khan, Balakrishnan, S. Celik, et al. Masked autoencoders are scalable learners of cellular morphology. arXiv preprint arXiv:2309. 16064, 2023.
C. Lance, M. D. Luecken, D. B. Burkhardt, R. Cannoodt, P. Rautenstrauch, A. Laddach, A. Ub- ingazhibov, Z.-J. Cao, K. Deng, Khan, al. Multimodal single cell data integration challenge: Results and lessons learned. In NeurlPS 2021 Competitions and Demonstrations Track, pages 162-176, 2022.
J. Liu, Y. Huang, R. Singh, J.-P. Vert, and W. Noble. Jointly Embedding Multiple Single-Ceth Omics Measurements, In 19th International Workshop on Algorithms in Bioinformatics (WABI 2019), 2019.
M.-Y. Liu, T. Breuel, and J. Kautz. Unsupervised image-to-image translation networks. NeurlPS, 2017.
R. Lopez. Regier, M. B. Cole, M. I. Jordan, and N. Yosef. Deep generative modeling for single-cell transcriptomics. Nature methods, (5(12):0053-1088 2018.
A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang. Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation pytorch. In NIPS- W, 2017.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830 2011.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
D. B. Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5):688, 1974.
J. Ryu, R. Lopez, C. Bunne, and A. Regev. Cross- matching and prediction of perturbation responses with labeled gromov-wasserstcin optimal transport. arXiv preprint arXiv:2-05.008838, 2024.
G. Schiebinger, J. Shu, M. Tabaka, B. Cleary, V. Subramanian, A. Solomon, J. Gould, S. Liu, S. Lin, P. Berube, et al. Optimal-transpea analysis of single-cell gene expression identifies developmental trajectories in reprogramming. Cell, 176(4):928- 943, 2019.
M. Spivak. Calculus on Manifolds: Modern Approach to Classical Theorems of Advanced Calculus. CRC press, 2018
C. Squires, A. Seigal, S. S. Bhate, and C. Uhler. Linear causal disentanglement via interventions. In ICML, 2023.
M. Stoeckius, C. Hafemeister, W. Stephenson, B. Houck-Loomis, P. K. Chattopadhyay, H. Swerdlow, R. Satija, and P. Smibert. Simultaneous epitope and transcriptome measurement in single cells. Nature methods, 14(9):865-868, 2017.
N. Sturma, C. Squires, M. Drton, and C. Uhler. Unpaired multi-domain causal representation learning. In NeurlPS, 2023.
A. Tong, Huang, G. Wolf, D. Van Dijk, and S. Krishnaswamy. Trajectorynet: dynamic optimal transport network for modeling cellular dynamics. In ICML, 2020.
12

---

A. van den Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1e0.110777.0535 2018
C. Villani. Optimal Trunsport: Old and New, volume 338. Springer, 2009.
J. von Kügelgen, M. Besserve, L. Wendong, L. Gresele, A. Kekić, E. Barcinboim, D. M. Blci, and B. Schölkopf. Nonparametric identifiability of causal representations from unknown interventions. In NewrlPS, 2023.
Q. Xi and B. Bloem-Reddy. Indeterminacy in generative models: Characterization and strong identifiability. In AISTATS, 2023.
K. D. Yang, A. Belyaeva, S. Venkatachalapathy, K. Damodaran, A. Katcoff, Radhakrishman, G. Shivashankar, and C. Uhler. Multi-domain translation between single-cell imaging and sequenc- ing data using autoencoders. Nature Communications, 12(1):31, 2021.
J.-Y. Zhu, T. Park, Isola, and A. A. Efros. Unpaired image-to-image translation using cycle- consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223-2232, 2017.
13

---

A
Relaxing (A1)
Relaxing Assumption Consider the propensity score
T(blee))) = P(1 x(c.0)
(11)
where we do not necessarily require (1e) (te) 11t 2(0), and thus we obtain
x(x(1.1))) P(t|z = (1) ( = (2)) + P(t = 20t, 200000 = =2(²)= T(x(2.0),
(12)
see the proof of Proposition 3.1 for details.
Suppose that the two observed modalities are indeed generated by shared {z>)==1, but where the indices of modality are potentially permuted, and with values differing by modality specific information:
{af = ( (x,
)) (=, - (12) uj (2) 13=
(13)
where j = (i) denotes permutation of the sample indices. Under (A1), we would be able to find some j such that π (a: ((()))))) = πaa (2.(2))) for each i.
Matching via OT can allow us to relax (A1) in a very particular way. Consider the simple case where t E {0, 1], so that π can be written in a single dimension, e.g., P(t = 1|X(e.) = o(e.)) ) [0. 1]. In this case, exact OT is equivalent to sorting (x:"")) and T(x(2.0)) and matching the sorted versions 1-to-1. Under (AI), the sorted versions will be exactly equal. relaxed version of (AI) that would still result the correct ground truth matching is to assume that affects U(1) and (122 differently,
but that the difference is order preserving, or monotone. Denote (T(I(1+t) ((.; = (-. (1) (2)
as the true pairing, noting that we use the same index We require the following:
(Ti
(1) Tie (=
(2) 12
0,
Vi,, 1,...,
(14)
This says that, even if T. (1) * (2) that their relative orderings will still coincide. Then, exact OT will still recover the ground truth matching. See Fig. for a visual example of this type of monotonicity.
For example, suppose that is chemical perturbation of a cell, and thus TT .T can be seen as measure of biological response the perturbation, e.g., in a treated population, Ta, > Ti, indicates samples i had stronger response than sample i2, as perceived by the first modality indexed by i. Then, this monotonocity states that we should see the same Th > Ja the other modality as well, if the samples i and i2 truly corresponded to jı and j2.
(2)
A.1 Cyclic Monotonicity
We can see the monotonicity requirement Eq. 14 as the monotonicity of the function with graph (m: (1) (2)) [0, 1j2. In higher dimensions, require that the "graph" satisfies the following cyclic monotonicity property [Villani] 2009]:
Definition A.1. The collection {(T,') T. (2) JE is said to be c-cyclically monotone for some cost
(1)
function c, if for any 1,..., N, and any subset of pairs (z((), क. )...., ( (1((1 .n (2)), we have
N
N S
Σ ( (1)
((2)
(1) ((T ==1
m.
(15)
Importantly, we define Āx+1 TT, so that the sequence represents cycle.
Note in our setting, the OT cost function the Euclidean distance, c(x, v) = || yl12. It is known that the OT solution must satisfy cyclic monotonicity. Thus, if the true pairing is uniquely cyclically monotone, we can recover it with OT. However, we are unaware common violations of (A1) that would satisfy cyclic monotonicity.
14

---

P
Pulza!o
Ground truth
Ground truth
Modality
masching
Modality
Modality
matchiig
Modiality
Figure 3: OT matching allows for / to have different effects on the modality specific information, here u: (1) and u ( as long as they can be written as transformations that preserve the relative order within modalities. Exact OT in 1-d always matches according to the relative ordering, and thus exhibits this type of "no crossing" behaviour shown the figure on the left. The figure on the right shows case where we would fail to correctly match across modalities because of the crossing shown in orange.
B Shared Nearest Neighbours Matching
Using the propensity score distance, we can compute nearest neighbours both within and between the two modalities. We follow Cao and Gao 2022 and compute the normalized shared nearest neighbours (SNN) between each pair of observations as the entry of the matching matrix. For each pair of observations (m] "
(1) (2)), define four sets:
(1) 11y: the k nearest neighbours of mi amongst of itself.
(T, Ji=1 m.
(1) an (1)
is considered neighbour
12y: the nearest neighbours of z) amongst
(2)
[T j(=1-
(1)
21 the k nearest neighbours of u." amongst { 1==1-
(1)
12)
22,y: the k nearest neighbours of i amongst { J==1- mi is considered neighbour of itself.
(2)
(2) (2)
Intuitively, if mT and π. (2) correspond to the same underlying propensity score, their nearest neigh- bours amongst observations from each modality should be the same. This measured as a set difference between 11, and 12;j, and likewise for 21,j and 22nj. Then, modified Jaccard index is computed as follows. Define
Jo= 11: n 12 + [21, n 22:yl.
(16)
the sum of the number of shared neighbours measured in each modality. Then, we compute the following Jaccard distance to populate the unnormalized matching matrix:
Jij
M,
(17)
4k Jky
where notice that 4k = |11,,|+ |12;|| + | + |22; since each set contains k distinct neighbours, and thus 0 < Myj 1, as with the standard Jaccard index. Then, normalize each to produce the final matching matrix:
Maj Dim Mij
Mi
(18)
Note Mij is always well defined because π. and TT selves.
(2)
are always considered neighbours of them-
Lemma B.1. My has at least non-zero entry in each of its rows and columns for any number of neighbours I.
Proof. We prove that Jıy > for least one j each i, which equivalent to My > Fix an arbitrary i. 21.j by definition is the same set for every By the assumption of k 2 it is non-empty,
15

---

(2) so there exists zj- E 21. Since -/ is neighbour of itself, we have (2)
--- E 22 showing that
Jág > O. The same reasoning applied to 11 and 12 also shows that Jij for at least one in each j.
C Proofs
C.1 Proof of Proposition 3.1
Proof. Let xle) denote the observed modality and 2, u(e) be the unique corresponding latent values.
By injectivity,
[(x(1)) = (() = x0(0)
P(ttZ =2,U = ule))
= P(ttZ = =2 = T(2),
(19)
for e = 1, since we assumed U 111 T (x) = (2) as random variables. Now, a classical result Rubin 1974 gives that Z.it T (Z), and that for any other function 6 (a balancing score) such that Z ILt b(2), we have ((2 g(((2)) The first property written in information theoretic terms yields,
Z. Since this holds pointwise, it shows that π x(x()) =
1lt, 2 क m()) (t,, 2 ((X(e))) 0,
(20)
since T ((X(e))= T (Z())) as random variables, required.
C.2 Proof of Proposition 3.2
Proof. In what follows, write π to be the restriction its domain where it is strictly positive, The i-th dimension of the propensity score can be written as
p(z|t i)p(t = i)
(m(²)), p(t i12)
(21)
E p(2lt 1)p(t i)
which, when restricted to be strictly positive, maps to the relative interior of the T-dimensional probability simplex. Consider the following transformation:
(T(2))
h(T(2)); log
(22)
(π(2)))
= log(p(z i)) log(p(z|t CC,
(23)
where C = log(p((: i)) log(p(t 0)) is constant in and that h(T(2))) = 0. Ignoring the constant first dimension, we can view as an invertible map to IR.". Under this convention, the map h _2 T is smooth (log is smooth, and the densities are smooth by assumption). Since is smooth, cannot be injective if T (Spivak 2018). Finally, since is bijective, this implies that cannot be injective.
D Experimental Details
D.I Evaluation Metrics
D.I.I Known Ground Truth
In the synthetic image and CITE-seq datasets, ground truth matching is known, and we can evaluate the quality of the synthetic matching directly against the truth. In these cases, the dataset sizes are necessarily balanced, so that = ni 7/2. In each case, we evaluate the quality our n n matching matrix M, which we compute within samples with the same L. Our reported results are then averaged over each cluster. Note we randomize the order of the datasets before performing the matching avoid pathologies.
16

---

Trace Metric Assuming the sample indices correspond to the true matching, we can compute the average weight on correct matches, which is the normalized trace of M:
Mr(M) = M
(24)
As baseline, notice that a uniformly random matching that assigns Mi = 1/n for each cell yields Tr(MF) and hence will obtain metric of 1/n. This metric however does not capture potential failure modes of matching. For example, exactly matching one sample, while adversarially matching dissimiliar samples for the remainder also yields a trace of 1/n, which is equal to that of a random matching.
Latent MSE On the image dataset, we have access to the ground truth latent values that generated the images, z = {z )= We compute matched latents as Mz, the barycentric projection according to the matching matrix. Then, to evaluate the quality of the matching in terms of finding similar latents, we compute the MSE:
MSE(M) = -11 Mzl3.
(25)
FOSCTTM We do not have access to ground truth latents in the CITE-seq dataset, so use the Fraction Of Samples Closer Than the True Match (FOSCTTM) [Demetci et al. 2022 Liu al. 2019
M to project x!²) { to x11) = Mx Then, we can compute a cross-modality distance as follows. For each point in x(1) we compute the Euclidean distance to each point in x()), and compute the fraction of samples in (!)
alternative matching metric. First,
{2; ;=1 as
that are closer than the true match, We also repeat this for each point in x(",, computing the fraction
of samples in (1) in this case. That is, assuming again that the given indices correspond the true matching, we compute:
(OS =
1dd(x) (x, < d(x(), xx) (211))
(26)
2n
n2
Σ inl
(d(x))) (1) (x(")) d(x)g) x ( ||
(27)
j=1
where notice that this evaluates M through the computation (1) Mx As a baseline, should expect a random matching, when distances between points are randomly distributed, to have an FOSCTTM of 0.5.
Prediction Accuracy We also trained a cross cross-modality prediction (translation) model fo.M to predict
CITE-seq protein levels from gene expression based on matched pseudosamples. Let ((1) = { x2)) = } denote protein and gene expression, respectively. We trained a simple 2-layer MLP minimizing either the standard MSE, using pairs (æi, &ty), ), â Mj., following the projected loss with unbiased estimates in Section 5 Each batch in general consists of samples from all f, but the î sampling step occurs within the perturbation. Let x {fo.M (x,)). We report the R randomly held-out test set of ground truth pairs (again, consisting of samples from all (), which is defined as the following:
MSE(x tteet &
R2 (fo.M)
(28)
MSE(x Lest (
where x (1)
(est is the naive mean (over all perturbations) estimator which acts as a baseline.
D.1.2 Unknown Ground Truth
We train a cross-modality prediction model to predict gene expression from cell images based on matched pseudosamples in the same way as in CITE-seq, but only using the projected loss with unbiased estimates. Denote this model for matching matrix M by fe.M.
17

---

Because we do not have access to ground truth pairs within each perturbation, we resort to distribu- tional metrics. Let x = { ( x(²) = { ) denote gene expression and cell images in
a held out test set respectively in perturbation t. Let x ( We compute empirical
versions of statistical divergences
D.(Jo.) == D(x (1) x(
(29)
where D is either the L2 norm of the difference in empirical mean, empirical Kullback-Leibler divergence or I-Wasserstein distance. We report these weighted averages of Dr over the perturbations t according to the number of samples in the modality of prediction interest.
D.2 Models
In this section we describe experimental details pertaining to the propensity score and VAE (Yang et al. 2021]. SCOT (Demetci et al.] [022] and scGLUE (Cao and Gao [022] are used according to tutorials and recommended default settings by the authors.
The propensity score approach minimizes the standard cross-entropy loss for both modalities. The VAE includes, in addition to the standard ELBO loss (with parameter À on the KL term), two cross-entropy losses based on classifiers from the latent space: one, weighted by parameter to classify as in the propensity score, and another, weighted by a parameter B, that classifies which modality the latent point belongs to.
Loss Functions
Hyperparameters and Optimization We the Adam optimizer with learning rate 0.0001 and one cycle learning rate scheduler. We follow Yang et al. (202T) and set a = 1, 3 = 0.1, but found that X = 10-9 9 (compared to x = 10-7 in Yang et al. [021] resulted in better performance. We used batch size 256 in both instances and trained for either 100 epochs (image) or 250 epochs (CITE-seq).
For the VAE and classifiers of experiment 3, we use an Adam optimizer with learning 0.001 and weight decay 0.001 and max epoch of 100 (PerturbSeq) and 250 (single cell images) using batch sizes of 256 and 2048 correspondingly. We follow similar settings as Yang et al. [2021] and implement a == 1 with = 10 99, and since we do not have matched data, 3 = O. For the cross-modal prediction models in experiment 3, we use Stochastic Gradient Descent optimizer with learning rate 0.001 and weight decay 0.001 with max epochs 250 and batch size 256. We implement early stopping with delay of 50 epochs which we then checkpoint the last model to use for downstream tasks
Architecture For the synthetic image dataset, we use an 5-layer convolutional network (channels = 32, 54, 128, 256, 512) with batch normalization and leaky ReLU activations, with linear heads for classification (propensity score and VAE) and posterior mean and variance estimation (VAE). For the VAF, the decoder consists of convolutional transpose layers that reverse those of the encoder.
For the CITE-seq dataset, we use a 5-layer MLP with constant hidden dimension 1024, with batch normalization and ReLU activations (adapted from the fully connected VAE in Yang et al. [202]] as both the encoder and VAE decoder. We use the same architecture for both modalities, RNA-seq (as we process the top 200 PCs) and protein.
For the PerturbSeq classifier encoder, we use a 2-layer MLP architecture. Each layer consists of a linear layer with an output feature dimension of 64, followed by Rectified Linear Unit (ReLU) activation, Batch Normalization, and dropout (p=0.1). A final layer with Leaky ReLU activation that brings dimensionality to 128 before feeding into a linear classification head with an output feature dimension of 25.
For the single-cell image encoder classifier, we use a proprictary Masked Autoencoder [raus et al. 2023 to generate (024-dimensional embeddings. Subsequently, 2-layer MLP is trained on these embeddings. Each MLP layer has linear layer, Batch Normalization, and Leaky ReLU activation. The output feature dimensions of the linear layers are 512 and 256, respectively, and the latent dimension remains at 1024 before entering a linear classification head with an output feature dimension of 25.
Optimal Transport We used POT Fllmary et al. 2021 to solve the entropic OT problem, using the log- sinkhom solver, with regularization strength ? 0.05.
18

---

Figure 4: Example pair of synthetic images with the same underlying
D.3 Data
Synthetic Data We follow the data generating process Eq. to generate coloured scenes of two simple objects (circles, or squares) in various orientations and with various backgrounds. The position of the objects are encoded in the latent variable x, which is perturbed by no-inneervention (setting to fixed value) randomly sampled for each L. Each object has an æ and coordinate, leading 4-dimensional z, for which consider 3 separate interventions each, leading to 12 different settings. The modality then corresponds to whether the objects are circular or square, and a fixed transformation of while the modality-speeific noise U controls background distortions. Scenes are generated using rendering engine from PyGame f((), Example images are given in Fig. 4
CTTE-seq Data We also use the CITE-seq dataset from Lance et al. [2022] as real-world benehmark (obtained from GEO accession GSE194122) These consist of paired RNA-seq and surface level protein measurements, and their cell type annotations over 45 different cell types. We used scanpy, standard bioinformatics package, to perform PCA dimension reduction on RNA-seq by taking the first 200 principal components. The protein measurements (134-dimensional) was processed raw form. For more details, see Lance et al. [2022]
PerturbSeq and Single Cell Image Data We collect single-cell PerturbSeg data (200 genes) and single-cell painting images in HUVEC cells with 24 gene perturbations and control perturbation, resulting 25 labels for matching across both modalities. The target gene perturbations are selected based on the 24 genes with the highest number of cells affected by the CRISPR guide RNAs targeting those genes. The PerturbSeq data filtered to include the top 200 genes with the highest mean count, then normalized and log- frg-transfommd. The single-cell painting images are derived from multi-cell images, with each single-cell nucleus centered within 32x32 pixel box. We use VI et al. (2088] to embed the raw PerturbSeq counts into 1 128-dimensional space before training the gene expression classifier. Similarly, we train our image classifier using 1024-dimensional embeddings obtained from pre-trained Masked Autoencoder Kraus et al. [2023) He et al. (2022). Following matching, we perform cross- moddaltyy translation from the single- embeddings to the transformed gene expression counts.
D.4 Supplementary Results
19

---

In Distribution
Out of Distribution
Wasserstein-1 ())
1.2 Norm (L)
Wasserstein-1 ()) 1,2 Norm (1)
Method
Med (Q1, Q3)
Med (Q1, Q3)
PS+OT
4.199
3,280
5.394
7.219
(4.173, 4.226)
(3.267, 3.284)
VAE+OT
4.339
3.490
5.629
7.444
(4,314, 4,348)
(3 486, 3.495)
Random
4,499
3,826
6.239
7.793
(4,478, 4.525)
(3.823, 3.828)
Table 3: Wasserstein-1 I and L2 norm distance values for PerturbSeq and single cell images experiments
where distance is evaluated between ross-modal predictions and actual gene expression values.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract simply describes the methodology and our experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We included a limitations section that explains the limitations of the method.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
20

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Our assumptions clearly stated, and the reasoning why they are required clearly justified, in (A1) and (A2).
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Full code, complete with scripts to obtain synthetic publicly available data for experimental datasets (1) and (2), is provided as supplementary material. Dataset (3) is not released publicly, but enough details given in the paper, and in descriptions of the method, to reproduce on similar data.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
21

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to deseribe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Same justification above.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. ec/
mublic/guidess httpgguddde//oeeeumiiieesse/ee
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.com www.c//wwwpgpcuudclocoeoooooppee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The hyperparameters are reported in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
22

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Experimental results are reported with quantiles corresponding to variations from random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emorr bars, confi-
dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative етгог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: The neural network architectures we use are rather small scale, using relatively shallow MLPs or small convolutional nets that do not require large computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http::::///uursss..... httphhgghhp.s.e...ppp...cc//apap
Answer: [Yes]
Justification: We do not see a potential ethical issue arising from the research conducted in this paper.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
23

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper describes methodology that may advance the sciences which can have many downstream societal impacts, but we do not otherwise see an immediate potential for societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
24

---

Answer: [Yes]
Justification: All packages and datasets used are properly versioned and cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswithcode
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The code in its current form is released only for reviewing purposes and not intended for public use. It is otherwise well-documented given the context.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
25

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
26


Reviews:
[{"review_id": "rOfJjelgDe", "reviewer": "Reviewer_iHqN", "summary": "This paper presents a new method for pairing unpaired examples across different modalities using the labels of the examples. The method essentially trains a classifier for predicting labels for examples in each modality, then uses the classifier's logits across two modalities to calculate a similarity matrix. Empirical results show that the method is promising in aligning data from different modalities.", "questions": "Is it possible that the use of labels as the sole signal to match examples from two modalities results in example clustering based on labels? i.e. examples from modality 1 that belong to class t are matched strongly against those from modality 2 that also belong to class t?\nIn that case, is the method still beneficial compared to not doing any matching (i.e. just using data from individual modalities)? Having a discussion on this in the paper will be useful.", "limitations": "There's a short section on limitations of the method, explaining (1) the reliance of the method on labels, and (2) the strong assumptions made in the method.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is written clearly.\n- The problem of aligning unpaird data in multimodal learning is quite important since in many cases there's limited or no access to paired data for representation learning. This paper proposes a new method for it, paving the way to future work in this area.\n- The method is relatively straightforward and easy to follow.\n- Empirical results show that the method perform well compared to the baseline.", "weaknesses": "The method uses the information in the labels in order to match labels via classifiers. There are two drawbacks with this approach: (1) the quality of matching depends directly on the choice of the classifier and its capacity/complexity. (2) more importantly, in many multimodal representation learning settings the label signal is unavailable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dependence on a classifier**\n>There are two drawbacks with this approach: (1) the quality of matching depends directly on the choice of the classifier and its capacity/complexity.\n\nIt is true that the quality of the matching depends on the choice of classifier and its capacity, but because you can easily validate the classifier’s performance on a validation set, this makes it relatively easy to choose a good classifier on which to base the matching. Training a classifier is not always trivial, but it is far easier than, e.g. training a variational autoencoder with a shared latent state as in Yang et al. [2021]. \n> more importantly, in many multimodal representation learning settings the label signal is unavailable.\n\nThe assumption of perturbation labels is a restriction but it is extremely common in the biological settings that we study (note the concurrent work we cited, Ryu et al. [2024], which was posted just before the submission deadline, also study the same setting motivated by its biological applications). Even when experimental data is not available, it is often possible to rely on weaker class labels. E.g. in our real data experiments on CITE-Seq data, we used cell type as a label.\n\n**Question about labels**\n> examples from modality 1 that belong to class t are matched strongly against those from modality 2 that also belong to class t” \n\nIn all of our experiments, we perform matching within a known class. So all the examples from class $t$ in modality 1 are restricted to only match to examples from modality 2 in class $t$ (and vice versa). This is why random matching still gives fairly decent results on cross modality prediction tasks (Table 2); we believe that this is roughly equivalent to the “not doing any matching (i.e. just using data from individual modalities)” baseline you refer to (but please let us know if you have something else in mind so that we can update the camera ready). The additional benefit over random comes from matching within a class. Note that matching results in a significant improvement in $R^2$ and the KL metrics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MMW1dbJ2VI", "reviewer": "Reviewer_XqRN", "summary": "The paper proposes a novel matching method for unpaired multimodal (bimodal) data. It aligns unpaired samples across modalities using a propensity score. Based on additional treatment information, the propensity score is learned and used to align multimodal samples.\nThe method is evaluated on three datasets: one imaging dataset and one single-cell dataset (both with available matching ground truth); and one additional single-cell microscopy dataset without available ground truth.", "questions": "- what are the labels/perturbations in the image-based dataset? I did not see the information in section 6 (but maybe I missed it. Apologies in that case)\n- section 6.2: what is the reason for using the first 200 principal components for the gene expression modality (and not for the protein data as well)?\n- does assumption A1 hold in the datasets used? Is there a way to check whether the assumption holds?\n- there are some typos:\n    - line 90: the our instead of the or our\n    - line 168: form instead of from\n    - line 227: against on instead of against or on", "limitations": "The authors address some of the limitations.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Interesting idea: using a propensity score that leverages (often) available labels/treatments to learn a classifier is interesting.\n- Important Problem: A long line of multimodal methods describes how to leverage best-paired samples, but little work has been done on aligning unpaired samples. Besides biology, there is (probably) a lot of unpaired multimodal data.\n- well written and easy to read", "weaknesses": "- Final purpose of the method: To me, it was not fully clear what the final goal/purpose of the method is. Is it (1) the aligning of unpaired multimodal samples? Or (2), learning from unpaired, multimodal samples while aligning the samples? If (1) is the goal, what can we conclude from the improved results? If (2) is the goal, it would be interesting to see if the improved alignment/matching results in better downstream task/learning from multimodal data compared to other works (e.g., Yang et al., 2021) or just unimodal methods without any pairing of data.\n- Evaluation: It is not straightforward to understand the used metrics (maybe also related to the point above) and relate the performance metrics to the quality of the method (besides a relative performance between different alignment methods). Maybe reporting some classification-based performances for alignment (is the assigned sample from the other modality the correct one) or some downstream task performance (what task can we solve using the trained method?) would help.\n- Although the method is advertised as multimodal, no experiments or proofs use more than two modalities. Maybe the term bimodal would be more accurate? Otherwise, an experiment or derivation for more than two modalities would be appreciated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Final purpose of the method**\n\nWe are most interested in (2) “learning from unpaired multimodal samples”. Our cross modality task (see Table 2) explicitly evaluate this and show that you do indeed see a significant improvement by learning from matched data (the difference in performance between our approach and Yang et al. shows that one gets a larger improvement from better matching).\n\n**Evaluation** \n\nWe agree that the metrics can be difficult to interpret, we will include more details in the main text for clarification. \n\n> Maybe reporting some classification-based performances for alignment (is the assigned sample from the other modality the correct one)\n\nWe report two metrics that measure how close the estimated matchings are to the correct one, when the ground truth is known (image and CITE-seq data). The trace metric corresponds exactly to a measure of how often the assigned sample from the other modality is the correct one ($Tr(M) = \\sum_{i} M_{ii}$, where $M_{ii}$ is the weight given to the true pairing).The FOSCTTM is a standard metric in the matching literature, where a value of 0.3 indicates that, on average, the true sample is within the closest 30% to the true sample.\n\nBoth of these metrics are standard (which is why we report them), but essentially evaluate your task (1), “aligning unpaired multimodal samples”. For what you called task (2) (learning from unpaired samples), what matters more is (i) how good a proxy the matched sample is for the true match, and (ii) performance on downstream tasks. On synthetic data, we can evaluate (i) by testing how close the latent state of the matched sample is to the ground truth match (MSE in table 1). This metric is a better proxy for downstream performance because if there are a number of samples that are very similar to the true match, then it is likely that you won’t get the exact match correct, but that matching to any of the close samples will still lead to good results (because they’re good proxies for the true match). As discussed above, we also report (ii) in Table 2 (see below for more detail). \n\n> or some downstream task performance (what task can we solve using the trained method?)\n\nWe emphasize that the metrics above that measure may not be indicative of downstream especially when using soft matching techniques such as OT. The $R^2$ metric on the prediction task (section 5) can be interpreted as the amount of information that is shared between modality 1 and the paired modality 2. E.g., an $R^2$ of 0.233 can be roughly interpreted as the paired modality 1 explains 23.3% of the variation in modality 2, while even the ground truth modality 1 only explains 22.4% (Table 2), indicating that the matched samples are statistically indistinguishable from the true matches in terms of shared information. \n\n**Multimodal vs. bimodal**\n\n> advertised as multimodal, no experiments or proofs use more than two modalities.\n\nUsing the term “multimodal” for settings with two or more modalities is standard in the literature. For example, Manzoor et al. 2024 state,\n> Multimodal systems utilize **two or more** input modalities ... which could be different from the inputs. [emphasis added]\n\nSimilarly, most of multimodal methods discussed in Guo et al [2019] involve only two modalities, in some cases with no obvious way of extending them beyond just two modalities.\n\nThat said, the multimodal matching could in theory be done by iteratively applying bivariate matching to a base modality, e.g., sampling a tri-modal observation $(x_i^{(1)}, x_{j}^{(2)}, x_{k}^{(3)})$ would involve sampling $x_j^{(2)}$ from $M^{(1,2)}$ and $x_k^{(3)}$ from $M^{(1,3)}$. One can also think of distance functions involving multi-pairs of propensity scores, e.g., a cost function $c(\\pi_i, \\pi_j, \\pi_k)$, which can be aligned with [multi-marginal optimal transport](https://arxiv.org/pdf/1406.0026), but our main contribution is to introduce the common space defined by the propensity score. \n\nManzoor, M. A., Albarri, S., Xian, Z., Meng, Z., Nakov, P., & Liang, S. (2023). Multimodality representation learning: A survey on evolution, pretraining and its applications.\n\nGuo, W., Wang, J., & Wang, S. (2019). Deep multimodal representation learning: A survey. \n\n**Questions**\n\n> what are the labels/perturbations in the image-based dataset? \n\nThe perturbations are do-interventions on the latents (location) of the objects in the rendered images, it is briefly described on l253.\n\n> section 6.2: reason for using the first 200 principal component?\n\nWe chose to use principal components for the GEX data due to sparsity (over ~ 20k dimensions), which is common pre-processing in bioinformatics pipelines. Note this was only used as deterministic pre-processing to train the classifiers and in theory could be avoided with a flexible enough classifier (e.g., a GEX-specific encoder). The protein data is only ~130 dimensions and the measurements are naturally more dense and continuous, and we found that a classifier trained directly on raw measurements performed adequately. \n\n>. does assumption A1 hold in the datasets used? Is there a way to check whether the assumption holds?\n\nWhether or not A1 holds, depends in part on the extent to which A2 holds: any part of Z that is not reflected in modality 1 essentially becomes part of $U^{(2)}$ and vice versa. Because CITE-Seq only measures surface proteins of the cell, there is likely part of the latent cell state that is not reflected in the proteomics assays, and some of that state is surely affected by the cell type, so we should not expect assumption A1 to hold exactly. That said, the assumptions remain useful for reasoning about the conditions under which matching is theoretically possible (see also our response to R3) and guiding data collection. E.g. our theory suggests that we will have better matching performance with an assay that measures all proteins, and not just surface proteins.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MAAqEBYspL", "reviewer": "Reviewer_qEyT", "summary": "This work addresses the problem of aligning unpaired samples from multimodal data, whereby the problem is to find sample $x_i$ from modality $i$ that is best “related” to sample $x_j$ from modality $j$, when those two samples come from the observation of the same phenomenon according to two different modalities (e.g. sensors).\n\nThis is an important problem that arise in many endeavors including biology, and that has been somehow overlooked in the literature of multimodal representation learning, whereby large quantities of paired data (images and their corresponding captions) are available.\nIn a nutshell, the idea proposed in this work is as follows. Provided the following assumptions are valid, that is there exists a common latent space $Z$, which is entangled with a “perturbation signal” or label $t$, but that it is independent of modality specific noise $U_{i,j}$, then it is possible to estimate a common space for matching samples through the “bridge” across modalities offered by the perturbation signal $t$.\n\nIn practice, this amounts to training a classifier to predict $t$ given one of the available modality, which allows the creation of a transformed dataset endowed with a Euclidean distance. Such distance can be used to compute a matching matrix $M$, whereby entry $(i,j)$ represents the likelihood of $x_i$ being matched to $x_j$. Matching matrix $M$ can be obtained efficiently through the lenses of optimal transport matching with entropic regularization.\nIt should be noted that $M$ can then be used to construct pseudo-samples for any learning task that requires paired samples $(x_i, x_j)$ by replacing them with $(x_i, \\hat{x}_j)$, \n\nwhere the second term is obtained through the application of the matching matrix to a structural equation $f_\\theta$, which is a parametric function with learnable parameters.\n\nSeveral experiments on synthetic data, and a dataset whereby ground truth pairing is available, as well as experiments where no ground truth is available, indicate that the proposed method, in combination with optimal transport matching, provides an improvement over state of the art matching methods.", "questions": "* Would it be possible to explain (and eventually add to the appendix) in more detail the setup from [Yang et al, 2021] which is used in this work? More precisely, it would be great to have the definition of the multimodal autoencoder, the existence of a shared latent space, and the details of the perturbations $t$ and their effects on $Z$. This would really help formalizing better the use of $t$ as a bridge between modalities.\n\n* Can you provide a thorough discussion about the scalability of the proposed approach, the impact of imbalance in the size of the multimodal data (e.g. $n_1 >> n_2$), and the possible application to other domains with more abundant data?\n\n* Can you elaborate more on the baselines you compare your method to? Lines 235-239 are not detailed enough to fully appreciate the differences w.r.t. the proposed method.\n\n======= POST REBUTTAL UPDATE =======\nDear authors, thank you for the rebuttal. I have raised my score.", "limitations": "Yes, the authors acknowledge that their work relies on strong assumptions, but suggest that empirical results indicate their method is robust to failures. Nevertheless, I could not find the discussion on the experimental results that clearly indicates such robustness.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* This work presents a simple method to produce a matching matrix that can be used to pair multimodal data. The algorithm only requires training of two classifiers and the execution of the Sinkhorn’s algorithm to compute the optimal transport matching.\n\n* Experiments show that the proposed method using propensity scores in combination with optimal transport matching outperform the literature in terms of MSE and other specific metrics.\n\n* Experimental result indicate, surprisingly, that the proposed method implicitly enforces invariant to modality specific information, which endows propensity score with an improved generalization, although this specific phenomenon is only hinted at and requires additional studies", "weaknesses": "* The proposed method relies on strong assumptions, which are only superficially discussed. Relaxation of the conditional independence assumption A1 is discussed in appendix A, but only for the case of a small perturbation set $t \\in \\{ 0,1 \\}$, which allows the authors conclude that under exact optimal transport, this is equivalent to operate on order preserving effects of $t$. The validity of the method also  relies on A2, which allows computing propensity score from data alone, and not the shared latent space $Z$.\n\n* The overall perturbation method, as well as the existence of a shared latent representation $Z$ is only vaguely described. I have looked up the key reference [Yang et al, 2021], but the lack of detail is uncomfortable. It is possible to get an intuition of how the method works, but I would prefer such important technical details to be spelled out clearly.\n\n* There is no mention of the scalability of the proposed approach. Building a matching matrix can be a daunting task for some domains whereby a large amount of unpaired training data is available. In this case, the matching matrix could be either a big square matrix, or a very skinny and tall matrix that could pose some computational challenges. It would also be important to discuss about what would be the best operating scenario for the proposed method in terms of the number of samples per modality. In very unbalanced scenarios, would the classifiers using one or the other modality really behave similarly as per Equation (5)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**“Strong” Assumptions**\n\nThe problem that we address in this paper is obviously impossible in general: for example, if I gave you data from two modalities that are completely unrelated to each other (e.g. text from the complete works of Shakespeare, and images of cells under a microscope), there would be no correct solution to the problem because there is no shared information between the two modalities. But not all multimodal matching problems are as pathological as this: if we have shared information and some way of tying them together, then matching is possible. \n\nTo our knowledge, there are no prior works tackling the matching problem for which any theoretical guarantees exist on the validity of the matching itself. E.g., Ryu et al., [2024] provides theory on obtaining OT solution itself, but does not describe whether the solution corresponds to a valid matching. In this case, the pathological example above may have an OT solution, but it would clearly not be a useful one. Similarly, Yang et al. [2021] gives no theoretical guidance on the limitations of the method. Our assumptions clearly delineate where we know the problem is solvable from settings where we have no theoretical guarantees. \n\nWe also note that our method empirically outperforms all of the published baselines, so we could have chosen to just present it as a state of the art method with no mention of the inherent limitations of matching in general, but we chose to be upfront about the limitations (which likely apply to all of the baselines). NeurIPS reviewer guidelines state, “Reviewers will be specifically instructed to not penalize honesty concerning limitations,” we ask you to reconsider whether the theory that is explicit about assumptions that are sufficient for the method to work is really a weakness (we think it’s a strength!).\n\n**Shared Latent Space**\n\nWe will include a more formal description of Yang et al., [2021]. Roughly, they assume the following data generating process. Each modality, indexed by $i$, is generated according to $X_i = f_i(Z, N_i)$, where $Z$ is a latent variable shared across all modalities. They fit a multi-modal generative model with a VAE assuming $P_Z$ the latent distribution is known, by enforcing that all modalities have the same latent distribution. If label information $t$ is available, they add the classification term requiring $Z$ to classify $t$ (they refer to “prior knowledge” of clusters rather than perturbations, but mathematically $t$ plays the same role).\n\nWe believe our data generating process (Eq. (1)) is significantly more general in describing multimodal settings. Yang et al., [2021] require a known latent distribution, while we do not even assume knowledge of the space of $Z$, nor of the functional form of $f$ (though additional structure can yield a richer theory, such as our Prop 3.2). In our biology experiments, $Z$ is just an abstract random variable representing the cells, and $U$ an abstract notion of measurement noise. For the VAE, any theoretical guarantees would require correctly specified latent spaces and decoder functional classes (both of which are also untestable). In [Yang et al., 2021], they state\n\n> The dimensionality of the latent distribution is a hyperparameter that is tuned to ensure that the autoencoders are able to reconstruct the respective data modalities well.\n\nBy contrast, our theory about propensity scores is valid with only (A1) and (A2), both of which, are easy to interpret scientifically for practitioners (e.g., (A1): interventions do not modify the measurement process, (A2): the measurement contains complete information about the biology). \n\n**Scalability**\n\nEntropy regularized OT is $\\mathcal(O)(n^2 / \\epsilon)$ (ignoring log factors, see [Pham 2020] for details), where in our setting, n corresponds to the number of samples in each domain (bounded by the max number of sample in the unbalanced case you refer to). Given that we only match within a treatment group, this is not unreasonable in the domains that we consider: for example, a well of cells in a cell-paint assay will typically have approximately 1000 cells. Constructing a 1000 x 1000 distance matrix and solving the resulting OT problem is very doable on current hardware. Of course, there may exist settings with even larger numbers of samples which would require more scalable unbalanced OT methods, or alternatively switching to shared nearest neighbors (at the cost of some performance; see our experiments). We emphasize that our contribution is in designing the distance metric that leverages propensity scores (and explaining theoretically when and why it works), rather than any particular matching approach that operates on that distance metric, so our approach can be combined with any scalable method that operates on pairwise distances, and we simply found that entropy regularized OT was most effective empirically.\n\nPham, K., Le, K., Ho, N., Pham, T., & Bui, H. (2020, November). On unbalanced optimal transport: An analysis of sinkhorn algorithm.\n\n**Elaborate on the baselines**\n\nGromov-Wasserstein OT (SCOT) uses the local geometry within each modality, assumed to lie in a metric space, to build a cross-modality distance by comparing the distances between pairs within modalities. We expect this approach to fail when one of the modalities does not have a simple metric structure, e.g., for images where euclidean distance in pixel space usually poorly captures semantic similarity. scGLUE is also a VAE approach that is tailored to biological applications, by enforcing that gene expression and their associated proteins (or, in their case, ATAC-seq locations) to have similar embeddings. As such, scGLUE is not applicable outside of genomics data. Importantly, neither of these methods utilize label information. For this reason, the VAE of [Yang et al., 2021] described above, which can use label information in addition to the VAE objective, is considered our main comparison.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a novel method for aligning unpaired multimodal data using label-based classifiers. The approach involves training classifiers to predict labels for each modality and using the resulting logits to calculate a similarity matrix between modalities. The method leverages a common latent space, influenced by a perturbation signal, to align samples across modalities. This alignment is further refined using optimal transport matching with entropic regularization. Empirical results on multiple datasets, including imaging and single-cell data, demonstrate that this method effectively improves sample alignment compared to existing techniques.

All reviewers agree that this work is interesting and important and the paper is well written and easy to follow. Experiments show that the proposed method outperforms the literature in terms of MSE and other specific metrics. I think this is an interesting paper that should be accepted for the conference.
Decision: Accept (poster)