Cross-Linked Unified Embedding for cross-modality representation learning
Authors: Xinming Tu
Zhi-Jie Cao
Chen-Rui Xia
Sara Mostafavi
Ge Gao
Venue: NeurIPS 2022 Accept (2022)
Keywords: Multimodal Learning
Representation Learning
Semi-supervised Learning
Deep Autoencoders
Computational Biology and Bioinformatics
Single-cell Genomics

Abstract: Multi-modal learning is essential for understanding information in the real world. Jointly learning from multi-modal data enables global integration of both shared and modality-specific information, but current strategies often fail when observa- tions from certain modalities are incomplete or missing for part of the subjects. To learn comprehensive representations based on such modality-incomplete data, we present a semi-supervised neural network model called CLUE (Cross-Linked Unified Embedding). Extending from multi-modal VAEs, CLUE introduces the use of cross-encoders to construct latent representations from modality-incomplete observations. Representation learning for modality-incomplete observations is common in genomics. For example, human cells are tightly regulated across multi- ple related but distinct modalities such as DNA, RNA, and protein, jointly defining a cell’s function. We benchmark CLUE on multi-modal data from single cell measurements, illustrating CLUE’s superior performance in all assessed categories of the NeurIPS 2021 Multimodal Single-cell Data Integration Competition. While we focus on analysis of single cell genomic datasets, we note that the proposed cross-linked embedding strategy could be readily applied to other cross-modality representation learning problems.

Cross-Linked Unified Embedding for cross-modality representation learning
Xinming Tu,.3.;, Zhi-Jie Cao 1:2 Chen-Rui Xial.2, Sara Mostafavi Ge Gao 1.21
PPekigg University, *Changping Laboratory, University of Washington
Abstract
Multi-modal learning is essential for understanding information the real world. Jointly learning from multi-modal data enables global integration of both shared and modality-specific information, but current strategies often fail when observa- tions from certain modalities are incomplete or missing for part of the subjects. To learn comprchensive representations based on such modality coodalty-incomplett data, we present supervisdd neural network model called CLUE (Cross-Linked Unified Embedding). Extending from multi-modal VAEs, CLUE introduces the use of cross-encoders construct latent representations from modality-incomplete observations. Representation learning for common in genomics. For example, human cells are tightly regulated across multi- ple related but distinct modalities such as DNA, RNA, and protein, jointly defining a cell's function. We benchmark CLUE on multi-modal data from single cell measurements, illustrating CLUE's superior performance in all assessed categories of the NeurIPS 2021 Multimodal Single-cell Data Integration Competition While we focus analysis of single cell genomic datasets, note that the proposed cross-linked embedding strategy could be readily applied to other cross- modality representation learning problems.
observations is
cross-modality
Introduction
Data from complex systems span multiple modalities. For example, to perceive the world we need see images, hear sounds, and read text. Learning how to represent multi-modal data in way that captures both shared and complementary information across modalities is a fundamental ML
task. [ 2 Previous works have demonstrated that multi-modal learning improves video and image classification 4), and cadality-translatiion [5
Multi-modal learning is of critical importance in biology. Biological processes in cells involve multiple regulatory layers that are composed different types of parts, such as DNA 6 7
g RNA(9],
and protein 10]. but these layers directly influence each other, resulting in intrinsic need for multi- modal understanding. In recent years, single-cell technologies have enabled measuring variety of data modalities at single-cell resolution, including specialized techniques capable of simultaneously measuring two or more modalities in the same cell (SHARE-seq Sci-CAR[12) (2), SNARE-seq [13] CITE-seq [4].. However, quality of multi-modal experiments is typically lower than unimodal ones. Further, unimodal datasets can be generated on a much larger scale (tens of millions of cells, as opposed thousands of cells). Therefore, in-silico integration of these unimodal data may provide great value for understanding inner workings of complex cell systems [5 [6].
"Contribute equally.
|Correspondence should be addressed to saramos @ es. washington.edu (for Dr. Sara Mostafavi).
(for Dr.
Ge Gao)
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Single-odailtt
Multi-modailit
Single-mddality Dataset 1,2,3
Multi-madalitt Daraser 4,5
Modality
Medally Representation
Modality
Embedding
Modaliyy
Figure I: Cross-Modality Representation Learning: Figure represents schematic example with three data modalities, where each study (dataset) has measured subset of the modalities. Our cross modality representation approach learns the complete set of modalities for each dataset.
particular, pressing computational question is how to utilize the limited and noisy datasets from multi-modal experiments as reference to pair the larger set of high quality unimodal data, in order to generate pseudo-muls mooo-immodal datasets that enable high-quality downstream analyses. To address this, propose to coss-modality representation learning (Figure [ to learn comprehensive representations from modality-incmmplet incomplete (unimodal) data. show that our approach results in a significant improvement compared competing approaches.
In this paper, we propose semi-supervised neural network model called CLUE (Cross-Linked Unified Embedding). The model uses paired multi-modal data as supervision to learn a combined representation by jointly training cross-encoders across data modalities. Based on this framework. representations, which are then combined to build effective
CLUE learns modality-specii embedding in the multi-modal space for each cell. As a result, CLUE can automatically pair single- cell data from different modalities. Further, while other methods in this domain seek to find common anformattion/repreentttioo across the modalities, CLUE preserves both shared and modality- specific information. CLUE achieved state-of ooffthe-art art performance in the modal single-cell data integration challenge as demonstrated by its top performance on all of the assessment categories in the NeurlPS 2021 Multimodal Single-cell Data Integration Competition Also, provide additional experiments illustrating CLUE's state-of-the-arr performance.
2
Related work
In this section, we will first introduce general, existing machine learning approaches and then focus on tailored approaches in single cell genomics.
2.1 The general multi-modality learning problem
CLUE aims to learn a coordinated representation 1 each modality has corresponding projection function, and constrains across the projection functions ensures some level of coordinated repre- sentation learning, Multi-modal machine learning has shown impressive performance in computer
vision and NLP 4
17 [8]. For example, aligned VAE [19] aligns modality-specific embedding space by minimizing the MSE of embeddings from paired data. Coupled VAE [20] uses "translator" decoder to reconstruct given modality from the embedding. Others methods, including CLIP and DALL-2(5) apply contrastive learning to form a coordinated space. Furthermore, attention- based methods focus on the learned interactions between modalities to improve the quality of the coordinated representation 21
5
2.2 Related work in single-cell genomics
common approach for integrating data in single cell genomics to represent multi-modal data in shared representation space. Various downstream analyses such as clustering, cell type annotation, and cell-fate trajectory reconstruction are then performed using the embedded representations. Existing methods can be categorized to unsupervised and semi-supervised approaches.

---

Unsupervised methods Unsupervised methods integrate cells from different modalities without paired multi-modal measurements. The key here is to build a feature transformation between different modalities so that all modalities are represented in the same feature space. For example, Seurat v3 23 transforms ATAC-seq data to RNA-seq space via the gene activity scores, computed as the normalized sum of cis open chromatin read counts within a certain distance cutoff from each gene. Standard approaches such as Canonical Correlation Analysis (CCA) can then be applied because of the computationally paired feature spaces. Online iNMF (24), LIGER (25) and MOFA+ (26) follows similar feature transformations, but apply Negative Matrix factorization (NMF) to separately model shared and modality -specffcc information. GLUE 27 adopts a graph variational autoencoder (VAE) to model the prior regulatory relationship between open chromatin regions and genes, which enables an improved feature transformation, yicldiin improved performance.
Semi-supervised methods Semi-supervised methods make use of paired multi-modal measure- ments. Dictionary learning in Seurat v4 (8 uses paired multi-modal data as a bridge to interconnect other unimodal data by representing each unimodal cell as the linear combination a common set of atoms (i.c.. paired multi-modal cells). Extending from the deep generative VAE model, Babel uses moodality-ppccifii encoders and decoders to learn the translation between different modalities. Similar to Babel, Polarbear (30] trains separate translator to model the translation between different modalities. Cobolt BI. MultiVI [3] and scMVP 33 also apply multiple VAEs and align the embedding space using joint embedding.
Common limitation of previous methods The common VAE-based approaches assume there is a "Reference" embedding space, where data from all modalities are projected into. As a byprod- uct of this assumption, existing methods mainly learn modality-shared information and discard modality-seecific information. As we show, CLUE can learn from both modality-shared and specific information, and this capability is important for its top performance on single cell genomic tasks.
3 Notations and Formulation
Let x E Im= represent sample e from modality & (k E KC ( KJ), where ma is the number of features. We denote the set of modalities observed for sample as K... For convenience, we also define 2 as a shorthand for {x le E Kee).
Data from each modality is generated from a latent variable 2y E RRd via generative distribution p(x= |zz; 0g). Here, Zμ represents section of the underlying global representation relevant to the corresponding modality, and 0₂ is the set of learnable parameters in the generative distribution. Note that the modality-specific representation Z 's are correlated, as they are jointly determined by common global representation. We represent the global sample representation with the concatenation of all sondallty-specifie latent variables (zz, Z2, ..., Zx) ), and denote the joint prior distribution as p(z) = p(z1, Z2, Zk).
The multi-modality integration task Given set of samples for which an arbitrary combination of modalities is observed (K,), the objective is to learn the most comprehensive representation z for all samples even when subset of modalities are not observed for all samples. In other words, we are interested in inferring the posterior distribution p(z (x) which is dynamically defined based on the set of modalities profiled in each sample.
4 CLUE
CLUE is a deep generative model that can align data from different modalities after training on partially paired multi--odal data. builds upon the multi VAE architecture where data from different modalities are projected into shared latent space using coddality-speciff VAEs. The key difference that CLUE models the shared latent space as combination of modality-specific sub-spaces, and extends the basic multi-VAB architecture by including cross-encoders that project data from each modality into the sub-spaces of all modalities. This feature facilitates learning from both modality-shatl shared and specific information. Below, we start by introducing the model architecture.
3

---

4.1 Multi-modal Variational Inference
For generative model, the marginal likelihood of sample e can be written as:
p(x
0)
II kE
p(x} (0) Ze; (z)
We can derive the following evidence lower bound (ELBO) by introducing variational posterior ((2) (22 x2)
Inp(x 0) = In f p(xxx ()p(z)dz
-In / p(x 2: 0)p(2) ((2) dz
(2)x
((2) dz
2 ((2)x
( Ro Φ)
の) In p(z)
a(z|x
Imp( 0) Oz +
g(z|x( (0)
(c) dz
9(z x
Zki 10dz0dzzzz KL (g(z)xE)) の) l)((())) - al21xE In p
(c)
kEKo
Regularization term
Reconstruction lernn
Maximizing the ELBO effectively maximizes the marginal data likelihood and minimizes the error of the variational posterior at the same time.
When a factorized variational posterior q(z x (c) term becomes:
xk= Φ) = IIkEx a(z)) xx: Φ) is used, the reconstruction
/II
II REAC
a( x ; ②)) Σ
Σ kEK
E
In Inp((( (zx;; 0)
(2)
kE
However, for the regularization term, the joint prior p(z) = p(z1 Z2, ... 3k) difficult learn due to the correlation between modality-speciff latent variables. Here we approximate it with factorized prior pzz) I[tex P(Zz), which simplifies the regularization term into:
Σ KL (1(2x|x() REK
(0) ())p(()))))))
The overall ELBO is thus:
(0,
I o(as (xl
In Inp(x (c) (ppx(())22,00) (xx;0) -KL (1(2)xx(;)) (01l((2)))))))
(4)
KEA
4.2 Self-Linked Variational Autoencoders
Intuitively, data from the same modality should be the most informative for inferring pondality-speciine (c), latent states, is reasonable to apply the approximation q [()) Φ) q(Zz, [x{ (x)(()) (). In that case, the ELBO becomes:
ELBO (x (,000) x,
Σ
E
q(ze g/ ((())))))))))) Inp(x (c) 1z:0) - KL (g(zz X (c)
; ())l((z)))] (5)
kEK
Following the approach of VAEs. we can implement the generative distributions p Iz 0) by decoder neural networks, and the variational posteriors g(za |x (c) Φ)) by encoder neural networks. However, the above approximation leads K independent VAEs. The full latent z (21, Z2, ...: kx) only be inferred when all modalities are observed.

---

4.3 Cross-Linked Variational Autoencoders
To allow full inference of latent z with modality-incomplet data, some the latent variables corresponding to unobserved modalities must be inferred from other observed ones, necessitating a cross-modality encoding scheme. To achieve this, avoid the previous approximation and model
the variational posteriors (zz, (e), Φ) directly. We use the following data-adaptive encoders to implement the dynamically-defined variational posterior:
g xx = II
(6)
E'E
The ELBO term is then:
ELBO (x 0x,
(7)
BeeEe ((ao)s)))))) in ( (xxx0 - KLL
II K'EK
q(zx| 6) p))lp(z)))
(8)
kEA
The model essentially introduces a matrix encoders g(Zx X Φ) that map between all modality pairs. The "self-encoders" (k k') map data to the latent variable of the corresponding modality while the "cross-encoders" (k: + k') map data to latent variables of other modalities. For given sample, the inferred latent of modality & combines self- and cross- inferences from all its observed modalities. For example, let modality 1 be RNA and modality 2 be protein. The cross-encoder q (22 æ1) encodes RNA data to the protein latent space, and the cross-encoder (21 x2)) encodes protein data to the RNA latent space. These cross-encoders will learn to translate information from one modality to another after being trained with multi- modal cells where both RNA and protein are observed. Thus, for unimodal cell where only the protein modality is observed, latent representation of the missing RNA modality can be imputed with cross-encoder q (#1 x2), and vice versa.
Self/Cross-Enodeer
Decoders
q(z, (x,)
p(x
Modality (RNA)
((22(x,))
1x211x))
p(x2(2g)
Modality 2
(ATAC)
((22(x2)
22
Figure 2: CLUE model architecture
The cross-linking architecture allows the model to learn useful information from multi-modal observa- tions, generalizing to cases where certain modalities are missing. For example, multi-modal samples where pair of modalities i.j are simultancously observed can be utilized to train the cross-encoders q(z, (x,) and (x,), enabling us to infer the latent z,, z, fully even when one x, and x, is missing.
4.4 Additional alignment losses
Beyond the ELBOs, we add the following two objectives to further align the learned representations:
5

---

Mean squared error loss The intuition behind the mean squared error loss is simple. Representa- tions inferred from different modalities of the same multi-modal sample should be "close" each other:
(o) E.
2 19)
[KCel
kEKo
2/
q(z) x ;6)
(e);
(10)
2
2 (c)
(11)
kEK.
We train modality discriminator p[k: 1b) tell which data modality latent representation z is inferred from. The encoders are then trained adversarially to fool the modality discriminator, so that the representations from different modalities are well-aligned:
Adversarial loss
(D( = Eg
;gt In (k:l2z;
(12)
Kr
KEN.
4.5 Overall objectives
Lec:
Loeet (0, = Ex
- -ELBO welf (x 0,
(13)
Ceross (0, 6) = Ec
(c) LELBO Kc ; 0,
(14)
The overall training objectives can be written as:
max ( 2/)
W
(15)
min Awett se () AeroesCernse () + (MSE + AnC 1b)
In summary, the mean squared error and the cross prediction use the pairing information to learn the local alignment, while the adversarial learning aligns these embedding in a global manner.
5 Result
We applied the CLUE model to the problem of multi-modal single-cell data integration. We first introduce the dataset and metric used to evaluate the performance of the model, followed by the results. Next, we additionally compare our model with MultiVI (f]22, Cobolt[31] and the "bridge integration" in Seurat v4. We show that CLUE outperforms previous methods.
5.1 Datasets
The NeurIPS multi-modal competition used data from two types of recent technologies for measuring single-cell multi-modal data: 10X genomics Multiome and CITE-seq (5) Multiome measures DNA accessibility and gene expression simultaneously. By dividing the genome into bins of fixed length, DNA accessibility information was converted into 100,000 0000-dmmeeelia matrix. After filer gene expression, the gene expression matrix includes measurements for 20,000 genes per cell. Due to technical limitations. these two matrices are highly sparse. CITE-seq simultancously captures proteins (n=134) surface and gene expression information inside cell. The Multiome training dataset includes 42,492 cells, and the test dattaset includes 20,009 cells,. The CITE-seq training dataset includes 66, cells, and the test dataset includes 15,066 cells. The data splitting scheme is identical to that used in the competition.
NeurIPS 2021 Multi-modality Competition Datasets 34
Share-seq dataset In addition to the competition data, we also use a dataset from mouse skin gen- erated with SHARE-seq [1]. SHARE-seq simultaneously measures gene expression and chromatin accessibility. In this dataset, there a total of 34,774 cells from four different batches (53, 54, 55, 56). We leave one batch (53) out for testing, and use the other three batches (54, 55, 56) for training data.

---

If true pairing information is 5.2 Metric
Matching Score Average confidence placed on correct matching. Specifically, moos--todality matching matrix M is first constructed by computing a Jaccard index of cross-modality nearest neighbors in the integrated embedding space:
(NN,, n NNN (1) U JN O (NN)
M
(16)
(NN U 1 U (NN3+ U NNN.)
where NN, is the nearest neighbor of profile i in the modality of profile j. The matching score is computed as follows:
Matching Score
ΣΣ M. * dG, N
1
(17)
where M is obtained by row-normalizing the matching matrix M. óz,5 is 1 if profile and j were measured in the same cell and 0 otherwise. N the number of observations.
FOSCTTM Fraction of Samples Closer than True Match 36 available for N cells then the FOSCTTM calculated as:
N
(6)
N
FOSCTTM
1
Σ
2
2N
N
(18)
11 = Hi d (x), < d (.) m2
(
Ki d (x,,y))
< d vxH
(6) represent the number of cells that closer to the ith cell than their true matches in the ni and n2
(1)
opposite dataset. d is the Euclidean distance. Lower FOSCTTM values indicate higher accuracy.
5.3 Competition results
Tyyee
#11
Lymph pros
<D44 T Tactivotes
HIGE prog
<D44 T Tiaaro
<D0+
C000-
UIKAP2
CDD Tislree
MMama
Proeyttrolloz
UMAP
URAP1
(a) Multiome: chromatin accessibility and gene expression
cel ape
CMC-
0016+ Mons
Nommsbaed
OKC-
Erytnrch
Flems ICCC
0244 Tachratte
G/9
Taame CCC-
0341 actirated intogin8)
HSC
MasmatMast LACC
C04+ ravve
Hasmatiist CMC
208+ T Co
&CC
Froerythenctee
AST
2084 7 COST+ CDSSRA
lampph prog
Ret ceste
GEX
COS 7 CD5T4 cDesRO-
MAIT
rag
CO01 C2651 CDASRA-
MICE prg
Tarsitioaa
C04+ C266. (D4iRo-
cDC2
203+ TITT CDI5RA.
co:5ax1+
paf CD11tb+
5084 1D/T+ CDASRO-
Natva co22-
gat TCRVD24
<Dd
Natve
URAP]
UMAP1
(b) CITE-seq: surface protein abundance and gene expression
Figure Integration result of both multi-modality task never seen test data
There were 280 teams participated in the competition. Our method CLUE won the competition the test dataset with decisive lead(Table T). The competition used matching score evaluate model
7

---

performance four sub-tasks: pairing ATAC(Chromatin accessibility) to GEX(gene expression), ADT(Protein abundance) GEX, and reversely, GEX ATAC and GEX ADT: then an overall score was calculated of these four sub-tasks. CLUE scored between 0.049 and 0.058, which was almost x000x higher than the random matching baseline model. Other top teams used several technologies including VAE, NMF and contrastive learning. Our score was higher than the Top2 method by 26% and the Top3 method by 158%.
We use non-linear dimension reduction method, UMAP[37], to visualize the embedding for test data. On the 2D scatter plot of UMAP visualization, each point a single modality profile. We first color the points by modality, which shows that CLUE integrates the two different modalities very well. Coloring by cell type indicates that the embedding captures the biological variation well(Figure 3
Table 1: The matching score of the Top 5 team in the competition.
Team
GEX2ATAC
ATAC2GEX
GEX2ADT
ADT2GEX
Overall
CLUE
0.0560
0.0583
0.0495
0.0516
0.0539
Novel
0.0482
0.0482
0.0373
0.0373
0.0427
SCEMMAPDTHCOFF"
0.0071
0.0099
0.0322
0.0344
0.0209
Liuz Lab BCM2
0.0120
0.0120
0.0250
0.0252
0.0185
SysMo
0.0124
0.0127
0.0153
0.0159
0.0141
VAE Contrastive learning
VAF-based methods
NMF Contrastive Learning
5.4 Integration Benchmark
In addition to the comparing CLUE to existing methods on the challenge dataset, we also compared CLUE's performance with three other methods (Bridge integration Seurat V4, MultiVI, and Cobolt). We performed hyperparameter search for all deep-learning- based methods. Bridge integration method able handle such large paired datasets due the memory limitation of So
in Seurat V4 we downscale the paired data to benchmark the performance of all these methods with CLUE. We also note that the Cobolt, MultiVI are not designed for the CITE-seq data so the result is significant worse than on the Multiome data.
Table 2: The matching score of benchmark methods in the competition data
Methods
GEX2ATAC ATAC2GEX
GEX2ADT
ADT2GEX
Overall
CLUE
0.0560
0.0583
0.0495
0.0516
0.0539
Cobolt
0.0349
0.0356
0.0147
0.0123
0.0244
MultiVI
0.0261
0.0296
0.0063
0.0056
0.0169
Bridge-integration
0.0158
0.0117
0.0126
0.0102
0.0116
Table 3: The matching score of the benchmark methods in the SHARE-seq data
Methods
GEX2ATAC
ATAC2GEX
Overall
CLUE
0.136
0.134
0.135
MultiVI
0,0579
0,0623
0.0601
Cobolt
0.0431
0.0486
0.0458
cedggeinntegration
0.0375
0.0296
0.0335
We first evaluated the performance using the matching score on competition dataset and the SHARE- seq Aataset.As shown in Table CL.EE is the best methods compared to other four methods. The Overall score is 0.0539 which higher than other methods from 121% 2 to 3644e 2 Table We also calculate the FOSCTTM score for all methods and compare the Performance(Table

---

Table 4: The FOSCTTM score of the benchmark methods in the competition data
Methods
Multiome
CITE
Overall
CLUE
0.0183
0.0140
0.0162
Bridge-integration
0.0377
0.0311
0.0344
Cobolt MultiVI
0.0296
0.0457
0.0377
0.0398
0.0872
0.0635
5.5 Ablation study
In this section, explore the contribution of various modeling choices using ablation. CLUE has three important components: the cross-loss, the mean squared error loss and the adversarial loss.
Table 5: The matching score with different combinations loss components
Ceross
OMSE
LD
GEX2ATAC
ATAC2GEX
GEX2ADT
ADT2GEX
Overall
0.0560
0.0583
0.0495
0.0516
0.0539
0.0499
0.0483
0,0400
0.0423
0.0451
0.0460
0.0457
0.0412
0.0448
0.0444
0.0224
0.0212
0,0286
0.0413
0.0284
0.0127
0.0135
0.0320
0.0429
0.0253
0.0292
0.0264
0.0210
0.0233
0.0250
0.0130
0.0117
0.0098
0.0114
0.0115
0.0001
0.0001
0.0002
0.0002
0.0001
The ablation experiments shows that all components contribute the final performance while the adversarial learning loss is the most important, without which the matching score will drop 47.3%. The performance will drop 16.5% without the square error loss and 17.6% without the cross loss.
Conclusion
We present a superiuper neural network called CLUE learn latent representations based on multi-modality datasets. We applied CLUE to integrate cellular information including gene expression, protein abundance, and DNA accessibility at single-cell resolution. Extensive assessment on benchmark datasets illustrate that CLUE achieves the state-of-the-ar performance on two standard evaluation metrics.. Our model provides principled approach to combining paired and unpaired measurements in single cell genomics applications. As future work, an online platform can be developed to enable users to query for 'cells' that are similar set pre-specified cells.
The proposed cross-linked embedding strategy described here can also have broader application other domains. We note that our model contains K2 cross-encoders for modalities, so for applications with very large K, scalability can pose a challenge. However, with shared preprocessing layer for each of the modalities, model size can be effectively reduced, allowing efficient training in most real world applications where K < 5.
7 Code Availability
CLUE for the single-cell multi-modality integration incorporated in the seglue package All source code is available in https:///itthin com/ gao-lab 1ab/ /GUUE
8
Acknowledgment
This work was supported by funds from the National Key Research and Development Program of China (2021 YFC2502000, 2016 2016YFC0901603), well as the State Key Laboratory of Protein

---

and Plant Gene Research, the Beijing Advanced Innovation Center for Genomics (ICG) at Peking University, the Changping Laboratory and the Shaw Foundation Hong Kong Limited. The research of G.G. was supported in part by the National Program for Support of Top-notch Young Professionals.
10

---

References
[1] Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learn- ing: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2):423 44.42233433,422 2018.
[2] Wenzhong Guo, Jianwen Wang, and Shiping Wang. Deep multimodal representation learning: survey. IEEE Access, 7:63373-63394 2019.
[3] Yanan Liu, Xiaoqing Feng, and Zhiguang Zhou. Multimodal video classification with stacked contractive autoencoders. Signal Processing, 1 120:761-766, 2016.
[4] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 7774-88763 PMLR, 2021.
[5] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[6] Darren A Cusanovich, Riza Daza, Andrew Adey, Hannah A Pliner, Lena Christiansen, Kevin L Gunderson, Frank Steemers, Cole Trapnell, and Jay Shendure. Multiplex single-cell profiling of chromatin accessibility by combinatorial cellular indexing. Science, 348(6237):910-914 2015.
[7] Evan Z Macosko, Anindita Basu, Rahul Satija, James Nemesh, Karthik Shekhar, Melissa Goldman, Itay Tirosh, Allison Bialas, Nolan Kamitaki, Emily M Martersteck, et al. Highly parallel genome-wide wide expression profiling of individual cells using nanoliter droplets. Cell, 161(5):1202-1144, 2015.
[8] Hongshan Guo, Ping Zhu, Liying Yan, Rong Li, Boqiang Hu, Ying Lian, Jie Yan, Xiulian Ren, Shengli Lin, Junsheng Li, al. The dna methylation landscape of human early embryos. Nature, 5 511(751 (:6611))60066100 2014.
[9] Fuchou Tang, Catalin Barbacioru, Yangzhou Wang. Ellen Nordman, Clarence Lee, Nanlan Xu, Xiaohui Wang. John Bodcau, Brian Tuch, Asim Siddiqui, et al. mmna-seq whole-transcriptone analysis of single cell. Nature methods, 6(5):377-382, 2009.
[10] Sarah Hainer, Ana Bošković, Kurtis N McCannell, Oliver Rando, and Thomas G Fazzio. Profiling of pluripotency factors in single cells and early embryos. Cell, 17765)1119-1132, 2019.
[11] Sai Ma, Bing Zhang, Lindsay M. LaFave, Andrew S. Earl, Zachary Chiang, Yan Hu, Jiarui Ding, Alison Brack, Vinay K. Kartha, Tristan Tay, Travis Law, Caleb Lareau, Ya-Chieh Hsu, Aviv Regev, and Jason D. Buenrostro. Chromatin potential identified by shared single-cell profiling of ma and chromatin. Cell, 183(4):1033-1116))
2020.
[12] Junyue Cao, Darren Cusanovich, Vijay Ramani, Delasa Aghamirzaie, Hannah A Pliner, Andrew J Hill, Riza M Daza, Jose L McFaline-Figuuee et al. Joint profiling of chromatin accessibility and gene expression in thousands of single cells. Science, 361(6409); 1380 1385, 2018.
Jonathan S Packer, Lena Christiansen,
[13] Song Chen, Blue Lake, and Kun Zhang. High-throughhut sequencing of the transcriptome and chromatin accessibility in the same cell. Nature biotechnology. 37(12):1452-1457 2019.
[14] Marlon Stoeckius, Christoph Hafemeister, William Stephenson, Brian Houck-Loomis, Pratip K Chattopadhyay, Harold Swerdlow, Rahul Satija, and Peter Smibert. Simultancous epitope and transcriptome measurement in single cells. Nature methods, 14(9):865 2017.
[15] Zizhen Yao, Hanqing Liu, Fangming Xie, Stephan Fischer, Ricky Adkins, Andrew Aldridge, Seth A Ament, Anna Bartlett, M Margarita Behrens, Koen Van den Berge, al. A transcriptomic and epigenomic cell atlas of the mouse primary motor cortex. Nature, 59877879):033-100, 2021.

---

[16] Peng He, Brian A Williams, Diane Trout, Georgi K Marinov, Henry Amrhein, Libera Berghella, Say-Tar Goh, Ingrid Plajzer-Frick. Veena Afzal, Len A Pennacchio, al. The changing mouse embryo transcriptome at whole tissue and single-cell resolution. Nature, 583(78188,760-767 2020.
[17] Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multimodal deep learning. In ICMI., 2011.
[18] Fangxiang Feng, Xiaojie Wang, and Ruifan Li. Cross-modal retrieval with correspondence autoencoder. In Proceedings of the 22nd ACM international conference on Multimedia, pages 7-16, 2014.
[19] Edgar Schonfeld. Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, and Zeynep Akata. General- ized zero- and few-shot learning via aligned variational autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[20] Rohan Gala, Nathan Gouwens, Zizhen Yao, Agata Budzillo, Osnat Penn, Bosiljka Tasic, Gabe Murphy, Hongkui Zeng, and Uygar Sümbül. A coupled autoencoder approach for multi-modal analysis of cell types. Advances in Neural Information Processing Systems, 32, 2019.
[21] Ying Cheng, Ruize Wang, Zhihao Pan, Rui Feng, and Yuejie Zhang. Look, listen, and attend: Co-attention network for self-supervised audio-visual representation learning. In Proceedings of the 28th ACM International Conference on Multimedia, pages 3884-3892, 2020.
[22] Xi Wei, Tianzhu Zhang, Yan Li, Yongdong Zhang, and Feng Wu. Multi-modality cross attention network for image and sentence matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10941-109500 2020.
[23] Tim Stuart, Andrew Butler, Paul Hoffman, Christoph Hafemeister, Efthymia Papalexi, William M Mauck III, Yuhan Hao, Marlon Stoeckius, Peter Smibert, and Rahul Satija. Compre- hensive integration of single-cell data. Cell, 177(7):1888-11))
2019.
[24] Chao Gao, Jialin Liu, April Kriebel, Sebastian Preissl, Chongyuan Luo, Rosa Castanon, Justin Sandoval, Angeline Rivkin, Joseph R Nery, Margarita M Behrens, et al. Iterative single-cell multi-omic integration using online learning. Nature biotechnology. 1000 2021.
[25] Jialin Liu, Chao Gao, Joshua Sodicoff, Velina Kozareva, Evan Z Macosko, and Joshua D Welch. Jointly defining cell types from multiple single-cell datasets using liger. Nature protocols, 3662, 2020.
[26] Ricard Argelaguet, Damien Amol, Danila Bredikhin, Yonatan Deloro, Britta Velten, John C Marioni, and Oliver Stegle. Mofat: a statistical framework for comprehensive integration of multi-modal single-cell data. Genome biology, 21(1):1-17, 2020.
[27] Zhi-Jie Cao and Ge Gao. Multi-omics single-cell data integration and regulatory inference with graph-linked embedding. Nature Biotechnology, pages 1-9, 2022.
[28] Yuhan Hao, Tim Stuart, Madeline Kowalski, Saket Choudhary, Paul Hoffman, Austin Hartman, Avi Srivastava, Gesmira Molla, Shaista Madad, Carlos Fernandez-Granda, et al. Dictionary learning for integrative, multimodal, and scalable single-cell analysis. bioRxiv, 2022.
[29] Kevin E Wu, Kathryn E Yost, Howard Y Chang, and James Zou. Babel enables cross-mood moss-modalitty translation between multiomic profiles at single-cell resolution. Proceedings of the National Academy of Sciences, 1 118(15) 2021.
[30] Ran Zhang, Laetitia Meng-Papaxanne Papaxanthos, Jhin-Phil Vert, and William Stafford Noble. Semi- supervised single-cell cross- moos-ttyy translation using polarbear. bioRxiv, 2021.
[31] Boying Gong, Yun Zhou, and Elizabeth Purdom. Cobolt: integrative analysis of multimodal single-cell sequencing data. Genome biology, 22(1):1-21, 2021.
[32] Tal Ashuach, Mariano Gabitto, Michael Jordan, and Nir Yosef. Multivi: deep generative model for the integration of multi-modal data. bioRxiv, 2021.
12

---

[33] Gaoyang Li, Shaliu Fu, Shuguang Wang, Chenyu Zhu, Bin Duan, Chen Tang, Xiaohan Chen, Guohui Chuai, Ping Wang, and Qi Liu. A deep generative model for multi-view profiling of single-cell ma-seq and atac-seq data. Genome biology, 23(1):1-23, 2022.
[34] Malte D Luecken, Daniel Bernard Burkhardt, Robrecht Cannoodt, Christopher Lance, Aditi Agrawal, Hananeh Aliee, Ann Chen, Louise Deconinck, Angela M Detweiler, Alejandro A Granados, et al. A sandbox for prediction and integration of dna, ma, and proteins in single cells. In Thirty-fifthh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[35] Marlon Stoeckius, Christoph Hafemeister, William Stephenson, Brian Houck-Loomis Pratip Chattopadhyay, Harold Swerdlow, Rahul Satija, and Peter Smibert. Simultaneous epitope and transcriptome measurement in single cells. Nature methods, 14(9):865-865) 2017.
[36] Ritambhara Singh, Pinar Demetci, Giancarlo Bonora, Vijay Ramani, Choli Lee, He Fang. Zhijun Duan, Xinxian Deng, Jay Shendure, Christine Disteche, et al. Unsupervised manifold alignment for single-cell multi-omics data. In Proceedings of the 1ith ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, pages 1-10, 2020.
[37] Etienne Becht, Leland McInnes, John Healy, Charles-Antoine Dutertre, Immanuel WH Kwok, Lai Guan Ng. Florent Ginhoux, and Evan W Newell. Dimensionality reduction for visualizing single-cell data using umap. Nature biotechnology. 37(1):38 44.33884442222 2019.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'le roe're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "x4syFDgduIQ", "reviewer": "Reviewer_Z9V2", "summary": "This paper proposes a semi-supervised neural network named CLUE (Cross-Linked Unified Embedding) for cross-modality representation learning. The core part is the proposed cross-encoders, which can learn modality-specific representations and combine them to build an effective feature embedding.", "questions": "1. Some works also try to explore the intersection between different modalities for cross-modal representation learning. What is the main difference between your method and other methods (e.g., “Look, Listen, and Attend: Co-Attention Network for Self-Supervised Audio-Visual Representation Learning” ) ?.\n2. In Eq.14, the weights of different losses are different. Please explain the exact reasons. How do you decide the weight values of different losses?\n3. Please describe the detailed experimental settings, such as learning rate, etc., so the reader can reproduce.\n4. IS the same experiments for all compared methods in Table1 and Table 2? \n5.What is the contribution of $q({{z}_{2}}|{{x}_{1}})$or $q({{z}_{1}}|{{x}_{2}})$,respectively.\n", "limitations": "We recommend enriching the experimental section, such as experimental setup and parameter analysis of the method.", "rating": 7, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper uses cross-encoders to learn comprehensive representation. However, the experiments are insufficient.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "bNjSvWFo_p6", "reviewer": "Reviewer_tEWF", "summary": "The paper proposes CLUE (Cross-Linked Unified Embedding) to construct multimodal representations from modality-incomplete datasets and demonstrates its application on multi-modal single-cell data integration.", "questions": "+ Can the model be generalized to other applications other than single-cell data integration?\n\n+ For the evaluation, is there other available dataset for the task? Consistent performance improvement over multiple datasets will help demonstrate the generalization ability of CLUE.  ", "limitations": "See questions above", "rating": 5, "confidence": 1, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\\+The paper is well written and clearly presented.\n\n\\+The paper extends the multi-VAE architecture following a good intuition (preserving modality-specific features rather than simply aligning multimodal data).\n\n\\+CLUE yields large improvement over baseline approaches.\n\n\n-The evaluation is conducted on one dataset for one specific application, which seems a bit limited.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JFjjwWG1ZOW", "reviewer": "Reviewer_dvS5", "summary": "This paper presents CLUE for cross-modality representation learning from incomplete observations using the cross-encoders. Specifically the focus of CLUE is to learn comprehensive representations from datasets with each having a subset of modalities possibly unimodal.\n\n", "questions": "Instead of concatenating the latent encoding of different VAEs, what is the effect of learning a shared latent encoding that reconstruct other modalities while using their respective parameters in decoder?", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nThe CLUE make use of the cross-linked VAEs to model the cross-modality encoding.\nIt leans a comprehensive latent encoding with partially available modalities.\nAchieve better performance than the other compared methods.\n\nWeaknesses:\nIt need K^2 cross-encoders for K modalities.\nScalability may be limited as it requires large-number of cross models to store and compute thus poses the space and computational computational challenges.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept