Title: Group Contrastive Learning for Weakly Paired Multimodal Data
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: multimodal learning, weakly paired data, contrastive learning, single-cell genomics

Abstract: We present GROOVE, a semi-supervised multimodal representation learning approach for high-content perturbation data where samples across modalities are weakly paired through shared perturbation labels but lack direct correspondence. Our approach integrates an on-the-fly backtranslating autoencoder with a novel group-level contrastive loss, GroupCLIP, which leverages shared labels to enforce group-level consistency. This approach encourages cross-modally entangled representations while maintaining group-level coherence within the shared latent space. Importantly, GroupCLIP bridges the gap between CLIP for paired cross-modal data and SupCon for uni-modal Supervised contrastive learning. Next, we introduce a comprehensive combinatorial evaluation framework that systematically assesses representation learners across multiple optimal transport aligners, addressing key limitations in existing evaluation strategies. Across simulated datasets with varying degrees of modality coupling and two real single-cell genetic perturbation datasets, GROOVE is the only method to show consistent outperformance in downstream cross-modal matching and imputation tasks. Our ablation studies demonstrate that GroupCLIP is the key component driving performance gains. These results highlight the importance of leveraging group-level constraints for effective multimodal representation learning in scenarios where only weak pairing is available.

TL;DR: A multi-modal semisupervised representation learning framework for weakly paired multi-modal perturbation data