Title: BaryBind: Binding All Modalities via Multimodal Wasserstein Barycenter Space
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Multimodal representation learning; Inter-modality balance; Wasserstein barycenter; Video understanding.

Abstract: Multimodal joint representation, which aligns multiple modalities in a shared latent space, has emerged as the foundation of recent multimodal understanding models. To scale beyond two modalities, existing models typically treat a specific modality (e.g., text) as the alignment center, or anchor, to bind other modalities via pairwise contrastive losses. However, the learned joint representation space tends to be sub-optimal and imbalanced, as it may insufficiently exploit the modality-agnostic semantics and holistic geometric structures within multimodal data. In this work, we are motivated by the intuition that multimodal representations arise from different shifts from an underlying modality-agnostic representation space. Based on this, we present **BaryBind**, a multimodal framework that aligns modalities in the multimodal Wasserstein barycenter (WB) space, which inherently models a modality-agnostic distribution by minimizing the average of Wasserstein distances to all modalities. We further construct a barycenter polytope, whose volume serves as a geometric metric for quantifying $n$-modality alignment.  This metric is integrated as a barycenter-anchored volumetric contrastive loss that contrasts the volumes of the $n$-dimensional polytopes, encouraging global alignment of non-anchor modalities to the barycenter while reducing inter-modality gaps. Extensive experiments show that BaryBind delivers more balanced zero-shot generalization performance in downstream tasks, e.g., cross-modal text/video retrieval and classification, highlighting its potential for general multimodal understanding.

TL;DR: We present BaryBind, which aligns all modalities to a Wasserstein barycenter with a volumetric loss for scalable and balanced multimodal learning.