M3-JEPA: Multimodal Alignment via Multi-gate MoE based on the Joint-Embedding Predictive Architecture
Authors: Hongyang Lei
Xiaolong Cheng
Qi Qin
Dan Wang
Huazhen Huang
Qingqing Gu
Yetao Wu
Luo Ji
Venue: ICML 2025 poster (2025)
Keywords: JEPA
MoE
multimodal
alignment
Primary area: deep_learning->selfsupervised_learning
TL;DR: We propose M3-JEPA, which implements a multi-gate MoE predictor to  JEPA, and shows theoretical optimality and reasonable performance on multimodal tasks..

Abstract: Current multimodal learning strategies primarily optimize in the original token space. Such a framework is easy to incorporate with the backbone of pretrained language model, but might result in modality collapse. To alleviate such issues, we leverage the Joint-Embedding Predictive Architecture (JEPA) on the multimodal tasks, which converts the input embedding into the output embedding space by a predictor and then conducts the cross-modal alignment on the latent space. We implement this predictor by a Multi-Gate Mixture of Experts (MMoE) and name the framework as M3-JEPA, accordingly. The gating function disentangles the modality-specific and shared information and derives information-theoretic optimality. The framework is implemented with both contrastive and regularization loss, and solved by alternative gradient descent (AGD) between different multimodal tasks. By thoroughly designed experiments, we show that M3-JEPA can obtain state-of-the-art performance on different modalities and tasks, generalize to unseen datasets and domains, and is computationally efficient in both training and inference. Our observation suggests that M3-JEPA might become a new basis to self-supervised learning in the open world.

M3-JEPA:
Multimodal Alignment via Multi-gate MoE based on the Joint-Embedding Predictive Architecture
Hongyang Lei"
Xiaolong Cheng" 1
Qi Qin * Dan Wang'
Huazhen Huang?
Qingqing Gu
Yetao Wu
Luo Ji
Abstract
information content
energy: F(x, y)
Current multimodal learning strategies primar- ily optimize the original token space. Such framework is easy to incorporate with the back- bone of pretrained language model, but might re- sult modality collapse. To alleviate such issues, leverage the chitecture (JEPA) the multimodal tasks, which converts the input embedding into the output em- bedding space by predictor and then conducts the cross- modal alignment on the latent space. We implement this predictor by a Multi-Gate Mix- ture of Experts (MMoB) and name the framework M3-JEPA, accordingly. The gating function disentangles the modality-speciff and shared in- formation and derives cetrmatiion-theoeee mality. The framework is implemented with both contrastive and regularization loss, and solved by alternative gradient descent (AGD) between dif- ferent multimodal tasks. By thoroughly designed experiments, show that M3-JEPA ffateoff-the-art performance on different modali- ties and tasks, generalize to unseen datasets and domains, and is computationally efficient in both training and inference. Our observation suggests that M3-JEPA might become a new basis to self- supervised learning in the open world.
max I
min H
min Lreg
min Led
L contrastive loss
MoE- predictor
Leg regularization loes
g
mutual information conditiona entropy
Predictive Ar-
x-encoder
y-encoder
X
y
provided
to predicted
any-to-any
opti-
Text
Image
Audio
Figure I. The paradigm of M3-JEPA on any-to- mny-to-any multi codliityd tasks. The self-supervised learning conducted with two encod- ing branches of input and output signals, as well as MoE pre- dictor which projects input embedding into the output latent spacc. M3-JEPA is energy-basee model that minimizes both contrastive and regularization losses. M3-JEPA is also conditioned the inherent information content which maximizes the tual information and minimizes the conditional entropy.
obtain
and integrate information from multiple modalities like text, image, audio and video, becoming important way to solve complex tasks involving heterogeneous data sources (Alayrac et al., 2022; Radford et al., 2021; Wang et al., 2022a; Liu et al.). Modern multi-modal studies are dominated by generative architecture, which can generally be classified into two main categories. The first category trains the model from scratch (Wang et al., 2022a; 2023a; Oquab et al., 2025), which asks for large-scale data therefore faces high training costs. Another category often employs pretrained Large Language Model (LLM) as the back- bone while finetuning lightweight cross- modal connector (Alayrac et al., 2022; et al., 2023; Wu et al., 2024a; Liu et al., 2023; Jun Zhan, 2024). In this manner, the prior
1. Introduction
Human perception is inherently multi-modal, seamlessly integrating diverse sensory inputs from vision, hearing, touch, and other senses to comprehend the world. Inspired by this capability, modern modeling techniques also pro-
'Equal contribution 'Geely AI Lab, Zhejiang, China *Peking University, Beijing, China 'Shenzhen Institutes of Advanced Tech- nology, Chinese Academy Sciences, China. Correspondence to: Luo Ji <il @ @emm
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

M3-JEPA: Multimodal Alignment via MMoE based on JEPA
knowledge of LLM can be well preserved while the compu- tational burden can also be alleviated (Zhang et al., 2024).
of MoE. This MoE predictor is initialized randomly and trained by two terms of losses, the contrastive loss and the regularization loss. These two loss terms together form an energy function, and are adapted with the two-gate output of the MoF. predictor. To avoid the representation collapse, the predictor is also driven by the optimal information content of an implicit latent variable, as discussed in (Dawid & LeCun, 2024). The paradigm of M3-JEPA is visualized by Figure 1, and the project code can be found at https: //gittubbb com/ /oongggggggggg -EPA..
Although these studies achieve remarkable success in mul- timodal learning, however, they may be often subject to the issue of modality collapse during the cross-modal align- ment, resulting from the conflicting gradients (Javaloy et al., 2022), missing modality or labels (Wu et al., 2024b), and mismatched data distribution or fusion (Ma et al., 2022). Es- pecially, LLMs take advantage of self-supervised learning (SSL), which predicts the hidden parts of the data grounded by the visible parts, in the discrete token space. Neverthe- less, this paradigm relatively struggles in continuous do- mains (e.g., image or video), which causes the cross- moosl alignment difficult to converge, and might fail to capture key information (Dawid & LeCun, 2024), especially in the exis- tence of information uncertainty, redundancy, or ambiguity (e.g., one picture can be described by two content different but semantically similar sentence).
To better alleviate the gradient conflict issue across various modalities, M3-JEPA switches multiple directional multi- modal tasks stepwisely, and solves them by Alternating Gradient Descent (AGD) (Akbari et al., 2023). To validate the reasonability of M3-JEPA, we conduct both theoretical study and empirical experiments. We provide an informa- tive theoretic analysis, and discuss the convergence and optimal typer-parametery We also conduct comprehensive experiments across multiple multimodal tasks, including vision-language, audio-language, and vision classification tasks, while keeping the model architecture and training strategy the same. Experimental results demonstrate that our M3-JEPA achieves competitive results across different modalities and tasks, as well as remarkable generalization on unseen domains, compared to current state-of -o-tthe-art (SoTA) multimodal models. Furthermore, our approach is also computationally efficient from both training and infer- ence aspects. We summarize our contributions as follows:
To mitigate these issues, one possible solution is to switch from generative and probabilistic modeling to the energy- based model (EBM) paradigm, which minimizes a non- negative energy function of both the input and output em- beddings. To better handle the uncertainty of information, the Joint-Embeddding Predictive Architecture (JEPA) (Dawid & LeCun, 2024) is proposed, which employs latent predic- tor that projects the necessary information from the input into the output embedding space, while filtering out irrel- evant or misleading input features. JEPA then aligns the input and output signals in the latent level instead of the token or pixel level, with a similar idea proposed by the "platonic representation" (Huh et al., 2024). JEPA has been applied in vision, namely I-JEPA (Assran et al.. 2023), in which an arbitrary part of an image is masked and predicted by the other image blocks; as well as the motion and con- tent features of videos (Bardes et al., 2023). Nevertheless, there has been no generalized framework that applies the idea of JEPA to generalized multi-modalit modeling, with any-to-any modality combinations either for the input or the output.
We propose novel any-to-any multi-modal alignment paradigm based on JEPA, to mitigate the potential modality collapse by aligning on the latent space.
We leverage a computationally efficient multi-gate MoE architecture as the cross-modal predictor of JEPA, while freeze most parameters of modality encoders.
We disentangle the gating function of MoF into the modality- and shared information, and also de- rive an coammation-theorttic. analysis.
We optimize M3-JEPA by alternating the gradient de- scent between different multi-directional multimodal tasks, and discuss its theoretical convergence.
In this work, we propose a novel Multimodal alignment paradigm via Multi-gate MoE based on JEPA, in short, M3-JEPA. It studies the dependency of the unobserved part (y) on the observed part (x) in the embedding space of y. where 2 and y may belong to different modalities or any combination of them. Their embeddings are encoded by pretrained uni-modality encoders, with possibly several layers being finetuned during the multimodal learning. We implement the latent predictor by the Mixture-of- -oxpeerr (MoE) structure, as a lightweight moos connector. To tackle the potential multimodal semantic discrepancy, decouple the cross-modal information into modality- specific and shared components through the gating function
The experimental results demonstrate remarkable mul- timodal alignment accuracy and efficiency, encompass- ing text, image. audio and other modalities.
The rest of the paper is organized as follows. We first intro- duce the preliminary knowledge in Section 2. The method- ology is stated in Section 3. The theoretical derivation is stated in Section 4. Experiment results and subsequent dis- cussions are summarized in Section 5. The connection with previous works is stated in Section 6. Finally Section 7 concludes this paper.
2

---

M3-JEPA: Multimodal Alignment via MMoE based on JEPA
2. Preliminary
3. Methodology
2.1. Loint-Embedding Predictive Architecture
This section illustrate our methodology, and Figure 2 pro- vides a detailed introduction to the framework of M3-JEPA.
Given an input-output pair (z, 3) which are encoded into (ez,Gy), a joint-embedding predictive architecture (JEPA) (Dawid & LeCun, 2024) first converts the input encoding into the output's dimensional space by predictor P(.)
3.1. Problem Formulation
We develop M3-JEPA as an any-to-any modality frame- work. Assume we have total M modalities and T tasks. For each modality m, a uni-modal encoder can be employed to produce its latent embedding 6mn- For the f-th task, denote its input and output with (x', v"),, which contain the set of ] and {mī) modalities, respectively, with S m my I M. Then the embedding of (x", y²) can be the corresponding or the combination of modality latents:
I= + R2, then minimizes its loss with ey:
min (les-ry: By) == y),,y),
- (I)
with - denotes the loss; F is nonnegative and can be viewed as the energy between and Subsequently, the JEPA framework belongs to energy-based model (EBM).
2.2. Mfxturs-of-Experts
e concat( (em));
m {mm)
The Mfxture-of-Experts (MoE) architecture (Garmash & Monz, 2016; Shazeer et al., 2017) uses N feed-forward networks (FFN), namely "experts". The output of expert (BN given by:
cg = concat concat((em)) (em)).
m E {m,}
Similar to Equation which behaves both the training loss and also the alignment score during the inference
formulate energy term F(x,, ( y)
(e) wnt o(w"" ee)
(2)
which B denotes N experts, is the input vector, wh and un woutpit are learnable weights of the n-th expert, and o the activation function (e.g., GeLU). An additional gating network G outputs an N-dimension normalized vector:
F' v) = L' (ei C (P((e)))
(8)
To make the formulation simple and clear, in the follow- ing derivations, we temporarily omit the superscript & until discussing the optimization method.
Gee) = softmax(g
where g is a learnable matrix. The output of G routes each input via few of the experts, and the output of MoE is:
3.2. Losses
We implement the loss of JEPA (C in Equation 8) from two aspects, the regularization and contrastive losses.
MoB(e)= = Σ (e))
(4)
Regularization loss. fined by the L-2 distance:
The regularization loss be de-
where G(e). denotes the probability of selecting expert E,,
Freg = 1x-+y - -
(9)
Top-K MoE. The above MoE can be implemented with a top-K mechanism, which first ranks the gating scores, then keeps and summarizes top K experts instead of N experts
where denotes the L-2 norm.
We conduct the contrastive learning by sampling the in-batch negatives:
Contrastive loss.
MoE-K(e) Σ top- (G))
(5)
B
exp(sim(e_--..)) e e/)/T)
La
Σ
log
(10)
Multi-gate Mixture-of-Experts. The Multi-gate Mixture- of-Expsrts (MMoE) (Ma et al., 2018) expand the MoE ar- chitecture to the multi-task setting. Given L tasks, MMoE generates the outputs simultancously by L parallel gates:
Σn exp(sim(c et)
where B the batch size, the superscripts i and represents the i-th or the j-th sample within the batch. (im( is the cosine similarity, and r a temperature parameter that controls the sharpness of the similarity distribution.
MMoE' (z)
Σ (E)) 1 1=1,-.. L
(6)
The total loss. The total loss is the linear combination between the regularization loss and contrastive loss:
then use them to calculate the loss of each task:
min L Σ c'(MMoE'(a))
(7)
L(ex-uy ey) = aLreg + Ca
(11)
in which a is the loss weight which determines the weight- ing balance between two loss terms.
where L and C'()) are the total loss and task losses.
3

---

M3-JEPA: Multimodal Alignment via MMoE based on JEPA
(C)
Misio-spaeific Information
Output
Output B
(D)
Text Information
Shared Information
ex-
Crea
MMoE
Expert
Expert
Expert
Expert n
F
Eyp
Cy
Ex
Ey
Gate A
Gate B
(B)
task 1:
w (2)
(A)
task 2:
Vision Encoder
Text Encoder
Audio Encoder
One-hot Encoder
miF
m [4]
AGD
task 3:
red and while egainst in dky.
01
[4]
task 4:
(1.9)
Figure 2. The entire framework of M3-JEPA with 4 modalities and 4 tasks indicated.
(A): Information encoded hy modality encoders (m 1, 1), forming the input and output signals.
(B): Training is conducted by the alternative gradient descent (AGD), with different multimodal tasks switched stepwise. Tasks 1, 2, 3,4 the figure represent text-to-image, audio-to-text, image classification, and visual question question-answering. respectively.
(C): Input and output are aligned on the latent space with an energy (F), consisting of both contrastive (cl) and regularization (reg) losses.
(D): A Multi-gat gatti-gatelul MoE predictor projects the input latent vector to output latent space, generates parallel outputs and B) for different loss terms. Gating functions automatically separate modality-speciife (em. and shared information (ạ).
3.3. Implementation of Predictor
Shared representation: the projection matrix g creates common subspace for the cross-modal gating.
In this paper, implement the predictor of JEPA with top-K' MoE module:
Multi-gating on losses. We further expand our formation to the setting of MMoE architecture. In M3-JEPA, the multi- gate functions are employed to represent two loss terms, the regularization loss Equation and the contrastive loss in Equation 10. As result, the number of gates L
Gy-y= = P = :Mo-Ke=M
(12)
For each modality, we implement N experts, which results in totally M * N experts (L Furthermore, we implement the gating function with the following formulation
3.4. Alternating Gradient Descent
G softmax(g [e, em)), (m²}
(13)
different training steps, we switch between tasks and conduct the forward pass and back-propagation of the train- able parameter sequentially:
where f1:M a learnable complementary matrices which convert the input of the gating function into constant latent dimension, As a result, the gate matrix is task-agnostic.
0 + 1) - 0(1) if mo =t t= 1,2,
(14)
Disentangle modality-specifff and shared information. Our MoFE-peedictor not only serves multimodal aligner, but also harmonize shared semantic information across modalities while preserving modality-specife details. This disentanglement is implemented with different terms in Equation 13:
where is the current step. 7) is the learning rate, and mod(-,-) is the reminder after integer division. This op- timization paradigm is called alternating gradient descent (AGD) with similar implementation on previous multimodal studies (Likhosherstov et al., 2022; Akbari et al., 2023).
sodality-speciffe paths: the perts, as well as the gating embedding 6m-
ex-
The conventional joint optimization requires simultancous updates overlapping parameter subsets (e.g., single con-

---

input (or M3-JEPA: Multimodal Alignment via MMoE based on JEPA
nector aligns both image-to-text and text-to-imaae), leading to potential gradient conflict (Javaloy et al., 2022). In con- trast, AGD decouples these updates, mirroring the success of alternating training in multi-task learning.
a 0.5. It is of significance to note that this theoretical conclusion is also validated by the subsequent empirical result (Figure 4).
Connection with the free energy. Loss in Equation 15 mirrors the free energy minimization principle
4. The Theoretical Analysis
F= U - -TS
In this section, we show that M3-JEPA optimizes the infor- mation content by simultaneously maximizing the mutual information and minimizing the conditional entropy. We further discuss the optimality of loss weight, as well as the convergence condition.
in which [(:; y) corresponds to the internal energy U, and H(yl=) + H(xxy) corresponds to the entropy TS. The critical temperature T. where the energy/entropy balance occurs corresponds to a = 0.5.
4.1. The Information- Theoretic Analysis
Derivation from the convergence assumption. For sim- plicity, we show this derivation with the case of two tasks, x + y and y + z. Given a large enough step i, it is rea- sonable to assume that the losses of two consecutive steps converge to each other. Then the stable total loss can be approximated by
We start this section with the information-theoredl analysis. To simplify the problem, we take the vision- sainunla learn- ing as an example', with M = =T = 2. We then propose a new perspective of loss objective:
6 f- arg min I[;; y) + a (H(y)=) H(xl))) (15)
9
L + (c(r - v) + Lly + x))
1
in which [(x: y) is the mutual information between r and y, while H ( æ)) and (xrly) are their conditional entropies. COMPLETER (Lin et al., 2021) proves the connection be- tween these terms and different loss types:
2
1
I[;;; v) + +H(ylx) I(y; + H((ly))
2
1
=-[(z; v) + 2 (H(y) [z) + H((zl))
(16)
Redundancy reduction: the contrastive loss Let maxi- mizes Z by separating negatives.
based on the fact [(x; y) I(y; z²). Comparison between Equation 11, 15 and 16 indicates the optimal a 0.5.
Uncertainty reduction: the regularization loss Greg minimizes H by regressing the positives.
4.3. Convergence of AGD on M3-JEPA
We assume for each task t, minimizing the task-specific loss in Equation 11 is independent and convex. From this assumption, from previous derivation (Jain & Kar, 2017; Pascal et al., 2022; Wibisono et al., 2022), AGD multiple tasks (Equation 14) is guaranteed the convergence to a local optimum if each subtask is convex and optimally solved.
Connection with the original loss. The above state- ments ensure that minimizing the averaged task-specific losses (F(xr Fly, æ)) in Equation 11 equivalent to the information-theonm thhorrtticc objective in Equation 15. Also, the loss weight in both equations is consistent, balaneing the compression (Cer and Z) and predictiveness (Creg and H).
5. Experiment
Information decomposition and coupling, Conditional entropy H(ylz) and H(x y)) represent the modality-specific information in text and image, respectively; while the mu- tual information I(x; y) quantifies the shared information between image and text. For reasonable alignment, high mutual information (more shared content) and low condi- tional entropy (less modality-speciff noises) are desirable, ensuring strong information coupling across modalities.
To empirically verify our theoretical claims of M3-JEPA, design the experiments to address the following research questions:
RQ1: Can M3-JEPA align well on the cross-modal repre- sentation on typical multimodal tasks?
RQ2: Can the same architecture be arbitrarily expanded to more forms of information (other than well-studied text, im- age and audio)), or generalized to unseen data and domains? RQ3: Can M3-JEPA take
4.2. The Optimal Loss Weight
In this subsection, we derive the optimal loss weight from two different aspects, both of which suggest the optimal
output), instead of single-modality RQ4: Are MoE, finetuning and AGD all reasonable compo- nents of M3-JEPA?
In this case, both x and consist of only one modality. Two related tasks (image - text and text- image) can be simply represented by x + y and y + T.
RQ5: Are both contrastive and regularization losses neces- sary to achieve the optimality, and how about their weights?
5

---

M3-JEPA: Multimodal Alignment via MMoE based on JEPA
Table 1. Finetuned results Vision-Languaage Retrieval tasks.
Flickr30K
COCO
Method
Trainable Params
Image
Text
Text - Image
Image
Text
Text Image
R@1
RØ5
R@10
RØ1
R@5
R@
RØ1
RØ5
R 10
RØ1
RØ5
R
Tightwnigho models
TinyCLIP et ol., 2023)
63M-31M
38.5
MobileCLIP (Vasu al., 2024)
<30.7M
85.9
67.7
58.7
40.4
Dual-encoder models
CLIP (Radford et al., 2021)
428M
88.0
98.7
99.4
68.7
90.6
95.2
ALIGN (Cohen, 1997)
93.5
96.9
59.9
FILIP (Yao ol., 2022)
417M
89.8
99.2
75.0
93.4
96.3
78.9
94.4
97.4
61.2
84.3
90.6
Florence (Yuan et al., 2021)
893M
90.9
99.1
76.7
93.6
81.8
95.2
63.2
85.7
BEIT-3 (Wang al., 20235)
198
94.9
99.9
100.0
81.5
95.6
97.8
84,8
96.5
98.3
67.2
87.7
92.8
Fuston-encoder models
UNITER
2020)
88.6
52.9
79.9
OSCAR (Li ot al., 2020)
345M
70.0
91.1
95.5
54.0
80.8
88.5
ViaVL (Zhang et al., 2021)
345M
75.4
92,9
96.2
58.8
83.5
90.3
Dual eneader Fusioo encoder
ALBEF (Li 2021)
96.3
94.3
97.2
60.7
BLIP (Li al., 2022)
446M
97.1
100.0
100.0
86.7
97.3
98.7
82.4
95.4
97.9
65.1
86.3
91.8
BLIP-2 w/ VIT-L Li ct al., 2023)
474M
96.9
100.0
100.0
88.6
97.6
98.9
83.5
96.0
98.0
66.3
86.5
91.8
BLIP-2 w/ ViI-g (Li et 2023)
1.2B
97.6
100.0
100.0
89.7
98.1
98.9
85.4
97.0
98.5
68.3
87.7
92.6
M3-JEPA
140M
97.8
100.0
100.0
97.8
100.0
100.0
87.7
99.6
99.9
89.7
99.7
99.9
RQ6: Is M3-JEPA a computational efficient framework for both training and inference?
test set. Table shows the experimental results, compared to previous state-of offte-of-ther fi-thhe-r art baselines across different archi- tectures. M3-JEPA obtained superior performance on both image-to-text and text-to-image tasks. This observation in- dicates that our framework achieves stronger cross-modal alignment on the latent space, capturing different levels of abstraction the cross-modal relationship.
In the following subsections, we discuss different experi- mental results and answer the above questions.
5.1. Setting
M3-JEPA employs pretrained uni-modal encoders and con- nects their latent spaces with an MoE. We use LLama3-SB
M3-JEPA also demonstrates superior computational effi- ciency by observing the size of training parameters. To partially address RQ6, notice that M3-JEPA has only 140M trainable parameters, which is significantly smaller than the 1,2B BLIP-2, the second-bes method Table
(Dubey al., 2024), 2inov2-Large (Oquab et al., 2025) and Lang (Zhu et al., 2024) to encode text, image and audio modalities, respectively. We select 3 layers of the modality encoders to be finetuned by LoRA (Hu et al., 2022) (with rank=64), while keep the rest of parameters frozen. We implement an MMoE predictor with = 12, K = and L = 2. The inner hidden size (h) is 2048 and dropout rate is set to 0.1. The MoE predictor is initialized randomly with full parameters updated during the training.
In Figure 3, we also visualize the similarity matrix for 10 text-image pairs from COCO, snapshot of the image- text retrieval performance. It shows that M3-JEPA can differentiate positive pairs (the diagonal grids) and negative pairs (the non din-giignnlll grids) well, which ensures retrieval performance.
All tasks have the batch size of 128, solved by the Adam optimizer with the Ir schedule of cosine, warmup of 0.1 and weight decay of 0.005. For retrieval tasks, we evalu- ate recall-basee metrics including R@1 1, R@5 and @ 10. For classification tasks, we provide metrics such as Accu- racy, Precision, Recall and Fl score. Further details of implementation, datasets and benchmarks summarized in Appendix A.
5.3. Audio-Language Retrieval
In address RQ2, we attempt to adapt M3-JEPA to new modality and simultaneously examine generalization abil- ity. In more detail, we experiment on audio-text retrieval tasks by replacing the image encoder with the audio en- coder. For fair comparison, inherit the same experi- mental setting from LanguageBind (Zhu et al., 2024), iee. training on a held-out audio-language dataset (including wavtext5k (Deshmukh et al., 2023) and freesound (Fonseca et al., 2022)) while evaluate on Clotho (Drossos et al., 2020)
5.2. Vision- Lisgua-Language Retrieval
To answer RQ1, first validate M3-JEPA image-text retrieval tasks, including COCO (Lin et al., 2014b) and Flickr30K (Plummer et al., 2015) and evaluate it on the
6

---

M3-JEPA: Multimodal Alignment via MMoE based JEPA
Table Audio-text retrieval results. Results of AVFIC, ImageRind and VAL.RR obtained from Zhu al. (2024) directly. We download the original model of LanguageBind and evaluate it by ourselves to collect the results of all metrics.
Clotho
Audiocaps
Method
Audio
Text
Text + Audio
Audio Text
Text + Audio
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@10
R@1
R@5
R@10
AVFIC (Nagrani et al., 2022)
3.0
17.5
8.7
37.7
ImageBind (Girdhar et ul., 2023)
6.0
28.4
9.3
42.3
VALOR (Liu et al., 2025)
8.4
LangungeBind (Zhu et al., 2024)
16.1
39.9
53,2
15.5
38.6
51.7
17.8
47.3
64.0
16.5
48.7
64.6
M3-JEPA (ours)
17.0
40.8
53.0
20.1
45.2
58.7
20.4
50.8
66.6
19.8
5144
66.8
Table 3. Image classification results on ImageNet-IK. All results in percentage.
1
2
4
5
7
1.00
0
Method
Accuracy
Precision
Recall
FI
1
CLIP--IT (Radford 2021)
82.1
82.4
82.0
82.0
Dino (Oquab al., 2025)
83.2
83.5
83.3
83.1
2
M3-JEPA (ours)
86.6
86,9
86.6
86.5
3
6:25
Texts
4
the sparsity of audio samples.
5
6-
5.4. Vision Classification
7
This subsection further addresses RQ2 by expanding to the image classification task. In this situation, we assume M3- JEPA can still learn well, by treating the classified labels as another modality. To validate this hypothesis, we train and evaluate M3-JEPA on ImageNet- 1K (Deng et al., 2009) and compare it to Dino (Oquab al., 2025) and CLIP-ViT ViT (Radford al., 2021). Specifically, Dino is vision-only, while CLIP-ViT ViT is text-guided vision model; they cap- ture distinct types of information and offer complementary perspectives of performance comparison, but may need ex- plicit classification heads. Unlike these methods, we encode the classification labels by one-hot, such that they can be involved in self-supervised manner separate modal- ity. As shown in Table 3, M3-JEPA outperforms DinoV2 and CLIP-VIT VIT on all classification metrics, indicating the potential of M3-JEPA to represent the inherent knowledge of the natural world, not limited to the traditionally studied modalities (text, image, audio, etc).
8
9-
Images
Figure 3. The similarity matrix of the text-image pairs. The colors indicate different levels retrieval scores. up-triangle part indicates text-to-image and the down-triangle part indicates image- to-text. The diagonal grids correspond to ground truth pairs.
and Audiocaps (Kim et al., 2019) in a zero-shot manner2. Since LanguageBind (Zhu et al., 2024) provides limited results of metrics (only @ and @ 10 text-to-audio), we re-run the evaluation test of LanguageBind on all text- to-audio and audio-to-text metrics As shown in Table 2, M3-JEPA still demonstrates superior performance over all baselines, showcasing its alignment capability in the au- dio modality and an exceptional level of generalization. this zero-shot setting, M3-JEPA also becomes extremely sensitive to the dataset bias (i.e., the ratio of freesound to wavtext5k) and their sampling intervals and lengths, due to
5.5. Vision-Language Understanding
To address RQ3, we examine M3-JEPA a more challeng- ing scenario, where the input or output consists of multiple modalities, instead of uni-modality. To test M3-JEPA adaptation ability to this situation, we conduct the Visual Question Answering (VQA) task, in which the model is concurrently prompted with an image and textual question, and is expected to provide a reasonable textual answer. In this situation, we integrate the image encoding and text en- coding by simple concatenation into the input, then feed into the MMoE predictor. The rest of the algorithm pipeline
"One can refer to Zhu (2024) for more details of training datasets other settings.
Forr overlapping metrics, our results are close to the origi- nally reported values in (Zhu al., 2024) and do not change the conclusion.

---

M3-JEPA: Multimodal Alignment via MMoE based on JEPA
is kept the same.
significant performance degradation, as indicated by the second row of Table 5.
Training and evaluation are performed VQAv2 (Goyal et al., 2017) and 2 (Suhr et al., 2019). Results are ex- hibited in Table 4. M3-JEPA adapts well to this multimodal input scenario, obtaining the second-best result obtained for each split of test sets. Although results of M3-JEPA are lower than (Wann et al.. 2023b), it might be due to the large pretrained corpus of BEYT-3, including MSCOCO (Lin et al., 2014a) and Visual Genome (Krishna et al., 2017)). Furthermore, M3-JEPA could be further improved by im- plementing smarter fusion of multimodal information (e.g., cross-attention between input modalities), instead of simple concatenation. We will leave this further attempt to future work. To better discuss this observation, we put a detailed bad case analysis in Appendix B.3.
Ablation on finetuning of modality encoder. During training, one can choose the uni-modality encoder to be completely frozen, finetuned on its full layers or part of layers (we test for 3 layers). We show this ablation result in Table 6. Results indicate that finetuning has positive impact on both performances of image-to-text and text-to- image, while full-layer LoRA generally performs better than 3 3-layer LoRA. Nevertheless, full-layer LoRA is also sub- ject to a higher time cost. As a result, we apply 3-layer LoRA in all the aforementioned formal experiments, and we achieve state-of-the-aa performance on vision- languaee and Laniouaa tasks with only 3 layers finetuned.
Table 6. Ablation of modality encoder finetuning COCO.
Table VQA scores on VQAv2 and NLVR-2. For each test set, the bold number indicates the best result and the underlined number indicates the second best.
Appeoach
Imuge
Text
Image
R@1
R@:5
R@ 10
R@1
R@5
R@ 10
Method
VQAv2
NLVR-2
freeze
75.4
88.6
94.5
84.3
90.1
97.8
test-dev
test-std
dev
test-P
3-layer LoRA
87.7
99,6
99.9
89.7
99.7
99.9
full-aayer LoRA
92.1
99.4
99.9
91.1
99.8
99,9
ALBEF (Li al., 2021)
75.8
76.0
82.6
83.14
BLIP (Li et al., 2022)
78.3
78.3
82.2
82.2
X-VLM (Zeng et al., 2022)
78.2
78.4
84.4
84.8
Sensitivity analysis on the loss weight. We then conduct the sensitivity study on an important hyper-parameter, the loss weight a between Let and Creg. The experiment is conducted on the text-to-image task of COCO, with the R@I I results shown in Figure 4. One can observe that the best performance is achieved when a = 0.5, indicating that both contrastive loss and regularization loss are necessary, We then select 0.5 as the formal choice of a in all formal experiments, which also empirically verifies the theoretical conclusion in Section 4.2.
SimVLM (Wang et al., 2022b)
80.0
80.3
84.5
85.2
OFA (Wang et al., 2022a)
82.0
82.0
Flamingo (Alayrac et al., 2022)
82.0
82.1
CoCa (Yu al., 2022)
82.3
82.3
86.1
87.0
BLIP-2 (Li et al., 2023)
82.2
82.3
BEiT-3 (Wang et al., 2023b)
84.2
84.0
91.5
92.6
M3-JEPA (ours)
82.3
82.5
86.8
87.6
5.6. Analysis and Discussions
We conduct further analysis, including ablation and sensitiv- ity studies, as well as an efficiency analysis, to address RQ4, RQ5 and RQ6, and obtain deeper insight on M3-JEPA.
90
89
Ablation on the M3-JEPA approach. To validate the design of the MoE predictor design, we replace it with a comparable-size MLP, with results shown in the first row of Table 5. Compared to the last row (the formal M3-JEPA), significant drop in performance is observed, demonstrating the effectiveness of MoE.
R@I
88
87 0
0.25
0.5
0.75
a
Figure 4. Sensitivity plot of the loss weight a.
Table 5. Ablation of the M3-JEPA approach on COCO.
Image + Text
Text + Image
MoE
AGD
R@1
R@S
R@ 10
R@1 1
R@s
R@ 10
Analysis of computational efficiency. In order answer RQ6, here discuss the computational efficiency of M3- JEPA from the aspects of training and inference costs, re- spectively. Although M3-JEPA has relatively heavy modal- ity encoders, its training cost is relatively low, since only the lightweight MMoE predictor and several layers of encoders (3 layers in the formal experiment) need be trained, while
x
V
74.4
86.0
92.2
82.3
89.5
92.6
v
x
68.2
68.7
81.1
74.2
88.7
92.4
87.7
99.6
99.9
89.7
99.7
99.9
Additionally, we examine the impact of the AGD approach. Ablating it to a non-alternating optimization also leads to
8

---

M3-JEPA: Multimodal Alignment via MMoE based on JEPA
the rest parts of encoder parameters are freeze. To elabo- rate the comparison of training costs baselines, we also include the # of trainable parameters in Table 1.
concept by interconnecting vision, text and audio encoders through multi-gate MoE. We design the gating function to separate the model-specific and shared information to optimize the cross-modal performance.
For the inference cost, one should notice that M3-JEPA supports the modality precomputing and online caching, while the cross-modality alignment only happens in the latent space of the MMoE predictor (which is relatively lightweight compared to the uni-modality encoders), sig- nificantly reducing the memory overhead. To illustrate this difference, we calculate the averaged retrieval time (RT) of M3-JEPA on COCO, a typical image-text retrieval task. We observe the RT of M3-JEPA is only 0.02 seconds, com- paring to CLIP, a classical dual-encoder approach, with an RT of 0. and BL.IP-,, which relies on a lightweight Q-former, with an RT of 0.05s. These results indicate M3- JEPA with modality pre-computing is much more compu- tationally efficient, and can be potentially applied as an efficient multimodal retriever on massive documents.
6.2. Multimodal Models with MoE Structures
There are also pioneer works that integrate multimodal mod- els with MoE-like structures. For instance, LIMoE (Mustafa et al., 2022) feeds both images and text signals into a single MoE encoder, trained by a contrastive loss. VL-MoB (Shen et al., 2023) uses mspality- experts for image-text modeling. MoE-LLaVA (Lin al., 2024) proposes MoE- Tuning. a training strategy for large vision-language models, and construct sparse model with constant computational cost. In contrast, M3-JEPA adopts the multi-gat -gte MoE ar- chitecture, which simultancously feeds parallel output terms to both contrastive and regularization losses.
For inference with dynamic inputs (i.e., user uroviddd im- ages/text) which are intractable to be precomputed, M3- JEPA's retrieval latency is dominated by the modality en- coder inference (approximately 1/ for DINOv2, 0.3s/text for LLaMA-3-8B) In such a situation, it is recom- mended to cache frequent queries (e.g., common prompts in retrieval systems), for
6.3. Multimodal Models Trained by AGD
Although most multimodal learning is trained jointly and concurrently, there have also been attempts that train the model by alternating gradient descent (AGD), i.e. alternat- ing propaaatttt across different tasks. Such efforts in- clude Polyvit (Likhosherstovv et al., 2022) and IMP (Akbari et al., 2023), revealing that the alternation of diverse modal- ities, tasks, and resolutions may enhance the model's cross- domain performance and generalization capabilities. In our study, we apply AGD on the joint-embedding - predictive architecture, in which the gradient descent happens on the latent space, which further mitigates the gradient conflict issue from cross- mooss-yy samples.
applications.
6. Related Works
6.1. Multimodal Learning
Modern multimodal Learning has achieved outstanding per- formance, which can be classified into two main categories: training end-to-end, or training connector based on pre- trained unimodal models. The end-to-end multimodal mod- els can have various architectures, such as dual-encoder (Radford et al., 2021; Jia et al., 2021) or fusion encoder (Li et al., 2020; Chen et al., 2020; Jia et al., 2021; Wang et al., 2022a). Such end-to-end pre-training consumes large- scale multi-modal datasets. As the model scale continues to increase, it may incur prohibitively high computational costs and often struggle to adapt to novel modalities or tasks without extensive re-training.
7. Conclusion
This paper introduces M3-JEPA, a novel any-to-any multi- modal training framework, which is lightweight, scalable, and computationally efficient. M3-JEPA effectively inte- grates pretrained unimodality encoders, projects the input embedding into the output embedding space through a multi- gate MoE predictor, and finally aligns the cross-modal infor- mation on the latent space. The MoE predictor includes two gates, corresponding to the contrastive and regularization losses, while each gating function is designed with sepa- rated model-specific and shared learnable encodings. The M3-JEPA optimized by alternating the multi multii-iirectonall multimodal tasks, facilitating the cross-modal alignment in multiple directions. By theoretical derivation, we demon- strate that M3-JEPA is information-theoretic optimal, while its convergence can be ensured with reasonable assumptions on the subtasks. Extensive experiments finally demonstrate that M3-JEPA can achieve state-of-the-a performance on different types of tasks and modalities, and generalize well on unseen datasets and domains.
Another category of multimodal learning focused on in- tegrating unimodal models to achieve high-quality multi- modal aligmment (Alayrac et al., 2022: Li al., 2023; Liu et al., 2023; Zhang et al., 2024). For example, Flamingo (Alayrac et al., 2022) integrates visual information into each layer of frozen Large L.M through the use of cross- attention. BLIP-2 (Li et al., 2023) introduces a Q-former before feeding into the LLM, and propose also a two-stage training process. LLaVA (Liu et al., 2023) aligns vision and language representation by employing a two-layer mul- tilayer perceptron (MLP). M3-JEPA further advances this

---

M3-JEPA: Multimodal Alignment via MMoE based on JEPA
Impact Statement
Bardes, A., Ponce, J., and LeCun, Y. MC-JEPA: A joint- embedding predictive architecture for self-supervised learning of motion and content features, 2023, URL https://aat /ttpss//aaxvvv./// httpps///aaxii..og/aa/////////// 2230g 12698.
In this paper, we apply JEPA to multimodal tasks, including vision-languaae retrieval, ludio-language retrieval, image classification, and vision-language understanding. We im- plement the predictor of JEPA with multi-gate MoE ar- chitecture, which attempts to separate the model-specific and model-shared information. Two gates of MoE generate parallel outputs which are used to calculate the contrastive loss and the regularization loss, respectively. We name this framework as M3-JEPA, and optimize by AGD, an alternative optimization strategy between different multi- modal tasks. An cnarmation-thhoretiee analysis is provided, which connects our loss with maximization of the mutual information and minimization of the conditional entropy. We also discuss the convergence of AGD, and the optimal loss weight in this formulation. M3-JEPA achieves state- of-the-art performance across various tasks and modalities, and also exhibits strong generalization and high compu- tational efficiency. This work opens new directions for supervipe multimodal learning and open-world under- standing.
Chen, Y.C.., Li, L.,, Yu, L., El Kholy, A., Ahmed, F., Gan, Z.., Cheng, Y., and Liu, J. Uniter: Universal image-text repre- sentation learning. In European conference on computer vision, pp. 104-120. Springer, 2020.
Cohen, G. H. Align: program to superimpose protein coor- dinates, accounting for insertions and deletions. Journal of applied crystallography, 3006):1100-1166 1997.
Dawid, A. and LeCun, Y. Introduction to latent variable energy-based models: a path toward autonomous machine intelligence. Journal of Statistical Mechanics: Theory and Experiment, 2024(10), October 2024. ISSN 1742- 5468. doi: 10.1088/17425546//aaaa right: ① 2024 The Author(s).
Publisher Copy-
Deng. J., Dong, W., Socher, R., Li, L.J.,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vi- sion and Pattern Recognition, pp. 248-255, 2009. doi: 10. 1..109///PR.0009 55068488
References
Akbari, H., Kondratyuk, D., Cui, Y., Hornung, R., Wang, H., and Adam, H. Alternating gradient descent and mixture- of-experts for integrated multimodal perception. In Ad- vances in Neural Information Processing Systems, vol- ume 36, pp. 79142-79154, 2023.
Deshmukh, S., Elizalde, B., and Wang. H. Audio retrieval with wavtext5k and clap training. In Interspeech 2023, pp. 2948-2952, 2023. doi: 10.21437/Intesspeech..o 2023-11102233133 1136.
Drossos, K., Lipping, S., and Virtanen, T. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 736-740. IEEE, 2020.
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: a visual language model for few- shot learning. Advances in neural information processing 2022.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
systems, 35:23716-2371
Antol, S., Agrawal, A., Lu, J., Mitchell, M.. Batra, D., Zitnick, C. L., and Parikh, D. VQA: Visual Question Answering. In International Conference on Computer Vision (ICCV), 2015.
Fonseca, E., Favory, X., Pons, J., Font, F., and Serra, X. FSD50K: an open dataset of labennl sound events. IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, 30:829-852. 2022.
Assran, M., Duval, Q., Misra, I., Bojanowski, P., Vincent, P., Rabbat, M., LeCun, Y., and Ballas, N. Self-supervised learning from images with joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15619- 15629, 2023.
Font, F., Roma, G., and Serra, X. Freesound technical demo. In Proceedings of the 21 ACM international conference on Multimedia, pp. 411-412, 2013.
Garmash, E. and Monz, C. Ensemble learning for multi- source neural machine translation. In Matsumoto, Y. and Prasad, R. (eds.), Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pp. 1409-1418, Osaka, Japan, December 2016. The COLING 2016 Organizing Committee. URL https: http//ttaaaattoogooooooooooooola .or// C1 C16-1133/.
Bao, H., Wang, W., Dong, L., Liu, Q., Mohammed, O. K., Aggarwal, K., Som, S., Piao, S., and Wei, F. Vlmo: unified vision-language pre-training with mixture-of- moxality-experts. In Proceedings of the 36th Interna- tional Conference on Neural Information Processing Sys- tems, NIPS '22, Red Hook, NY. USA, 2022. Curran As- sociates Inc. ISBN 0781713871088
10

---

M3-JEPA: Multimodal Alignment via MMoE based on JEPA
Girdhar, R., El-Nouby, A., Liu, Z.., Singh, M., Alwala, K. V., Joulin, A., and Misra, I. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15180-15190, 2023.
Kim, C. D., Kim, B., Lee, H., and Kim, G. AudioCaps: Generating captions for audios in the wild. In Burstein, J., Doran, C.,, and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pp. 119- 132, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N199 1011. URL https: : http///caaaaacooogyy.oo.o - org 9-1-1991100 91101.
Goyal, Y., Khot, T., Summers-Star Stay, D., Batra, D., and Parikh, D. Making the in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-69999999 6913, 2017.
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K.,
Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J,, Shamma, D. A., Bernstein, M. S., and Fei-Fei, L. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Int. J. Compur. Vision, 123 (1):32-73, May 2017. ISSN 0920-5691. doi: 10.1007/ $11263-016-0981-7 URL https: ptttts:://oli org/110..... 0007/s11263-011--9900
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https:// openreview net/ wwwumoiiureoooo.eooo.oo. www.mmiieddddeeeeeee....
Huh, M., Cheung, B., Wang, T., and Isola, P. Position: The platonic representation hypothesis. In Salakhut- dinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceed- ings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learn- ing Research, pp. 20617-20644 PMLR, 21-27 Jul 2024. URL https: /ttp///roeeeeeeeeeeeeececccocccc ngs m.l..p........ r.peess//// v235 1235//uu2445 html.
Li, J., Selvaraju, R., Gotmare, A., Joty, S., Xiong, C., and Hoi, S. C. H. Align before fuse: Vision and language representation learning with momentum distillation. Ad- vances in neural information processing systems, 34: 9694-9705, 2021.
Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language Linuugg-imagg pre-training for unified vision- languala understanding and generation. In International confer- ence on machine learning, pp. 12888-12900 12900. PMLR, 2022.
Jain, P. and Kar, Non-convex Optimization for Machine Learning. 2017. doi: 10. 10.1561/22000000588
Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th In- ternational Conference on Machine Learning, ICML'23. JMLR.org. 2023.
Javaloy, A., Meghdadi, M., and Valera, I. Mitigating modality collapse in multimodal VAEs via impartial op- timization. In Chaudhuri, K., Jegelka, S., Song, L.., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Pro- ceedings of the 39th International Conference Ma- chine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 2022. URL https: httpppp:://rroooe /ttp///rrceeeececcceeee mlr. prr.s/sss/ v1
X., Yin, Li, C.,, Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F., et al. Oscar: Object- semantics aligned pre-training for vision-language tasks. In Computer Vision-ECCV 2020: 16th European Con- ference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16, pp. 121-137. Springer, 2020.
PMLR, 17-23 Jul
html.
Jia, C., Yang, Y., Xia, Y., Chen, Y.TT., Parekh, Z., Pham, H., Le, Q., Sung, Y.H.,, Li, Z., and Duerig, T. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp. 4904-4916. PMLR, 2021.
Likhosherstov, V., Arna,, A., Choromanski, K., Lucic, M., Tay, Y., Weller, A., and Dehghani, M. Polyvit: Co- training vision transformers on images, videos and audio. Transactions on Machine Learning Research, 2022.
Jin, P., Zhu, B., Yuan, L., and Yan, S. Moh: Multi-head attention as mixture-of-head attention. arXiv preprint 11Xiv:2410.11842. 2024.
Lin, B., Tang, Z., Ye, Y., Cui, J., Zhu, B., Jin, P., Zhang, J., Ning, M., and Yuan, I.. MoE-LLLaVA Mixture of experts for large vision-lann -laguuge models, arXiv preprint arXiv:2401.1594/ 2024.
Jun Zhan, Junqi
L. L. H. Y. F. T. G. T. S. V.J.X.X Q. Anygpt: Unified multimodal IIm with discrete sequence modeling, 2024.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft COCO: Common objects context. In Fleet, D., Pajdla,
URL httph:tt//////tt //ops:://as/ //tps:////xvvvv/// www.riivvvvavv.eeeeeooeeee www./bs//////.oo. 2btp//2..2222. 1226.

---

M3-JEPA: Multimodal Alignment via MMoE based on JEPA
T., Schiele, B., and Tuytelaars, T. (eds.), Computer Vi- sion ECCV 2014, pp. 740-755, Cham, 2014a. Springer International Publishing, ISBN 778-3-319-10602-1
Nagrani, A., Seo, P. H., Seybold, B., Hauth, A., Manen, S., Sun, C., and Schmid, C. Learning audio-video modal- ities from image captions. In European Conference on Computer Vision, pp. 407-426. Springer, 2022.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P.. and Zitmick, C. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12,62 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014b.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby. A., et al. Dinov2: Learning robust visual features without supervision. In International Conference on Learning Representations, 2025.
Lin, Y., Gou, Y., Liu, Z., Li, B., Lv, J., and Peng, X. Com- pleter: Incomplete multi-view clustering via contrastive prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.
Pascal, L., Michiardi. P., Bost, X., Huet, B., and Zuluaga, M. A. Improved optimization strategies for deep multi- task networks, 2022. URL https: /ttp:///axxvvooo abs abs//109... 111/21199..11...
Liu, D., Zhang, R., Qiu, L., Huang, S., Lin, W., Zhao, S., Geng, S., Lin, 7.,, Jin, P., Zhang, K., et al. Sphinx- Scaling data and parameters for a family of multi- modal large language models. In Forty-first International Conference on Machine Learning.
Plummer, B. A., Wang. L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., and Lazebnik, S. Flickr30k entities: Collecting region-to region-to-phrase correspondences for richer amage-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pp. 2641- 2649, 2015.
x:
Liu, H., Li, C., Wu, Q.. and Lee, J. Visual instruction tuning. In Proceedings of the 37th International Confer- ence on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Liu, J., Chen, S., He, X., Guo, L., Zhu, X., Wang, W., and Tang, J. VALOR: Vision-audio-languaa omni- perception pretraining model and dataset. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 47 22:7088724, 2025. doi: 10.109TTPAM..200.44477777
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference Learning Representations, 2017. URL https::/// /wppneeveeeeeeeeeeeeeeeeeee net/ forum? i id=B1 ckMDqlg.
Ma, J., Zhao, Z., Yi, X., Chen, J., Hong, L., and Chi, E. H. Modeling task relationships in multi-task learn- ing with multi-gate mixture-of mixture-of-experts. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, KDD '18, pp. 1930-1939, New York, NY, USA, 2018. Associa- tion for Computing Machinery. ISBN 9781450355520. doi: 10. 10.1115//00 (0.11445/221981993222000 org/10. 145 1tt.1/0..115//21111...22222 3220007.
Shen, S., Yao, Z. Li, C., Darrell, T., Keutzer, K., and He, Y. Scaling vision-language models with sparse mix- ture of experts. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 11329-11344, Singa- pore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2022 http://...innngseemmeiii 758. URL https: / aaachhhhllll ogy. orgg/003.. E Findings-emnlp... 758/.
URL https: //doi.
Ma. M., Ren, J.. Zhao, L., Testuggine, D., and Peng. X. Are multimodal transformers robust to missing modality? In 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp- 18156-18165, 2022. doi: 10. (UU9/CVPR52688.022..000
Suhr, A., Zhou, S., Zhang, A., Zhang, I., Bai, H., and Artzi, Y. A corpus for reasoning about natural language grounded in photographs. In Korhonen, A., Traum, D., and Mårquez, L. (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguis- tics, pp. 6418-6428, Florence, Italy, July 2019. Associ- ation for Computational Linguistics. doi: 10. 18653/v1l P19-1644. URL https: / /aclanthooooyyy.....ooc .org/ P19-1644/.
Mustafa, B., Riquelme, C., Puigcerver, J., Jenatton, R., and Houlsby, N. Multimodal contrastive learning with limoe: the language-image mixture of experts. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 9564-95675 Inc., 2022.
Curran Associates,
12

---

M3-JEPA: Multimodal Alignment via MMoE based on JEPA
Vasu, P. K. A., Pouransari, H., Faghri, F., Vemulapalli, R., and Tuzel, O. Mobileclip: Fast image-text models through multi-modal reinforced training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15963-15974, 2024.
Wu, Z., Dadu, A., Tustison, N., Avants, B., Nalls, M., Sun, J., and Faghri, F. Multimodal patient representation learn- ing with missing modalities and labels. In The Twelfth International Conference on Learning Representations, 2024b.
httpes//ppppeeeee /wpp//ppeeeeeeeeeeeeeee net/ forum? id-Je5SHCKpPa.
Wang, P. Yang, A., Men, R., Lin, J., Bai, S., Li, Z., Ma, J., Zhou, C., Zhou, J., and Yang, H. Ofa: Unifying architec- tures, tasks, and modalities through a simple sequence- to-sequence learning framework. In International con- ference machine learning, pp. 23318-23340. PMLR, 2022a.
Yao, L., Huang, R., Hou, L., Lu, G., Niu, M., Xu, H., Liang, X., Li, Z., Jiang, X., and Xu, C. Filip: Fine- grained interactive language-image pre- pre-training. In In- ternational Conference on Learning Representations, 2022. URL https: pattps://openreeett /ttp////ppeee.eeeeeeeeepee http:/nee//eeeeeeeeeeeeeeee formm? i id=ggDhesEDC2.
Wang, P., Wang, S., Lin, J., Bai. S., Zhou, X., Zhou, J., Wang, X., and Zhou, C. One-peace: Exploring one gen- eral representation model toward unlimited modalities, 2023a. URL 11172.
Yu, J., Wang, Z.., Vasudevan, V., Yeung, L., Seyedhosseini, M. and Wu, Y. Coca: Contrastive captioners are image- text foundation models. arXiv preprint 2rX11:2205.01911, 2022.
org. abs 2305.
Yuan, L.. Chen, D., Chen, Y.-L., Codella, N., Dai, X., Gao, J., Hu, H., Huang, X., Li, B., Li, C.,, et al. Florence: A new foundation model for computer vision. arXiv preprint 1 2021.
Wang, W., Bao, H., Dong, I.., Bjorek, J., Peng, 7., Liu, Q. Aggarwal, K., Mohammed, O. K., Singhal, S., Som, S., and Wei, F. Image as a foreign language: BEIT pretraining for vision and vision-language tasks. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19175-19186, 2023b. doi: 10. WWWCVVPRS272..0223.000
Zeng. Y., Zhang, X., and Li, H. Multi-grainee vision language pre-training: Aligning texts with visual con- cepts. In Chaudhuri, K., Jegelka, S., Song, L., Szepes- vari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learn- ing, volume 162 of Proceedings of Machine Learn- ing Research, pp- 25994 26009, PMLR, 17-23 Jul 2022. URL htts::::: / http:///oooeddeeeeeeooooooocc ngs ml m.lr. press/ v162 we////engn...cccc html.
Wang, 7., Yu, J., Yu, A. W., Dai, 7.,, Tsvetkov, Y., and Cao, Y. SimVLM: Simple visual language model pre- training with weak supervision. In International Confer- ence on Learning Representations, 2022b. URL https: //openreview, net/ / forum? i id=GUrhfTuf_....
Wibisono, A.,
Tao, M., and Piliouras, G. Alternating mir- ror descent for constrained min-max games. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https: ///p//peeeveeweeeeeeeeeeeee wtwpiieww.eeeeooeeeeeeeeeeeeeeee forum? Ad=w4x7GLhhiu.
Zhang. D., Yu, Y., Dong, J., Li, C., Su, D., Chu, C., and Yu, D. MM-LLMs: Recent advances in Multi- Modal large language models. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 12401- 12430, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 111538/v1/202/ findings-acl.738. URL https: aa/aacaaan aclanthology. org 22//2024.22 w.........iing...eeeee.....e..ee . 738/.
Wu, K., Peng, H., Zhou, Z., Xiao, B., Liu, M., Yuan, L., Xuan, H., Valenzuela, M., Chen, X. S., Wang, X., al. Tinyclip: Clip distillation via affinity mimicking and weight inheritance. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 21970- 21980, 2023.
Zhang, P., Li, X., Hu, X., Yang. J., Zhang, L., Wang, L., Choi, Y., and Gao, Vinvl: Revisiting visual representa- tions in languaee models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5579, 2021.
Wu, S., Fei, H., Qu, L., Ji, W., and Chua, T.SS NEXT- GPT: Any-to-any multimodal LLM. In Salakhutdi- nov, R., Kolter, Z., Heller, K., Weller, A. Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceed- ings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 53366-53397, PMLR, 21-27 Jul 2024a. URL https:/ http::///oooeeeeeeee wwi.nngsmmm.... mlr. press/v233 wu24e. htmll
Zhu, B., Lin, B., Ning, M., Yan, Y., Cui, J., Wang, H., Pang, Y., Jiang, W., Zhang, J., Li, Z., al. Lan- guagebind: Extending video-language pretraining to n- modality by language-based semantic alignment. In International Conference on Learning Representations, 2024. URL https::///// /openreviewwweeeeeeeeeeeeee net/ forum? i id=Qm2ke7UZcy.
13

---

M3-JEPA: Multimodal Alignment via MMoE based on JEPA
A. Extra Implementation Details
the Clotho dataset; then train M3-JEPA on the mixture of Clotho, WavText5K, and Freesound then test Audio- Caps dataset. The initial learning rate is 5.0c-4 and the final learning rate is 2.5e-6.
A.1. Image-Texi Retrieval
For image-text retrieval, we examine M3-JEPA on two fa- mous datasets, COCO and Flickr30K, with the detailed introductions below:
A.3. Image Classification
We train and test M3-JEPA the popular benchmark, ImageNet- IK (Deng et al., 2009), which contain 1,281,167 training images, 50,000 validation images, and 100,000 test images across totally 1,000 classes. For image classification, the initial learning rate is 1.0e-3 and the final learning rate is 5.5e-6.
COCO (Lin et al., 2014b): 330,000 images with object annotations and captions, providing a rich resource for multi-label image classification and visual understand- ing. encompasses 80 object categories with 5,000 training and 1,000 testing images per category. The test set contains 5,000 samples.
A.4. Visual Question Answer
Flickr30K (Plummer et al., 2015): comprises over 30,000 images, each paired with five descriptive sen- tences. It sourced from Flickr and reflects the diver- sity and complexity of real-world data, making it suit- able for tasks such as image annotation, visual question answering, and image retrieval. The test set contains 1,000 samples.
We consider the following two datasets for the VQA task:
VQAv2 (Antol et al., 2015): 265,016 images with mul- tiple questions per image, assessing the ability of mod- els understand and answer questions about visual content.
For each dataset, finetune M3-JEPA its training set and evaluate it its test set, respectively. The initial learn- ing rate is 0.001 and the final learning rate is 5.5e-6.
NLVR-2 (Suhr et al., 2019): 107,292 pairs of images with corresponding sentences, testing the visual rea- soning capabilities of models.
A.2. Audio-Text Retrieval
Specifically, the VQA task requires the model to answer textual questions about input images. We starts from the pretrained version of M3-JEPA by the COCO datasets, then finetune it following previous works (Wang et al., 2023b;a; Bao et al., 2022), which formulate the VQA task as a textual answer retrieval problem. The initial learning rate 2.0e-4 and the final learning rate 2.4e-5.
Details of our audio-language datasets are follows:
Clotho (Drossos et al., 2020): 4,981 audio samples with 24,905 descriptions, sourced from the Freesound platform and crowdsourced from English-speaking contributors.
B. More Experimental Results
Audiocaps (Kim et al., 2019): curated subset of AudioSet, focusing on audio captions and enabling the study of audio-text relationships.
B.1. Parameter Comparisons
To provide a more comprehensive discussion, here we sum- marize the parameter statistics of M3-JEPA, compared to lisguanlanguage model baselines, as listed in Table 7. Specif- ically, M3-JEPA can also be viewed as a lightweight knowl- edge connector, similar BLIP-2. Therefore, although our M3-JEPA has large total parameter size (around 8.5B), its trainable parameter is only 140M, even smaller than BLIP-2, which ensures its training efficiency.
Wavtext5k (Deshmukh et al., 2023): The WayText5K data was sourced from two main websites: BigSound- Bank and SoundBible 3 (details can be found in (Desh- mukh et al., 2023)). WavText5K contains 4505 audios, 4348 descriptions, 4525 audio titles, and 2058 tags.
Freesound(Font et al., 2013): The Freesound dataset contains 363,618 samples, totaling 2,162.10 hours of audio.
B.2. Ablation of MoE Hyperparameters
To zero-shot evaluate the performance of M3-JEPA audio- text retrieval, similar to Similar to LanguageBind (Zhu et al., 2024), we zero-shot evaluate the performance of M3-JEPA on Clotho and Audiocaps based on the knowledge of other datasets. In more details, we train M3-JEPA on the mixture of AudioCaps, WavText5K, and Freesound then test it on
Here we conduct the ablation studies on the structural pa- rameters of MoF, including the number of experts and the top-k ranking mechanism. For n, we compare different values on VQAvq2, with results shown in Table 8. One can observe that a larger n can benefit the VQA performance, where 12 outperforms 2 and 8. However, increasing
14

---

M3-JEPA: Multimodal Alignment via MMoE based on JEPA
Table Parameter statistics of vision-language methodologies.
Method
# total parameter
trainable parameter
CLIP
428M
the same the same the same the same the same the same 474M 140M
ALIGN
820M
FLIP
417M
BEÎT-3
1.9B
UNITER
303M
OSCAR
345M
BLIP-2
4.1B
M3-JEPA
8.5B
also amplifies the computational cost, therefore we do not attempt larger n. For k, conduct the ablation study COCO, with image-text retrieval results shown in Table 9. The results validate the optimality of our formal settings,
Figure 5. A typical failure case on the VQA task. The input image contains multiple visual objects (a horse and a staircase).
issues.
which is k = 4.
C. Limitation and Future Direction
Table 8. Ablation of on the validation set of VQAvq2. The reported score is the accuracy of VQA answers.
In this paper, we propose M3-JEPA and demonstrate its effectiveness across different modalities and tasks. However, the following limitations still persist and hinder M3-JEPA from a more generalized foundation model:
2
8
12
score 55.15
59.84
68.03
Generative capability: the joint-embedding--reee architecture (JEPA) reformulate the next-token predic- tion into the alignment on the latent space, to filter the modality noise and capture the core cross-modal infor- mation. As a cost, JEPA is not a generative framework which prevents its wider applications. Nevertheless, we believe there might be potential solutions that are able incorporate generative learning with JEPA.
Table Ablation & on COCO. The reported metric R@L I.
Flickr30K
COCO
k
Image + Text
Text + Image
Image + Text Text + Image
2
96.0
95.5
85.0
82.0
4
89.7
87.9
97.8
97.8
6
88.0
86.5
97.5
97.0
B.3. Typical Failure Case
Modality expansion: different modalities have differ- ent information intensity. Therefore, popular modality encoders generally encode different modalities into various embedding dimensions. Built upon these pre- trained modality encoders, M3-JEPA needs to adapt with these different input embedding dimensions. Al- though we have disentangled the modality-speciff and shared components inside the architecture, the entire M3-JEPA framework can not be claimed as modality- agnostic. Introducing new modality need to manu- ally select its modality encoder, add the corresponding condality-speciff expert, and redetermined the subse- quent training pipeline. Incorporating the idea of meta or hyper networks into the MoE predicter may increase its adaptability to new information or modality, making it a true modality-agnostic framework.
Currently, we found that M3-JEPA may suffer performance degradation in the existence of multimodality input or out- put, e.g. the VQA task. As discussed before, part of reasons may be the insufficient pretraining, as well as its simple con- catenation strategy of modality input embeddings. To better demonstrate this phenomenon, here we exhibit a typical bad case of M3-JEPA on VQAv2. The input image in Figure 5, while the textual questions, answers, and corresponding scores are shown in Table 10.
This example is challenging to M3-JEPA since it contains multiple visual objects as well as corresponding questions. As indicated in Table 10, M3-JEPA successfully predicts the number and color of the horse, but fails identify the num- ber of steps. This may be due the simple concatenation of visual and question embeddings before passing to the MoE predictor. The MoE tends to focus dominant objects (c.g., the horse), while failing to capture minor details like the stairs. A specifically designed mechanism, such as the cross-nae modal attention, or a unified positional embedding, may help capture finer-grained objects and alleviate such
Generality world model: in the future, one may mask out an arbitrary part of world, and ask a gener- alized world model to predict the masked part, given the information of unmasked content. Without los- ing the generality, the masked and unmasked parts
15

---

M3-JEPA: Multimodal Alignment via MMoE based on JEPA
Table 10. A typical failure on the VQA task, which corresponds the image in Figure 5. The bold indicated the model's
predicted answer. M3-JEPA correctly answers questions about the house, but fails to identify the number of steps.
Question
Answer
Score
brown and white
0.6
What kind of horse is this?
clydesdale
1.0
others
How many horses are in the picture?
others
0.0
10
0.3
0.3
How many steps to the building?
0.3
20
0.9
others
should be assumed as combination of different modal- ities, instead of a single modality. In this paper, we explore such a case, the VQA task. We show that simply concatenating the visual and question embed- dings as the input can have reasonable performance, but is worse than the best baseline, BEÎT-3. Better cross-modality encoding techniques may be needed to further enhance M3-JEPA'S 's performance in such cases, such as Mfxture-of- (MoH) (Jin et al., 2024). Furthermore, the spatial-temporal representation and corresponding positional embedding may also be cru- cial expand M3-JEPA in related scenarios, e.g., and video comprehension.
Possibility of other optimization: this paper, we mainly apply AGD, an alternative optimization strat- egy between tasks, to train M3-JEPA. We conduct the ablation study of AGD and compare its result to joint optimization, and suggests AGD's superiority. Never- theless, we admit there are many details (e.g., various tasks, modalities, losses) and different possible joint optimization implementations, which has not been ex- plored by us. The ultimate optimization strategies can still be experimented.
16


Reviews:
[{"review_id": "prjOIi22wP", "reviewer": "Reviewer_eSmg", "summary": "The paper \"M3-Jepa: Multimodal Alignment via Multi-directional MoE based on the JEPA framework\" introduces a novel modality-agnostic multimodal alignment paradigm called M3-Jpea, which utilizes the Joint Embedding Prediction Architecture (JEPA) to align representations of different modalities into the same latent space and achieves multimodal alignment via a multi-directional mixture of experts (MoE) structure. The authors propose this approach to address the limitations of current multimodal alignment strategies, which typically rely on a single or unified modality encoder and may lead to information bias. M3-Jpea aims to achieve alignment of different modalities by alternating between optimizations of different unidirectional tasks to maximize mutual information and minimize conditional entropy. The main contributions of the paper include: (1) a scalable and modality-agnostic architecture for multimodal alignment. (2) optimization of different unidirectional alignment tasks using alternating gradient descent via multi-directional MoE as a cross-modal connector. (3) information-theoretic analysis to verify the optimality of the framework. (4) Experiments verify that M3-Jpea achieves the best performance in different modalities and tasks (including image-text retrieval, audio-text retrieval, image classification, and visual question answering) while maintaining computational efficiency. The experimental results highlight the model's ability to generalize to unknown datasets and domains.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> doesn't compare with other lightweight cross-modal methods\n\nWe include such baselines on the R@1 where M3-Jepa still outperforms:\n| Model          | # Param | Flickr30K |      | COCO  |      |\n| -------------- | ------------ | --------- | ---- | ----- | ---- |\n|                |              | T → I     | I→T  | T → I | I→T  |\n| MobileCLIP [1] | <30.7M       | 67.7      | 85.9 | 40.4  | 58.7 |\n| TinyCLIP [2]   | 63M+31M      | 66.0      | 84.9 | 38.5  | 56.9 |\n| M3-Jepa        | 140M         | 97.8      | 97.8 | 87.9  | 89.7 |\n> how MoE disentangles shared and modality-specific information\n\nMoE implements information disentanglement through **differentiable routing based on latent factors** (Section 2.2). Formally, for modality embeddings $z_m^c$ with modality encoding $E_m$, the expert selection probability is:\n$$p(E_k|z_m) = \\text{Softmax}(W_r[z_m \\oplus E_c])$$\nwhere $W_r$ and $E_m$ are learnable matrices. This allows:\n- **Modality-specific paths**: Experts specialize in processing unique modality patterns $E_m$\n- **Shared representation**: Projection matrix $W_r$ creates a common subspace for cross-modal alignment\n\n> rigor of theoretical derivation (formula 9 to 10):\n\nWe appologize that our original proof lacks some details; here we provide a more completed version. For paired modalities $(I,T)$, the total loss can be derived from the information bottleneck (Eq. 10) \n$$\\mathcal{L} =-I(I;T) + \\lambda \\big(H(T|I) + H(I|T)\\big)$$  a trade-off requires balancing:  \n- **Compression**: Maxmize $\\mathcal{I}(I;T)$ (redundancy reduction)\n- **Predictiveness**: Minimize $\\mathcal{H}(T|I) + \\mathcal{H}(I|T)$ (uncertainty reduction)\n\nCOMPLETER [3] proves that the two implemented types of losses\n- $\\mathcal{L}_{CL}$ maximizes $\\mathcal{I}$ by separating negatives  \n- $\\mathcal{L}_{Pred}$ minimizes $\\mathcal{H}$ by regression\n\n> inadequate experimental evidence for the weight parameter lambda\n> theory of senstivity for alpha\n\nSorry for the redundant notations: actually, **$\\lambda$ in Eq.10 (theoretically derived) is equivalent with $\\alpha$ in Eq.3, Eq.4 and Fig.4 (emphiricially optimally obtained).** Re-derivation from Eq. 9 to Eq. 10 further validates the emphiricial optimum ($\\alpha=0.5$): \n- Loss in Eq.10 mirrors the **free energy minimization** principle $$F = U - TS$$ in which $\\mathcal{I}(I;T)$ corresponds to the iternal energy $U$, and $\\mathcal{H}(T|I) + \\mathcal{H}(I|T)$ corresponds to the entropy $TS$. The critical temperature $T_c$ where energy/entropy balance occurs corresponds to α=0.5.\n- For two consecutive steps with step large enough, a reasonable assumption is that $\\mathcal{I}$ converge to each other, then\n$$\\mathcal{L} → \\frac{1}{2} (\\mathcal{L}(i → t) + \\mathcal{L}(i → t)) \\rightarrow \\frac{1}{2} \\big( -\\mathcal{I}(I;T) +\\mathcal{H}(T|I)   -\\mathcal{I}(T; I) + \\mathcal{H}(I|T)  \\big) = -\\mathcal{I}(I;T) + \\frac{1}{2} \\big(\\mathcal{H}(T|I) + \\mathcal{H}(I|T)\\big)$$\nwhich indicates the optimal $\\lambda = \\alpha = 0.5$\n\n> convegence of AGD\n\nSorry here we clarify here:\n\n- Key Assumptions: **Modality Independence during Alternation**. At each step $t$, we update only one alignment direction (e.g., $\\theta_{I→T}$ or $\\theta_{T→I}$), with the other fixed. This is standard in alternative optimization (e.g., EM).\n\n- Proof Sketch for Optimality:  Eq. 9's alternation between $\\mathcal{L}(I→T)$ and $\\mathcal{L}(T→I)$ is equivalent to a **block coordinate descent** on Eq. 10, and **convergence theorem of alternating optimization** [4][5] guarantees convergence to a local optimum if each subproblem (Eq. 9) is optimally solved.\n\n> hyperparameter for reproducing LanguageBind\n\nWe use the model downloaded from their website and strictly followed their inference parameters.\n\n> whether different tasks require differentiated alpha\n\nalpha=0.5 is an optimal choice (emphirically) for most tasks except for VQA, where one image correlates with multiple questions. The occurance of  CL loss is different with pred loss, therefore alpha changes by replacing batch-average with batch-sum. \n\n> ablation of expert number and routing strategy\n\nWe have n and k's ablations. Please see response to Reviwer JJf4 for n. R@1 results of k (default 4) is:\n\n| $k$  | Flickr30K |      | COCO  |      |\n| ---- | --------- | ---- | ----- | ---- |\n|      | T → I     | I→T  | T → I | I→T  |\n| 2    | 95.5      | 96.0 | 82.0  | 85.0 |\n| 6    | 97.0      | 97.5 | 86.5  | 88.0 |\n| 4    | 97.8      | 97.8 | 87.9  | 89.7 |\n\n> Further Literature Discussion; key details; types\n\nWe appreciate insightful suggestions of related works and will include them in CR. We will also move more key implementation details from appendix to the mainpart, and fix the typos. \n\n------\nReference:\n\n[1] Vasu & Anasosalu, CVPR2024\n\n[2] Wu & Peng, ICCV2023\n\n[3] Lin, Gou, et al. CVPR2021\n\n[4] Jain & Kar. Non-convex optimization for machine learning. 2017\n\n[5] Akbari, Hassan, et al. Alternating gradient descent and mixture-of-experts for integrated multimodal perception. NIPS2023", "claims_and_evidence": "Overall, the paper presents convincing evidence in most experimental results. However, the supporting evidence for the following claims is somewhat insufficient:\n（1）Cross-modal alignment performance: The paper doesn't compare with other lightweight cross-modal methods, which may affect the comprehensiveness of the parameter advantage. More visualization results are needed to strengthen the persuasiveness of the cross-modal alignment effect.\n（2）Multi-directional MoE module design: While ablation studies show the superiority of multi-directional MoE over simple MLP or non-alternating optimization strategies, there's no formal explanation of how MoE disentangles shared and modality-specific information. This part lacks systematic theoretical analysis and multi-dimensional experimental comparisons, making the connection between theory and practice less robust.\n（3）Rigor of theoretical derivation: The paper derives the alternating optimization strategy from an information-theoretic perspective (see formulas 9 to 11), showing that modality alignment can be achieved by maximizing mutual information and minimizing conditional entropy. However, the derivation is somewhat brief, with key assumptions and proof steps not fully elaborated. This makes it hard to be entirely convinced of the method's theoretical optimality. Supplementing with detailed proofs and clear assumptions would better validate this theoretical claim.", "methods_and_evaluation_criteria": "The experiments selected well-known benchmark datasets such as COCO, Flickr30K, ImageNet-1K, Clotho, Audiocaps, VQA, and NLVR-2, covering a wide range of tasks including image-text retrieval, audio-text retrieval, image classification, and visual question answering. This comprehensive approach allows for a thorough evaluation of the model's performance across different scenarios. The evaluation metrics used, such as Recall, Accuracy, and F1 score, are standard in the field and facilitate direct comparison with existing methods. Overall, the experimental setup is well-reasoned and highly relevant to current multimodal alignment issues and their associated applications.", "theoretical_claims": "I've reviewed the theoretical proof section, particularly the part in section three about proving the alternating optimization strategy's effectiveness using mutual information maximization and conditional entropy minimization (e.g., formulas 9 - 11). While the overall reasoning is sound, I found these issues:\n（1）Insufficient mathematical derivation details are provided for the transformation from formula 9 to 10, and inadequate experimental evidence is given for the weight parameter λ.\n（2）No clear explanation is offered on how alternating optimization of uni-directional tasks (like i→t and t→i) ensures convergence to a global optimal solution. There's no citation or proof of AGD's convergence theorem in multi-objective optimization, nor is there discussion of potential conflicts in the optimization process (such as inconsistent gradient directions across tasks).\n（3）The sensitivity analysis for parameter α lacks a theoretical explanation for why α=0.5 is the optimal choice to balance mutual information maximization and conditional entropy minimization. Relying solely on experimental observation isn't rigorous enough.", "experimental_designs_or_analyses": "While the overall design is reasonable, further verification or analysis is needed in these areas:\n（1）In the zero - shot generalization evaluation of the audio - text retrieval task, the hyperparameter alignment process for reproducing LanguageBind's results isn't detailed, raising concerns about the consistency of the reproduction method.\n（2）The sensitivity analysis of loss weight α indicates that α=0.5 is optimal, yet there's no exploration of whether different tasks require differentiated weights (for instance, whether audio tasks need a higher contrastive loss weight). Also, there's a lack of ablation experiments on the number of MoE experts and routing strategies (such as dynamic top - k) to verify the impact of sparsity.", "relation_to_broader_scientific_literature": "The paper's key contributions and their relation to existing literature are as follows:\n（1）Extension of the JEPA Framework: The paper builds on the Joint Embedding Predictive Architecture (JEPA) by Dawid & LeCun (2023) and Assran et al. (2023), expanding it to multimodal scenarios. Unlike the original JEPA, which focuses on single - modality latent space prediction, M3-Jepa uses a multi - directional MoE connector for cross - modal latent space alignment. This approach aligns with recent trends in multimodal representation learning, such as ImageBind and Flamingo. Compared to methods like BLIP-2 (Li et al., 2023b) that rely on heavy cross - modal fusion modules, M3-Jepa employs a lightweight MoE and alternating optimization strategy.\n（2）Application of Mixture of Experts (MoE): The paper leverages the sparsification advantages of MoE in cross - modal tasks, similar to MoE-LLaVA (Lin et al., 2024) and Pathways (2023). However, M3-Jepa further explores multi - directional routing mechanisms, such as dynamically selecting experts based on input modalities, contrasting with traditional single - directional MoE approaches like the Switch Transformer.\n（3）Alternating Optimization Strategy (AGD): The idea of alternating training for multi - task learning in M3-Jepa is similar to IMP (Akbari et al., 2023) and PolyViT (Likhosherstov et al., 2021). Yet, M3-Jepa applies this strategy to cross - modal bidirectional alignment rather than multi - resolution or multi - task learning.\n（4）Information - Theoretic Alignment Objective: The objective of maximizing mutual information I(I; T) in M3-Jepa is consistent with the contrastive learning framework of CLIP (Radford et al., 2021). But M3-Jepa more explicitly disentangles shared and modality - specific information by introducing the minimization of conditional entropy H(T∣I) and H(I∣T).", "essential_references_not_discussed": "(1) For multi-directional MoE and sparse expert models, though the paper cites some relevant works, recent findings on Switch Transformer or MoE-LLaVA offer new insights into multimodal alignment and cross-modal fusion. Discussing these (e.g., progress in applying MoE strategies in large - scale multimodal models) would enhance the comprehensiveness of this paper's positioning.\n(2) From the information - theoretic perspective, the paper justifies alternating optimization using mutual information and conditional entropy, but similar ideas have been discussed in works like Dawid & LeCun (2023). Supplementing the literature comparison would help clarify the theoretical innovation and limitations of this paper's method.", "other_strengths_and_weaknesses": "Strengths:\nInnovation: The paper combines the JEPA framework with multi-directional MoE, presenting a novel approach for cross-modal alignment in latent space. This combination helps solve information bias issues in traditional unified encoders.\nComprehensive Experimental Validation: The authors conduct extensive experiments across multiple tasks, including image-text retrieval, audio-text retrieval, image classification, and visual question answering. Ablation studies and sensitivity analyses demonstrate the effectiveness of each component, highlighting the method's practicality.\nWeaknesses:\nInsufficient Theoretical Proof: The explanation of the alternating optimization strategy from an information-theoretic perspective lacks detailed mathematical derivations and assumptions, reducing the persuasiveness of the theoretical argument.\nInadequate Detail Description: Key experimental details (e.g., data preprocessing, training strategies, hyperparameter configurations) are mainly described in the appendix. Brief descriptions in the main text may hinder readers' understanding of the experimental design and reproducibility.\nFurther Literature Discussion Needed: The paper could benefit from a more in-depth discussion of recent advances in multi-directional MoE and sparse activation strategies in multimodal learning. Additional comparisons with relevant literature would help better position the paper's contributions.", "other_comments_or_suggestions": "The paper contains some minor errors that need polishing. For instance, in formula 4, the learnable matrix W2 is referred to as W3 and W4 in the explanation, which is unclear. In section 5.1, the description \"d Firstly\" in the sentence \"However, as model scale continues to increase, several potential challenges emerge:d Firstly,\" is ambiguous. Additionally, there is no reference to Figure 5. It is recommended to carefully correct these grammatical errors in the paper.", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "alX7ivNc8J", "reviewer": "Reviewer_XJZs", "summary": "The paper presents M3-Jepa, a multimodal alignment framework leveraging the Joint-Embedding Predictive Architecture (JEPA) and Mixture-of-Experts (MoE) for aligning diverse modalities in a shared latent space. It introduces a multi-directional MoE predictor that alternates between unidirectional alignment tasks using contrastive and predictive learning. The primary goal is to enhance multimodal alignment efficiency, reduce information bias, and improve generalization to unseen modalities and tasks. The study demonstrates state-of-the-art (SoTA) performance in multimodal tasks, including vision-language retrieval, audio-text retrieval, image classification, and visual question answering (VQA).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's insightful comments.\n\n> additional insights into failure cases, model robustness across different domains\n> How does M3-Jepa handle modality imbalances (e.g., missing or weakly correlated modalities in training)?\n\nWe found primary failure cases on the VQA tasks, i.e. when there is more than one modality in the input or output. Currently M3-JEPA is worse than BEiT-3 and is the second-best. The reason is, currently we only use simple concatenation between different modality encodings.\n\nA detailed example:\n-The image is a horse before a 5-step stairs to a building.\n-M3-JEPA correctly answers the number and color of the horse, but fails to answer the number of steps.\n-The reason might be the simple concatenation of V and Q embeddings before feeding to MoE. MoE might only be able to focus on the main part of image (horse) while hard to extract the minor details (stair). \n-Smart and unified positional embedding between Q and A, or cross-modality attention inside MoE may alleviate the problem.\n\n\n> does not provide direct comparisons of training costs with other SoTA models\n> additional insights into ... real-world latency analysis would further strengthen the study.\n\nWe apologize that our current statement is somewhat confusing and here we further clarify it. Although M3-JEPA has relatively heavy modality encoders (which impacts inference), the training cost is relatively low since only the lightweight MoE predictor and N=3 (LORA rank=64) encoder layers are needed64, reducing memory overhead. We have elaborated the comparison of \\# Trainable Params in Table 1. To further compare the training cost in Table 1, we will also explicitly list direct comparisons of FLOPs, GPU hours, and memory usage.\n\nFor the inference cost, we conduct further **Inference Efficiency** analysis to address the reviewer's concerns:\n- Pre-Computation vs. Real-Time:\n  - For **static datasets** (e.g., COCO), pre-computed embeddings enable **0.02s retrieval** (vs. CLIP’s 0.16s), as shown in Section 4.6.\n  - For **dynamic inputs** (user-provided images/text), the latency is dominated by encoder inference (~0.1s/image for DINOv2, ~0.3s/text for LLaMA-3-8B). We will clarify this distinction.\nWe will also include the inference efficiency analysis in the Discussion.\n\n> one limitation is that the alternative optimization approach for training MoE is not compared with other optimization techniques\n> does not explicitly compare the information-theoretic advantages of JEPA vs. traditional generative models\n\nWe acknowledge the importance of exploring other optimization strategies, and we are sorry about the possible confusion due to the different notations between \"AGD\", and \"ALT\" used in the ablation (Table 5).\n- Our ablation experiments (**Section 4.6, Table 5**) explicitly demonstrate the necessity of AGD. Removing AGD (\"ALT\" in Table 5) leads to a significant performance drop:  \n   - **Image→Text Retrieval (COCO)**: R@1 declines from **88.1% → 68.2%**  \n   - **Text→Image Retrieval (COCO)**: R@1 drops from **90.1% → 74.2%**  \n   This empirically validates that AGD is critical for bidirectional alignment, as the standard optimization (randomly mixing task samples) fails to capture mutual dependencies between modalities.\n- **Theoretical Grounding**:The information-theoretic analysis (**Section 3**) demonstrates that AGD inherently:  \n   - Maximizes mutual information $ I(I; T) $  \n   - Minimizes conditional entropy $ H(T|I) $ and $ H(I|T) $  \n   This aligns with the theoretical framework of JEPA [1], where alternating optimization ensures the latent space retains **shared cross-modal information** while filtering modality-specific noise. Competing strategies like joint optimization risk collapsing the latent space or overfitting to a single modality direction.\n- Joint optimization would require simultaneous updates to conflicting parameter subsets (e.g., image→text and text→image predictors), leading to **gradient interference**. AGD decouples these updates, mirroring the success of alternating training in multi-task learning [2,3].\n\n- **Future Directions**: While AGD is currently effective, we agree that exploring alternatives could be informable and will address it in future work.\n\n> the impact of dataset biases on model performance is not addressed.\n\nWe will add a discussion paragraph about the zero-shot text-audio retrieval experiment. In short, we train the model on wavtext5k and freesound, while test on cloth and AudioCaps. We find the fraction of freesound over wavtext5k is extremely sensitive to the final result, due to the sparsity of audio data, and their different sampling intervals and lengths.\n\n---\n\n**References:**  \n[1] Dawid & LeCun (2023). *Introduction to Latent Variable Energy-Based Models*.  \n[2] Akbari et al. (2023). *Alternating Gradient Descent and Mixture-of-Experts for Integrated Multimodal Perception*.  \n[3] Likhosherstov et al. (2021). *Alternating Mirror Descent for Constrained Min-Max Games*.", "claims_and_evidence": "The paper's claims are largely supported by experimental results. It argues that M3-Jepa provides better multimodal alignment and efficiency than existing methods, with empirical evidence showing superior retrieval accuracy and classification performance. The MoE predictor's effectiveness is demonstrated through ablation studies, highlighting its role in improving cross-modal representation learning. However, the paper does not deeply analyze the failure cases, making it difficult to understand where the model struggles. Additionally, while the study claims computational efficiency, it does not provide direct comparisons of training costs with other SoTA models, leaving some uncertainty about real-world deployment feasibility.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are generally well-founded. M3-Jepa employs pretrained unimodal encoders and a multimodal MoE-based connector, which are evaluated through benchmark datasets such as COCO, Flickr30K, Clotho, Audiocaps, and ImageNet-1K. The use of contrastive and predictive losses in training aligns with current best practices in multimodal learning. However, one limitation is that the alternative optimization approach for training MoE is not compared with other optimization techniques, making it unclear if this is the most efficient strategy.", "theoretical_claims": "Theoretical claims in the paper include an information-theoretic justification for M3-Jepa, showing that it maximizes mutual information while minimizing conditional entropy for improved multimodal alignment. These claims are sound and mathematically derived, but the paper does not explicitly compare the information-theoretic advantages of JEPA vs. traditional generative models. Further theoretical exploration could strengthen its justification.", "experimental_designs_or_analyses": "The experimental design is robust, covering vision-language and audio-language retrieval, classification, and VQA tasks. M3-Jepa consistently outperforms prior methods in retrieval accuracy and classification metrics, demonstrating its generalization capabilities. The ablation studies confirm the importance of the MoE structure, alternative optimization, and contrastive/predictive loss balancing. However, the study does not analyze failure cases, and the impact of dataset biases on model performance is not addressed.", "relation_to_broader_scientific_literature": "M3-Jepa contributes to the broader multimodal learning literature by enhancing alignment efficiency through JEPA and MoE-based predictive modeling. It builds upon previous self-supervised multimodal methods while introducing a novel multi-directional MoE predictor. Compared to Flamingo, BLIP-2, and MoE-LLaVA, it offers a lighter-weight alignment method with improved retrieval and classification accuracy.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "A potential area for improvement is a more detailed failure analysis, particularly regarding cases where M3-Jepa underperforms in complex reasoning tasks or domain shifts. Additionally, a comparative efficiency analysis against computationally intensive SoTA models would help validate its claims of efficiency.", "questions_for_authors": "How does M3-Jepa perform in real-world deployment scenarios, particularly in terms of inference latency on edge devices?\n\nDoes the alternative optimization approach provide significant advantages over standard MoE training?.\n\nWhat are the common failure cases, and how does the model handle ambiguous multimodal inputs?\n\nWould incorporating temporal information (e.g., video-based tasks) further improve multimodal alignment?\n\nHow does M3-Jepa handle modality imbalances (e.g., missing or weakly correlated modalities in training)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ovqrzBdzIo", "reviewer": "Reviewer_9vZ6", "summary": "This paper proposes a novel JEPA-based framework, M3-JEPA, for multimodal data (image, text, audio, etc.). Specifically, M3-JEPA follows the encoder-predictor architecture in JEPA, but replaced the target embedding with other modalities. M3-JEPA designed a novel predictor based on multi-directional MoE, which consists of projection layers and learnable task encodings to guide the cross embedding prediction. The proposed method is trained with contrastive loss and predictive L2 loss. For experiments, the authors conducts several experiments to effectively answered 8 key questions, with good results on image-text retrieval, audio-text retrieval, image classification, and VQA tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s insightful comments. Below we clarify our methodology and provide additional context to address the concerns.\n\n> RQ1: even though the trainable parameters in M3-JEPA is small, it needs heavy image/text encoders. While other baselines, such as BLIP-2 VIT-g, does not include extra parameters. Therefore, reporting only the number of trainable parameters in Table 1 might not be a fair and convincing comparison with other baselines.\n> RQ8: M3-JEPA uses heavy encoders for both images and text (DINOv2 and LLama-8B). During training, these encoders need to be partially fine-tuned by LoRA, which could incur high computational cost. In addition, the author compared the retrieval efficiency with CLIP, but with the M3-JEPA features pre-computed. This could lead to (1) unfair comparison, (2) the user-provided image/prompt might not exactly match with the pre-computed embeddings, which means the DINOv2 and LLama8B encoders are still needed during inference. Therefore, I would kindly cast doubt on the claim on training and inference efficiency.\n\nWe apologize that our current statement is somewhat confusing and does not differentiate the training and inference costs clearly. Although M3-JEPA has relatively heavy modality encoders (which impacts inference), the training cost is relatively low since only the lightweight MoE predictor and N=3 (LORA rank=64) encoder layers are needed64, reducing memory overhead. We have elaborated the comparison of \\# Trainable Params in Table 1. We conduct further **Inference Efficiency** analysis to address the reviewer's concerns:\n- Pre-Computation vs. Real-Time:\n  - For **static datasets** (e.g., COCO), pre-computed embeddings enable **0.02s retrieval** (vs. CLIP’s 0.16s), as shown in Section 4.6.\n  - For **dynamic inputs** (user-provided images/text), the latency is dominated by encoder inference (~0.1s/image for DINOv2, ~0.3s/text for LLaMA-3-8B). We will clarify this distinction.\n- Mitigation Strategy:\n  - For latency-sensitive applications, we recommend **caching frequent queries** (e.g., common prompts in retrieval systems).\n  - The MoE connector’s **0.02s latency** still outperforms cross-attention baselines (e.g., BLIP-2’s Q-Former: ~0.05s).\n\nFurthermore, we need to clarify that M3-JEPA primarily works on multimodal alignment tasks and is not a generation-based method. There is no sample-level prompt needed, therefore the pre-computation of textual input can be done before a specific task is conducted online.\n\nNevertheless, we also agree with the reviewer that this comparison needs to be more clear and comprehensive. To remedy this, we will make the modification to the camera-ready version:\n\n- In Table 1, also exhibit the column of parameter of encoders.\n- Add a **total parameter/FLOPS table** comparing M3-Jepa and baselines (including encoder costs).\n- Clarify **use-case assumptions** (pre-computation for static datasets vs. real-time constraints).\n- Add a paragraph to discuss the  **trade-offs** between connector efficiency and encoder overhead in Section 6.\n\n\n> For the ablation experiment on connector architecture, is the MLP using comparable number of trainable parameters?\n\nYes, the MLP in the ablation study uses a comparable number of trainable parameters to ensure a fair comparison. We are sorry that we did not make this point clear. We will add the detailed number of MLP parameters in Table 5 of the CR version.", "claims_and_evidence": "The authors addresses 8 key questions as mentioned at the beginning of Section 4. Most of the claims are validated effectively through experiments. However, I have a concern about the first and last statement about training and inference efficiency. \n\n- RQ1: even though the trainable parameters in M3-JEPA is small, it needs heavy image/text encoders. While other baselines, such as BLIP-2 VIT-g, does not include extra parameters. Therefore, reporting only the number of trainable parameters in Table 1 might not be a fair and convincing comparison with other baselines.\n\n- RQ8: M3-JEPA uses heavy encoders for both images and text (DINOv2 and LLama-8B). During training, these encoders need to be partially fine-tuned by LoRA, which could incur high computational cost. In addition, the author compared the retrieval efficiency with CLIP, but with the M3-JEPA features pre-computed. This could lead to (1) unfair comparison, (2) the user-provided image/prompt might not exactly match with the pre-computed embeddings, which means the DINOv2 and LLama8B encoders are still needed during inference. Therefore, I would  kindly cast doubt on the claim on training and inference efficiency.", "methods_and_evaluation_criteria": "The proposed method and evaluation make sense.", "theoretical_claims": "The theoretical claims in the paper looks sound to me.", "experimental_designs_or_analyses": "The paper contains detailed explanation of the experiment setup. The experiment designs look sound.", "relation_to_broader_scientific_literature": "The paper is related to creating a unified embedding space for multimodal data. Prior works including ImageBind, LanguageBind, etc, has shown that creating a shared embedding space across different modalities achieves competitive results on modality-specific tasks.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "- For the ablation experiment on connector architecture, is the MLP using comparable number of trainable parameters?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JJzEuccOnA", "reviewer": "Reviewer_JJf4", "summary": "- The work proposed a method that aligns multimodal information in the latent space with a MoE setup, treating single-modality encoder as experts, and is optimised by single modality alignment loss with alternating input/output modality each iteration. To avoid representation collapse over alternative training, the author included mutual information loss as a regularisation term.\n- The conceptual ideal is good — an MoE encoder is trained such that any modality latent x can be projected to latent y, and if the assumption is true, it can enable flexible handling of any unknown modality.\n- Results demonstrate that the proposed method outperforms baselines across a variety of multimodal tasks and modalities.\n\n## update after rebuttal\n I would like to keep my scores, as the authors address most of my concerns.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the feedback. \n\n> limited evidence to support ... the limitation of prior works\n\nIn the CR version, we will add:\n- why self-supervise learning and energy-based model have less information bias than supervised learning and probabilistic model (refer to Section 2.4, Dawid & LeCun (2023))\n- why aligning on the latent space is less biased than aligning on the token space (Section 4, Dawid & LeCun (2023))\n\n> more recent work using Llama3-8B should be included as the baseline\n\nM3-JEPA primarily works on multimodal alignment and is not generation-based, while LLama3-based MM studies  (e.g. Cambrian, MiniCPM, LLaVA, etc) mainly report generation metrics. So it is difficult to include them as baselines right now. We consider testing them for typical cases. \n\n> The terms ‘input’ and ‘output’ corresponding to Zi, Zt are very confusing. ...what about m1 and m2 for modality 1 and modality 2?\n\nWe admit that the terms `(I, T)` were denoted for image-text tasks but are intended to generalize to any modality pair `(m1, m2)`. We will:\n- Replace `(I, T)` with `(m1, m2)` in all equations and figures.\n- Add a notation table to unify symbols (e.g., `z_m1`, `z_m2` for latent representations, `L_{m1→m2}` for task-specific losses).\n- Revise Figures 1–3 to reflect this generalization.\n\n> The task losses in section 2 and alignment losses in section 3 use identical notation (Li→t and Lt→i)\n\n- We will explicitly distinguish them in Section 2:\n  - **Task losses**: `L_pred` (Eq. 5) and `L_CL` (Eq. 6) are *component-level* losses for individual tasks.\n  - **Alignment loss**: `L_align` (Eq. 10) is the *meta-objective* combining mutual information and conditional entropy.\n- Add a subsection (2.5) to clarify their hierarchical relationship.\n\n> it is limited by what the modality-specific encoders (expert) been trained with, this leads to marginal advantage over existing work \n\nM3-Jepa’s flexibility stems from:\n- **Scalable Architecture**: MoE is agnostic to encoder choices. Any modality with a pretrained encoder (e.g., LiDAR) can be integrated, as demonstrated by audio experiments (Section 4.3).\n- **Empirical Evidence**: Table 2 shows strong zero-shot performance on unseen audio-text tasks.\n- **Further Revision**: for camera-ready, we will discuss extending to novel modalities (e.g., tactile sensors).\n\n> lack of experiment verifying that the proposed losses\n\nThe first row of Table 5 experiments on this concern. By replacing MoE with an MLP, while keeping the loss unchanged, the performance degrades on COCO, compared to the formal M3-JEPA. We are sorry that Table 5 is not clear enough.\n\n> Table 1 is quite saturated...at least clarify the embeddings of each baseline.\n\nWe will explicitly clarify this in Table 1:\n- **Dual-encoder models** (CLIP, ALIGN, FILIP) use ViT-B/32 + text transformers.\n- **Fusion-encoder models** (UNITER, OSCAR) typically use ResNet-50 + BERT.\n- **BLIP-2** and **BEiT-3** leverage ViT-L + OPT-2.7B.\n\n> the classification label is very well-studied.\n\nWe thank the reviewer for highlighting prior work on this. We will cite those prior works and add them in Table 3. However, unlike prior studies (e.g., CLIP’s linear probe), M3-Jepa treats labels as a *modality* and aligns them *in the latent space* without supervised fine-tuning.\n\n> Is the MoE essentially learning to ‘map an unknown modility latent to the target modilty latent space’?\n\nMoE in M3-Jepa serves as a ​**multi-directional alignment module** that learns to harmonize *shared semantic information* across modalities while preserving *modality-specific details*. This is achieved through **Multi-Directional Tasks** and **Information Decomposition**.\n\n> Ablation of MoE and Loss Components: \n\nThese studies are conducted in Section 4.6.\n- **MoE vs. MLP**: In the first row of Table 5, replacing MoE with a standard MLP caused a significant performance drop\n- **Loss Combination**: Figure 4 (sensitivity analysis of loss weight α) shows that the best performance occurs at α=0.5 (equal weighting of predictive and contrastive losses). Using only contrastive loss (α=0) or only predictive loss (α=1) led to a 15–20% relative degradation in R@1. \n\n> Scaling and Number of Experts\n\nThe default setting is number of expert **$n$=12**. We expeirment on different number of experts (n = 2, 8, 12) during the series of VQA experiements. Below are their accuracies on the validation set accuracy of VQAv2:\n\n $n$  | VQAv2(val) |\n| ---- | --------- |\n| 2    | 55.15     |\n| 8    | 59.84     |\n| 12   | 68.03     |\nThe number of expert contributes positively to the VQA performance. \n\n> Failure Cases / Limitations\n\nWe found typlical failure cases on the VQA tasks, i.e. when there is more than one modality in the input or output. M3-JEPA is worse than BEiT-3 and is the second-best. The reason is we only use simple concatenation between different modality encodings, further solutions (e.g. unified positional embedding across V and Q) may strengthen M3-JEPA's performance.", "claims_and_evidence": "- The claim that to alternatively optimise each single-modility encoder (i.e. expert) is better than to simultaneously optimise single/unified encoders as existing work did, is supported by Table 5.\n- The claim on the proposed method generalising well to domains and modalities is supported by the audio results in Table 2.\n- The claim on training- and inference-efficiency is partially supported by comparing the trainable parameter in Table 1.\n- The motivation is not strong, given limited evidence to support the argument on the limitation of prior works, i.e. optimising alignment with single or unified encoders may lead to information bias.", "methods_and_evaluation_criteria": "- Using a mixture-of-experts is appropriate given that different modality pairs and tasks may require different mappings. The MoE allows specialization and presumably helps maximize mutual information more effectively.\n- The author argued that a different framework is needed to resolve ‘representataion collapse’, then proposed to solve with the mutual information loss, but the ablation experiment on this seems missing.\n- The evaluation criteria encompass a range of tasks to reflect general multimodal alignment quality. However, given that the author used Llama3-8B as a text encoder, more recent work using Llama3-8B should be included as the baseline (e.g. Cambrian, MiniCPM, LLaVA, LLaVA-Next, etc).", "theoretical_claims": "- The method description includes multiple misused and un-introduced symbols. The notation across figures (1-3) and formulas is inconsistent and hard to follow.\n    - The terms ‘input’ and ‘output’ corresponding to $Zi$, $Zt$ are very confusing. I suppose i and t represent the image and text, but you want generalised terms; what about m1 and m2 for modality 1 and modality 2?\n    - The task losses in section 2 and alignment losses in section 3 use identical notation ($L_{i \\to t}$ and $L_{t \\to i}$), but to my understanding, they are different, so they should be expressed differently.\n- The method is still not flexible to any modality as it is limited by what the modality-specific encoders (expert) been trained with, this leads to marginal advantage over existing work either train single or unified modality encoders without MoE.", "experimental_designs_or_analyses": "- There is a lack of experiment verifying that the proposed losses (mutual information and conditional entropies) would also work in the usual setting, i.e. separate encoder without MoE.\n- Table 1 is quite saturated, and many weak baselines are not using strong embeddings as this work (i.e. use LLama3-8B, Dinov2-Large and LanguageBind (Zhu et al., 2023) to encode text, image and audio modalities). It would be helpful to at least clarify the embeddings of each baseline.\n- Table 3 aims to verify RQ4 - can the method handle less-studied information, but the classification label is very well-studied, and many prior studies already verified text-to-image or image-to-text models learn representations that adapt well to classification tasks.", "relation_to_broader_scientific_literature": "- It references JEPA (Joint Embedding Predictive Architecture) by LeCun (Dawid & LeCun 2023, Assran et al. 2023), acknowledging the conceptual foundation.\n- The authors cited recent works on multimodal alignment, such as BLIP-2 (Li et al., 2023b), and included it in Table 1 as an important baseline.", "essential_references_not_discussed": "Given that the author used Llama3-8B as a text encoder, more recent work using Llama3-8B should be included as the baseline (e.g. Cambrian, MiniCPM, LLaVA, LLaVA-Next, etc.).", "other_strengths_and_weaknesses": "- The abstract and Figure 1 are hard to follow (over complicated). Is the MoE essentially learning to ‘map an unknown modility latent to the target modilty latent space’?\n- The overall writing is not well structured and is confusing from time to time. The ideal is simple and could be introduced with simple terms in general.", "other_comments_or_suggestions": "- l.075-076: ‘optimizing by alternating’ → ‘optimized by alternating’.\n- l.97: ‘input-output $(I,T)$’, ‘$e_i (·)$ and $e_t (·)$., and formular (1), these are confusing as the author keep alternating between set symble (captionised $I,T$) with invidual symble $(i,t)$, and the notation of $i,t$ are not introduced.\n- l.135-136 ‘modality encodings’ → ‘modality encoders’, and ‘task encodings’ …", "questions_for_authors": "- Ablation of MoE and Loss Components: How crucial is the mixture-of-experts design and the combination of predictive + contrastive losses to M3-Jepa’s performance? Did you run experiments with a single predictor (no MoE) or with only contrastive or only predictive loss?\n- Scaling and Number of Experts: How does the number of experts influence performance? Did you experiment with different counts of experts n in the MoE?\n- Failure Cases / Limitations: Beyond the note that MoE struggled with complex relationships, were there any particular failure cases you observed? For instance, are there specific modality pairs or tasks where M3-Jepa underperforms a simpler baseline?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Four knowledgeable reviewers went over this manuscript. The reviewers found the experiments comprehensive, covering a wide range of tasks (eSmg, XJZs), and most claims supported by experimental evidence (XJZs, 9vZ6). The reviewers raised concerns about:

1) Some parts of the experimental evidence:
    - Comparisons with lightweight cross-modal methods missing (eSmg)
    - Key experimental details to understand comparisons missing (eSmg, JJf4)
    - Ablations missing (alpha, number of moe, routing strategies, representation collapse, optimization strategies) (eSmg, XJZs)
    - Failure cases not characterized (XJZs)
    - Some recent generative models could have been used as comparison points (JJf4)
2) The theoretical analysis which appeared limited, with missing details, and not clearly connected to the practice (eSmg)
3) The training and inference efficiency claims, which did not appear well supported by evidence (XJZs, 9vZ6, JJf4)
4) The clarity of the notation (JJf4)

The authors' response addressed the vast majority of these concerns. In particular, the response included baseline comparisons, discussed the missing experimental details, added ablations, discussed failures, and shared additional theoretical details. The response also clarified the notation. The response partially addressed the efficiency concerns by providing an inference efficiency analysis along with possible mitigations. The response promised to provide additional efficiency comparisons. After discussion, the reviewers are overall satisfied the received answers and recommend to accept. The AC agrees with the reviewers' assessment and recommends to accept.
Decision: Accept (poster)