Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models
Authors: Weixin Liang
LILI YU
Liang Luo
Srini Iyer
Ning Dong
Chunting Zhou
Gargi Ghosh
Mike Lewis
Wen-tau Yih
Luke Zettlemoyer
Xi Victoria Lin
Venue: Accepted by TMLR ()

Abstract: The development of large language models (LLMs) has expanded to multi-modal systems capable of processing text, images, and speech within a unified framework. Training these models demands significantly larger datasets and computational resources compared to text-only LLMs. To address the scaling challenges, we introduce Mixture-of-Transformers (MoT), a sparse multi-modal transformer architecture that significantly reduces pretraining computational costs. MoT decouples non-embedding parameters of the model by modality -- including feed-forward networks, attention matrices, and layer normalization -- enabling modality-specific processing with global self-attention over the full input sequence. We evaluate MoT across multiple settings and model scales. In the Chameleon 7B setting (autoregressive text-and-image generation), MoT matches the dense baseline's performance using only 55.8% of the FLOPs. When extended to include speech, MoT reaches speech performance comparable to the dense baseline with only 37.2% of the FLOPs. In the Transfusion setting, where text and image are trained with different objectives, a 7B MoT model matches the image modality performance of the dense baseline with one third of the FLOPs, and a 760M MoT model outperforms a 1.4B dense baseline across key image generation metrics. System profiling further highlights MoT's practical benefits, achieving dense baseline image quality in 47.2% of the wall-clock time and text quality in 75.6% of the wall-clock time (measured on AWS p4de.24xlarge instances with NVIDIA A100 GPUs).

Published in Transactions on Machine Learning Research (04/2025)
Mixture-of-Transforress A Sparse and Scalable Architec- ture for Multi-Modal Foundation Models
Weixin Liang*
wrtkann@sstnfoor..ee
Deportment of Computer Science Stanford University
Lili Yu, Liang Luo', Srinivasan Iyer, Ning Dong, Chunting Zhou Gargi Ghosh, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer Xi Victoria Lin
victorialin@mett..com com
Meta AI
Reviewed on OpenReview: https: openreview. normm? id= NuSN6918SB
Abstract
The development large language models (LLMs) has expanded to multi-modal systems capable processing text, images, and speech within unified framework. Training these models demands significantly larger datasets and computational resources compared to text- only LLMs. To address the scaling challenges, introduce Mixture-of- Transforrrs (MoT), sparse multi-modal transformer archilecture that significantly reduces pretraining com- putational costs. MoT decouples non-embedding parameters of the model by modality including feed-forward networks, attention matrices, and layer normalization -anaaiiaatooncnaalinng mopelity-specific processing with global self-attention over the full input sequence. We evaluate MoT across multiple settings and model scales. In the Chameleon setting (autoregressive text-and-image generation), MoT matches the dense baseline's performance using only 55.8% of the FLOPs. When extended to include speech, MoT reaches speech performance comparable to the dense baseline with only 37.2% of the FLOPss In the Trans- fusion setting, where text and image are trained with different objectives, 7B MoT model matches the image modality performance of the dense baseline with one third of the FLOPs, and a 760M MoT model outperforms a L.4B dense baseline across key image generation met- rics. System profiling further highlights MoT's practical benefits, achieving dense baseline image quality in 47.2% of the wall-clock time and text quality in 75.6% of the wall-clock time (measured on AWS p4de.24xlas 24xlarge instances with NVIDIA A100 GPUs).
Wookk done during a Meta internship
Joint second authorship

---

Published in Transactions on Machine Learning Research (04/2025)
Autoregressive loss diffusion-based loss
Autoregressive loss
Autoregressive loss
Image Transformer
Speech Transformer
Text Transformer
[OO]
[EOS]
complete specimen measures 3.7-4 -44 m (12- 13 ft) in hip height. Tyrannosaurus could have exceeded sizes of 13 (43 ft)..
How tall is the dinosaur ? Answer 12
(MoT) architecture. MoT is generative model architecture de- signed to process sequences consisting of arbitrarily interleaved modalities (e.g. text, image and speech). Each modality employs a separate set of non-embedding transformer parameters including the feedfor- ward network, attention matrices and layer normalization, while global self-attention is applied to the full sequence. During training, each modality be supervised using modality-specifie losses.
Figure 1:
2

---

Published in Transactions on Machine Learning Research (04/2025)
1 Introduction
The development of foundation models has expanded to multi-modal large language models (LLMs) ca- pable of processing diverse data types such as text, images, and speech within a unified framework. Recent advancements, such as Chameleon (Chameleon Team, 2024). demonstrate the potential of early- fusion, mixed-modal models to generate diverse media types within single architecture. These models hold promise for advancing applications such as content crcation and cross-modal translation but pose significant computational challenges due to the complexity of simultaneously learning represenlations across multiple modalities.
Training carly-fusion multi-modal LLMs demands significantly larger datasets and computational resources compared to single-modality models. For example, Chameleon (Chameleon Team 2024) is trained on 9.2 trillion training tokens (including image tokens) to match LLaMA2 (Touvron et al. 2023b), ; which is trained on 2 trillion training tokens for text performance. Each modality introduces unique optimization challenges, which must be addressed concurrently within unified model. Empirically, these modalities often exhibit conflicting training dynamics in dense transformer model (Figure 15 complicating optimization and increasing computational load. Despite processing inputs as uniform tokens without modality-specifh priors, different modalities occupy distinct regions in the feature space (Figure Appendix Figure 23). indicating the inherent differences in how modalities are processed.
To address this scaling challenge, promising approach is model sparsity, such as Mixture of Experts (MoE), which enables scaling by activating only subset of model components for each input, reducing the overall computational load. In MoE, learned router in each transformer layer sparsely activates one of multiple
MLPs, allowing different experts to focus on different aspects of the data (Jacobs et al. 1991 Eigen et al.
2013 Shazeer et al. 2017 Lepikhin et al.) 2020 Fedus et al.) 2022 Jiang et al. 2024 Sukhbaatar et al. However, MoF. introduces a number of challenges: the learned router often results in imbalanced expert utilization, requiring additional baad-balancin techniques during training. Furthermore, the bi-level optimization nature of MoE complicates training dynamics, which can become unstable as model sizes scale up. Addressing these challenges in MoE remains an open area of research.
2024
In multi-modal contexts, previous work (Bao et al. 2022b Wang et al. 2022 Shen et al. 2023 Lin et al. 2024) has introduced modality-aware sparsily in the MoE layers of transformers. or furt her fine- uned modality-specifi modules on LLM backbones during post-training (Wang et al. 2023 He et al. 2024 These approaches have shown promising results, suggesting that a simple rulc-basod routing by modality outperforms the learned routing commonnl used in MoE. This success might be attributed to more stable training dynamics, avoiding the instability that arises when both experts and routers are under-trained in the early stages.
Inspired by these insights, we propose Mixture-of-TTransfrnee (MoT), a sparse multi-modal trans- former architecture that introduces modality-aware sparsity for all non-embedding transformer parameters (Figure 2p). Different from previous approaches, MoT applies modality aware sparsity across the entire transformer, rather than specific layers or modules. MoT takes an interleaved multi-modal sequence (e.g., text, image, speech) as input and dynamically applies distinct, modality-specifie parameters to each to- ken, including feed-forward networks, attention projection matrices, and layer normalization. Therefore, the MoT design yields a sparse model with the exact same computational structure and FLOP count its dense transformer counterpart.
We evaluated MoT by pretraining thirteen instances, including three 7B models, from scratch across various multi-modal settings. This comprehensive setup allowed us to assess MoT's performance in multiple ex- perimental configurations, each progressively introducing more complex training objectives and modalities. Specifically, we conducted experiments on the following multi-modal scenarios to evaluate MoT's adaptability and efficiency gains:
Autoregressive objectives for both text and images ( (Chameleon). In the Chameleon setting (Chameloon Team 2024), our 7B MoT matched the performance of a 7B dense bascline while using only 35.8% of the FLOPs as evaluated on multiple data distribution (Figure B). Results are consistent across multiple other model scales (37M, 94M, 443M, 1.5B) (Figure i Appendix Figure 24p.
3

---

Published in Transactions on Machine Learning Research (04/2025)
2. Introducing speech third modality ( (Chameleon: Teat+ Image+ When ex- tended to include discrete speech tokens as the third modality in the Chameleon setting, MoT achieves similar performance across all modalities, with even fewer (37.2%) training FLOPs re- quired for the speech modality (Figure Results are also consistent across multiple other model scales (Figure Appendix Figure 25
3. Autoregressive objectives for text and diffusion- objectives for images (Transfu- sion). In the Transfusion selting, where lext and image are trained with different objectives autoregressive for text but diffusion-based for images 760M MoT model, which utilizes half the training/ iiiniing/iffreneee finference FLOPs of the 1.4B dense baseline (Transfusion), outperforms the dense model across multiple metrics, including CLIP score and FID score for image generation, CIDEr score for image captioning, and image modality training loss (Figure 11.. A 7B MoT model matches the image performance of the dense baseline with less than one third of the FLOPs on diffusion vali- dation loss for image generation and CIDEr score for image captioning (Figure 10 Additionally, across three different model scales (163M, 760M, 1.4B) in the Transfusion setting, MoT consistently achieves substantial specdup in the image modality, outperforming the dense model by wide margin (Figure 12
To provide a deeper and more comprehensive evaluation of MoT, we extended our analysis with additional experiments to validate MoT's advantages across multiple dimensions. These experiments assessed MoT's computational efficiency,
efffiiiii reductions in wall-clock time, and effectiveness relative to other sparse architectures:
4. Wall-Clock Time Comparison Furthermore, system profiling (on AWS p4de.24x 24xlare instances with NVIDIA A100 Tensor Core GPUs) demonstrated that MoT's efficiency translates into signifi- cant reductions in wall-clock training time. Our 7B MoT matches the image performance of the 7B dense model in just 47.2% of the time, and the text performance in 75.6% of the time (Figure
Comparing MoT against Mixture-of- To validate that MoT's observed gains are not merely due to additional sparse parameters (although these additional sparse parameters do not in- crease the training/ /nnferencee FL,OPs), we incorporated a 4-expert mixture-of-expert model (MoF-4x) as additional baseline throughout the experiments. MoE-4x, which includes more sparse parameters than MoT across all experiment settings, consistently underperformed compared to MoT especially in non-text modality (image, speech). The advantage of MoT over MoE-4x is even larger when measured in wall-clock time (Figure 19h.
6. Combining the Best of Both Worlds Mixing Heterogeneous Transformers As an early proof of concept, we explored a hybrid approach that integrates sparse transformers in the MoT framework. Specifically, we adopt the MoE-4x architecture for the text transformer MoT, while preserving the original MoT architecture for image tasks. Preliminary results validate that this combination can further enhance text modality performance in both the Chameleon and Transfusion settings without compromising image generation quality (Figure 16 Figure 17
2 Method:
Mixture-of- Transformers Traauureof-Taansfoorness Architecture
2.1 Background: Foundation Models for Multi-Modal Generation
Recent advances large language models have expanded to modalitics beyond text. A key approach tok- enizes non-text data such as images and speech into discrete token sequences, and applies aulo-regressive se- quence modeling to the data similar to text-based models (Figure 2b. For example, Chameleon (Chameleon Team 2024) tokenizes images into 1,024 discrete tokens using pre-trained image tokenizer (Gafni et al.
2022 a allowing unified training across text and images. Similar methods have been applied to speech (Nguyen
et al. 2024). Alternative approaches like Transfusion (Zhou et al., 2024) use continuous image tokens and
diffusion- based training objective to improve generation of continuous modalities such as image (Section 3.4
To probe the internal representations of multi-modal foundation models, we analyzed their feature space. Results reveal clustering by modality (text, speech, image) across layers (Figure 2b. Appendix Figure 23
4

---

Published in Transactions on Machine Learning Research (04/2025)
Autoregressive Objective
Next Token Prediction
Discrete Tokens
Foundation Model for Multi-Modal Generation
T
Ta
Ta
Tx
h
1z
hrzz3
Inc04
Ts
To
Ty
Ts
Image Token
Image Token
Image Token
Image Token
Token
Token
Token
Token
Token
Token
Token
Token
Sequence Dimension: Interleaved Multi-Modal Tokens Stream
(a) Foundation models for multi-modal generation
(b) Layer 1
(c) Layer
(d) Layer 17
(0) Layer 32
Figure 2: Multi-modal foundation model architecture and feature space analysis. a, Typical multi-
modal foundation model processing interleaved text (T) and image (I) tokens (e.g., Chameleon (Chameleon)
Team 2024)). Image tokens derived from pre-trained VQGAN model, converting an image into 1,024 discrete tokens. b, Principal Component Analysis of latent feature space for Chameleon+-Speal 7B Dense model across layers 1, 5, 17, and 32. Despite the model's architecture processing all inputs as uniform discrete tokens without modality-specific priors, distinct clustering by modality (text, speech, image) is observed in the feature space. This natural clustering highlights the inherent differences between modalities, suggesting that the model might have processed them differently.
Principal component analysis (PCA) shows distinct regions for different modalities the feature space, de- spite uniform processing of inputs as discrete tokens without modality-specific priors. This natural clustering suggests inherent differences in modality processing, informing our subsequent approach.
2.2 Mixture-of-Transffrmmes Architecture: Modality-Specific Parameter Decoupling
We present Mixture-of- Transsooo-rrtttomerrss (MoT), a novel architecture designed to accelerate multi-modal pretrain- ing while reducing computational costs. MoT extends the standard transformer architecture by incorporating modality-specific weights for all non-embedding model parameters, including feed-forward networks, allen- tion matrices, and layer normalization. This approach allows the model to process different modalities more efficiently while preserving the ability to learn cross-modal interactions. Let x (II,..., σn.) be the input sequence tokens, where each z; belongs to a modality m, E {text, image, speech). typical transformer layer can be expressed as:
Attn(z, Oussn)
h = x + Layer (a)
(1)
output = h + LayerNorm (FFN(h, 0ma))
5

---

Published in Transactions on Machine Learning Research (04/2025)
(a)
AutoRegressive
Diftusion Objective
(b) Autoregressive Objective
Output Queue
Next Taken Prediction
Ti
S+
lrase
Discrete Tokens
Sequence Re- Or-deriinng Buffer
Transformers Minlure-oofTTanssommes
Mixture-of
Add & LayerNom lost
Add & SayerNormSeocch
Add & LayerNorm
Speech: FFN, Waiv, LayerNorm
FFN, Waes, LayerNorm
FFN, Wans, LayerNorm
FFN
FFN
FFN
Add & LayerNor
Add & Leye
Add & LayerNorm
Inooa
Inoos
Modality Indexing Logic
(c) Diffusion Objective
Joint Attention Shared Feature Space
Next Diffusion TimeStep
Continous Tokens
Sequence Re-Ordening Buffer
Mixture of Transformers
WaT
Wom
Text: FFN, Wors LayerNorm
FFN, WaRs LayerNorm
Vocab Embedding
Vocab Embedding
Vocabimon Embedding
l:M
lass!!
Modality Indexing Logic
1,000 diffusion timestaps
2010, I
Initialize from Gaussian Noise
Speech
Image Token
Text
Speech Token
Speech Token
Image Token
Image
Text
Token
Token
Token
Token
Token
Figure 3: Mixture-of- Transformers Mf architecture for multi-modal generative AI. a: Schematic of the sparsely activated Centuure-o--raasffomoe (MoT) architecture. For each input token, MoT activates modality-specifie weights (including feed-forward networks, attention projection matrices, and layer normal- ization), then applies self-attention across the entire sequence. T, S, and I indicate text, speech, and image tokens, respectively. b-c: Flexibility in modality representation and training objectives. Images can be rep- resented as (b) a sequence of discrete tokens trained with an autoregressive objective (Chameleon setting) or (c) a sequence of continuous tokens trained with diffusion objective (Transfusion setting). This allows integration of diverse learning tasks, such as autoregressive objectives for text and diffusion based objectives for images.

---

Published in Transactions on Machine Learning Research (04/2025)
Algorithm Mixture-of- Transformers (MoT) Computation
1: Let = (#1,.... In) be the input sequence, where 24 E R2 and m, E {text, image, speech) is the modality of D;
2: Let M = {lexl, image, speech) be the set of modalities for each modality m M do
Im (i: m:=m== m)
Indices of tokens for modality > Group tokens by modality -speliit-speccfic projections
Xm - foz i Im}
6:
Qm - W Xm: Km f- WR Xm, Vn + W Xm
7: end for
8: Q + UMEM Qm K + UmEM Km, V t- UMEM Vm
> Restore original sequence order
A + softmax QK" V
> Global self-attention
vil
10: for each modality m M do
11:
Om Wo A
Modality-spedde output projection Residual connection and layer norm Modality-speciffe feed-forward network Residual connection and layer norm
12:
Hm Xm LayerNorm attn (0
13: Fm + FFN (1lm)
14: Ym f Hm LayerNormfh (Fm)
15: end for
16: return (Ym : m E M}
> Return transformer layer outputs
In our proposed MoT, we decouple the parameters by modality while maintaining global self-attention
GlobalAttn(r, (0 J hm (mmgg...ein))))))))))
me h; + LayerNorm atto (a:)
(2)
output, = h, LayerNorm
my (FFN(h, m)))
The global self-attention mechanism operates across all modalities, capturing cross-modal relationships de- spite the modality-specifie parameter decoupling:
QKT
GlobalAttn(r, (0mm))) (softmax
V
W
Vdz
(3)
Q: = w.
K; = xz Wma,
V, V x Wi
Here, W WHy, W and W are modality-specific projection matrices, and LayerNorm and LayerNorm m. are sopality-specific layer normalization.
This approach allows MoT to adapt its processing to the specifie characteristics of each modality while maintaining unified architecture for multi-modal learning. The computation process in MoT begins with grouping input tokens by modality (Algorithm lines 3-5). Modality-specific projections are then applied for attention (line 6), followed by global self-attention across all modalities (lines 8-9). Subsequently, modality- specific output projections (line 11), layer normalization, and feed-forward networks are applied (lines 12- 13). The process concludes with the combination of outputs, incorporating residual connections and layer normalization (lines 14-16).
'Comparing works that utilize cross-attention fuse information from different modalities Alayrac ct 2022 Aicllo
eli 2023). formulation using global self-attentoon normalizes attention weights across takens of different modalities while reducing the number of layers the architecture.

---

Published in Transactions on Machine Learning Research (04/2025)
3 Experiments
3.1 Results Overview
We evaluated the Mixture-of- Transformers MristureoofTraasfoomme (MoT) architecture across three multi-modal experiment settings, each progressively incorporating more complex training objectives and modalities. For each setting, we compared MoT against two baselines: a dense transformer model and a Mixture-of Experts model with 4 experts (MoE-4x). All model implementations, built upon the dense model, maintain identical FLOPs for both training and testing, enabling direct efficiency and performance comparisons.
1. Multi-modal experiment setting with autoregressive objectives (Chameleon, Figure 4). Both modalities trained using autoregressive objectives. Images represented as 1,024 discrete tokens via a pre-trained VQ-VAE model Gaffi et al. 2022). We compared MoT's performance to baselines across training and evaluation metrics for both modalities.
2. Extended multi-modal experiment with speech modality (Chameleon: Figure 7). Extended the previous setting by incorporating speech as a third modality, represented by discrete tokens via pre-trained speech tokenizer. All modalities are trained with autoregressive objectives. This setting evaluated MoT's ability to handle an additional modality while maintaining efficiency and performance.
Beat++mmage+-peeee
3. Multi-modal experiment with modality-speciffc objectives (Transfusion, Figure 10 Ex- plored multi-objective training with text using autoregressive objectives and images using diffusion- based objectives. This experiment highlighted MoT's capacity to manage distinct training objectives for different modalities, potentially improving image generation qualily while maintaining lext gen- eration capabilities.
The following sections present detailed results for each setting: Chameleon (Section 3.2), Chameleon+Speech (Section 3.3 and Transfusion (Section 3.4). Each section provides comprehensive comparisons of MoT against the baselines across various multi-modal generative evaluation metrics. In Section 3.5 we report an ablation study demonstrating the impact of model performance when introducing modality-speciffc decou- pling to different components of transformer.

---

Published in Transactions on Machine Learning Research (04/2025)
Text: Autoregressive Objective
Image: Autoregressive Objective
Next Token Prediction
Next Taken Pred ction
Discrote Tokens
Discrate Tokens
Mixurn of Transformers
Midurn of Transtormers
Text: FFN Wors
Imager PFM, Wars Layerlorm
Image: FFN, Wow: LayerNorm
FFN, Woos LayerNorm
LayerNorm
Troca
Trosk
Figure 4: Multi-modal experiment sctting with autoregrossive objectives (Chameleon). Both text and images are trained using autoregressive objectives. Images are tokenized into 1,024 discrete tokens using a pre-trained VQ-VAB model. This setting demonstrates unified processing across modalities with a single objective function.
3.2
Performance in the Chameleon Setting: Autoregressive Objectives for Text and Image Generation
In this subsection, we evaluated the Mfxturee-ofTrrnssoomee (MoT) architecture in the Chameleon setting, where text and image modalities are trained using autoregressive objectives.
3.2.1 Experiment Setup
Data and Pre-processing. We use the same mixed-modal training data and the same text and pre- trained image tokenizers as Chammeleon Team 2024). The training data comprises roughly equal amount of text and image tokens. We evaluated the 7B model performance using validation losses on held-out sets of the Obelisc (Laurençon et al.) 2023). MS-COCO Lin et al., 2014), Flickr30k (Plummer et al., 2015), and Shutterstock datascts. More specifically, for MS-COCO and Flickr30k, we take the Karpathy test split of MS-COCO Lin et al., 2014) and the Karpathy lest split of Flickr30k (Plummer el. al. 2015 and report text-to-image to-eingge and image-to-text conditional perplexity using these two datasets.
Model Hyperparameters. We evaluated MoT across multiple model scales ranging from 37M to 7B parameters, comparing it to dense transformer and MoE-4x baselines. All models were pre-trained from scratch with controlled FLOPs for fair comparison. Table [ details the architoctural specifications and training configurations for each model scale. Model architectures were scaled progressively, with hidden dimensions increasing from 256 to 4096, and layer counts from 4 to 32. Attention heads scaled from 8 to 32, while sequence length remained constant at 4096 tokens across all scales. As model size increases, we reduce batch sizes per GPU from 12 to 2, while increasing the number of GPUs from 32 to 384. Training steps set at 160,000 for smaller models (37M to 443M) and 120,000 for larger models (880M to 7B). Total training tokens ranged from 0.168 to 0.377 trillion, with most configurations processing approximately 0.252 trillion tokens. This allowed us to examine MoT's performance across wide range of model scales and training FLOPs, providing insights into its effectiveness at different computational scales 3
Mixture-of- Implementation. For our MoE-4x baselines, we employed Expert Choice (EC) (hhou et al. 2022) routing, a state-of-the-art routing method that ensures balanced load during training by having each expert select top-k inputs based on routing weights. However, EC cannot be directly applicd to auto-regressive generation, as it violates the causal dependency between lokens in a sequence, where each token is generated based solely on the previous ones. Previous work have proposed various inference- time adjustment to ensure generation causality for MoE models trained with EC routing. For example, some recent works have explored using expert choice routers out-of-the-box token choice routers during
hattpss/www.euuttrsttcckk wwwtterstock.cc.omoooooo com/
With this setup, we on evaluating relative performance of the proposed architecture and the baseline various FLOPs budgets, rather than conducting scaling law study.

---

Published in Transactions on Machine Learning Research (04/2025)
Steps to Match Dense
120000
MoE-4x (s 0.778)
3.3
100000
Training Loss
22
Speedup
MoT (ours) (s 0.455)
80000
60000
Dense
40000
MoT (ours)
20000
MoE-4x
B
120000
120000
Training Steps
Training Step for Dense
(a) Chameleon 7B: Training Loss
(b) Training Steps Matching
Stops Norch m
Tainng 0005
"2.6% Speedup
10300
MoT (ours) (s 0.348)
McE-Ax (s 1.012)
Trainin Loss
MaT
Dense
treace
MoE- 4x Is 0.6351
MoT (ours] 0.5580
Meten
MoE-4x
Dange
8900
11.8% Speedup
MoT (ours)
MaE-4x
23000
2000
120000
Trining Steps
Training Stop Dense
Trainingg
Traiiing Stop for Dense
(c) Image Training Loss
(d) Training Steps Matching
(e) Text Training Loss
(f) 'Trining Steps Matching
4.00
6.15-
6.20
3.0
3.85
Wit ww1005ie--haaooo (5550--heeppoo O000
Do0s wit met-nne
1N07
44
DarG4 Hit eet nnt-nndounnt
Hoff
Densh Hil Ans.neddeprr
MIT
(g) Image Eval Loss: Obelisc
(h)
Image Eval Loss: COCO
Image Eval Loss: Flickr
(i)
Image Eval Loes: SSTK
WARCU caloo Lana
(29.0 www.tecapnnt
Dar5e
05 6n chichin
(3s 0% chorapoic)
(k) Text Eval Loss: Obelics
Text Eval Loes: COCO
(m) Text Eval Loss: Flickr
(n) Text Eval Loss: SSTK
Figure 5: Pre-training acceleration of MoT for 7B Chameleon multi-modal model. a, Global training loss curves. MoT reduces loss faster than dense and MoE-4x models, matching dense model's final loss at 120,000 steps in 60,000 steps. b, Step matching plot for training loss in a. MoT requires 45.5% of dense model's training steps for comparable performance. c,d, Image modality training loss and corresponding step matching plot. e,f, Text modality training loss and corresponding step matching plot. MoT particularly effective for image modality, requiring 34.8% of dense model's training steps to match final loss. Both MoT and MoF-4x outperform dense model for text modality. g-j. Image modality validation losses. k-n, Text modality validation losses. Comparison of final validation losses for all models and MoT at 55.8% training checkpoint. MoT at 55.8% training stops achieves comparable or lower validation losses than donse model's final loss, indicating 44.2% reduction in required training FLOPs. Model sizes for sparse models indicate activated parameters. All runs are FLOPs-controlled and pre-trained from scratch.
inference (Zhong et al. 2024), or training small auxiliary MLP predictors post-training for routing (Raposo
et al. 2024 Lin et al. 2024).
We evaluated all models using the same EC routing as during training, focusing exclusively on validation perplexity. This approach guarantocs an isoFLOP inference setting as the dense bascline. However, it also introduces two confounding factors. First, it may overestinate MoE-4x's validation performance, as the router access future tokens, potentially leading to information leakage. Second, may also underestimate
10

---

Published in Transactions on Machine Learning Research (04/2025)
Trentrg 1o65
Dense
0.794)
McB-3x 05770995
Rel lours]
225000
Mob-4x (s 0.496)
Moe-2x (.
NoE
NoE 0.342)
Sepp Morch
McE-4x (5-0.1811
NoT (ours) is 0.2601
(ours)
(ours)
MoE 4x
50000
MoE 3x
MoE 2x
Locoe
Trining Steps
10000
(a)
37M
Image
Training
Dense
Triining
Tranning
Loss
(b) Image Loss Matching
(c) Text Training Loss
(d) Text Loss Matching
Traniing Le3s
Deniee
MoE Is= 0.457)
Derse
Deoe
McE:2x 2x (s 0.752!
MoT [ours)
NaB.3x (5- 0.448)
MoT (ours)
MoE-3x (s 0.619)
Mot-2X
120000
McB-2x
Suep Maren
(0.531))
7500
Iours) is 0.199)
McE-3x
Not (ours) Is 0.539)
McE 4x
1000
Training Steps
3008
(e)
94M
Image
Training
Training Step
Thiniing
Traniing
Loos
(f) Image Loss Matching
g
Text Training Loss
(h) Text Loss Matching
Tanin LoGs
41
McT (ours)
MoE-2x
220000
100000
MoE-4x
MoE 0.5
Mot-2x is
Dense MoE-Ix
Crepa to Match Dende
3000E
McE: 2x (s 1.127)
Mob-cx (s 0,/331
MoF-3r (s-0.893))
MoT (ours) is 0.271!
MoT (ours) is 0.610)
McE 4x
50000
Mol-3x MoE 4x
Training Steps
100000
Lo0000
(i) 443M
Image Training
Traiingg
Loss
(i) Image Loss Matching
(k) Text Training Loss
(l) Text Loss Matching
Tailing Lone
Donse
McE 4x (S 0.745)
Dense
MoE: 2x 0.990)
MUT. tourst
McE-3x 0.6721
McT
0.830)
NoE 2X
0.613!
McF-2x
MoT toursi (s 0. 200)
McE-5x
MoT (surs) 0.68)
NuE-4x
Mcl-4x
Steps
13000
Training Steps
(m) 1.5B
Image
Training
Training Step Derose
Training Staps
Faining Step for Denuie
Loss
[n) Image Loss Matching
(o) Text Training Loss
(p) Text Loss Matching
Mol--xx 1.012]
Stuce March Derge
Moli-ix (K (.635)
Trentig Looe
2003
MoT lours) (s- 0.368)
Look Saorng
MoT (ours)
MoT (ourrs) (s 0.558)
McE-4x
MoE 4x
NOT
Stops
L000
128009
Tranning Stots
Training Stop for Denge
Triiing Steps
Toining Stop for Donse
(q) 7B Image Training Loss
(r) Image Loss Matching
(s) Text Training Loss
(t) Text Loss Matching
Figure 6: -peelic pre-training loss and step matching plots across model scales (Chameleon setting). MoT shows consistent, significant speedup in image modality across all scales (37M, 94M, 443M, 1.5B, 7B), outperforming dense and MoF.-4x models. MoE-4x exhibits diminishing gains in image modality as scale increases, with advantages disappearing at 7B. In text modality, both MoT and MoE-4x outperform dense model, with MoT showing comparable or slightly better gains. Validation loss results in Appendix Figure 24]. Model sizes for sparse models indicate activated parameters. All runs are FLOPP--cntrooled and pre-trained from scratch.
MoE-4x's validation performance when the evaluation data distribution differs significantly from the training data, resulting in uneven token distribution among exports. We acknowlodge these limitations and provide additional discussion on the results compared to MoE-4x each individual experiment to provide a more comprehensive understanding.
11

---

Published in Transactions on Machine Learning Research (04/2025)
Model Size
Hidden Dim.
Layers
Heads
Seq.
Length
Batch Size/GPU
GPUs
Tokens/ /Batch
Steps
Tokens (T)
37M
256
4
4096
12
32
1.57M
160k
0.252
94M
512
8
4096
8
32
1.05M
160k
0.168
443M
1024
24
4096
1.57M
160k
0.252
24
4096
0.252
4096
32
4096
3.15M
120k
0.377
Table 1: Architectural specifications and training configurations of models across different pa- rameter scales (Chameleon setting). The table lists the hidden dimension, number of transformer layers, attention heads, and sequence length for each model size. Additionally, provide the batch size used per GPU, the total number of GPUs, training steps, and the corresponding total number of training tokens (in trillions).
3.2.2 Accelerated Pre- Teaaninn at 7B Scale
The MisturceoffTraansfonncss (MoT) architecture demonstrated significant pre-training acccleration at the 7B parameter scale (Figure 5a). MoT achieved the dense model's final loss (at 120k steps) in half the time, reaching equivalent performance at just 60k steps. We quantified this acceleration using step matching analysis (Figure 5).. This method plots the training steps required by MoT and MoE-4x to reach equivalent loss values as the dense model. The analysis revealed that MoT consistently required only 45.5% of the dense model's training steps to achieve comparable pre-training loss, indicating substantial and sustained acccleration throughout training. poddality-speciffe analysis showed MoT's particular effectiveness in the image modality, requiring only 34.8% of the dense model's training steps to match final loss (Figure - MoE-4x showed limited improvement in this domain. For text, both MoT and MoE-4x outperformed the dense model, with MoT showing comparable or slightly better gains.
Validation loss results (Figure g-n) further supported these findings. MoT at 55.8% of training steps achieved validation losses comparable to or lower than the dense model's final validation loss across both modalitics. This indicates that MoT requires only 55.8% of the training FLOPs to match the dense modcl's validation metries, offering substantial computational savings.
3.2.3 Performance Across Multiple Model Scales
We extended our analysis of MoT to five additional model scales (37M, 94M, 443M, 1.5B, and 7B) within the Chameleon setting (Figure Gp. MoT consistently delivered significant speedups the image modality across all scales, outperforming both the dense model and MoF-4x.
We also observed that MoE-4x exhibited diminishing returns model size increased. While it showed some speedup in image modality at smaller scales, this advantage diminished at the 7B scale. In contrast, MoT maintained its performance edge across all scales. For text modality, both MoT and MoF-4x outperformed the dense model, with MoT showing comparable or slightly better gains.
Validation loss results across these scales (Appendix Figure 24 remains consistent with the trend observed in training loss. In both image and text modalities, MoT consistently achieved lower validation losses with fewer training steps compared to the dense model and MoE-4x. This demonstrates that MoT's benefits extend across wide range of model sizes, highlighting its versatility and efficiency for large-scale multi-modal generative tasks.
12

---

Published in Transactions on Machine Learning Research (04/2025)
3.3 Extending to Third Modality: Chameleon Tex-- Results
We evaluated MoT's performance in multi-modal setting by introducing speech as third modality along- side text and images. Experiments focused on the 7B model and smaller scales, comparing MoT against dense and MoE-4x models pre-trained from scratch under FLOPs-controlled conditions.
3.3.1 Experiment Setup
We utilized the training dataset from SpiRit-LM (Nguyen et al. 2024 (Table 2 as our speech dataset. The training data included both speech-only samples and interleaved speech text data in the SpiRit-LM format. Speech imput was converted to discrete tokens using an in-house tokenizer, variant of DinoSR (Liu et al., 2024a), which extracts semantic tokens with vocabulary size of 500. Each token represents 40ms of audio content (25Hz). Model architectural specifications and training configurations of models are shown in Table
To create the three- moreeeyyy training dataset, we combine the speech training dataset with the Chameleon text-and-image training dataset with sampling ratio of 1:6. Within each dataset, we adopt the same data mix ratio utilized by Nguyen el. al. (2024] and Chameleon Team (2024).
This experimental setup allows us to evaluate MoT's capacity to handle complex multi-modal inputs, in- cluding the temporal and semantic challenges inherent in speech processing, while maintaining efficiency and performance across text, image, and speech modalities. We followed the evaluation setup in the aforemen- tioned Chameleon setting (Section b and additionally reported the speech modality validation losses on held-out sets of LibriLight (LL60K) and People's 's Speech Dataset (PPL30K).
Dataset
Modality
Hours
# Speech Tokens'
# Text Tokens
People's Speech Dataset Galvez et al. 2021
Speech-only
16,404
1.2B
Voxpopuli (English) (Wang al. 2021
Speech-only
23,166
1.6B
LigriLight (Kahn et al. 2020
Speech-only
55,308
4B
Multilingual LibriSpeech (English)
Pratap al.
2020
Speech+Test 44,585
3.2B
0.5B
Spotify CCifton et al. 2020
Speech+Text
57,290
4.2B
0.7B
Table 2: Dataset information for spoech pro-training. plication.
TThe speech token counts are computed after dedu-
Model Size
Hidden Dim.
Layers
Heads
Seq.
Length
Batch Size/GPU
GPUs
Tokens/ Baaens
Steps
Tokens (TT)
413M
1021
21
1096
61
1.57M
160k
0.252
880M
1536
21
24
4096
2.10M
120k
0.252
1.5B
2048
24
2.10M
0.252
4096
32
4096
381
3.15M
120k
0.377
Table 3: Architectural specifications and training configurations of models across different pa- rameter scales CChameleon+-Speee setting). The table lists the hidden dimension, number of trans- former layers, attention heads, and sequence length for cach model size. Additionally, provide the batch size used per GPU, the total number of GPUs, training steps, and the corresponding total number of training tokens (in trillions).
3.3.2 Performance with Speech Integration at 7B Scale
The 7B MoT model with added speech modality (Figure 8) demonstrates substantial pre-training accelera- tion. In the speech modality, MoT' speeds up pre-training substantially compared to the dense and MoE-4x models (Figure bh). Step matching analysis (Figure 5p) shows MoT achieving equivalent speech pre-training loss to the dense model in just 22.9% of the training steps, indicating considerable computational efficiency.
MoT also consistently outperforms baselines according to the validation loss results on speech datasets LL60K and PPT30K (Figure 8 -f). Notably, MoT maintained its efficiency across image and text modalities
13

---

Published in Transactions on Machine Learning Research (04/2025)
(a) Text: Autoregressive Objective
(b) Speech: Autoregressive Objective
(c) Image: Autoregressive Objective
Prodiction
Tokan
Discrell Tolkens
&
Discriie Tokkens
Discrvil Tokkens
Mixture of Transformors
Midure Transforaars
Mixuue Transtomans
Speech: FFN, Warss Layerlorm
Text: FFN, Wass Layerloorm
Speecht: FFN, Layerklam
Image: FFN, Wars Layerlorm
Speech: FFN, Wors. Layenlorm
FFN, Layerhoor
FFN, Layerform
FFN Layerform
FFN, Won, LayerNorm
Figure 7: Extended multi-modal experiment with speech modality CChamcleon+-Speechhh Build- ing on the previous setting, a third modality (speech) is incorporated. All three modalities text, images, and speech-are are trained using autoregressive objectives. Speech is represented as discrete tokens via pre- trained speech tokenizer, showcasing the model's ability to handle diverse imput types uniformly.
(Figure 8g Bg-n), achieving comparable or lower validation losses than the dense model's final loss at only 55.8% of training steps.. This demonstrates MoT's robust. performance in multi-modal settings.
3.3.3 Scalability Across Model Sizes
We extended our evaluation to smaller model scales (443M, 880M, 1.5B) in the Chameleon: Sppet setting (Figure MoT consistently delivered significant acceleration across all three modalities, with pronounced improvement in speech processing. MoT required only 15.1% to 33.6% of the dense model's training steps to match speoch training loss across all scales.
MoF-4x exhibited inferior performance in speech tasks, showing improvements in training loss but unstable generalization in validation, particularly for speech (Figure g]. We observed mixed performance of MoB-4x across all scales studied (443M, 880M, 1.5B), where it underperforms the dense baseline in speech valida- in speech pre-training loss. We believe this instability stems from two
tion loss, despile outperforiing reasons: first from the learned routing mechanism (specifically, the Expert Choice routing in Section 3.2.1| which is sensitive to data distribution shifts. In heterogeneous multi-modal settings, the routing network can become imbalanced; for example, when speech tokens are less frequent or display a distinct distribu- tion relative to text and images, the gating mechanism may under-utilize some experts. These result in suboptimal performance the speech validation datasets, LL60K and PPK30K, which exhibit significantly different data distributions compared to the combined text and speech training data. Second, MoE-4x s large number of raw parameters could make it prone to overfitting, hence contributing to its underperformance in speech validation loss, especially given the smaller amount of unique specch tokens the combined dataset (Section 3.3.1
In contrast, MoT leverages a deterministic, modality-aware partitioning that decouples non-embedding pa- rameters (such as FFNs and projection matricos) by modality. This design inherently avoids the load- balancing issues of MoE and offers a more stable signal across modalities, allowing it to consistently outper- form both dense and MoF-4x models across all scales in speech modality, for both training and validation metrics. This consistency demonstrates Mol's effective adaptation to multi-modal tasks, highlighting its reliability and scalability in generative AI applications across speech, image, and text modalities.
We highlight that our contleedd experiments, we ensured that the dense bascline and MoT models have nearly identical parameter counts in the non-embedding portions of the network. The MoT's efficiency does not come from reducing the overall number of parameters but from decoupling mpeelity-speciffc com- ponents. In contrast, MoE-4x inherently has larger parameter count due to multiple expert branches. Although this increased count might suggest a potential for improved performance, our results indicate that the extra parameters in MoE do not translate inlo stable or superior performance across modalities largely because of the aforementioned routing challenges.
14

---

Published in Transactions on Machine Learning Research (04/2025)
Steps to Match Dense
120000
4x
Speedup
MoE-4x (s 0.681)
100000
Training Loss
1.4
MoT (ours) (s = 0.229)
80000
60000
Dense
40000
MoT (ours)
20000
MoE-Ax
B
120000
120000
Training Steps
Training Step for Dense
(a) Speech Modality Training Loss
(b) Training Steps Matching
McE 4x (5- 0.9791
lourst
120000
MoE 4x (5 0.840)
Wildenbe 1065
2,45
Danse NoT (ours)
030
4030
Dense
MoT (ours)
MuE-4x
Staps to
-00
Dense
Taining Siece
Theiring Step Derne
(d) Speech Eval Loss Match-
(e)
Speech
Eval
Loas:
(c) Speech Eval Loss: LL60K
ing
PPL30K
Speech Eval Loss Matching
6:35
ues
6.15
(53 05 (oappo))
03 8h cwwttwott
Maf
25.4% chddsiit
M-7
(g) Image Eval Loss: Obelisc
(h) Image Eval Loss: COCO
Image Eval Loss: Flickr
(j)
Image Eval Loss: SSTK
2,50
1a lne
2.10
2.25
3,20
MaT Civ
Drese Ma7 chechhcccee
Mai
36E-A4
MuT shachmrrtt
Dane
HOT
yuE-A
SWIS HM7 09 coddooo
(k) Text Eval Loss: Obelics
Text Eval Loss: COCO
(m) Text Eval Loss: Flickr
(n) Text Eval Loss: SSTK
third modality. a, MoT accelerates pre-training for speech modality, reducing loss faster than dense and MoE-4x models. b, Step matching plot shows MoT achieves equivalent loss in 22.9% of dense model's training steps, indicating substantial computational effi- ciency. c-f, Validation losses on LL60K and PPL30K speech datascts confirm MoT's consistent performance. MoT reaches baseline speech performance in 37.2% of the FLOPs (f). g-n, MoT maintains efficiency across image and text modalities when speech is added. At 55.8% of training steps (determined from Figure 5 Chameleon 7B), MoT achieves comparable lower validation losses than dense model's final loss for image and text tasks. Model sizes for sparse models indicate activated parameters. All runs are FLOPs-controlled and pre-trained from scratch.
Figure 8: Performance of MoT with speech
15

---

Published in Transactions on Machine Learning Research (04/2025)
MoF-Ax (8- 0.566] MoT lours) Is 0.202!
Dense McT (ours)
Mof--xx (s 0.749)
Trahing
NoT (ours) is 0.557)
Danie MoT lours)
5X00
l0xce
MoE-Hn 4k
5000
5X01
Training Steps
19000
Traninng Stop for
Traning
Training
(a)
443M
Image
Training
Loss
(b) Image Loss Matching
(c) Text Training Loss
(d) Text Loss Matching
Thiringto Lox
MoT (ours) is 0.151)
Valddaten Lons
1770m
1.24
Dense MuT (ours)
narch
330000
Dense NoT [OLEs)
2.30
Dense
MoE-4x
1000
MoT tours]
McE-4x
Tririing
Tranting Step for
(e)
443M
Speech
TTraiiing
(h)
Speech
Eval
Losst
Loss
(f) Speech Loss Matching
(H) Speech Eval Loss: LL60K PPL30K
Mc6-4x 0.6221
Dense
SWAP to Phath Dalue
223000
MoE-4x
Toaiing Loes
tours) (s- 0.311)
Mot lours)
MoT (ours) (s- 0.572)
McB-4x
Tranngg
Donse NoT lours!
40:03
NuE-4x
D200
3200
Taining Stees
Thaining
Trininng
Tsining Step
880M Image Training Loss
(j) Image Toss Matching
(k) Text Training Loss
(1) Text Loss Matching
Thining Lon
lours!
Validatton Losa
3500
Denik
1.30
Denisc
MoT (ours)
Derie MoE-4x
2,00
NoT
Mall-xx
Maf-4x
Liboe
1390
Traiiin Stops
Traning Stass
Training Stop Dense
Traiting Stess
(m
880M Speech Training
(p) Speoch
Eval
Loss:
Loss
(n) Speech Loss Matching
(0) Speech Eval Loss: LL60K PPL30K
McE-4x 0.763)
Dense
Stops Watch Darge
2200
MoE-Ax (s- 0.77)
Thaning Loos
MoT lours] (s- 0.336)
MoT lours) McE-4x
Mol (ours) (x- 0.505)
Traning
Dense MoE 4x
1200
120000
Triniing Steps
Training Step Dense
Training Steps
Taining Step for Dence
(q) 1.5B Image Training Loss
(r) Image Loss Matching
(8) Text Training Loss
(t) Text Loss Matching
Ualaasaan Loua
MoT joursi (s 199)
Tainin Loce
1.0
Mol--ox
Denis
MoT (ours)
Stepa
2003
vaidaiton
220
2220-
MoT tours)
Mof-4x
NoT tours)
MoE-4x
Training Stops
Training Stops
Training Stop for Dense
Training Steee
(x) Speech
Eval
Loss:
(u) 1.5B Speech Training Loss
(v) Speech Loss Matching
(w) Specch Eval Loss: LL60K PPL30K
Figure 9: Speech, image, and text modality performance across model scales. MoT demonstrates consistent speedup across image and text modalities for models ranging from 443M to 1.5B parameters (see Appendix Figure 25 for validation losses). Speech modality shows even greater acceleration, with MoT matching dense model training loss in 15.1%-3..6% of steps across all scales. MoT also consistently outperforms MoE-4x in speech modality. Sparse model sizes indicate activated parameters. All runs are FT.OPs-controlledd and pre-trained from scratch.
16

---

Published in Transactions on Machine Learning Research (04/2025)
3.4 Multi- Multi-Objective Training in the Transfusion Setting: Autoregressive Text and Diffusion- Based Image Generation
(a) Text: Autoregressive Objective
(b) Image: Diffusion Objective
Non Token Prodiction
Discorete Thkiane
Midure Tranfformns
Faot:
Toxts Worcs
Image: Layerhorm
PPN Wark
FPN, Worls
110000 difusion timestops
MB II
Goucsion Noite
aa) Multi-Modal Experiment Modality with Diffusion Objective.
Setting 3: Enhancing Image
Steps Match Dense
250000
0.220
MoE-4x (s 0.784)
Training Loss
0.215
200000
MoT (ours) (s 0.309)
0.210
150000
0.205
Dense
100000
MoT (ours)
50000
0.195
MoE-4x
250000
250000
Training Steps
Training Step for Dense
Transfusion 7B Image Modality Training Loss
(c) Image Modality Training Loss vs dense steps
Mici de dp- 3.343] Mell Oound fom 0.374)
0321
12300
pance
Deruz
Mut touns
123
28.00
C35
Tuiding
Thaiing Step
(d) Image Modality (e) Image validation (f) Image Eval: CLIP (g) Image Eval: FID (h) Captioning Eval:
validation Loss
Loss vs dense steps Score(t)
Score(1)
CIDEr Score(t)
Figure 10: Multi-objective training performance in MoT. a, Schematic of multi-objective setup: text trained with autoregressive objectives, images with diffusion-based objectives, as described in Transfusion [hhou et al. 2024). b-e, MoT accelerates pre-training beyond Transfusion, particularly for image modality. The 760M MoT model, using half the baaniing/nnfreeee FLOPs of the 1.4B dense baseline (Transfusion). consistently outperforms the dense model across metrics: CLIP score (0.214 0.206, higher is better), FID score (21.145 vs 24.688, lower is better), CIDEr score for image captioning (0.320 0.286, higher is better), and image modality training loss. All runs FLOPs-controlled and pre-trained from scratch, demonstrating MoT's superior efficiency and performance across various model scales.
Transfusion [Zhou et al. 2024 introduces unified framework that enables single transformer model to process both discrete text and continuous image modalities (Appendix A The key innovation is the utilization of separale loss functions for each modality language modeling loss for text and diffusion loss for images whhiee sharing data and parameters within a single architecture. In this subsection, we evaluate the performance MoT under the multi-objective training setup the Transfusion setting. Here, text is trained using autoregressive objectives, while images are trained using diffusion- objectives. All models pre-trained from scratch under FT.OPs-cottoolled conditions.
17

---

Published in Transactions on Machine Learning Research (04/2025)
0.225
Steps to Match Dense
250000
Training Loss
0.220
163M MoT (ours) (s 0 0.883)
200000
1.4B MoE-4x (s 0.626)
0.215
1.4B Dense
150000
760M MoT (ours) (s = 0.259)
1.4B MoE-4x
0.210
163M MoT (ours)
100000
0.205
760M MoT (ours)
50000
250000
Training Steps
250000
Training Step for Dense
(a) 760M MoT vs.
1.4B Dense Image Modality
Training Loss
(b) Image Modality Training Loss vs dense steps
9.215
26.920
0.325
0.320
9.200
0.206
FID Score:
24-
0275
CLIP
0.200
0.195
21.145
0,232
0.195
0225
0200
1.46 DazdA
H83M MoT (ours)
160M MOT (965)
1.48 Danss
363M NoT (ouns)
7ROM MOT (scas)
1,48
183M Mot (ours)
IROH McT LOU5)
(c) Image Eval: CLIP Score(1)
(d) Image Eval: FID Score(1)
(e) Captioning Eval:
CIDEr Score(1)
In the Transfusion setting, a 760M MoT model outperforms a 1.4B dense baseline across key image generation metrics, while using only half the FLOPs for both training and inference. a-b, MoT accelerates pre-training beyond Transfusion, particularly for image modality. The 760M MoT model, using half the training/ Draining/inference /iffrrnce FLOOPs of the 1.4B dense baseline (Transfusion), consis- tently outperforms the dense model across metrics: CLIP score (0.214 vs 0.206, higher is better, (c)), FID score (21.145 vs 24.688, lower is better, (d)), CIDEr score for image captioning (0.320 0.286, higher is better, (e)), and image modality training loss. All runs are FLOPs-controlled and pre- pre-trained from scratch, demonstrating MoT's superior efficiency and performance across various model scales.
Figure 11:
3.4.1 Experiment Setup
Data and Pre-processing. We adopt the same data setup as Zhou et al. (024). For text, we utilize the Llama 2 tokenizer and corpus Touvron et al. 2023b), which contains 2 trillion tokens across diverse domains. Images are encoded into latent patches using a Variational Autoencoder (VAE) (Kingma & Welling 2022 where each patch corresponds to a continuous vector. We use a collection of 380 million licensed Shutterstock images and captions. Each image is center-croe center-cropped and resized to 256 256 pixels. Our VAE model does 8 x 8 spatial downsampling of the image. For multimodal examples, we enclose each image sequence with special Lokens-beginniin of image (BOT) and end of image (FOT))before before integrating into the text sequence. This approach results in single sequence that may contain both discrete elements (text tokens) and continuous elements (image patches). We randomly order the images and captions, placing the caption first 80% of the time. In most of our experiments, we sample 0.5 trillion tokens (or patches) from two modalities at a 1:1 ratio.
Model Hyperparameters. To investigate scaling trends, train models at five different sizes 0.16B, 0.76B, 1.4B, and 7B parameters. Model architectural specifications and training configurations of models are shown in Table ! We keep U-Net patch encoding parameters fixed as 0.27B additional parameters across all configurations. We randomly init ialize all model parameters, and optimize them using AdamW (3) =0.9, 32 =0.95, e =le-8) with a learning rate of 3e-4, warmed up for 4000 steps and decaying to 1.5e-5 using a cosine scheduler. We train on sequences of 4096 tokens batches of 2M tokens for 250k steps, reaching 0.5T tokens in total. We regularize with weight decay of 0.1 and clip gradients by norm (1.0). We conduct 250 diffusion steps during inference.
18

---

Published in Transactions on Machine Learning Research (04/2025)
Evaluation Benchmarks. We evaluate the model's performance collection of standard unimodal and cross-modal benchmarks. For text-to-text tasks, we measure perplexity on 20 million held-out tokens from Wikipedia and the C4 corpus (Raffel et al. 2020). For text-to-image tasks, we report the diffusion validation loss following SD 3 Esser et al. 2024) on held-out Conceptual 12M (CC12M; Changpinyo et al. (2021)) data. We also use the MS-COCO benchmark (Lin et al., 2014), where we generale images based on 30,000 randomly selected prompts from the validation set. We measure the photorealism of these images using zero-shot Fréchet Inception Distance (FID) (Heusel et al., 2017) and their aligmment with the prompts using CLIP score (Radford et al. 2021 We also assess the model's ability to generate image captions by
reporting CIDEr Vedantam et al. 2015 scores on the Karpathy test split of MS-COCO LLin et al. 2014
Model Size
Hidden Dim.
Layers
Heads
Seq. Length
Batch Size/GPU
GPUs
Tokens /Bachh
Steps
Tokens (T)
163M
768
16
4096
128
2.10M
250k
0.524
760M
1536
24
24
4096
4
128
2.10M
250k
0.524
1.4B
2048
24
16
4096
2
256
2.10M
250k
0.524
7B
4096
32
32
4096
2
256
2.10N
250k
0.524
Table 4: Architectural specifications and training configurations of models across different pa- rameter scales (Transfusion setting). The table lists the hidden dimension, number of transformer layers, attention heads, and sequence length for each model size. Additionally, we provide the batch size used per GPU, the total number of GPUs, training steps, and the corresponding total number of training tokens (in trillions).
3.4.2 Mixture of Transformers Enhances Multi- Obtti-Obeeetivee Training Efficiency
In the Transfusion setting (Figure 10), the dissture-oF-ransffrmmes (MoT) architecture demonstrates siguif- icant acceleration in pre-training for the image modality at the 7B parameter scale (Figure 10p). Compared to dense and MoF-4x models (Figure =0 MoT substantially speeds up pre-training. Step matching analysis indicates that MoT achieves equivalent image pro-training loss to the dense model in only 30% of the training steps. MoT's efficiency in image generation further evidenced by superior diffusion validation loss (Figure [0Hl,e,, higher COCO-30k CLIP scores (Figure [0F), and lower FID scores (Figure lOg). To compare our 7B model with external models, we compute the COCO-30k FID at a guidance level of 1.6 and obtain score of 8.14. In contrast, dense model trained on 1T tokens with richer data achieves COCO-30k FID of 9.22 under the same guidance level. This comparison further validates the efficiency of MoT over dense models. In image understanding tasks, MoT exhibits more than a threefold speedup compared to the dense model, achieving a final score of 40.6 versus 31.5. We exclude the MoE-4x caption performance due to potential information leakage from expert choice training. These results extend our findings from the Chameleon 7B setting, where MoT matched the dense baseline's image pre-training loss using only 34.8% of the FLOPs (Figure 5).
The text performance improvement of MoT in the Transfusion setting was less pronounced compared to the Chamelcon setting. In Chamelcon, MoT required only 54.6% to 66.2% of training steps to match the dense model's text modality training loss. In Transfusion, the text performance improvement was marginal to none (see Appendix 26). We believe several factors contribute to this observation: (1) this discrepancy may be attributed to the separate training objectives for image and text modalities in the Transfusion sctting, which leads to close to optimal text performance. The decoupling of training objectives in the dense model might confer benefits similar to MoT's decoupling of modality weights. This hypothesis is supported by observations in Zhou et al. 2024), where only changing the image training representation and objective led
4SD 3 (Esser et al. 2024) and Movic Gen (Polyak et al. 2024) show that diffusion validation loss is a strong predictor of overall model performance. Validation loss is well correlated with human evaluations of text alignment and overall quality, as well as with holistic image evalustion metrics, including GenEval (Ghosh et al. 2023 and 12l-CompBench (Huang et al. 2023).
For clarity, unless otherwise noted, do not classifier-free guidance for image generation (Ho & Salimans 2022) for the case of comparisioon. Although classifier-free guidance offers immediate improvements image quality as indicated by better FID and CLIP scores requires complex hyperparameter tuning to find the optimal guidance value for each individual model, complicating the evalutation process. For 7B MOT and dense model, use classifier guidance of to generate example images.
19

---

Published in Transactions on Machine Learning Research (04/2025)
to dramatic improvements in text performance compared to Chameleon [Chameleon Team 2024 despite no direct changes to text. training. (2) Since our experiments are FI.OP-controiled, the limited improvement margin on text tasks could reflect the fact that text generation is already efficient and may require less model sparsity. (3) The modality decoupling in MoT is especially beneficial for modalities that demand heavier computation (like image diffusion), so the relative advantage for which is less compute-intensivee is smaller. None the less, we acknowledge the need for further investigation into text performance and plan to explore additional modifications or hybrid strategies (e.g, integrating MoE elements selectively) in future work.
At smaller scales, MoT demonstrated significant performance gains, particularly in the image modality (Figure 11 A 760M parameter MoT model, operating at half the FLOPs of a 1.4B parameter dense baseline, consistently outperformed its larger counterpart across multiple metrics. Image quality improved,
as evidenced by CLIP (0.214 vs 0.206) and FID (21.145 vs 24.688) scores (Figure 11 (d.)). Image captioning capability, measured by CIDEr score, also improved (0.320 vs 0.286; Figure 11p).
When comparing models with 8-fold difference in size (163M MoT 144B dense/MoE-4y: the 163M MoT achieves comparable image modality training loss. While the 163M MoT still slightly lags behind the 1.4B dense model in evaluation metrics, the more than 8-fold reduction in both training and inference compute highlights MoT's strength in the image modality.
3.4.3 Scalability Across Model Sizes
MoT consistently outperformed basclines in image modality tasks across all scales (163M, 760M, 1.4B, Figure 12 FID scores showed substantial improvements for MoT over dense models: 21.58 vs 27.42 (163M), 15.75 vs 25.58 (760M), and 15.85 vs 19.32 (1.4B). CLIP scores also consistently improved with MoT: 0.195 vs 0.185 (163M), 0.214 vs 0.202 (760M), and 0.217 vs 0.206 (1.4B).
In text modality, MoT matched dense models training and validation loss on text-only datasets. However, MoT demonstrated significantly better generalization in captioning tasks, with consistently higher CIDEr scores: 0.232 vs 0.147 (163M), 0.320 vs 0.251 (760)), and 0.335 vs 0.286 (1.4B). As discussed in 7B results, MoT shows little improvement text accross the scales.
3.4.4 Performance with fine-tuning
Following the original Transfusion (Zhou et al. 2024) sctup, we finc-tune the 7B MoT and dense models an internal visually appealing dataset and on image editing lasks, as shown in Figure 13 The fine- tuned models are capable of generating text, detailed hand features, fictional images, and photorealistic images. After fine-tuning, MoT demonstrates better quality and higher faithfulness compared to the fine- tuned dense models (see Appendix B). indicating that the performance gain of MoT over the dense bascline is well maintained after fine-tuning. Notably, we train on only 0.5 trillion tokens, which is significantly less than other state-of-the-an (SOTA) image generation models. Zhou et al. (2024) shows that the model is not yet saturated even at trillion tokens; we leave the scaling up of our model and training with more data as future work. We also finetune 7B MoT on 8k image eiditing data (Zhang et al.) (202)), as show in Figures 13 (i.j) The Transfusion MoT fine-tuned image editing tasks extends its capabilities to generate images based other inages by following text instructions.
20

---

Published in Transactions on Machine Learning Research (04/2025)
Mol-4K jx- C.6911
20000
Mot-dax (..1444)
Mol (ours) (s 0.254)
Mol (ours) 0.285)
Steps Murch
0.211
c215
MoT [ours]
MoE-4x
MoT (ours)
S0009
Twihing Steas
10003
0.21
0.004
MoE 4x
24000
Trainin Stop
Traiing Staps
Fairing Dense
(1) 163M
Image
Training
Loss
Image Loes Matching
(3) Image Val. loss
(1) Image Val. Loss Matching
322
2.1
0.42
(5) 163M Image Eval:
(6)
Image
Evall:
FID (7) (1)
Text. Val. Loss: C4
Text Val. Wikipeddia (4)
(
Loss:
(9)
Captioning
Eval:
CLIP Score(t)
SLore(L)
CIDEr (T)
MoE-áx 0.588)
2elun lax
25000
MoE
0,29
200090
20000
SINES Malco
Detiew MoT (ours)
6.23
MoE 4x
33000
Deniee MoT (ourss)
2009
Steae
Taihing Stucs
0,03
MrE-4x
Farning Step for Dense
Trainin Step for Denser
Tharning Staps
(10)
760M
Image Training
(13) Image Val. Loss Match-
Loss
(11) Image Loss Matching
(12) Image Val. loss
ing
cm
19245
117
6.41
(14) 760M Image Eval:
(15) Image Eval: FID (16) (1)
Text Val. Loss: CA
(17) Text Val. Wikipedia (1)
Loss:
Captioning Eval: CIDE: (t)
(18)
CLIP Score(1)
(core(L)
7V/00
(ours) 0-231)
3200
MoT fours)
Denise HOT (ours) MoE
11:00
30000
1000
Dinise
Sees
MoT tours)
Taining Stass
Toining Step for Dense
Trainin Stap for Dense
Trairing Stopa
(19)
1.4B
Image
Training
(22) Image Val. Loss Match-
Loss
(20) Image Loss Matching
(21) Image Val. loss
ing
(23) 1.4B Image Eval: (24) Image Eval: FID
(25) Text Val. Loss: C4 (1)
(26) Text Val. Wikipedia (4)
Loss: (27)
Captioning
Eval:
CLIP Score(1)
Score(1)
CIDE: (f)
Figure 12: -spelity- training loss and step matching plots Transfusion setting across model scales. MoT consistently achieves substantial speedup in image modality (trained with diffusion- based objectives) across 163M, 760M, and 1.4B models, outperforming dense model and MoE-4x. In image modality, MoT reaches comparable training loss to dense model in 17.5%-26.44% of steps across all scales. For text modality, MoT matches dense model in training and validation loss C4 and Wikipedia datasets, with improved generalization in captioning tasks (CIDEr score). MoE-4x shows unstable performance: lower training losses (Appendix F Figure 26 but poorer generalization than dense model on text evaluation metrics. Model sizes for sparse models indicate activated parameters. All experiments FT.OP--contoolled and pre- trained from scratch.
21

---

Published in Transactions on Machine Learning Research (04/2025)
GO BIG OR GO MOT
(a) 'GO BIG OR GO MOT is written on the
(b) A close up photo human hand, hand model. Figh quality
(c) An angry duck doing heavy weightlifting at the gym.
(d) A car made out of veg- etables.
blackboard.
can't bear it
(e) photo of bear
(f) A photo of corgi dog wearing wizard hat play- ing guitar on the top mountain.
(g) Three spheres made of glass falling into ocean. Water splashing. Sun is setting.
(h) A tranquil, anime- style koi pond in a serene Japanese garden, featuring blossoming cherry trees.
ing a suit and tophat in river the middle of for- est holding sign that says "I cant bear it'.
STOP
GO
Put cat the seat.
(i) Change the stop sign to say 'GO".
Figure 13: Image generation and image editing (last row) examples from 7B Transfusion MOT model trained with 0.5T tokens.
22

---

Published in Transactions on Machine Learning Research (04/2025)
800m Dense
laldm, HoHAa,
200
-4
22.5
Isc
S5TK
280
62.5
230
Steps
Steps
Steps
Steps
Figure 14: Ablation results of modality unlying in different transformer components in the Chameleon set- ting, evaluated using the held-out sets of Obelisc and Shutterstock. Modality untying in the feedforward modulc alonc significantly improves model performance, with substantial gains on the image modality. Fur- ther unlying Q K, V matrices in the attention module yields significant performance improvements, whereas untying layer norms has a negligible impact on evaluation performance.
3.5 Impact of Modality Untying Different Transformer Components
We conduct ablation experiments to understand the impact of modality untying in different transformer components. We conduct the experiments using architectures with FLOPSs controlled to match the dense architecture in Table We compare four model variations:
(1) the dense baseline; (2) model with modality untying only in the feedforward module as in Lin et al. (2024); (3) model with modality untying in both the feedforward module and the Q, K, V weight matrices (excluding LayerNorms); (4) the full modality-untier
MoT model.
Model Size
Hidden Dim.
Layers
Heads
Seq. Length
Batch Sixe/GPU
GPUs
Tokens Tokens/Batch Batch
Steps
Tokens (T)
880M
1536
24
24
4096
128
2.10M
120k
0,252
Table 5: Architectural specifications and training configurations of model used in ablation experiments.
As shown in Figure 14 modality untying in the leedforward module alone (Lin et al. 2024 significantly improves model performance, with substantial gains on the image modality. Further untying the Q K, V weight matrices in the attention module yields significance performance improves. On the Obelisc (auurenoon et al. 2023) held-out set, this leads to approximately 33.3% FLOPs saving for the image modailty and 10% FLOPS saving for the text modality compared to only performing unlying in the feedforward module. Notably, the FLOPs savings from adding attention untying to feedforward untying are smaller than those from adding foodforward untying to the dense model. We attribute this to two factors: (1) the foedforward module accounts for a larger proportion FLOPs in the transformer architecture given our context length (4096), and (2) the foedforward module serves as a memory component transformers, where employing separate memory parameters for each modality is effective. Finally, we observe that further untying the LayerNorm parameters top of feedforward and attention untying has negligible impact evaluation performance 0
6This finding does not suggest that untying LayerNorm parameters entirely incffective. Our experiments only examine its impact when combined with atitention and feedforward untying. leave the understanding of its individual effectiveness future work.
23

---

Published in Transactions on Machine Learning Research (04/2025)
Three Transformer Towers
Two Transformer Towers
Two Transformer Towers
Two Transformer Towers
One Transformer Tower
Speech
Image
Speech
Toxt
Text
Imager
Speech
Speech
Imager
Text
Speech
Images
Texte
Good for All Three Modalities:
Good Image Modality
for Modality
for Speech Modality
Hurts Thrre Modalities:
Image, Text, Speech
Hurts Text and Speech
Hurts Image and Speech
Hurts Image and Text
Image, Text, Speech
(a) MoT (ours)
(b) LOO: image
(c) LOO: Text
(d) LOO: Speech
(e) Dense
2.8
Dense
150000
LOO: speech (s 0.875)
MoT (ours)
2.34
125000
LOO: image (s 0.684)
2.5
Training
LOO: image
2.32
MoT (ours) (s 0.557)
2.4
LOO: text
LO0: text (s 0.548)
100: speech
2.30
50000
25000
2:28
160000
15:0000
Training Steps
150:
LOO:
LDO:
Training Step for Dense
(I) Text Training Loss.
(g) Text Loss Comparison
(h) Text Loss Matching
5:25
150000
LOO: speech (s 0. 0.892)
125000
LO0: text (s - 0.321)
Dense
LOO: image (s 0.272)
MoT tours)
LOO: image
75000
MoT (aurs) (s 0.262)
50000
5.00
4.95
LOO: text
LOO: speech
25000
4.90
100000
1500000
Training Steps
L00:
Training Step for Dense
(1) Image Training Loss.
(j) Image Loss Comparison
(k) Image Loss Matching
1.55
150000
LOO: Image (s= 0.653)
Training Loss
1.500
125000
LOO: text (s - 0.304)
1.50
Dense
Spaachtos
1.475
MoT (aurs) (s 0.151)
MoT tours)
3.490
75000
LOO: speech (s 0.147)
LOO: image
2.423
Steps to
50000
25000
speech
160000
1.355
150000
Training Steps
L00,teat
speech
Training Step for Dense
()) Speech Training Loss.
(m) Speech Loss Comparison
(n) Speech Loss Matching
Figure 15: Modality Leave-One-Out (LOO) analysis of MoT variants in Chameleon+ setup. a, Proposed MoT architecture with separate transformer towers for image, text, and speech. b-d, Two-tower MoT variants for three modalities: b, Text and speech combined (LOO: image). c, Image and speech combined (LOO: text). d, Text and image combined (LOO: speech). e, Dense transformer with single tower for all modalities. All models (a-e) have equivalent FLOPs. f-n, Performance results across modalities. Combining modalities in single tower consistently degrades performance, while separation improves results. LOO: text, LOO: image, and LOO: speech achieve lowest losses in their respective isolated modalities mg. Analysis highlights the importance of modality-speciff parameter allocation for optimal performance. All models are 443M in size. For sparse models, size indicates activated parameters. All runs are FLOPS-controlled and pre-trained from scratch.
4 Modality Separation in MoT: Leave-One-Our Analysis
4.1 Experiment Setup
To evaluate the efficacy of MoT's modality-specific architecture, we conducted a Leave- Cuaee-One-Modality--a (LO0) analysis with the Chameleon: Text+ Soadt-mmae+--ppeee framework. This analysis aimod quantify the benefits of separating modalities into distinct transformer towers, implemented in MoT, compared combining multiple modalities within single tower.
24

---

Published in Transactions on Machine Learning Research (04/2025)
Figure 15 illustrates the MoT variants and their performance across different configurations. The bascline
MoT architecture (Figure 15a) comprises three separate transformers for image, text, and speech modalities.
We tested three LOO variants, where two out of the three modalities share the same transformer:
LO0-image (Figure 15
text and speech combined, image isolated
L00-text (Figure
15 image and speech combined, text isolated
L00-speech (Figure 15
text and image combined, speech isolated
A dense transformer architecture with all modalities in single tower (Figure 15 served baseline. All models maintained equivalent FLOPs to ensure fair comparison.
4.2 Results
The
(LOO) analysis revealed advantages of modality separation in the MoT archi- tecture (Figure 15ā-n). Combining modalities consistently degraded performance, as evidenced by higher training and validation losses across configurations. The LOO-text configuration achieved the lowest text loss (Figure 15 while LOO-image and LOO-speech yielded the lowest losses in their respective isolated modalities (Figure 15 m)). These results support the importance of modality-specifio parameter allocation
in MoT.
The impact of modality combination varied across configurations. In LOO-speech, separating the speech modality preserved MoT's benefits for speech but eliminated improvements in image and text modalities. Similarly, LOO-image retained most of MoT's improvements in image performance, while merging text and speech led to performance declines in both. In LOO-text, speech performance deteriorated when combined with image, yet the image modality largely maintained the gains realized by MoT. This differential impact suggests non-reciprocal modality competition effects. By dedicating separate transformer towers to each modality, MoT is able to optimize for the unique characteristies of each modality, resulting in better overall performance across all modalities.
25

---

Published in Transactions on Machine Learning Research (04/2025)
5 Combining the Best of Both Worlds Mixing Heterogeneous Transformers
5.1 Combining MoT and MoE-4x in the Chameleon Setting
In this subsoction, present preliminary results exploring the potential of combining key features of Mixture- of Transssformrss (MoT) and Mixture-of-Experts (MoE-4x) within the Chameleon setting. Specifically, we modify the MoT architecture by incorporating MoE-4x into the text transformer tower. The text forward network (FFN) of MoT is replaced with the MoE-4x mechanism, which introduces multiple expert layers the model. The image transformer lower remains unchanged and follows the original MoT architecture. This experiment seeks to assess whether integrating MoF-4x's expertise mechanism can further enhance MoT's performance in multi-modal generative tasks.
As shown in Figure 16 the combination of MoT and MoE-4x significantly accelerates the reduction of text training loss compared to the dense model, MoF-4x, and MoT alone (Figure 16p-b). The results demon- strate that introducing MoE-4x into the text transformer provides additional speedup without sacrificing the efficiency gains of MoT in the image modality (Figure 16p-d). The averaged training losses across both text and image modalities confirm that the combined model maintains or exceeds the performance of MoT in both tasks (Figure 16p-f).
When evaluating the combined model's performance on validation datasels, we observe consistent gains in the text modality, As shown in Figure [6g--j, the combination of MoT and MoE-4x achieves the best text validation performance across multiple datasets, outperforming both MoT and MoE-4x. Importantly, the image modality performance remains comparable to or slightly better than that of MoT, indicating that the incorporation of MoE-4x into the text tower does not hinder MoT's efficiency in image generation tasks, These early results suggest that combining the strengths of MoT and MoE-4x offers a promising avenue for improving multi-modal models, particularly in tasks requiring simultaneous text and image generation.
5.2 Combining MoT and MoE-4x in the Transfusion Setting
We exlend the experiment the Transfusion setting, where distinct objectives are applied different modalities- autoregressive objectives for text and diffusion-based objectives for images. Similar to the Chameleon setting, we replace the FFN layer of the text transformer in MoT with 4-expert MoE layer, while the image transformer remains unchanged.
As shown in Figure 17 this approach continues to accelerate text training loss reduction compared to the dense model and MoT alone (Figure 17a-b). Notably, the combined model retains MoT's advantage in the image modality, with comparable training loss and speedup relative to MoT (Figure I7p-d). The averaged training loss across both modalities (Figure 17p-f) highlights the potential of this approach in handling multi-objective training with balance between efficiency and performance.
Validation results on representative text-only datasets and image generation tasks confirm the consistency of this approach (Figure 17g-)). "MoT' + Text MoE-4x' achieves the best text performance, maintaining the efficiency of MoT in the image modality while improving text generation. On the other hand, despite MoE-4x demonstrating lower text training loss than the dense model, il shows little to no improvement in text valtidation losses (Figure 17p-i, 1-n), further emphasizing the effectiveness of the MoT approach.
Overall, these preliminary results in both the Chameleon and Transfusion settings provide proof-of-concept for combining the key strengths of MoT and MoF-4x. The integration of MoF-4x into the text modality enhances text performance, while MoT continues to deliver strong results in image generation. Further investigations will explore the scalability and generalizability of this approach across additional tasks and modalities.
"This observation diverges from our observation in the Chaneleon setting (Figure 16 where MoE-4x improved text and image losses during both training and validation. Contrary conventional understanding of MoE, MoE-Ax's training loss gains in Transfusion didn't translate inference time. The discrepancy may stem from the fact that Transfusion processes discrete text. tokens and continuous image tokens, which complicates router generalization during inference. We leave further exploration of integrating MoF and Transfusion future work.
26

---

Published in Transactions on Machine Learning Research (04/2025)
Steps to Match Dense
Dense
150000
MoE-4x (s = 0.731)
Training Loss
MoT (ours)
125000
MoT (ours) (s = 0.610)
MoE-4x
MoT + Text Moe-4x (s = 0.431)
2.4
MoT + Text Moe-4x
75000
2.3
50000
25000
2.2
0
160000
150000
Training Steps
Training Step for Dense
(a) Chameleon 373M: Text Training Loss
(b) Text Loss Matching
Dense
suo
Dense
Mol-4x
Tallung Loos
MoT
MoT (ouns) (5= 0271))
NoT tours)
(ours! (5 0.435)
200
MoT Text Moe-4x 0263)
MoE-4x
100
McT Text Moe-4x & 356)
MoT Text Moc
Text
50000
2000
Thiiiin
Traning Stop for Donge
Theiring Steps
Training Stop for Dense
(c) Image Training Loss
(d) Image Loss Matching
Averaged Training Loss
(I) Training Step Matching
UNitzation bee
Mol' (ours)
Dense
100
0.694) (ours) 1s 294)
Dense
jours)
Srap 10 Mone Dee
NiEE 0.766)
(ours) 0.635)
MoE-4x
230
MoT Text Moe-Rx (o- 0.284)
Maf-4x
MoT + Text Mon-4x 502)
70000
Text Moe 4x
2000
2000
1000
250900
199000
15300
Training Steps
Training Step for Dense
Training Steps
Training Step Dence
(g) Image Eval Loss
(h) Image Loss Matching
(1) Text Eval Loss
(j) Text Loss Matching
Figure 16: Combining MoT and MoE-4x in the Chameleon setting. hybrid model "MoT + Text MoE-4x' was created by replacing the text feed-forward network MoT's text transformer tower with MoE- 4x. a-b, Text training loss reduction is significantly accelerated compared dense model, MoE-4x, and MoT. c-d, Image modality performance benefits of MoT are retained. e-f, Averaged training loss across both modalities. g-j, Validation losses Obelisc dataset: "MoT + Text MoE-4x" achieves best text performance while maintaining comparable or slightly improved image performance relative to MoT. Both significantly outperform dense model and MoE-4x.
27

---

Published in Transactions on Machine Learning Research (04/2025)
2.30
Steps to Match Dense
250000
MoT (ours) (s = 0.970)
2.25
Training Loss
200000
MoT + Text Moe-4x (s (s= 0.504)
2.20
2.15
Dense
150000
MoE-4x (s = 0.495)
2.10
MoT (ours)
100000
MoE-4x
2.05
50000
MoT + Text Moe-4x
2.00
250000
250000
Training Steps
Training Step for Dense
(a) Transfusion 760M: Text Training Loss
(b) Text Loss Matching
Is- NoT (ours) (s= 0.176)
Stape Matc Din
(ours) (s- 0.7361
mekking Looe
1:0E-4x11 15 0.517)
Dense
NoT Text Moc is 0.1711
Dente
MoT Text Noc 4x 0.413)
Mos (ours)
Hol (ours)
2300
MoT Text Moe-fx
Text Moe-4x
20000
2000
35800
Training Steps
Thaining Step Dense
Traiinn Steps
Triiing Step for Dence
(c) Image Training Loss
(d) Image Loss Matching
(e) Averaged Training Loss
(f) Training Step Matching
Valettion Loos
k29
Dense
110 Soos
Derne
MoT
(ours)
Deriee McT lours)
MaE 4x
MoT Text Moe-4x
McT Text Noc 4x
McT Text Moc 4x
Taiing Stops
Tairing Stops
mangn Studs
Training Stops
(g) Text Eval Loss: C4
(b) Text Eval Loss: Wikipodin
(i) Image Eval: CLIP
(i) Image Eval FID
Stepe Marth Datce
HoE
NoE 0.993)
Text
Text
(ours)
(ours) 0.436)
(ours) 0.5822
MoT Text Moesax (s C.511
199
NoT Text Moe-dx
3b00se
120000
208000
S1000 1200e
5000
Grepe
50000
$800
Training Step for Dense
Taining Stop for Denge
Training Stop for Dange
Trainng Step Donse
(k) Steps Matching:
Steps Matching: Wikipodia
(m) Steps Matching: CLIP
(n) Steps Matching: FID
Figure 17: Combining MoT and MoE-4x the Transfusion setting. a.b, Text training loss reduction significantly accelerated compared to dense model and MOT. c,d, Image modality performance benefits of MoT retained. e,f, Averaged training loss across both modalities. g-n, Validation losses on multiple reprosentative text-only datasets and image generation evaluation results: 'MoT + Text MoE-4x" achieves best text. performance while maintaining comparable or slightly improved image performance relative to MoT. Both significantly outperform dense model and MoE-4x. MoE-4x shows better text training loss than dense model (a,b) but minimal improvement in text validation losses (g-h, k-l).
28

---

Published in Transactions on Machine Learning Research (04/2025)
6 ML Systems Aspects of
This section highlights few system properties of MoT, and demonstrates how they translate to real-world benefits in typical training environment.
6.1
Throughput Scaling Properties
Communication Volume The modality-based scaling method by MoT maintains lower Parameter to FLOPs (PpF) ratio. PpF is crucial metric that governs training throughput in large-scale cloud-based training environments, where distributed learning is highly sensitive to communication overhead (Llama tcam 2024 Luo et al. 2020). This is particularly relevant in recent years, as compute capacity has increased significantly faster than nelwork bandwidth (Luo el al., 2018 2024). Consequently, models with smaller parameter sizes have advantage in terms of training throughput at a large scale.
To quantify this effect, we compare the added paramcters each transformer layer by adding E experts in MoE versus untying K modalities in MoT, in a typical transformer setup with Swiglu FFN layers, with token hidden and embedding dimension both D, feed forward layer has |FFN| = 3D² parameters, and the KQVO projections in attention layers introduces |ATTN] 4D parameters. We can ignore the normalization layer weights as they are usually in the shape of D.
Thus, each MoE layer comprises E feod forward layers and a router paramater of size ROUTER = K D. and a MoE layer has an additional (E 1) x [FFN|+ + |ROUT ER| = 3 1)D2 + ED 3 I)D2 parameters. In contrast, MoT incurs an added parameter count (K 1)x ([ATTN]+]]FNN 7(K-1)D
Since E can range from few to hundrods (Dai et al. 2024 Muennighoff et al., 2024 DeepSeck-AT et al. 2025,, typically much smaller K compared to E implies that MoT can have a lower Ppl ratio compared to MoF. in general, which can prove beneficial in real world scenarios, as we show later.
Compute Efficioncy Both MoE and MoT incur additional overhcads when routing tokens to the appro- priate parameters.
1. MoEs suffer from overheads due to the additional operations of performing Top-K selection, indexing tokens, and scattering and adding expert outputs. These operations are sequentially dependent on each other, making it challenging to hide the resulting latency.
In contrast, MoT's overheads primarily come from two sources. First, the CPU-GPU synchronization required for grouping tokens by modality for element-wise projections and reassembling them for attention results in significant overhead, mostly attributed to frequent GPU-CPU synchronization due to masking for specific modalities. Second, the sequential processing of modalities can also lead to underutilization of GPU resources and imbalance, particularly when tokens of different modalities are unevenly distributed across local batches and GPUs.
It's worthwhile to note that. the overheads in MoT can be minimized via diligent. engineering: for example, caching sequence indices for each modality can substantially reducing indexing costs and needs to be done only once per iteration, as the modalities of tokens do not change. Specialized Group GEMMs (Nvidia) or Megablock-style block sparse matrix multiplication can be employed (Gale et al. 2022) to perform imba lanced projections in one shot across all modalities. Since we did not observe these overheads on the critical path in our training setup, we leave these as future directions to further improve MoT.
6.2 Empirical Analysis
We conducted our experiments and system profiling on AWS, using p4de.24xe 2Axlare instances equipped with NVIDIA A100 Tensor Core GPUs. Distributed training of all models are powered by Fully Sharded Data Parallel (Zhoo al.] (003) in full shard mode. We enable Pytorch 2 Compiler (Ansel al.. 2024h to optimize the model whenever applicable.
29

---

Published in Transactions on Machine Learning Research (04/2025)
6.2.1 Horizontal Scaling-Mon Benefits Increase with GPU Count
(MoT) in large-scale dis- tributed training. As large language models (LLMs) typically employ larger global batches across increasing numbers of GPUs, we examined MoT scaling trends by varying GPU count during training. Global batch size and total training tokens were increased proportionally to GPU count, while maintaining constant training steps. We conducted the experiments in the Chameleon setting under the 443M model scale.
We investigated the horizontal scaling capabilities of
Figure 18 shows Obelisc dataset evaluation losses as GPU count scales from 16 to 256. MoT performance gains increase substantially with GPU count. For image validation loss, the percentage of training steps required for MoT match the dense model decreases from 42.1% (16 GPUs) to 21.6% (256 GPUs). For text, this percentage decreases from 75.7% to 50.9%. These results suggest MoT's efficiency and performance benefits grow with increasing pre-training compute resources.
This analysis was conducted under specific AWS infrastructure conditions. Further investigation is needed to generalize these findings across different hardware configurations and training environments.
6.2.2 Speed Advantage of MoT in Wall-Clock Time
We investigated the wall-clock time performance of MoT in a specific environment. This analysis is crucial for understanding MoT's practical benefits in real-world training scenarios, where achieving the best model quality within fixed GPU training time budget is the primary objective. To ensure the accuracy of our claims, we note that our results were obtained using specific AWS setup, specified above. Therefore, we expect the relative performance of MoF, MoT, and dense models to vary across different clusters. Neverthe- less, we believe that our setup represents a typical deployment that leverages cloud computing (e.g., AWS), and thus our experiences and findings will be relevant and beneficial to readers.
Figure 19 illustrates MoT's wall-clock time acceleration over dense Transformer and MoF-4x baselines in terms of GPU training time 256 GPUs. MoT demonstrates significant improvements in both image and text modalities for a fixed amount of GPU training time. Specifically, MoT matches the dense model's 's image
training loss in 47.2% of the total GPU training time and continues to improve thereafter (Figure 19 p). For
text, MoT requires 75.6% of the dense model's time to achieve comparable performance (Figure 19 In
contrast, MoE-4x exhibits no speed advantage the lext modality (Figure 19 ()) and even results in a 1.7x
slowdown the image modality compared to the dense model (Figure 19 Results are consistent, on the
evaluation losses on the Obelisc dataset (Figure 19 h))
6.3 Deployment Considerations
So far, our discussion on the MLSys aspect focuses mainly on training. Since mixod-modality requests might not fill full batch for each modality branch, the inference system could aggregate similar modality operations across multiple requests (when latency budgets allow) to ensure that the specialized GEMM kernels are utilized efficiently, i.e, via dynamic batching (git Yu et al. 2022) continuous batching (any). Orthogonally, we note that when sequence length is not prohibitively long, or when the input modalities are mostly uniform, the modality with fewer number of tokens can be padded to align with the modality with the most tokens, thereby we can compute all modalitics' GEMM at once with a single batched matrix multiplication, trading off slightly wasted compute with lower latency.
30

---

Published in Transactions on Machine Learning Research (04/2025)
Vallsation Los3
2300000
Mol-ixx (5 0.8899
Dense MoT (ours)
Mob--xx (s 0.9300
Derse
124an0
McE
Valiagn LoGa
12500
MoE
MoT (ours)
MoT (ours) 0.421)
McE: 3x
toursi (x- 0.757)
73900
McE-4k
70000
MoE-4x
2500
Thaining Steps
3000
(a) 16 GPUs Image Eval
Thiining
Thaining
Loss
(b) Image Loss Matching
(c) Text Eval Loss
(d) Text Loss Matching
Valazzen Los
McB-3x 0.8191
Denze
SWEs Me leec oond
150000
MoE-3xx (5 0.946)
Danse
121000
McT
MoT
130000
McT is 0.345)
McE 3x
MoT tourst (s- 0. 694)
McE: 3x
McE-4x
75000
McE 4k
25000
Training Steps
300000
Lo000s
(e) 32 GPUs Image Eval
Training Step for
Traiingg
Trining Stop
Loss
(f) Image Loss Matching
g
Text Eval Loss
(h) Text Loss Matching
valoudon LoG
Denie
150000
McE 0.767)
Steps to Mond Dente
120000
MoT
MoB-3x 0.747)
Dense MoT (ours)
MoE-3x 0.9733)
Mol-4xx (.6609
0.694]
10000
Mot-Ix (s 0.766;
42-
MoF-3x
MrE-4k
70800
McT (ours) (8-
MaE 2x
MoE-3xx
MoT tours) (s 0.639)
22000
MaE 4x
Training
350900
190000
150000
(i) 64 GPUs
Image Eval
Training Step for Dense
Traiing Steps
Training Step for Dence
Loss
(j) Image Loss Matching
(k) Text Eval Loss
(1) Text Loss Matching
Vallogeton Laes
McE-3x 0.601!
Denie
MoE: 3x (s 0.6600
120000
McE
MoT (ours)
0.531) (ours) (x- 0.509)
(oTr(ours)
McT (ours) 0.20666
McE: 3K
Shops Melcce
MoE-3x
Moli-ix
MoE-4K
2500
Traiing Reps
350300
15000
(m) 256 GPUs Image Eval
Training Stop for Donse
Training Stons
Trining Stop
Loss
(n) Image Loss Matching
(o) Text Eval Loss
(p) Text Loss Matching
Figure 18: Horizontal scaling The benefits MoT increase with the number of GPUs (Chameleon setting, 443M model scale). As modern LLMs are typically trained on larger global batches using more GPUs, we conduct pilot study on the scaling trends of MoT by varying the number of training GPUs. This effectively scales the global batch size and the total number of training tokens, while keeping the nunber of steps constant. Shown are the evaluation losses on the Obelisc dataset. The performance gains of MoT increase substantially as the number of GPUs grows. For instance, when scaling from 16 to 256 GPUs, the percentage of steps required for MoT to match the image validation loss of the dense model (trained with the same number of GPUs) decreases from 42.1% to 21.6%, and for text validation loss, from 75.7% 50.9%. This suggests that scaling pre-training compute resources further enhances the efficiency and performance gains of MoT.
31

---

Published in Transactions on Machine Learning Research (04/2025)
5.0
Dense MoT (ours)
Time to Match Dense
Training Loss
150
MoE-4x (s 1.707)
MoE-4x
MoT (ours) (s 0.472)
100
4.7
4.6
175
Normalized Training Wall Clock Time
100
Normalized Time for Dense Model
(a)
Chameleon 7B: Image Training Loss by Wall-Clock
Time
(b) Image Loss Matching
Dense MoT (ours)
Time to Match Dense
MoE-4x (s = 1.070)
100
Training Loss
MoT (ours) (s 0.756)
80
MoE-4x
2.0
40
1.9
20
175
100
Normalized Training Wall Clock Time
Normalized Time for Dense Model
(c) Text Training Loss by Wall-Clock Time
(d) Text Loss Matching
Dense (ours) MoE 4x
McE-Oox
Dense Net (ours)
Fimaa March bersse
Mot: ts 1.0033
homalized
Normalzed Cock
homalized Model
(e) Image Eval Loss
(f) Image Loss Matching
(g) Text Eval Loss
(h) Text Loss Matching
Figure 19: Speed advantage of MoT in wall-clock time (Chameleon Setting). For fixed amount of GPU training time, MoT significantly outperforms both the dense Transformer bascline and MoE-4x. MoT matches the image training loss of the dense model in just 47. 47.2% of the GPU training time, with continued improvement. For the text modality, MoT requires 75.6% of the time to achieve the same quality. In contrast, MoE-4x shows no speed advantage in the text modality and results in a 1.7x slowdown in the image modality. Evaluation losses on the Obelisc dataset show consistent results.
32

---

Published in Transactions on Machine Learning Research (04/2025)
7
Related Work
7.1
Foundation Models for Multi-Modal Generation
Recent advances in large language models (LLMs) have extended to multi-modal applications. Early multi- modal LLMs focused on understanding rather than generation, using late fusion techniques to merge sepa-
rately encoded images and text (Alayrac et al., 2022 Liu et al. 2023 Laurençon et al. 2023 Chen et al. 2022 While benefiting from lightweight training, these models lacked multi-modal generation capabilitics.
To enable multi-modal generation, a key strategy involves tokenizing non-text modalities into discrete se- quences (Aghajanyan et al.) 2022 Yu et al., 2023; Bao et al.) 2021 Ramesh et al. 2021 Liu et al.) 2024ch (Figure 20). For instance, Chameleon (Chamcleon Team 2024 and related approaches
Aghajanyan et al.
2022 tokenize images into 1,024 discrete tokens using pretrained models like VQGAN [Esser et al. 202
training over combined text-image token sequences. Similar tokenization has been applied to speech Nguyen
et al. 2024). Recent models like Transfusion (Zhou et al.) 2024) have explored continuous image tokens and diffusion- based loss functions to enhance visual generation quality. Our proposed mixture of transform- ers method is compatible with these approaches and can be integrated as a drop-in replacement for dense transformer architectures. We demonstrate substantial improvements across diverse multi-modal settings, including both Chameleon (Chameleon Team 2024) and Transfusion (Zhou et al 2024).
7.2 Sparse Architectures for Multi-Modal Generation
Sparse architectures, particularly Mixture of Experts (MoE), have shown promise in text-based models, allowing dynamic paramcter selection for each input (Jacobs ct al. 1991 Eigen et al. 2013 Shazcer et al., 2017
Lepikhin et al.) 2020: Fedus al. 2022 Jiang et al. 2024 Sukhbaatar et al.) 2024). Recent efforts
have adapted MoF. for multi-modal tasks, addressing the challenges posed by inherent feature space gaps
between modalities (Wang et al. 2022 Shen et al.) 2023 [ao et al. 2022a Long et al. 2023 Lin et al,, 2024). These approaches suggest that modality-specific parameter allocation can improve performance by addressing distinct data type (i.e., modality) characteristics (Liang et al. 2022]. Different from previous works, in this paper, we propose the Cecttrre-o-Trranformne (MoT) framework, which generalizes the MoE concept by decoupling all non-embedding parameters within the transformer architecture. MoT consistently outperformed MoE in molti-modal pretraining when the amount of total parameters is controlled (Figure 16] and demonstrated complementarity with MoF-4x (Figure 17
While recent works have extended MoE beyond feedforward layers to attention mechanisms (Wagg et al. 2023 Shen et al.) 2024 Liu et al. 2024b), our approach differs in several key aspects. Unlike Cog M [Wang et al. 2023) which is limited to generating text outputs, Mo' is capable of both image and text generation. Concurrent to our work, Playground v3 (PGv3) (Liu et al. 2024b) integrates a DiT-style image transformer with Llama3-8B as the text backbone using global self-attention, and achieves stale-of-the-art performance in text and image generation. During training, the text LLM is frozen and only the image transformer component is updated. While both CogVLM and PGv3 conduct multi-modal training top of pre-trained LLM, we establish MoT a general sparse architecture that can be trained from scratch. MoT also decouples every non-embedding parameter across transformer layers, including layer normalization, whereas previous approaches maintain shared layernorm parameters. Our findings position MoT as a flexible and scalable solution for multi-modal pretraining, demonstrating its potential to complement MoE-bason architectures and offering pathway for more computationally efficient large-scale multi-modal models.
8 Conclusion
In this work, we present Mixturo-of-Tranann Transformers (MoT), a sparse and scalable architecture designed to address the computational challenges of multi-modal model pretraining. By decoupling non-embedding parameters by modality and retaining global self-attention across multi-modal sequences, MoT optimizes modality-speeiffic processing while preserving cross-modal interactions. Our experiments demonstrate that MoT achieves significant reductions in training costs across various settings and model scales. In the Chameleon and Chameleon+Speecch settings, MoT matched or exceeded the performance of dense base-
33

---

Published in Transactions on Machine Learning Research (04/2025)
lines while using substantially fewer FLOPs. Furthermore, MoT maintained these improvements in more complex setting (Transfusion), where distinct training objectives were applied to different modalities, demon- strating consistent efficiency gains and enhanced performance in tasks such as image generation. In addi- tion to FLOP reductions, system profiling highlights the practical benefits of MoT, inchuding reductions in wall-clock time for both text and image tasks. When scaled across GPUs, MoT demonstrated further im- provements, indicating its suitability for large-scale distributed training environments. Preliminary results combining MoT with Hixture-offExpert (MoE-4x) suggest the potential for hybrid models that further improve performance without increasing computational costs. These findings suggest that MoT could serve as an effective framework for future multi-modal LLMs, enabling more efficient large-scale training while maintaining competitive performance across diverse modalities.
References
Achieve 23x LLM Inference Throughput & Reduce p50 Latency anyscale.com. https://www. anwscale............
can///llg/contttmmoussttaaaaaagp haaaachhhhnn------ffeeeeeeee [Accessed 22-03-2025).
resource_utiiization Loaiiaal/CCncpppuaalaa Guide/Part 2-improving resource utilization at main triton-inference-
Cuide/Partt22impovvvnn
server/tutorials gom.
is dynamic-batching main Sain/Conceptual...m Satpeptuaa_ddddePaaaaeaa wttpaaa/atttttttttpapoooooeeo http://:ww...uoooooooooooooooooa otg.aaaiiaaaaaiioooooooooooaaaaa
Accessed 22-05.2025).
Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, et al. Cm3: A causal masked multimodal model of the internet. arXiv preprint orXiv2201..07500, 2022.
Emanuele Aiello, Lili Yu, Yixin Nie, Armen Aghajanyan, and Barlas Oguz. Jointly training large autore- gressive multimodal models, 2023. URL https::/arxiv.coo org. atggabs/2309..... 15564
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Autoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 23736, 2022.
Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, et al. Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation. 2024.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint ar Xiv:2106. 08254, 2021.
Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. Vlmo: Unified vision-language pre-training with mixture-of experts. Advances in Neural Information Processing Systems, 35:32897 32912, 2022a.
Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, and Furu Wei. Vlmo: Unified language pre-training with ooddurr-- http:://arxxv..orr//a///11111 02358
2022b. URL
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models, 2024. URL https: btops://arxizn
org/
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Concept 12m: Pushing web-scale
image-text pre-training to recognize tail visual concepts. CoRR, abs/2102.08981, 2021.
URL https:
Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhosciny. Visualgpt: Data-cfficient adaptation of pretrained language models for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18030-180480 2022.
34

---

Published in Transactions on Machine Learning Research (04/2025)
Ann Clifton, Sravana Reddy, Yongze Yu, Aasish Pappu, Rezvanch Rezapour, Hamed Bonab, Maria Eskevich, Gareth Jones, Jussi Karlgren, Ben Carterette, and Rosie Jones. 100,000 podcasts: A spoken English document corpus. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 5903-5917, Barcelona, Spain (Online), December 2020. International Committce on Computational Linguistics. URL https: htops://wwww... h/tp://wwwwwbbb....//./ aclweb. org httg//nthologgy222222 wttp:///........oooooo mellinng-miin.5 .59
Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint ar xrXi:2401.06000 2024.
DeepSeck-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Danai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawci Wang, Jin Chen, Jingchang Chen, Jingvang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang. Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia Hc, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Deepseck-v3 technical report, 2025. URL RRLLhhtps::// / /axiiv..vvvv org/ httg/ab//2412.... /.t../2.....999......
David Figen, Marc' Aurelio Ranzato, and Tlya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1812.431, 2013.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the LEEE/CVE conference on computer vision and pattern recognition, pp. 12873 12883, 2021.
Patrick Essor, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis, 2024. URL
William Fedus, Barret Zoph, and Noam Shazcor. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022. URL
03961
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. arXiv preprint arXiv:2208. 18131, 2022.
Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. Megablocks: Efficient sparse training with mixture-of-experts, 2022. URL https: wrg. 2211 15841 1281.15841
35

---

Published in Transactions on Machine Learning Research (04/2025)
Daniel Galvez, Greg Diamos, Juan Ciro, Juan Felipe Cerón, Keith Achorn, Anjali Gopi, David Kanter, Maximilian Lam, Mark Mazumder, and Vijay Janapa Reddi. The people's speech: A large-scale diverse english speech recognition dataset for commercial usage, 2021. URL attps///arri..orr//ab///11111 wrg/abs///11..... 09344
Dhruba Ghosh, Hannanch Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36, 2023.
Wanggui He, Siming Fu, Mushui Liu, Xierui Wang, Wenyi Xiao, Fangxun Shu, Yi Wang, Lei Zhang, Zhelun Yu, Haoyuan Li, et al. Mars: Mixture of auto-regressive models for fine-gres tne-grained text-to-image synthesis. arXiv preprint arXic:2407.07614 2024.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two lime-scale update rule converge to local nash equilibrium. Advances in neural infor- mation processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXie:2207.12500 2022.
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2icomppench:: A comprehensive bench- mark for open-world compositional text-to-image generation. arXiv preprint :XXw:2307.06950 2023.
Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural compulation, 3(1):79-8) 87, 1991.
Albert Q- Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024. URL https: //arxiv org/ abs//401...2.. 2401.44088
J. Kahn, M. Rivière, W. Zheng, E. Kharitonov, Q. Xu, P.E. Mazaré, J. Karadayi, V. Liptchinsky, R. Col- lobert, C. Fuegen, T. Likhomanenko, G. Synnaeve, A. Joulin, A. Mohamed, and E. Dupoux. Libri- light: A benchmark for asr with limited or no supervision. In ICASSP 2020 2020 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7669-7673, 2020. doi: C..109/ICCSSP40070 001.7766.220090599440
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arxiv preprint ar xxXi::1312.6114, 2013.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2022. URL https://arxiv.org
hbss/332..612.
Hugo Laurengon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M Rush, Douwe Kiela, et al. Obelisc: An open web-scale filtered dataset of interleaved image-text documents. arXie preprint arXiv:2306. 16527, 2023.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding, 2020. URL https://arxiv.. wrwpii..rrg/bbba herg/aba/2000 16668
Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 35:17612 17625, 2022.
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740 755. Springer, 2014.
36

---

Published in Transactions on Machine Learning Research (04/2025)
Xi Victoria Lin, Akshat Shrivastava, Liang Luo, Srinivasan Iyer, Mike Lewis, Gargi Gosh, Luke Zettlemoyer; and Armen Aghajanyan. Moma: Efficient early-fusion pre-training with mixture of modality-aware experts. arXiv preprint arX.i::207..11770 2024.
Alexander H. Liu, Heng-Jui Chang, Michael Auli, Wei-Ning Hsu, and James R. Glass. Dinosr: Self-distillation and online clustering for self-supervised speech representation learning, 2024a. URL https:/aaxxi..orrg
10005
Bingchen Liu, Ehsan Akhgari, Alexander Visheratin, Alcks Kamko, Linmiao Xu, Shivam Shrirao, Chasc Lambert, Joao Souza, Suhail Doshi, and Daiqing Li. Playground v3: Improving text-lo-image alignment with deep-fusion large language models, 2024b. URL https://arxiv.. http://axxxv..rg/bbb/aaaaaaaa/a/
10695
Hao Liu, Wilson Yan, Matei Zaharia, and Picter Abbecl. World model on million-length vidco and language with blockwise ringallention. aixi preprinl arXiu:2402.08266 2024c.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint ar wrxxx:2300003111 2023.
Llama team. The llama 3 herd of models, 2024. URL http:://arat...rr//a/////.11111
Zijum Long, George Killick, Richard McCreadie, and Gerardo Aragon Camarasa. Multiway-adapater: Adapt- ing large-scale multi-modal models for scalable image-text retrieval. arXiv preprint arXiv:2309.01516, 2023.
Liang Luo, Jacob Nelson, Luis Ceze, Amar Phanishayee, and Arvind Krishnamurthy. Parameter hub: a rack-scale parameter server for distributed deep neural network training. In Proceedings of the ACM Symposium on Cloud Computing, SoCC '18, pp. 41 54, New York, NY, USA, 2018. Associa- tion for Computing Machinery. ISBN 4701450360111. doi: 10.1145/3200800.322777 32007 URL https:
7/doi. org/10. 11111114677899322222
Liang Luo, Peter West, Jacob Nelson, Arvind Krishnamurthy, and Luis Ceze. Plink: Dis- covering and exploiting locality for accelerated distributed training on the public cloud. T. Dhillon, D. Papailiopoulos, and V. Sze (eds.), Proceedings of Machine Learning and Systems, vol- ume 2, pp. 82 97, 2020. URL https://proceedding soneiiiinngmmsy.........m org/ htgg/papr__aee files/ppppr/2200/fi1ee 7aa96665bb0033000122777772266
pdf
Liang Luo, Buyun Zhang, Michael Tsang, Yinbin Ma, Ching-Hsiang Chu, Yuxin Chen, Shen Li, Yuchen Hao, Yanli Zhao, Guna Lakshminarayanan, Ellie Wen, Jongsoo Park, Dheevatsa Mudigere, and Maxim Naumov. Disaggregated multi-tower: Topology-aware modeling technique for efficient large scale recommendation. In P. Gibbons, G. Pekhimenko, and C. De Sa (eds.), Proceedings of Machine Learning and Systems, volume 6, pp. 266 278, 2024. URL htttps://proceedinngg wwwpeesssssssssaaeeeeeeeeeeeeeee org/paper www.:/s.orrr/ppprrrr.eeee//////s filess/paper/222///ii 7B524333ec2200144662222266260663
Paper- hatps:3222eeaaeeeeeeee----eeeeee
Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafford, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kicla, Ali Farhadi, Noah A. Smith, Pang Wei Koh, Amanpreet Singh, and Hannaneh Hajishirzi. Olmoe: Open mixture-of-experts language models, 2024. URL https://arxiv. http://aaxivv.../abbbaa/ org/abb//2499.00063
Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R. Costa-jussa, Maha Elbayad, Sravya Popuri, Paul- Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, Gabriel Synnaeve, Juan Pino, Benoit Sagot, and Emmanuel Dupoux. Spirit-Im: Interleaved spoken and written language model, 2024. URL
https://arxiv. www/absg////oooooooooo..oo....rt 22022020255572
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models.
International conference on machine learning, pp. 3162-8171. PMT.R, 2021.
37

---

Published in Transactions on Machine Learning Research (04/2025)
Nvidia. Introducing
grouped updates nvidia technical
gemm apis in blog.
cublas and
more performance
https://developprt .vidia.. com/blog/ /
seredducinnggroupeer gemm- -ewneagie--aaaaaae-- 10/04/2024).
and moreammoreeeee peeree-eerfoooe perrrepprrrrmaaae--uddeeteee
(Accessed on
Bryan A Plummer, Liwci Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image- toseeteoeeee models. In Proceedings of the IEEE international conference on computer vision, pp.
2015.
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Yhih Ma, Ching-Yaa Yao Chuang, David Yan, Dhruv Choudhary, Dingkang Wang, Geet Sethi, Guan Pang, Haoyu Ma, Ishan Misra, Ji Hou, Jialiang Wang, Kiran Jagadeesh, Kunpeng Li, Luxin Zhang, Mannat Singh, Mary Williamson, Matt Le, Matthew Yu, Mitesh Kumar Singh, Peizhao Zhang, Peter Vajda, Quentin Duval, Rohit Girdhar, Roshan Sumbaly, Sai Saketh Rambhatla, Sam Tsai, Samaneh Azadi, Samyak Datta, Sanyuan Chen, Sean Bell, Sharadh Ramaswamy, Shelly Sheynin, Siddharth Bhat- tacharya, Simran Motwani, Tao Xu, Tianhe Li, Tingbo Hou, Wei-Ning Hsu, Xi Yin, Xiaoliang Dai, Yaniv Taigman, Yaqiao Luo, Yen-Cheng Liu, Yi-Chiao Wu, Yue Zhao, Yuval Kirstain, Zecheng He, Zijian He, Albert Pumarola, Ali Thabet, Artsiom Sanakoyeu, Arun Mallya, Baishan Guo, Boris Araya, Breena Kerr, Carlcigh Wood, Ce Liu, Cen Peng, Dimitry Vengertsev, Edgar Schonfeld, Elliot Blanchard, Felix Juefci- Xu, Fraylie Nord, Jell Liang, John Hoffman, Jonas Kohler, Kaolin Fire, Karthik Sivakumar, Lawrence Chen, Licheng Yu, Luya Gao, Markos Georgopoulos, Rashel Moritz, Sara K. Sampson, Shikai Li, Simone Parmeggiani, Steve Fine, Tara Fowler, Vladan Petrovic, and Yuming Du. Movie gen: A cast of media foundation models, 2024. URL fttpp:://axxi..rr/abbb... 13720
Vineel Pratap, Qiantong Xu, Amuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. Mls: A large-scale multilingual dataset for speech research. ArXiv, abs/2012.000
2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103, 00020, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. In JMLR, 2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and llya Sutskever. Zero-shot text-to-image generation. arXiv preprint arXix:2102. 12002, 2021.
David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam San- toro. Mixture-of-deephhs: Dynamically allocating compute in transformer-based language models. arXiv preprint (rXie:2404.02258, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition, pp. 10684 10441,,055 2022.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E, Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture--f-experts layer. CoRR, abs/1701.06538, 2017. URL
wttp:///wwww///bb/o/..0.000. 06538
Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrell, Kurt Keutzer, and Yuxiong He. Scaling vision- language models with sparse mixture of experts. arXiv preprint arXiv:2303.072: 07226, 2023.
Yikang Shen, Zhen Guo, Tianle Cai, and Zengyi Qin. Jetmoe: Reaching Ilama2 performance with 0.1 m dollars. aix preprint arXiv:2444.07413, 2024.
Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Rozière, Jacob Kahn, Daniel Li, Wen tau Yih, Jason Weston, and Xian Li. Branch mrxin-mix: Mixing expert 1lms into a mixture-of-experts 11m, 2024. URL https://arxxi: http:/:////axxwooo////////////// atrg/bbs/4403... 07816
38

---

Published in Transactions on Machine Learning Research (04/2025)
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXim:2302.1399 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and finc-tuned chat models. Xiv preprinl or orXiv:2307009888 2023b.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-basse image description evaluation. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 4566- 4575, 2015.
Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. VoxPopuli: large-scale multilingual speech corpus for represenlation learning, semi-supervised learning and interpretation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 993-1003, Online, August 2021. Association for Computational Linguisties. doi: 10.18653/v1 /2021 Lacl-long.80. URL htttss:/acclattholo.go arwp://..//.2....11ooo...... 80
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:231100077) 2023.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as foreign language: Beit pretraining for all vision and vision-language tasks, 2022. URL https: org wwtp://////...../a/./// 10442
Gyeong-In Yu, Joo Seong Jeong, Geon- Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orea: distributed serving system for Transformer-Basse generative models. In 16th USENIX Symposium on Operating Sys- tems Design and Impiementation (OSDI 22), pp. 521-538 Carlsbad, CA, July 2022. USENIX Association. ISBN 978-1-939133-28.1 28-18-1. URL https://wwe wwwp///wwww... usenix. ottp://oontrrreeee//a//e//eeaaaa
Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et al. Scaling autoregressive multi-modal models: Pretraining and instruction tuning. arXiv preprint ar xrXi::2309.022 02501, 2023.
Kai Zhang, Lingho Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for intaructton-guieed image editing. Advances in Neural Information Processing Systems, 36, 2023.
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch fsdp: Experiences on scaling fully sharded data parallel. Proc. VLDB Endow., 16(12):3848 3860, August 2023. ISSN 2150-8097. doi:
10.14778/388 000///666111100000 1569. URL https:://oo: org/ 10. 147.1444777661551 3611569
Zexuan Zhong, Mengzhou Xia, Danqi Chen, and Mike Lewis. Lory: Fully differentiable mixture-of-experss for autoregressive language model pre-training. In First Conference on Language Modeling, 2024. URL
httpss//opeereviiz
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kalm, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint ar arxi:22408. 1 11039, 2024.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoe Le, and James Laudon. -fxture-ofeexperts with expert choice routing, 2022. URL https: //ar:iv../x... org/ ab3//222/220..... 09368
39

---

Published in Transactions on Machine Learning Research (04/2025)
A Tranfusion: Preliminaries
A.1 Diffusion for Image Generation
Diffusion models have emerged as a powerful class of generative models capable of producing high-fidelity data across various modalities. These models utilize a Markov chain that progressively adds Gaussian noise to data a forward process and then learns to reverse this process to generate new data samples.
In the forward diffusion process, the data Xo is perturbed over T timesteps by sequentially adding Gaussian noise. The transition from X,-1 to x2 is defined by the conditional probability distribution: g(x, xx-1) = (xx; X,-1, (1 (t)T), where a2 E (0, 1) controls the rate of noise addition at each timestep t. The cumulative product of a up to timestep is denoted by ā. = II' ag.. Using this notation, we can express x, directly in terms of the original data Xn: X, Vā X + vT- C4 6, where N/(0,) I) is standard Gaussian noise. As & approaches T,, the data distribution transitions towards isotropic Gaussian distribution.
In the reverse diffusion process, the goal is to recover the original data Xo from the noisy observation XT by iteratively denoising. The reverse process is parameterized by a neural network Eg(X,, t, c) trained to predict the added noise at each timestep, where c is extra context, such as text prompt. The denoising step can be expressed as: Xt-1 1 added during the reverse step, and z (O, is auxiliary noise introduced for stochasticity in the sampling,
1-a Va: (xt vi-a: Es ex,,,, t)) 2, where,o, is the standard deviation of the noise
The neural network E, is trained by minimizing the objective function LDDPM = Exo.e,t [||e co(x,, t. c) e) Once optimized, a new data point Xa can be sampled by initializing x ~ (O), following the above denoising steps.
In this work, we use cosine scheduler Nicchol & Dhariwal 2021 to set the value of az- We show qualitative image generation results from the 7B model. In this case, we use Classifier-free guidance (CFG) (Ho & Salimans 2022) to improve generation by contrasting the prediction of the model conditioned on the context c with the unconditioned prediction.
To reduce computational requirements, we adopt latent diffusion models (LDMs) (Rombach el al. 2022 which perform the diffusion process in lover-dimensional latent space (e.g. represent every 8x8 pixel patch first
8-dimensional vector.) rather than directly in the high-dimensional data space. Specifically, encode the original data Xy into latent representation Zu using a Variational autoencoders (VAEs) (Kingma & Welling, 2013)) ). The diffusion (forward and reverse) process is then applied to 2o, significantly reducing computational cost due to the lower dimensionality of the latent space. This approach allows efficient training and sampling while preserving the quality and fidelity of the generated multimodal outputs.
A.2 Transfusion Model Architecture
The model primarily consists of a single transformer to processes the combined sequence regardless of modal- ity. We follow Llama's architecture [Touvron et al. 2023a to build transformer layers, and add lightweight modality-specific module to map the inputs into a shared high-dimensional vector space Rd. For text, em- bedding matrices convert input integers to vectors and output vectors back into token probabilities. For images, we employ U-Net to compress local windows of 2 2 patch vectors in VAE latent space into single vectors suitable for the transformer (and vice versa). In this setting, image is represented as 256 continous tokens. The transformer uses a hybrid attention mechanism: causal attention is applicd across the entire sequence to preserve the autoregressive property, while bidirectional attention is used within each image to capture intra-image dependencies. This means that image patches can attend to all other patches within the same image but only to preceding tokens or image patches outside their own image.
The model is trained by minimizing a combined loss function:
Lranssusioo LLM + X LDDPM,
(4)
where À is a balancing coefficient. The language modeling loss LLM computed per token, encouraging the model to predict the next token in the sequence. The diffusion loss LDDPM is computed per image. We set
40

---

Published in Transactions on Machine Learning Research (04/2025)
the À coefficient in the Transfusion objective to following preliminary experiments; leave further tuning of À to future work.
During inference, the model alternates between language modeling and diffusion sampling modes. In language modeling mode, it generates text by sequentially sampling tokens from the predicted probability distribution. When a beginning of image (BOT) token is generated, the model switches diffusion mode. In this mode, pure noise Xy appended to the input sequence as a series of image patches corresponding to the desired image size. The model then iteratively denoises this input. Once the diffusion process concludes, of image (EOI) token is appended to the sequence, and the model returns to language modeling mode. At the mcan time, an image is generated using a VAE decoder. This scamless switching mechanism allows Transfusion to generate sequences containing any mixture of text and images, leveraging shared parameters and modality-speciiff processing within a unified architecture.
end
B MoT Transfusion Fine-tuning Results
START
Dense
START
MOT
(a) A photo of person with the head of a cow, wearing a tuxedo and black bowtie. Beach wallpaper the background.
(b) the word 'START" on a blue t-shirt.
(c) translucent pig, inside is a smaller pig.
(d) A single beam of light enter the room from the ceiling. The beam of light is illuminating an easel. On the easel there is Rembrandt painting raccoon.
Figure 20: Example easy prompts
We compare the image generation capabilities of fine-tuned TransFusion MoT and dense models by prompting them with variety of text imputs, as illustrated in Figures 21 20 and 22 In Figure 21 both MoT and dense fine-tuned models succossfully follow the prompts. However, in Figure 20 the MoT fine-tuned model demonstrates superior performance, producing images that are either more visually appealing or more faithful the prompts. In Figure 22, both models struggle to perfectly follow the text prompts and fail to capture all the details accurately. Our study suggests that text faithfulness can greatly improve with extended training and we leave it future work to scale up training with bigger model and more data.
41

---

Published in Transactions on Machine Learning Research (04/2025)
Dense
MOT
(a) Photo of rrheeeinsspired spherical chair, with bumpy white exterior and plush interior, set against a tropical wallpaper.
(b) tilt shift aerial photo of a cute city made of sushi on wooden table the evening.
(c) Film still of a long-legged cute big-eye anthroponorphiic cheeseburger wearing sneakers relaxing the couch a sparsely decorated living room.
(d) A chrome-plated duck with a golden beak arguing with an angry turtle forest.
Figure 21: Example prompts where MOT yields better image generation than Dense
Dense
Jot edtiii ddo.
MOT
(a) An illustration of an avocado sitting therapist's chair, saying 'I just feel so empty inside' with pit-sized hole in its center. The therapist, spoon, scribbles notes.
(b) A photo of a crocodile made water.
(c) A dsir picture of colorful graffiti showing hamster with moustache.
(d) espresso machine that makes coffee from human souls, high-contrast painting.
Figure 22: Example hard prompts
42

---

Published in Transactions on Machine Learning Research (04/2025)
C
Supplementary Figures
(a) 4% checkpoint, Layer
(b) 4% checkpoint, Layer 5
(c) 4% checkpoint, Layer 17
(d) checkpoint, Layer
(e) 24% chockpoint, Layer
(f) 24% chockpoint, Layer
(g) 24% chockpoint, Layer 17
(h) chockpoint, Layer 32
(i) 50% checkpoint, Layer 1
(i) 50% checkpoint, Layer
(k) 50% checkpoint, Layer 17
(1) 50% checkpoint, Layer 32
(m) 100% checkpoint, Layer (n) 100% checkpoint, Layer (o) 100% checkpoint, Layer 17 (p) 100% checkpoint, Layer 32
Figure 23: Visualization of latent feature space for Chaneelenn+Speech 7B Dense model across training checkpoints and layers. Principal Component Analysis (PCA) of model activations shows clustering by modality (text, speech, image) at different stages training (4%, 24%, 50%, 100% checkpoints) and across layers (Layer 1, Layer 5, Layer 17, Layer 32). The PCA plots show that different modalities consistently occupy distinct regions of the feature space. This natural clustering highlights the inherent differences between modalities, suggesting that they processed differently by the model. These findings architecture, where modality
motivate the need for decoupled weights in our parameters can better capture and leverage the distinct statistical properties of each modality, leading to improved performance and efficiency compared to a dense baseline.
43

---

Published in Transactions on Machine Learning Research (04/2025)
10090
MoE is 2.750)
Deniee
103000
MoE 0.827)
Valldton
Danse MoT lours)
32000
Mof-2x is 1.549)
MoB-3x 1.567)
MoT lours)
Mofi-2x
1900
MoE-2x (5- 0.746)
Mot--xx (s 0.637)
McE 2x
(oT
MoT lours! 0.597)
Mct-4x
MoE 4x
2930
1000
350900
2008
15000
Training Steps
Training Step for Dense
Trainnng Stops
Training Step for Dence
37M Image Eval Loss
(b) Image Loss Matching
(c) Text Eval Loss
(d) Text Loss Matching
150000
Mo 1.
Dense
Mot-2x is 0.834)
Vallatto USS
47-
Danse
MoF-3x 1.050)
MeE
McE: 2x
(ours)
120000
MoF-3x (s 0.6355
lours! (s= 0.570)
MoE: 2x
McT (ours) is- 0.225)
McE 3X
7500
MoE 4x. (s 0.557)
McE 3x
Stops
Sheps
MoE
20000
18000
Steps
Triining Step
Thaining
94M Image Eval Loss
(f) Image Loss Matching
(g) Text Eval Loss
(h) Text Loss Matching
McE (x 0.767)
22
Steps March Derie
(5 1.301)
Valozen Los4
Deriee
125000
McB-3x (5 0.747)
Dense
MoE:3x (s- -0.973)
MoT lours)
Mol-4x 0.694)
MoT (purs)
MoF-Ax (x- 0.7661
MoE
(4TT(ours)
MoE
(s-
x2
MoF-3x
5000
MoF-3x
MoE 4x
2000
MoE 4x
Thaining Steps
Training Stop for Dense
Traningg Steps
Training Step for Dence
(i) 443M Image Eval Loss
(i) Image Loss Matching
(k) Text Eval Loss
(l) Text Loss Matching
Valtatton Lo32
Denie
2000
McE 3x Es 0.853)
0.8561
McT (ours)
Dense
50ap Masco Denee
12000
MoE: 3x (s- 0.819)
(s
MoT (ours)
McE:2x (s 0,747)
24
McE-2x
MoE: MoE-Ax 0.738)
MoE 2X
15-
McE 3K
MoT (s-2))))
Mol-3x -3x
2003
L20000
Thaining Steps
Triniig Step Dense
Trining Steps
Training Shep for Derne
(m) 1.5B Image Eval Loss
(n) Image Loss Matching
(o) Text Eval Loss
(p) Text Loss Matching
Vaitation car
McT 0.431)
MoE-4 1,187)
Wanuue Losa
McT (ours)
Dense
2008
MoT 0.553)
MoE-tt (s (601)
1100 Mrath
Dense MoT (ours)
MoE-4x
2000
Traning 120000
Training Steps
Traninn Dense
Trairing Step
(q) 7B Image Eval Loss
(r) Image Loss Matching
Text Eval Loss
(t) Text Loss Matching
Figure 24: Training and validation losses for image and text modalities across model scales (37M, 94M, 443M, 1.5B, 7B) in the Chameleon setting evaluated on the Obelisc dataset. For the image modality, MoT consistently delivers a substantial speedup relative to the dense model and MoE- 4x, with the advantage growing across scales. In contrast, MoE-4x exhibits diminishing gains as the model scales increase, particularly at 7B, where the benefits disappear in the image modality. In the text modality, both MoT and MoE-4x outperform the dense model, with MoT demonstrating comparable or slightly better performance. FLOOs-controlled across all runs in the same model scale and pre-trained from scratch.
44

---

Published in Transactions on Machine Learning Research (04/2025)
2.675)
Denie
0.635)
1X00
(ours) (s-m 0.227)
Jatoationsss
Mot lours)
MoT touns) (s- 0.528)
Valdzzon
Steps Marc
13000
5090
50
McT teurs)
5300
-oE
2000
Thgining Steps
Training Step for Dense
Taining Steps
Training Step for Dence
37M Image Eval Loss
(b) Image Loss Matching
(c) Text Fval Loss
(d) Text Loss Matching
Volostoon LoGS
123900
Mot-4x 1.362)
MoT lours)
Dense
15000
Mol--x (s 0.555)
jours]
McE 4x
Worch
Dense MoT
75000
7000
Thaining Steps
Thiining Step for Dense
Thaining Steps
Training Step
(e) 94M Image Eval Loss
(I) Image Loss Matching
(g) Text Eval Loss
(h) Text Loss Matching
McE 4x 0.790)
Dense
MoE 4x (s= 0.7581
UNi200 1000000on. Lon
123000
McT (oTrsu 15 0.202)
McT (ours)
12500
MoT tours) (s= 0,577)
McF-4x
Shept March
Dense
75000
MoT (ours)
5000
MoE-4x
150000
Training
Training Step for
Traiinn
(i) 443M Image Eval Loss
(i) Image Loss Matching
(k) Text Eval Loss
Text Loss Matching
McE-4x 0,789)
Dense
[ours] (x- 0,343)
MoT (ours)
MoT (ours) 0.606)
Valisaation
McE-4x
Dente (ours)
2008
30000
120095
Thiining Reps
Thaiiing
Thiiiing Step
(m) 880M Image Eval Loss
[n) Image Loss Matching
Text Eval Loss
(p) Text Loss Matching
Wilderen Love
MBE 4x (s 0,9761 tourst (s 0,369)
McT (ours)
Dense
MoE Ax 0.7871
Mot (ours) 0.6301
McE
Dense
MoT (ours)
4033
MoE-4x
Trainig Steps
Traningg Dense
Training Staps
Training Step for Derne
(q) 1.5B Image Eval Loss
(r) Image Loss Matching
Text Eval Loss
(L) Text Loss Matching
Vollatton Lose
2000
MoT [ours] (s- 0.402)
McE-4x L.060!
Sanaaann Losa
MoT lours) McE-4x
Dense
SUMEE to Match Daroe
MoT (cuure) (s- (.536)
(2-
Dense MoT
MoF-4x
1300
120080
338009
Thaining Steps
Triiiin Step Dense
Triining Steps
Thaining Step Demee
(u) 7B Image Eval Loss
(v) Image Loss Matching
(w) Text Eval Loss
(x) Text Loss Matching
Figure 25: Training and validation losses for image and text modalitics across model scales (37M, 94M, 443M, 1.5B, 7B) in the Chameleon+Speech setting evaluated on the Obelisc dataset. MoT exhibits consistent and significant improvements validation loss for the image and text modalities, demonstrating its efficiency and robustness across scales. FLOPs-controlled across all runs in the same model scale and pre-trained from scratch.
45

---

Published in Transactions on Machine Learning Research (04/2025)
1000
MuT FeaRS ta- 6055)
Denie
2:14
3141
0012
503
Tuiling Pao Dene
(L) 163M Text Training
(2) Text Training Loss Matching
(3)
163M Text Val. (4) Text Val.
Loss:
(5) Captioning
Eval:
loss
Loss: C4 (4)
Wikipedia (4)
CIDEr (t)
1000
round 009
Medt 15- (u8t))
De 10
1000
201-
(6) 760M Text Training Matching
(7) Text Training Loss
(8)
760M Text Val. (9) Text Val.
Loss:
(10) Captioning Eval: CIDEr (t)
loss
Loss:
C4 (1)
Wikipedia
(4)
2218
15 0.390
NoT McTibural
Takiing
(11) L.4B Text Training loss
(12) Text Training Loss
(13) 1.4B Text
Val.
(14) Text Val. Wikipodia (1)
Loss:
(15) Captioning
Eval:
Matching
Loss: C4
CIDE: (f)
Figure 26: Text -odelityyspeecffic training loss and step matching plots in Transfusion setting across model scales. For text modality, MoT matches dense model in training and validation loss on C4 and Wikipedia datasets, with improved generalization in captioning tasks (CIDEr score). MoE-4x shows unstable performance: lower training losses but poorer generalization than dense model on text evaluation metrics. Model sizes for sparse models indicate activated parameters. All experiments FLOPs-controlled and pre-trained from scratch.
46


Reviews:
[{"review_id": "xiN0narLc4", "reviewer": "Reviewer_oCiV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. **Efficient Sparse Architecture**: Decoupling transformer components by modality is straightforward yet highly effective. The global self-attention mechanism still supports cross-modal interactions.\n2. **Extensive Experiments**: The authors validate MoT across multiple settings (Chameleon, Chameleon+Speech, Transfusion) and multiple model scales, offering strong evidence of consistency and scalability.\n3. **Real-World Throughput Gains**: Beyond theoretical FLOP savings, the paper includes system profiling that shows significant wall-clock time reductions compared to dense baselines.\n4. **Ablations & Analysis**: Clear ablations (e.g., which transformer components are untied) and results on combining MoT with MoE give additional insights into architectural choices.\n\n**Weaknesses**\n1. **Cross-Modal Reasoning**: While global self-attention in MoT naturally supports cross-modal interaction, it would be interesting to see if more complex cross-modal fusion layers or specialized heads would further improve tasks requiring deeper reasoning (e.g., multi-step visual QA).\n2. **Production Deployment**: The paper’s system analysis is strong, but real-world deployments can face variable batch sizes and latency-sensitive requirements. More details on how MoT handles highly variable or multi-modal streaming inputs in production would enhance practical utility.\n3. **Long-Range Cross-Modal Tasks**: The paper demonstrates standard benchmarks and tasks like image generation and captioning. It might be valuable to consider long-range tasks (e.g., audio-visual alignment or multi-step referencing in text+image+speech).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Mixture-of-Transformers (MoT), a sparse architecture for multi-modal foundation models. The key idea is to decouple non-embedding parameters by modality while applying global self-attention across all tokens in a unified sequence. This design addresses the high FLOPs demands of multi-modal training by selectively activating modality-specific parameters (feed-forward layers, attention projections, and layer norms), thereby reducing computation. \n\nThe work thoroughly evaluates MoT in three settings:\n\n1. Chameleon\n2. Chameleon+Speech\n3. Transfusion\n\nExperiments at multiple model scales consistently show that MoT matches or exceeds the performance of dense baselines while requiring substantially fewer FLOPs. The paper also provides system profiling to demonstrate training speedups in wall-clock time. Additionally, there are discussions on combining MoT with Mixture-of-Experts (MoE) and ablations on modality separation in different parts of the transformer (FFNs, attention matrices, layer norms).\n\nOverall, the paper contributes:\n- A new multi-modal sparse design with global self-attention.  \n- Comprehensive experimental evaluation.  \n- System-oriented analysis for communication overheads and practical throughput benefits.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. **Extended Cross-Modal Experiments**:  \n   - Consider testing MoT on more challenging cross-modal tasks (e.g., reasoning-heavy VQA or multi-turn dialogue tasks) to confirm that global self-attention alone is sufficient or to see if specialized cross-modal fusions improve performance.\n\n2. **Implementation Details for Production**:  \n   - Provide a deeper discussion on dealing with variable sequence, dynamic batch sizes, and latency constraints.  \n   - Comment on the cached indexing or potential specialized GEMM kernels to optimize the overhead from modality-based partitioning.\n\n3. **Inference-Time Profiling**:  \n   - Expand on how MoT’s multi-modal decoupling impacts inference speed under real-world conditions (e.g., single-stream vs. batch inference, mixed-modality requests).", "broader_impact_concerns": "No immediate ethical red flags beyond standard concerns for generative models are evident.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jnt12rz5Lk", "reviewer": "Reviewer_fTvF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "• Strengths:\n\n◦Novel Architecture: The MoT architecture is a novel and significant contribution to the field of multi-modal models, presenting a distinct approach from prior methods, and is shown to outperform MoE models while using fewer FLOPs.\n\n◦ Modality-Aware Sparsity: The core concept of modality-aware sparsity is well-motivated by the inherent differences in how modalities are processed, and is supported by PCA analysis of latent feature spaces, which show distinct clustering by modality.\n\n◦ Scalability and Efficiency: The MoT architecture demonstrates significant efficiency gains by achieving comparable or better performance than dense models while using fewer FLOPs. The paper also shows that the benefits of MoT increase with the number of GPUs during training.\n\n◦ Strong Experimental Results: The paper provides strong empirical results across a variety of model sizes and multi-modal tasks, including text, image, and speech generation, showing that MoT is versatile for large-scale multi-modal generative tasks.\n\n◦ Comprehensive Analysis: The inclusion of ablation studies, a leave-one-out analysis, and exploration of combining MoT with MoE provide a thorough understanding of the MoT architecture and its potential.\n\n◦ Clear Presentation: The paper presents complex information in a clear and well-structured manner. Figures and tables are effective for communicating results and key concepts.\n\n•Weaknesses:\n\n◦ Limited Generalization: While MoT shows strong performance gains, the paper notes that MoE-4x's performance can be unstable when the evaluation data distribution differs significantly from the training data, which may lead to uneven token distribution among experts. This highlights the potential for MoT to also struggle with more significant data distribution changes.\n\n◦ Inconsistent MoE Performance: The paper mentions that MoE-4x shows mixed performance across all scales, sometimes underperforming the dense baseline in speech validation loss despite outperforming it in pre-training loss. This suggests a need for further investigation into why MoE performance is unstable.\n\n◦ Text Performance: The text performance of MoT in the Transfusion setting shows little improvement across scales. This is a potential area for future research.\n\n◦ Parameter count: Although the model shows great improvements in training with fewer FLOPs, parameter counts are not explicitly compared with other models in the study.\n\n◦ Limited Real-World Validation: The experiments are primarily conducted in a controlled environment. The paper would benefit from further real-world validation and testing in more diverse deployment environments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The Mixture-of-Transformers (MoT) architecture builds multi-modal foundation models for text, images, and speech. MoT uses modality-aware sparsity, decoupling parameters like feed-forward networks and attention projection matrices, while applying global self-attention. Experiments show MoT achieves comparable performance to dense models with reduced computational costs, in settings including: Chameleon (text and images), Chameleon+Speech, and Transfusion (autoregressive text and diffusion images). Ablation studies and analyses were also conducted.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "• More detailed analysis of MoE: Provide further analysis into the instability of the MoE models and the conditions under which MoE performs poorly compared to both dense and MoT models. This analysis may shed light on the mechanisms by which MoT is able to perform so well and why similar gains are not achieved in MoE models.\n\n• Parameter Count Comparisons: Explicitly compare the parameter counts between the dense, MoE, and MoT models to further clarify the efficiency gains in MoT. It is unclear from the results whether the total number of parameters are similar, even though the FLOPs is controlled in the experiment.\n\n• Text performance improvement: Investigate further how MoT could be improved for text-based tasks, particularly in the Transfusion setting. Since MoT performs so well in image generation, more explicit analysis into what contributes to the lack of improvement on text tasks would be helpful.\n\n• Generalization Analysis: Explore the generalization capabilities of MoT in settings where the distribution of data during evaluation differs substantially from the training data. Test MoT on different datasets to further assess how well it can generalize in different domains and settings.\n\n• Real-World Validation: Conduct further real-world validation to understand how the MoT model performs outside of the controlled lab settings. Such experiments would help make the claims of the paper even more robust.\n\n• Hyper parameter Tuning: Provide more details on hyper parameter tuning for the MoT and baseline models, especially for the diffusion objective, and justify the choice of hyper parameters used in the study.", "broader_impact_concerns": "• Reduced Computational Costs: The primary positive impact of MoT is the potential to reduce the computational cost of training large multi-modal models. This can significantly democratize access to advanced AI, making it accessible to a wider range of researchers and developers.\n\n• Energy/Power Consumption: By reducing computational costs, the MoT architecture may also lead to a reduction in the energy consumption associated with training large models, contributing to more sustainable AI development.\n\n• Potential Misuse: As with any advanced AI technology, there is a potential for misuse. The ability to generate convincing text, images, and speech could be exploited for malicious purposes, such as generating misinformation or creating deepfakes. It is important to consider the ethical implications and develop safeguards to prevent misuse.\n\n• Accessibility: By making multi-modal models more efficient, MoT could enable the deployment of advanced AI applications on devices with limited resources, increasing their accessibility to a broader audience.\n\n• Data Bias: The models are still trained on existing datasets, which may contain biases. While the architecture itself may be bias-neutral, it may still be susceptible to perpetuating biases in its training data. Further research is needed to mitigate this.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BTRpyUo1El", "reviewer": "Reviewer_KhiW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1) As mentioned in the summary part, the experiments are pretty comprehensive and the results are very strong. The LOO ablation study in Section 4 is also very helpful to understand the contribution of this model arch.\n2) Balancing different modalities is challenging in general, including the data resource constrains, training stabilities.... This idea has a great potential to alleviate this problem.\n3) Although this idea introduce more parameters, the final model is actually not fully required for many of the downstream tasks. For instance, for the text only applications, people can deploy only a subset of the model (text-only parameters) so that greatly reduce the memory requirement. However, MoE-based model must deploy the whole model for inference. (Seems that authors did not mention this in the draft, please add this into your paper if agree.) \n4) It is good to see the improvement from MoE and MoT can be stacked together.\n\n\n\nWeakness:\n1) (Minor) Could authors please fix the white background of the font in their Figure 1? It is a bit weird.\n2) As stated in the paper, there is a relatively large overhead for gathering the tokens for each modality. And this paper did not show how large the overhead is in terms of the training throughput (Tokens/Second) or MFU. I understand that, due to faster convergence, it is still better, but it is very important to show how large the overhead is and maybe people can get some ideas to speed it up.\n3) Seems that the MoE baseline is a bit weak? The EC is good but I dont think a well-tuned MoE should be worse than Dense given the same compute budget. Any further justification on this?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores a new model architecture, Mixture of Transformers for multimodal foundation models. The core design is using independent trainable transformer for the tokens from different modalities, but conduct cross-modality computation with attention activations.\n\nThis proposed model achieved very promising results on all three modalities, i.e. text, speech, and vision. For the vision part, authors consider both discrete token modeling (Chamelon) and also diffusion modeling (Transfusion). And achieved promising results on both of them.\n\nThe comprehensive ablation studies further supports the effectiveness of the proposed model.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weakness", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After the author rebuttal, all reviewers are supportive and recommend acceptance. However, the authors have not yet incorporated the requested changes into their paper.
Decision: Accept (poster)