4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities
Authors: Roman Bachmann
Oğuzhan Fatih Kar
David Mizrahi
Ali Garjani
Mingfei Gao
David Griffiths
Jiaming Hu
Afshin Dehghan
Amir Zamir
Venue: NeurIPS 2024 poster (2024)
Keywords: multimodal learning
multitask learning
representation learning
transfer learning
foundation models
generative models
computer vision
Primary area: machine_vision

Abstract: Current multimodal and multitask foundation models, like 4M or UnifiedIO, show promising results. However, their out-of-the-box abilities to accept diverse inputs and perform diverse tasks are limited by the (usually small) number of modalities and tasks they are trained on. In this paper, we develop a single any-to-any model trained on tens of highly diverse modalities and by performing co-training on large-scale multimodal datasets and text corpora. This includes training on images and text along with several semantic and geometric modalities, feature maps from recent state of the art models like DINOv2 and ImageBind, pseudo labels of specialist models like SAM and 4DHumans, and a range of new modalities that allow for novel ways to interact with the model and steer the generation, for example, image metadata or color palettes.

A crucial step in this process is performing discrete tokenization on various modalities, whether they are image-like, neural network feature maps, vectors, structured data like instance segmentation or human poses, or data that can be represented as text.
    
Through this, we show the possibility of training one model to solve at least 3x more tasks/modalities than existing models and doing so without a loss in performance. In addition, this enables more fine-grained and controllable multimodal generation capabilities and allows studying the distillation of models trained on diverse data and objectives into one unified model.
We scale the training to a three billion parameter and different datasets. The multimodal models and training code are open sourced at https://4m.epfl.ch/.

4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities
Roman Bachmann li-
Oguzhan Fatih Karl*
David Mizrahi
Ali Garjani'
Mingfei Gao
David Griffiths
Jiaming Hu
Afshin Dehghan²
Amir Zamir!
1 Swiss Federal Institute Technology Lausanne (EPFL) 2 AAplle
https://4n epfl. cofl.ch. ch
Edge
300
Coler palette
OAM edoe
Ouany sdge
Captic
Whb
Any-to-ary modal
Sensetic modalliee
Mataebeis mochellee
Dvendin
NAM
Feature modiel
Glcbal fearure Bise
Leagline
Leaglioc
(Sica)
Figure 1: We demonstrate training single model tens of highly diverse modalities without loss in performance compared to specialized single/few task models. The modalities are mapped discrete tokens using modality-specifit tokenizers. The model can generate any the modalities from any subset of them.
Abstract
Current multimodal and multitask foundation models, like 4M [65] Uni- fiedIO [61, 60]. show promising results. However, their out-of-the the-box abilities to accept diverse inputs and perform diverse tasks are limited by the (usually small) number modalities and tasks they are trained on. In this paper, we develop single any-to-any model trained on tens of highly diverse modalities and by performing co-training on large-scale multimodal datasets and text corpora. This includes training images and text along with several semantic and geometric modalities, feature maps from recent state of the art models like DINOv2 and ImageBind, pseudo labels of specialist models like SAM and 4DHumans, and a range of new modalities that allow for novel ways to interact with the model and steer the generation, for example, image metadata or color palettes. A crucial step in this process is performing discrete tokenization on various modalities, whether they are image-like, neural network feature maps, vectors, structured data like instance segmentation or human poses, or data that can be represented as text.
Through this, we show the possibility of training one model to solve least 3x more tasks/modalities than existing models and doing so without loss in performance. addition, this enables more fine-grained and controllable multimodal generation capabilities and allows studying the distillation of models trained on diverse data and objectives into one unified model. We scale the training to a three billion parameter and differentt datasets. The multimodal models and training code are open sourced at https: //4m. epf....f..... ch.
"Equal contribution & corresponding authors. Randomized order. Work partially done while EPFL Apple.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Having single neural network handle wide and varied range of tasks and modalities has been a longstanding goal. Such a model, especially when capable of any-to-any predictions, brings notable advantages, such as test-time computational efficiency, model size, and enabling modality fusion.
However, multitask learning has commonly faced significant challenges. For example, the training often suffers from negative transfer, leads to reduction in performance compared to single-task models, and typically requires careful strategies for balancing losses or gradients [48, 99, 86, 100, 33]. Moreover, training single network on tasks and modalities that vary greatly in terms of dimensionality, data type, and value ranges presents additional complexities" Recent notable efforts in the space of multimodal and multitask training, such as Pix2Seq [17, 18], OFA [93], 4M [65], or Unified-IO [61, 60] have made significant strides in unifying the representation space for conceptually different inputs and targets. A large part of their success can be attributed transforming different modalities into a common representation, namely sequences of discrete tokens, and training relatively standard Transformer architectures on them. While these works show promising results, they are typically trained on a small set of modalities. This raises the question if increasing the set of tasks/modalities the models can solve will lead to a degradation of performance.
We build upon the multimodal masking pre-training scheme [65] and increase its capabilities by training on tens of highly diverse modalities. Concretely, we add SAM segments [49], 3D human poses and shapes from 4DHumans [37], canny edges extracted from RGB and SAM instances, color palettes, multiple types of image, semantic and geometric metadata, embeddings, in addition to more common modalities. On top of that, we include dense feature maps of the recent state of the art models DINOv2 [68] and ImageBind [35], well their global embedding vectors to enable multimodal retrieval abilities. Please see fig. I for an overview.
well T5-XXI. [71] text
We able to train single unified model on diverse modalities by encoding them with modality- specific discrete tokenizers (see fig. 3). For image-like modalities, e.g. RGB or edges, we train ViT-based [25] VQ- VQEVAE [66] tokenizers to map the imputs into small grid of discrete tokens. For modalities like 3D human poses or image embeddings, we train MLP-based discrete VAEs to compress them into small of discrete tokens. All other modalities that can be mapped a text representation, such as captions or metadata, are encoded using a WordPiece tokenizer [24].
The resulting model demonstrates the possibility of training single model large number of diverse modalitics/tasks without any degradation in performance and significantly expands the out-of- the-box capabilities compared to existing models. Adding all these modalities enables new potential for multimodal interaction, such as retrieval from and across multiple modalities, or highly steerable generation of any of the training modalities, all by single model.
In short, we expand the capabilities of existing models across several key axes:
Increase from 7 in the existing best mny-to-any models [65] to 21 diverse modalities, enabling new capabilities like cross-modal retrieval, controllable generation, and strong out-of-the-box performance. This is one of the first times in the vision community that single model can solve tens diverse tasks in an any-to-any manner (see fig. 2), without sacrificing performance and especially do without any of the conventional multitask learning difficulties [77, 48, 99, 86, 100, 33].
Modalities:
Diversity: Add support for more structured data, such as human poses, SAM instances, metadata, and color palettes for controllable generation.
Tokenization: Investigate discrete tokenization of diverse modalities such global image embeddings, human poses, and semantic instances using modality-specifft approaches.
Scale: Scale the model size to 3B parameters and dataset to 0.5B samples using [12].
Co-Training: Demonstrate co-training on vision and language modeling simultaneously.
'Modality vs task: "Modalities" usually denote the inputs to model (e.g. sensory signals), and "tasks" usually denote the outputs (e.g. semantics). The adopted architecture in multimodal masked modeling enables a symmetric onput-output structure, thus modalities and tasks used interchangeably in this paper.

---

Villi
Shill
Galget
++
Figure 2: One-to-all generation. 4M-21 can generate all modalities from any given input modality and can benefit from chained generation [65]. Notice the high consistency among the predictions of all modalities for input. Each starts from different modality coming from the seene. Highlighted in green new input/output pairs that 4M [65] cannot predict accept as input. Note that, while this figure shows predictions from single input, 4M-21 can generate any modality from any subset of all modalities.
2
Method
We adopt the 4M pre- vre-training scheme [65] as it has been shown to be a versatile approach that can be efficiently scaled to diverse set of modalities. We keep the architecture and the multimodal masked training objective the same, but expand upon the model and dataset size, the types and number of modalities with which we train the model, and train jointly multiple datasets. All modalities first transformed into sequences of discrete tokens using modality-specine tokenizers (See fig. 3). During training, random subsets of these tokens selected from all modalities inputs and targets, and the objective is to predict one subset from the other. We rely on pseudo labeling to create large pre-training dataset with multiple aligned modalities. See appendix I.1 for discussion on different architecture choices.
2.1 Modalities
We train a large and diverse set of modalities that group into the following categories: RGB, geometric, semantic, edges, feature maps, metadata, and text modalities. Below we provide summary of them (See fig. I and appendices D and E for details, and fig. for generation examples).
RGB: We include both tokenized and pixel versions of RGB images to facilitate transfer learning. In particular, discrete tokens enable iterative sampling, making them useful for generative tasks [96, 15].

---

On the other hand, using RGB pixels input is more suitable for visual perception tasks. By avoiding the discrete bottleneck, there no information loss during the tokenization step, and the projection layer can be more lightweight. Given these tradeoffs, we follow 4M by training on both and treating them as separate modalities, with RGB pixels as an input-only modality. We also extracted color palettes from RGB images using PyPalette [2], at varying number of colors. This enables us to perform conditional generation using desired colors for better artistic control.
These contain surface normals, depth, and 3D human poses & shape which provide important information about the scene geometry. For the first two, we used Omnidata models from [27, 46] for pseudo labeling due to their strong generalization performance. For 3D human poses and shape, we leverage a recent state-of- saate-of-the-are art model, DD-mmmans [37].
Geometric modalities:
Semantic modalities: We include semantic segmentation and bounding boxes to capture the scene semantics and leverage Mask2Former [20] and ViTDet [56] models for pseudo labeling. Next to these, we also incorporated pseudo labels extracted from Segment Anything Model [49] (SAM) as SAM instances for its strong object representation.
Edges: As recent generative methods such as ControlNet [104] showed, edges carry important information about the scene layout and semantics that are also useful for conditioning. abstraction, and sketching. We consider two types of edges, specifically Canny edges and SAM edges. The former is extracted from the RGB images with OpenCV [1]. As Canny edges may contain low-level information, e.g. shading edges, we also include edges extracted from SAM instances to get a more semantic boundary map. We tokenize Canny and SAM edges with shared tokenizer.
We extract embeddings from CLIP [70], DINO,2 [68] and ImageBind [35] as they demonstrated strong transfer learning and retrieval capabilities. Previously, tokenized CLIP features were shown to be an effective target for masked image modelling 194, 65] that enables distilling a useful semantic representation of the scene. We follow similar approach and tokenize the feature maps from pre-trained CLIP-BI6, DINOv2-B14 14 and ImageBind-HH models. We also included the global embeddings of DINOv2 and ImageBind models and tokenized them separately.
Feature maps:
Metadata:
We extract several useful pieces of information from the RGB images and other modalities, that can be categorized into semantic metadata, geometric metadata, and image processing metadata. For this, we use functionalities from Pillow [3] OpenCV [1], and Omnidata [27].
The following semantic metadata are extracted from bounding boxes, poses, and segmentation maps:
Crowdedness score: number of humans (extracted from 4DHumans instances)
SAM clutter score: number of SAM instances
COCO clutter score: number of COCO [57] instances
COCO instance diversity: number of unique COCO instance classes
Objectness score: % of pixels that belong to countable COCO semantic classes
Walkability score: % of pixels belonging to walkable COCO semantic classes such as 'road'
Semantic diversity: number of unique COCO semantic classes
Caption length: length of the caption in characters, words, and sentences
These are aimed to capture the semantic regularities of the scene at a more holistic level opposed to pixel-based representations.
Similarly, geometric metadata captures the scene geometry more globally. They are extracted from surface normals and depth maps:
Geometric complexity: angular variance of surface normals
Occlusion score: % of occlusion edges over a fixed threshold
Finally, image processing metadata contains several aspects of images such as original image height and width before cropping, which can be used as conditioning to generate higher quality images [69], brightess, contrast, saturation, entropy, and colorfulness [39]. Similar to color palette, these help with encoding low-level image representations into the model and enable more steerable generation.
Text:
Large language models (L.LMs) trained on large text corpora learn strong representations as shown by several works [24, 71, 89, 67]. We include captions from CC12M [16] and COYO700M [12]

---

Spatial discrete with diffusion decoder: ROB, normaa, deguth, edgee
Spatial discrete VAE: Segrmentation, CLUP, DEROvE, Imageehund, BAM mnat.
encoder
decader
encoder
decoder
VG Ait
Yia Wec
on
Recondhaatann
MLP discrete VAE Ruman pases, DIMOv2 & ImaggaRind global tokens
Sequence tokenizer: Text, boundiigg boxes, matarisa, color palatta
MLP ancodar
MLP deccddrr
be von cendo
WordPlece
WordPlece
nel canta' porto
Figure 3: Tokenization overview. We employ suitahle tokenization schemes for different modalities based on their format and performance. For image-like modalities and feature maps, we spatial VQ-VAE; [66] with optional diffusion decoders for detail rich modalities like RGB. For non-spatial modalities like global tokens or parameterized poses, we compress them to a fixed number of discrete tokens using Memcodes [62] with MLP encoders and decoders. All sequence modalities are encoded as text using WordPicce [24]. The shown examples real tokenizer reconstructions. Notice the low reconstruction error. See appendix D for more details and Fig. 13 for visualizations.
datasets, well as web text from C4 [71] for language modeling. Next, we employ both a standard WordPiece [24] tokenizer for captions as [65] as well as caption embeddings obtained from T5-XXL [71] encoder to capture better text representations, which have been shown improve text-to-image generation fidelity [80, 14] (See fig. 4).
2.2 Tokenization
Tokenization consists of converting modalities and tasks into sequences or sets of discrete tokens, thereby unifying their representation space. This is critical for training large multimodal models as confers the following key benefits: 1) It enables training multimodal and multitask models with single pre-training objective. After tokenization, all tasks are formulated cation problem using the cross-entropy loss. This improves training stability, enables full parameter sharing, and removes the need for task-specific heads, loss functions, and loss balancing. 2) It makes generative tasks more tractable by allowing the model to iteratively predict tokens, either autoregres- sively [72, 96] through progressive unmasking [15, 14]. It reduces computational complexity by compressing dense modalities like images into a sparse sequence of tokens. This decreases memory and compute requirements, which is crucial when scaling up larger dataset and model sizes.
per-token classifi-
We use different tokenization approaches to discretize modalities with different characteristics. See fig. 3 for overview. To summarize, we mainly three different types of tokenizers, as explained below. Please see appendices D and H for more details and insights on tokenizer design choices.
VIT tokenizer (with optional diffusion decoder): We trained modality-specific ViT [25] based VQ-VAE [66] tokenizers for image-like modalities such as edges and feature maps. The resulting tokens form small grid of size 14 x 14 or 16 x 16, according to the pseudo-labeler patch size. The edge tokenizers use a diffusion decoder [82, 65] to get visually more plausible reconstructions.
MLP tokenizer: For human poses and global embeddings from DINOv2 and ImageBind, we use Bottleneck MLP [6] based discrete VAEs with Memeodes quantization [62] to tokenize them into small number of tokens, e.g. 16.
Text tokenizer: We leverage WordPiece [24] tokenizer which is used to encode not only text, but also other modalities such as bounding boxes, color palettes and metadata using a shared set of special tokens encode their type and values (See appendix D.6 for details).
2.3 Training details
We perform the training in two stages, namely a 4M pre- vre-training stage on a significantly larger image dataset, followed by a fine-tuning phase on a smaller dataset containing a larger number of modalities. Since the 4M-XL model showed signs of overfitting on sequence modalities when trained on CC12M [16), we re-trained the models on COY0700M [12], containing 50 times more samples. COYO700M was pseudo labeled with the same modalities used for 4M. To cut down
Datasets:
5

---

Fine-graned muttimodal conditioning control
Improved understanding capabiilios
Cayrin
Hurmon
Coplen badkgooun puuei
Probing with grounded generation
Steerable multimodal data generation from metadala
Metadate
instariia
17/6
Edald
Caption
201
mountinn
220
Figure 4: Fine-grained & steerable multimodal generation. Top left: 4X-21 generate variants of images that grounded in any input modality, here human poses. Bottom left: This enables to perform multimodal edits (e.g, editing the shape a polygon or grounding generation with edges) and probe learned representation. For example, by only changing the shape of the ellipse, 4M-21 renders the bowl from different angles. Top right: By pre-training on 21 types of modalities, including T5-XXL embeddings, and co-training with language modeling input is captions instead of language model embeddings). Bottom right: Compared to generating images from captions only, metadata provides more direct and stecrable way of controlling the multimodal data generation process, enabling exciting further research into generative dataset design.
large text corpus, show improved text understanding capabilities (even when the
pseudo labeling cost when expanding the number of modalities, decided to pseudo label CC12M instead COY0700M, and fine-tune the models with both new and old modalities. To avoid overfitting the larger models, co-train them with samples from COY0700M. In addition to the previously mentioned multimodal datasets, we also included the C4 [71] text corpus in training. We perform the training by randomly sampling elements of each batch from any of these datasets, given a pre-determined set of sampling weights, and perform language modeling on them. Exact details on the training mixture are given in appendix E.2.
Architecture: We adopt 4M's encoder-decoder based transformer architecture with additional modality embeddings to accommodate new modalities. Similar to 4M, besides RGB tokens, the encoder directly accepts RGB pixels with a learnable patch-wise projection to enable use as a ViT [25] backbone for transfer learning.
Masking strategy: We used both multimodal random [7, 65] and span masking [71] strategies that mask input and target tokens. We invoke dataset mixing ratios and Dirichlet sampling parameters, ensure stable training on multiple modalities and datasets, detailed in appendix E.2.
3 Multimodal capabilities
We demonstrate broad range of capabilities unlocked by 4M-21, including steerable multimodal generation (Sec. 3.1), multimodal retrieval (Sec. 3.2) and strong out-of-the-bon capabilities (Sec. 3.3). Please the project website for more visualizations demonstrating these capabilities.
3.1 Steerable multimodal generation
4M-21 can predict any training modality by iteratively decoding tokens [65, 15, 14]. This is shown in fig. 2 where we can generate all modalities from given input modality in consistent manner. Furthermore, as we can generate any of the training modalities from any subset other modalities, both conditionally and unconditionally. it enables several ways to perform fine-grained and multimodal generation, as shown in fig. This includes diverse capabilities such as performing multimodal edits, probing the learned representations, and steering multimodal data generation.

---

Any-to-RGB retrieval
Any-to-any retrieval
Multimodal retrieval
Query
Top Retrooass
Query
Reriowas
Guery
Siglewe 26:00
Adily
JAN7 WOOO
Figure 5: Different modes of multimodal retrieval. We perform multimodal retrievals by predicting global embedkdings (here shown DINOv2) from a given input (of any modality) using 21 and comparing the cosine distances between the query and retrieval set embeddings. Left: Retrieving RGB images from distinctly different query modalities (here RGB, segmentation map, edges, depth map, color palette, and caption). Middle: Retrieving any modality using any other modality as the query input. Each query modality constrains the retrievals differently, c.g. here the RGB image and caption queries always yield Neuschwanstein castle retrievals. contrast, depth and semantic queries, the scene is more ambiguous, thus they retrieve other buildings with similar characteristics. Right: We can also combine any subset of modalities to define the query input, e.g.
surface normals and a color palette, better control the retrieval. See appendix B.2 for more results.
Moreover, 4M-21 exhibits improved text understanding capabilities leading to geometrically and semantically plausible generations, both when conditioning on T5-XXL embeddings and on regular captions (fig. 4, top right). Please see appendix I.2 for additional results.
3.2 Multimodal retrieval
Our model can also perform multimodal retrievals by predicting global embeddings of DINOv2 and ImageBind, from any (subset) of the input modalities. Once the global embeddings are obtained, the retrieval is done by finding the retrieval set samples with the smallest cosine distance to the query [68, 35J. As shown in fig. 5, this unlocks retrieval capabilities that were not possible with the original DINOv2 and ImageBind models such as retrieving RGB images or any other modality via using any other modality the query. Furthermore, one can combine multiple modalities predict the global embedding, resulting in better control over retrievals, as shown on the right. Please see appendix 1.3 for additional results.
3.3 Evaluating out-of-the-box capabilities
4M-21 is capable of performing a range of common vision tasks out-of-the-box, as demonstrated visually in fig. 6. In table 1, we evaluate the performance DIODE [90] surface normal and depth estimation, COCO [57] semantic and instance segmentation, 3DPW [91] 3D human pose estimation, and do ImageNet- IK [79] KNN retrieval using predicted DINOv2 global tokens. We compare against the pseudo labeling networks, strong baselines, and the 4M model from [65] trained on 7 modalities. For surface normal estimation and semantic segmentation, we observed that ensembling multiple predictions significantly improves performance, see appendix F for more details and results.
Our model consistently achieves strong out-of-the-box performance, and often matches or outperforms the pseudo labelers and other specialist baselines, while being a single model for all tasks. Notice the large performance gap with other multitask models like Unified-IO [61] and Unified-1O-2 [60]. For KNN retrieval, 4M-21 XL performance approaches the tokenizer bound, i.e. the retrieval performance using the DINOv2 tokenizer reconstructions. While the smaller models lag behind 4M models, we observe that 4M-21 XL is able to match the performance of 4M-7 XL, while being trained solve three times more tasks. The trend over the model size needing to be larger is expected as the number of tasks increase.
7

---

Guare
human Figure 6: Out-of-the-box vision tasks. Given an RGB image, 4M-21 can predict all tasks successfully. as can be seen from their high consistency with the pseudo labels. See fig. 7 for more results.
Table 1: Out-of-the-box (zero-shot) performance. We show the performance for common subset of tasks: surface normals and depth estimation DIODE [90], semantic and instance segmentation COCO [57], k-NN retrieval ImageNet- [79], and human keypoint estimation 3DPW [91]. We compare strong baselines and specialist models, including our pscudo labelers. The model learned solve all the tasks without loss of performance, is significantly better than the baselines, and is competitive with pscudo labelers, while being single model for all tasks. Compared to 4M-7, the 4X- 21 model preserved its performance while solving 3x more tasks. X denotes that given model cannot solve the task out-of out-of-the-boo thx-box shows the tokenizer reconstruction quality and provides an estimate on the performance upper bound due to tokenization. fig. 14 for qualitative comparisons. Best results are bolded, second best underlined.
set
Method
Normals
Depth 1
Sem. seg.
Inst. seg.
INIK KNN
Pseudo labelers
Omnidata [46]
22.5
0.68
X
M2F-B [20]
45.7
SAM [49]
X
X
32,9
X
X
DINOv2-B14 [68]
X
X
X
/93.9
X
ImageBind-H14 [35]
/94.4
X
4D-Humane [37]
X
X
81.3
OASIS [19]
34.3
X
X
MiDaS DPT [73]
X
0.73
X
X
M2F-S [20]
44.6
X
M2F-L [20]
X
48.0
HMR [45]
X
X
X
130.0
UnifiedIO-B [61]
35.7
1.00
32,9
UnifiedIO-LL [6]]
33,9
UnifiedIO-XX [61]
31.0
0.82
44.3
X
UnificdIO 2-L [60]
37.1
0.96
38.9
UnifiedIO 2-XL [60]
34.8
Unified1O 2-XXL [60]
37.4
0.84
41.7
X
4N-7 [65]
21.9
0.71
43.3
X
4M-21
21.7
0.71
42.5
15.9
73.1
89.7
108.3
4M-7 L. [65]
21.5
0.69
47.2
x
X
4M-21
21.1
0.69
46.4
31.2
77.0791.9
97.4
4M-7 XL [65]
20.6
0.69
48.1
X
4N-21
20.8
0.68
48.1
32.0
78.3
92.0
Tokenizer bound*
4.0
0.06
90.5
91.2
80.2//930
17.5
Transfer experiments
To study the scaling characteristics of pre-training any-to-any models larger set of modalities, train models across three different sizes: B, L, and XL. We then transfer their encoders to downstream tasks and evaluate both unimodal (RGB) and multimodal (RGB Depth) settings. The decoders are discarded for transfer experiments, and we instead train task task-specific heads. We perform seff-comparisons in similar manner to [65, 7J, well comparing
of strong baselines.
Unimodal transfers. For unimodal transfers we leverage the RGB patch embeddings learned during the pre-training, as RGB pixel inputs used alongside the tokenized modalities. For the XL models

---

Table 2: Unimodal transfer study. We transfer 4M-21 baselines ImageNet-IK [79] classification. ADE20K [106] semantic segmentation, NYUv2 [84] depth estimation, and ARKitScenes [9] (ARKS) 3D object detection. We observe that 4M-21 21 1) docs not lose performance for the transfer tasks that are similar to the seven modalities of 4M, i.c. first three columns of the results, while being able to solve many more, and 2) novel tasks that are more different from 4M modalities, e.g. object leads improved performance improvements are further verified in the multimodal transfer results (Table 3)
detection (last column).
showing the usefulness of new modalities. Best results per task are bolded, second best underlined.
Pre-training
Enc.
INIK
ADE20K mloUt
NYUv2-D
ARKS Ap
Method
data
param.
Ace.1
6 acc.1
MAE B [40]
INIK
84.2
46.1
89.1
30.9
DeiT III B [88]
IN21K
85.4
49.0
87.4
36.1
MultiMAE B [7]
INIK
84.0
46.2
89.0
34.2
DINOv2 [68]
LVD142M
86M
85.3
51.6
92.2
38.1
4M-7 B [65]
CC12M
84.5
50.1
92.0
40.3
4M-7 7 B (Ours)
COYO
84.4
49.4
91.4
38.6
4M-21 B
CC12M+COYO+CA
84.5
50.1
90.8
42.4
MAE L [40]
INIK
86.8
51.8
93.6
36,2
DeiT III L [88]
IN2IK
87.0
52.0
89.6
40.3
DINOv2 1. [68]
IVDI42M
86.7
53.4
94.1
42.8
303M
4X-7 L [65]
CC12M
86.6
53.4
94.4
46.8
4M-7 L (Durs)
COYO
86.7
53.5
94.3
45.2
4M-21 21
CC12M+COYO+C4
86.5
53.4
93.7
47.0
DINOv2 g [68]
LVD142M
1.1B
88.0
58.7
92.5
45.3
4M-7 XL [65]
CC12M
87.0
55.0
96.1
48.1
4X-7 XL (Dura)
COYO
1.2B
87.1
56.1
96.5
47.3
4M-21 XL
CC12M+COYO+CA
87.1
56.0
96.5
48.4
and DINOv2 g, we perform parameter-efficient fine-tuning using LoRA [42] instead of full fine- tuning, which significantly improves results for XL. models. We did not observe similar performance gains for the smaller models. Further training details are described in appendix G.
We evaluate on ImageNet-11 KK classification [23, 79], ADE20K semantic segmentation [106]. NYUv2 depth estimation [84], and ARKitScenes [9] 3D object detection tasks. Some transfer tasks are completely unseen during pre-training, e.g. object classification or 3D object detection, while others are included as different instantiations, e.g. absolute depth instead of relative depth, or using ADE20K instead of COCO classes. We follow the best practices and commonly used settings from other papers [65].
The results are shown in table 2. We make the following observations: 1) for the transfer tasks that are similar to the seven modalities of 4M, e.g. semantic segmentation depth, 4M-21 does not lose performance due to being trained on many more modalities, 2) for novel transfer tasks like 3D object detection that are sufficiently different from 4M modalities, we observe an improved performance. Moreover, the performance improves with larger model sizes, showing promising scaling trends. These trends can further seen in the multimodal transfer results, which we explain next.
Multimodal transfers. We perform multimodal transfers on NYUv2, Hypersim [75] semantic seg- mentation, and 3D object detection on ARKitScenes. We compare transfers using RGB images only, and RGB pixels + tokenized sensory depth inputs. As table shows, 4M-21 makes strong use of optionally available depth inputs and significantly improves upon the baselines.
Table 3: Multimodal transfer study. We trans- both 4M-21 21 4M (pre-trained on CC12M) to NYUv2 and Hypersim segmentation, and object detection on ARKitScencs. All models are able to optionally available depth when it is of high quality (Hypersim & ARKitScenes), while our model achieves best results. Best results are bolded, second best underlined.
NYtIv--S mloU f
Hypersim mloU
ARKitScenes AP
5 Related Work
Method
RGB-D
RGB.D
RGB
RGB-D
Multitask learning in vision involves training a single model to perform multiple visual tasks efficiently [13, 78]. Earlier methods [28, 64, 50] combined multiple dense vision tasks into a single model but faced challenges scaling to larger variety
4H-7 B
56.6
57.5
40.2
43.9
40.3
46.5
59.7
38.6
46.4
42.4
48.1
4H-7 1.
61.2
61.4
48.7
50.5
468
49.5
4M-21 L
61.8
61.8
47.3
50.7
47.0
50.1
3M-7
62.1
61.2
48.6
51,0
48.1
50.1
4H-21 XL
63.9
63.9
48.6
52.5
48.4
51.3

---

of tasks and modalities, limited by training instabilities and the need for careful task selection and loss balancing to reduce negative transfer [48, 102, 86, 100].
Recently, discrete tokenization has enabled a shift towards integrating numerous vision tasks into unified multimodal and multitask models such Gato [74], OFA [93], Pix2Seq [17, 18], Uni- fiedIO [61, 60], 4M [65], and more [51, 83, 5, 41, 26, 44, 4, 97, 87, 108, 54, 105]. These methods first transform various modalities and tasks into sequences or sets of discrete to- kens [24, 52, 66, 31, 95], and then train a single Transformer on these tokens using either a sequence modeling [74, 93, 61, 17. 18, 51, 83] or masked modeling objective [65, 94]. Some methods (e.g. Gato [74], UnifiedIO [61, 60)) perform co-training on multiple disjoint datasets and are capable of performing wide range of tasks, but not jointly. In contrast, methods like 4M [65] train on a single aligned dataset through the use of pseudo labeling, enabling any-to-any modality prediction but on a typically more limited set of modalities. We significantly expand upon them by adding the ability to use this framework for an even greater amount of modalities and capabilities.
Furthermore, masked modeling has proven effective for learning useful representations in both NLP [24, 71] and vision [40, 8, 107, 29]. Extending it to multimodal domains [7, 36, 94, 65] enables strong cross-modal representations which is critical for multimodal learning. When combined with tokenization, masked modeling also enables generative applications [15, 55, 14, 85, 65J. Our work highlights the ability of masked modeling to expand to much greater set of modalities than previously shown, improving upon the out-of-the tt--heebox and multimodal generation capabilities of previous works.
6 Limitations and Discussion
We developed an any-to-any model on tens of diverse modalities and tasks. This was achieved by mapping all modalities to discrete sets of tokens via modality-specific tokenizers and using a multimodal masked training objective [65]. We successfully scaled the training to 3 billion parameters and to 21 modalities and different datasets, without a degradation in performance compared to the existing expert single/few task models. This results in strong th- capabilities as well new potential for multimodal interaction, generation, and retrieval, all by single unified model. Below, we discuss limitations and future work.
Transferlemergent capabilities: One hope from training a single network on several tasks is leading model that can solve novel tasks, often referred to as "transfer" or "emergent" capabilities. While, as we showed, a multitask model brings several key advantages even without transfer/emergence (e.g., efficiency, using a single model for broad out-of-the-box capabilities, modality fusion, etc.), we observe that the potential for transfer/emergent remains largely untapped. In general, compared to LLMs, vision/multimodal models have not shown exciting results in terms of transfer/emergence yet. We find this to be an important point to address in the future, e.g., via designing multitask architectures that have emergence, in contrast to out-of-the-boo capabilities, as their main objective.
Better tokenization: Like any token-based model, ours can directly benefit from progress on tokeniz- ers, e.g. higher reconstruction fidelity.
Co-training on partially aligned datasets: We showed the possibility of training on partially aligned datasets, e.g. text data from C4 and other modalities from CC12M, yet further investigations and a larger mixture of datasets are expected to bring stronger capabilities.
Acknowledgement: This work was supported as part of the Swiss AI initiative by grant from the Swiss National Supercomputing Centre (CSCS) under project ID a08 on Alps. We also acknowledge the support of the ETH4D and EPFL. EssentialTech Centre Humanitarian Action Challenge Grant.
References
[1] OpenCW.https://pppe org/ 4
[2] PyPalette. http:://gthhubbbooooo///ggggg//e
4,25
[3] Python Pillow, http:tt:://pythhopppllee org/ 4
[4] Aghajanyan. A., Huang, B., Ross, C., Karpukhin, V., Xu, H., Goyal, N., Okhonko, D., Joshi, M., Ghosh, G., Lewis, M.. Zettlemoyer, L.: CM3: A causal masked multimodal model of the internet, ArXiv abs/2201.07520 (2022) 10
10

---

[5] Aghajanyan, A.. Yu, L., Conneau, A., Hsu, W.N., Hambardzumyan, K., Zhang, S., Roller, S., Goyal, N., Levy. O., Zettlmmyer, L.: Scaling laws for generative mixed-modal language models. ArXiv abs/2301.033228 (2023) 10
[6] Bachmann, G., Anagnostidis. S., Hofmann, T.: Scaling mips: tale inductive bias. arXiv preprint arXiv:2306. 13575 (2023) 2,
[7] Bachmann, R., Mizrahi. D., Atanov, A., Zamir, MulúMAAE Multi-modal multi-task masked autoen- coders. European Conference on Computer Vision (2022) 6, 8, 9, 10, 30, 31
[8] Bao, H., Dong, L., Piao, S., Wei, F.: BEÎT: BERT pre-training of image transformers. In: International Conference on Learning Representations (2022) 10
[9] Baruch, G., Chen, Z., Dehghan, A., Dimry, T., Feigin, Y., Fu, P., Gebauer, Joffe, B., Kurz, D., Schwartz, A., Shulman, E.: Arkitscenes a diverse real-world dataset for 3d indoor scene understanding using mobile rgh-d data. In: NeurIPS (2021), https: /aaxivv wwweeii.orr//oeeeeeeee org/pdft /2111 08897.888888888 pdf 9
[10] Bavishi, R., Elsen, E.. Hawthorne, C., Nye. M., Odena, A., Somani, A., Taşırlar, S.: Introducing multimodal models (2023), https: htpp:://www adept ai/ awtblog/fuyu--aaaaa/a/aaa
[11) Burgess, N., Milanovic, J., Stephens, N., Monachopoulos, K., Mansell, D.: Bíloat16 processing for neural networks. In: 2019 IEEE 26th Symposium on Computer Arithmetic (ARITH), pp. 88-91 (2019). https://doi.org/11.. 109/ARITH.2019.0000
29
[12] Byeon, M., Park, B., Kim. H., Lee, S., Baek, W., Kim, S.: COYO-700M: Image-text pair dataset. https://gáthiee www.coo///cce.e http///akkbbaanccooo-daaaee -atasett (2022) 5, 23,
[13] Caruana, R.: Multitask learning. https://oi..or//10 http://11.10/1000033333300 1023/A.1000339906774
Machine
Learning
28, 41-75
(1997).
[14] Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, L., Yang, M., Murphy, K.P., Freeman, W.T., Rubinstein, M.. Li, Y.. Krishnan, D.: Muse: Text-to-image generation via masked generative transformers. In: International Conference on Machine Learning (2023) 6, 10, 24,
[15] Chang, H.. Zhang. H., Jiang, L., Liu, C., Freeman, W.T.: MaskGIT: Masked generative image transformer. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 11305 11315 (2022) 3, 5, 6, 10
[16] Changpinyo, S., Sharma, P.K., Ding, N., Soricut, R.: Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 3557-3567 (2021) 4, 5, 33,
[17] Chen, T., Saxena, S., Li. L., Fleet, D.J., Hinton, G.: Pix2seq: A language modeling framework for object detection, In: International Conference Learning Representations (2022) 2, 10, 25, 26
[18] Chen, Saxena, S., Li, L., Lin, T.Y., Fleet, D.J., Hinton, G.E.: A unified sequence interface for vision tasks. Advances in Neural Information Processing Systems 35, 31333-31346 (2022) 2, 10
[19] Chen, W., Qian, S., Fan, D.. Kojima, N., Hamilton. M., Deng, J.: Oasis: large-scale dataset for single image 3d in the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 679-688 (2020) 8
[20] Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: -atsetd-ttenttion mask transformer for universal image segmentation. CVPR (2022) 4, 8, 24
[21] Clark, K., Luong, M.T., Le, Q.V., Manning, C.D.: Electra: Pre-training text encoders as discriminators rather than generators. In: International Conference Learning Representations (2020) 30, 31
[22] Cubuk, E.D., Zoph, B.. Shlens, J., Lc. Q.V.: Randaugment: Practical automated data augmentation with
a reduced search space. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. pp. 702-703 (2020) 30
[23] Deng. J., Dong, W.. Socher, R.. Li. L.J. Li, K., Fei-Fei, L.: ImageNet: A large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition pp. 248-255 (2009) 9
[24] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding, In: North American Chapter of the Association for Computational Linguistics (2019) 2, 4, 5, 10

---

[25] Dosovitskiy, A.. Beyer, L., Kolesnikov, A., Weissenborn. D., Zhai, X., Unterthiner, T.. Dehghani, M.. Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image worth 16x16 words: Transformers for image recognition scale. International Conference on Learning Representations (2021) 2, 6
[26] Driess, D., Xia, F., Sajjadi, M.S., Lynch, C.. Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q. Yu, T., et al.: Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 (2023) 10
[27] Eftekhar, A., Sax, A., Bachmann, R., Malik, J., Zamir, A.R.: Omnidata: scalable pipeline for making multi-task mid-level vision datasets from scans. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) pp. 10766-10776 (2021) 4
[28] Eigen, D., Fergus, R.: Predicting depth, surface normals and semantic labels with common multi-scale convolutional architecture. In: Proceedings of the IEEE international conference on computer vision. pp. (650-2658 (2015)
[29] El-Nouby, A., Izacard, G., Touvron, H., Laptev, I,, Jegou, H., Grave, E.; Are large-scale datascts necessary for self-supervised pre-training? ArXiv abs/2112.10440 (2021)
[30] El-Nouby, A., Klein, M., Zhai, S., Bautista, M.A., Toshev, A., Shankar, V., Susskind. J.M., Joulin, A.:
Scalable pre-training of large autoregressive image models. arXiv preprint arXiv:2401.08:5 001444 (2024) 32
[31] Esser. P., Rombach, R., Ommer, B.: Taming transformers for high-resolution image synthesis. 2021 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR) pp.
(2020)
10
[32] Feichtenhofer, C., Li, Y., He, K., et al.: Masked autoencoders as spatiotemporal learners. Advances neural information processing systems 35, 35946-35958 (2022) 29
[33] Fifty, C., Amid. E., Zhao, Z. Yu, T., Anil, R., Finn, C.: Efficiently identifying task groupings for
multi-task learning, Advances in Neural Information Processing Systems 34, 27503-27516 (2021) 2
[34] Ghiasi, G., Cui, Y., Srinivas, A., Qian, R., Lin, T.Y., Cubuk, E.D., Le, Q.V., Zoph, B.: Simple copy-paste is a strong data augmentation method for instance segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2918-2928 (2021) 30
[35] Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K.V., Joulin, A., Misra, L:: ImageBind: One embedding space to bind them all. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15180-15190 (2023) 2, 4. 7, 8, 24
[36] Girdhar, R., El-Nouby, A., Singh, M., Alwala, K.V., Joulin, A., Misra, 1:: OmniMAE: Single model masked pretraining on images and videos. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10406-10417 (2023)
[37] Goel, S., Pavlakos, G., Rajasegaran, J., Kanazawa, A., Malik, J.: Humans in 4D: Reconstructing and tracking humans with transformers. In: ICCV (2023) 2, 4, 8, 24, 30
[38] Goyal, P., Dollár, P., Girshick, R.B., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., He. K.: Accurate, large minibatch sgd: Training imagenet in I hour. ArXiv abs/1706.02677 (2017) 29, 30
[39] Hasler, D., Suesstrunk. S.E.: Measuring colorfulness in natural images. In: Human vision and electronic imaging VIII. vol. 5007, pp. 87-95. SPIE (2003) 4, 24
[40] He, K., Chen, X., Xie, S., Li, Y., Doll'an ar, P., Girshick, R.B.: Masked autoencoders are scalable vision learners. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 15979-
15988 (2021) 9, 10, 24, 32
[41] Hu, A., Russell, I.., Yeo, H., Murez, 7., Fedoseev, G., Kendall, A., Shotton, J., Corrado, G.: Gaia-1: A generative world model for autonomous driving. arXiv preprint arXiv:2309. 7788 (2023) 10
[42] Hu, J.E., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, Chen, W.: LoRA: Low-rank adaptation large language models. ArXiv abs/2106.09685 (2021), attps://apts semanti iommantccscholatt .rgg CorpusID : 235458009 9, 30, 31
[43] Huang, G. Sun, Y., Liu, Z.. Sedra, D., Weinberger, K.Q.: Deep networks with stochastic depth. In: European conference computer vision. pp. 646-661. Springer (2016) 30, 31
12

---

[44] Huang. S., Dong, L.,. Wang. W. Hao. Y. Singhal, S.. Ma, S., Lv, T.,, Cui, L., Mohammed, O.K., Liu, Q.. Aggarwal, K.. Chi, Z., Bjorck. J., Chaudhary, V., Som, S., Song, X., Wei, F. Language is not all you need: Aligning perception with language models. ArXiv abs/2302.14045 (2023) 10
[45] Kanazawa, A., Black, M.J., Jacohs, D.W., Malik, J.: End-to-end recovery of human shape and pose. In: Proceedings of the IEFF conference on computer vision and pattern recognition, pp. 7122-7131 (2018)
[46] Kar, O.F. Yeo, T., Atanov, A., Zamir, A.: common corruptions and data augmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18963-18974 (2022) 4, 8
[47] Ke, I.., Ye, M., Danelljan, M., Liu, Y., Tai, Y.W., Tang, C.K., Yu, F:: Segment anything in high quality (2023) 24
[48] Kendall, A., Gal, Y., Cipolla, R.: Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition pp. (2017), https://appp ap1. http://app.ssamanicccooo semanticscholar. org/ II: 2, 10
[49] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, I.., Xiao, T. Whitchead, S., a
Berg, A.C., Lo, W.Y., Dollár, P., Girshick, R.B.: Segment anything. ArXiv abs/2304.02643 (2023),
petps://aptt semanticscholar wwwpolaarrrrrrrrrreeeeeeeee org/Corpus 257952310 4, 8, 24, 29
[50] Kokkinos, L: Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6129-6138 (2017)* 9
[51] Kolesnikov, A., Susano Pinto, A., Beyer, L.,, Zhai, X., Harmsen, J., Houlsby, N.: UViM: A unified modeling approach for vision with learned guiding codes. Advances in Neural Information Processing Systems 35, 26295-26308 (2022) 10
[52] Kudo, T., Richardson, J.: Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv: 808.06226 (2018) 10
[53] Levine, Y., Lenz, B., Lieber, O., Abend, O., Lryton-Brown, K., Tennenholtz, M., Shoham, Y: (PMI)- masking: Principled masking of correlated spans. In: International Conference on Learning Representa- tions (2021), https: / h/tp///pppe/eeeeeeee/eeeeeeeeeee net htte///ormmmdddaeeeeee wwo. 27
[54] Li, H., Zhu, J., Jiang, X., Zhu, X., Li, H., Yuan, C., Wang, X., Qiao, Y., Wang, X., Wang, W., Dai, J.: Uni-perceiver v2: A generalist model for large-scale vision and vision-language tasks (2022) 10
[55] Li, T., Chang, H., Mishra, S., Zhang, H., Katabi, D., Krishnan, D.: [mage]: MAsked generative encoder to unify representation learning and image synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2142 2142-2152 (2023) 10, 33
[56] Li, Y., Mao, H., Girshick, R., He, K.; Exploring plain vision transformer backbones for object detection. In: Computer Vision-ECCY ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX. pp. 280-296. Springer (2022) 4, 24
[57] Lin, T.Y., Maire, M., Belongie, S.J., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: Common objects context. In: European Conference on Computer Vision (2014) 4, 7,8
[58] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2021) 24
[59] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (2019) 29, 30, 31,32
[60] Lu, J., Clark, C., Lee, S., Zhang, Z., Khosla, S., Marten, R., Hoiem, D., Kembhavi, A.: Unified-IO 2: Scal- ing autoregressive multimodal models with vision, language, audio, and action. ArXiv abs/2312.11172 (2023), hctps::/ppp.semmaticccee.. semanticscholes
org/CorpusID:2e5e :ttrppusI::2667355555
266573555 7, 8, 10, 27, 28, 29
[61] Lu, J., Clark, C., Zellers, R., Mottaghi, R., Kembhavi, A.: Unified-IO: unified model for vision, language, and multi-mode tasks. In: The Eleventh International Conference on Learning Representations (2023) 1, 2, 7, 8, 10, 27, 28, 29
[62] Mama, R., Tyndel, M.S., Kadhim, H., Clifford, C., Thurairatnam, R.: Nwt: Towards natural audio-to- vidoo generation with representation learning. ArXiv abs/2106.04283 (2021), https://api. semanticscholar. org/ Cttp//ooppuDDD::: :33336579- 235367982 5
13

---

[63] Mentzer, F.. Minnen, D.C., Agustsson, E., Tschannen, M.: Finite scalar quantization: Vq-vae made simple. ArXiv abs/2309.15505 (2023), https://appp http:///aap.ssaaiiciesseeeeee www.nticsmcoole..e...e..ee. org/ Srgg/CrrpsIDD :::rpussD::2313399 263153393 33
[64] Misra, 1., Shrivastava, A., Gupta, A., Hebert, M.: Cross-stitch networks for multi-task learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3994 4003 (2016)
[65] Mizrahi, D., Bachmann, R., Kar, O.F, Yeo, T., Gao, M., Dehghan, A., Zamir, A.: 4M: Massively multimodal masked modeling. In: Advances in Neural Information Processing Systems (2023) 1, 2,
5, 6, 7, 8, 9, 10, 23, 25, 26, 27, 29, 30, 31,
[66] van den Oord, A., Vinyals, O., Kayukcuoglu, K.: Neural discrete representation learning. Advances in neural information processing systems 30 (2017) 2, 10
[67] OpenAl: GPT-4 technical report (2023) 4
[68] Oquah, M., Darcet, T., Moutakanni, T., Vo, H.Q., Szafranice, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby. A., Assran, M., Ballas. N.. Galuba, W., Howes, R., Huang. P.Y.B., Li. S.W.. Misra, L., Rabbat, M.G., Sharma, V., Synnaeve, G., Xu, H.. Jégou, H., Mairal, J., Labatut, P., Joulin, A.. Bojanowski, P:: DINOv2: Learning robust visual features without supervision. ArXiv abs/2304.07193
(2023) 2, 4, 7, 9, 24,23 29
[69] Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Muller, J., Penna, J., Rombach, R.: SDXL: Improving latent diffusion models for high-resolution image synthesis. ArXiv abs/2307.01952 (2023), http::/ppp...smtticcccooo.oo
semanticscholas org/ CorpusID :ttp:/ss:::2299133300 259341735 4, 24
[70] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A.. Mishkin, P., Clark, J., al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning (2021) 4, 24
[71] Raffel, C., Shazeer, N.M., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, PJ::: Exploring limits transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research (1),,, 5485-5551 (2020) 2, 4, 5, 6, 10, 23, 24, 27, 32, 33
[72] Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., Sutskever, I:: Zero-shot text-to-image generation. In: International Conference on Machine Learning (2021) 5
[73] Ranftl, R., Bochkovskiy, A., Koltun, V:: Vision transformers for dense prediction. 2021 IREF/CVF
International Conference Computer Vision (ICCV) pp. 12159-12168 (2021) 8, 24
[74] Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S.G., Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky, Y., Kay, J., Springenberg, J.T., et al.: A generalist agent. arXiv preprint M.Xi:22255.0677. (2022) 10, 32
[75] Roberts. M., Paczan, N.: Hypersim: A photorealistic synthetic dataset for holistic indoor under- standing. 2021 TEEE/CV International Conference on Computer Vision (ICCV) pp. 10892-10902 (2020)
[76] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. 2022 IEER/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 10674-10685 (2021) 33
[77] Ruder, S.: An overview of multi-task learning in deep neural networks. ArXiv abs/1706.05098 (2017). https: //api. http//pp..mmanicccoooooooo semanticscholar: org/ /Crr/rrppsID::::
10175374 :
[78] Ruder, S.: An overview of multi-task learning in deep neural networks. arXiv preprint 170.:01706.05098 (2017)
[79] Russakovsky. O.. Deng, J.. Su, H., Krause, Satheesh, S., Ma. S., Huang, Z., Karpathy. A., Khosla, A.. Bernstein, M.S., Berg, A.C., Fei-Fei, L.: ImageNet large scale visual recognition challenge. International Journal of Computer Vision 115, 211-252 (2014) 8,
[80] Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E.I., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al.: Photorealistic texi-to-image diffusion models with deep language understanding, Advances in Neural Information Processing Systems 35, 36479-36499 (2022) 5, 24
[81] Shazeer, N.: GLU variants improve transformer. ArXiv abs/2002.05202 (2020) 29
[82] Shi, J., Wu, C., Liang. J., Liu. X.. Duan. N.: DiVAE: Photorealistic images synthesis with denoising
diffusion decoder. ArXiv abs/2206.00386 (2022) 5
14

---

[83] Shukor, M., Dancette, C., Rame, A., Cord, M.: Unified model for image, video, audio and language tasks. arXiv preprint arXiv:2307.16184 (2023) 10
[84]
Silherman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from RGBD
images. In: European Conference on Computer Vision (2012) 9
[85] Sohn, K., Ruiz, N., Lee, K., Chin, D.C., Blok, I., Chang, H., Barber, J. Jiang, L.,, Entis, G., Li, Y., et al.: Styledrop: Text-to-image generation in any style. arXiv preprint arXiv:2306.00988 (2023) 10
[86] Standley, T., Zamir, A., Chen, D., Guibas, L., Malik, Savarese, S.: Which tasks should be learned logether in multi-task learning? In: International Conference on Machine Learning. pp. 9120-9132.
PMLR (2020) 2, 10
[87] Team, C.: Chameleon: Mixed-modal early-fusion foundation models (2024) 10,
[88] Touvron, H., Cord, M., Jégou, H.: DeiT III: Revenge of the vit. In: European Conference on Computer Vision. pp. 516-533 (2022) 9
[89] Touvron, H., Lavril. T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint 2Xiv:2302.13977 (2023) 4
(90] Vasiljevic, I., Kolkin, N., Zhang, S., Luo, R., Wang. H., Dai, F.Z., Daniele, A.F., Mostajabi, M., Basart, S., Walter, M.R., et al.: Diode: A dense indoor and outdoor depth dataset, arXiv preprint arXiv:1908.004:: (2019) 7,
[91] Von Marcard, T., Henschel, R., Black, M.J., Rosenhahn, B., Pons-Moll, G.: Recovering accurate 3d human pose in the wild using imus and a moving camera. In: Proceedings of the European conference on computer vision (ECCV). pp. 601-617 (2018) 7, 8
[92] Wang, H., Tang, H., Jiang, L., Shi, S., Nacem, M.F. F. Li, H., Schicle, B., Wang. L.: Git: Towards generalist vision transformer through universal language interface. arXiv preprint arXiv:2403.09300 (2024) 32
[93] Wang, P., Yang, A., Men, R., Lin, J., Bai, S., Li, Z., Ma, J., Zhou, C., Zhou, J., Yang, H.: Ofa: Unifying architectures, tasks, and modalities through simple sequence-to-sequeee learning framework. In: International Conference on Machine Learning (2022) 2, 10
[94] Wang. W., Bao, H., Dong, I.., Bjorck, J., Peng, Z., Liu, Q., Aggarwal, K., Mohammed, O.K., Singhal, S., Som, S., Wei, E: Image as a foreign language: BEiT pretraining for vision and vision- laiinnla tasks. ArXiv abs/2208.1044 10442 (2022) 4, 10
[95] Yu, Li, X., Koh, J.Y., Zhang, H., Pang, R., Qin, J., Ku, A., Xu, Y., Baldridge, Wu, Y.: Vector- quantized image modeling with improved VQGAN. International Conference on Learning Represen- tations (2022) 10
[96] Yu, J., Xu, Y., Koh, J.Y., Luong, T., Baid, G., Wang, Z., Vasudevan, V., Ku, A., Yang, Y., Ayan, B.K., Hutchinson, B., Han, W., Parekh, Z., Li, X., Zhang, H., Baldridge, J., Wu, Y.: Scaling autoregressive models for content-rich text-to-image generation. Transactions on Machine Learning Research (2022) 3, 5. 33
[97] Yu, L.. Shi, B., Pasunuru, R., Muller, B., Golovneva, O.Y., Wang, T., Babu, A., Tang, B., Karrer, B.. Sheynin, S., Ross, C., Polyak, A., Howes, R., Sharma, V., Xu, P., Tamoyan, H., Ashual, O., Singer, U., Li, S.W. S.W., Zhang, S., James, R., Ghosh, G., Taigman, Y., ZaraniZ M., Celikyilmaz, A., Zettlemoyer, L., Aghajanyan, A.: Scaling autoregressive modti- models: Pretraining and instruction tuning. ArXiv abs/2309.02591 (2023) 10
[98] Yu, L., Lezama. J., Gundavarapu, N.B.. Versari, L., Sohn, K., Minnen, D.C., Cheng, Y., Gupta, A., Gu, X., Hauptmann, A.G., Gong, B., Yang, M.H., Essa, L,, Ross, D.A., Jiang, L.: Language model beats diffusion tokenizer key visual generation (2023), aattps://ap. semanticscholar.co. org/
2633337733 33
[99] Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., Finn, C.: Gradient surgery for multi-task lcarning. ArXiv abs/2001.06782 (2020), https: /tttp://api.... semanti cscholar org/ CorpusID 210839011
[100] Yu, T., Kumar. S., Gupta, A., Levine. S., Hausman, K., Finn, C.: Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems 33, 5824-5836 (2020) 2, 10
[101] Yun. S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y.: Cutmix: Regularization strategy to train strong classifiers with localizable features. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 6023-6033 (2019) 30
15

---

[102] Zamir, A.R., Sax, A.. Shen, B.W., Guibas. LJ., Malik, J., Savarese, S.: Taskonomy: Disentangling task transfer learning. 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition pp. 3712-3722 (2018)
[103] Zhang, H., Cisse, M.. Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical risk minimization. In: International Conference on Learning Representations (2018) 30
[104] Zhang, L.., Rao, A., Agrawala, M.: Adding conditional control text-to-image diffusion models. In: Proceedings of the IEEE/CVF International Conference Computer Vision. pp. 3836 3847 (2023) 25
[105] Zhang, Y., Gong, K., Zhang, K., Li H.. Qiao, Y., Ouyang, W., Yue, X.: Meta-transformmrr: A unified framework for multimodal learning (2023)
[106] Zhou, B., Zhao, H., Puig, X., Fidler, Barriuso, A., Torralba, A.: Scene parsing through ADE20K dataset. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp. 5122-50 5130 (2017) 9
[107] Zhou, J., Wei, C., Wang. H., Shen, W., Xie, C., Yuille, A., Kong, T.: iBoT: Image BERT pre-training with online tokenizer. In; International Conference on Learning Representations (2022) 10
[108] Zhu, X., Zhu, J., H.. Wu. X., Wang, X., Li, H., Wang. Dai, J.: Uni- uni-Perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 16783-16794 (2021) 10
16

---

Appendix
Code, Pre-trained Models & Interactive Visualizations
18
Multimodal Capabilities
18
B.1 Additional multimodal generation & probing visualizations
18
B.2 Additional retrieval visualizations
22
C Additional Ablations
23
C.1 Ablation of pre-training data and modalities
23
C.2 Ablation of ensembling the predictions
23
Multimodal Dataset & Tokenization Details
24
D.I Pseudo labeled multimodal training dataset
24
D.2 Tokenization of human poses...
25
D.3 Tokenization SAM instances
................................
25
D.4 Tokenization global feature maps
25
D.5 Tokenization of dense feature maps
25
D.6 Tokenization of modalities
Seeee...........................
25
D.7 Tokenization Canny and SAM edges
26
D.8 Tokenizer reconstruction quality
26
Training Details
26
E.1 Modality-speciffc accommodations...
27
E.2
Multidataset co-training and diversified multimodal masking strategy
27
F Out-of-the-box Evaluation Details
27
E. Surface normal and depth estimation on DIODE
29
F2 Semantic and instance segmentation on COCO
29
F3 kNN retrieval on ImageNet-IK.
29
F.4 3D human pose prediction on 3DPW
30
G Transfer Evaluation Details
30
H Investigating Different Tokenization Schemes
31
I Additional Discussions and Results
32
L1 Encoder-decoder vs decoder-only architectures
32
1.2 Quantitative evaluation of image generation
33
1.3
Quantitative evaluation of cross-modal retrieval
34
J Broader Impact
34
J.1
Computational costs
34
J.2 Social impact
17

---

A Code, Pre-trained Models & Interactive Visualizations
Please see our website for documented open-source code, pre-trained model and tokenizer weights, as well as an overview video and additional interactive visualizations.
B Multimodal Capabilities
B.1 Additional multimodal generation & probing visualizations
Please see Figures 7, 8, 9, 10 for additional qualitative results any-to-any generation, controlled generation, and text understanding capabilities our model.
ROB Iepu
Predictions
Cuator
Chiold
Figure 7: RGB-to-any generation. This is extension of fig. and visualizes the model's out-of- the-box capabilities on various vision tasks, compared to the pseudo labeler outputs.
18

---

Image metadata
Geometric metadata
Original Rlee. Brightness
Saturation Colorfuiness
Contrest 16%
Entropy
Goometric complexity
Occlusion score
10%s
30%
0016
40%
2040 2040
0009
9016
00:95
98%
9.5
50%
80%
Semantic metadata (semanttcc classes)
Walkability score
Semantic diversity
Objectness score
10%
20%
40%
90%
25
100%
Semantic metadata (instances)
Crowdedness
SAM
GOCO
COGO Instance diversity
100
10
5U%
500
20
80%
Figure 8: Steerable multimodal generation using metadata. This an extension fig. 4 and shows model's capability of generating multimodal data by conditioning wide set of controls. The common caption for all examples is "a painting of a bridge in a lush forest".
19

---

Varying SAM polygon instances
SAM polygon instance inputs RGB generations
Caption input
A framed painting of mountains inside bedroom
Color palette input
Color palette input
Varying color palette
Color palette inputs & RGB generations
Caption input
Outside view of an apartment
Normals input
Figure 9: Probing with grounded generation. This an extension of fig. and further shows our model's capability performing generation by conditioning on multimodal input. The top row varies SAM instances and combines them with a fixed caption and color palette input. The bottom row fixes the normals and caption inputs and varies the color palette.
20

---

Caption Imput: a photto of toddy bear made of water
Caption Input xbrant portes painting Salaador Del with robedc haffface
436. dram captort
435-1 From captior
437-21 poom T5-00, amb))
435. (rom caplior
48C A1 Jaom captier)
455-8 Prom TS-00L, ambb)
Caption input: diowing of house on mountain
Caption Input the sthouerte of me Millonamm Kitzel of duse
4369 drom ception
From capsiory
435-83
wmbh)
435- from cepttr)
trom cantert
48:81 (hom T5-00, ane)
Imput: stap sign caslevound
Caption Input
teacup
PITGE
HIIE
STOP
STOP
STOP
STOP
EIIIES
H400
STOP
STOP
STOP
SOTP
paintig
Caption guend
rpp Klspin
4M5-9
Figure 10: Text understanding. This is an extension of fig. and further demonstrates improved text understanding capabilities of our method compared to 4M for several caption inputs.
21

---

B.2 Additional retrieval visualizations
Please see Figures 11 and 12 for additional qualitative results on RGB-to-Any and Any-to-RGB retrievals.
Tepa Barieeads
Tapel Teetrenck
Tep-3 Fariesals
Top3 feriesak
baing Tha
weiting any time
Figure RGB-to-Any retrieval. This an extension of fig. and further demonstrates cross-modal retrieval capabilities of our model. Here model successfully retrieves several modalities (RGB, depth, normals, segmentation) using the RGB image as the query input.
22

---

Depth-to
Segmentation
Ouey
fariecea
Query
HAVE
Figure 12: Any-to-RGB retrieval. This is an extension of fig. 5 and further demonstrates cross-modal retrieval capabilities our model. Here our model successfully retrieves RGB images when the query imputs from depth, normals, segmentation, and caption modalities.
C
Additional Ablations
C.1 Ablation of pre-training data and modalities
For training 4M-21, initialize the training using 4M models that pre-trained on COY0700M [12]. We ablate in Table different choices of training data and modalities. We can see that performing co- training on C4 [71] and COY0700M [12] has the potential to slightly improve transfer performance on average.
Table 4: Pre-training data and modality mixture ablation: modality and dataset choices on B models. represents the models initialized from the corresponding 4M models trained on COY0700M.
We ablate different pre-training
Pre-training
ImageNet-IK Top-1 acc. f
ADE20K mloU 1
NYUv2 depth
ARKitScenes Ap
Method
data
61 acc.
4M-7 [65]
CC12M
84.5
50.1
92.0
40.3
4M-7
COY0700M
84.4
49.4
91.4
38.6
4M-7
CC12M
84.5
49.2
91.0
39.5
4M-21
CC12M
84.4
4M-21
CC12M+C4
84.6
49.5
90.4
41.2
4M-21
CC12M+COYO700M4C4
84.5
50.1
90.8
42.4
C.2 Ablation of ensembling the predictions
Unlike the deterministic pseudo labeler and other state of the networks we compared against in table 1, our model can produce multiple prediction given the same RGB input through repeated sampling with different seed. As shown Table 5, ensembling ten samples of predicted surface normals and semantic segmentation maps can significantly improve the reported metrics. While ensembling improves upon these metrics, we note that the ensembled predictions can comparatively blurrier around object edges than any individual sample.
23

---

Table Ensembling ablation: We ensembling multiple predictions on DIODE normals and COCO semantic segmentation compared to no ensembling. As the results suggest, ensembling in all cases improves the quantitative results.
DIODE Normals
COCO Semseg
mean angle error
mean IoU f
Method
No Ensemble
Ensemble
No Ensemble
Ensemble
4M-21
22.3
21.7
39.0
42.5
4M-21
21.7
21.1
43.8
46.4
4M-21 XL
21.3
20.8
46.5
48.1
D Multimodal Dataset & Tokenization Details
D.1 Pseudo labeled multimodal training dataset
Similar to 4M, to have an aligned multimodal dataset, we pseudo label the CC12M dataset using strong specialized models for each task. The pseudo labeling of existing modalities is done in the same fashion as 4M, using Omnidata DPT-Hybrid [73] for surface normals and depth estimation, COCO Mask2Former [20] with a SwinB [58] backbone for semantic segmentation, COCO ViTDet ViT-H model [56] initialized from MAE weights [40] for bounding boxes, and CLIP-B16 [70] with ViT-B/16 visual backbone backbone for CLIP feature maps.
3D human poses. We use 4D-Humans [37] to extract 3D pose and shape parameterized by an SMPL model. For the images in CC12M without humans, we set the pose label to a "none" token. For the images with humans, we form sequence by concatenating the bounding box, body pose, camera, and shape values in sequence for each human instance. As data augmentation, we randomly shuffle the order of each component in the sequence.
SAM instances. Besides semantic segmentation and bounding boxes, SAM [49] instance seg- mentation also provides some level of semantic information from an image by clustering together semantically similar pixels in it. Unlike semantic segmentation, SAM instances are not restricted to specific set of classes and can segment in more detail. We use the SAM H model and query with points in grid format to obtain the instances. We also considered the SAM-HQ [47] H model, however in the grid-point querying format, it yields very similar results to SAM. We found x 32 query points to be the optimal choice both for pseudo labeling speed and quality.
DINOv2 and ImageBind global features & feature maps. We extract both dense feature maps and global embeddings, i.e. cls token embeddings, from DINOv2-B11 B14 [68] and Image [35] pre-trained models. For the latter, we only extracted the image embeddings, incorporating other modality embeddings such as thermal or audio could be interesting future work.
T5-XXL embeddings. Language model embeddings, such from T5-XXL [71], have been shown to improve the generation fidelity and text understanding capabilities of text-to-image generative models [80, 14]. Consequently, we use the T5-XXL encoder to extract text embeddings from all CC12M captions, without any preprocessing of the text. Unlike other modalities, we do not convert these text embeddings to sequence of discrete tokens treat them as targets (similar to the RGB pixel modality variant). Instead, we only provide them as inputs using a linear projection from the T5-XXL embedding dimension (dTs.xxI. 4096) our model's embedding dimension.
Image metadata. From RGB images, we directly extract different types of metadata like the original height and width before cropping [69], brightness, contrast, saturation and entropy. We additionally extract a notion of colorfulness, following [39].
Semantic metadata. We compute the crowdedness score as the number of humans in the pseudo la- beled human poses, the SAM clutter score as the number of SAM instances, the COCO clutter score as the number of COCO instances, the COCO instance diversity as the number of unique COCO instance classes, and the semantic diversity as the number of unique COCO semantic classes in an image. For caption length, we count the number of characters, words, and sentences. As objecíness score, we count the percentage of pixels in the COCO semantic segmentation map that belong to countable classes (indices 87, 90, 96, 97, 98, 99, 100, 101, 102, 103, 105, 106, 109, 110,
24

---

111, 112, 113, 117, 118, 119, 122, 123, 124, 125, 126, 129, 131, 132), and for the walkability score we count classes such as 'road' (indices 87, 90, 97, 100, 102, 105, 106, 122, 123, 125, 126, 132).
Geometric metadata. To compute the occlusion score, first generate occlusion edges from depth images by applying a Sobel filter, followed by counting the percentage of occlusion edge pixels that surpass threshold of 0.3. As notion of geometric complexity, we project surface normal pixels onto the unit sphere, and compute their angular variance. Note that images of indoor scenes or caves featuring large surfaces pointing in all different directions receive high score in this metric, while ones with a more localized geometric variance get somewhat lower score. Exploring other potential notions of geometric complexity can be an interesting future addition.
Color palette. For every RGB image, we extract between one and seven color palettes using PyPalette [2]. During training, we randomly sample one of the color palettes to enable to input palettes with different levels of granularity.
SAM edges and canny edges. Edges are convenient way of grounding image generation shapes contained in images [104]. To pseudo label edges, we apply the OpenCV canny edge detector on SAM instance maps and RGB, to obtain SAM edges and canny edges respectively.
D.2 Tokenization of human poses
We use a BottleneckMLP [6] with 6 blocks and 1024 width to compress pose into 8 tokens. We use 1024 vocabulary size, and trained using smooth L1 loss for 15 epochs on CC12M training data. We also binned the global orientation, body shape, and bounding boxes into 1000 discrete bins similar [17]. The final sequence is obtained by also adding identifiers, i.e. "bbox". "pose", "shape", before the corresponding sub-sequence.
D.3 Tokenization of SAM instances
The SAM instance tokenizer is a ViT-based VQ-VAE that tokenizes 64 x 64 binary masks into 16 tokens using vocabulary size of 1024. The tokenizer is trained using the cross-entropy loss for 24 epochs on CC12M training data, by resizing individual masks into square aspect ratio image of 64 64 pixels. To preserve the SAM instances' original location, width, and height in the image, their bounding boxes are extracted. The final sequence for each instance is formed by appending the identifier "polygon" to numbers that specifyy the bounding box of the instance, along with the 16 token IDs.
D.4 Tokenization of global feature maps
Similar to human poses, we use BottleneckMILP with blocks and 1024 width to compress DINOv2- B14 and Iind-Bind-H14 global embeddings into 16 tokens. We use 8192 vocabulary size, and trained using cosine similarity loss for 15 epochs.
D.5 Tokenization of dense feature maps
We follow [65] and tokenize CLIP-B16, DINOv2-B14, and ImageBind-H14 dense feature maps into 196, 256, and 256 tokens, respectively, using ViT-based VQVAE with 8192 vocabulary size and smooth LI loss.
D.6 Tokenization of sequence modalities
We tokenize text, color palette, metadata, and bounding boxes using WordPiece tok- enizer by fitting it on all captions and 4000 "special value" tokens, with joint vocab- ulary size of 30k. These special tokens are divided into four groups, each with 1000 v1=999, v2=0, v2=1, values, i.e. v0=0, v0=1, v3=999. For bounding boxes, we follow 4M [65] and represent v2=999, v3=0, v3=1,
v0=999, v1=0, v1=1,
xmin, ymin, xmax, ymax coordinates using v0, v1, v2, v3 tokens respectively. Other modal- ities are tokenized by binning their values into corresponding bins, e.g. color palette sequence formed as color c R G =gl. b R r,... where takes value between and 7 and
25

---

specifies the number of colors in the palette and r. b takes values between 0-255. We chose to model metadata using interleaved pairs of special tokens, where the first one specifies the type of metadata modality, and the second specifies its value. For example, a crowdedness score of 3 and brightness of 120 would be specified as the sequence v1=5 v0=3 v1=10 v0=120. During training the number of metadata entries and their order is randomized. All of this results in sequence prediction formulation, following [65, 17].
D.7 Tokenization of Canny and SAM edges
We use a VQ-VAE with diffusion decoder, similar to [65] tokenize the edge modalities. We use the same tokenizer as reconstructs both edges similarly well.
D.8
Tokenizer reconstruction quality
We provide qualitative results for tokenizer reconstructions in fig. 13.
Ground truth
Tokenizer
Ground truth
Tokentzer
Ground fruth
Tokenizer
Ground truth
Tokenber
RGB
Depth
Sufface Hntrrnl
SAM Inttia
Cantr Edgoa
Figure 13: Tokenizer reconstruction quality. Our multimodal tokenizers can reconstruct the ground truth well. Here show sample reconstructions of RGB, depth, surface normals, semantic segmentation, SAM instances, and canny edges on (pseudo) labels from the CC12M validation set 224 224 resolution. Quantitative evaluations provided in table 1 for different tasks and datasets (last row, Tokenizer bound), confirming the reconstruction quality.
E
Training Details
Please see Tab. 6 for an overview of pre-training settings. For more accurate model comparisons, the architecture and overall training objective of our B, L, and XL models are the same as those of 4M models. However, we modify and improve various aspects of the training process that allow us to significantly increase the number of training modalities. These changes concern modality-speciffe accommodations the masking strategy, the ability co-train on several datasets, and the use of more diversified multimodal masking strategy. We describe these modifications below:
26

---

E.1 -peclity-specificc accommodations
Positional and modality embeddings. As with 4M, 4M-21 incorporates both learnable modality beddings and fixed sine-cosine positional embeddings for each modality. The positional embeddings are either ID 2D depending on the modality type.
Metadata grouping and chunk-based masking. To address the sparsity and number of different types of metadata, the metadata modalities are all grouped together as single modality during training. This prevents the over-allocation of tokens to sparse metadata, enabling a more balanced distribution of the token budget across modalities. However, the standard span masking from T5 [71] and 4M [65] performs random uniform masking at the token level, which can lead to pre-training inefficiencies [53] and make conditioning specific metadata difficult, as conditioning on just one of them would rarely occur during pre-training with this masking strategy. Instead, we propose to mask chunks of sequence (similar to PMI-Masking [53]), where the span masking is performed per chunk of metadata instead of at the token level.
E.2 Multidataset co-training and diversified multimodal masking strategy
Multi-dataset support. Unlike 4M which was only trained on single aligned dataset, we train 4M-21 on multiple datasets simultaneously. This flexibility allows for the inclusion of datasets with varying numbers of modalities, which enables training on both large-scale datasets with a smaller number of modalities and smaller datasets with larger diversity of modalities.
Sampling and masking strategies. Our data sampling process involves selecting training dataset based on its sampling weight, followed by choosing a masking strategy from the dataset-specific mixture of masking strategies. Input and target tokens are then sampled using the selected strategy.
Co-training datasets. We co-train on several datasets to improve the model's performance and data diversity. These include CC12M [16], which comprises about 10 million text-image samples fully pscudo labeled with all 21 modalities, and accounts for 60% of our training samples. Additionally, we include COY0700M [12], with approximately 500 million text-image samples pseudo labeled with the 7 modalities of 4M, and accounts for 20% of our training samples. Lastly, the Colossal Clean Crawled Corpus (C4) [71], large text-only dataset, is used for language model co-training, also making up 20% of our training samples.
Diverse mixture of masking strategies. As with 4M [65], the masking strategy is governed by Dirichlet distribution with parameter a. This distribution influences the sampling of tokens from modalities: lower results in samples dominated by one modality, while higher a leads more balanced representation across all modalities. For both CC12M and COYO datasets, we implement multiple masking strategies to cater to specific training needs, and randomly sample from them for every sample in the batch:
All-to-all masking: Involves four masking strategies with symmetric input and target a set to 0.01, 0.1, 1.0, and 10.0 respectively.
RGB-to-all masking:
Consists of only RGB tokens as input, with target a all set to 0.5.
Caption- masking: Includes two strategies. heavily skewed towards either unmasked captions or T5-XXL embeddings as input. These masking strategies are particularly benefi- cial for tasks involving text-to-image generation
F Out-of-the-box Evaluation Details
Below, we provide further details on out-of-the-box evaluations we performed. Please also see fig. 14 for a qualitative comparison between our XL model and Unified-IO XL [6]], as well as Unified-IO 2 XXL [60]. Furthermore, table 7 compares Unified-IO, Unified-IO 2, and our model's out-of-the- box capabilities on surface normal estimation, depth estimation, and semantic segmentation. As demonstrated, model outperforms Unified-IO and Unified-10 2 in all the mentioned tasks.
27

---

DIODE depth
RGB Input
Ground Truth
4M 21 XL
Unified-IO XL
Unified-IO XXL
DIODE surface normals
RGB Input
Ground Truth
4M-21 XL.
Unified-IO
Unified-IO XXL
COCO semantic segmentation maps
RGB Input
Ground Truth
4M-21 XL
Unified-IO XL
Unified-I0 XXL
Figure 14: Comparing 4M-21 XL, Unified-IO XL [61], and Unified-IO XXL [60] out-of-the-box. 4M-21 XL demonstrates strong generalization to inputs from different datasets and tasks out-of-the- box (zero shot), significantly improving over Unified-IO and 2.
28

---

Table Pre-training settings. Training configuration for 4M-21 used in the transfer experiments and generation results.
Configuration
4M-21 B
4M-21
4M-21 XL
Weight initialization
4M (COYO)
Training length (n tokens)
500B
Warmup length (n tokens)
10B
Optimizer
Adam W [59]
Opt. momentum
81 92 le-4
0.9, 0.95
Base learning rate [38]
le-4
2e-5
Batch size
8192
Weight decay
0.05
Gradient clipping
X
x
3.0
Learning rate schedule
Cosine decay SwiGLU [81]
Feedforward activation
Input token budget
256
Target token budget
256
Input and target a
Mixture (see Sec. E.2)
Masking strategy
Mixture (see Sec. E.2)
Dataset
Mixture (see Sec. E.2)
Image resolution
224
Augmentation
None (Center Crop)
Repeated sampling [32]
Data type
bfloat16 [11]
Table Out-of-the-bos capabilties. Comparison between Unified-IO and our model out-of-the- task capabilities across surface normal estimation, depth estimation, and semantic segmentation. We use the LI score as the metric for surface normal and depth estimation, and mean IoU for semantic segmentation.
Method
Normals
Depth
Sem. seg. f
Unified-IO [61]
35.7
32.9
Unified-IO L
33.9
0.87
41.6
Unified-IO XL
31.0
0.82
44.3
Unified- 10 2 L [60]
37.1
0.96
38.9
Unified-1O 2 XL
34.8
0.86
39.7
Unified-UU -ifedd--222 2 XXL
37.4
0.84
41.7
4M-21
21.7
0.71
42.5
4M-21
21.1
0.69
46.4
4M-21 XL
20.8
0.68
48.1
F1
Surface normal and depth estimation on DIODE
We follow the evaluation setup [65] and evaluate on DIODE validation set at 224 x 224 input resolution.
F.2 Semantic and instance segmentation on COCO
employ a similar approach as SAM [49] by querying our model on the bounding boxes to obtain the instances. To predict the instances, only the target bounding box is provided in the input final sequence, and the tokens are masked for our model to predict them.
F.3 kNN retrieval on ImageNet-1K
We follow the evaluation setup from DINOv2 68Jand set A: 20 and temperature to 0.07.
29

---

F.4 3D human pose prediction on 3DPW
We follow the evaluation implemented in the 4D-Humans [37] codebase, with the difference that we use 224 x 224 as input image resolution opposed to 256 x 256.
G Transfer Evaluation Details
We provide the transfer settings in Tables 8, 9. 10. We also note that after an extensive hyper parameter search for the DINOv2-g baseline NYUv2, using ConvNeXt head, it achieved only 92.5 acc., which is lower than the reported 95.0 with frozen encoder and DPT head.
Table Image classification settings. Configuration for intermediate tnn-tuning on ImageNet-2IK and fine-tuning ImageNet- IK, the settings follow MultiMAE [7] and 4M [65].
Configuration
ImageNet-21K
ImageNet-IK Large
Base
Large
XL
Base
XL
Fine-tuning epochs
50
20
Warmup epochs
2
Optimizer
Adam W [59]
Adam W [59]
Opt. momentum
81 82 = 0.9, 0.95
B1, 82
0.9, 0.999
Base learning rate [38]
le-4
le-4
5e-5
le-4
Batch size
4096
4096
4096
1024
Weight decay
0.05
0.05
Learning rate schedule
Cosine decay
Cosine decay
Layer-wise decay [21]
0.75
0.85
0.85
0.75
0.85
0.85
Drop path [43]
0.1
0.4
0.1
0.2
Input resolution
2242
2242
Augmentation
RandAug(9, 0.5) [22]
RandAug(9, 0.5) [22]
Random resized crop
(0.5, 1)
(0.08, 1)
Label smoothing e
0.1
Mixup [103]
0.1
0.1
Cutmix [101]
1.0
1.0
Table 9: Semantic segmentation settings, Configuration for semantic segmentation fine-tuning ADE20K, the settings follow MultiMAE [7] and 4M [65].
Configuration
ADE20K
Base
Large
XL
Fine-tuning epochs
64
64
128
Warmup epochs
Optimizer
Adam W [59]
Opt. momentum
B1 B2 = 0.9, 0.999
Learning rate
2e-4
2e-4
3e-4
Batch size
64
Weight decay
0.05
Learning rate schedule
Cosine decay
Layer-wise Ir decay [21]
0.75
0.85
0.95
Drop path [43]
0.1
0.2
0.3
LoRA [42] rank scale
X
X
64 1.0
Input resolution
5122
Augmentation
Large-scale jitter (LS..) [34]
Color jitter
30

---

Table 10:
Depth estimation settings. Configuration for depth estimation fine-tuning on NYUv2, the
settings follow MultiMAE [7] and 4M [65].
NYUv2 Large
Configuration
Base
XL
Fine-tuning epochs
1000
Warmup epochs
100
Optimizer
Adam W [59]
Opt. momentum
81
0.9, 0.999
Learning rate
le-4 128
1e-4
5e-5
Batch size
128
16
Weight decay
le-4
Learning rate schedule
Cosine decay
Layer-wise Ir decay [21]
0.75
0.85
0.9
Drop path [43]
0.1
0.2
0,0
LoRA [42] rank scale
x
X
8/00
Input resolution
256
Random crop
Color jitter
Path Tokenization
Mask Tokenization
100.0-
Number of Toiens
100.0
Voc Wbullayy Size
100.0
Loss
100.0
Missk Size
95.0
95.0
92.5
925
92.5
92.5
90.0
90.0
87.5
85.0
36
25
85.0
512
1024*
2048
85.0
MSE
CE"
0.00
85.0
22
x
64.7
64"
120
Figure 15: Ablating tokenization choices: We ablate the impact of different tokenization choices. Performance reported as reconstruction loU CC12M validation used in the final tokenizer. See appendix H for details.
shows mask tokenization configuration we
H Investigating Different Tokenization Schemes
As develop several tokenization strategies for each modality, ablating their performance against all possible design choices would be prohibitively expensive. Thus, we focus on one modality, namely SAM instances, and provide more detailed look into the impact of different tokenization strategies. We study two approaches for SAM instances: path tokenization and mask tokemization.
We represent each instance in the image as list of polygon coordinates. Then tokenize these coordinates using a Bottleneck MLP-based VQ- VAE tokenizer. To achieve a fixed-size input, the polygons are either simplified extended to have the same number of corner points. We found that fixing the maximum number corners to 128 results a minimal change in the overall polygon shape, thus
Path tokenization:
this value for all the path tokenization ablations.
Mask tokenization: this scheme, we first convert each instance binary masks and resize them to fixed mask size. Then, we tokenize them using a ViT-based VQ-VAE tokenizer, similar to the way we tokenize image-like and feature map modalitics.
We investigated LI and MSE losses for both tokenization schemes, and additionally cross-entropy and Dice loss for the mask tokenization. We also investigated the effects of the total number of tokens, token vocabulary size, and mask size. To compare the performance of the resulting tokenizers, we use the loU between the pseudo-labeled and reconstructed instances as our metric. fig. 15 illustrates the results of different ablated configurations. For each configuration, the remaining unspecified parameters are by default set to for number tokens, 1024 for the vocabulary size, L11 for the loss, and 64 x 64 for the mask size. The ablations show that using mask tokenization with tokens, 1024 vocabulary size, and x 64 mask size performs well and sets a good balance between reconstruction quality and total sequence length.
Ablations:
31

---

In all ablations, the tokenizers are trained for 24 epochs starting with 5 warmup epochs using the AdamW [59] optimizer with 81. 32 0.9, 0.999 and batch size of 128. For all the experiments except the Dice loss, learning rate of le-5 is used. Since using this learning rate for the Dice loss experiment resulted in instabilities, we reduced increasing the number of tokens results in better reconstruction quality both for the mask tokenizer and the path tokenizer. Compared to LI loss, the cross-entropy loss training obtains reconstructions with smoother edges and better coverage.
learning rate to 1e-6. As demonstrated fig. 16,
Mask tokenizer
Path tokenizer
Instance ground truth
loss
entropy
L1
3xx tokens
tokens
tokens
tokkess
Figure 16: Different tokenization schemes for SAM instances. We compare different tokenization
schemes to tokenize SAM instances for pre-training. Please see Sec. for details.
Additional Discussions and Results
I.1 Encoder-decoder vs decoder-only architectures
In our work, we decided to use an encoder-decoder architecture to achieve any-to-any multimodal capabilities, while other approaches [74, 87, 92, 10] decoder-only architectures in multimodal settings. Both architectural choices are valid depending on specific use cases and priorities, and future work could explore these trade-off's more sy systematically. Below, we explain our choice of an encoder-decoder architecture and discuss how a similar model might look in a decoder-only setting.
encoder-decoder architecture it is directly compatible with masking approaches (e.g.. T5 [71], MAE [40]), which is at the core of our method and enables any-to-any capabilities with single training objective. In addition, after training, the encoder be extracted and used as a VIT or multimodal encoder. Notably, the bidirectional self-attention in the encoder has been shown have slight benefits causal attention for representation learning & transfer tasks [71, 30].
Following 4M, we
If we were design 4M-like model using decoder-only architecture, there two main approaches to consider:
The first is causal decoder (i.e. next token prediction without span [71] or MAE [40] masking).
This approach is similar to multimodal LLMs operating on interleaved data [74, 87, 92, 10] and is the easiest to unify with LLMs. However, the 4M masking strategy would not be directly compatible with this approach. A naive strategy would modality after another, but this leads to much larger sequence lengths per example and redundancy between modalities. As many of the capabilities shown in our paper rely on cross-modal masking (c.g. the any-to-any capabilities and the ability to predict outputs from partial modalities), it is unclear
keep everything unmasked and concatenate one
32

---

Table 11: Quantitative evaluation of image generation. We compare our model's 's text-to-image generation performance against 4M, Stable Diffusion 2.1-base, as well as controlled text-to-ine text-to-image specialist variant of 4M (T2I-B) trained with full text and masked RGB token inputs, conceptually similar to Muse. We compute FID and CLIP-II/14 scores on COCO validation set after resizing the generations to 256x256. All runs use guidance scale 3.0. 4M-21 consistently outperforms 4M by a good margin. 4M can match the performance of the controlled Muse baseline, but not that of dedicated text-to-image models like SD-2.1, which are trained on orders of magnitude more data and compute.
COCO val (30k)
Model
Res.
Dataset
A100 hours
Text encoder
FID 4
CLIP score
4M-7
224
CC12M
2.3k
37.5
21.4
4M- 21 B
224
CC12M & COYO & C4
3.1k
34.5
22.1
4M-7 L
224
CCI2M
9.2k
30.1
23.1
4M-21 L.
224
CCI2M & COYO & C4
12.3k
27.2
23.7
4M-7 XL
224
CCI2M
24.5k
27.
23.7
4M-21 XL
224
CCI2M & COYO & C4
33.8k
26.0
24.3
T2I-B "Muse"
224
CC12M
1.6k
39.8
20.3
Curated
CLIP VIT-L/14
SD-2.1-base
512
200k
10.1
25.5
LAION-5B
whether those would be achievable with a causal decoder and what engineering challenges this would involve.
The second approach using a prefix LM-like decoder (see Fig. 4 of T5 [71]), where unmasked inputs (i.e., encoder inputs in the 4M formulation) and masked inputs/targets (i.e., decoder inputs in the 4M formulation) are concatenated. The entire sequence then given to a single decoder LLM. This approach allows preserving the masking strategy and training objective within decoder-only architecture, but has seen less adoption than encoder-decoder approaches the masking literature. However, it is more amenable to multi-tury temporal inputs, as multiple sequences of unmasked inputs and masked targets can be concatenated one after the other.
In summary, we use an encoder-decodee architecture as it provides straightforward way to achieve any-to-any capabilities through masking, and allows for downstream reuse of the trained encoder as a singlemodal or multimodal backbone. While decoder-only approaches could potentially be adapted for similar purposes, we are unaware of work demonstrating this the same scale (in terms of number of modalities) and we believe it to be very impactful and exciting research direction.
1.2 Quantitative evaluation of image generation
In table I.2, we quantize our model's conditional generation capabilities by performing caption-to- image generation on COCO. We compare against 4M-7 [65] across all model sizes, Stable Diffusion 2.1 [76], and a controlled text-to- image specialist baseline. The controlled text-to-image baseline (T2I-B), conceptually similar Muse [14], uses the same architecture and RGB tokenizer as our model and 4M-7, and was trained for total of 300B tokens on CC12M [16]. We test for image fidelity using FID and image-text alignment using CLIP score, computed using 30'000 validation set images, and resizing all images to 256x256. We used guidance scale 3.0 for all experiments.
Our models are able consistently outperform 4M-7 across model sizes on COCO, both in terms of FID and CLIP score. While there is still a seizable gap between dedicated text-to-image models like Stable Diffusion 2.1 and our models on out-of-distribution data, we note that these models are usually trained on orders of magnitude more data and compute. Our T2I-B baseline attempts to control for factors such as the tokenizer that can have significant influence on FID, and we see that 4M-7 performs similar to the specialist T2I-B, Optimizing for image generation quality was not the focus of our work, but considering the scaling trends of token-based masked (e.g. Muse [14], MAGE [55]) and auto-regressive models (e.g. Parti [96]), we expect significant improvement with larger model sizes. Furthermore, we expect that recent advances RGB tokenization (e.g. MAGVIT-v2 [98), FSQ [63]) will translate to significant gains in FID for large enough models.
33

---

1.3 Quantitative evaluation of cross-modal retrieval
We provide additional quantitative results for cross-modal retrieval in table I.3. Our model has notable performance for different retrieval tasks (RGB-Tec Text, RGB- Depth, RGB- (GB-Semantic) while being only trained on global embeddings extracted from the RGB images.
Table 12: Cross-modal retrieval quantitative results. We report the performance on ImageNet K and Flickr30k benchmarks (no fine-tuning). For all evaluations, Top- / Top-5 accuracies reported. For cross-modal retrieval on Flickr30k, we perform both RGB-+Text (i.e. R-T) and vice versa. For ImageNet, we the pseudo-labeled depth (D) and semantic segmentation (S) since the dataset does not come with labels. Retrieval is performed using the method described in section 3.2. Cross (x) means the model is not capable of performing that task due to being multimodal model. Dashed (-) means no official result was reported. We make the following observations: 1) Our model successfully matches the performance of DINOv2 and ImageBind for RGB-RGB (R+RR) retrieval, which is their primary domain, despite being a multitask model trained to do several other tasks as well. This indicates that our model distilled their embeddings well via masked modeling objective. 2) On top of that, our model can perform cross-modal retrieval with notable performance across different modalities as shown below. Despite ImageBind being technically capable cross-modal retrieval like R-D, no official result was reported in the paper. We tried to produce that number which led to poor results by ImageBind, so we refrain from reporting.
I KNN
Flickr30k
Model
R-R
T
4M-21
56.89/79.32
66.82/85.95
78.3/92.4
58.0/82.5
42.4/71.6
ImageBind
81.1/94.4
DINOv2
X
82.1/93.9
x
CLIP
79.59/94.0
88.0/98.7
68.7/90.6
Broader Impact
J.1
Computational costs
All models were trained on Nvidia A100 GPUs. The 4M-21 B model was trained for 2 days using 64 A100s. The 4M-21 L model was trained for days using 128 A100s. The largest 4M-21 XL model required 11 days using 128 A100s. Fine-tuning and transfer learning experiments for each model used approximately 20% additional compute compared to its pre- pre-training. Training the various tokenizers (RGB, depth, normals, CLIP, DINOv2, ImageBind, semantic segmentation, SAM edges, and Canny edge detection, SAM instances, and 3D human poses) required roughly days using 8 A A100s each, totaling approximately 60 A A100-days. In total, the primary experiments reported the paper used approximately 120 000 A100-hours, not including additional preliminary experiments and ablations. We estimate the total compute for the full research project, including preliminary and unreported experiments, be 150'000 A100-hours.
J.2
Social impact
We are open sourcing our code and models to support researchers with the democratization of the tools and enable transparent inspection and safeguarding. 4M-21 models trained publicly available datasets with some curation, e.g. people's 's names are redacted CC12M [16]. However, this process still noisy, hence advise caution when using the models for generation.
34

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
We discussed the contributions and claims of our work both the abstract and in the introduction section (Sec. 1). Our experiment sections (Sec. and 4) include several results to reflect them.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations are provided in Sec. of the main paper. broader impact section is also provided in Appendix Sec. J.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
35

---

Answer:
[NA]
Justification:
The paper does not include any theoretical assumptions, proofs or results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide details of experiments in Sections 2, 3 and 4 of the main paper as well as Sections F, G, of appendix including training data, model architecture, objectives and hyperparameters.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
36

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
Justification:
We open source our models and training code at website.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
Sewiiddee/CoeSSumiiiiiiicc.o sooooiiiiiiiooiiciiiicco for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /CodeSubmii
/ntps.cc/publll httpccccccpccccccccceeeccccccccc htt.biiiiggdddeeeddeeubbiiiiii ssi aesionPooo onnocicy)) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The required information provided in sections G, H, I and of the supple- mentary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We follow the best practices from previous work like 4M and Unified IO. The error bars not reported due to the high computational cost of running each experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
37

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes)
Justification:
We discuss the compute resources used in Section of the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: / /ttp///eersp//eeeee co httpppllttttisptsiiiiplipappppee
Answer: [Yes]
Justification:
We follow the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts are discussed section K of the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
38

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We discussed broader impacts of our models in Sec. K of the appendix, we will release our models and code with standard safeguards.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cited the assets used in the paper accordingly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
39

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release any assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
40


Reviews:
[{"review_id": "wVNqDeK56J", "reviewer": "Reviewer_b2ju", "summary": "The authors present a new vision model that can generate tokens in all directions that represents multiple modalities like RGB images, depth map, segmentation maps, color palette, DINOv2 features etc. given a conditioning on a subspace of the modalities.\nThey develop a new way to tokenize certain modalities and scale both model’s parameters and dataset’s size.", "questions": "- What are the performances of the trained tokenizers? Given that your model will be somewhat bounded by the tokenizers’ performances, could you provide a table including the performances of all tokenizers versus the original model and yours?\n- How did you chose each vocabulary size?\n- The metadata picked by the authors seem arbitrary, could you explain your decision process?\nl.223 I don’t understand why you treat RGB tokens and pixels differently: ‘we leverage RGB patch embeddings learned during the pre-training, as RGB pixel inputs are used alongside the tokenized modalities’. Could you expand on this please?\n\nThe paper is interesting nonetheless but could be way, way stronger with quantitative evaluations on the new claimed capabilities.", "limitations": "The authors talk about all the limitationsof their work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "- Good performances on a lot of downstream tasks. The authors evaluate the model on a wide variety of tasks and show promising results that are either on-par or above very strong baselines.\n- Showing scaling to bigger models isn’t easy\n- Multimodal retrieval is interesting to see and could be leveraged for a lot of new interesting tasks.", "weaknesses": "- The added value over 4M is mostly due to engineering.\n- No multimodal retrieval nor conditional generation metrics. The actors emphasize a lot the generative/multimodal part of the model but the only quantitative experiments are shown in Table 3. Qualitative figures are interesting but should be supporting quantitative results. The conditional generation evaluation needs to be compared to other models like ControlNet [1] and you should at least provide precision/recall@k on some modalities for the multimodal retrieval (like comparing DINOv2/ImageBind with one modality against your model with one modality and your model with multiple ones).\n- The tokenization might reduce the burden of finding good losses’ weight, but tokenizers trained on a specific domain (like CC12M) might not generalize well. Moreover, tokenizers might not capture high frequency details that are very important like text for OCR, good quality faces, or smaller details. In [2], the authors speak about the fact that “quality limitations of the VQ image reconstruction method inherently transfer to quality limitations on images”. They also focus on the different domains that are not well captured by the tokenizers. The current paper only includes one ablation on the tokenizer, and I’m skeptical of the tokenizers’ performances on out-of-distribution images.\n\n[1] Adding Conditional Control to Text-to-Image Diffusion Models, Zhang, Rao and Agrawala\n[2] Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors, Gafni et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer b2ju for the constructive feedback. We address the main concerns and questions in the following response:\n\n> What are the performances of the trained tokenizers?\n\nPlease see the following items as the response: 1) Figure 1 in the rebuttal `PDF` for a visual comparison of multimodal tokenizer reconstructions (*resolution 224x224*), 2) the “tokenizer bound” reported in the main paper's Table 1 that represents the performance upper bound — it clearly shows that the prediction errors of all models are notably worse than the tokenization error indicating that the **lossiness of discrete tokenization is not the bottleneck**.\n\n> Generalization of tokenizers\n\nWe train tokenizers on CC12M to match our training data distribution, and find them to perform well within that distribution (see above answer). For more niche domains not captured in web-scale image distributions (medical, satellite, etc.), we recommend training domain-specific tokenizers. That said, we note that **tokenizers trained on web-scale data perform significantly better in areas like human face and text reconstruction** compared to popular tokenizers like VQ-GAN [1] that were trained on ImageNet-1k; a finding supported by MAGVIT-v2 [2]. \n\n> Capturing high-frequency details\n\nThis is challenging for discrete tokenizers, but can be addressed by increasing token density and performing an additional token super-resolution step, similar to 4M and Muse. Furthermore, recent advances in VQ-free tokenization [2,3] show strong scaling trends in reconstruction and generation performance relative to the vocabulary size.\n\nWe would like to stress that **discrete tokenization is one of the key design decisions that allows us to scale a unified model to such a large and diverse set of modalities**, without multi-task loss-balancing, task-specific heads, or encountering training instabilities. We expect the above-mentioned advances in discrete tokenization to show direct benefits in multimodal generation and out-of-the-box performance.\n\n> Multimodal retrieval numbers\n\nPlease refer to the `PDF` Tab. 2 for additional evaluations on cross-modal retrievals. **Our model has notable performance for different retrieval tasks (RGB-Text, RGB-Depth, RGB-Semantic) while being only trained on global embeddings extracted from the RGB images.** \n\nRegarding DINOv2, it is trained only on RGB images and is not a multimodal model, thus capable of only RGB-RGB retrieval. We also compared our model’s RGB retrieval performance with DINOv2/ImageBind using ImageNet kNN classification, (Tab. 1 of the main paper, also included in the `PDF`). The results in the table show that **our model’s RGB retrieval capability successfully matches DINOv2 and ImageBind, and on top of that, it can perform multimodal retrieval which DINOv2 and other similar models cannot.**\n\n> Conditional generation numbers\n\nPlease see the `PDF` for detailed caption-conditioned generation metrics on COCO, as well as Section 2 of the common response for a discussion.\n\n> How did you choose each vocabulary size?\n\nWe follow the best practices from community and tune the vocabulary size to obtain low reconstruction error while keeping the vocabulary size as small as possible. Please see Fig. 15 and Appendix J for an ablation.\n\n> Metadata decision process\n\nOur choice of which metadata to include was guided by Omnidata [4] and SDXL [5]. Omnidata extracts several types of metadata from a multimodal dataset to give researchers a high-level set of controls to steer the data distribution towards high/low walkability, occlusion, etc. Our goal was to expose similar parameters, but for a multimodal generative model. **We believe that such capabilities provide a path to answer questions on what the ideal multimodal data distributions are for different downstream tasks.**\nFurthermore, we looked towards SDXL and more broadly ControlNet [6] as inspiration for metadata that can be used for image generation, e.g. original image size, colorfulness, etc. While many types of metadata can be included as text in a prompt, we observe strong condition adherence with metadata. In summary, our aim was to provide a broad set of control knobs by extracting various parameters from RGB images and the different pseudo labels. **We note that this set can be easily extended towards scores like aesthetics, NSFW, watermark, etc.**\n\n> Treating RGB tokens and pixels differently\n\nThank you, this is an important design choice in current multimodal models.\n\nDiscrete tokens enable iterative sampling, making them useful for generative tasks, which is why most token-based generative methods (e.g., Parti [7], MaskGiT [8]) use them. However, discretization leads to information loss, which is not ideal for visual perception tasks.\nOn the other hand, using RGB pixels as input is more suitable for visual perception tasks. By avoiding the discrete bottleneck, there is no information loss during the tokenization step, and the projection layer can be more lightweight (in 4M's case, it is just a simple linear projection).\nGiven these tradeoffs, we follow 4M by training on both and treating them as separate modalities, with RGB pixels as an input-only modality. **This allows us to choose the most appropriate representation for a given task - discrete RGB for generation or RGB pixels for perception.**\n\n> The added value is mostly due to engineering.\n\nWe find it hard to concretely respond to this comment as a weakness since, first, many undeniably key contributions in the community are basically “engineering”, and second, what would be “engineering” is too vague and broad. **Engineering doesn’t mean unimpactful.** All reviewers positively recognized that this paper required a significant amount of exploration and work, e.g., in terms of scaling, and the workload was viewed as a strength of the paper. Besides those, the paper made several contributions and introduced new capabilities across key axes, as summarized in L52-L70 in the main paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CQElXJCPVL", "reviewer": "Reviewer_PHF3", "summary": "The paper addresses the limitations of current multimodal and multitask foundation models, such as 4M and UnifiedIO, which are constrained by the limited number of modalities and tasks they can handle. The authors present a model trained on a wide variety of modalities and tasks using large-scale multimodal datasets and text corpora. This includes training on images, text, semantic and geometric modalities, feature maps from state-of-the-art models, and new modalities like image metadata and color palettes. A key technique used is discrete tokenization of various data types. The new model can handle at least three times more tasks and modalities than existing models without losing performance. This approach also enhances fine-grained and controllable multimodal generation and explores the unification of diverse models into a single one.", "questions": "1. The paper uses an encoder-decoder paradigm, whereas we know that the commonly used LLMs nowadays typically employ a decoder-only structure or simply a multi-layer transformer structure. Why does the paper adopt the encoder-decoder paradigm? The decoder-only approach[1][2] seems simpler and better suited for unifying with the LLM structure.\nThe authors are advised to include some discussion on encoder-decoder and decoder-only approaches, referencing papers such as the following.\n\n[1] GiT: Towards Generalist Vision Transformer through Universal Language Interface. (ECCV 2024)\n\n[2] Fuyu-8B: A Multimodal Architecture for AI Agents. (Blog)\nThe two articles mentioned above also tokenize everything within a simple unified framework.\n\n2. Do different feature maps count as different modalities? I thought different modalities referred to the basic levels such as image, language, and speech, etc. The formulation in the paper is somewhat strange, and I hope the authors can explain this a bit. After all, these features are generated by cost-intensive multi-modal encoders, which differs from the lightweight multimodal tokenizer proposed in the paper. I also hope the authors can discuss this in comparison with the approaches of GiT[1] and Fuyu-8B[2], which use light-weight yet simple tokenizers on original image and language.\n\n3. Regarding the challenge of negative transfer, you can also refer to the GiT paper. It demonstrated that joint training on five tasks has better performance than single-task training. (Considering that this paper is also recent, a direct comparison may not be necessary, but it is still recommended to discuss it.) :)\n\nOverall, I am inclined to accept this paper, but I still hope the authors can address my concerns. If these issues are addressed well, I will accordingly raise my score. :)", "limitations": "There are no comments regarding the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper has a good starting point, noting that the network structures across various AI fields are converging (mostly to transformers).\n\n2. This paper involves a significant amount of engineering work. Organizing large amounts of data and conducting large-scale training is not easy.\n\n3. The experiments and performance in the paper are quite good.", "weaknesses": "1. The writing of the paper has room for improvement. Not all reviewers have read the 4M paper. The paper does not detail the training framework, such as how to control different tasks and how to conduct multimodal masked training. These aspects are not simply described and are only mentioned in section 2 as following the 4M paper. Including a simple diagram would significantly improve clarity.\n\n2. Some claims are somewhat inaccurate. There are works that use very simple methods to unify modeling for multiple tasks, maybe not as many as 10. It is necessary to discuss the difference and challenges between completing 5 tasks and 10+ tasks, as some tasks can actually be categorized as a single task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer PHF3 for the positive feedback. We address the main concerns and questions in the following response:\n\n> Encoder-Decoder vs Decoder-only.\n\nThank you for your question. **Both encoder-decoder and decoder-only architectures are valid design choices, depending on the specific use cases and priorities, and this is somewhat that can be explored more solidly in future work.** We explain the reasons for our choice of an encoder-decoder architecture below and discuss how a similar model might look like in a decoder-only setting.\n\nFollowing 4M, we use an encoder-decoder architecture as it is **directly compatible with masking approaches (e.g., T5, MAE), which is at the core of our method and enables any-to-any capabilities with a single training objective**. In addition, **after training, the encoder can be extracted and used as a ViT or a multimodal encoder**.  Notably, the bidirectional self-attention in the encoder has been shown to have slight benefits over causal attention for representation learning & transfer tasks (e.g., Table 3d of AIM [9], Table 2 of T5 [10]).\n\nIf we were to design a 4M-like model using a decoder-only architecture, there are two main approaches to consider:\n\nThe first is a causal decoder (i.e. next token prediction without span or MAE masking). This approach is similar to multimodal LLMs operating on interleaved data (e.g. Gato [11], Fuyu [12], GiT [13], Chameleon [19]) and is the easiest to unify with LLMs. However, the 4M masking strategy would not be directly compatible with this approach. A naive strategy would be to keep everything unmasked and concatenate one modality after another, but this leads to much larger sequence lengths per example and redundancy between modalities. **As many of the capabilities shown in our paper rely on cross-modal masking (e.g. the any-to-any capabilities and the ability to predict outputs from partial modalities), it is unclear whether those would be achievable with a causal decoder and what engineering challenges this would involve.**\n\nThe second approach is using a prefix LM-like decoder (see Fig. 4 of T5 paper [10]), where unmasked inputs (i.e., encoder inputs in the 4M formulation) and masked inputs/targets (i.e., decoder inputs in the 4M formulation) are concatenated. The entire sequence is then given to a single decoder LLM. **This approach allows preserving the masking strategy and training objective within a decoder-only architecture, but has seen less adoption than encoder-decoder approaches in the masking literature.** However, it is more amenable to multi-turn or temporal inputs, as multiple sequences of unmasked inputs and masked targets can be concatenated one after the other.\n\nIn summary, **we use an encoder-decoder architecture as it provides a straightforward way to achieve any-to-any capabilities through masking, and allows for downstream reuse of the trained encoder**. While decoder-only approaches could potentially be adapted for similar purposes, we are unaware of work demonstrating this at the same scale (in terms of number of modalities) and we believe it to be a very impactful and exciting research direction. \n\nAs suggested, we will include additional references to decoder-only multimodal LLMs as well as parts of this discussion in our camera-ready version.\n\n> Do different feature maps count as different modalities? Comparison with approaches of GiT and Fuyu-8B.\n\nThis highlights an important point; thanks for the question. Three clarifications: \n\n1) our primary goal in this paper is ‘modeling’ work, e.g. **demonstrating the possibility of having one model that effectively does any-to-any prediction across many diverse modalities**. Once that is established, the community can adopt the model on their own modalities that are justified based on the use case. **Our goal is not committing a specific dictionary of modalities or ruling out others, but using a set that provides diversity and is relatively large to enable evaluation.** \n\n2) **‘what qualifies as a modality’ is an interesting nontrivial question.** Many modalities, e.g., depth/3D, heat, etc., can be sensed using sensors, but also, they can be effectively inferred out of RGB images with a powerful task-specific prediction model. Being inferable with a task-specific neural network does not question if depth/3D qualifies as a modality of information. **In general, in theory, any specific mode of information about an underlying scene qualifies as a modality.** Which modalities are ‘useful’ is a different deeper question, often application specific, and out of our scope as the paper’s focus is making progress on the model. \n\n3) adopting neural network feature maps as a modality is useful as, they can be **used for chained conditioning [18], and also they provide a way for distilling popular single networks (e.g. DINO) into one multitask network**. \n\nSimilar to GiT, we employ tokenization of several modalities via language, e.g. bounding boxes, metadata and color palette. However, this strategy is limited for tokenizing dense representations such as depth, normals, edges, instance segmentation masks, or feature maps. **Thus, we employ different tokenization schemes to handle such modalities as depicted in Figure 3.** We will add a discussion on this to the paper.\n\n> Challenge of negative transfer.\n\nThanks for the pointer, we will add a discussion. While we didn’t observe negative transfer when more modalities are included, investigating this direction further could be worthwhile. \n\n> Detail the training framework\n\nThanks. We will add a diagram to camera-ready to improve the clarity as suggested.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9wxn2K38Hu", "reviewer": "Reviewer_66Uu", "summary": "The paper presents an advanced vision model capable of handling a wide range of tasks and modalities, demonstrating the potential to train a single model on tens of diverse modalities without a loss in performance compared to specialized models. Specifically, the model is trained on a multitude of modalities, including RGB images, depth, semantic segmentation, CLIP features, surface normals, and more, enabling it to perform various tasks such as image generation, retrieval, and understanding.", "questions": "see weakness", "limitations": "see weakness", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The presentation is clear and detailed.\n+ The performance of the proposed model is good.\n+ The workload of this paper is impressive.", "weaknesses": "1. Do the authors try to measure the quality (e.g. FID ) of generated images (based on caption input) and can the proposed method surpass some existing diffusion-based models?\n\n2. The authors should illustrate the detailed configurations of B, X, and XL models, for example, the number of layers and the width of channel dimensions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 66Uu for the positive feedback. We address the main concerns and questions below:\n\n> Quality of generated images\n> \n\nPlease see the `PDF` for detailed caption-conditioned generation metrics on COCO, as well as Section 2 of the common response for a discussion.\n\n> Detailed configurations of B, L, and XL models\n> \n\nThank you for pointing this out. We've listed the model configurations in the table below. These configurations are derived from 4M and closely resemble those of T5 and UnifiedIO to ensure comparability with other encoder-decoder models in the literature.\n\nNote that we use SwiGLU [15] in the feedforward, which employs three weight matrices instead of the typical two. To maintain equivalent parameter and operation counts, we've adjusted the feedforward dimension by scaling it by 2/3 (4 x 2/3 instead of 4x).\n\n| Model    | Encoder Blocks | Decoder Blocks | Model Dim | Feedforward Dim | Num Heads | Total Params |\n|----------|----------------|----------------|-----------|-----------------|-----------|--------------|\n| Ours B  | 12             | 12             | 768       | 2048            | 12        | 198M         |\n| Ours L  | 24             | 24             | 1024      | 2730            | 16        | 705M         |\n| Ours XL | 24             | 24             | 2048      | 5461            | 32        | 2818M        |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## **Response to all reviewers**

We thank the reviewers for their insightful comments and constructive feedback. We are pleased that they commended our performance with remarks such as: **“good performances on a lot of downstream tasks”** (b2ju), **“The experiments and performance in the paper are quite good”** (PHF3), and “**The performance of the proposed model is good”** (66Uu).

The reviewers also recognized our scaling efforts: **“Showing scaling to bigger models isn’t easy”** (b2ju), **“This paper involves a significant amount of engineering work. Organizing large amounts of data and conducting large-scale training is not easy”** (PHF3), and **“The workload of this paper is impressive”** (66Uu).

Additionally, we are glad that the reviewers found our results promising that **“could be leveraged for a lot of new interesting tasks”** (b2ju), and praised our presentation as **“clear and detailed”.** (66Uu)

### 1. Additional results overview

We address the reviewers’ remaining questions and concerns in the individual responses and rebuttal `PDF` . We discuss the questions on the quality of generated images below. We also provide a list of additional results to address reviewer questions:

- b2ju, 66Uu: Conditional generation numbers (`PDF` Tab. 1)
- b2ju: Tokenizer performance (`PDF` Fig. 1)
- b2ju: Multimodal retrieval numbers (`PDF` Tab. 2)

### 2. Common questions

> b2ju, 66Uu: Quality of generated images
> 

In Tab. 1 of the rebuttal `PDF`, we quantize our model's conditional generation capabilities by performing caption-to-image generation on COCO. We compare against 4M [18] across all model sizes, Stable Diffusion 2.1, and a controlled text-to-image specialist baseline. The controlled text-to-image baseline (T2I-B), conceptually similar to Muse [16], uses the same architecture and RGB tokenizer as our model and 4M, and was trained for a total of 300B tokens on CC12M. We test for image fidelity using FID and image-text alignment using CLIP score, computed using 30'000 validation set images, and resizing all images to 256x256. We used guidance scale 3.0 for all experiments.

**Our models are able to consistently outperform 4M across model sizes on COCO, both in terms of FID and CLIP score**. 

While there is still a seizable gap between dedicated text-to-image models like SD2.1 and our models on out-of-distribution data, we note that **these models are usually trained on orders of magnitude more data and compute**. Our T2I-B baseline attempts to control for factors such as the tokenizer that can have a significant influence on FID, and we see that Ours-B performs similar to the specialist T2I-B.

Optimizing for image generation quality was not the focus of our work, but considering the scaling trends of token-based masked (e.g. Muse [16], MAGE [17]) and auto-regressive models (e.g. Parti [7]), **we expect significant improvement with larger model sizes**. Furthermore, we **expect that recent advances in RGB tokenization** (e.g. MAGVIT-v2 [2], FSQ [3]) **will translate to significant gains in FID** for large enough models.

### 3. References used in the rebuttal:

[1] Taming Transformers for High-Resolution Image Synthesis, Esser et al., 2020

[2] Language Model Beats Diffusion – Tokenizer is Key to Visual Generation, Yu et al., 2023

[3] Finite Scalar Quantization: VQ-VAE Made Simple, Mentzer et al., 2023

[4] Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans, Eftekhar et al., 2021

[5] SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis, Podell et al., 2023

[6] Adding Conditional Control to Text-to-Image Diffusion Models, Zhang et al., 2023

[7] Scaling Autoregressive Models for Content-Rich Text-to-Image Generation, Yu et al., 2022

[8] MaskGIT: Masked Generative Image Transformer, Chang et al., 2022

[9] Scalable Pre-training of Large Autoregressive Image Models, El-Nouby et al., 2024

[10] Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, Raffel et al., 2019

[11] A Generalist Agent, Reed et al., 2022

[12] Fuyu-8B: A Multimodal Architecture for AI Agents (Blog), Adept, 2022

[13] GiT: Towards Generalist Vision Transformer through Universal Language Interface, Wang et al., 2024

[14] ImageBind: One Embedding Space To Bind Them All, Girdhar et al., 2023

[15] GLU Variants Improve Transformer, Shazeer, 2020

[16] Muse: Text-To-Image Generation via Masked Generative Transformers, Chang et al., 2023

[17] MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis, Li et al., 2022

[18] 4M: Massively Multimodal Masked Modeling, Mizrahi et al., 2023

[19] Chameleon: Mixed-Modal Early-Fusion Foundation Models, Chameleon Team, 2024

Meta-review: This submission presented a multimodal framework trained on a wide range of modalities and tasks. The authors propose to use large-scale datasets including images, text, semantic and geometric modalities, feature maps from state-of-the-art models and new modalities such as image metadata or color. Processing at least three times as many tasks and modalities as existing models, this unified single model delivers excellent results and also improves fine-grained, controllable multimodal generation.  
Reviewers asked for several clarifications. After the rebuttal, all were satisfied with the answers.  
The reviewers and AC are ultimately convinced by this submission, considering that the novelty and interest of the proposed multimodal model and learning framework are clearly evident.  
The authors are strongly encouraged to take all comments into account in their final version.
Decision: Accept (poster)