4M: Massively Multimodal Masked Modeling
Authors: David Mizrahi
Roman Bachmann
Oguzhan Fatih Kar
Teresa Yeo
Mingfei Gao
Afshin Dehghan
Amir Zamir
Venue: NeurIPS 2023 spotlight (2023)
Keywords: multimodal learning
multitask learning
representation learning
transfer learning
foundation models
generative models
computer vision

Abstract: Current machine learning models for vision are often highly specialized and limited to a single modality and task. In contrast, recent large language models exhibit a wide range of capabilities, hinting at a possibility for similarly versatile models in computer vision.
In this paper, we take a step in this direction and propose a multimodal training scheme called 4M. It consists of training a single unified Transformer encoder-decoder using a masked modeling objective across a wide range of input/output modalities  – including text, images, geometric, and semantic modalities, as well as neural network feature maps. 4M achieves scalability by unifying the representation space of all modalities through mapping them into discrete tokens and performing multimodal masked modeling on a small randomized subset of tokens.

4M leads to models that exhibit several key capabilities: (1) they can perform a diverse set of vision tasks out of the box, (2) they excel when fine-tuned for unseen downstream tasks or new input modalities, and (3) they can function as a generative model that can be conditioned on arbitrary modalities, enabling a wide variety of expressive multimodal editing capabilities with remarkable flexibility.

Through experimental analyses, we demonstrate the potential of 4M for training versatile and scalable foundation models for vision tasks, setting the stage for further exploration in multimodal learning for vision and other domains.

4M: Massively Multimodal Masked Modeling
David Mizrahi .2+
Roman Bachmann 1+
Oguzhan Fatih Kar!
Teresa Yeo Mingfei Gao
Afshin Dehghan
Amir Zamir
Swiss Federal Institute of Technology Lausanne (EPFL.
AAplle
https://4n.ep:: epfl.cl.
Abstract
Current machine learning models for vision are often highly specialized and limited to single modality and task. In contrast, recent large language models exhibit a wide range of capabilities, hinting at a possibility for similarly versatile models in computer vision. In this paper, we take step in this direction and propose multimodal training scheme called 4M. It consists of training single unified Transformer encoder-decoder using a masked modeling objective across a wide range of input/output modalities - innlluding text, images, geometrie, and seman- tic modalities, as well as neural network feature maps. 4M achieves scalability by unifying the representation space of all modalities through mapping them into dis- crete tokens and performing multimodal masked modeling small randomized subset of tokens.
4M leads to models that exhibit several key capabilities: (I) they can perform a diverse set of vision tasks out of the box. (2) they excel when fine-tuned for unseen downstream tasks new input modalities, and (3) they can function as generative model that can be conditioned on arbitrary modalities, enabling a wide variety of expressive multimodal editing capabilities with remarkable flexibility.
Through experimental analyses, we demonstrate the potential of 4M for training versatile and scalable foundation models for vision tasks, setting the stage for further exploration in multimodal learning for vision and other domains.
Introduction
In recent years, the field of natural language processing (NLP) has seen a shift toward training large language models (LLMs) that are inherently capable of performing wide range of tasks without requiring extensive task-specific adaptations [12, 25]. While these models have demonstrated remarkable success in NLP, there remains a need to develop similarly versatile and scalable models for vision. A crucial aspect of scalability and versatility in vision is the ability to handle multiple (input) modalities and (output) tasks, as vision models must deal with a diverse range of sensory inputs, such as images, 3D, and text and solve wide range of tasks. Unlike NI.P, where language modeling on raw text has led multitask capabilities [84, 86], training on only RGB images with single objective has not exhibited the same behavior for vision. Therefore, it is deemed important to incorporate multiple modalities and tasks in training. It has been indeed suggested by psychophysical studies that multimodality is one key driver behind the development of biological intelligence [104].
To create a model that exhibits the desirable properties foundation models in vision, is important to consider three key aspects in terms of scalability: data, architecture, and training objective. For data, scalability means being able to benefit from more training samples toward improving performance.
For clarity, "modalitie"" usually denote the inputs to model (e.g. sensory signals), and "tasks" usually denote the outptt (e.g. semantics). Our method enables symmetric input-output structure, thus use "modalities" and "tasks" interchangeably in this paper. "Equal contribution & corresponding authors.
37th Conference on Neural Information Processing Systems (NeurIPS 2023).

---

A generalist vision model that can...
multimodal generative model that can...
perform diverse of sision tasks out the
generate any modalities conditioned any othertsi
Satao
Sarande
the
Capriier potel ia espa
RG0
deptn nomaa
Nammila CLP + FGB
easily fine-tuned specialist variants
with varying conditoning weights and from partal inputs...
paietiy Paania Crain
Capioo:
4M
Castion betongi & (nya)) Hits
rabite namals
transfor unsean tasks and modelities
anabling precise usar control through muttimodal aditing chains
Organal Iaae
FoB and acd caption
Dc sod image
Caprioe: yoall coniaanse
Malll trordww, RGB degth semande
6 anges - cuundruu
4M enables training a versatile multimodal and multitask model, capable of performing a diverse set of vision tasks out of the box, as well as being able to perform multimodal conditional generation. This, coupled with the model's ability to perform in-painting, enables powerful image editing capabilities. This generalist model transfers well to a broad range of downstream tasks or to novel modalities, and can be easily fine-tuned into more specialized variants of itself.
Figure I:
In terms of architecture, scalability implies increased performance with growing model size and remaining stable when trainingd at large sizes. Lastly, a scalable training objective should efficiently handle a growing number of modalities without incurring excessive computational costs. In our approach, we target scalability across these three aspects while maintaining compatitiiiii multiple modalities.
compatibility with
We address these challenges by proposing method consisting of training a single unified Transformer encoder-decoder using multimodal masked modeling objective. We name this approach 4M (short for "Massively Multimodal Masked Modeling") to emphasize its ability to scale to many diverse modalities. Our method unifies the benefits of multimodal learning and masked modeling, such as (I) serving as an effective pre-training objective for learning rich representations [30, 48], (2) leading to
strong cross mross-modal predictive coding abilities and shared scene representations [5. 44], (3) enabling
models to be used for generative tasks through iterative sampling [17, 18]. Crucially, 4M combines these benefits while remaining efficient via a number of mechanisms.
To enable training a single Transformer on modalities with different formats, like text, bounding boxes, images, or neural network features, we choose to unify their representational spaces by mapping them into sets or sequences of discrete tokens [21, 22, 74, 64] using modality-speciffc tokenizers [110]. This tokenization approach enhances compatibility, scalability, and sharing by removing the need for task-specific encoders and heads, allowing the Transformer to be compatible Also, although 4M operates on a large set of modalities, it can train in a highly efficient manner through the use of inpul and target masking. This involves randomly selecting a small subset of tokens from all modalities as inputs to the model, while another small subset of the remaining tokens is treated as targets. Decoupling the number of input and target tokens from the number of modalities prevents the computational cost from rapidly escalating with increasing modalities, allowing for a scalable training objective.
with all modalities and maintain full parameter-sharing
Leveraging the availability of single-modal or text-image pair datasets, such as CC12M [19], we employ strong pseudo labeling networks to generate aligned binding data across various modalities. This pseudo labeling approach enables training on diverse and large-scale datasets without demanding them to come with muliimoda//multitakk annotations.
4M models capable of performing many key vision tasks out of the box and can also be fine-tuned to achieve highly competitive performance on unseen downstream tasks and input modalities. In
'e sometimes also refer the trained models 4M (short for "Massively Multimodal Masked Model").

---

addition, training using multimodal masked modeling objective leads to steerable generative models that can be conditioned on arbitrary modalities, enabling the user's intent be expressed in versatile manner (as depicted in Figure 4) as well as various multimodal editing tasks (see Figure 1).
We further perform an extensive ablation analysis that studies the factors affecting 4N's performance. This thorough examination, along with the simplicity and generality of our approach, demonstrates the potential of 4M for a wide range of vision tasks and further expansions.
Our main contributions and results can be summarized as follows:
we introduce 4M, a framework for training versatile and scalable foundation models for vision tasks using a multimodal masked modeling objective. Our approach results in models that learn rich representations and perform well on wide range of tasks without requiring task-specific adaptations.
Method:
Performance: we demonstrate the efficacy of our approach through extensive experiments and benchmarks, showcasing the ability of these models to perform many key vision tasks out of the box, as well as achieving highly competitive performance when fine-tuned on unseen downstream tasks.
3. Generative Capabilities: we showcase the flexible and steerable generative capabilities of models trained using 4M, enabling variety of multimodal editing tasks utilizing conditioning arbitrary modalities.
4. Experimental Study:
we conduct an extensive ablation analysis to study the factors affecting 4M's performance, providing important insights into these models' behavior and design.
Code, models, and additional interactive visualizations are available at https: : //4m. epfl ch.
2 Method Description
The 4M architecture and training objective (depicted in Figure 2) were designed with focus on being as compatible and scalable possible in terms of the number and type of modalities it accepts, while being conceptually simple and computationally efficiens. We enable these through the conjunction of the following key aspects:
1. Tokenizing modalities: We abstract away modality-specifff intricacies by mapping all modalities into sequences or sets of discrete tokens, whether they are images, text, sparse data, or neural network feature maps. This allows every possible mapping between modalities to be seen as predicting one sequence or set of tokens from another. In Section 2.1, we discuss what types of modalities we train on, how we generate the training data, and how we enable training a model on different modalities through tokenization.
2. Training single compatible network all modalities: Different tasks in vision, NLP and other domains traditionally required vastly different modeling choices, architectures, and losses, making the joint training on multiple modalities challenging. Tokenizing all modalities into unified representation space allows us to train single Transformer encoder-decoder (see Figure 2) to map between different modalities through (parallel or serialized autoregressive) token prediction. In Section 2.2, we provide more details the 4M architecture.
Multimodal masked pre-training objective: Transformers have demonstrated excellent scalabil- ity with data and model size across a diverse set of tasks [60, 2]. particularly when paired with scalable pre-training objective such as masked reconstruction [30, 48, 56]. In Section 2.3, we detail our approach to training 4M using multimodal masked modeling objective on randomized token subsets to learn strong cross-modal predictive coding abilities.
2.1 Modalities & data
Pre-training modalities. We train 4M models diverse set of modalities, namely RGB, captions, depth, surface normals, semantic segmentation maps, bounding boxes, and tokenized CLIP feature maps [85, 82, 114]. These modalities were chosen to cover several key aspects: First, they contain a mix of semantic information (captions, semantic segmentation, bounding boxes. CLIP). geometric information (depth, surface normals), and RGB. When used as input modalities, these modalities can be used informative priors about the scene geometry and its semantic content [98, 69], and when used as target tasks, they allow us to steer what kind representations are learned [42, 5, 82, 1 114]. Second, these modalities are diverse in terms of the format they use to encode information. They consist of dense visual modalities (RGB, depth, surface normals, semantic segmentation), sparse and/or sequence-base modalities (captions, bounding boxes), as well as neural network feature maps (CLIP [85]). Finally, these modalities allow for diverse and rich interaction with the model for
3

---

Tokenization
4M multimodal masked pre- wre-training
RGB
Depti
donny semoil number bageet tollens
Segmertation
Transformer encoder
Transformer decoder
CLIP features
Captton
conference Chaa allo and
Bounding
a faxed number Inout Takors
Maskc tokeee for left-shiitee tokenss forr sequericet)
Figure 2: Method overview. (Left): 4M is framework for training multimodal and multitask models that operate on tokenized versions multiple image-like modalities (such as RGB, depth, etc.) and sequence modalities (such captions and bounding boxes). (Right): The 4M pre-training objective consists of training Transformer encoder-decoder predict randomly selected subset tokens, which is sampled from all modalities, based on another random subset of tokens.
generative purposes, For example, captions, segmentation maps, and bounding boxes allow for semantically conditioned generation, while geometric modalities enable grounding the generation on 3D information. 4M's versatility in handling various modalities, its capacity to benefit from cross-training, and its ability to learn cross-modal predictive representations (as demonstrated in Sections 3 and 4) suggest its potential for extension even more modalities.
Pseudo labeled multimodal training dataset. Training 4M models requires a large-scale and aligned multimodal/mmltiing
dataset that contains all the above modalities/tasks and is sufficiently diverse. Most multimodal datasets, however, either do not contain all our pre-training modalities [32], are too small [94], or are not diverse enough [126]. For those reasons, we resort to pseudo labeling [42, 5] the publicly available Conceptual Captions 12M (CCI2M) [19] as binding dataset using powerful off-the-shelf models. Because this approach only requires access to a dataset RGB images, may scale to even larger web-scale image datasets [99, 14, 39].
Tokenization. All modalities are mapped to sets sequences of discrete tokens (indices of vocabulary) through the use of modality-speciife tokenizers. Captions and bounding boxes are both treated as text and encoded using WordPiece [30]. For modeling bounding boxes, we follow the approach of Pix2Seq [21], that turns the task of object detection into sequence prediction problem. RGB, depth, normals, semantic segmentation maps, and CLIP feature maps are tokenized using learned vector quantized autoencoders (VQ- VAE) [110]. Unlike Unified-IO [74] that represents all image-like modalities using RGB pre-trained VQ-GAN [35], instead modality-specific tokenizers. This allows us to incorporate neural network feature maps that would otherwise be difficult to represent with existing image tokenizers. While mapping modalities tokens and vice versa incurs a small computational overhead during inference. we avoid this overhead during pre-traing pra-training by pre-computing the tokens while assembling our multimodal dataset.
We provide a detailed overview of the multimodal dataset, pseudo labeling procedure, and tokenization in Appendix B.
2.2 Multimodal Transformer
We design the architecture of 4M with efficiency, scalability, and simplicity in mind. 4M's architecture closely resembles standard Transformer [112] encoder-decoder but includes few crucial modifica- tions to enable joint modeling of multiple different image-like like modalities, such as RGB or semantic segmentation, but also of sequence modalities, such as captions or bounding boxes.
Multimodal encoder. The encoder is standard Transformer encoder but features podality-specifff learnable input embedding layers to map token indices vectors. To each token of specific modality, we add a learnable modality embedding and either ID (for sequences) or 2D (for dense modalities) sine-cosine positional embeddings. To facilitate transfer learning, the encoder is additionally designed to accept RGB pixels using a learnable patch-wise wise linear projection, enabling it to double as a Vision Transformer [3]] backbone.

---

4M chained multimodal generation
Detokenization
Iterration Igenerate / inppaint RGB with MeskGrt)
Iterstion (genenate caption vutoregresssvvel))
RGB
[
Iverattinn
Iteration
Caption (S0) harso
Iteration
Iterstion
Bounding boxes
3maAr-0.6 0.80
Figure 3: Chained multimodal generation. This simplified example illustrates the generation of full RGB image from partial RGB and bounding box input using the MaskGIT [17] decoding scheme, followed by autoregressive generation of caption. Note that through chaining (i.e. using fully generated modalities as conditioning when generating subsequent modalities), we can predict multiple modalities in self-consistent manner. This is in contrast to independently generating each modality from the original conditioning, where each generated output consistent with the input but not necessarily with other outputs. See Figures 9 and 10 for visual examples of chained generation. Generated tokens can be turned back into images, text, and other modalities, using the detokenizers.
Multimodal decoder. The decoder handles tokens from both dense image-like and sequence-like modalities, with each type requiring different approach. However, aspects are common to all tokens: First, they can all freely attend to any encoder tokens in the cross-attention layers, ensuring full access to the encoded information. Second, we employ attention masks to separate decoder tokens of different modalities. This ensures that the decoder produces consistent outputs for each specific modality, irrespective of what other outputs are being generated simultaneously. For dense mike modalities, the decoder input consists of mask tokens along with modality and positional information. The decoder's role predict this masked content. For sequence- modalities, the input to the decoder comprises modality, positional, and content information. The decoder is tasked to predict the next token in the sequence. To ensure that each token is only influenced by preceding tokens (and not by any future tokens), we apply a causal mask to the self-attention, as is standard in autoregressive models. Since all target tasks consist of discrete tokens, we can use the cross-entropy loss for all of them, which we found removes the need for task-specifie loss balancing and improves training stability. Further details on the architecture are provided in Appendix C.
2.3 Multimodal masking strategy
For multimodal pre-training, use pre-training strategy similar to MultiMAE [5], in that we sample and encode small set of visible tokens/patches from all modalities, and train the model to perform cross cross-modal predictive coding.
Input & target masking. Dropping masked-out tokens and only encoding the small set of visible ones when performing masked image modeling has been shown yield significant increases in training efficiency [48], and is crucial when training on multiple modalities [5]. The imbalance of the usually low number of input tokens and the much higher number of target tokens
induce significant computational costs the decoder, even if they are small. We propose to use largel masking, meaning that we do not decode all masked-out tokens, but only randomly sampled subset. By fixing the number of randomly sampled input and target tokens (see Figure 2), 4M enables pre-training many modalities while keeping training costs low. Similar to MultiMAE [5], we sample the number of input tokens per modality using symmetric Dirichlet distribution with concentration parameter We follow the same approach also sample the number target tokens per modality. After sampling the per-modaa modality number of input and target tokens, we sample tokens from dense modalities uniformly random, and perform span masking [86] sequence modalities. The experimental consequences of these design choices are studied in Section 5, and in more detail in Appendix E.4.
Transfer Experiments
To assess the effectiveness of 4M as pre-training strategy, we train two models: base version 4M-B with 86M encoder parameters and a large version 4M-L with 303M encoder parameters (for more details, see Appendix C). We then transfer these trained models to several common downstream tasks and compare their performance against relevant basclines. To better control for the dataset, augmenta-
5

---

Table 1: Transfer learning study: We transfer 4M models to semantic and geometric downstream tasks and compare several baselines, For transfers ImageNet-IK [29, 96], we first perform intermediate fine-tuning on ImageNet-21K [29, 93]. 4M outperforms the baselines on all tasks except for ImageNet-IKK IK, surpassed by DeiT IIII which is specialized model. In contrast to 4M, all of the baselines employed data augmentations to achieve their results. Best results per category are bolded.
Pre-training
Data
Extra labels
ImageNet-Ile IK Top-1 1 acc.
COCO Apmask
ADE20K mloU 1
NYU depth ō acc. 1
Method
data
aug.
Apha
MAE [48]
IN-IK
X
84.2
48.3
41.6
46.1
89.1
DeiT III [108]
IN-21K
V
X
85,4
46.1
38.5
49.0
87.4
MultMME B [5]
IN-IK
V
V
84,0
44.1
37.8
46.2
89,0
4M-B (RGB- RGB only)
CC12M
X
X
82.8
42.3
36.6
38.3
80.4
4M-B (RGB CLIP only)
CC12M
X
V
83.4
46.6
39.9
43.0
85.7
4M-B
CC12M
X
V
84.5
49.7
42.7
50.1
92.0
MAE [48]
IN-IK
V
X
86.8
52.8
45.3
51.8
93.6
DeiT III [108]
IN-21K
V
X
87.0
48.7
41.1
52.0
89.6
4M-L
CC12M
X
86.6
53.7
46.4
53.4
94.4
tions, model architecture, and compute, which can significantly affect downstream performance, additionally show self-baseling that are conceptually similar to MAF [48] (Masked RGB -> RGB) and BE [82] (Masked RGB + CLIP).
The transfer tasks include ImageNet- K classification [29, 96], COCO detection and instance seg- mentation [67]. ADE20K semantic segmentation [131], and NYUv2 depth estimation [102]. While some transfer tasks have similarities to our pseudo labeled tasks, they are different instantiations (e.g.. ADE20K instead of COCO semantic classes, absolute depth instead of relative depth).
To make 4M models comparable to other ViT backbones, we train all methods by attaching transfer task-specific heads (e.g. Cascade Mask R-CNN [15, 47J) to the encoder, and discard any decoders. Note that For comparability, we perform the transfers of all 4M and baseline models in the same controlled manner, by closely following commonly used settings from other papers. Exact training details are provided in Appendix D.
also choose to keep the decoder for transfer learning, which explore in Section 5,
Results in Table show that 4M transfers exceptionally well to all downstream tasks, outperforming the baselines on detection, segmentation and depth estimation. While on ImageNet- K, 4M is outperformed by more specialized models such as DeiT III, the results demonstrate 4M to be versatile vision model that can strongly benefit from being pre-trained on multiple (pseudo labeled) tasks. We note that preliminary experiments with an even larger 4M XL model with 2.7B parameters showed overfitting to the pseudo labeled CC12M dataset, resulting limited additional improvement on downstream tasks. While a larger dataset or adding data augmentations are therefore necessary to fully benefit from larger 4M models, we still observe significant improvements in generation quality and use the 4M-XL model in the next section.
Generative Capabilities & Probing the Learned Representation
4M can directly be used for generation of all pre-training modalities, through iteratively decoding tokens [17, 65, 18], as illustrated in Figure 3. In addition, we enable several other generative capabilities by utilizing two key aspects of 4M: The first is the fact that 4M is crucially able to generate any of the training modalities, either unconditionally or conditioned on any other set of modalities (see Figure 4 top). The second is the fact that 4M is trained using masking, which enables (conditional) in-painting and out-painting (see Figs. and 4). Combining these key aspects enables several multimodal editing tasks, such as semantic editing, geometrically grounded generation, or guiding the generation with multiple strong and weak conditions (via weighting).
To improve image generation fidelity, trained 4M-XL version and all subsequent images were generated with it. While 4M can be directly used for generation, there are certain common improve- ments we perform to improve image fidelity in the results shown. These include specializing 4M-L into super-resolution variant that maps tokens of low-resolution generations to a higher resolu- tion [18, 123], and specializing 4M models by fine-tuning them to be more aligned with specific generative use-cases (c.g. for text-to-iman image, or in in-painting) [95]. See Appendix A for more details on these specializations.
Probing learned representations through generation. In addition to performing transfers, we can get a glimpse into what kind of (predictive) representations 4M learned by manipulating one part of

---

Any-lo-any generation
Grounded generation of image variations
CUP
Segrariide
Soending Feers Gople1
Segreridtnn Cattion
nachel chary
teusting
A chain detailed multimodil edits allows the to better generate what had mind
Capiion Bounding
Segmenbation
Segmeriation
RGB
Gagtion Masked RGE Segimrrationn
Mesked RGB RGB
Castis Frgut
Capson FREE Iepigsand
ongrentation
Weighting of different medalities (cclassifier-free guidance") for added steering power to guide edits
Dirron
Roiiiigg 11a
JAR C1 Righttast
Kheely
strorgy
stoongly on baption bounding bor
on caption
masnod RGB
Figure 4: Multimodal generation and editing. 4M 's in-painting and any-to-any prediction abilities unlock suite of multimodal generation and editing capabilities, which allow for fine-grane fine-grained creative control. We show several key capabilities such grounding the generation in predicted geometry, performing semantic edits, and being able control how much certain input modalities influence the generation via weighting.
4M implicitly infers object context
can perform semantic manipulations
Figure 5: Probing the learned representation through manipulation: (Left): All of the condition- ing bounding boxes are fixed except the marked "potted plant" ones that are changing. Depending on where the "potted plant" bounding boxes are placed, 4M infers how they can be part of the scene (e.g.. as painting or real plant) in a geometrically and physically plausible manner. (Right): Changing a single semantic class accordingly affects how 4M predicts the overall image. See more examples in Appendix A.4 4 and interactive visualizations our website.
the input and keeping the remainder fixed [5]. Figure shows series manipulations, in which 4M shows intriguing capabilities predicting geometrically and physically plausible arrangements while taking into account the semantic context the scene.
Multimodal editing. By combining the multimodal conditional generation and in in-painting capabili- ties of 4M, we can perform various multimodal editing tasks, such performing semantic edits or in- painting grounded by geometric conditioning (see Figure top and middle). Drawing parallels to ControlNet [130], these allow for steering the generation using more than just text. However, 4M is able to perform these tasks with just single network and condition multiple (partial) modalities individually or simudtaneously. The conditions can either be hand-specified, or extracted from an image using 4M itself, thereby removing the need for specialist models to create the conditions.
Multimodal weighted guidance. Classifier-free free guidance [50] has been shown improve image fidelity in token-based generative models [40, 123, 18]. Inspired by Liu et al. [68] that perform compositional generation on multiple text conditions. we can guide our generation by weighting different (parts of) modalities by different continuous amounts even negatively. This unlocks further

---

multimodal editing capabilities (see Figure 4 bottom), such as being able to weakly condition on certain modalities, or using negative weighing avoid a certain concept in the generation. Multimodal guidance can be achieved by computing a weighted sum of the logits of an unconditional and each conditional case: logits guided = logot + Ei= wi (logits,ood. logits
We provide additional visualizations covering 4M's wide range of generative capabilities our website and in Appendix A.4.
Ablations
Pre-training on large set of modalities using a masking objective creates a large design space that raise questions such as: what modalities should we pre-train on, what is the optimal masking ratio, or how do we select the number of tokens to mask from each modality? By performing a thorough ablation of these design parameters, we aim to find out which ones matter the most for multimodal pre-training and which ones do not. We used these findings to decide on the settings for the models shown in Section 3 and 4.
We first choose a reference setting and measure the deviation of the model performance as result of the one aspect under study while keeping the rest fixed. Performance is measured through transferring the models to a large set of downstream tasks and measuring the validation set performance. The aim of the benchmarks is to measure how good certain instantiation of 4M is at transferring both to new targel tasks, but also to unseen inpul modalities. For that, we include tasks that use RGB (pixels) as inputs and transfer to a new target, such as COCO [67] object detection, ADE20K [131] semantic segmentation and ten transfers from RGB to dense tasks in Taskonomy [126] and Hypersim [94] (e.g. depth, curvature, segmentation). Furthermore, we include eleven single-modal and twelve multimodal transfers from some Taskonomy or Hypersim modalities to others (e.g. curvature + occlusion edges, or RGB depth- - segmentation).
For simplicity, and similar to pre-training, we model each of the benchmark tasks as predicting one set of tokens from another. This is achieved by training tokenizers on these modalities and tasks in the same way comparability across modalities and tasks, we measure the validation set cross-entropy performance rather than lask-specific metrics. Note that in the interest of space, we aggregate the results of the Taskonomy and Hypersim tasks, and denote them RGB +x, Y and X + See Appendix E for further training details.
did for pre- pre-training. All transfers
performed at 224 x 224 resolution. For
5.1 Reference model
Following common practice for the reference model size [30, 86], our model consists of 12 encoder and 12 decoder layers. We train it on CC12M [19] using all modalities as both inputs and targets, at resolution 224 x 224 pixels (corresponding to 14 x 14 tokens per dense modality), ), and using no augmentations such cropping nor color augmentations. The total training length is fixed at 100B tokens, corresponding to roughly 400M masked samples. We set the number of randomly sampled input and target tokens to 12 each, and sample each using a symmetric Dirichlet distribution with parameter a 0.2. To determine the significance of various modeling choices, adopt the approach used by Raffel et al. [86] that calculates the standard deviation of the transfer results of ten independently trained reference models.
5.2 Input modalities and target tasks
Importance of target tasks for representation learning. 4M is both multimodal and a multitask training scheme and the choice of target task(s) is a powerful way of steering what representation the model learns [10, 109, 42, 107, 5]. To ablate this for 4M, we fix the input modalities to be either RGB or all the modalities (denoted as ")l and vary the target tasks. The results in Table 2 mirror the findings of Sax et al. [98]. that the optimal choice of pre-training setting depends highly on the type of transfer that is performed, and that there is single pre- mre-trainin task that performs best all transfers. However, pre- mre-rraniin on all target modalities consistently outperforms the other single-task and multitask alternatives in terms of average loss, no matter what input modalities were used during pre-training. This makes it the preferred configuration for generalist models, especially when their future applications are unknown or varied.
Importance of multimodal pre-training for transferring to new input modalities. Table 2 shows that multimodal pre-training can significantly help with transferring to new input modalities (X-+Y Y and X+Y - transfers), but comes at a performance loss at transfers that use RGB as the sole input modality. In Appendix F.4, we explore pre-training using mixtures of different masking strategies, which enables us to train models that perform well both regimes.
8

---

Table 2: Pre-training imput and target modalities ablation: The choice of pre-training tasks and modalities influences what representations the model learns, and how well can be transferred to novel tasks and modalities. Here, Geometric = RGB Depth Normals and Semantic = RGB + Segmentation CLIP + Detection Captions. We show the average losses (4) for several task categories and compute the average rank and best losses for "RGB" and "All" inputs separately. The reference model setting is indicated by & and results that lie within two reference model standard deviations the best result are bolded. Performing
pre-training on all input and target modalities is the most versatile choice, the optimal set of pre-training modalities for any given downstream task unknown.
Pre-training
Pre-training targets
COCO Det.
ADE20K Seg.
RGB-X
X-Y
X+Y-Z
Avg. Loss
Avg. Rank
imputs
RGB
RGB
3.14
6.21
5.03
6.94
6.15
5.49
8.00
RGB
Depth
3.11
6,06
4.72
6.89
5,84
5.32
4.00
RGB
Normals
3.12
6.02
4.66
6,83
5.87
5.30
3.20
RGB
Segmentation
3.17
5.94
4.84
6.86
5.89
5.34
4.60
RGB
CLIP
3.07
6.11
4.83
6,85
5.94
5.36
4.80
RGB
Detection
2.78
6.11
5.03
7.07
6.24
5.45
7.20
RGB
Captions
3,45
5.92
7.35
6.86
6.03
10.00
RGB
Geometrie
3.11
6.08
4.70
6.88
5.85
5.32
3.80
RGB
Semantic
2.88
5.99
4.86
6.97
6.06
5.35
5.40
RGB
All
2.90
5.99
4.74
6.91
5.93
5.29
4,00
All
RGB
3,21
6.20
5,07
6.75
5.85
5.42
3.80
All
CLIP
3.19
6.18
5.06
6.80
5.88
5.42
3.80
All
Geometric
3,20
4,98
5,36
1.80
All
Semantic
3,05
6.13
5.16
6.77
5.87
5.39
3.40
All
All
3.06
6.11
3,07
6.75
5.80
5.36
2.20
6.0
Average Loss
5.0
0.1
0.2 0.5
1.0
14 174
255
512
64
17A
756 Tokens
512
1/54
1716
1/4
50
100
200
400
406-11
406-5
481-8
48-4
Input Taraggt
Input Tokens
Target
Dataset
Fraction
Train Tokens
Model Size
Figure 6: Ablations results: We ablate several key design choices of the multimodal masking objective (in blue), and study how well 4M scales (in green). We show the overall average losses (L) and highlight the reference model setting in blue / green. A detailed breakdown of the task losses is provided in Appendix E.4 and Appendix E.5.
5.3 Multimodal masking strategy
Multimodal masking is at the core of 4M, in this section, we ablate how modality tokens should be sampled, and how many tokens should encode and decode.
Modality proportions masking. We ablate various choices of Dirichlet parameter a, both for the input and target sampling. If a is low, the sampling procedure will often select cases where most of the tokens are sampled from only one modality. If high, however, most samples will contain tokens from all modalities to equal proportions. Results in Figure 6 (a) show that uniformly sampling over the simplex performs best average, but not by a large margin.
Input masking budget. The difficulty of the multimodal masked modeling task is largely determined by the number visible (non- mon-masked) input tokens. Encoding only the small set of visible tokens can significantly improve training efficiency [48]. Figure (b) shows that with a fixed training token budget, training with 128-256 input tokens performs well.
Target masking budget. We can decide to decode merely a small random subset of all remaining masked-ou out tokens, which is especially important for enabling efficient multimodal training with large decoders. As Figure (c) shows, decoding only a small random subset of all targets performs well (for a fixed number of total training tokens), while also reducing computational costs.
5.4 How well does 4M scale?
Scalability is key property that models and training objectives should have. We therefore ablate the following three axes: To ablate the dataset size, train 4M models various subsets of CC1 2MM, down to 1/64th of the full dataset. To ablate the training length, we vary the total number of tokens seen. We define tokens seen the total number of both input tokens and target tokens the model was

---

trained on. To ablate the model size, we train different sizes of 4M models, ranging from Tiny (4M-Ti) with 24M parameters to Large variants (4M-L) with 705M parameters. Exact model specifications are given in Appendix C.I. Figure (d), (e), (f) show that 4M scales with dataset size, training length, and model size, respectively.
For additional ablations on the architectural and training design choices of 4M, see Appendix E.
Related Work
Large language models have been demonstrated to be capable of performing a diverse range of tasks out of the box [86, 12, 81, 25, 45] by training on large datasets with simple objectives [30, 84, 86, 106]. In vision, however, many scaling efforts have instead focused on training specialized models on a single task and modality, such as predicting masked RGB pixels [20, 31, 4, 48, 118, 34], discrete tokens [7, 132], other (deep) features (117, 6, 82, 1 11, 36, 70] from RGB inputs. Training models instead on multiple tasks [16, 33, 63, 42, 92, 11] and modalities [85, 122, 76, 59, 133, 1, 3, 57, 43),
or both [54, 103, 114, 5, 44] usually requires modality-speciin modeling choices, making it difficult to extend these methods.
While some recent works aim to consolidate various modalities and tasks by representing them as images [77, 8, 115], these approaches have limitations when dealing with modalities that cannot be readily converted into images, such as text or neural network feature maps. Instead, 4M adopts the approach of Pix2Seq [21, 22] and Unified-OO [74] which addresses these issues by unifying the representation space on which models are trained through tokenization [110, 35, 30]. However, unlike methods like In which operate on single RGB image tokenizer, 4M's ability to work tokenizers enables scaling to visual modalities beyond those that can
with multiple modality-specii be represented as images, such as neural network feature maps. 4M also builds upon the multimodal masking approach of MultiMAE [5] and extends it beyond image-like modalities.
Both oaken-based generative models [88, 123, 17, 18, 65] and diffusion models [89, 79, 95, 97] have been mostly limited text-to-image generation. While there works that enable a greater amount of control by conditioning on additional modalities, they are either very limited in the number of ways they can be conditioned on [58, 40, 113, 119, 62, 9, 128], or require training separate model for each new modality [130]. 4M flexibly allows for conditioning on any subset of the training modalities, and can, likewise, generate all these modalities, unlocking powerful generative editing capabilities.
7 Conclusion and Limitations
4M is generalist framework for training multimodal and multitask models that not only perform many key vision tasks out of the box, but also demonstrate strong transfer results to a wide range of downstream tasks. 4M's in-painting and any-to-any generation capabilities enable it to perform wide range of multimodal generative and expressive editing tasks all using a single model. In the following, we discuss some limitations of our approach and potential future work.
Additional modalities. While 4M already includes number of modalities, from semantics-bases and geometry- based to text-based, bringing additional modalities could vastly improve its usability. For example, training on features extracted from a large language model has been shown to significantly boost text-to-imaan to-image generation capabilities [97, 123, 18]. Introducing modalities like edges, sketches, or human poses has the potential to greatly improve the expressiveness [130] of 4M, but it may also be expanded to videos or multi-view imagery to unlock capabilities. We anticipate 4M to conveniently extend such new modalities.
generation and editing
Tokenizer quality. 4M can benefit from better tokenizers. both in terms of generation and transfer results, but there are limits to the amount of information that can be encoded in tokenized patches. Operating on tokens that cover a smaller image region, or operating on higher-resolution images may improve image quality, but is expected to be computationally more expensive. Generally, improvements along this direction are expected to directly boost the performance of 4M.
Dataset size and quality. While our binding dataset choice of CC12M is standard, 4M can benefit from training on significantly larger datasets [99, 14, 39]. Web-scraped image-text datasets contain many low-quality images, as well as captions that are not related to the image content. Fine-tuning 4M on more curated dataset like LAION-Aestheties V2 [99], or tuning the model using reinforcement learning [83] could significantly improve generation quality and diversity. We leave this for the future.
10

---

Acknowledgements. We thank Hanlin Goh and Elmira Amirloo Abolfathi for their valuable feedback on earlier versions of this manuscript.
References
[1] Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, and Luke Zettlemoyer. CM3: A causal masked multimodal model of the internet. arXiv:2201.0733
2022. 10
[2] Armen Aghajanyan, L. Yu, Alexis Conncau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Scaling laws for generative mixed- modal language models. In International Conference on Machine Learning, 2023.
[3] Jean- San-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Tain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katic Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei. Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh. Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Systems, 2022. 10
[4] Sara Atito. Muhammad Awais, and Josef Kittler. arXiv:2104:03602 2021. 10
SIT: Sell-supervised vision transformer.
[5] Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. MultiMAE: Multi-modal multi-task masked autoencoders. In European Conference Computer Vision, 2022. 2, 3, 5, 6, 7, 10, 32, 35, 38, 39,
[6] Alexci Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vee: A general framework for self-supervised learning in speech, vision and language. International Conference on Machine Learning, 2022. 10
[7] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BETT: BERT pre- pee-training of image transformers. In International Conference on Learning Representations, 2022. 10, 40
[8] Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image inpainting. In Advances Neural Information Processing Systems, 2022. 10
[9] Omer Bar-Tal, Lior Yariy, Yaron Lipman, and Tali Dekel. MultiDiffusion: Fusing diffusson paths for controlled image generation. In International Conference on Machine Learning, 2021. 10
[10] Jonathan Baxter. A model of inductive bias learning. Journal of Artificial Intelligence Research, 2000. 8,
38
[11] Deblina Bhattacharjee, Tong Zhang, Sabine Süsstrunk, and Mathieu Salzmann. MulT: An end-to-end multitask Icarning transformer. In Conference Computer Vision and Pattern Recognition, 2022. 10
[12] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, T. Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Matcusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv:2005. 14165, 2020. 1, 10
[13] Neil Burgess, Jelena Milanovic, Nigel Stephens, Konstantinos Monachopoulos, and David Mansell. Bfloat16 processing for neural networks. In Symposium on Computer Arithmetic, 2019. 32, 33
[14] Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Back, and Saehoon Kim. COYO-700M: Image-text pair dataset. https:// tttps://githube /ithhb......... com http///aaabbrrat//oooodaaaaaaaaa 10
2022.
[15] Zhaowei Cai and Nuno Vasconcelos. Cascade R-CNN: Delving into high quality object detection. In Conference on Computer Vision and Pattern Recognition, 2018. 6, 34
[16] Rich Caruana. Multitask learning. Machine Learning, 1997. 10
[17] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. MaskGIT: Masked generative image transformer. In Conference Computer Vision and Pattern Recognition, 2022. 2, 5, 6, 10. 21
11

---

[18] Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, José Lezama. Lu Jiang, Ming Yang, Kevin P. Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text-to- image generation via masked generative transformers. In International Conference on Machine Learning, 2023. 2, 6, 7, 10, 20, 21,45
[19] Soravit Changpinyo, Piyush Kumar Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-texi pre-training to recognize long-tail visual concepts. In Conference on Computer Vision and Pattern Recognition, 2021. 2, 4, 8, 30, 37, 46
[20] Mark Chen, Alec Radford, Rewon Child. Jeffrey Wu, Heewoo Jun. David Luan, and llya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, 2020. 10
[21] Ting Chen, Saurabh Saxena, Lala Li, David Fleet, and Geoffrey Hinton. Pix2seq: A language modeling framework for object detection. International Conference on Learning Representations, 2022. 2, 4, 10,
30, 36
[22] Ting Chen, Saurabh Saxena, Lala Li, Tsung-Yi Lin, David Fleet, and Geoffrey F. Hinton, A unified sequence interface for vision tasks. In Advances in Neural Information Processing Systems, 2022. 2, 10, 30
[23] Weifeng Chen, Shengyi Qian, David Fan, Noriyuki Kojima, Max Hamilton, and Jia Deng. OASIS: A large-scale dataset for single image 3d in the wild. In Conference on Computer Vision and Pattern Recognition, 2020. 44
[24] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Conference on Computer Vision and Pattern Recognition, 2022. 30
[25] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sehastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran. Emily Reif. Nan Du, Benton C. Hutchinson, Reiner Pope. James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengeheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Gareía, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Ömernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat. Aitor Lewkowyez. Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen Mcier-Hellsterm, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: Scaling language modeling with pathways. wrXiv:2204.023111 2022. 1, 10, 42
[26] Kevin Clark, Minh-Thang Luong, Quoe V. Lee, and Christopher D. Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Repre- sentations, 2020. 34, 35
[27] Ekin Dogus Cubuk, Barret Zoph, Jon Shiens, and Quoc Le. Randaugment: Practical automated data augmentation with a reduced search space. In Advances in Neural Information Processing Systems, 2020. 34
[28] Mostafa Dehghani. Josip Djolonga, Basil Mustafa. Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos Ibrahim Alahdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Amab, Xiao Wang. Carlos Riquelme Ruiz, Matthias Minderer, Joan Puigcerver. Utku Evei, Manoj Kumar, Sjoerd Van Steenkiste, Gamaleldin Fathy Elsayed, Aravindh Mahendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Collier, Alexey A. Gritsenko, Vighnesh Birodkar, Cristina Nader Vasconcelos, Tay, Thomas Mensink, Alexander Kolesnikov, Filip Pavetic, Dustin Tran, Thomas Kipf, Mario Lucic, Xiaohua Zhai, Daniel Keysers, Jeremiah Harmsen, and Neil Houlsby. Scaling vision transformers to billion parameters. International Conference on Machine Learning, 2023. 42
[29] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. ImageNet: large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition. 2009, 6.
[30] Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding, In North American Chapter of the Association for Computational Linguistics, 2019. 2. 3, 4, 8, 10, 30. 37,40
[31] Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer. Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. 4, 10, 31, 33
12

---

[32] Ainaz Efiekhar, Alexander Sax. Roman Bachmann, Jitendra Malik, and Amir Roshan Zamir. Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans. In International Conference Computer Vision, 2021. 4, 30
[33] David Eigen and Rob Fergus. Predicting depth, surface normals and semantic labels with common multi-scale convolutional architecture. In International Conference on Computer Vision, 2015. 10
[34] Alaacldin El-Nouby. Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave. Are large-scale datasets necessary for self-supervised pre-training? 12. 2021. 10
[35] Patrick Esser, Robin Rombach, and Bjöm Ommer. Taming transformers for high-resolution image synthesis. In Conference on Computer Vision and Pattern Recognition, 2021. 4, 10, 31
[36] Yuxin Fang, Quan Sun, Xinggang Wang. Tiejun Huang, Xinlong Wang, and Yue Cao. EVA-02: A visual representation for neon genesis. arXiv:2303.11331, 2023. 10, 30
[37] Yuxin Fang, Wen Wang, Binhui Xie, Quan-Sen Sun, Ledell Yu Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. EVA: Exploring the limits of masked visual representation learning at scale. Conference on Computer Vision and Pattern Recognition, 2023. 30
[38] Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, and Kaiming He. Masked autoencoders as spatiotempo- ral leamerr. In Advances in Neural Information Processing Systems, 2022. 33, 38, 43
[39] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang. Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mchdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander J. Ratner, Shuran Song, Hannanch Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alexandros G. Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. DataComp: In search of the next generation of multimodal datasets. arXiv:2304. 14108, 2023. 4, 10
[40] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-A- Scene: Scene-based text-to-image generation with human priors. In Europea Conference Computer Vision, 2022. 7, 10, 21,30
[41] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung- Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph. Simple copy-pasts is strong data augmentation method for instance segmentation. In Conference on Computer Vision and Patter Recognition, 2021. 34, 35,
[42] Golnaz Ghiasi, Barret Zoph, Ekin Dogus Cubuk, Quae V. Le, and Tsung-Yr Yi Lin. Multi-task self-training for learning general representations. International Conference on Computer Vision, 2021. 3, 4,4, 8, 10, 38
[43] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. ImageBind: One embedding space to bind them all. In Conference on Computer Vision and Pattern Recognition, 2023. 10
[44] Rohit Girdhar, Alaacldin El-Nouby, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. OmaiMAE: Single model masked pretraining on images and videos. In Conference on Computer Vision and Pattern Recognition, 2023. 2, 10
[45] Google, PaLM 2 technical report. https: //ai: google/ /ttp:///////a////at///a///a/aeee
pdf, 2023. 10
[46] Priya Goyal, Piotr Dollár, Ross B. Girshick. Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming Hc. Accurate, large minibatch SGD: Training ImageNet in hour. arXiv: 100:v:1706:0667 2017. 31, 33, 34, 36, 37, 38, 43
[47] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask r-cnn. Transactions on Patiern Analysis and Machine Intelligence, 2017. 6, 34
[48] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll'ar ar, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In Conference Computer Vision and Pattern Recognition, 2022. 2, 3, 5, 6, 9, 10, 30, 32, 38, 40, 43
[49] Dan Hendrycks and Kevin Gimpel. Gaussian error lincar units (GELUs), 16X.v:1606.08415, 2016. 33, 38, 42
[50] Jonathan Ho and Tim Salimans. Classifier-fre free diffusion guidance. arXiv:2207.12550 2022. 7.21
13

---

Multimodal multitask learnng with a unified transformer. In [51] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advunces in Neural Information Processing Systems, 2020. 32
[52] Jonathan Ho, Chitwan Saharia, William Chan, David J. Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. The Journal of Machine Learning Research, 2022. 20
[53] Emiel Hoogeboom, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. In International Conference on Learning Representations, 2022. 21
[54] Ronghang and Amanpreet Singh. UniT: International Conference on Computer Vision, 2021. 10
[55] Gao Huang. Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European Conference on Computer Vision, 2016. 34, 35
[56] Po- Yao Huang, Hu Xu, Juncheng Billy Li, Alexci Baevski, Michael Auli, Wojeiech Galuba, Florian Metze, and Christoph Feichtenhofer. Masked autoencoders that listen. In Advances in Neural Information Processing Systems, 2022. 3
[57] Shaohan Huang, Li Dong, Wenhui Wang. Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, and Furu Wei. Language is not all you need: Aligning perception with language models. arXiv:2302. 0e23302.144443 14045, 2023. 10
[58] Xun Huang, Arun Mallya, Ting-Chun Wang, and Ming- Yu Liu, Multimodal conditional image synthesis with product-of experrs gans. In European Conference on Computer Vision, 2022. 10
[59] Andrew Jacgle, Schastian Borgeaud, Jean- Sann-Bapistc Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier Henaff, Matthew Botvinick, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver IO: A general architecture for structured inputs & outputs. In International Conference on Learning Representations, 2022. 10
[60] Jared Kaplan, Sam McCandlish, T. J. Henighan. Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models. arXiv:2001.08331 08361, 2020. 3
[61] Oğuzhan Fatih Kar, Teresa Yeo, Andrei Atanov, and Amir Zamir 3d common corruptions and data augmentation. In Conference Computer Vision and Pattern Recognition, 2022. 30
[62] Yuval Kirstain, Omer Levy, and Adam Polyak. X&Fuse: Fusing visual information in text-to-image generation. 23X002230..01000, 2023. 10
[63] Iasonas Kokkinos. Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In Conference on Computer Vision and Pattern Recognition, 2017. 10
[64] Alexander Kolesnikov, André Susano Pinto, Lucas Beyer, Xiaohua Zhai, Jeremiah Harmsen, and Neil Houlsby, UViM: A unified modeling approach for vision with learned guiding codes. In Advances in Neural Information Processing Systems, 2022. 2
[65] Tianhong Li. Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. MAGE: Masked generative encoder to unify representation learning and image synthesis. In Conference on Computer Vision and Pattern Recognition, 2023. 6, 10, 21
[66] Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In European Conference on Computer Vision, 2022. 30, 33, 34
[67] Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision, 2014. 6, 8, 30, 36
[68] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B. Tenenbaum. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision, 2022. 7, 21
[69] Shikun Liu, Linxi (Jim) Fan, Edward Johns, Zhiding Yu, Chaowei Xiao, and Anima Anandkumar. Prismer: vision-language model with ensemble of experts. NXXv:23003002006 2023.
[70] Xinghin Liu, Jinghao Zhou, Tao Kong, Xianming Lin, and Rongrong Ji. Exploring target representations for masked autoencoders. wxXi::2209003117 2022. 10
14

---

[71] Ze Liu, Yuiong Lin. Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In International Conference on Computer Vision, 2021. 30
[72] Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer. Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Conference Computer Vision and Pattern Recognition, 2022. 35
[73] llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2019. 31, 33, 34, 35, 36, 37, 38
[74] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-100 IO: A unified model for vision, language, and multi-modal tasks, In International Conference on Learning Representations, 2023. 2, 4, 10
[75] Troy Luhman and Eric Luhman. 04X1012207.04366, 2022. 32
Improving diffusion model efficiency through patching.
[76] Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun, Attention bottlenecks for multimodal fusion. In Advances in Neural Information Processing Systems, 2021. 10
[77] Charlic Nash, Joao Carreira, Jacob Walker, lain Barr, Andrew Jaegle, Matcusz Malinowski, and Peter Battaglia. Transframer: Arbitrary frame prediction with generative models. arXiv:2203.0044000 2022. 10
[78] NegPrompt. Negative prompt. https: thub. vikkiNegative-eeee pttp://eeaaiveeerpp..... 2022. 21
[79] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, llya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning, 2021. 10
[80] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In
International Conference on Machine Learning, 2021. 31
[81] OpenAI. GPT-4 technical report. arXiv:2303. 087.040.877,4000 2023. 10
[82]
Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. BEiT v2: Masked image modeling with veetor-quantized visual tokenizers. 225v::2208.063666 2022. 3, 6, 10, 30, 31, 38, 43
[83] André Susano Pinto, Alexander Kolesnikov, Yuge Shi, Lucas Beyer, and Xiaohua Zhai. Tuning computer vision models with task rewards. In International Conference on Machine Learning, 2023. 10
[84] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. 1, 10
[85] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021. 3, 10, 30
[86] Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 2020. 1, 5,5, 8, 10, 32, 33, 36, 37, 42
[87] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. ZeRO: Memory optimizations toward training trillion parameter models. In International Conference for High Performance Computing, Networking, Storage and Analysis, 2020. 33
[88] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, 2021. 10 31
[89] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with CLIP latents. arXiv:2204.06125, 2022. 10
[90] René Ranftl, Katrin I Lasinger. David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. Transactions on Pattern Analysis and Machine Intelligence, 2020. 44
[91] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun, Vision transformers for dense prediction. In International Conference on Computer Vision, 2021. 30, 35
15

---

[92] Scott Reed, Konrad Zolna, Emilio Parisotto. Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-maron, Mai Giménez, Yury Suisky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. Transactions on Machine Learning Research, 2022. 10
[93] T. Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manon ImageNet-21K pretraining for the masses, arXiv:2104. 10972,, 2021. 6, 34
[94] Mike Roberts and Nathan Paczan. Hypersim: photorealistic synthetic datasct for holistic indoor scene understanding. In International Conference on Computer Vision, 2020. 4, 8, 36
[95] Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Conference Computer Vision and Pattern Recognition, 2022. 6, 10,45 45
[96] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjecv Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 2014. 6
[97] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li. Jay Whang. Emily Denton, Seyed Kam- yar Scyed Ghasemipour, Raphael Gontijo-Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems, 2022. 10
[98] Alexander Sax, Bradley Emi, Amir Roshan Zamir, Leonidas J. Guibas, Silvio Savarese, and Jitendra Malik. Mid-level visual representations improve generalization and sample efficiency for learning visuomotor policies. arXiv:1812.11971 2018. 3, 8, 38
[99] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5B: An open scale dataset for training next generation image-text models. In Advances in Neural Information Processing Systems, 2022. 4, 10
[100] Noam Shazeer. GLU variants improve transformer. arXiv:2002.05202, 2020. 33, 42
[101] Jie Shi, Chenfei Wu, Jian Liang. Xiang Liu, and Nan Duan. DiVAE: Photorealistic images synthesis with denoising diffusion decoder. arXiv:2206, 2206002000 2022. 31
[102] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from RGBD images. In European Conference on Computer Vision, 2012. 6
[103] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. FLAVA: A foundational language and vision alignment model. In Conference on Computer Vision and Pattern Recognition, 2022. 10
[104] Linda B. Smith and Michael Gasser. The development of embodied cognition: Six lessons from babies. Artificial Life, 2005.
[105] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations. 2021. 32
[106] Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang. Hyung Won Chung. Dara Bahri, Tal Schuster, Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. UL2: Unifying language learning paradigms. In International Conference on Learning Representations, 2023. 10
[107] Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B. Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: a good embedding is all you need? In European Conference on Computer Vision, 2020. 8, 38
[108] Hugo Touvron, Matthieu Cord, and Hervé Jégou. DeiT III: Revenge of the vit. In European Conference on Computer Vision, 2022. 6
[109] Nilesh Tripuraneni, Michael I. Jordan, and Chi Jin. On the theory of transfer learning: The importance of task diversity. In Advances in Neural Information Processing Systems, 2020. 8, 38
[110] Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Advances in Neural Information Processing Systems, 2017. 2, 4. 10,
[111] Igor Vasiljevic, Nick Kolkin, Shanyi Zhang, Ruotian Luo, Haochen Wang, Falcon Z Dai, Andrea F Daniele, Mohammadreza Mostajabi, Steven Basart, Matthew R Walter, et al. DIODE: A dense indoor and outdoor depth dataset. arXiv: : 19119998.000 00463, 2019. 44
16

---

[112] Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017. 4
[113] Andrey Voynov, Kfir Aberman, and Daniel Cohen-Or. Sketch-guided text-to-image diffusion models. In ACM SIGGRAPH Conference, 2023. 10
[114] Wenhui Wang. Hangbo Bao, Dong, Johan Bjorek, Zhillang Peng. Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojjt Som, and Furu Wei. Image as a foreign lan- guage: BEiT pretraining for all vision and vision-language tasks. arXiv:2208. 10442, 2022. 3, 10, 34
[115] Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. Conference on Computer Vision and Pattern Recognition, 2023, 10
[116] WebDataset. Webdataset.
2022. 43
[117] Chen Wei, Haoqi Fan, Saining Xie, Chaoxia Wu, Alan Loddon Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training, In Conference on Computer Vision and Puttern Recognition, 2022. 10
[118] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. SimMIM: A simple framework for masked image modeling. In Conference on Computer Vision and Pattern Recognition, 2022. 10
[119] Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. ReCo: Region-controlled text-to-image generation. In Conference on Computer Vision and Pattern Recognition, 2023. 10
[120] Zhilin Yang. Zihang Dai, Yiming Yang, Jaime G, Carbonell, Ruslan Salakhutdinov. and Quoc V. Le. XLNet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, 2019. 21
[121] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved VQGAN. In International Conference Learning Representations, 2022. 31
[122] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung. Mojtaba Seyedhosseini, and Yonghui Wu. CoCa: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research, 2022. 10
[123] Jiahui Yu, Yuanzhong Xu, Jing Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfci Yang, Bureu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. Transactions on Machine Learning Research, 2022. 6, 7, 10, 20, 21
[124] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. International Conference on Computer Vision, 2019. 34
[125] Amir R Zamir, Alexander Sax, Nikhil Cheerla, Rohan Suri, Zhangjie Cao, Jitendra Malik, and Leonidas J Guibas. Robust learning through cross boss-task consisteney. Conference on Computer Vision and Pottern Recognition, 2020. 30
[126] Amir Roshan Zamir, Alexander Sax, Bokui (William) Shen, Leonidas J. Guibas, Jitendra Malik. and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Conference Computer Vision and Pattern Recognition, 2018. 8, 30, 36
[127] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi, SoundStream: An end-to-end neural audio codec. Transactions on Audio, Speech, and Language Processing, 2022. 31, 32
[128] Fangneng Zhan, Yingchen Yu, Rongliang Wu, Jiahui Zhang, and Shijian Lu. Multimodal image synthesis and editing: A survey. arXiv:21 2021. 10
[129] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations. 2018. 34
[130] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In International Conference on Computer Vision, 2023. 7, 10. 24
17

---

[131] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ADE20K dataset. In Conference Computer Vision and Pattern Recognition, 2017. 6, 8, 36
[132] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. iBoT: Image BERT pre-training with online tokenizer. In International Conference on Learning Representations, 2022.
10
[133] Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Xiaogang Wang, Hongsheng Li, Xiaohua Wang, and Jifeng Dai. Uni-Perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. In Conference on Computer Vision and Pattern Recognition, 2022. 10
18

---

Appendix
A Generative Capabilities & Probing the Learned Representation
20
Token super-resolution
20
A.2 Generation-specffic specializations of 4M
20
A.3 Generation procedure details
21
A.4 Additional visualizations
22
B Multimodal Dataset & Tokenization
30
B.1 Pseudo labeled multimodal dataset details
iea.............................
30
B.2 Tokenization captions bounding boxes
................................
30
B.3 Tokenization dense modalities
30
Method & Training Details
32
C.1 Additional 4M architecture details
32
C.2 Training details
32
Transfer Experiments Details
33
D.I Architectural differences with the baselines
33
D.2 Image classification ImageNet-IK
34
D.3 Object detection and instance segmentation on COCO
................................
D.4 Semantic segmentation on ADE20K
35
D.5 Depth estimation on NYUv2
35
Ablation Details Results
36
E.1
Benchmark tasks
36
E.2 Reference model
37
................................
E.3 Input modalities and target tasks
38
E.4
Multimodal masking strategy
39
E.5
How well does 4M scale?
E.6
Architectural design choices
42
E.7
Training design choices
43
E.8
Self-baselines
43
E,9
Comparison the final models
Additional Evaluations
44
F.I Out of the box (zero-shot) performance
44
F2
Text-to-image performance
45
Broader Impact
46
G.I Computational costs
46
G.2 Social impact
46
19

---

A Generative Capabilities & Probing the Learned Representation
A.1 Token super-resolution
For generating high-resolution images, it is common to first generate images at a lower resolution, followed by super resolution step [52, 18, 1233. These two-stage approaches often large model to generate a low-resolution image that contains most of the semantically and geometrically important information, followed by one
smaller super-resolution models that fill in higher resolution details. Directly generating high-resolution images with Transformers is both computationally expensive due the long sequence lengths, but has also been shown to perform worse at generating semantically coherent images [18] compared to cascade approaches. 4M is trained at a base resolution of 224 x 224 pixels, which corresponds 14 x tokens. this low image resolution, fine details are difficult to model. We therefore train super-resolution model, 4M-SR, starting with 4M-L as the base and fine-tune
map from row-rssouution image tokens higher-resolution tokens. All images shown are generated at the base resolution using 4M-XL and subsequently up-sampled to double the resolution using 4M-SR.
Training details. For the high-resolution tokens, each 448 x 448 pixel image is represented by 28 x 28 tokens. The super resolution scchen very similar the base resolution 4M pre-training scheme, but the inputs to the model consist of a random subset of low-resolution and high-resolution tokens of all modalities, while the target is a random subset of all high-resolution tokens. At inference-time, one or multiple full low-resolution modalities (and sequence-like modalities) are used as conditioning to decode the high-resolution targets step-by-step.
To train 4M-SR, we start from the pre-trained 4M-L and continue training it for an additional 100B tokens (20% of the pre-training length). For each sample, we randomly sample the input budget uniformly between 64 and 1024 to better accommodate for the varying number of tokens throughout the generation process, and keep the target budget fixed 1024. As full captions seen during training, we train on a mixture of masking strategies where 1/4 of the input samples are heavily skewed towards fully unmasked captions (a 5.0 for captions with Pmask 0, and 0.05 for all other modalities).
otherwise rarely
RGB
Surface normals
284
224
224
448
448
Depth
Semantic Segmentation
224 224
824X 224
448 x:43
448 4
Figure 7: Token super-resolution: We specialize a version of 4M-L for performing super-resolution
in the space of tokens (14 + 28 tokens) to double the resolution (224² - 448" pixels) and in the process add more details the generated images. Just like the base model, it can generate any high-resolution modality conditioned on any low-resolution modality, but we use it to super-resolve modalities we generated the base resolution. Best viewed when zoomed
A.2 Generation-specific specializations of 4M
Adapting 4M to specific use-cases by fine-tuning it either on a subset of pre- pre-training modalities, or fine-tuning with a different token sampling distribution can improve performance on select downstream tasks. In similar manner, to optimize 4M model for text-to-imane generation, we can fine-tune by skewing the token sampling distribution include more captions as input. To train this caption-skewed model, we start from pre-trained 4M model and continue training it for an additional 50B tokens (10% the pre-training length). For each sample in a batch, we randomly sample the input budget uniformly between 64 and 256 to better accommodate for the varying number of tokens throughout the generation process, and keep the target budget fixed at 256. To skew the model towards full captions, train mixture of masking strategies where 1/3 of the input
20

---

samples are heavily skewed towards fully unmasked captions (a = 5.0 for captions with Pmask 0, and a 0.05 for all other modalities).
All text-to-image results in this section were generated using a caption-skewed 4M specialization.
A.3 Generation procedure details
Token-based masked image models that were trained with variable masking rates can be directly used for image generation [17, 65, 18]. These models can be seen as order-agnostic autoregressive models [53] in which generation is performed by iteratively decoding tokens in any order. Unlike traditional autoregressive models which need to generate tokens one-by-one in pre-determined order, these models are able to speed up inference by parallelizing the decoding process because the distribution over each masked token can be predicted at the same time.
Generation schedule. The generation process is guided by a generation schedule that outlines a sequence of generation steps. Each step in the sequence determines the target modality, the decoding scheme (described below), the number of tokens to decode, the temperature, and the top-k and top-p sampling values.
Decoding schemes. A crucial feature of our method is the flexibility incorporate different decoding schemes, such as for different target modalities. We describe these decoding schemes below:
MaskGIT [17] (for image-like modalities). This parallel decoding scheme generates the entire image in pre-determined number of steps (see Figure 3). At every prediction step, encode all visible tokens and decode all masked out tokens. We then sample from their predicted distributions and choose the most confident tokens, where n is determined by the generation schedule. Finally, we add the n predicted tokens to the input and repeat the previous steps.
Random order autoregressive (ROAR) (for image-like modalities). This scheme is concep- tually similar to the decoding scheme proposed in XLNet [120]. Unlike MaskGIT, we do not decode all masked-out tokens at every step, but instead randomly select tokens to decode.
Left-to-right autoregressive (for sequences). Besides images, we able generate sequence modalities such as captions and bounding boxes by autoregressively decoding them with the Transformer decoder of 4M (see Figure 3)-
Chained generation. Because any generated modality can be used as conditioning too, we can perform chained generation of several modalities, one after another, with each fully generated one being added to the conditioning of the next (see Figure 3). Performing generation in this chained manner results in each additional modality being generated in a consistent manner, as shown in Figures 9 and 10. In addition, we found that for certain generative tasks, such caption-to-RCe generating intermediate modalities such as CLIP tokens can further improve image fidelity (see Figure 9).
Classifier-free guidance. Classifier-free guidance is crucial for improving both image fidelity and how well the generation matches the conditioning. It is most commonly used in diffusion models [50], but can be applied in token-based models as well [40, 123, 18]. We perform classifier-fre guidance by computing weighted combinations of the logits of a forward pass with the conditioning and one without the conditioning:
logits guided
logits uncond + w (logits cond lngotsunnood)
Here, w is the guidance scale. When performing chained generation, we add each fully generated modality to the set of guided modalities.
Multimodal guidance. While guidance has been shown to significantly improve image quality, it can still happen that generative models ignore parts of the input, unpredictably focus some parts more than others, or generate undesired concepts. Negative prompting [78] is a popular way of keeping the model from generating undesired concepts. Liu et al. [68] show that performing compositional guidance on multiple conditions can further improve text-image similarity. In a similar way, can perform compositional generation by weighting different (parts of) modalities by different continous amounts even negatively. We can do this by computing a weighted sum of the logits of an unconditional case and the logits of each conditional case: wi (logitS - logits
logits guided = logits uncond
21

---

Here, Wi are the guidance scales for the different conditions. For example, this allows 4M to generate semantically or geometrically similar variants of images by weakly conditioning on their extracted segmentation, normal, or depth maps (see Figure 13). It can further be used for fine-grained steerability multimodal edits (see Figs. 16, 14, and 15), or to use negative weighting to avoid certain concepts from being generated (see Figure 17).
A.4 Additional visualizations
RGB-> -X. The model can solve several common RGB + X vision tasks out of the box, such as predicting normals, depth, semantic segmentation maps, or performing object detection and captioning. In Figure 8, we show examples of this functionality.
RGB
Surtece
segmenatton
Object
Ceptioning
narmal
segnentation
detecttón
PIN dive Captioning
STAIN NAIS
ANOUA
WETE CEE DOVE
Figure 8: RGB-XX 4M can perform several common vision tasks, such as predicting surface normals, depth, semantic segmentation maps, or performing object detection and captioning.
Chained generation. For certain generation tasks, like text-to-image observed that generating intermediate modalities, such as CLIP, before generating the RGB image can improve image fidelity. This leads form of progressive self-conditioning. which model particularly enables includes several modalities. We demonstrate this phenomenon in Figure 9.
Caption
CLIP
Depth
Normals Segmentation
None
Caption
Depth
Normals
Segmentation
None
rebbem OnE poen
Jorn STANE
Figure 9: Chained generation: First, we generate intermediate modalities from the caption (CLIP, depth, etc.). Then we generate the RGB images by conditioning on both the caption and the intermediate modality, i.e., caption intermediate modality RGB. We compare this to directly predicting the RGB image from the caption, i.c., caption - RGB, in the 'none' column. We observe that certain intermediate modalities like CLIP can improve image fidelity. As 4M includes multiple modalities, it particularly enables this form of progressive self-conditioning and engineering a sequence of modalities for better final generation.
22

---

Any-to-any generative modeling, 4M can be used to generate any modality from any other (and partial) subset of modalities, Figure 10 displays examples where all modalities are generated from single input, and Figure 11 shows several examples of generating different modalities using two full inputs. Subsequent figures will make use of this functionality and show masked predictions and examples with 1-3 different conditionings.
RGB
Caption
Bounding boxes
Surface normals
Depth
Semantic segimnntttin
CLIP
RGB
the
Caption
CONETTY COIS
Bounding
Sister der 2o the
Surface
the
Depth
Vitlage Sunse!
Sementic segnisntation
GENCCOMS TARS
phono 2
CLIP
arionsen CONG DIV
Figure 10: One-to-all generation: Starting from single input modality, we use 4M to generate the corresponding outputs for all others. Each row shows the outputs from different starting modality, all from the same original sample. Chained generation is used to ensure consistent outputs.
23

---

Begmoriorion RGB
Segnentatinn RBB
Nomrals
Sogmeetatton
Nomalls
-Begmerratoo
nover cerenneee
Aoarte of
IN reNeET
DAST AAT Wron
Any-to-any generation: 4M can perform any-to-any generation, meaning that any modality can generated by conditioning on any subset of modalities.
Figure 11:
Conditional in-painting. By virtue of being trained on a masking objective, can be used for conditional and unconditional in tn-painting (see Figure 12). In-painting coupled with 4M 's ability to predict any modality from any other and flexible multimodal conditioning is the basis for several multimodal editing capabilities (see Figure 4 middle and bottom).
soadan
Seeulhl camts
Sthayy
glaama
Image
Input image
C open D00 on
Dosten Reet sppy
Cankeinged bg
Capkotinnd
Input image
Input imaga
Figure 12: Conditional in- in-painting: 4M can be used for conditional in-painting by masking out selected image region and generating the remaining tokens using the masked image and a caption (or other modalities) conditions.
Grounded generation of image variations. 4M be used generate different versions of images by grounding the generation in some aspect extracted from a reference input image (see Figure 13). For example, we can take an image and generate variations that respect the geometry of the original image but may contain completely different colors by conditioning only extracted surface normals. This conceptually similar to ControlNet [130], but 4M is able to perform this using single model trained in one single run, and 4M is able to flexibly use any combination of input modalities (see Figure 11). In addition, unlike ControlNet which requires external networks to extract the conditioning modalities, 4M models are able to generate all of them by themselves.
24

---

Condttioing
Generated images
Consttoning
Generated images
Condioring
Gererated imague
Insd image
Inud Image
bod imare
Donditoring
Conrrstted imaeges
Donditooing
Generated Imagee
Conditining
Gararaled ups
Irout Emigge
Inout Imagee
Ipd magee
Coraltioring
Inpit
Figure 13: Grounded generation of image variations: can be used to generate variations of reference image by conditioning modality extracted from the original image. This allows for generation of diverse images that similar geometrically, semantically, in another (user-specified) aspect.
25

---

Multimodal guidance. Multimodal guidance, as explained in Appendix A.3, can be used for various fine-grained editing tasks. We show here several demonstrations of how can be used to (1) use weak/strong weighling to generate images that loosely/strongly match certain input modality, and (2) avoid the generation of certain undesired concepts via negative weighting.
Surface sormals
Geneated imagee
Surface nommals
Genargted Images
Sem, segment.
images
Sam, sugnar.
mages
imagas
Surface nomale
Genargted Images
Sam, segment,
images
Sam augmani
mages
Surface nommal
Generated Images
Sum, segment,
Images
Sarn sugmari
mages
Figure 14: Multimodal guidance: By weakly conditioning geometry (here surface normals) or semantics (here semantic segmentation maps), and strongly conditioning on a caption, we can generate image variations that only loosely grounded in some aspect of the original image.
26

---

Surface normals
Generated Images
Input
Gapila drowing oF besvice
Waster osomabic semantic conditwonng
Shotger geametic condkborung
segment.
Cbed
Generated Images
Surface normals
Inout image
Fivdd
Capilan Fiai:: bead
Wealer goomestle aomart's conditiiit
Sbonger gearebic gemerdi canchooneg
Sem. segment.
Pand wnaee
Surface normals
Generated Images
Input image
Cupilon FEuE afa bethoyy cabe
condioorvng
Sem, segment.
Doet 2ettt
Multimodal guidance: Similar to Figure 14, we show here multimodal guidance with captions and either normal semantic segmentation map. We keep the weight of the caption fixed and vary the weight of the other modality. The examples demonstrate how a user can effectively control to which degree the generation should use certain conditioning.
Figure 15:
27

---

Imput image
Images
Input Image
Images
pabli werorgg 5f4
Do gract JAIY
Impul image
Generaled images
Irouu Image
Generated imagee
Castion ROAR
Captin texd foa
Irpul image
Generaled images
Inpul image
Generated images
Cappos FOeE
Capler EgAs
Figure 16: Multimodal guidance: In this example, weakly condition masked RGB image, and strongly on a caption and bounding box. The generations follow the caption and bounding box precisely, while showing some degree of variation for the rest of the generated image.
images conditioned on
images on
Poslhs
bi akry abatt
INSSCIEN
Pertler
Nepothv
Heyuthv
mages conditioned on captions
images on captions
Peslhss
doiing
shing
Negithe
Negathe
Det
Chil:
Figure 17: Multimodal guidance: Multi-captioning with negative weighting. (Top of each figure): Standard text-to-image generation can create pleasing results, but the user might want greater control over the generation to counteract dataset biases (e.g., most generated buses are yellow), or may have envisioned a different image. (Bottom of each figure): Enabled by the process described earlier, by adding an additional caption with a negative guidance weight, we can steer which concepts are generated, and which are not.
28

---

Probing the learned representation. We can probe what representations 4M models learned either by performing transfers to various downstream tasks or manipulating parts of the input and observe how 4M predicts some readout modality. Concretely, we perform these representation probes by keeping the entire input fixed and manipulating only single aspect of it.
NNV
Figure 18: Probing 4M representations: In this example, condition the generation several bounding boxes. For each of the generations inside the blue cells, we keep all bounding boxes fixed, except for (marked in red shade), which move around and resize. Depending bounding box is placed, 4M implicitly infers the semantic and geometric properties of the object. For example, placing bicycle bounding box in front of bed bounding box generates real bicycle, while placing it above generates pictures of bicycles.
where the
Figure 19: Probing representations: Here, we condition the generation on segmentation maps in which we manipulate single segment and keep all others fixed. 4M creates plausible generations, no matter what class the segment was changed to.
29

---

B Multimodal Dataset & Tokenization
We pseudo label large aligned multimodal dataset using CC12M [19] and several pre-trained networks to map RGB images to pseudo labels. To enable training single Transformer on large number of vastly different modalities, we resort tokenization. Concretely, we train modality-specine tokenizers, allowing us to map different modalities to sequences or sets of tokens.
B.1 Pseudo labeled multimodal training dataset details
We trained all 4M models using an aligned multimodal dataset that we created by pseudo labeling CC12M. CC12M already contains aligned RGB images and captions, and we pseudo labeled the remaining tasks using powerful off-the-shelf models.
Surface normals & depth. To introduce geometric priors and to enable generation and transfer learning conditioned on measured estimated geometric modalities, pseudo labeled both surface normals and scene depth. We used DPT-Hybrid [91] models trained on Omnidata [32] using cross- task consistency [125] and 3D data augmentations [61] predict normals and depth from the CC12M RGB images. To encode images using DPT, we resized their heights and widths to the closest multiple of 32, while keeping the maximum side length below 768 to avoid prediction degradation from too large image sizes. We found that even though the Omnidata DPT models were mostly trained on simulated and indoor scenes, they generalize well a wide range of scenes.
Semantic segmentation. For introducing dense semantic priors, semantic segmentation maps can be useful allow for fine-grain fine-grained creative control when conditioned during generation [40]. We pseudo labeled semantic segmentation labels using a Mask2Former [24] with a SwinB [71] backbone trained on COCO [67] panoptic segmentation. We select the labels by taking the argmax over the predicted semantic classes.
To add object bounding boxes to the training data, we use ViTDet ViT-H model [66] initialized from MAE weights [48] and fine-tuned on COCO [67]. We filter the detected bounding boxes by removing all instances with confidence score below 0.6.
Bounding boxes.
CLIP feature maps. Tokenized CLIP [85] feature maps have been demonstrated be powerful targets for masked image models [82, 37, 36]. We use the ViT-B/16 visual backbone of CLIP-B16 model to extract dense feature maps from its last Transformer layer. To visualize dimensional CLIP feature map, we project all H + W entries onto the first three principal components computed from said feature map, interpreting the normalized values as R, G, and colors [126].
x W D
B.2 Tokenization of captions & bounding boxes
We follow Pix2Seq [21, 22] and treat detection ample. a scene containing a bounding box around a cat and a plant could be parameter- ized as xmin=0. 15 ymin=0.3 xmax=0. .6 ymax=0. 5 cat xmin=0. 75 ymin=0.3 3 xmax=0. 95 ymax=0.8 8 potted plant [EOS]. In practice, we model the corner coordinates (i.e. the minimum and maximum and y coordinates) using resolution of 1000 special tokens per each of those cases namely we represent bounding boxes using xmin=0, ymin=999, xmax=0, method involves masking parts of the object sequence, which requires two modifications to the sequence construction pipeline to make this task more manageable: First, we order the objects in the sequence by their distance to the origin. Second, unlike in Pix2Seq where corner coordinates share tokens, we assign separate tokens for each of the corner coordinates, resulting in vocabulary size 4 times larger than in Pix2Seq. Assigning separate tokens for each corner coordinate ensures that the meaning of each token
sequence prediction problem. For ex-
xmin=999, ymin=0, xmax=999, ymax=0, .... ymax=999. In contrast to Pix2Seq. our
made ambiguous when surrounding coordinates are masked out.
We jointly fit WordPiece [30] text tokenizer on all captions and the 4000 special bounding box tokens. COCO class labels for the bounding boxes are treated special tokens, meaning that if they appear in any caption, they get mapped to the same tokens. The joint text and bounding box vocabulary has size of 30K.
B.3 Tokenization of dense modalities
We tokenize dense modalities using variants of vector-quantized autoencoders (VQ-VAEs, [110]. To avoid blurry and unrealistic images, we train the tokenizers for RGB, normals and depth using diffusion decoders. All tokenizers are first trained on 100 epochs of an ImageNet- 1K version of the pseudo labeled dataset (see Appendix B.),, after which they continued training for 15 epochs on the
30

---

pseudo labeled CC12M dataset. We follow up the training at the base resolution of 224 with short multi-resolution fine-tuning step to adapt the tokenizers up to resolution 448².
The base resolution training specifics of our tokenizers are shown in Table 3. We employ several common practices: Following Yu et al. [121], heavily reduce the dimensionality of the codebook and Ig-normalize the codes and encoded vectors to improve codebook utilization and reconstruction quality. In addition, solution, we found it to be crucial to restart stale codebook entries, similar to Zeghidour et al. [127]. To that end, we count the number of encoded vectors in batch that map to a given codebook entry after every iteration, and replace (randomly from the batch) any codes that have an exponential moving average (EMA) count less than specified threshold thresh Because this value depends on the total batch size B, number of tokens per image Ninknnss and codebook vocabulary size N vocah. we specify this EMA threshold using a coefficient Creplace by:
avoid the joint VQ-VAE and diffusion training collapse to degenerate
BN
thresh
Creplace N Vvocab
Table 3: Tokenizer training settings, Base resolution (224²) training configuration for the tokenizers used for training 4M. The base learning rate is specified for batch size 256.
Configuration
RGB
Normals
Depth
Segmentation
C1.IP
Codebook size
16384
8192
8192
4096
8192
Patch size
16
Latent dimension
EMA stale code coef. Creplace
32
Codebook EMA
0.99
ln-nomalized codes [121]
Codebook weight
1.0
Commitment weight
1.0
Encoder architecture
ViT-B/16 Patch-UNet
Decoder architecture
Patch-UNet / MSE
Putch-UNet / MSE
VIT-B/16 Cross-entropy
VIT-B/16
Diffusion decoder
X Smooth
Loss function
Smooth L1
Optimizer
Adam [73]
Opt. momentum
8 82 = 0.95
Weight decay
0.05
Base learning rate [46]
5e-5
Batch size
640
640
640
640
1024
Learning rate sched.
Cosine decay 100 (INIK) + 15 (CC12M)
Training epochs
Warmup epnchss
(INIK) + (CC12M)
Resolution
224 (0.8, 1.0)
Random crop scale
(0.8, 1.0)
(0.8, 1.0)
(0.8, 1.0)
(0.2, 1.0)
Random crop ratio
(0.75, 1.3333)
Horizontal flip
Data type
float16
float16
float16
bfloat16
bfloat16
Semantic segmentation & CLIP feature maps. We tokenize segmentation maps and CLIP feature maps using VIT-B [31] encoders and decoders (without the patch projection on the latter). The reconstruction loss for segmentation maps cross-entropy, while for CLIP we choose the smooth LI loss. While we do not use any data augmentations for training 4M, we decided to train the CLIP tokenizer using a minimum crop scale of 0.2 to keep open the option of performing fine-tuning with CLIP targets and strong augmentations [82]. For all other tokenizers, we use more moderate minimum crop scale of 0.8. to avoid training them on very low-resolution but upscaled images that could affect image fidelity negatively.
RGB, normals & depth. Training VQ-VAE by simply minimizing a reconstruction loss can lead to blurry and unrealistic looking results [88], which is why VQ-GAN [35] proposes to use discriminator to get more realistic results. We found VQ-GAN training be unstable and decided to instead train diffusion model as the decoder, similar to DiVAE [101]. Unlike DiVAE, which trains diffusion model decoder using a frozen and pre-trained VQ-GAN encoder, we train both the encoder and diffusion decoder end-fo-end and from scratch. As the decoder, we use UNet [80] with four down and up layers, each consisting of three residual blocks and one up/downsampling block, with self-attention blocks at the two smallest down and up layers.
31

---

For improved training and inference efficiency, we process C-channel image patches of size C' x 4 x 4, rather than individual pixels, similar Patched Diffusion Models [75]. We condition the diffusion decoder by concatenating the 32-dimensional codebook entries of the 14 x 14 encoded tokens with the noised input after patching, i.e. we concatenate the noised and patched image of shape 16C x 56 x 56 with the upsampled latent tensor of shape 32 x 56 x 56. The output of the UNet gets reshaped back to C' x 224 x 224 before computing the loss.
We found that predicting the noise leads to undesirable color shifts during inference, which is why we predict the clean image instead. In addition, we found that avoid the training to collapse to a degenerate solution. it is crucial to restart dead / unused codebook entries [127]. We train the diffusion decoder using 1000 DDPM [51] steps and linear noise schedule. At inference time, we sample from the decoder using 25 DDIM [105] steps.
Multi- training. To train super-resolution specializations of 4M (see Appendix A.1), need to be able to use the tokenizers both at the base resolution of 224 x 224, and at our chosen higher resolution of 448 448. Unlike convolutional VQ- however, our ViT based tokenizers do not perform well on resolutions different from the training resolution. For that reason, we perform multi-resolution training as a short fine-tuning step, and initialize from the weights trained at the base resolution. During training, the resolution of every batch is sampled randomly between 224 and 448, in increments of 32. All tokenizers were fine-tuned for one CC12M epoch, except for RGB, which was fine-tuned for five epochs. We lowered the batch size per GPU to 16, except for semantic segmentation, for which we lowered it to 10. The remaining settings are the same in Table 3.
Method & Training Details
C.1 Additional 4M architecture details
For both the ablations and the longer training runs we train 4M models of different sizes. Table 4 lists their respective encoder and decoder depths, model dimension, number of attention heads, as well as the number of trainable parameters (excluding the embedding layers).
Embedding details. Modality embeddings shared between the encoder and decoder. Additionally, as is commonly done for autoregressive models, the parameters of the input and output embeddings (i.e., the final linear layer) of the decoder are shared for sequence modalities. Similar to MAE [48] and MultiMAE [5], the positional and modality embeddings are re-injected into the encoder output before it is passed on to the decoder.
RGB pixels as input. All 4M models are trained accept both RGB pixels and tokenized RGB as input. This is implemented by treating RGB pixels and tokenized RGB as two entirely separate modalities, with RGB pixels being an input-only modality (as it isn't tokenized). This simple approach makes it easy to incorporate additional non-tokenized modalities as input in the future.
Span masking for sequences. We perform span masking [86] as follows: Given the probability of masking a token (Pmask), we randomly mask out tokens in the sequence and replace each consecutive span of masked-out tokens by sentinel token (e.g., [S_1], [S_2], [S_3] [s_3]...). The target sequence then consists of the masked-out spans delimited by the sentinel tokens, followed by a final sentinel token to signal the end of the sequence.
Unlike dense modalities, it is not possible to strictly respect the token budget when sampling from sequences due their variable length. Instead, treat the token budget strict upper bound and mask sequences as follows: For the input, we sample a masking probability Pmask from a uniform distribution and use it for span masking. If the sequence length after masking is greater than the input budget, we progressively increase Pmask until it fits within the assigned budget. For the target, if the sequence does not fit within the budget. we randomly truncate it while also ensuring that the first token of the truncated sequence is a sentinel token.
C.2 Training details
The training details for the 4M models used for the transfer experiments (Section 3) and the generation results (Section 4) are shown in Table 5. These hyperparameters were chosen using insights from the ablation study (Section 5), more detailed ablation results are shown in Appendix E. All 4M models were trained using bfloat 1 mixed-precisins
[13]. Training durations varied with model size; 4M-B was trained in 1.5 days on 64 A100 GPUs, 4M-L was trained in 3 days on 128 A
GPUs, while training 4M- XL took 8 days on 128 A100 GPUs. For 4M-XL. XL, we reduced GPU memory
32

---

Table 4: Model size variants: The model sizes and naming scheme follow T5 [86]. We exclude the size of the embedding layers in the parameter count.
Model
Encoder Blocks
Decoder Blocks
Model Dim
Num Heads
Total Params
4M-Ti
6
384
6
24M
4M-S
8
512
8
59M
4M-B
12
12
768
12
198M
4M-L
24
1024
16
705M
4M-XL
24
24
2048
32
28 1818M
consumption through the use of activation checkpointing along with optimizer state and gradient sharding (ZeRO-2 (87]), via PyTorch's Fully Sharded Data Parallel (FSDP).
Table 5: Pre-training settings. Training configuration for the 4M models used in the transfer experiments and generation results.
Configuration
4M-B
4M-L
4M-XL
Training length (n tokens)
500B
Warmup length (n tokens)
10B
Optimizer
Adam W [73]
Opt. momentum
31. 8/
0.9, 0.95
Base learning rate [46]
le-4
le-4
2e-5
Batch size
8192
Weight decay
0.05
Gradient clipping
x
X
3.0
Learning rate schedule
Cosine decay SwiGL. [100]
Feedforward activation (Appendix E.6)
Input token budget (Appendix E.4)
128
Target token budget (Appendix E.4)
128
Input and target (Appendix E.4)
0.5, 0.5
Masking strategy (Appendix E.4)
RGB + All & All - All
Image resolution
224
Augmentation
None (Center Crop) 4 bfloat16 [13]
Repeated sampling [38] (Appendix E.7)
Data type
D Transfer Experiments Details
In this section, we report the fine-tuning details for all transfers shown in Section 3. Whenever possible, we follow commonly used settings from other papers performing such transfers. However, to avoid excessive computational costs, adjust some transfer settings such that single transfer requires more than 20% of the pre-training compute. These adjustments include reducing the training resolution when fine-tuning on COCO, and reducing the number of epochs for intermediate fine-tuning on ImageNet-2IK.
D.1 Architectural differences with the baselines
4M 's encoder and all baselines be viewed ViT-Base ViT-Large backbones [31], and nearly identical in terms of parameter count and model FLOPS. Despite this, a few minor architectural differences could potentially affect their performance. These include: (I) 4M lacks bias terms and uses SwiGLU [100] in the feedforward, while the baselines include bias terms and use GELU [49]. (2) All baselines include additional learnable [CLS] token input, which is absent in 4M. (3) DeiT III uses learnable positional embeddings, whereas all other methods including 4M use fixed dine-cosine positional embeddings. (4) When fine-tuning on COCO, MAE uses both absolute and relative positional embeddings (following ViTDet's approach [66]), while all other methods only use absolute positional embeddings.
33

---

D.2 Image classification on ImageNet-1K
We use settings inspired by [114] and first perform intermediate fine-tuning on ImageNet-21K [93], followed by full fine-tuning on ImageNet-IK [29]. Intermediate fine-tuning helps improve overall performance for models that were not originally trained on ImageNet-21K (all 4M models and baselines aside from DeiT III). Details are shown in Table 6.
Table 6: Image classification settings. Configuration for intermediate fine-tuning on ImageNet-2IK and fine-tuning ImageNet-IK.
Configuration
ImageNet-21K
ImageNet-IK
Base
Large
Base
Large
Fine-tuning epochs
20
50
20
Warmup epochs
2
Optimizer
Adam W [73]
Adam W [73]
Opt. momentum
31, 32
0.9, 0.95
B1, B2
0.9, 0.999
Base learning rate [46]
le-4
le-4 4096
Batch size
4096
Weight decay
0.05
0.05
Learning rate schedule
Cosine decay
Cosine decay
Layer-wise Ir decay [26]
0.75
0.85
0.75
0.85
Drop path [55]
0.1
0.2
0.1
0.2
Input resolution
2242
224
Augmentation
RandAug(9, 0.5) [27]
RandAug(9, 0.5) [27]
Random resized crop
(0.5, 1)
(0.08, 1)
Label smoothing
0.1
0.1
Mixup [129]
0.1
0.1
Cutmix [124]
1.0
1.0
D.3 Object detection and instance segmentation COCO
We follow the settings from ViTDet [66] with simplified Cascade Mask-RCNN head [47, with two major changes: First, we reduce the image resolution from 1024x 1024 to 512x512, which reduces computational costs during training by over 4x but lowers the performance. Second, we do not use window attention, and instead keep the attention layers unchanged (i.e. global) for all models and rely on activation checkpointing to reduce the GPU memory usage. Details are shown in Table 7.
Table 7: Object detection and instance segmentation settings. Configuration for object detection and instance segmentation fine-tuning on COCO, the settings follow ViTDet [66].
Configuration
COCO
Base
Large
Fine-tuning epochs Optimizer
100
AdamW [73]
Opt. momentum
81,02 0.9, 0.999
Weight decay
0.1
Learning rate
0.0001
Learning rate schedule
Multi-step decay [Epoch 89, Epoch 96]
Lr schedule milestones
Lr schedule decay values
[1.0, 0.1, 0.01]
Warmup epochs
0.01
0.007
Batch size
512
320
Layer-wise decay [26]
0.7
0.8
Drop path [55]
0.1
0.4
Input resolution
5122
Augmentation
Large-scale jitter (LSJ) [41]
34

---

D.4 Semantic segmentation on ADE20K
follow the settings from MultiMAE [5] and use ConvNext [72] prediction head depth on top of the encoder to perform semantic segmentation. Details are shown in Table 8.
Table 8: Semantic segmentation settings. Configuration for semantic segmentation fine-tuning on ADE20K, the settings follow MultiMAE [5].
Configuration
ADE20K
Base
Large
Fine-tuning epochs
64 1
Warmup epochs
Optimizer
Adam W [73]
Opt. momentum
B1 B2
0.9, 0.999
Learning rate
2e-4
Batch size
64
Weight decay
0.05
Learning rate schedule
Cosine decay
Layer-wise Ir decay [26]
0.75
0.85
Drop path [55]
0.1
0.2
Input resolution
5122
Augmentation
Large-scale jitter (LSJ) [41]
Color jitter
D.5 Depth estimation on NYUv2
We closely follow the settings from MultiMAE [5] but replace the DPT [91] prediction head by ConvNeXt [72] prediction of depth 2. We find that this ConvNeXt head reaches comparable performance DPT on the MultiMAE baseline, while also removing the need to extract and rescale features from intermediate layers of the network. Details are shown in Table 9.
Table Depth estimation settings. Configuration for depth estimation fine-tuning on NYUv2, the settings follow MultiMAE [5].
Configuration
NYUv2
Base
Large
Fine-tuning epochs
1000
Warmup epochs
100
Optimizer
Adam W [73]
Opt. momentum
B1, 82
0.9, 0.999
Learning rate
lc-4
Batch size
128
Weight decay
le-4
Learning rate schedule
Cosine decay
Layer-wise Ir decay [26]
0.75
0.85
Drop path [5S]
0.1
0.2
Input resolution
256
Random crop
Color jitter
35

---

E
Ablation Details & Results
E.1 Benchmark tasks
All transfer tasks in the ablation are cast as token-to-token prediction problems, similar to the way 4M was pre-trained. We perform several transfers with RGB (pixels) as the input, test how well 4M variants perform adapting to unseen input modalities, and ablate their ability make use of multiple input modalities.
All models fine-tuned with constant learning rate and evaluated using exponential moving average of the weights. As different models may overfit to the transfer tasks at different speeds, we measure the validation loss 50 times per transfer and report the best loss, as in T5 [86].
For all tasks and modalities, we train new tokenizers on the respective training sets, which means that they are all either completely unseen, or a new instantiation of a training task. When transferring from an unseen modality (e.g.. in X-> and X+Y-ZZ transfers). the input embeddings are not initialized, which can lead to instabilities and lower the overall performance. To address this problem, we initially freeze the Transformer encoder-decoder and only train the new input and output embedding layers. Afterwards, we unfreeze the Transformer and train the entire model.
As described in Section 5, we report the validation set cross-entropy performance instead of task- specific metrics. The reasons for this are twofold: First, downstream performance hinges on A) how well the tokens are able to represent the downstream tasks (i.e. tokenizer reconstruction error), and B) how well 4M is able to predict these tokens (i.e. cross-entropy loss). Since the tokenizers are the same across all settings, we abstract away aspect for this ablation and only report B. Second, reporting the cross-entrope loss makes comparisons more uniform and avoids having to scale and average wildly different task-specific metrics such as mAP, mloU, MSE, etc. that are not comparable with each other.
COCO transfers. We perform object detection the COCO dataset [67] in the same manner as during pre-training (see Appendix B.2), i.c. by casting it as sequence prediction problem [21]. The input modality for the transfer task is RGB (pixels). Exact training settings are provided in Table 10.
ADE20K transfers. We perform semantic segmentation on ADE20K [131] by fine-tuning the 4M models to predict VQ-VAE tokens of the targets. Since our semantic segmentation tokenizer during pre-training was trained with COCO classes, we train a new tokenizer for ADE20K in the same manner (see Appendix B.2). The tokenizer is trained for 1500 epochs with a warmup period of 150 epochs. In order to perform random crop augmentations at transfer time, we also pre-trained the tokenizer with a crop scale of (0.2, 1.0). The input modality for the transfer task is RGB (pixels). Exact training settings are provided in Table 11.
Table 11: ADE20K semantic segmentation transfer settings.
Table 10: COCO detection transfer settings.
Configuration
COCO Det.
Configuration
ADE20K Seg.
Fine-tuning epochs
100
Fine-tuning epochs
200
Warmup epochs Optimizer
5
Warmup epochs
20
AdamW [73]
Optimizer
Adam W [73]
Opt. momentum
B1.88 82 = 0.9, 0.999
Opt. momentum
81, 82
0.9, 0.999 Ie-4
Base learning rate [46]
Base learning rate [46]
1c-4
Batch size
256
Batch size
256
Weight decay
0.05
Weight decay
0.5
Learning rate schedule
Constant
Learning rate schedule
Constant 0.998
EMA decay
EMA decay
0.998
Eval. freq (epochs)
2
Eval. freq (cpochs)
4
Input resolution
224
Input resolution
224 Random crop (0.2, 1.0) (0.75, 1.333) X
Augmentation
Large-scale jitter (LSJ) [41]
Augmentation
Color jitter
/
Crop scale
Crop ratio
Color jitter
Taskonomy-20k & Hypersim RGB-+X transfers. We perform several transfers on Hypersim [94] and subset of Taskonomy [126] Tiny, denoted here as Taskonomy-20K, The subset contains 20' 0000 training images and 2000 validation images.
36

---

For Diskonomy-20K, we transfer to depth, principal curvature. reshading. occlusion edges, edges, 2D keypoints, and 3D keypoints. For Hypersim. we transfer to surface normals, semantic segmentation, and 2D edges. The input modality for all transfer tasks is RGB (pixels). Exact training settings are provided in Table 12.
T & Hypersim X-Y transfers. There are 72 possible transfers between the Taskonomy tasks, and 12 possible transfers between the Hypersim tasks. To reduce the computa- tional cost of the ablations, we subsample these transfer tasks, trying to cast a diverse net. For Taskonomy-20K, we perform curvature + 2D keypoints. depth - normals, + depth, surface normals + 2D edges, occlusion edges + principal curvature, and RGB (tokens) + surface normals. For Hypersim we perform RGB (tokens) + segmentation. 2D edges + segmentation, normals 7 segmentation, segmentation - 2D edges, and segmentation > normals. Exact training settings are provided in Table 13.
edges
T0K & Hypersim X+Y transfers. As before, we subsample all pos- sible transfer tasks that use two modalities as the input, aiming for diverse set of transfers. For Taskonomy-20K, we perform RGB (pixels) + normals - depth, RGB (pixels) depth + reshading, reshading 2D keypoints + normals, 2D edges + 3D keypoints + RGB (tokens), 2D edges + curvature + occlusion edges, and occlusion edges 2D keypoints + curvature. For Hypersim we perform RGB (pixels) + depth + segmentation. RGB (pixels) + normals - segmentation, RCB (pixels) + segmentation + depth. depth segmentation + RGB (tokens), 2D keypoints + normals + segmentation and 2D edges + depth segmentation. Exact training set- tings are provided in Table 13.
Table 12: RGB -X transfer settings.
Table 13: X-> and X+Y-Z transfer settings.
Configuration
Taskonomy- 20K
Hypersim
Configuration
Taskonomy 20K
Hypersim
Warnup spochs
Vine-tuning epochs
100
50
Fine tuning epochs
Warmup epodts
100
50
Frozen Transfomer epochs
Frozen Transomme spochs
25
Optimiece
Adam [73]
Optimizer
Adama [?3]
Base learning rite [46]
Opt. momentum
91.% 0.9, 0.389
lo-4
Base learingg ratee [46]
momentumm
A, 0.9, 0.9,0.900 0,3.8
Frozen Transfomer basc Ir
le-1 256
Provznn Trandomer
26-3
Batch size
Weight decay
0.05
Weight decay
Batch size
0.05
256
Learning rale schedule
EMA decay
Consttant 0.998
Learning rale schedule
decay
Constrat 0.998
Eval. freq (epocts)
Eval. froy (ppochs)
2
Crop scale
Crop ratio
Color jitier
Augmentation
Input resolution
2247 Random crop (0.2, (0.75, L.333)
Crop ratio
Augmentation
Crop scale
Input resolution
Color jitter
Only if RGB is in the input
Random crop (0.2,
(0.75. 1.333)
224"
E.2 Reference model
In this section, we specify the reference model and its training settings. These choices are made by following common practices and performing educated guesses. The settings include model size, number of encoding and decoding layers, additional modifications on the architecture (e.g. no bias), choice of Dirichlet sampling parameter a, input and target modalities, and training length. Following common practice for the base model size [30, 86), our base model consists of 12 encoder and 12 decoder layers. We train it on CC12M [19] using all modalities as both inputs and targets, at resolution 224 x 224 pixels (corresponding to 14 14 tokens per dense modality), and using no augmentations such as cropping nor color augmentations. The total training length is fixed at 100B tokens seen, corresponding to roughly 400M masked samples seen for the base setting. We set the number of randomly sampled input and target tokens both to 128, and sample each using symmetrie Dirichlet distribution with parameter a = 0.2. Full details are shown in Table 14.
To determine the significance of various modeling choices, we adopt the approach used by Raffel et al. [86] that calculates the standard deviation of the transfer results of ten distinct reference models. each trained with different seeds. In the subsequent ablation results, the average performance of the reference model setting is indicated by the symbol D. Furthermore, we highlight transfer results falling within two standard deviations (shown in Table 15) of the lowest achieved losses, enabling us to estimate which settings are significant. Table 15 further contains from-scratch results (training a randomly initialized model) for all the transfer tasks. This serves as a lower bound for the performance
37

---

that can be achieved on a given transfer task, and allows us to calibrate how difficult each transfer is, and by how much pre-training with given setting improves upon random initialization.
Table 14:
Reference model pre-training settings. Training configuration for the reference 4M model used in the ablation. All other models in ablation are obtained by changing just one of these hyperparameters.
Configuration
Reference model
Model size
Base (see Table 4)
Training length (n tokens)
100B
Warmup length (n tokens)
10B
Optimizer
AdamW [73]
Opt. momentum
B1, B2 0.9, 0.95
Base learning rate [46]
le-4
Batch size
4096
Weight decay
0.05
Gradient clipping
x
Learning rate schedule
Cosine decay GELU [49]
Feedforward activation
Input token budget
128
Target token budget
128
Input and target a
0.2, 0.2
Masking strategy
All + All
Image resolution
2242
Augmentation
None (Center Crop) 4
Repeated sampling [38]
Table 15: Reference model results: We report the average and standard deviation of the transfer results achieved by our reference 4M model. In addition, we show performance achieved by randomly initialized model. All results are reported on the validation sets of each dataset.
COCO Det.
ADE20K Seg.
RGB >X
X +++Y
X+Y +Z
Avg. Loss
Reference model avg. No pre- tre-training
3.06
6.11
5.07
6.75
5,80
5.36
3.31
6.59
6.03
7.52
6.62
6.01
Reference model std. dev.
0.008
0.008
0.021
0.018
0.016
0.010
E.3 Input modalities and target tasks
Importance of target tasks for representation learning. 4M is both a multimodal and a multitask pre-training scheme and the choice of target task(s) is a powerful way of steering what representation the model learns [10, 109]. Furthermore, multitask pre-training has been shown to improve down- stream task transfers [42, 107, 5J. In this ablation, we would like learn how the choice of target tasks affects transfers to various downstream tasks. For that, we fix the input modalities to be either RGB or all the modalities (denoted as "All"), and vary just the target tasks. The results in Table 16 (or equivalently, Table 2) mirror the findings of Sax et al. [98], that the optimal choice of pre-training setting depends highly on the type of transfer that is performed, and that there no single pre-training task that performs best on all transfers. That said, pre-training on all target modalities outperforms the other single-task and multitask alternatives in terms of average loss, no matter what input modalities were used during pre-training. Note that some of the settings in Table 16 are conceptually similar to well-known pre-training methods, and can serve as basclines for such methods, while abstracting away implementation details that would make them difficult to compare otherwise. E.g., RBG - RGB (first row) is similar to MAE [48], while RGB + CLIP (fifth row) is similar to BEITv2 [82].
Importance of multimodal pre-training for transferring to new modalities. Just as multitask pre-training improves transfers to new target tasks, we would hope that, likewise, multimodal pre- training improves transfers to new input modalities. Indeed, pre-training with multiple modalities
38

---

that may be available at transfer time has been shown to be beneficial [5], but it is not clear how well multimodal models transfer to completely new, unseen imput modalities. Table 16 shows that multimodal pre-training can significantly help with transferring to new input modalities (X-+Y transfers), but comes performance loss at transfers that use RGB as the sole input modality. In Appendix E,4 we will explore pre-training using mixtures of different masking strategies, which enables us to train models that perform well in both regimes.
Table 16: Pre-training input and target modalities ablation: The choice of pre- pre-training tasks and modalities influences what representations the model learns, and how well it can be transferred to novel tasks and modalities. The average rank and best losses shown for "RGB" and "All" inputs separately. Here, Geometric = RGB + Depth + Normals and Semantic RGB + Segmentation + CLIP Detection Captions. Performing 4M pre-training on all input and target modalities the most versatile choice, if the optimal set of pre-training modalities for any given downstream task is unknown. Note that the results shown here identical those of Table and included for completeness.
Pre-training
Pre-training targets
COCO Det.
ADE20K Seg.
RGB-X
X-Y
X+Y-Z
Avg. Loss
Avg. Rank
imputs
RGB
RGB
3.14
6.21
5,03
6,94
6.15
5.49
8.00
RGB
Depth
3.11
4.72
5.32
4.00
RGB
Normals
3,12
5,30
3.20
RGB
Segmentation
3.17
5.94
4.84
6.86
5.34
4.60
RGB
CLIP
3.07
6.11
4.83
6.85
5.94
5.36
4.80
RGB
Detection
2.78
6.11
5.03
7.07
6.24
5,45
7.20
RGB
Captions
3,45
6.55
5.92
7.35
6.86
6.03
10.00
RGB
Geometric
3.11
6.08
4,70
6.88
5.85
5.32
3.80
RGB
Semantic
2,88
5,99
4,86
6.97
6.06
5.35
5,40
RGB
2.90
4.74
5,29
4.00
All
RGB
3,21
6,20
5.07
6.75
5,85
5.42
3.80
All
CLIP
3.19
5.06
5.42
3.80
All
Geometric
3,20
5,36
1.80
All
Semantic
3.05
5.16
5.39
3.40
All
All
3.06
6.11
5.07
6.75
5.80
5.36
2.20
E.4
Multimodal masking strategy
Multimodal masking is at the core of 4M, so in this section, we ablate how modality tokens should be sampled and mixed, and how many tokens we should encode and decode. When performing multimodal training, it can be computationally challenging to deal with the large number of tokens from modalities. For that reason, the choice of the number input and target tokens has the potential to greatly reduce the computational burden, and setting them low possible is desirable.
Mask sampling parameter. The number tokens to sample from each modality determined by a symmetric Dirichlet distribution with parameter [5]. If a is low, the sampling procedure will often choose cases where most of the tokens are sampled from only one modality. If a high, however, most samples will contain tokens from all modalities to equal proportions. We ablate here various choices of values, both for the input and target sampling. Table 17 shows that models trained with higher values generally transfer better to RGB tasks, but methods trained with lower input values perform better at transferring to novel input modalities.
Table 17: Mask sampling parameter ablation: The choice of Dirichlet sampling parameter influences how many tokens are sampled from each modality. Low values lead to many samples consisting of entire modalities, while high values result in equal numbers of tokens be sampled from each modality. We ablate various input and target a values.
Input a
Target a
COCO Det.
ADE20K Seg.
RGB >X
X-+Y
X+Y
Avg. Loss
Avg. Rank
0.1
0.1
3.11
6.14
5.18
6.81
5.89
5.43
4.80
0.2
0.2
3.06
6.11
5.07
6.75
5.80
5,36
3.20
0.5
0.5
3.02
6.76
5,33
1.0
3,00
6.80
5.32
00
00
3.00
6.08
4.93
6.84
5.83
5.34
2.60
39

---

Input masking budget. The difficulty of the multimodal masked modeling task is largely deter- mined by the number of visible mon-masked) input tokens, with fewer tokens used making the task more challenging. Indeed, since the modalities contain lot of spatial information about each other, it is necessary lower the number of visible tokens to keep the objective difficult enough. Furthermore, encoding only small set of visible tokens (as opposed to encoding both visible and mask tokens [30, 7J) can significantly improve training efficiency [48]. This is especially important when training multiple modalities [5], the input length would otherwise grow too large.
Table 18: Input masking budget ablation: The number of visible input tokens decides the difficulty of the masked modeling task (the fewer given, the harder the task), and influences the computational cost of the encoder (the fewer, the less expensive). We fix the target masking budget at 128 tokens and vary the number of input tokens.
Input Tokens
COCO Det.
ADE20K Seg.
RGB-X
X-Y
X+Y
Avg. Loss
Avg. Rank
64
3.15
6.21
5.27
6.89
5.99
5.50
4.00
b 128
3.06
6.11
5.07
6.75
5.80
5.36
2.00
256
3.03
6.08
5.04
6.71
5.75
5.32
1.00
512
3.07
6.12
5.11
6.78
5.83
5.38
3.00
Target masking budget. Just like how we could improve the training efficiency by only decoding a small set of visible tokens, so can we decide to decode merely subset of all remaining masked-out tokens. Decoding all masked-out tokens, like MAE [48], in the multimodal case MultiMAE [5], can quickly become infeasible as the number of modalities grows and the masking ratio is kept high. While they attempt to address this issue by decreasing the size of the decoder, such strategy might ultimately hurt generation quality if the decoder is used for generative tasks. As Table 19 shows, decoding only a small random subset of all targets performs better (for fixed training duration) than decoding a larger number of tokens, while also significantly reducing the computational costs.
Table 19:
Target masking budget ablation: Similar to passing only visible tokens to the encoder, can decide to only decode subset of masked-out tokens, which improves computational efficiency considerably. We fix the input masking budget at 128 and vary the number of target tokens.
Target Tokens
COCO Det.
ADE20K Seg.
RGB-X
X-+
X+Y-Z
Avg. Loss
Avg. Rank
64
3.06
6.11
5.09
6.73
5.79
5.36
1.40
> 128
3.06
6.11
5.07
6.75
5,80
5,36
1.60
256
3.10
6.16
5.15
6.81
5.91
5.43
3.00
512
3.16
6,20
5.27
6.84
5.95
5.48
4.00
Mixture of masking strategies. As shown in Appendix E.3, pre-training using RGB as the sole input modality instead training on all of them performs significantly better at transfers that, likewise, use RGB as input. On the flip-side, pre-training with all modalities performs significantly better when transferring to unseen input modalities. This difference can explained by the fact that these two mask sampling strategies are opposite extremes the number of RGB inputs in the multimodal pre-training case relatively small, while the RGB-only case never trained on any other modalities. We find compromise by sampling batch elements using these two strategies uniformly at random, creating a pre- pre-training strategy, where approximately half the time input tokens RGB-only and half the time they sampled from all modalities. Table 20 shows that while the mixture approach does not perform better than either individual mask sampling strategy at any of the transfers, it is good compromise if the use-case of the model is open the time of pre-training.
40

---

Table 20: Mixture of masking strategies ablation: Mixing different masking strategies by randomly sampling between the two provides good compromise in downstream performance.
Masking strategy
COCO
ADE20K Seg.
RGB +X
X-+Y
X+Y
Avg. Loss
Avg. Rank
RGB - All
2.90
5.99
4.74
6.91
5.93
5.29
1.80
All + All
3.06
6.11
5.07
6.75
5,80
5.36
2.20
RGB - All & All + All
2.95
6.00
4.86
6.86
5.87
5.31
2.00
E.5
How well does 4M seale?
We argue that scalability is key property that models and training objectives should have. To ablate how well 4M scales, we ablate the following three axes: dataset size, training length, and model size.
Does 4M benefit from more data? We train the base configuration various subsets of CC12M, down to 1/64th of the full dataset. The results in Table 21 show that 4M is able to scale with the pre-training dataset size.
Table 21:
Dataset size ablation: To estimate how well 4M scales with pre- pre-trrining dataset size, train it on different subsets of CC12M ranging from 67 of the dataset to the entire dataset..
Dataset fraction
COCO Det.
ADE20K Seg.
RGB +
X-+Y
X+Y +Z
Avg. Loss
Avg. Rank
1/64
3.22
6.23
5.31
6,86
5.94
5.51
4,00
1/16
3.15
6.17
5.23
6.80
5.87
5.44
3,00
1/4
3.08
6.13
5.11
6.79
5.84
5.39
2.00
3.06
6.11
5.07
6.75
5.80
5.36
Does 4M benefit from longer training? We train the base configuration for different amounts of total tokens seen. We define tokens seen as the total number both input tokens and target tokens the model was trained on. Table 22 shows that 4M scales to longer training schedules.
Table 22: Training duration ablation: To whether 4M benefits from longer training schedules, we train for up to 400B tokens seen.
Train tokens [B]
COCO Det.
ADE20K Seg.
RGB X
X-Y
X+Y-IZ
Avg. Loss
Avg. Rank
50
3.13
6.17
5.23
6.86
5,93
5,46
4,00
3.06
5.07
6.75
5.80
5.36
3.00
200
3.00
6.05
4.93
6.71
5.74
5.29
2,00
400
2.96
6.02
4.84
6.70
5.71
5.25
1.00
Does scale with model size? We train models of different sizes, ranging from Tiny (4M-T1) to Large variants (4M-L). Exact model specifications are given Table 4. Table 23 shows that 4M scales with model size.
Table 23: Model size ablation: To see how 4M scales with model size, we train Tiny, Small, Base and Large versions.
Model
COCO Det.
ADE20K Seg-
RGB >X
>Y
X+Y >Z
Avg. Loss
Avg. Rank
4M-Ti
3,28
6.38
5.57
6.93
6.15
5.66
4.00
4M-S
3.16
6.25
5.34
6.84
5.97
5.51
3.00
h 4M-B
3.06
6.11
5.07
6.75
5.80
5,36
2.00
4M-L
2.93
5.97
4.86
6.63
5.75
5.23
1.00
41

---

E.6 Architectural design choices
the following, we ablate several design choices related to the Transformer architecture.
Number of encoder & decoder layers. We ablate whether one should train 4M using a balanced allocation of encoder and decoder layers rather go with a deeper the total number of Transformer layers to 24 and ablate three different configurations in Table 24. The results show that all configurations perform comparably, with the balanced and encoder-heaw settings having slight edge over the decoder-heavy one.
shallow decoder. We fix
Table 24:
Encoder & decoder depth ablation: We ablate three different ways of allocating 24 Transformer layers between the encoder and decoder.
Enc. depth
Dec. depth
COCO Det.
ADE20K Seg.
RGB +X
X-+Y
X+Y-IZ
Avg. Loss
Avg. Rank
8
16
3.10
6.14
5.09
6.75
5.82
5.38
2.80
12
12
3.06
6.11
6.75
5.36
1.40
16
8
3.05
6.11
5.11
6.75
5.36
1.80
Per-token vs. per-modality loss. Since we predict multiple modalities at the same time, there are compute their losses. We ablate the following We treat every predicted token the same and average all their losses. We call this setting per-token loss. This setting is biased against modalities that don't contain a lot of tokens, such as captions. (2) We first average the loss for every target modality individually and then average those. We call this setting per-modeee per-modality loss. Computing the loss per per-modality noticeably outperforms the per-toker loss, as shown in Table 25.
several ways
loss weighting strategies: (1)
Table 25: Multitask loss aggregation ablation: We ablate weighting the loss contribution of every modality equally vs. weighting every token equally.
Loss type
COCO Det.
ADE20K Seg.
RGB +X
X-+Y
X+Y-
Avg. Loss
Avg. Rank
Per Per-modality loss
3.06
6.11
5.07
6.75
5.80
5.36
1.00
Per-token loss
3.07
6.78
5.86
5.40
2.00
Transformer architecture modifications. ablate several modifications to the Transformer architecture, and show results in Table 26. (1) Following Shazeer [100], we ablate the use of the SwiGLU activation function
feed-forward layers, and reduce the feed-forward dimension by factor of 3 (from 4d to 4d) to keep the parameter count and amount of computation constant. (2) Following T5 [86] and PaLM [25], we ablate removing all bias terms from the Transformer. (3) Following ViT-22B [28], we ablate the of query/key normalization, which has been shown to help the stability of very large Vision Transformers trained to perform image classification.
We find SwiGLU slightly outperforms GELU while also removing the need for bias terms, and therefore use train the final model. As we do not observe noticeable stability issues with our training objective with current model scales, we refrain from using query/key normalization find that it negatively impacts the transfer performance.
Table 26: Architecture modifications ablation: We ablate the use of different activations in the forward [49, 100], removing bias terms [86, 25], and performing query-key normalization [28].
FEN act.
Bias
QK Norm.
COCO Det.
ADE20K Seg.
RGB-X
X-Y
X+Y+Z
Avg. Loss
Avg. Rank
GELU /
X
3.06
6.11
5.07
6.75
5.80
5.36
2.20
GELU
X
X
3.07
6.11
5.11
6.80
5.86
5.39
4.00
GELU
x
/
3.08
6.13
5.16
6.92
5.96
5.45
5.20
SwiGLU
X
3.08
5.16
6.94
6.26
5.51
5.80
SwiGLU
X
X
3.07
6.11
5.03
6.73
5.79
5.34
1.80
SwiGLU X
V
3.06
6.09
5.03
6.79
5.80
5.35
2.00
42

---

E.7
Training design choices
Base learning rate. ablate several base learning rates [46] and show results in Table 27.
Table 27:
Base learning rate ablation. We ablate the choice learning rate used during training. 4M not too sensitive to the choice of learning rate, although observe some performance degradation and instabilities when strongly increasing the learning rate.
Base Ir [46]
COCO Det.
ADE20K Seg.
RGB -X
X-Y
X+Y-Z
Avg. Loss
Avg. Rank
5c-5
3.06
6.11
5.07
6.78
5.82
5.37
2.80
> le-4
3.06
6.11
5.07
6.75
5.80
5.36
1.20
2e-4
3.05
6.11
5.10
6.75
5.86
5.37
2.20
3e-4
3.07
6.11
5.11
6.77
5.88
5.39
3.80
5c-4
3.10
6.14
5.17
6.81
5.89
5.42
5.00
Training with repeated sampling, Data loading can be a significant bottleneck when training efficient masked models. For that reason, 1000 samples instead of loading individual samples. We keep buffer loaded images in RAM and randomly sample from it repeatedly each time applying different random masking. Whenever an element the buffer has been sampled more than the specified number of repeats, we replace with fresh sample. Using repeated sampling [38] has the potential significantly improve training efficiency at no loss in performance (see Table 28).
webdataset [116] to load tar files consisting of
Table 28: Repeated sampling ablation: To improve training efficiency, we reuse samples from buffer multiple times, at no significant loss performance.
Num. repeats
COCO Det.
ADE20K Seg.
RGB >X
X-+Y
X+Y
Avg. Loss
Avg. Rank
3.06
6.11
5.06
6.73
5.80
5.35
1.40
D
3.06
6.11
5.07
6.75
5.80
5.36
1.60
8
3.07
6.12
5.10
6.76
5.81
5.37
3.00
E.8 Self-baselines
When comparing 4M to other pre-training methods the transfer learning study, we also report performance of two haff-baselins to better control for the dataset, augmentations, model architecture, compute, and use of tokemizers, all of which can significantly affect downstream performance. These two self-baselines were chosen to be conceptually similar to MAE [48] (Masked RGB - RGB) and BEiT-v2 [82] (Masked RGB - CLIP). Following these methods, we train these self-baselines with a relatively high masking ratio (in case, 50%) and ensure that the target tokens do not spatially overlap with the inputs. In this section, we ablate the masking strategy for these self-baselines by also training a version with lower masking ratio and with spatial overlap between the input and the targets (as in 4M), and a version without any masking and where all targets are predicted (standard RGB- training).
Results are shown Table 29, We find that while the RGB - RGB setting does benefit from higher masking ratios and spatial overlap (as shown in MAE), RGB - CLIP benefits from lower masking ratio and from allowing targets to spatially overlap with the input.
Table 29: Self- Seff-baselines ablation: The RGB + RGB self-baseline performs best with a higher masking ratio and spatial overlap between inputs and targets, while RGB CLIP benefits from lower masking ratio and spatial overlap.
Target
Input
Target
Spatial overlap
COCO Det.
ADE20K Seg.
RGB-
X-+
X+Y-
Avg. Loss
Avg. Rank
mod.
tok.
tok.
RGB
98
98
x 3.12
6.16
6.91
6.14
5.47
1.40
RGB
128
/
3.14
6.15
RGB
196
196
/
3.24
6.37
5.42
6.87
6.13
5.61
2.20
CLIPP
98
98
x
3.08
6.09
4.93
6.94
6.03
5.41
2.40
CLIP
128
128
/
3.07
6.11
4.83
6.85
5.94
5.36
1.20
CLIP
196
196
/
3.12
6.20
4.93
6.88
6.01
5.43
2.40
43

---

E.9 Comparison to the final models
We gather insights from ablation to train the "final" 4M models described in Appendix C. Here, we compare these models to the reference models used throughout the ablation. Unsurprisingly, we find that these models perform significantly better than the reference models on the benchmark tasks. The large performance gap for tasks that take input RGB images in large part be attributed to the mixture masking strategies described Appendix E.4.
Table 30: Comparison to the final models: The final models obtained by combining insights from the ablations significantly outperform the reference models all benchmark tasks.
Size
Setting
COCO Det.
ADE20K Seg.
RGB -
X
X+Y -Z
Avg. Loss
> Base
Reference (Table 14)
3,06
6.11
5.07
6.75
5.80
5.36
Base
Final (Table 5)
2.85
5.91
4.69
6.74
5.76
5.19
Large
Reference (Table 14)
2.93
5.97
4,86
6,63
5.75
5.23
Large
Final (Table 5)
2.68
5.76
4.48
6.67
5.63
5.04
F Additional Evaluations
F1
Out of the box (zero-shot) performance
In this section, we evaluate the out of the box capabilities of 4M models in three RGB-XX tasks: surface normals estimation, depth estimation, and semantic segmentation. The results are shown Table 31. Our findings demonstrate that 4M performs competitively in these tasks even without fine-tuning, when compared to both task-specific baselines and the pseudo labelers used for 4M training. Additionally, the tokenization quality, as shown in Table 31 and Figure 20, does not seem to limit the overall performance of 4M in these zero-shot scenarios. We therefore anticipate that the zero-shot performance 4M can be further improved if not limited by the current pseudo labels, either through the use of ground truth data or stronger pseudo labeling networks.
Table 31: Out of the box (zero-shot) performance: We report the out of the box performance of
4M models and of baselines on several tasks, all at 224 x 224 resolution. We use the DIODE [111]
validation set for normals and depth, and COCO validation set for semantic segmentation. [P]
denotes the pseudo labeler used to train 4M. Best results are bolded, second best are underlined. 4M's
zero-shot performance matches outperforms strong baselines such OASIS [23] and MiDaS [90],
and is competitive with pseudo labelers. * To estimate the performance upper bound of 4M due to tokenization, we also report the tokenizer reconstruction quality on validation images from CC12M.
Surface Normals
Depth (standardized L1 error).
Semantic Segmentation
Method
(mean angle error)
(mean loU)/
OASIS
34.3
MiDaS DPT Hybrid
0.73
3DCi [P]
22.5
0.68
Fask2Former Swin-S
44.6
Fask2Former Swin-B [P]
45.7
Fask2Former Swin-L
48.0
4M-B
21.9
0.71
41.3
4M-L
21.4
0.69
45.8
4M-XL
20.8
0.68
46.5
Tokenizer *
4.0
0.06
90.5
44

---

Ground Truth
Tokenizer
Ground Truth
Tokenizer
Ground Truth
Tokenizer
Surface Normals
Depth
Semantic Segmentation
Figure 20: Tokenizer reconstructions: We show sample reconstructions of surface normals, depth, and semantic segmentation pseudo labels from the CC12M validation set 224 224 resolution (14 x 14 tokens). Quantitative evaluations are provided in Table 31 (last row), confirming that the reconstruction quality does not bottleneck 4M's out of the box performance.
F.2 Text-to-image performance
In this section, we evaluate 4M's text-to-image capabilities, with detailed quantitative results shown in Table 32. To perform a controlled comparison, train pure text-to-image variant of 4M-B, akin Muse [18], for a total of 300B tokens on CC12M, and using the same RGB tokenizer as the 4M models. 4M trained on all modalities achieves comparable FID and CLIP scores this specialist model, and at the same time can be conditioned on any pre-training modality and can solve several common vision tasks of the box. We also compare against the 512 512 base model of Stable Diffusion 2.1 (SD-2.1) 1) [95] and observe considerable gap SOTA generative models on OOD data. It is important to highlight that SDD-2.1 was trained on datasets that were two orders of magnitude larger and with compute budget one order of magnitude greater than 4M-XL. When taking into account the scaling behaviors of comparable token-based ext-tto-image models such as Muse MAGE, we anticipate substantial improvement in the generation quality of 4M models if they were trained under a similar data and computational regime, alongside the use of improved image tokenizers.
Table 32: Quantitative evaluation of text-to-image generative capabilities: We evaluate the
text-to-image capabilities 4M and compare them with a controlled text-to-image version of 4M (similar to Muse), and Stable Diffusion 2.1-ba We compute FID and CLIP-L/14 scores on 30K images of the CC12M and COCO validation sets after resizing the generations to 256x256. All runs use guidance scale 3.0. In-domain, 4M can approach SD>2.1 and matches the Muse baseline in
controlled setting. COCO, we observe a larger gap to SD-2.1 which was trained on orders of magnitude more data and compute.
CC12M val (30K) CLIP score
COCO val (30K)
Model
Res.
Dataset
A100-hours
Text encoder
FID
FID 1
CLIP score
4M-B
224
CC12M
2.3k
15.9
20.6
37,5
21.4
4M-L
224
CC12M
9.2k
11.9
18.9
30.1
23.1
4M-XL
224
CC12M
24.5k
10.7
22.1
27.0
23.7
MM-B "Muse"
224
CC12M
1.6k
18.3
18.8
39.8
20.3
Curated
CLIP
SD-2.1-base
512
LAION-SB
> 200k
VIT-1/14
9.1
25.5
10.1
25.5
45

---

G
Broader Impact
G.1
Computational costs
The training time of the 4M models used in the transfer and generative results depends on their size: 4M-B was trained in 1.5 days on 64 A100 GPUs, 4M-L was trained in days on 128 A100 GPUs, and training 4M-XL XL took 8 days on 128 A100 GPUs.
The reference model used in the ablations be trained in 12 hours 32 A A100 GPUs, 2 days 8 A100 GPUs. Transferring a pre-trained model to the 35 benchmark tasks used for the ablations takes approximately 3 days on 8 A100 GPUs.
G.2 Social impact
We developed framework for training be conveniently re-purposed for various tasks practitioner may be interested in. We also committed to open-sourcing our code and models. These actions support the public with the democratization of the tools and the possibility of transparent inspection and safeguarding. While particularly poised for negative use compared to the alternatives, it should be noted that powerful generative models a general tool and have the potential be used ways authors did not intent. In addition, the data they are trained on may incorporate various societal biases or contain samples gathered in different ways from the internet. We trained our models on CC12M [19] which is an open-sourced dataset and has been curated to some degree (e.g., people's names are redacted), yet, due to the imperfections in this process, we still advise caution when using the models for generative purposes.
foundation models that, as demonstrated, can
model is not
46


Reviews:
[{"review_id": "CvR7zpDPWP", "reviewer": "Reviewer_RJ3x", "summary": "This paper proposes a multimodal pre-training framework named 4M, which employs the masked data modeling style to train a transformer encoder-decoder archtecture that is capable of performing different downstream tasks. Experiments show that 4M delivers competitive transfer ability on these tasks compared with MAE / DEiT III / BEiT v2.", "questions": "1. In the Table.2 of ablation studies, why use loss instead of corresponding task metrics as the mesure?  \n2. Could the authors provide some quantative results on the generative capabilities of 4M?", "limitations": "The model size and the data size could be further scaled up. The authors have discussed some limitations in their paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The baseline settings in the experiments are fair and sound, especially the self-baselines to control other variables.\n2. The ablation studies provide insightful discussions on the design choices of the pre-training strategy.", "weaknesses": "1. The multi-modal and multi-task training of 4M needs datasets with all required modalities and labels. However, this kind of well-annotated dataset is hard to obtain and not scalable. This research employs pseudo labeling to extend existing image-text datasets such as CC12M. Therefore, the performance of the off-the-shelf labelers are important. The authors should provide more detailed and careful discussions and ablations on that.  \n2. The downstream tasks are limited, especially considering the target of this paper, i.e. ``massively pre-training''. Quantative results on more diverse tasks / datasets should be examined, especially on the transfer ability to novel tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer RJ3x for their positive feedback. We address the main weaknesses, questions, and limitations in the following:\n\n> The multi-modal and multi-task training of 4M needs datasets with all required modalities and labels. However, this kind of well-annotated dataset is hard to obtain and not scalable. This research employs pseudo labeling to extend existing image-text datasets such as CC12M. Therefore, the performance of the off-the-shelf labelers are important. The authors should provide more detailed and careful discussions and ablations on that.\n\n- We generally agree with this summary. However, **Pseudo labeling is inherently more scalable than other approaches** due to the high availability of RGB images and off-the-shelf models. Combining datasets with incomplete annotations (like UnifiedIO [1] does) is limited by the relatively smaller number of those datasets, and the end result is not an aligned large dataset -- however, our method is not incompatible with that approach (even for the fine-tuning phase), and it'd be an interesting experiment to run. Existing annotated multi-task datasets like Taskonomy [2], Omnidata [3], Hypersim [4], etc, are also too limited in terms of their domain. Overall, Pseudo labeling is a good enabling strategy at the moment, and we acknowledge the value of training on incomplete data for the future. We will update the camera-ready with a discussion. \n- Performance of the off-the-shelf labelers: As shown in Tab. 1 in the rebuttal `PDF`, **the larger the 4M model, the more it approaches the performance of the original pseudo labeler, and may even slightly surpass it**. Indeed, we would expect higher-quality pseudo labels to translate well to better zero-shot performance. In addition, we use pseudo labeling to enable massively multi-modal pre-training, but it is conceivable that a fine-tuning stage with a much smaller dataset of high-quality data can significantly improve zero-shot performance. This practice of pre-training on large-scale, noisy and uncurated data and then tuning the model on a clean dataset or using reinforcement learning has worked well in the field of natural language processing (e.g., ChatGPT, or GPT-4), and we expect a similar approach to work well for multi-modal foundation models too.\n\n> The downstream tasks are limited, especially considering the target of this paper, i.e. ``massively pre-training''. Quantitative results on more diverse tasks / datasets should be examined, especially on the transfer ability to novel tasks.\n\nWe agree that the transfer learning study could benefit from a more diverse set of novel downstream tasks. That said, our ablation that contains **data- and compute-controlled baselines of MAE (RGB->RGB) and BEiT v2 (RGB->CLIP) style models** contains an extensive set of both **novel downstream tasks and datasets, spanning 35 tasks over 4 different datasets**. Many of the transfers contain input modalities and target tasks that were **not seen during 4M pre-training**. 4M pre-trained on all modalities performs **better than the baselines** on most of these tasks.\n\n> In Table.2 of ablation studies, why use loss instead of corresponding task metrics as the measure?\n\nThe aim of Tab. 2 is to show **how well certain instantiations of 4M transfer to arbitrary new distributions of tokens**. Since downstream performance comes down to A) how well the tokens are able to represent the downstream tasks (i.e. tokenizer reconstruction error), and B) how well 4M is able to predict these tokens, we abstract away A for this ablation since the tokenizers are the same for all settings, and only report B. In addition, reporting the cross-entropy losses, or equivalently log-perplexity, is a standard practice in NLP, and aligns well with the aim of this ablation and with the way 4M is pre-trained and fine-tuned. Reporting the cross-entropy loss makes **comparison more uniform** and avoids having to scale and average wildly different task-specific metrics such as mAP, mIoU, MSE, etc. that are not comparable with each other.\n\n> Could the authors provide some quantitative results on the generative capabilities of 4M?\n\nFor a quantitative analysis of 4M’s generative capabilities, please see the section 2 of the common response and the `PDF`.\n\n> The model size and the data size could be further scaled up. The authors have discussed some limitations in their paper.\n\nScaling the model and dataset size beyond 4M-XL is out of the scope for this paper, but we agree that 4M could benefit from exploration in that area. Our scaling trends (Fig. 5 in the main paper) and evidence from token-based text-to-image models like Muse [5] and Parti [6] paint a promising picture and suggest scaling the data and model size to be an exciting future direction.\n\n[1] Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks, Lu et al., 2022\n\n[2] Taskonomy: Disentangling Task Transfer Learning, Zamir et al., 2022\n\n[3] Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans, Eftekhar et al., 2021\n\n[4] Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding, Roberts et al., 2020\n\n[5] Muse: Text-To-Image Generation via Masked Generative Transformers, Chang et al., 2023\n\n[6] Scaling Autoregressive Models for Content-Rich Text-to-Image Generation, Yu et al., 2022"}, {"review_id": "ysdLHtBetw", "reviewer": "Reviewer_f31a", "summary": "The paper presents a unified transformer model by using an effective multi-modal pre-training scheme. The authors propose to perform masked modeling across different modalities. This is made possible by unifying the representation space of the considered modalities by mapping them into discrete tokens and then performing multi-modal masked modeling on a small subset of tokens. Experimental results demonstrate several promising results.", "questions": "N/A", "limitations": "The authors discussed some of the limitations, but this is more like descriptions of future work. No method is developed in address the limitations.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- This paper is technically valid and interesting. By conditioning on arbitrary modalities, the model can have great potential for a variety of multimodal intelligence capabilities.\n- The authors present comprehensive experiments and ablations, providing insightful discussions. The paper can be a good reference for future researchers.\n- The paper is well-written and easy to follow.", "weaknesses": "- The multi-modal masking strategy is highly similar to prior works, like MultiVAE. The mask-modeling part of this paper is somewhat less interesting and less innovative. The innovation is more in the developed system framework. \n- I don't find other significant concerns in the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer f31a for their positive feedback. We address the main concerns and questions in following response:\n\n> The multi-modal masking strategy is highly similar to prior works, like MultiVAE. The mask-modeling part of this paper is somewhat less interesting and less innovative. The innovation is more in the developed system framework.\n\nWhile 4M’s multi-modal masking strategy is inspired by MultiMAE, 4M proposes several important changes that enable further scaling of multi-modal models beyond three image-like modalities:\n\n- MultiMAE’s masking strategy assumes that all modalities are image-like, in their case RGB, depth and semantic segmentation maps. Applying the same masking strategy to sequence modalities such as captions or bounding boxes would not allow 4M to generate these modalities at inference time. We therefore propose to **use span-masking [1] to both benefit from masked pre-training and enable generation of sequence modalities**.\n- MultiMAE requires separate decoders for each target modality, and **always decodes all input and all mask tokens**. The overhead of this is manageable with three modalities and shallow decoders, but may not scale to much larger number of modalities. We propose target masking as a means to overcome this. For example, if we trained 4M without target masking, the number of tokens to decode would be around 1000, which would incur a significant compute overhead (see rebuttal `PDF` Fig. 2). In addition, our ablations in Appendix Tab. 17 show that **lower target masking budgets are more compute efficient** (for a fixed number of total training tokens).\n- In addition, we propose **mixtures of masking strategies** to train 4M on a diverse set of inputs and targets (see Appendix Tab. 18), with the resulting models striking a compromise between either masking scheme.\n\n> The authors discussed some of the limitations, but this is more like descriptions of future work. No method is developed in address the limitations.\n\nIn the Conclusion and Limitations section of the main paper **we discuss several limitations** (limited number of modalities, tokenizer quality, dataset size and quality) and **propose potential solutions** to address them. In addition, we list here several other limitations and possible ways to overcome them:\n\n- **Text understanding**: Recent state of the art text-to-image models like Imagen [2], Parti [3], Muse [4], or Stable Diffusion 2.1 [5] commonly train on powerful text encoders (e.g. T5-XXL or OpenCLIP-ViT/H) instead of classical text tokenizers to significantly improve their image generation fidelity and text understanding. 4M is trained directly on the text tokenizer, but we expect similar text-to-image improvements when training on LLM embeddings instead.\n- **Fine-grained editing**: Since 4M operates on discrete sets of tokens, in-painting is roughly constrained to the relatively coarse grid of tokens. Since every token affects an area slightly larger than its relative size on the image, the area that needs to be selected to remove a certain object is even larger. In addition, tokenization is a lossy process and destroys fine-grained details. Unlike diffusion models, we are not able to perform pixel-level edits. To remedy this, we can consider in-painting at a higher resolution (with a finer grid of tokens) or training specialized in-painting adapters.\n- **Alignment to downstream objectives**: 4M is trained by simply minimizing the cross-entropy loss between predicted and ground truth tokens. If this objective is misaligned with a certain downstream objective (e.g. aesthetics or a downstream task metric), we are not optimizing what we care for directly. To address this, we can consider fine-tuning the pre-trained 4M model using reinforcement learning on downstream objectives that are otherwise hard to optimize for directly. [6]\n- **Flexible image resolution**: As most other models, 4M is trained on square images of a fixed resolution — in our case 224x224. We trained a token super-resolution model that can map 4M outputs to 448x448, but to work out of the box at different resolutions and aspect ratios, we can consider approaches similar to FlexiViT [7] or NaViT [8]. This can be done as a fine-tuning step after training the model at the base resolution.\n\n[1] Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, Raffel et al., 2019\n\n[2] Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding, Saharia et al., 2022\n\n[3] Scaling Autoregressive Models for Content-Rich Text-to-Image Generation, Yu et al., 2022\n\n[4] Muse: Text-To-Image Generation via Masked Generative Transformers, Chang et al., 2023\n\n[5] https://huggingface.co/stabilityai/stable-diffusion-2-1-base\n\n[6] Tuning computer vision models with task rewards, Pinto et al., 2023\n\n[7] FlexiViT: One Model for All Patch Sizes, Beyer et al., 2022\n\n[8] Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution, Dehghani et al., 2023"}, {"review_id": "zeaJ7mlpQv", "reviewer": "Reviewer_qLD7", "summary": "The paper presents a foundation model for a variety of vision tasks. The authors show it can perform many key vision tasks out of the box and can also be fine-tuned to achieve highly competitive performance on unseen downstream tasks and input modalities. To handle the variety of modalities, the inputs/outputs are encoded into sequences of discrete tokens, and the model is trained on all the tasks simultaneously via a multi-modal masked modeling objective.", "questions": "One suggestion is to mention in the related work other papers that deal differently with multimodal inputs/outputs for solving various vision tasks altogether. This line of work includes:\n\n- Wang et al., \"Images Speak in Images: A Generalist Painter for In-Context Visual Learning\", CVPR'23\n\n- Bar et al. \"Visual prompting via image inpainting\", NeurIPS'22\n\n\nOne question that I had (and I am not sure how to evaluate) - what is more helpful for downstream performance - inputting during training tokens that correspond to the same image position but from different domains, or using more tokens from the same domain but from different locations in the image?", "limitations": "The limitations of the paper are discussed and addressed in the last section.", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper presents strong results and a scalable method to perform a variety of vision tasks. The ablation study covers almost all the aspects of the model. The results indeed prove the superiority of multimodal training over the baselines, without any need for augmentations. The paper is well-written and paves the way for a variety of research questions about the interactions between different modalities. ", "weaknesses": "While to tokenization method allows the model to train on a variety of tasks with a single architecture and cross-entropy loss, it also introduces quantization of the space of inputs/outputs. While this quantization does not harm the results for text, it might decrease the quality of the results for other domains (the segmentation boundaries might not be fully aligned with the objects for example). This idea induces an upper bound on the performance of such an algorithm and should be discussed. One way to evaluate this upper bound is by encoding and decoding back the ground-truth results of different domains (e.g. - quantizing the ground-truth segmentation masks and decoding them back), to verify the reconstruction quality and the downstream task-specific performance.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer qLD7 for their positive feedback. We address the main concerns and questions in following response:\n\n> While this quantization does not harm the results for text, it might decrease the quality of the results for other domains (the segmentation boundaries might not be fully aligned with the objects for example). This idea induces an upper bound on the performance of such an algorithm and should be discussed. One way to evaluate this upper bound is by encoding and decoding back the ground-truth results of different domains (e.g. - quantizing the ground-truth segmentation masks and decoding them back), to verify the reconstruction quality and the downstream task-specific performance.\n\nThank you for the suggestion. We **measure the reconstruction quality of the surface normal, depth, and semantic segmentation tokenizers** on 5000 CC12M validation images at a resolution of 224x224. Note that we cannot evaluate the performance of the tokenizers on datasets like DIODE or OASIS due to the presence of masks/holes in the dense labels. Since the tokenizers were trained on pseudo labeled data, they do not handle such masks. This is, however, not an inherent limitation, as the tokenizers could be trained with masked inputs, which we did for the Taskonomy and Hypersim tokenizers in the ablations.\nFig. 1 in the rebuttal `PDF` shows qualitative examples and Tab. 1 (last row) shows reconstruction metrics. While measuring the reconstruction on CC12M is not fully comparable to the DIODE and COCO zero-shot performance of 4M and baselines shown in Tab. 1, we note that **the tokenizer reconstruction errors are of a significantly lower magnitude than the prediction errors from RGB**. This indicates that the tokenization is not a strong bottleneck when considering the difficulty of predicting these tasks from RGB. That said, tokenization may remove or change fine details present in images, which may not be as clear in these metrics but can be visually apparent (see Fig. 1). A remedy to this could be to perform tokenization on higher resolution images or to decrease the patch size. We also generally expect future advances in tokenizer training to translate directly to zero-shot and downstream performance improvements.\n\n> […] mention in the related work other papers that deal differently with multimodal inputs/outputs for solving various vision tasks altogether.\n\nThank you for the suggestions, we’ll make sure to include a discussion of works that unify different vision tasks in this way in the camera ready version upon acceptance.\n\n> […] what is more helpful for downstream performance - inputting during training tokens that correspond to the same image position but from different domains, or using more tokens from the same domain but from different locations in the image?\n\nThis is indeed an interesting question, and our ablation of the input and target mask sampling parameters $\\alpha$ (see Appendix Tab. 15) provides a partial answer. Setting the input and target alphas to a very low value (e.g. 0.1) corresponds to sampling the number of tokens per modality from a “spiky” Dirichlet distribution, i.e. most of the time tokens are sampled from single modalities. This is close to the latter case mentioned, i.e. using more tokens from the same domain but from different locations in the image and predicting across domains. **This setting performs slightly worse on downstream transfers compared to more random mask sampling approaches (alphas >= 0.2), or mixtures of sampling strategies (see Appendix Tab. 18).**\nNote that on the other hand, the use of higher alphas (> 1.0) does not correspond to the former case (sampling tokens that correspond to the same image position but from different domains). Restricting the sampling in that way has been studied in masked video pre-training, notably spatio-temporal MAEs [1] and VideoMAE [2], and it has been observed that more random masking strategies perform similar or better. Extending this analysis to the multi-modal case is interesting future work.\n\n[1] Masked Autoencoders As Spatiotemporal Learners, Feichtenhofer et al., 2022\n\n[2] VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training, Tong et al., 2022"}, {"review_id": "IaXr58F2uJ", "reviewer": "Reviewer_QEFU", "summary": "The paper proposes a multi-modal masked modeling pre-training scheme (4M) that unifies a several modalities – including text, images, geometric, and semantic modalities, neural network feature maps. The tokenization and masked modeling enable the efficient pretraining of 4M. The pretrained model can 1) achieve reasonable finetuned performance on vision tasks, 2) achieve conditional generation under different modalities.", "questions": "1.\tClarification of the novelty of this paper.\n2.\tThe performance comparison and experimental results to support the claims in this paper.", "limitations": "The limitation of this work is mainly on the lacking novelty and the experimental results cannot well support the claims.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "This work is a good practice on multi-modal masked modeling pre-training and achieve reasonable performance on both finetuned downstream tasks and generative tasks.", "weaknesses": "1.\tThis work is a combinational work of existing methods and lacks technical novelty.\n-\tThe multi-modal masked modeling has been utilized by various existing works. E.g. MultiMAE has proven the feasibility of multi-modal masked modeling. Vision-NLP multi-modal is also explored by exsting works,e.g. MAGVLT.\n-\tThe tokenization is claimed to be a key part for the efficient pretraining of 4M, while the tokenization has been widely used existing works, and the tokenizers used in this work are mostly borrow from other works.\n-\tThe major contribution seems to be the the multi-modal pretrained data based on the CC12M dataset. But this is no novelty on the pseudo-labeling of the data.\n2.\tThe performance is not impressive given the large-scale data used in this work.\n-\tThe paper claims 4M can perform a diverse set of vision tasks out of the box, but no experimental results are showed in the manuscript.\n-\tIn Tab.1, the finetuned performance is relatively weak compared other pretrained methods. In addition, many stronger pretrained methods are not included in this paper.\n-\tThe conditional generation results are interesting. While there is no quantitative performance comparison with other methods such as contorlnet. Also, one drawback of 4M based conditional generation is that the model accepts fixed modal after pertraining, while controlnet can be easily extended to new modal.\n-\tThe paper claims the pretraining efficiency is an advantage, yet no experimental result is given to prove this point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer QEFU for their feedback. We address the main concerns and questions in following response:\n\n> This work is a combinational work of existing methods and lacks technical novelty. \nThe multi-modal masked modeling has been utilized by various existing works. E.g. MultiMAE has proven the feasibility of multi-modal masked modeling. Vision-NLP multi-modal is also explored by exsting works,e.g. MAGVLT. \nThe tokenization is claimed to be a key part for the efficient pretraining of 4M, while the tokenization has been widely used existing works, and the tokenizers used in this work are mostly borrow from other works.\n> \nFirst, we want to clarify that the simplicity of our method is a highly desirable property given its competitiveness and novel capabilities. Nevertheless, **our approach includes several technical innovations**, which we believe distinguish 4M from existing methods in meaningful ways.\n- **Masking:** While 4M's multi-modal masking strategy is inspired by MultiMAE, we introduce several key changes that are crucial for scaling our models beyond the three image-like modalities of MultiMAE. For a comprehensive overview of these changes, please see our response to reviewer f31a. Additionally, please note that MAGVLT, although relevant, focuses only on text-image pairs and was published within two months of the submission deadline, making it concurrent work.\n- **Tokenization:** The novelty lies in 4M's ability to work with multiple modality-specific tokenizers. Unlike methods like Unified-IO, which operates on a single RGB image tokenizer, our approach enables scaling to modalities beyond those that can be represented as images, including neural network feature maps. This key distinction is not about the tokenizers themselves but how 4M leverages them to jointly operate on diverse modalities.\n- **Architecture:** 4M's architecture was intentionally designed to be as close as possible to a standard Transformer encoder-decoder to take advantage of their scalability and flexibility. However, we also had to include some crucial modifications to enable joint modeling of both image-like and sequence-like modalities within a single encoder-decoder architecture, as described in Section 2.2 of our paper.\n- **Importance of combinational work:** It's worth noting that the act of bringing together these specific methods in itself introduces novelty. **Our combination of techniques leads to new capabilities and improved results that wouldn't be possible otherwise**. For example, unlike MultiMAE which can’t be used for generative tasks due to lack of tokenization, 4M can function as a generative model while also showing much better transfer performance.\n\n> The paper claims 4M can perform a diverse set of vision tasks out of the box, but no experimental results are showed in the manuscript\n> \nPlease see Table 2 of the rebuttal `PDF` where we show the out of the box (zero-shot) performance of 4M on surface normals, depth, and semantic segmentation on the DIODE and COCO datasets. On this data, **4M matches or even surpasses the pseudo labeler networks and other strong baselines**.\n\n> In Tab.1, the finetuned performance is relatively weak compared other pretrained methods. In addition, many stronger pretrained methods are not included in this paper.\n> \nIn our transfer study (Tab. 1 of our paper), **the pretraining methods used as comparison are recognized as strong models**. For example, MAE serves as the backbone for task-specific foundation models such as ViTDet [1] and SAM [2].  Note that 4M outperforms all reported baselines including MAE on all tasks except ImageNet classification. \n\nFurthermore, in Tab. 3 of the rebuttal `PDF`, we provide a comparison with DINOv2-Base, one of the strongest publicly available ViT-B models. However, please note that these models are not directly comparable as DINOv2:\n\n1. uses an **order of magnitude more training data.** Furthermore, the data was curated to be similar to evaluation datasets, including those that used in our downstream tasks (IN1K, ADE20k, NYUv2). Thus, there is less of a distribution shift for DINOv2 when transferring with these datasets. \n\n2. requires **orders of magnitude more compute.** \n\n3. is distilled from a significantly larger model (DINOv2 ViT-g, 1.1B params), which gives a boost in performance compared to training from scratch.\n\nWhile DINOv2 is able to attain slightly better performance on downstream tasks, 4M-B is still able to approach DINOv2’s performance on several tasks despite notable differences in computational cost and dataset size.\n| Model | Dataset size | Compute cost (A100 hrs) |\n| --- | --- | --- |\n| 4M Base | 12M | 2300 |\n| DINOv2 Base Distilled (from ViT-g) | 142M | 22000 + 5300 |\n\n> The paper claims the pretraining efficiency is an advantage, yet no experimental result is given to prove this point.\n> \nThe 4M training scheme produces a model that can predict any task from any subset of full or partial modalities — all in a single and highly efficient pre-training run. Input and target masking are crucial to make this work efficiently. Already MAE has shown that dropping masked tokens at the input level can significantly improve pre-training efficiency. MultiMAE further demonstrated its importance for multi-modal training, but as the number of tasks increases, so does the computational cost of the decoders since all masked and non-masked patches of all modalities are always decoded. 4M addresses this issue by decoding only a random subset of the masked tokens (as ablated in Appendix Tab. 17), at no cost to downstream performance. \n\nFurthermore, in Fig. 2 of the rebuttal `PDF`, **we quantitatively demonstrate that target masking can significantly improve training efficiency**, especially when training on a large number of modalities or on modalities with a large sequence length.\n\n[1] Exploring Plain Vision Transformer Backbones for Object Detection, Li et al., 2022\n\n[2] Segment Anything, Kirillov et al., 2023"}, {"review_id": "EzlG3Rg8Zc", "reviewer": "Reviewer_q9W9", "summary": "The paper addresses the importance of a versatile model that is not limited to single modality and task and proposes a multi-modal pre-training scheme called 4M. 4M is a single encoder-decoder architecture trained on a large set of image and sequence-like modalities. The modalities including text, images, geometric and semantic are brought into a joint representation as tokens through modality-specific tokenizers. The training procedure relies on multi-modal masked training, with only a small set of tokens used as inputs and targets. Extensive experimentations showed that 4M can solve many common vision tasks out of the box, can be fine-tuned to unseen tasks as well as perform multi-modal controllable generation.", "questions": "The paper in its current form lacks discussion on the robustness of 4M to dataset quality as well as evaluation of its generation capabilities. Please refer to the “Weakness” section for details.", "limitations": "Yes, the authors discuss the limitations in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "This publication has several strengths including:\n\n1) The writing is very clear and easy to understand.\n\n2) The proposed approach is scalable across three key aspects - data (more training samples increase performance), architecture (improve performance with model size as well as remain stable) and training objective (handle growing number of modalities without incurring excessive computational costs). \n\n3) Good experimental methodology with carefully designed ablations that justifies architectural design decisions especially impacts of  input modalities and target tasks, multi-modal masking strategy as model and data scaling.\n\n4) Very exhaustive in-depth experimentation showcasing the key capabilities - zero-shot generalization to diverse set of vision tasks, fine-tune to unseen tasks, multi-modal controllable generation.", "weaknesses": "1) The paper seemingly lacks any quantitative evaluation of its generation capabilities or comparison with existing state-of-the-art methods. I'd appreciate it if the authors can elaborate on this.\n\n2) Another concern is that the paper lacks any discussion on the robustness of the proposed approach to the quality of the datasets, since low-quality data is usually readily available compared to high-quality data. This will be critical for data scaling as well as will align with model scaling to 4M-XL and beyond.\n\n3) Minor comment: I am curious if the authors have performed any out-of-distribution analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer q9W9 for their positive feedback. We address the main concerns and questions in following response:\n\n> The paper seemingly lacks any quantitative evaluation of its generation capabilities or comparison with existing state-of-the-art methods. I'd appreciate it if the authors can elaborate on this.\n\nFor a quantitative analysis of 4M’s generative capabilities, please see the section 2 of the common response and the `PDF`.\n\n> Another concern is that the paper lacks any discussion on the robustness of the proposed approach to the quality of the datasets, since low-quality data is usually readily available compared to high-quality data. This will be critical for data scaling as well as will align with model scaling to 4M-XL and beyond.\n\nAssessing the influence that the “quality” of a dataset has on a pre-training strategy is a **highly interesting and important question**. Works like DataComp [1] aim to achieve this by developing filtering strategies for multi-modal datasets, but it is an **open question and research direction of its own**.\nTo ablate the influence of the choice of pre-training dataset to some extent, **we pseudo labeled ImageNet-21K, as well as a 15M subset of COYO-700M** [2], and trained 4M-B models on each. Tab. 3 in the rebuttal `PDF` shows that the 4M-B models trained on these datasets achieve a **similar performance to the CC12M version, while still surpassing previously reported baselines**.\nTo add to that, and related to reviewer RJ3x’s question on pseudo labeling quality, we argue that **the use of pseudo labeling is a strength of the approach as it is inherently more scalable than using high-quality off-the-shelf datasets**.\nSimilar to the common practice in NLP of pre-training on large uncurated datasets and tuning the model with higher-quality data, it is conceivable that similar approaches can work well for 4M too.\n\n> Minor comment: I am curious if the authors have performed any out-of-distribution analysis.\n\nPlease see Tab. 4 in the rebuttal `PDF` for an OOD analysis of several ImageNet-1k transfers to IN-A, IN-R, IN-S, IN-C, and IN-3DCC. **4M shows strong robustness to various OOD domains and corruptions, and is competitive with DeiT III**, which is a specialist model.\nFurther, we show the **strong zero-shot performance of 4M** on surface normal, depth, and semantic segmentation in Tab. 1 in the rebuttal `PDF`. The performance is evaluated on the DIODE and COCO datasets which are not part of the 4M training dataset. **We note that on this OOD data, 4M matches or even surpasses the pseudo labeler networks and other strong baselines.**\n\n[1] DataComp: In search of the next generation of multimodal datasets, Gadre et al., 2023\n\n[2] https://github.com/kakaobrain/coyo-dataset"}]

Author rebuttal:
# Response to all reviewers

We thank the reviewers for their insightful comments and are glad to hear that the reviewers found the paper to be **“technically valid and interesting”** with **“great potential for a variety of multimodal intelligence capabilities”** (f31a), provide **“insightful discussions on the design choices of the pre-training strategy”** (RJ3x), and appreciate the **“very exhaustive in-depth experimentation showcasing the key capabilities”** (q9W9). We are also glad that the reviewers recognized the **“strong results and a scalable method to perform a variety of vision tasks”** (qLD7) and commended our writing as being **“very clear and easy to understand / follow”** (q9W9, f31a) / **"well-written and paves the way for a variety of research questions about the interactions between different modalities"** (qLD7).



## 1. Additional results overview
We address the reviewers’ remaining questions and concerns in the individual responses and rebuttal `PDF`. We discuss general questions on the generative capabilities below the following list of new experiments and major addressed questions:
- QEFU: Out of the box (zero-shot) performance evaluation (`PDF` Tab. 1)
- qLD7: Tokenizer reconstruction quality (`PDF` Tab. 1 and Fig. 1)
- RJ3x, QEFU, q9W9: Quantitative evaluation of generative capabilities (`PDF` Tab. 2)
- QEFU, q9W9: Comparison to strong baselines and robustness to dataset quality (`PDF` Tab. 3)
- q9W9: OOD analysis (`PDF` Tab. 4)
- QEFU: Pre-training efficiency (`PDF` Fig. 2)

## 2. Common questions

> RJ3x, QEFU, q9W9: General quantitative evaluation of generative capabilities

Please see Tab. 2 in the rebuttal `PDF` for a quantitative comparison of 4M across model sizes, a controlled text-to-image baseline, as well as Stable Diffusion 2.1. The metrics shown are computed on 30k subsets of CC12M and COCO validation sets, and we interpolate all generated images to 256x256.
To perform a controlled comparison, we train a pure text-to-image variant of 4M-B, in spirit similar to Muse [1], for a total of 300B tokens on CC12M, and using the same RGB tokenizer as used for 4M. **4M trained on all modalities achieves comparable FID and CLIP scores to this specialist model, and at the same time can be conditioned on any pre-training modality and can solve several common vision tasks out of the box.**
We also compare against the 512x512 base model of Stable Diffusion 2.1 (SD-2.1) [2] and observe a considerable gap to SotA generative models on OOD data. We note here, however, that **SD-2.1 was trained on datasets two orders of magnitude and a compute budget one order of magnitude larger than what was used to train 4M-XL**. 4M is a general multi-modal pre-training strategy and considering the scaling curves of similar token-based text-to-image models like Muse or MAGE, we expect 4M’s generation quality to significantly improve given a similar data and compute regime, and better image tokenizers.

> QEFU: Also, one drawback of 4M based conditional generation is that the model accepts fixed modal after pertraining, while controlnet can be easily extended to new modal.

Adapting multi-modal Transformers like 4M to new modalities / tasks is an exciting future research direction. Parameter-efficient fine-tuning techniques like Low-Rank Adaptation (LoRA) have been shown to work well on LLMs and diffusion models, and we would expect similar techniques to allow 4M to be efficiently adapted to additional modalities. Since our pre-training dataset is pseudo labeled, this would require only a dataset of the new modality and RGB images, which would also be the case for training a new ControlNet.

[1] Muse: Text-To-Image Generation via Masked Generative Transformers, Chang et al., 2023

[2] https://huggingface.co/stabilityai/stable-diffusion-2-1-base

Meta-review: This paper presents 4M: A single unified Transformer encoder-decoder trained using masked modeling objective across a wide range of modalities including text, images, geometric, and semantic modalities. The paper received nearly unanimous accept reviews, including 1x Weak accept, 2x accept, and 1x very strong accept. The reviewers appreciate the well written paper with extensive analysis and experimentation, insightful discussions, and an interesting and scalable approach. ACs concur with the reviewer consensus.
Decision: Accept (spotlight)