4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 19:57:05

Abstract: Current multimodal and multitask foundation models, like 4M or UnifiedIO, show promising results. However, their out-of-the-box abilities to accept diverse inputs and perform diverse tasks are limited by the (usually small) number of modalities and tasks they are trained on. In this paper, we develop a single any-to-any model trained on tens of highly diverse modalities and by performing co-training on large-scale multimodal datasets and text corpora. This includes training on images and text along with several semantic and geometric modalities, feature maps from recent state of the art models like DINOv2 and ImageBind, pseudo labels of specialist models like SAM and 4DHumans, and a range of new modalities that allow for novel ways to interact with the model and steer the generation, for example, image metadata or color palettes.

A crucial step in this process is performing discrete tokenization on various modalities, whether they are image-like, neural network feature maps, vectors, structured data like instance segmentation or human poses, or data that can be represented as text.
    
Through this, we show the possibility of training one model to solve at least 3x more tasks/modalities than existing models and doing so without a loss in performance. In addition, this enables more fine-grained and controllable multimodal generation capabilities and allows studying the distillation of models trained on diverse data and objectives into one unified model.
We scale the training to a three billion parameter and different datasets. The multimodal models and training code are open sourced at https://4m.epfl.ch/.

Main paper: 
 introduction: 
 Having a single neural network to handle a wide and varied range of tasks and modalities has been a longstanding goal. Such a model, especially when capable of any-to-any predictions, brings notable advantages, such as test-time computational efficiency, model size, and enabling modality fusion. However, multitask learning has commonly faced significant challenges. For example, the training often suffers from negative transfer, leads to reduction in performance compared to single-task models, and typically requires careful strategies for balancing losses or gradients [48, 99, 86, 100, 33] . Moreover, training a single network on tasks and modalities that vary greatly in terms of dimensionality, data type, and value ranges presents additional complexities † . Recent notable efforts in the space of multimodal and multitask training, such as Pix2Seq [17, 18] , OFA [93] , 4M [65] , or Unified-IO [61, 60] have made significant strides in unifying the representation space for conceptually different inputs and targets. A large part of their success can be attributed to transforming different modalities into a common representation, namely sequences of discrete tokens, and training relatively standard Transformer architectures on them. While these works show promising results, they are typically trained on a small set of modalities. This raises the question if increasing the set of tasks/modalities the models can solve will lead to a degradation of performance.We build upon the multimodal masking pre-training scheme [65] and increase its capabilities by training on tens of highly diverse modalities. Concretely, we add SAM segments [49] , 3D human poses and shapes from 4DHumans [37] , canny edges extracted from RGB and SAM instances, color palettes, multiple types of image, semantic and geometric metadata, as well as T5-XXL [71] text embeddings, in addition to 7 more common modalities. On top of that, we include dense feature maps of the recent state of the art models DINOv2 [68] and ImageBind [35] , as well as their global embedding vectors to enable multimodal retrieval abilities. Please see fig. 1 for an overview.We are able to train a single unified model on diverse modalities by encoding them with modalityspecific discrete tokenizers (see fig. 3 ). For image-like modalities, e.g. RGB or edges, we train ViT-based [25] VQ-VAE [66] tokenizers to map the inputs into a small grid of discrete tokens. For modalities like 3D human poses or image embeddings, we train MLP-based discrete VAEs to compress them into a small set of discrete tokens. All other modalities that can be mapped to a text representation, such as captions or metadata, are encoded using a WordPiece tokenizer [24] .The resulting model demonstrates the possibility of training a single model on a large number of diverse modalities/tasks without any degradation in performance and significantly expands the out-ofthe-box capabilities compared to existing models. Adding all these modalities enables new potential for multimodal interaction, such as retrieval from and across multiple modalities, or highly steerable generation of any of the training modalities, all by a single model.In short, we expand the capabilities of existing models across several key axes:• Modalities: Increase from 7 in the existing best any-to-any models [65] to 21 diverse modalities, enabling new capabilities like cross-modal retrieval, controllable generation, and strong out-of-the-box performance. This is one of the first times in the vision community that a single model can solve tens of diverse tasks in an any-to-any manner (see fig. 2 ), without sacrificing performance and especially do so without any of the conventional multitask learning difficulties [77, 48, 99, 86, 100, 33] .• Diversity: Add support for more structured data, such as human poses, SAM instances, metadata, and color palettes for controllable generation.• Tokenization: Investigate discrete tokenization of diverse modalities such as global image embeddings, human poses, and semantic instances using modality-specific approaches.• Scale: Scale the model size to 3B parameters and dataset to 0.5B samples using [12] .• Co-Training: Demonstrate co-training on vision and language modeling simultaneously. † Modality vs task: "Modalities" usually denote the inputs to a model (e.g. sensory signals), and "tasks" usually denote the outputs (e.g. semantics). The adopted architecture in multimodal masked modeling enables a symmetric input-output structure, thus modalities and tasks are used interchangeably in this paper. benefit from chained generation [65] . Notice the high consistency among the predictions of all modalities for one input. Each row starts from a different modality coming from the same scene. Highlighted in green are new input/output pairs that 4M [65] cannot predict nor accept as input. Note that, while this figure shows predictions from a single input, 4M-21 can generate any modality from any subset of all modalities.On the other hand, using RGB pixels as input is more suitable for visual perception tasks. By avoiding the discrete bottleneck, there is no information loss during the tokenization step, and the projection layer can be more lightweight. Given these tradeoffs, we follow 4M by training on both and treating them as separate modalities, with RGB pixels as an input-only modality. We also extracted color palettes from RGB images using PyPalette [2] , at varying number of colors. This enables us to perform conditional generation using desired colors for better artistic control.Geometric modalities: These contain surface normals, depth, and 3D human poses & shape which provide important information about the scene geometry. For the first two, we used Omnidata models from [27, 46] for pseudo labeling due to their strong generalization performance. For 3D human poses and shape, we leverage a recent state-of-the-art model, 4D-Humans [37] .Semantic modalities: We include semantic segmentation and bounding boxes to capture the scene semantics and leverage Mask2Former [20] and ViTDet [56] models for pseudo labeling. Next to these, we also incorporated pseudo labels extracted from Segment Anything Model [49] (SAM) as SAM instances for its strong object representation.Edges: As recent generative methods such as ControlNet [104] showed, edges carry important information about the scene layout and semantics that are also useful for conditioning, abstraction, and sketching. We consider two types of edges, specifically Canny edges and SAM edges. The former is extracted from the RGB images with OpenCV [1] . As Canny edges may contain low-level information, e.g. shading edges, we also include edges extracted from SAM instances to get a more semantic boundary map. We tokenize Canny and SAM edges with a shared tokenizer.Feature maps: We extract embeddings from CLIP [70] , DINOv2 [68] and ImageBind [35] as they demonstrated strong transfer learning and retrieval capabilities. Previously, tokenized CLIP features were shown to be an effective target for masked image modelling [94, 65] that enables distilling a useful semantic representation of the scene. We follow a similar approach and tokenize the feature maps from pre-trained CLIP-B16, DINOv2-B14 and ImageBind-H14 models. We also included the global embeddings of DINOv2 and ImageBind models and tokenized them separately.We extract several useful pieces of information from the RGB images and other modalities, that can be categorized into semantic metadata, geometric metadata, and image processing metadata.For this, we use functionalities from Pillow [3] OpenCV [1] , and Omnidata [27] .The following semantic metadata are extracted from bounding boxes, poses, and segmentation maps:• Crowdedness score: number of humans (extracted from 4DHumans instances) These are aimed to capture the semantic regularities of the scene at a more holistic level as opposed to pixel-based representations.Similarly, geometric metadata captures the scene geometry more globally. They are extracted from surface normals and depth maps:• Geometric complexity: angular variance of surface normals • Occlusion score: % of occlusion edges over a fixed threshold Finally, image processing metadata contains several aspects of images such as original image height and width before cropping, which can be used as conditioning to generate higher quality images [69] , brightness, contrast, saturation, entropy, and colorfulness [39] . Similar to color palette, these help with encoding low-level image representations into the model and enable more steerable generation.Text: Large language models (LLMs) trained on large text corpora learn strong representations as shown by several works [24, 71, 89, 67] . We include captions from CC12M [16] and COYO700M [12] their format and performance. For image-like modalities and feature maps, we use spatial VQ-VAEs [66] with optional diffusion decoders for detail rich modalities like RGB. For non-spatial modalities like global tokens or parameterized poses, we compress them to a fixed number of discrete tokens using Memcodes [62] with MLP encoders and decoders. All sequence modalities are encoded as text using WordPiece [24] . The shown examples are real tokenizer reconstructions. Notice the low reconstruction error. See appendix D for more details and Fig. 13 for visualizations. datasets, as well as web text from C4 [71] for language modeling. Next, we employ both a standard WordPiece [24] tokenizer for captions as [65] as well as caption embeddings obtained from a T5-XXL [71] encoder to capture better text representations, which have been shown to improve text-to-image generation fidelity [80, 14] (See fig. 4 ). 
tokenization: 
 Tokenization consists of converting modalities and tasks into sequences or sets of discrete tokens, thereby unifying their representation space. This is critical for training large multimodal models as it confers the following key benefits: 1) It enables training multimodal and multitask models with a single pre-training objective. After tokenization, all tasks are formulated as a per-token classification problem using the cross-entropy loss. This improves training stability, enables full parameter sharing, and removes the need for task-specific heads, loss functions, and loss balancing. 2) It makes generative tasks more tractable by allowing the model to iteratively predict tokens, either autoregressively [72, 96] or through progressive unmasking [15, 14] . 3) It reduces computational complexity by compressing dense modalities like images into a sparse sequence of tokens. This decreases memory and compute requirements, which is crucial when scaling up to larger dataset and model sizes.We use different tokenization approaches to discretize modalities with different characteristics. See fig. 3 for an overview. To summarize, we mainly use three different types of tokenizers, as explained below. Please see appendices D and H for more details and insights on tokenizer design choices.ViT tokenizer (with optional diffusion decoder): We trained modality-specific ViT [25] based VQ-VAE [66] tokenizers for image-like modalities such as edges and feature maps. The resulting tokens form a small grid of size 14 × 14 or 16 × 16, according to the pseudo-labeler patch size. The edge tokenizers use a diffusion decoder [82, 65] to get visually more plausible reconstructions.MLP tokenizer: For human poses and global embeddings from DINOv2 and ImageBind, we use Bottleneck MLP [6] based discrete VAEs with Memcodes quantization [62] to tokenize them into a small number of tokens, e.g. 16.Text tokenizer: We leverage a WordPiece [24] tokenizer which is used to encode not only text, but also other modalities such as bounding boxes, color palettes and metadata using a shared set of special tokens to encode their type and values (See appendix D.6 for details).Datasets: We perform the training in two stages, namely a 4M pre-training stage on a significantly larger image dataset, followed by a fine-tuning phase on a smaller dataset containing a larger number of modalities. Since the 4M-XL model showed signs of overfitting on sequence modalities when trained on CC12M [16] , we re-trained the models on COYO700M [12] , containing 50 times more samples. COYO700M was pseudo labeled with the same modalities used for 4M. To cut down on pseudo labeling cost when expanding the number of modalities, we decided to pseudo label CC12M instead of COYO700M, and fine-tune the models with both new and old modalities. To avoid overfitting the larger models, we co-train them with samples from COYO700M. In addition to the previously mentioned multimodal datasets, we also included the C4 [71] text corpus in training. We perform the training by randomly sampling elements of each batch from any of these datasets, given a pre-determined set of sampling weights, and perform language modeling on them. Exact details on the training mixture are given in appendix E.2.Architecture: We adopt 4M's encoder-decoder based transformer architecture with additional modality embeddings to accommodate new modalities. Similar to 4M, besides RGB tokens, the encoder directly accepts RGB pixels with a learnable patch-wise projection to enable use as a ViT [25] backbone for transfer learning.Masking strategy: We used both multimodal random [7, 65] and span masking [71] strategies that mask input and target tokens. We invoke dataset mixing ratios and Dirichlet sampling parameters, α, to ensure stable training on multiple modalities and datasets, as detailed in appendix E.2. 
multimodal capabilities: 
 We demonstrate a broad range of capabilities unlocked by 4M-21, including steerable multimodal generation (Sec. 3.1), multimodal retrieval (Sec. 3.2) and strong out-of-the-box capabilities (Sec. 3.3). Please see the project website for more visualizations demonstrating these capabilities.4M-21 can predict any training modality by iteratively decoding tokens [65, 15, 14] . This is shown in fig. 2 where we can generate all modalities from a given input modality in a consistent manner. Furthermore, as we can generate any of the training modalities from any subset of other modalities, both conditionally and unconditionally, it enables several ways to perform fine-grained and multimodal generation, as shown in fig. 4 . This includes diverse capabilities such as performing multimodal edits, probing the learned representations, and steering multimodal data generation. embeddings (here shown for DINOv2) from a given input (of any modality) using 4M-21 and comparing the cosine distances between the query and retrieval set embeddings. Left: Retrieving RGB images from distinctly different query modalities (here RGB, segmentation map, edges, depth map, color palette, and caption). Middle: Retrieving any modality using any other modality as the query input. Each query modality constrains the retrievals differently, e.g. here the RGB image and caption queries always yield Neuschwanstein castle retrievals. In contrast, for depth and semantic queries, the scene is more ambiguous, thus they retrieve other buildings with similar characteristics. Right: We can also combine any subset of modalities to define the query input, e.g. surface normals and a color palette, to better control the retrieval. See appendix B.2 for more results.Moreover, 4M-21 exhibits improved text understanding capabilities leading to geometrically and semantically plausible generations, both when conditioning on T5-XXL embeddings and on regular captions (fig. 4 , top right). Please see appendix I.2 for additional results.Our model can also perform multimodal retrievals by predicting global embeddings of DINOv2 and ImageBind from any (subset) of the input modalities. Once the global embeddings are obtained, the retrieval is done by finding the retrieval set samples with the smallest cosine distance to the query [68, 35] . As shown in fig. 5 , this unlocks retrieval capabilities that were not possible with the original DINOv2 and ImageBind models such as retrieving RGB images or any other modality via using any other modality as the query. Furthermore, one can combine multiple modalities to predict the global embedding, resulting in better control over retrievals, as shown on the right. Please see appendix I.3 for additional results.4M-21 is capable of performing a range of common vision tasks out-of-the-box, as demonstrated visually in fig. 6 . In table 1 , we evaluate the performance on DIODE [90] surface normal and depth estimation, COCO [57] semantic and instance segmentation, 3DPW [91] 3D human pose estimation, and do ImageNet-1K [79] kNN retrieval using predicted DINOv2 global tokens. We compare against the pseudo labeling networks, strong baselines, and the 4M model from [65] trained on 7 modalities. For surface normal estimation and semantic segmentation, we observed that ensembling multiple predictions significantly improves performance, see appendix F for more details and results.Our model consistently achieves strong out-of-the-box performance, and often matches or even outperforms the pseudo labelers and other specialist baselines, while being a single model for all tasks. Notice the large performance gap with other multitask models like Unified-IO [61] and Unified-IO-2 [60] . For kNN retrieval, 4M-21 XL performance approaches the tokenizer bound, i.e. the retrieval performance using the DINOv2 tokenizer reconstructions. While the smaller models lag behind 4M models, we observe that 4M-21 XL is able to match the performance of 4M-7 XL, while being trained to solve three times more tasks. The trend over the model size needing to be larger is expected as the number of tasks increase. Table 1 : Out-of-the-box (zero-shot) performance. We show the performance for a common subset of tasks: surface normals and depth estimation on DIODE [90] , semantic and instance segmentation on COCO [57] , k-NN retrieval on ImageNet-1K [79] , and 3D human keypoint estimation on 3DPW [91] . We compare to a set of strong baselines and specialist models, including our pseudo labelers. The model learned to solve all the tasks without a loss of performance, is significantly better than the baselines, and is competitive with pseudo labelers, while being a single model for all tasks. Compared to 4M-7, the 4M-21 model preserved its performance while solving 3x more tasks. ✗ denotes that a given model cannot solve the task out-of-the-box. * shows the tokenizer reconstruction quality and provides an estimate on the performance upper bound due to tokenization. See fig.  
transfer experiments: 
 To study the scaling characteristics of pre-training any-to-any models on a larger set of modalities, we train models across three different sizes: B, L, and XL. We then transfer their encoders to downstream tasks and evaluate on both unimodal (RGB) and multimodal (RGB + Depth) settings. The decoders are discarded for all transfer experiments, and we instead train task-specific heads. We perform self-comparisons in a similar manner to [65, 7] , as well as comparing to a set of strong baselines.Unimodal transfers. For unimodal transfers we leverage the RGB patch embeddings learned during the pre-training, as RGB pixel inputs are used alongside the tokenized modalities. For the XL models and DINOv2 g, we perform parameter-efficient fine-tuning using LoRA [42] instead of full finetuning, which significantly improves results for XL models. We did not observe similar performance gains for the smaller models. Further training details are described in appendix G.We evaluate on ImageNet-1K classification [23, 79] , ADE20K semantic segmentation [106] , NYUv2 depth estimation [84] , and ARKitScenes [9] 3D object detection tasks. Some transfer tasks are completely unseen during pre-training, e.g. object classification or 3D object detection, while others are included as different instantiations, e.g. absolute depth instead of relative depth, or using ADE20K instead of COCO classes. We follow the best practices and commonly used settings from other papers [65] .The results are shown in table 2 . We make the following observations: 1) for the transfer tasks that are similar to the seven modalities of 4M, e.g. semantic segmentation or depth, 4M-21 does not lose performance due to being trained on many more modalities, 2) for novel transfer tasks like 3D object detection that are sufficiently different from 4M modalities, we observe an improved performance. Moreover, the performance improves with larger model sizes, showing promising scaling trends. These trends can be further seen in the multimodal transfer results, which we explain next. Multimodal transfers. We perform multimodal transfers on NYUv2, Hypersim [75] semantic segmentation, and 3D object detection on ARKitScenes. We compare transfers using RGB images only, and RGB pixels + tokenized sensory depth as inputs.As table 3 shows, 4M-21 makes strong use of optionally available depth inputs and significantly improves upon the baselines. 
related work: 
 Multitask learning in vision involves training a single model to perform multiple visual tasks efficiently [13, 78] . Earlier methods [28, 64, 50] combined multiple dense vision tasks into a single model but faced challenges scaling to a larger variety of tasks and modalities, limited by training instabilities and the need for careful task selection and loss balancing to reduce negative transfer [48, 102, 86, 100] .Recently, discrete tokenization has enabled a shift towards integrating numerous vision tasks into unified multimodal and multitask models such as Gato [74] , OFA [93] , Pix2Seq [17, 18] , Uni-fiedIO [61, 60] , 4M [65] , and more [51, 83, 5, 41, 26, 44, 4, 97, 87, 108, 54, 105] . These methods first transform various modalities and tasks into sequences or sets of discrete tokens [24, 52, 66, 31, 95] , and then train a single Transformer on these tokens using either a sequence modeling [74, 93, 61, 17, 18, 51, 83] or masked modeling objective [65, 94] . Some methods (e.g. Gato [74] , UnifiedIO [61, 60] ) perform co-training on multiple disjoint datasets and are capable of performing a wide range of tasks, but not jointly. In contrast, methods like 4M [65] train on a single aligned dataset through the use of pseudo labeling, enabling any-to-any modality prediction but on a typically more limited set of modalities. We significantly expand upon them by adding the ability to use this framework for an even greater amount of modalities and capabilities.Furthermore, masked modeling has proven effective for learning useful representations in both NLP [24, 71] and vision [40, 8, 107, 29] . Extending it to multimodal domains [7, 36, 94, 65] enables strong cross-modal representations which is critical for multimodal learning. When combined with tokenization, masked modeling also enables generative applications [15, 55, 14, 85, 65] . Our work highlights the ability of masked modeling to expand to a much greater set of modalities than previously shown, improving upon the out-of-the-box and multimodal generation capabilities of previous works. 
limitations and discussion: 
 We developed an any-to-any model on tens of diverse modalities and tasks. This was achieved by mapping all modalities to discrete sets of tokens via modality-specific tokenizers and using a multimodal masked training objective [65] . We successfully scaled the training to 3 billion parameters and to 21 modalities and different datasets, without a degradation in performance compared to the existing expert single/few task models. This results in strong out-of-the-box capabilities as well new potential for multimodal interaction, generation, and retrieval, all by a single unified model. Below, we discuss limitations and future work.Transfer/emergent capabilities: One hope from training a single network on several tasks is leading to a model that can solve novel tasks, often referred to as "transfer" or "emergent" capabilities. While, as we showed, a multitask model brings several key advantages even without transfer/emergence (e.g., efficiency, using a single model for broad out-of-the-box capabilities, modality fusion, etc.), we observe that the potential for transfer/emergence remains largely untapped. In general, compared to LLMs, vision/multimodal models have not shown exciting results in terms of transfer/emergence yet. We find this to be an important point to address in the future, e.g., via designing multitask architectures that have emergence, in contrast to out-of-the-box capabilities, as their main objective.Better tokenization: Like any token-based model, ours can directly benefit from progress on tokenizers, e.g. higher reconstruction fidelity. Depth-to-RGB retrieval Normals-to-RGB retrieval Segmentation-to-RGB retrieval Caption-to-RGB retrieval For training 4M-21, we initialize the training using 4M models that we pre-trained on COYO700M [12] . We ablate in Table 4 different choices of training data and modalities. We can see that performing cotraining on C4 [71] and COYO700M [12] has the potential to slightly improve transfer performance on average. Similar to 4M, to have an aligned multimodal dataset, we pseudo label the CC12M dataset using strong specialized models for each task. The pseudo labeling of existing modalities is done in the same fashion as 4M, using Omnidata DPT-Hybrid [73] for surface normals and depth estimation, COCO Mask2Former [20] with a SwinB [58] backbone for semantic segmentation, COCO ViTDet ViT-H model [56] initialized from MAE weights [40] for bounding boxes, and CLIP-B16 [70] with ViT-B/16 visual backbone backbone for CLIP feature maps.3D human poses. We use 4D-Humans [37] to extract 3D pose and shape parameterized by an SMPL model. For the images in CC12M without humans, we set the pose label to a "none" token. For the images with humans, we form a sequence by concatenating the bounding box, body pose, camera, and shape values in a sequence for each human instance. As data augmentation, we randomly shuffle the order of each component in the sequence.SAM instances. Besides semantic segmentation and bounding boxes, SAM [49] instance segmentation also provides some level of semantic information from an image by clustering together semantically similar pixels in it. Unlike semantic segmentation, SAM instances are not restricted to a specific set of classes and can segment in more detail. We use the SAM H model and query it with points in a grid format to obtain the instances. We also considered the SAM-HQ [47] H model, however in the grid-point querying format, it yields very similar results to SAM. We found 32 × 32 query points to be the optimal choice both for pseudo labeling speed and quality.DINOv2 and ImageBind global features & feature maps. We extract both dense feature maps and global embeddings, i.e. cls token embeddings, from DINOv2-B14 [68] and ImageBind-H14 [35] pre-trained models. For the latter, we only extracted the image embeddings, incorporating other modality embeddings such as thermal or audio could be interesting future work.T5-XXL embeddings. Language model embeddings, such as from T5-XXL [71] , have been shown to improve the generation fidelity and text understanding capabilities of text-to-image generative models [80, 14] . Consequently, we use the T5-XXL encoder to extract text embeddings from all CC12M captions, without any preprocessing of the text. Unlike other modalities, we do not convert these text embeddings to a sequence of discrete tokens or treat them as targets (similar to the RGB pixel modality variant). Instead, we only provide them as inputs using a linear projection from the T5-XXL embedding dimension (d T5-XXL = 4096) to our model's embedding dimension.Image metadata. From RGB images, we directly extract different types of metadata like the original height and width before cropping [69] , brightness, contrast, saturation and entropy. We additionally extract a notion of colorfulness, following [39] .Semantic metadata. We compute the crowdedness score as the number of humans in the pseudo labeled human poses, the SAM clutter score as the number of SAM instances, the COCO clutter score as the number of COCO instances, the COCO instance diversity as the number of unique COCO instance classes, and the semantic diversity as the number of unique COCO semantic classes in an image. Geometric metadata. To compute the occlusion score, we first generate occlusion edges from depth images by applying a Sobel filter, followed by counting the percentage of occlusion edge pixels that surpass a threshold of 0.3. As a notion of geometric complexity, we project surface normal pixels onto the unit sphere, and compute their angular variance. Note that images of indoor scenes or caves featuring large surfaces pointing in all different directions receive a high score in this metric, while ones with a more localized geometric variance get a somewhat lower score. Exploring other potential notions of geometric complexity can be an interesting future addition.Color palette. For every RGB image, we extract between one and seven color palettes using PyPalette [2] . During training, we randomly sample one of the color palettes to enable users to input palettes with different levels of granularity.SAM edges and canny edges. Edges are a convenient way of grounding image generation on shapes contained in images [104] . To pseudo label edges, we apply the OpenCV canny edge detector on SAM instance maps and RGB, to obtain SAM edges and canny edges respectively.We use a BottleneckMLP [6] with 6 blocks and 1024 width to compress pose into 8 tokens. We use 1024 vocabulary size, and trained using smooth L1 loss for 15 epochs on CC12M training data. We also binned the global orientation, body shape, and bounding boxes into 1000 discrete bins similar to [17] . The final sequence is obtained by also adding identifiers, i.e. "bbox", "pose", "shape", before the corresponding sub-sequence.The SAM instance tokenizer is a ViT-based VQ-VAE that tokenizes 64 × 64 binary masks into 16 tokens using a vocabulary size of 1024. The tokenizer is trained using the cross-entropy loss for 24 epochs on CC12M training data, by resizing individual masks into a square aspect ratio image of 64 × 64 pixels. To preserve the SAM instances' original location, width, and height in the image, their bounding boxes are extracted. The final sequence for each instance is formed by appending the identifier "polygon" to 4 numbers that specify the bounding box of the instance, along with the 16 token IDs.Similar to human poses, we use BottleneckMLP with 6 blocks and 1024 width to compress DINOv2-B14 and ImageBind-H14 global embeddings into 16 tokens. We use 8192 vocabulary size, and trained using cosine similarity loss for 15 epochs.We follow [65] and tokenize CLIP-B16, DINOv2-B14, and ImageBind-H14 dense feature maps into 196, 256, and 256 tokens, respectively, using a ViT-based VQVAE with 8192 vocabulary size and smooth L1 loss.We tokenize text, color palette, metadata, and bounding boxes using a WordPiece tokenizer by fitting it on all captions and 4000 "special value" tokens, with a joint vocabulary size of 30k. These special tokens are divided into four groups, each with 1000 values, i.e. v0=0, v0=1, ..., v0=999, v1=0, v1=1, ... v1=999, v2=0, v2=1, ..., v2=999, v3=0, v3=1, ..., v3=999. For bounding boxes, we follow 4M [65] and represent xmin, ymin, xmax, ymax coordinates using v0, v1, v2, v3 tokens respectively. Other modalities are tokenized by binning their values into corresponding bins, e.g. color palette sequence is formed as color = c R = r G = g B = b R = r, ... where c takes a value between 1 and 7 and specifies the number of colors in the palette and r, g, b takes values between 0-255. We chose to model metadata using interleaved pairs of special tokens, where the first one specifies the type of metadata modality, and the second specifies its value. For example, a crowdedness score of 3 and a brightness of 120 would be specified as the sequence v1=5 v0=3 v1=10 v0=120. During training the number of metadata entries and their order is randomized. All of this results in a sequence prediction formulation, following [65, 17] .We use a VQ-VAE with a diffusion decoder, similar to [65] to tokenize the edge modalities. We use the same tokenizer as it reconstructs both edges similarly well.We provide qualitative results for tokenizer reconstructions in fig. 13 . Please see Tab. 6 for an overview of pre-training settings. For more accurate model comparisons, the architecture and overall training objective of our B, L, and XL models are the same as those of 4M models. However, we do modify and improve various aspects of the training process that allow us to significantly increase the number of training modalities. These changes concern modality-specific accommodations to the masking strategy, the ability to co-train on several datasets, and the use of a more diversified multimodal masking strategy. We describe these modifications below:Positional and modality embeddings. As with 4M, 4M-21 incorporates both learnable modality embeddings and fixed sine-cosine positional embeddings for each modality. The positional embeddings are either 1D or 2D depending on the modality type.Metadata grouping and chunk-based masking. To address the sparsity and number of different types of metadata, the metadata modalities are all grouped together as a single modality during training. This prevents the over-allocation of tokens to sparse metadata, enabling a more balanced distribution of the token budget across modalities. However, the standard span masking from T5 [71] and 4M [65] performs random uniform masking at the token level, which can lead to pre-training inefficiencies [53] and make conditioning on specific metadata difficult, as conditioning on just one of them would rarely occur during pre-training with this masking strategy. Instead, we propose to mask chunks of sequence (similar to PMI-Masking [53] ), where the span masking is performed per chunk of metadata instead of at the token level.Multi-dataset support. Unlike 4M which was only trained on a single aligned dataset, we train 4M-21 on multiple datasets simultaneously. This flexibility allows for the inclusion of datasets with varying numbers of modalities, which enables training on both large-scale datasets with a smaller number of modalities and smaller datasets with a larger diversity of modalities.Sampling and masking strategies. Our data sampling process involves selecting a training dataset based on its sampling weight, followed by choosing a masking strategy from the dataset-specific mixture of masking strategies. Input and target tokens are then sampled using the selected strategy.Co-training datasets. We co-train on several datasets to improve the model's performance and the data diversity. These include CC12M [16] , which comprises about 10 million text-image samples fully pseudo labeled with all 21 modalities, and accounts for 60% of our training samples. Additionally, we include COYO700M [12] , with approximately 500 million text-image samples pseudo labeled with the 7 modalities of 4M, and accounts for 20% of our training samples. Lastly, the Colossal Clean Crawled Corpus (C4) [71] , a large text-only dataset, is used for language model co-training, also making up 20% of our training samples.Diverse mixture of masking strategies. As with 4M [65] , the masking strategy is governed by Dirichlet distribution with parameter α. This distribution influences the sampling of tokens from modalities: a lower α results in samples dominated by one modality, while a higher α leads a more balanced representation across all modalities. For both CC12M and COYO datasets, we implement multiple masking strategies to cater to specific training needs, and randomly sample from them for every sample in the batch:• All-to-all masking: Involves four masking strategies with symmetric input and target α set to 0.01, 0.1, 1.0, and 10.0 respectively.• RGB-to-all masking: Consists of only RGB tokens as input, with target α all set to 0.5.• Caption-biased masking: Includes two strategies, heavily skewed towards either unmasked captions or T5-XXL embeddings as input. These masking strategies are particularly beneficial for tasks involving text-to-image generationBelow, we provide further details on out-of-the-box evaluations we performed. Please also see fig. 14 for a qualitative comparison between our XL model and Unified-IO XL [61] , as well as Unified-IO 2 XXL [60] . Furthermore, table 7 compares Unified-IO, Unified-IO 2, and our model's out-of-thebox capabilities on surface normal estimation, depth estimation, and semantic segmentation. As demonstrated, our model outperforms Unified-IO and Unified-IO 2 in all the mentioned tasks. We follow the evaluation setup in [65] and evaluate on DIODE validation set at 224 × 224 input resolution.We employ a similar approach as SAM [49] by querying our model on the bounding boxes to obtain the instances. To predict the instances, only the target bounding box is provided in the input final sequence, and the tokens are masked for our model to predict them.We follow the evaluation setup from DINOv2 [68] and set k = 20 and temperature to 0.07. Cosine decay Layer-wise lr decay [21] 0.75 0.85 0.9 Drop path [43] 0.1 0.2 0.0 LoRA [42] As we develop several tokenization strategies for each modality, ablating their performance against all possible design choices would be prohibitively expensive. Thus, we focus on one modality, namely SAM instances, and provide a more detailed look into the impact of different tokenization strategies. We study two approaches for SAM instances: path tokenization and mask tokenization.Path tokenization: We represent each instance in the image as a list of polygon coordinates. Then we tokenize these coordinates using a Bottleneck MLP-based VQ-VAE tokenizer. To achieve a fixed-size input, the polygons are either simplified or extended to have the same number of corner points. We found that fixing the maximum number of corners to 128 results in a minimal change in the overall polygon shape, thus we use this value for all the path tokenization ablations.Mask tokenization: In this scheme, we first convert each instance to a binary masks and resize them to a fixed mask size. Then, we tokenize them using a ViT-based VQ-VAE tokenizer, similar to the way we tokenize image-like and feature map modalities.Ablations: We investigated L1 and MSE losses for both tokenization schemes, and additionally cross-entropy and Dice loss for the mask tokenization. We also investigated the effects of the total number of tokens, token vocabulary size, and mask size. To compare the performance of the resulting tokenizers, we use the IoU between the pseudo-labeled and reconstructed instances as our metric. fig. 15 illustrates the results of different ablated configurations. For each configuration, the remaining unspecified parameters are by default set to 16 for the number of tokens, 1024 for the vocabulary size, L1 for the loss, and 64 × 64 for the mask size. The ablations show that using mask tokenization with 16 tokens, 1024 vocabulary size, and 64 × 64 mask size performs well and sets a good balance between reconstruction quality and total sequence length. In all ablations, the tokenizers are trained for 24 epochs starting with 5 warmup epochs using the AdamW [59] optimizer with β 1 , β 2 = 0.9, 0.999 and a batch size of 128. For all the experiments except the Dice loss, a learning rate of 1e-5 is used. Since using this learning rate for the Dice loss experiment resulted in instabilities, we reduced its learning rate to 1e-6. As demonstrated in fig. 16 , increasing the number of tokens results in better reconstruction quality both for the mask tokenizer and the path tokenizer. Compared to L1 loss, the cross-entropy loss training obtains reconstructions with smoother edges and better coverage. In our work, we decided to use an encoder-decoder architecture to achieve any-to-any multimodal capabilities, while other approaches [74, 87, 92, 10] use decoder-only architectures in multimodal settings. Both architectural choices are valid depending on specific use cases and priorities, and future work could explore these trade-offs more systematically. Below, we explain our choice of an encoder-decoder architecture and discuss how a similar model might look in a decoder-only setting.Following 4M, we use an encoder-decoder architecture as it is directly compatible with masking approaches (e.g., T5 [71] , MAE [40] ), which is at the core of our method and enables any-to-any capabilities with a single training objective. In addition, after training, the encoder can be extracted and used as a ViT or a multimodal encoder. Notably, the bidirectional self-attention in the encoder has been shown to have slight benefits over causal attention for representation learning & transfer tasks [71, 30] .If we were to design a 4M-like model using a decoder-only architecture, there are two main approaches to consider:The first is a causal decoder (i.e. next token prediction without span [71] or MAE [40] masking). This approach is similar to multimodal LLMs operating on interleaved data [74, 87, 92, 10] and is the easiest to unify with LLMs. However, the 4M masking strategy would not be directly compatible with this approach. A naive strategy would be to keep everything unmasked and concatenate one modality after another, but this leads to much larger sequence lengths per example and redundancy between modalities. As many of the capabilities shown in our paper rely on cross-modal masking (e.g. the any-to-any capabilities and the ability to predict outputs from partial modalities), it is unclear The second approach is using a prefix LM-like decoder (see Fig. 4 of T5 [71] ), where unmasked inputs (i.e., encoder inputs in the 4M formulation) and masked inputs/targets (i.e., decoder inputs in the 4M formulation) are concatenated. The entire sequence is then given to a single decoder LLM. This approach allows preserving the masking strategy and training objective within a decoder-only architecture, but has seen less adoption than encoder-decoder approaches in the masking literature. However, it is more amenable to multi-turn or temporal inputs, as multiple sequences of unmasked inputs and masked targets can be concatenated one after the other.In summary, we use an encoder-decoder architecture as it provides a straightforward way to achieve any-to-any capabilities through masking, and allows for downstream reuse of the trained encoder as a singlemodal or multimodal backbone. While decoder-only approaches could potentially be adapted for similar purposes, we are unaware of work demonstrating this at the same scale (in terms of number of modalities) and we believe it to be a very impactful and exciting research direction.In table I.2, we quantize our model's conditional generation capabilities by performing caption-toimage generation on COCO. We compare against 4M-7 [65] across all model sizes, Stable Diffusion 2.1 [76] , and a controlled text-to-image specialist baseline. The controlled text-to-image baseline (T2I-B), conceptually similar to Muse [14] , uses the same architecture and RGB tokenizer as our model and 4M-7, and was trained for a total of 300B tokens on CC12M [16] . We test for image fidelity using FID and image-text alignment using CLIP score, computed using 30'000 validation set images, and resizing all images to 256x256. We used guidance scale 3.0 for all experiments.Our models are able to consistently outperform 4M-7 across model sizes on COCO, both in terms of FID and CLIP score. While there is still a seizable gap between dedicated text-to-image models like Stable Diffusion 2.1 and our models on out-of-distribution data, we note that these models are usually trained on orders of magnitude more data and compute. Our T2I-B baseline attempts to control for factors such as the tokenizer that can have a significant influence on FID, and we see that 4M-7 B performs similar to the specialist T2I-B. Optimizing for image generation quality was not the focus of our work, but considering the scaling trends of token-based masked (e.g. Muse [14] , MAGE [55] ) and auto-regressive models (e.g. Parti [96] ), we expect significant improvement with larger model sizes. Furthermore, we expect that recent advances in RGB tokenization (e.g. MAGVIT-v2 [98] , FSQ [63] ) will translate to significant gains in FID for large enough models.• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: Broader impacts are discussed in section K of the appendix. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: We discussed broader impacts of our models in Sec. K of the appendix, we will release our models and code with standard safeguards.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cited the assets we used in the paper accordingly.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix:  


Decision: This submission presented a multimodal framework trained on a wide range of modalities and tasks. The authors propose to use large-scale datasets including images, text, semantic and geometric modalities, feature maps from state-of-the-art models and new modalities such as image metadata or color. Processing at least three times as many tasks and modalities as existing models, this unified single model delivers excellent results and also improves fine-grained, controllable multimodal generation.  
Reviewers asked for several clarifications. After the rebuttal, all were satisfied with the answers.  
The reviewers and AC are ultimately convinced by this submission, considering that the novelty and interest of the proposed multimodal model and learning framework are clearly evident.  
The authors are strongly encouraged to take all comments into account in their final version.

Reviews: review: paper_summary: paper_summary: The authors present a new vision model that can generate tokens in all directions that represents multiple modalities like RGB images, depth map, segmentation maps, color palette, DINOv2 features etc. given a conditioning on a subspace of the modalities.
They develop a new way to tokenize certain modalities and scale both model’s parameters and dataset’s size.
strength_weakness: strength_weakness: - The added value over 4M is mostly due to engineering.
- No multimodal retrieval nor conditional generation metrics. The actors emphasize a lot the generative/multimodal part of the model but the only quantitative experiments are shown in Table 3. Qualitative figures are interesting but should be supporting quantitative results. The conditional generation evaluation needs to be compared to other models like ControlNet [1] and you should at least provide precision/recall@k on some modalities for the multimodal retrieval (like comparing DINOv2/ImageBind with one modality against your model with one modality and your model with multiple ones).
- The tokenization might reduce the burden of finding good losses’ weight, but tokenizers trained on a specific domain (like CC12M) might not generalize well. Moreover, tokenizers might not capture high frequency details that are very important like text for OCR, good quality faces, or smaller details. In [2], the authors speak about the fact that “quality limitations of the VQ image reconstruction method inherently transfer to quality limitations on images”. They also focus on the different domains that are not well captured by the tokenizers. The current paper only includes one ablation on the tokenizer, and I’m skeptical of the tokenizers’ performances on out-of-distribution images.

[1] Adding Conditional Control to Text-to-Image Diffusion Models, Zhang, Rao and Agrawala
[2] Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors, Gafni et al.
questions: questions: - What are the performances of the trained tokenizers? Given that your model will be somewhat bounded by the tokenizers’ performances, could you provide a table including the performances of all tokenizers versus the original model and yours?
- How did you chose each vocabulary size?
- The metadata picked by the authors seem arbitrary, could you explain your decision process?
l.223 I don’t understand why you treat RGB tokens and pixels differently: ‘we leverage RGB patch embeddings learned during the pre-training, as RGB pixel inputs are used alongside the tokenized modalities’. Could you expand on this please?

The paper is interesting nonetheless but could be way, way stronger with quantitative evaluations on the new claimed capabilities.
limitations: limitations: The authors talk about all the limitationsof their work.
paper_summary: paper_summary: The paper addresses the limitations of current multimodal and multitask foundation models, such as 4M and UnifiedIO, which are constrained by the limited number of modalities and tasks they can handle. The authors present a model trained on a wide variety of modalities and tasks using large-scale multimodal datasets and text corpora. This includes training on images, text, semantic and geometric modalities, feature maps from state-of-the-art models, and new modalities like image metadata and color palettes. A key technique used is discrete tokenization of various data types. The new model can handle at least three times more tasks and modalities than existing models without losing performance. This approach also enhances fine-grained and controllable multimodal generation and explores the unification of diverse models into a single one.
strength_weakness: strength_weakness: 1. The writing of the paper has room for improvement. Not all reviewers have read the 4M paper. The paper does not detail the training framework, such as how to control different tasks and how to conduct multimodal masked training. These aspects are not simply described and are only mentioned in section 2 as following the 4M paper. Including a simple diagram would significantly improve clarity.

2. Some claims are somewhat inaccurate. There are works that use very simple methods to unify modeling for multiple tasks, maybe not as many as 10. It is necessary to discuss the difference and challenges between completing 5 tasks and 10+ tasks, as some tasks can actually be categorized as a single task.
questions: questions: 1. The paper uses an encoder-decoder paradigm, whereas we know that the commonly used LLMs nowadays typically employ a decoder-only structure or simply a multi-layer transformer structure. Why does the paper adopt the encoder-decoder paradigm? The decoder-only approach[1][2] seems simpler and better suited for unifying with the LLM structure.
The authors are advised to include some discussion on encoder-decoder and decoder-only approaches, referencing papers such as the following.

[1] GiT: Towards Generalist Vision Transformer through Universal Language Interface. (ECCV 2024)

[2] Fuyu-8B: A Multimodal Architecture for AI Agents. (Blog)
The two articles mentioned above also tokenize everything within a simple unified framework.

2. Do different feature maps count as different modalities? I thought different modalities referred to the basic levels such as image, language, and speech, etc. The formulation in the paper is somewhat strange, and I hope the authors can explain this a bit. After all, these features are generated by cost-intensive multi-modal encoders, which differs from the lightweight multimodal tokenizer proposed in the paper. I also hope the authors can discuss this in comparison with the approaches of GiT[1] and Fuyu-8B[2], which use light-weight yet simple tokenizers on original image and language.

3. Regarding the challenge of negative transfer, you can also refer to the GiT paper. It demonstrated that joint training on five tasks has better performance than single-task training. (Considering that this paper is also recent, a direct comparison may not be necessary, but it is still recommended to discuss it.) :)

Overall, I am inclined to accept this paper, but I still hope the authors can address my concerns. If these issues are addressed well, I will accordingly raise my score. :)
limitations: limitations: There are no comments regarding the limitations.
paper_summary: paper_summary: The paper presents an advanced vision model capable of handling a wide range of tasks and modalities, demonstrating the potential to train a single model on tens of diverse modalities without a loss in performance compared to specialized models. Specifically, the model is trained on a multitude of modalities, including RGB images, depth, semantic segmentation, CLIP features, surface normals, and more, enabling it to perform various tasks such as image generation, retrieval, and understanding.
strength_weakness: strength_weakness: 1. Do the authors try to measure the quality (e.g. FID ) of generated images (based on caption input) and can the proposed method surpass some existing diffusion-based models?

2. The authors should illustrate the detailed configurations of B, X, and XL models, for example, the number of layers and the width of channel dimensions.
questions: questions: see weakness
limitations: limitations: see weakness

Comments: title: Thank you!
Thank you!
comment: We thank the reviewer for the valuable suggestions and positive feedback. We will include the contents of the rebuttal in the camera ready.
We thank the reviewer for the valuable suggestions and positive feedback. We will include the discussions and results in the camera ready.
Thank you for the detailed response. I hope the author will include the contents of the rebuttal in the camera-ready version, including the discussion on Decoder-only models and feature maps as a modality. This will provide more insights to those who read the paper. I will accordingly raise my score.
Thanks for your detailed answer.
I really appreciate the tables 1 and 2 in the rebuttal pdf and the references about tokenizers' quality.

A CLIP baseline in the Table 2 for image and text retrieval would perfect it.

**Treating RGB tokens and pixels differently**
Adding a small discussion about this in the camera ready would help readers understand your thought process and why it is done like this.
A small table comparing results with RGB vs tokenizer as input would also help readers chose the best way for their own task when using your model.

**The added value is mostly due to engineering.**
I apologize for the confusing term I used. I was pointing out that your comparison uses 4M, but you also utilized more compute and data, which I wrongfully referred to as engineering. The new capabilities you've added are impressive, particularly with the quantitative results to compare against the pseudo-labelers. However, it's unclear whether the improvements on previous capabilities (wrt. 4M) are due to the additional data or because increasing the number of modalities leads to compounding gains on the other ones.

I think that the paper now pass the bar for acceptance as long as you include the Table 1 and Table 2 from rebuttal.
Thanks again for resolving my issues and I'll update my score.