UNIFIED-IO: A Unified Model for Vision, Language, and Multi-modal Tasks
Authors: Jiasen Lu
Christopher Clark
Rowan Zellers
Roozbeh Mottaghi
Aniruddha Kembhavi
Venue: ICLR 2023 notable top 25% (2023)

Abstract: We propose Unified-IO, a model that performs a large variety of AI tasks spanning classical computer vision tasks, including pose estimation, object detection, depth estimation and image generation, vision-and-language tasks such as region captioning and referring expression, to natural language processing tasks such as question answering and paraphrasing. Developing a single unified model for such a large variety of tasks poses unique challenges due to the heterogeneous inputs and outputs pertaining to each task, including RGB images, per-pixel maps, binary masks, bounding boxes, and language. We achieve this unification by homogenizing every supported input and output into a sequence of discrete vocabulary tokens. This common representation across all tasks allows us to train a single transformer-based architecture, jointly on over 90 diverse datasets in the vision and language fields. Unified-IO is the first model capable of performing all 7 tasks on the GRIT benchmark and produces strong results across 16 diverse benchmarks like NYUv2-Depth, ImageNet, VQA2.0, OK-VQA, Swig, VizWizGround, BoolQ, and SciTail, with no task-specific fine-tuning. Code and pre-trained models will be made publicly available.

Published as a conference paper at ICLR 2023
UNIFIED-IO: A UNIFIED MODEL FOR VISION, LANGUAGE, AND MULTI-MODAL TASKS
Jiasen Lu"; Christopher Clark!", Rowan Zellers
Roozbeh Mottaghi'o Aniruddha Kembhavito
Allen Institute for AI. UUniversity of Washington, Seattle
ABSTRACT
We propose UNIFIED-IO, a model that performs large variety of Al tasks span- ning classical computer vision tasks, including pose estimation, object detection, depth estimation and image generation, laio--and-language tasks such as region captioning and referring expression, natural language processing tasks such as question answering and paraphrasing. Developing single unified model for such large variety of tasks poses unique challenges due to the heterogeneous in- puts and outputs pertaining to each task, including RGB images, per-pixel maps, binary masks, bounding boxes, and language. We achieve this unification by ho- mogenizing every supported input and output into a sequence of discrete vocab- ulary tokens. This common representation across all tasks allows us to train single transformer-base
architecture, jointly on over 90 diverse datasets in the vision and language fields. UNIFED-IO the first model capable of performing all 7 tasks on the GRIT benchmark and produces strong results across 16 diverse benchmarks like NYUv2-Depth, ImageNet, VQA2.0, OK- KQA, Swig, VizWiz- Ground, BoolQ, and SciTail, with no task-specific fine-tuning, Code and demos for UNIFIRD-IO IO are available at: uni f fied-fie o lenai..... .org
INTRODUCTION
We present UNIFIED-IO, the first neural model to jointly perform a large and diverse set of AI tasks spanning classical computer vision (such as object detection, segmentation, and depth estimation), image synthesis (such as image generation and image in-painting). vision-and-lang language (like visual question answering, image captioning, and referring expression) and NLP (such as question an- swering and paraphrasing). Unified ceneral-purposs models avoid the need for task-specific design, learn and perform a wide range tasks with a single architecture, can utilize large, diverse data corpora, can effectively transfer concept knowledge across tasks, and even perform tasks unknown and unobserved at design and training time.
Building unified models for computer vision has proven to be quite challenging since vision tasks have incredibly diverse input and output representations. For instance, object detection produces bounding boxes around objects in image, segmentation produces binary masks outlining regions in an image, visual question answering produces an answer as text, and depth estimation produces map detailing the distance of each pixel from the camera. This heterogeneity makes it very chal- lenging to architect single model for all these tasks. In contrast, while the landscape of natural language processing (NLP) tasks, datasets, and benchmarks is large and diverse, their inputs and desired outputs can often be uniformly represented as sequences of tokens. Sequence to sequence (Seq2Seq) architectures (Raffel et al., 2020; Brown et al., 2020), specifically designed to accept and produce such sequences of tokens, are thus widely applicable to many tasks. Unified models employing such architectures have been central to much recent progress in NLP.
Unified models for computer vision typically use shared visual backbone to produce visual em- beddings but then employ individual branches for each of the desired tasks. These include models
"Equal contribution. Correspondence ji asen @@al 1enai.

---

Published as a conference paper at ICLR 2023
Binary Mask
Segmentation
Language
Image
Tasks
What they playing
Image Classification
Object Detection
Sementic Segmentation
*puutsequuncce emberdings
Depth Estimation
Surface Normal Estimetion
Bounding Boxes
Language
nocdee shatl
Segment based Image Generation
rilid win calortal
Image inpainting
Pose Estimation
Unified-IO
Relationship Detection
Image Captioning
Visual
Segmentation
Depth
Binary Mask
Image
Referring Expressions
Situation Recognition
Text-based Image Generation
Commonsense
Classification
Key Surface
Bounding
Region Captioning
GLUE Benchmark
Reading comprehensiin
Natural Language Inference
Grounded Com Comsonsesse Inference
Figure 1: UNIFIED-IO is single eequence-to-sequenee model that performs a variety of tasks in computer vision and NLP using unified architecture without a need for either task modality- specific branches. This broad unification is achieved by homogenizing every task's 's input and output into a sequence of discrete vocabulary tokens. UNIFTED-IO supports modalities as diverse im- ages, masks, keypoints, boxes, and text, and tasks varied as depth estimation, inpainting, semantic segmentation, captioning, and reading comprehension.
like Mask R-CNN (He al., 2017) for classical visual tasks that use an ImageNet pre-trained en- coder followed by branches for detection and segmentation, trained in fully supervised manner. In vision and language (V&L) domain, CNN backbones feed visual features to transformer archi- tectures that also combine language, followed by task-specific heads for visual question answering, referring expression, visual commonsense reasoning, etc. (Lu et al., 2019; Li et al., 2019; Tan & Bansal, 2019). A more recent trend has the emergence of unified architectures that do away with task-specific heads and instead introduce moodlltty-specinf al., 2021; Gupta et al., 2022a; Wang et al., 2022b) for instance, single language decoder that serves multiple tasks requiring language output like captioning and classification. However, most progress unified models continues be centered around V&L tasks, owing the simplicity of building shared language decoders and is often limited to supporting just handful of tasks.
heads (Hu & Singh, 2021; Cho
UNIFIED-IO is Seq2Seq model capable of performing variety of tasks using unified architecture without a need for either task or even modality-secciffc branches. This broad unification is achieved by homogenizing every task's output into sequence of discrete tokens. Dense structured outputs such as images, segmentation masks and depth maps are converted to sequences using vector quantization variational auto-encoder (VQ-GAN) (Esser al., 2021), 1. sparse structured outputs such as bounding boxes, and human joint locations are transcribed into sequences of coordinate tokens, and language outputs are converted to sequences using byte-pair encoding. This unification enables Unified-IO to jointly train on over 90 datasets spanning computer vision, &&L, and NLP tasks with a single streamlined transformer encoder-decoder architecture (Raffel et al., 2020).
Our jointly trained UNIFIED-IO is the first model to support all 7 tasks in the General Robust Im- age Task (GRIT) Benchmark (Gupta et al., 2022b) and obtains the top overall score 64.3 when averaging across all tasks, handily beating the second best model by 32.0. We further evaluate UNIFIED-IO on 16 diverse benchmarks across computer vision and NLP, without any fine-tuning towards any individual benchmark, and find that it performs remarkably well compared special- ized (or fine-tuned) state-of-the-arr models.
2 VISION, LANGUAGE AND MULTI-MODAL TASKS
UNIFIED-IO is designed to handle a wide range of language, vision and language, and classic vision tasks in unified way. To fully test this capability, gather 95 vision, language, and multi-modal
2

---

Published as a conference paper at ICLR 2023
Example Source
Size
Input Modalities
Output Modalities
Datnsets
Size
Percent
Rate
Text
Image
Spurse
Dense
Text
Image
Sparse Dense
Image Synthesis
43,0
Image Synihesis from Text
RedCapp
Image Inpainting
1.2m
Image Syuthesis from Sog.
LVIS
220k
Sparse Labelling
8.2m
Object Detection
Open Images
1.9m
3.6
Object Localization
VG
6m
4.6
7.1
Keypoint Estimation
COCO
140k
0.1
0.7
Reforting Expression
RefCoco
130k
0.1
1.1
Dense Lubelling
2,4m
1.8
6.2
Depth Estimation
NYU Depti
48k
0.1
0.4
Surface Normal Fstimation
Framenet
2
210k
0.2
1.1
Object Segmentation
LVIS
3
2.
1.6
4.7
Imuge Clussiicattinn
22m
16.8
12.5
Image Clussiiicttoon
ImageNer
16m
12.2
8.1
Object Categorization
COCO
6m
4,6
4.4
Image Captioning
31m
23.7
12.5
Webly Supervised Captioning
CC12M
Supervised Captioning
Wi:Wiz
Region Captioning
Vision & Language
Visual Question Answering
VQ4 2.0
13
3.3m
10.4
Relationship Detection
640%
1.9
Grounded VQA
V:Wi
6.5%
0.1
NL.
31
7.1m
5.4
12.5
Text Classification
MNLI
17
1.6m
1.2
4.8
Question Answering
SQuAD
13
1.7m
1.3
5.2
Text Summarization
Gigaword
1
3.8m
2.9
2.5
Langunge Modelling
2
12.5
Masked Language Modelling
C4
2
12.5
All Tasks
95
130m
100
100
Tasks UNIFIED-IO learns to complete. From left right, columns show example of one of the sources used for the task, the number of datasets, total number and percent of examples relative the entire training corpus, and sample rate during multi-task training. Subsequent columns show what modalities are required for the tasks, and highlighted rows show aggregated statistics for groups of similar tasks.
Table I:
datasets from 62 publicly available data sources as targets for our model to learn during multi-task training. These datasets cover wide range of tasks, skills, and modalities.
We categorize the input and output modalities of each task into 4 different types: Text natural small number of location coordinates within the image; Dense per-pixel labels such as depth maps, surface normal maps, etc. We group related datasets into groups and 22 tasks to facilitate our training and analysis:
language tokens; Tmage RGB images; Sparse
Image Synthesis. Given a text description, partially occluded image and inpainting target, or seg- mentation map containing a semantic class for some pixels, generate matching image. Data sources with image and text pairs (Desai et al., 2021), bounding boxes (Krishna et al., 2017) or semantic seg- mentation (Gupta et al., 2019) can be used to build these tasks.
Sparse Labelling. Given image and natural language query, identify the target regions key- point locations that are being referred to. Tasks include object detection (Kuznetsova et al., 2020), object localization (Rhodes al., 2017), human pose estimation (Lin al., 2014), and referring expression (Kazemzadeh et al., 2014).
Dense Labelling. Given image, produce per-pixel labels for that image. Labels include the distance that pixel to the camera (Nathan Silberman & Fergus, 2012), surface orientation (Bac et al., 2021) or semantic class (Lin et al., 2014).
Image Classification. Given an image and optionally a target bounding box, generate a class name tag of that image target region. This group includes image classification (Deng et al., 2009) and object categorization (Pinz et al., 2006) datasets.
Image Captioning. Given image and optionally bounding box, generate a natural language description of that image or target region. We include both crowd-sourced (Chen et al., 2015) and webly supervised (Changpinyo al., 2021) captions.
3

---

Published as a conference paper at ICLR 2023
VOGAN
angmak quantization
MOGANI
(CNN0
Image serializatio using VQ-GAN
VOVAE eaccadrr frozen
SentPlace
sequence target
Segment the cat
encoder
VOGAN decodar
Whet of the car?
Sentileace encoder
Unified-1O
SentPlace decoder
brown
Patch embedding
Transfomerr Encoder
Transfomerr Decoder
ipput
outpott sequencon
VOGAN decoder
What is the depth map of the image?
SentPlace ancoder
SentPlece decoder
loc/ho loci90
SentPlace ancoder
Localize the cut
Figure 2: Unified-IO. schematic of the model with four demonstrative tasks: object segmentation, visual question answering, depth estimation and object localization.
Vision & Language. broad category for other tasks that require joint reason over image con- tent and a natural language query. There are many popular vision and language datasets, and we categories these datasets into 3 tasks visual question answering (Antol al., 2015); relationship detection (Lu et al., 2016) and grounded VQA (Chen et al., 2022a).
NLP. Tasks with text the only input and output modalities, including text classification (Williams et al., 2018), question answering (Rajpurkar et al., 2016) and text summarization (Graff et al., 2003).
Language Modeling. The masking language modeling pre-training task (See Section 3.3) using text from C4 (Raffel et al., 2020) and Wikipedia (Foundation), which we include to ensure the knowledge gained from language pre-trate pre-training not lost during multi-tasi training. Other pre-training tasks are not included because the relevant datasets are already used in other supervised tasks (e.g., for captioning or classification).
Table shows the details of tasks and groups. We list an example dataset source, number of datasets, number of examples, percent of the total number of examples, and sampling rate during training (Section 3.3) for each group and task. Subsequent columns show what modalities are required for the inputs and outputs. We defer additional task details, inference details, the complete list of datasets and visualizations to the Appendix A.I.
UNIFLED-I0
Our goal is to build a single unified model that can support diverse of tasks across computer vision and language with little to no need for task-specific customizations and parameters. Such unified architectures can applied to new tasks with little to no knowledge the underlying ma- chinery, enable general pre-training to benefit many diverse downstream applications, be jointly trained on a large number of tasks, and better allows knowledge to be shared between tasks.
3.1 UNIFIED TASK REPRESENTATIONS
Supporting a variety of modalities such as images, language, boxes, binary masks, segmentation masks, etc. without task-specific heads requires representing these modalities in shared and unified space. To do this, we discretize the text, images, and other structured outputs in our tasks and represent them with tokens drawn from unified and finite vocabulary.
4

---

Published as a conference paper at ICLR 2023
Text representation. Following Raffel et al. (2020), text inputs and outputs are tokenized using SentencePiece (Kudo & Richardson, 2018). Following past works such as McCann et al. (2018); Raffel et al. (2020); Gupta et al. (2022a); Wang et al. (2022b) we also specify each task with natural language prompt (excluding some tasks like VQA, which are fully specified by their text inputs) in order indicate what task should be performed. For example, "What is the depth map of the image?" for depth estimation or "What region does "cat" describe?" for object localization.
Images and dense structures representation. variety of tasks in computer vision requires the model to produce high-dimensional outputs such as images (e.g. image in-painting) or per-pixel labels (e.g.. depth estimation). To handle these modalities, we first convert per-pixel labels into RGB images. For depth, we construct grayscale image by normalizing the depth map. For surface normal estimation, we convert the x/ x/y/z orientations into r/g/b values. For segmentation, we map each instance present in the image to unique color. We randomly select colors for each instance and specify the color-to-class mapping in the text instead of using universal color-to-class mapping- This avoids requiring a fixed list of classes and avoids having colors that may only be marginally different due to the presence of a large number of classes.
Then we encode these images discrete tokens using VQ-GAN. In particular, we use the maaagenet-pretraiig
VQ-GAN from Esser et al. (2021) with 256 x 256 resolution, a compres- sion ratio of 16, and 16384 codebook size. The VQ-GAN codebook is added to the vocabulary as additional tokens that can be generated by the decoder. During training, the tokens for the target image are used as targets. During inference, the VQ-GAN decoder is used to convert the generated image tokens into an output image.
Sparse structures representation. We encode sparse structures such as bounding boxes hu- man joints by adding 1000 special tokens to the vocabulary to represent discretized image coordi- nates (Chen et al., 2022b). Points are then encoded with sequence of two such tokens, one for the x and one for the y coordinates, and boxes are encoded using sequence of four tokens, two for the upper right corner and two for the lower left corner. Labeled boxes are encoded as a box followed by a text class label, and joints are encoded as a sequence of points followed by a text visibility label. This allows us to handle a wide variety of tasks that use these elements in their inputs or output (see Appendix A. for examples).
3.2 UNIFIED ARCHITECTURE
Universally representing wide variety of tasks input and output sequences of discrete tokens en- ables us to employ architectures that have been proven successful in natural language processing. In UNIFIED-IO, propose a pure transformer model largely following the design of T5 (Raffel et al., 2020). In particular, UNIFIED-IO is an mncoder-decoder architecture where both the encoder and decoder are composed of stacked transformer layers, which in turn are composed of self-attention transformers, cross-attention transformers (in the decoder), and feed-forward neural networks. The layers are applied residually, and layer norms are applied before each transformer and feed-forward network. See Raffel et al. (2020) for details.
We make few architectural changes adapt the T5 architecture to our setting, First, handle input images, we reshape the image into a sequence of patches that are embedded with linear projection similar to Dosovitskiy et al. (2021). Second, we expand the vocabulary to include the location tokens and the image tokens used in the VQ-GAN. Third, we extend the 1-d relative embedding (Dosovitskiy et al., 2021) to 2-d with fixed number of learned embeddings. We also add abso- lute position embedding to the token embedding following Devlin et al. (2019), since the absolute position information is essential to image tasks.
We use maximum of 256 and 128 text tokens for inputs and outputs respectively, and maximum length of 576 (i.e. 24 x 24 patch encoding from a 384 x 384 image) for image inputs and 256 (i.e. 16 x 16 latent codes from a 256 x 256 image) for image outputs. In this work, we present four versions of UNIFIED-IO ranging from 71 million to 2.9 billion parameters, as detailed in Table 2.
3.3 TRAINING
UNIFIED-IO 10 is trained in two stages - A pre-training stage that uses unsupervised losses from text, image, and paired image-text data, and massive multi-task stage where the model is jointly trained
5

---

Published as a conference paper at ICLR 2023
Model
Encoder Layers
Decoder Layers
Model Dims
MLP Dims
Heads
Total Params
UNIFIED- 10 SHALL
8
8
512
1024
6
71M
UNIPIED- BRSSE
12
12
768
2018
12
241M
UNIFIED- 10 LARGE
24
24
1024
2816
16
776M
UNIFIED-10
24
24
2048
5120
32
2925M
Table 2: Size variant of UNIFIED-IO. Both encoder and decoder are based on implementation (Raffel
et al., 2020). Parameters of VQ-GAN (Esser et al., 2021) are not included in the total parameter count.
on a large variety of tasks, Since our goal is to examine whether a single unified model can solve variety of tasks simultaneously, we do not perform task-specific fine-tuning although prior work (Lu et al., 2020; Wang et al., 2022b) shows it can further improve task performance.
Pre-training. To learn good representations from large-scale webly supervised image and text data, we consider two pre- training tasks: text span denoising and masked image denoising. The text span denoising task follows Raffel et al. (2020) randomly corrupt 15% of the tokens and replace the consecutive corrupted tokens with a unique mask token. The masked image denoising task follows Bao al. (2022) and He et al. (2022) - randomly masked 75% of the image patches, and the goal is to recover the whole image. When another modality is present, i.e. image or text, the model can use information from that modality to complete the tasks.
We construct the pre-training dataset by incorporating publicly available language data (i.e., plain texts from Common Crawl), vision data (i.e., raw images from different datasets), and V&L data (i.e.. image caption and image label pairs). For V&L data, we add simple prompt "An image of at the beginning of the caption or categories to indicate it is multi-modal data (Wang et al., 2022d).
We pre- train UNIFIED-IO equally sample data with the text and image denoising objective. For text denoising, half of the samples from pure text data, i.e. C4 and Wikipedia. The other half is constructed from image and class data, such as Imagenet2 k (Ridnik et al., 2021) or image and caption data, such as YFCC15M (Radford et al., 2021). For image denoising, image-only data from datasets for our vision tasks. We sample from datasets in proportion to dataset size. See Appendix A.2 for details.
this combination of datasets with in- mixing strategy. We
also the same caption and class data and some
To build single unified model for diverse vision, language, and V&L tasks, we construct a massive muuli-tasking dataset by ensembling 95 datasets from 62 publicly available data sources. See Section 2 for task details and Appendix A. for dataset visualizations.
Multi-tasking,
We jointly train UNIFIED-IO on this large of datasets by mixing examples from these datasets within each batch. We equally sample each group (1/8) except for image synthesis (3/16) and dense labeling (1/16) since dense labeling has significantly fewer data and image synthesis has significantly more data than other groups Within each group, we sample datasets proportional to the square root of their size to better expose the model to underrepresented tasks. Due to the large variance in dataset size, some tasks are still rarely sampled (e.g. depth estimation only has 0.43% chance of being sampled). See Appendix A.3 for details and visualizations.
Implementation Details. Due to space limitation, see Appendix A.4 for implementation details.
EXPERIMENTS
We now present results for UNIFIED-IO on the GRIT benchmark (Sec 4.1), ablate training data via the GRIT ablation benchmark (Sec 4.2) and evaluate UNIFIED-IO on 16 other benchmarks in computer vision and NLP (Sec 4.3). Appendix A.5 shows the evaluation of the same concept and new concept on GRIT and A.6 shows the prompt generalization. Qualitative examples are in A.9.
4.1 RESULTS ON GRIT
The General Robust Image Task (GRIT) Benchmark (Gupta et al., 2022b) is an evaluation-only benchmark designed to measure the performance of models across multiple tasks, concepts, and data sources. GRIT aims to encourage the building of unified and general purpose vision models and is thus well suited to evaluate UNIFIED-IO. GRIT has seven tasks that cover range of visual skills with varying input and output modalities and formats: categorization, localization, VQA, refer expression, segmentation, keypoint, and surface normal estimation.

---

Published as a conference paper at ICLR 2023
Calegorization
Localization
VQA
Retexp
Segmentatin
Keypoint
Normal
All
ablution
test
ablation
lest
ablution
lest
ablation
test
ablation
test
ablation
test
ablation
test
ablation
test
0 NLL-AngMM [4]
49.6
50.5
7.2
7.1
Mask R-CNN (42)
44.7
45.1
26.2
26.2
70.8
70.6
20.2
20.3
2 GPV-1 [39]
33.2
33.2
42.8
42.7
50.6
49.8
25.8
26.8
21.8
21.8
3 CLIP [87]
48.1
6.9
OFA sooe [108]
22.6
72.4
61.7
22.4
GPV-2 [53]
54.7
55.1
53.6
53.6
63.5
63.2
51.5
52.1
31.9
32.0
UNIFIED-1O AMAEE
42.6
50.4
52.9
51.1
40.7
46.5
33.5
45.4
UNIFIED-IO...
53.1
59.7
63.0
68.3
49.3
60.2
37.5
55.9
UNIFIED-1OLaacE $7.0
64.2
67.4
74.1
54.0
67.6
40.2
60.7
UNIPED-IO.z 61.7
60.8
67.0
67.1
74.5
74.5
78.6
78.9
56.3
56.5
68.1
67.7
45.0
44.3
64.5
64.3
Table 3: Comparison our UNIFED-IO models to recent SOTA on GRIT benchmark. UNIFIED-OO is the first model to support all seven tasks in GRIT. Results of CLIP, OFA obtained from GRIT challenge.
UNIFIED-IO is the first model to support all seven tasks in GRIT. As seen in Table 3. UNIFIED- outperforms prior submissions GRIT obtaining average accuracy of 64.3 on test. The next best submission is GPV-2 (Kamath et al., 2022) which obtains 32.0 and can only support 4 out of 7 tasks. UNIFIED-IO xL. also outperforms the multi-task checkpoint of OFALARGZ (Wang et al., 2022b) on VQA, refer expression and categorization.
Mask R-C R-CNN (He et al., 2017) is a strong baseline for core vision tasks. UNIFIED-IO,, outperforms Mask R-CNN on localization and segmentation. The reason is UNIFID- 1Ox1 shows little degrada- tion in performance between the same concept and the new concept as discussed in Appendix A.5. On keypoint, our model is worse compared Mask R-CNN (68.1 70.8). The reason is we have 2-stage inference for keypoint- - first locate the person using the object localization prompt, then find keypoints for each detected region.
NLL-AngMF (Bae et al., 2021) is a SOTA model for surface normal estimation. Our model gets strong results compared NLL-AngMF (44.3 vs. 49.6). Since our image tokenizer is only pre- trained on ImageNet without any surface normal data, the upper bound of our method through reconstruction is 59.8 on FrameNet (Kazemzadeh et al., 2014). This suggests our score could be considerably improved by training stronger image tokenizer.
4.2 ABLATIONS
To better understand how multi-tasking affects learning, we perform ablations by leaving out individual task groups from multi-task training. Due to computational constraints, we ablate UNIFTED-IO and train for 250k steps. If ablating task group. we reduce the number of training steps so that all models are trained on approximately the same number of examples for each of the remaining task groups. Results are shown in Table 4 on GRIT and MNLI (Williams et al., 2018).
In spite of supporting large number of heterogeneous tasks, Unified-IO is able to perform well across all tasks. Reducing this heterogeneity by removing task groups does not impact the perfor- mance of individual tasks significantly. This is notable since removing task group significantly reduces the scope of what a model needs to learn while keeping the model capacity fixed. This empirically demonstrates the effectiveness of the proposed unified architecture for massive hetero- geneous task support.
An exception that removing the NLP group significantly boosts categorization, which might indi- cate that the sentence classification task interferes with image classification. Removing captioning also boosts performances on VQA and few other tasks, which might be caused by captioning re- quiring relatively large amount of model capacity to learn free-form text generation, in contrast VQA that requires short answer phrases from limited vocabulary. Removing image synthesis causes major regression in keypoint. Manual inspection shows that the model predicts standing- human shaped keypoints even for people in very different postures, suggesting the model learned to rely priors instead of the image content. We also see minor regressions in localization and referring expression, suggesting that image synthesis tasks, possibly image in in-painting in particular, had surprising positive transfer understanding sparse structured outputs. It is possible that an ablation analysis on the XL model may yield different outcomes, but we are unable to perform an xI-baaedd analysis due limited compute.
7

---

Published as a conference paper at ICLR 2023
Model
Step
Categorization
Localization
VQA
Refexp
Segmentation
Keypoint
Normal
MNLI
UNIFED-IOO.oo
250k
50.3
63.4
65.7
73.4
51.8
69.2
40.7
85.1
win Image Synthesis
200k
52.7
62.9
64.2
72.0
53.6
18.1
42.2
84.5
Spunse
220k
52.6
64.1
51.3
38.5
Domse
62.4
65.6
72.9
66.7
84.8
wlo Classification
220k
63.1
61.0
73.7
52.1
66.8
39.1
84.6
wlo Captioning
220k
49.7
65.0
68.0
74.7
54.2
67.4
39.2
85.3
wlo V&L.
220k
50.9
72.5
51.9
70.0
38.2
84.4
64.3
65.9
74.6
52.0
69.1
39.9
Language Modelling
64.7
66.7
70.2
39.9
83.5
Table 4: Ablation study on holding out tasks groups and evaluating GRIT and MNLI (Williams et al., 2018)
NY002
ImagoNay
Placetts
OKVQA
Voe
VieWod
Swig
SALL-VE
Voc
Maggs
CO00
CO0D
ARPC
Rool
SoTal
Spitt Meric
val
val
val
tod-dov
lest
1291
Iest-dev
I0s1
val
val
val
val
val
val
test Acc
RMSE
Acc.
10C
Acc.
CIDER
CIDER
CIDE:
CIDER
Acc
United SUTA
Planingo
Planingo
PaLM
0.467
57.8
49.8
92.20
92.2
UNIEED
42.8
38.2
31.0
24.3
42.4
17.3
45.1
30.1
86.9
65.9
67.4
UNItED-1Owox
41.2
91.8
78.5
45.8
10.7
85.6
66.9
104.0
87.9
70.8
90.8
UNITID-10 10usa
71.8
50.5
67.8
42.7
33.4
47.7
54.7
40.4 86.1
87.2
117.5
87.5
73.1
98.1
UNITEDD 2
79.1
53.2
57.4
126.8
122.3
79.7
95.7
Sieple or doe-
BanFormer CoCa MAE
CoCu
KAT
GPV2
Flaningo MAC-Caps JSL
ORA
SVT
CoCa
OFA
Tuiny NLR
STMOE
DeBERTa
tused SOTA
0,350
91.00
60.3
54.4
38.1
65.7
39.6
145.3
92.4
97.7
results for the 9HALL/BASE//NSS models NYU2 and report results for model retrained with slightly different distribttion tasks, see the Appendix A.8
DDue datuset
able
discussion.
Comparing the jointly trained UNIFIED-IO specialized and unified fine-tuned SOTA models across Vision, V&L and Language tasks. Benchmark datasets are: NYUv2 (Nathan Silberman Fergus, 2012), Ima- geNet (Deng et al., 2009), Places365 (Zhou et al., 2017). VQA 2.0 (Goyal et al., 2017), A-OKVQA (Schwenk et al., 2022), VizWizVQA (Gurari et al., 2018), Viz WizG (Chen et al., 2022a), Swig (Pratt et al., 2020). SNLI- VE (Xie al., 2019). VisComet (Park al., 2020). Nocaps (Agrawal et al., 2019), COCO Captions (Chen al., 2015), MRPC (Dolan Brockett, 2005), BoolQ (Clark al., 2019), and SciTail (Khot et al., 2018).
Table 5:
4.3 RESULTS ON ADDITIONAL TASKS
We report results on 16 additional tasks used in our training setup. For these tasks, we do not expect to get state-of-the-an thrt results since specialized models are usually designed and hyper-parameter tuned for a single task, while we are evaluating a single jointly trained model. We also avoid ex- tensive task-specific tricks like color jittering, horizontal flipping, CIDEr optimization, and label smoothing, which are often responsible for considerable gains in individual task performance. We leave such task-specific tuning for future work. See Table 5 for the results. When possible, additionally report the best prior result on these tasks from a unified model, mcaning a model that trained in multi-task setting and unified architecture (no spec-fp head or customizations) with at least three other tasks.
UNIFIED-IO provides strong performance on all these tasks despite being massively multi-taske We review more fine-graie fine-grained results below.
Depth Estimation. On depth estimation, UNIFIED-IO achieves 0.475 rmse, which is behind SOTA (Li et al., 2022e) but similar the recently proposed unified model, UViM (Kolesnikov et al., 2022), despite being trained do far more tasks. More discussion can be found in A.8.
Image Classification. UNIFIED-IO achieves 79.1 on ImageNet and 53.2 on Places365, showing the model was able to retain the knowledge of many fine-grained classes despite being massively multi-tasked. Notably, we achieve this without the extensive data augmentations methods typically used by SOTA models (Yu et al., 2022a; He et al., 2022).
Visual Question Answering. UNIFIED-IO is competitive with fine-tuned models on VQA (Alayrac et al., 2022; Kamath et al., 2022; Gui et al., 2021), and achieves SOTA results on A-OKVQA. Relative to Flamingo, UNIFIED-IO performs better VizWiz-QA but worse OK-VQA.
Image Captioning. Despite the lack of CIDEr optimization, UNIFIED- 10 is a strong captioning model and generalizes well to nocaps. Since UNIFIED-IO is trained on many captioning datasets, is likely the use of style tags following Cornia et al. (2021) would offer additional improvement by signaling UN to specifically generate COCO-style captions during inference.
8

---

Published as a conference paper at ICLR 2023
NLP tasks.: UNIFIED-IO achieves respectable results three NLP tasks but lags behind SOTA models (Smith et al., 2022; Zoph et al., 2022; He et al., 2021). This can partly be attributed to scale. Modern NLP models contain 100 billion+ parameters and with more extensive NLP pre-training
4.4 LIMITATIONS
For object detection, while UNIFIED-IO generally produces accurate outputs (see Appendix A.9), we find the recall is often poor in cluttered images. Prior work (Chen et al., 2022b) has shown this can be overcome with extensive data augmentation techniques, but these methods are not currently integrated into UNIFEDDIO.. Our use of pre-trained VQ-GAN greatly simplifies our training and is surprisingly effective for dense prediction tasks. However, does mean UNIFIED-IO has limited image generation capabilities (recent works (Yu et al., 2022b) have shown this method can be greatly improved but was not available at the time of development). We also found in a small-scale study that our model does not always understand prompts not in the training data (see Appendix A.6).
5 RELATED WORK
Constructing models that can learn to solve many different tasks has been of long-standing interest to researchers. A traditional approach to this problem is to build models with suecispeei heads on top of shared backbones (He et al., 2017; Liu et al., 2019; Lu et al., 2020). However, this requires manually designing specialized head for each task and potentially limits transfer across tasks. An alternative is to build unified models models that can complete many different tasks without task- specialized components. In NLP, this approach has achieved great deal of success through the use of pre-trained generative models (Raffel et al., 2020; Brown et al., 2020; Chowdhery et al., 2022).
Inspired by this success, there has been a recent trend to build unified models that can be additionally applied to tasks with visual or structured inputs and outputs. Many models have been proposed for tasks with text and/or image input and text output (Cho al., 2021; Wang et al., 2022d: Li al., 2022b; Wang et al., 2021; Kaiser et al., 2017; Sun et al., 2022; Chen et al., 2022d; Wang et al., 2022c). However, these models can not produce any structured or visual output.
More recent unified models can additionally support image locations, which allows tasks like object detection or region captioning. This can be done by using bounding boxes proposed by an object detector (Cho et al., 2021; Kamath et al., 2022) or including a bounding box output head (Gupta et al., 2022a; Dou et al., 2022; Chen et al., 2022c; Kamath et al., 2021; Li et al., 2022d). Alterna- tively, image locations can be encoded special tokens in the input/output text (Yang et al., 2021; Yao et al., 2022; Zhu et al., 2022) following Chen et al. (2022b). UNIFIED-IO follows this design, but applies it to wider set of tasks than previous works, including key-point estimation, image in-painting, and region captioning.
Concurrent to our work, OFA (Wang et al., 2022b) proposes similar approach that also supports im- age locations and text-to-image synthesis. However, OFA does not support dense labeling tasks such as depth estimation, segmentation, and surface normal estimation. Other closely related models in- clude UViM (Kolesnikov et al., 2022) which generates discrete guiding code for a D- VAE to build an autoregressive model for panoptic segmentation, depth prediction, and colorization; Pix2Seq v2 (Chen et al., 2022c) which extends Pix2Seq to segmentation, keypoint estimation, and image cap- tioning; Visual Prompting (Bar et al., 2022) adapts the pre-trained visual model to novel downstream tasks by image inpainting. UNIFIED-IO covers all these tasks, and focuses multi-tasking rather then task-specific fine-tuning. Additional discussions are presented in Appendix A.
6 CONCLUSION
We have presented UNIFIED-IO, a unified architecture that supports a large variety of computer vision and NLP tasks with diverse inputs and outputs, including images, continuous maps, binary masks, segmentation masks, text, bounding boxes, and keypoints. This unification is made possible by homogenizing each of these modalities into a sequence of discrete tokens. The 2.9B parameter UNIFIED-IO XL model is jointly trained on 90+ datasets, is the first model perform all 7 tasks on the GRIT benchmark and obtains impressive results across 16 other vision and NLP benchmarks, with benchmark fine-tuning task-specific modifications.

---

Published as a conference paper at ICLR 2023
REFERENCES
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlci Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. nocaps: novel object captioning at scale. In ICCV, 2019.
Sean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhi- tao Gong, Sina Samangooci, Marianne Monteiro, Jacob Menick, Schastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning. arXiv, 2022. 8.27
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual Question Answering. In International Conference Computer Vision (ICCV), 2015. 4
Gwangbin Bae, Ignas Budvytis, and Roberto Cipolla. Estimating and exploiting the aleatoric uncertainty in surface normal estimation, ICCV, 2021. 3,
Hangho Bao, Li Dong, and Furu Wei. BEiT: BERT pre-training of image transformers. In ICIRR, 2022. 6
Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei A Efros. Visual prompting via image inpainting. arXiv preprint wrXiv:2299006644 2022. 9
Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro Danilo Giampiccolo, Bemardo Magnini, and Idan Szpek- The second pascal recognising textual entailment challenge. In Proceedings of the second PASCAL challenges workshop on recognising textual entailment, volume 6, pp. 6-4, Venice, 2006.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entail- ment challenge. TAC, 2009. 22
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. The SNLI corpus. 2015. 22
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Win- Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NewrlPS, 2020. 1,9
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task 1: Semantic textual similarity-muleen mulliinnnull and cross-lingual focused evaluation. arXiv preprint (rXiv:1708.000 2017. 22
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image- text pre-training to recognize long-tail visual concepts. In CVPR, 2021. 3, 17
Chongyan Chen, Samreen Anjum, and Danna Gurari. Grounding answers for visual questions asked by visually
impaired people. In CVPR, 2022a. 4, 8, 21,
Ting Chen, Saurabh Saxena, Lala Li, David Fleet, and Geoffrey Hinton. Pix2seq: A language modeling framework for object detection. In ICLR, 2022b. 5, 18
Ting Chen, Saurabh Saxena, Lala Li, Tsung- Yi Lin, David Fleet, and Geoffrey Hinton. A unified sequence interface for vision tasks. arXiv preprint (xXxx:2206.0766) 2022c. 9
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint At0: 06794, 2022d. 9
Xinlei Chen, Hao Fang, Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zimicck. Microsoft coco captions: Data collection and evaluation server. arXiv, 2015. 3, 8, 17
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and-language tasks via text generation. In ICML, 2021. 2,
Aakanksha Chowdhery, Sharan Narang. Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts. Paul Barham, Hyung Won Chung, Charles Sutton, Sehastian Gehrmann, al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
10

---

Published as a conference paper at ICLR 2023
Christopher Clark, Kenton Lee, Ming- Wei Chang. Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. NAACL, 2019, 22
Marcella Cornaa, Lorenzo Baraldi, Giuseppe Fiameni, and Rita Cucchiara. Universal captioner: long-tail
vision-and-lange and-language model training through content-style separation. arXiv, 2021. 8
Ido Dagan, Oren Glickman, and Beraaro Magnini. The pascal recognising textual entailment challenge. In
Machine Learning Challenges Workshop, pp. 177-190. Springer, 2005. 22
Marie-Catheriie Marneff, Mandy Simons, and Judith Tonhauser. The commitmentbank: Investigating projection in naturally occurring discourse. proceedings of Sinn und Bedeutung 23, 2019. 22
Deng, W. Dong, R. Socher, L.-J. Li, Li, L. Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009. 3, 8,8,22 20
Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. RedCaps: Web-curated image-text data created by the people, for the people. arXiv, 2021. 3, 17
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding, In NAACL, 2019. 5, 27,
William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In /WP, 2005. 8, 22
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakoh Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Zi- Yi Dou, Aishwarya Kamath, Zhe Gan, Pengchuan Zhang, Jianfeng Wang, Linjie Li, Zicheng Liu, Ce Liu, Yann LeCun, Nanyun Peng, al. Coarse-to-fine vision-language pre-training with fusion in the backbone. arXiv preprint wiXi:220660064... 2022. 9
Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. Searchqa: new q&a dataset augmented with context from a search engine. arXiv preprint 10X.vv:704.05/// 2017, 21
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021. 2, 25
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Bunsol Choi, and Danqi Chen. Mrqa 2019 shared task: Evaluating generalization in reading comprehension. arXiv preprint 19100:191, 2019. 21
Wikimedia Foundation. Wikimedia downloads. URL https:t//// /twppsdd/p/////////////////////. wi med a org. 4, 22
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, Bill Dolan. third pascal recognizing textual
entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pp. 1-9. Association for Computational Linguistics, 2007. 22
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017. 8.21
David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. English gigaword. Linguistic Data Consortium, Philadelphia, 4(1):34, 2003. 4, 22
Liangke Gui, Borui Wang. Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, and Jianfeng Gao. Kat: A knowl- edge augmented transformer for aiioo-and-languagee arXiv, 2021. 8
Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation. In CVPR, 2019. 3, 17
Tanmay Gupta, Amita Kamath, Aniruddha Kembhavi, and Derek Hoiem. Towards general purpose vision systems: An end-to-end lask-agnostic vision-language architecture. In CVPR, 2022a. 2, 7,7,9
Tanmay Gupta, Ryan Marten, Aniruddha Kembhavi, and Derek Hoiem. GRIT: General robust image task benchmark. arXiv, 2022h. 2,6
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffree P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, 2018. 8,
Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask R-CNN. In ICCV, 2017. 2,7,9
11

---

Published as a conference paper at ICLR 2023
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In CVPR, 2022. 6,8
Pengcheng He, Xiaodong Liu. Jianfeng Gao, and Weizhu Chen. DeBERTa: Decoding-enhaneed bert with
disentangled attention. In ICLR, 2021. 9
Ronghang Hu and Amanpreet Singh. Unit: Multimodal multitask learning with a unified transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2021. 2
Jingwei Huang, Yichao Zhou, Thomas Funkhouser, and Leonidas Guibas. Framenet: Learning local canonical frames of 3d surfaces from a single rgb image. In ICCV, 2019a. 19
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos qa: Machine reading comprehen- sion with contextual commonsense reasoning. arXiv, 2019b.
Shankar Iyer, Nikhil Dandekar, and Kornel Csemnai First quora dataset release: Question pairs, 2017. URL https:/ / /aata quoruuuorr....r. com/ htte//irsstttt Quora- -eweeeeeeeeeeeeeeeeeee Ps 22
Andrew Jaegle, Sehastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier J Henaff, Matthew Botvinick, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver io: A general architecture for structured inputs outputs. In ICLR, 2022.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly su- pervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 1601-1611, Vancou- ver, Canada, July 2017. Association for Computational Linguistics. doi: 10.1863//1/11-11447. URL https: : / aacllcccc/ anthol hogy www.ooggyo/oooooooooooooo..o...o wrg/ P1 21
Lukasz Kaiser, Aidan N Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, and Jakob Uszko- reit. One model to learn them all. arXiv preprint 1rXiv:1706.051 05137, 2017. 9
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr- modulated detection for end-to-end multi modal understanding. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 7780-17901 2021. 9
Amita Kamath, Christopher Clark, Tanmay Gupta, Eric Kolve, Derek Hoiem, and Aniruddha Kembhavi. Webly supervised concept expansion for general purpose vision models. arXiv, 2022. 8,
Sahar Kazemzadeh, Vicente Ordonez. Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP, 2014. 3. 7.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay. and Dan Roth. Looking beyond the surface:a challenge set for reading comprehension over multiple sentences. In Proceedings of North American Chapter of the Association for Computational Linguistics (NAACL), 2018. 22
Tushar Khot, Ashish Sabharwal, and Peter Clark. Scitail: A textual entailment dataset from science question answering. In AAAI, 2018, 8, 22
Alexander Kolesnikov, André Susano Pinto, Lucas Beyer, Xiaohua Zhai, Jeremiah Harmsen, and Neil Houlsby. Uvim: A unified modeling approach for vision with learned guiding codes. arXiv, 2022. 8, 94
Ranjay Krishna. Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein. and Fei-Fei Li. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 2017. 3, 17, 20, 21
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In EMNLP, 2018.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJCV, 2020. 3. 17. 21
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Ming-Wei Wei Chang. Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguis- tics, 7:452-466, 2019. doi: 10.1 162//111llaaaa000 00026.. URI. https: : /cc1 anthol ogy .org/Q1 019 9- 1026. 21
12

---

Published as a conference paper at ICLR 2023
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning, 2012. 22
Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hehong Chen, Guohai Xu, Zheng Cao, et al. mplug: Effective and efficient vision-language learning by cross-modal skip-connections. arXiv preprint Xiv:22205 1200, 2022a. 27
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694-9705, 2021. 28
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In JCML, 2022b. 9, 28
Linjie Li, Zhe Gan, Kevin Lin, Chung-Ching Lin, Zicheng Liu, Ce Liu, and Lijuan Wang. Lavender: Unifying video-language understanding as masked language modeling. arXiv preprint arXiv:2206.07/00 160, 2022c. 27
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert:
simple and
performant baseline for vision and language. In arXiv, 2019. 2,27
Liunian Harold Li, Pengchuan Zhang. Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-iman pre-training. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 10965-10975, 2022d. 9
Zhenyu Li, Xuyang Wang, Xianming Liu, and Junjun Jiang. Binsformer: Revisiting adaptive bins for monoc- ular depth estimation. arXiv. 2022e. 8
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Shengtong Mo, Dani Yogatama, Louis-Philippe Morency, and Rus-
lan Salakhutdinov. Highmmt: Towards modality and task generalization for high-modality representation learning. arXiv preprint 22X3v:2203.01311, 2022. 27
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zimicck Microsoft coco: Common objects in context. In ECCV, 2014. 3, 17,
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for nat- ural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pp. 4487-4496. Association for Computational Linguisties, 2019. URL https:
// //wwe. a web. wtg. antttntthhoooooooooooooooo
ogy/ hot.ttoooooooooooooooooooo 9- 1441.9 9
Cewu Lu, Ranjay Krishna, Michael Bersstein, and Li Fei-Fei. Visual relationship detection with language priors. European Conference on Computer Vision, 2016. 4
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic repre- sentations for vision-and-laree languuge tasks. In NeurlPS, 2019. 2,27
Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, D. Parikh, and Stefan Lee. 12-in-1: Multi-task vision and language representation learning. CVPR, 2020. 6, 9
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learn- ing word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http: www. aclweb org/ anthoogyy/ P11-1015. 22
Junhua Mao, Jonathan Huang, Alexander Toshev. Oana Camburu, Alan Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object deseriptions. In CVPR, 2016. 18
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Conference on Computer Vision and Pattern Recog- nition (CVPR), 2019. 21
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. arXiv, 2018. 5
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018. 22
Pushmeet Kohli Nathan Silherman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from RGBD images. In ECCV, 2012. 8, 19, 24
13

---

Published as a conference paper at ICLR 2023
Jae Sung Park. Chandra Bhagavatula. Roozbeh Mottaghi. Ali Farhadi, and Yejin Choi. Visualcomet: Reasoning about the dynamic context of a still image. In In Proceedings of the European Conference on Computer Vision (ECCV), 2020. 8, 21
Zhiliang Peng. Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers. ArXiv, atlp:aaa..222522.2 06366, 2022, 28
Mohammad Taher Pilehvar and os'e Camacho-Collados Wie: 10, 000 example pairs for evaluating context-
sensitive representations. CoRR, 001/1808.09121, 2018. URL attp://arzie org/ aas///bss/....... 09121. 22
Axel Pinz et al. Object categorization. Foundations and Trends in Computer Graphics and Vision, 1(4): 255 353, 2006.
Sarah Pratt, Mark Yatskar, Luca Weihs, Ali Farhadi, and Aniruddha Kembhavi. Grounded situation recognition. In ECCV, 2020. 8
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. 6, 7, 23,27
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yangi Zhou, Wei Li. and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020. 2, 4, 5, 6, 9, 21, 22, 24
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In EMNLP, 2016. 4, 21
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv, 2022. 17
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth- Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg. Tom Eecles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. arXiv, 2022. 27
Anthony D Rhodes, Max II Quinn, and Melanie Mitchell. Fast on-line kernel density estimation for active object localization. In 2017 international joint conference on neural networks (IJCN), pp. 454-462. IEEE, 2017. 3
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor, Imagenet-21k, pretraining for the masses. arXiv, 2021. 6, 20, 23
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evalu- ation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series. 2011. 22
Alexander M. Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence
summarization. Proceedings of the 2015 Conference on Empirical Methods in Natural Langugee Processing, 2015. doi: 10.18653/v1/d15- 1044. URL http: //tp:///xxxx dx doi wwwp://oooo..ooooooooooo//oo/ote
22
Dustin Schwenk, Apoory Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. arXiv, 2022. 8, 21
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Interna- tional Conference on Machine Learning, pp. 4596-4604. PMLR, 2018. 25
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon. Wojeiech Galuba. Marcus Rohrbach, and Douwe Kiela. Flava: foundational language and vision alignment model. Proceed- ings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 15638-15650, 2022. 28
Shaden Smith. Mostofa Patwary, Brandon Norick. Patrick LeGresley, Samyam Rajbhandari. Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zheng, Rewon Child, Reza Yaz- dani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shocybi, Yuxiong He, Michael Houston, Saurahh Tiwary, and Bryan Catanzaro. Using deepspeed and megatron train megatron-turing nig 530b, a large- scale generative language model. arXiv, 2022. 9
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceediggs of the 2013 conference on empirical methods natural language processing, pp. 1631-1642, 2013. 22
14

---

Published as a conference paper at ICLR 2023
Benyuan Sun, Jin Dai, Zihao Liang, Congying Liu, Yi Yang, and Bo Bai. Gppf: A general perception pre- training framework via sparsely activated multi-task leamnng. arXiv preprint arXiv: 02/40, 2022. 9, 28
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encader representations from transformers, In EMNLP, 2019. 2, 27
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. NewsQA: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Represen- tation Learning for NLP, pp. 191-200, Vancouver, Canada, August 2017. Association for Computational URL
Linguistics. doi:
org /W17- 21
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pictro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In CVPR, 2018. 20
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: multi-task benchmark and analysis platform for natural language understanding. arXiv, 2018. 22
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpat language understanding systems. In NeurlPS, 2019. 21,22
Jianfeng Wang. Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205. 2022a. 27, 28
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalitics through simple eequence-to-sqquene learning framework. arXiv, 2022b. 5, 6, 7,9, 21,
Wenhui Wang, Hangho Bao, Li Dong, and Furu Wei. Vlmo: Unified vasion-language pre-training with mixture- of-modality-expett. arXiv preprint arXiv:2111.022
2021.
Wenhui Wang, Hangho Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image foreign language: Beit pretraining for all vision and rision-langusee tasks. arXiv preprint arXiv:2208, 10442, 2022c. 28
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. In ICLR, 2022d. 6, 21,
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint arXiv: /805. 12471, 2018. 22
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongic, and P. Perona. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-001, California Institute of Technology, 2010. 20
Adina Williams. Nikita Nangia. and Samuel Bowman. A broad-coverage challenge corpus for sentence un- derstanding through inference. In Proceedings the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Pa- pers), pp. 1112-1 122. Association for Computational Linguistics, 2018. URL httt:://accwweeeeee o anthology /N 8- 8-18--111-00 1101. 4,7,8 7,8,22 22
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: scene recognition from abbey to zoo. In CVPR, 2010. 20
Large-scale
Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine-grained image understanding. arXiv, 2019. 8, 21
Zhengyuan Yang. Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. Unitab: Unifying text and box outputs for grounded vision-language modeling. 2021. 9
Zhilin Yang, Peng Qi. Saizheng Zhang, Yoshua Bengio, William Cohen. Ruslan Salakhutdinov, and Christo- pher D. Manning. HotpoiQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 2369- 2380, Brussels, Belgium, October-November 2018. Association for Computational Linguisties. doi: 10. 10.1866//1/D18 1259. URL https: httttp::/aclaatthtttt http///aaaaaaaooaya../... wrg/po/g//00 DDDD-111100 112-- 21
Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blended- mvs: A large-scale datasct for generalized multi-view stereo networks. In CVPR, 2020. 19
15

---

Published as a conference paper at ICLR 2023
Yuan Yao. Qianyu Chen, Ao Zhang, Wei Ji, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Pevl: Position- enhanced pre-training and prompt tuning for vision-language models. arXiv preprint 1rXi::2205111169 2022. 9
Mark Yatskar, Luke Zettlemoyer, and Ali Farhadi. Situation recognition: Visual semantic role labeling for image understanding. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5534-5542, 2016. 21
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung. Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Con- trastive captioners are image- text foundation models. arXiv preprint arXiv: 2022a. 8, 27, 28
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022b. 9
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432. 2021. 28
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition cognition: Visual commonsense reasoming. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp- 6720-6731, 2019a.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentenee? In Proceedings of the 57th Annual Meeting of the Association for Computational
Linguistics, 2019b. 22
Rowan Zellers, Ximing Lu, Jack Hessel, Youngjac Yu, Jac Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. Advances in Neural Information Processing Systems, 34:23634-2366 2021. 27
Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and lan- guage and sound. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11375-16387 2022. 27
Yuan Zhang, Jason Baldridge, and Luheng He. PAWS: Paraphrase Adversaries from Word Scrambling. In Proc. of NAACL, 2019. 22
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. TPAMI, 2017. 8, 20
Chaoyang Zhu, Yiyi Zhou, Yunhang Shen, Gen Luo, Xingjia Pan, Mingbao Lin, Chao Chen, Liujuan Cao,, Xiaoshuai Sun, and Rongrong Ji. Sequr: simple yet universal network visual grounding. arXiv preprint arXiv:2203.16265 2022.
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. Designing effective sparse expert models. arXiv, 2022. 9
16

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 TASKS DETAILS
UNIFIED-IO is jointly trained on a large and diverse set of vision, language and vision language tasks. In this section, we describe these tasks in detail and show the prompts we use during training and inference (text on the left of example cards). We also provide qualitative examples both the ground truth and the predictions made by UNIFIED-IO.
A.1.1 IMAGE SYNTHESIS TASKS
Image Synthesis from Text. This task requires generating an image that matches a sentence. Train- ing data comes from 4 captioning datasets: COCO Caption (Chen et al., 2015), Conceptual Captions 3M and 12M (Changpinyo et al., 2021), and RedCaps (Desai et al., 2021) as well datasets used for image classification using the object class the input caption. Specialized image generation models like DALL-E 2 (Ramesh et al., 2022) use an order of magnitude more data, but we limit our sources to these sets for training efficiency.
IMAGE GENERATION
PRED PREDCCTION
What is the complete image? small tiled bathroom"
TRUTH
Text:
Image Inpainting. This task requires filling in region of an image with target object. Training data for this task is built from object bounding box annotations from Open Images (Kuznetsova al., 2020), Visual Genome (Krishna et al., 2017) and COCO (Lin et al., 2014). For each object, the input image becomes the source image with the object's 's bounding box blanked out. The input prompt provides the bounding box's location and the target category. The target output is the original image.
INPAIN PPPIIIING TING
PREDIC TION
IMAGE
regian "lc257
loc425 loc575
loc758" person"?
Image Synthesis from Segmentation. This task involves generating image that matches an input semantic segmentation, i.e., set of class labels for some or all of the pixels in the image. UNIFIED-IO is trained for this task using segmentation annotations from COCO (Lin et al., 2014), Open Images (Kuznetsova et al., 2020), and LVIS (Gupta et al., 2019) as input. Following the method from Section 3.1 the segmentation input is converted into RGB image paired with prompt listing the tolr-ss- mapping, and the target output the source image.
SEG BASED IMAGE GENERATION
What is the complete image? Segmenntation color: "white: knob, silver: cupboerd, olive: drawer,
PRED ICTION
TRUTH
A.1.2 SPARSE LABELLING TASKS
Object Detection. UNIFIED-IO trained on object detection annotations from Visual Genome, Open Images, and COCO. For this task the input is static prompt and an image, and the output
17

---

Published as a conference paper at ICLR 2023
text includes the bounding boxes and class names of all objects in the image. We randomize the order the output objects during training, but for simplicity leave integrating more complex data- augmentation techniques (Chen et al., 2022b) to future work.
OBJECT DETECTION
Vi
loc100 14c745 loc495
PREDICTION
What objects are the Image?? ?
1oc991 chair loc293
TRUTH
loc753 loc763
bed loc262 loc103
loc841 loc109% bicycle
outgars
Object Localization. Object localization requires returning bounding boxes around all objects of a given category. Training data derived from our object detection training data by constructing training example from each category of objects present in image. The imput is then the image, prompt specifying the target class, and the output is list of all boxcs that contain an instance that class. The class for each box (which is always the class specified in the prompt) is included in the output for the sake of keeping the output format consistent with the object detection output. Object localization
input categories which negative samples by randomly selecting categories not present in the image to use as input, in which case the output an empty sequence.
present in the image. To handle this, we construct
LOCAL LOCALIZATIO ZATIONAIONN
PRED CTION
OBJECT
loo431 loc207 loo827 loc325 giraffe
TRUTH
Which ragions does the text "giralle"
loc109 loc337 loc934
loc954 giraffe
baxt outputt on stred for brevity
Referring Expression Comprehension. The task requires the model to localize image region described by natural language expression. The annotation is similar to Object Localization, except that the target is specified with natural language expression instead of class name. Datasets for this task include RefCOCO (Kazemzadeh et al., 2014), RefCOCO+ (Kazemzadeh et al., 2014) and RefCOCOg (Mao al., 2016).
EXPRESSIONS
REFERRING
PREDIC TION
Which rergiont dloe solid black" describe ?
TRUTH
loc270 loc856
loc6-10 loc952
Keypoint Estimation. Keypoint estimation requires returning the location of keypoints on human body (e.g., eyes, nose, feet, etc.) for each person in an image. While it is possible to perform this task in one pass by listing the keypoints of all people in the image in single output sequence, this result in an extremely long output sequence, so UNIFIED-IO uses multi-step approach instead. To do this UNIFLED-IO is trained to complete the subtask of detecting the keypoints for single person in given region. For this subtask, the input prompt specifies the target region and points (a pair of locations tokens for the r and coordinates) along with visibility labels (1 for not visible, 2 for partly visible, for fully visible). Non-visible points are preceded by two copies a new special tokens that indicate there are no valid coordinates. The keypoint metric does not award points for correctly identifying non-visible points, during inference we mask that special token so the model makes a best-effort guess for the coordinates of every single point. Training data for this subtask comes from COCO human pose data (Lin et al., 2014) with the ground-truth person regions as input. During inference we locate person regions using the object localization prompt, then apply UNIFIED-IO again find keypoints for each detected region.
and the output a list
18

---

Published as a conference paper at ICLR 2023
foc307 for487 1oc299
ESTMATION MATION
KEYPOINT
joints the region
Find human
toof?9 1oc295 foc181
1oc315 loc507 toc305
TRUTH
PREDICTION
"loc660 loc375 lo:726 1oc545".
foo453
foc359 toof focd57
loc527 1oc429 loc387
NNxt sulpoii coactted Aor briwiyy
A.1.3 DENSE LABELLING TASKS
Object Segmentation. Object segmentation requires finding the binary segmentation mask of each instance a particular category in an image. The input an image and prompt that includes target class, while the output is an RGB image with black background and instances of that class filled with unique colors following the method in Section 3.1. The output image resized to match the input image if needed using nearest-neighbo resizing method, and binary masks built from each unique color. In practice the output image from UNIFIED-OO can have slightly non- uniform colors extraneous background pixels, likely due to limitation in what the D-VAE can decode/encode, so the output pixels are clustered by color and and connected components of less than 8 pixels removed to build cleaned instance masks. Segmentation annotations from Open Images LVIS, and COCO.
ORJECT TEGMENTATION
PRED ITIION
What segrientation "spple"? ?
Depth Estimation. Depth estimation requires assigning each pixel in an image depth value. This task uses static prompt input, and the output is grayscale image representing the normalized depth at each pixel. The generated output image is reiszed to the same size as the input image and then pixel values rescaled to the maximum depth in the training get output depth map. Training data comes from the NYU Depth Dataset V2 (Nathan Silberman & Fergus, 2012).
ESTIMATION
PREDICTION
DEPTH
What depth map Image?
TRUTH
Surface Normal Estimation. UNIFIED-IO is trained on FrameNet (Huang et al., 2019a) and Blend- edMVS (Yao et al., 2020) surface normal estimation datasets. For this task the input is static prompt and an image and the output is an RGB representation of the x r/u/z orientation of the surface at each pixel. The generated output image resized match the input image and converted back to x x/u/z orientations to produce the final output.
SURFACE NORMAL EST ESTIMATION IITION
PREDICTION
What is the surface the image?
19

---

Published as a conference paper at ICLR 2023
A.1.4 IMAGE CLASSIFICATION TASKS
Image Classification. UNIFIED-IO trained on image classification datasets: ImageNet 2012 (Deng et al., 2009), ImageNer21k (Ridnik et al., 2021), Places365 (Zhou et al., 2017). Sun397 (Xiao et al., 2010), iNaturalist (Van Horn et al., 2018) and Caltech birds 2011 (Welinder et al., 2010). For this task the input is image and static prompt, and the output is a class name. During inference we compute the log log-probability turn the highest scoring one. This ensures UNIFIED-IO does not return category from different categorization dataset that is synonym or hypemny of the correct label.
each class label in the dataset being evaluated and re-
CLASSIFICATION
PREDICTION CTIION
IMAGE
What image?
snake
snake
Object Categorization. This task identifies which label, from given set, best corresponds to an image region defined by an input image and bounding box. The input is the image, prompt the output
specifying the image region
target class name. We convert object detection annotations from Visual Genome, Open Images, and COCO for this task. Inference is constrained to return a valid label for the target label set just as with image classification.
CL CLASSIFICATION
IN CONTEXT
PREDICTION CITIOON
What catagory of the region "occ389 loc loc684 loc684"
TRUTH
horse
A.1.5 IMAGE CAPTIONING TASKS
Image Captioning. Image captioning data comes from the same manually annotated and unsuper- vised sources used for Image Generation. In this case the inputs and output reversed, the input an image and the static prompt, and the output is caption that matches the image.
CAPTIONING
PREDICT ON
MAGE
What does the image describe
A clock that hanging undermeeth glass arch
TRUTH
large clock large room with gless celling
Region Captioning. Region captioning tasks model with generating caption that describes specific region in the image. Our format for this task is identical to Image Captioning except the region included in the input prompt. Visual Genome (Krishna al., 2017) used for the training
data.
CAPTIONING
PREDIC TION
REGION
What does the region
parson falling the water
TRUTH
a man falling in the water
1o0780 loc323 loc 1001 descrbe
20

---

Published as a conference paper at ICLR 2023
A.1.6 VISION & LANGUAGE TASKS
Visual Question Answering. UNIFIED-IO trained collection of VQA datasets including VQA 2.0 (Goyal et al., 2017), Visual Genome, VizWizVQA (Gurari et al., 2018), OKVQA (Marino al., 2019) and A-OKVQA (Schwenk et al., 2022). For VQA, the question is used as the prompt, and the output is the answer text. For VQA, it is common to constrain the model to predict an answer from fixed last of common VQA answers (Wang et al., 2022b;d) during inference, but we avoid doing this since we find it does not benefit UNIFIED-IO in practice.
We additionally convert data from several other datasets VQA format, including imSitu (Yatskar al., 2016), where we treat predicting the verb and then the related slots separate VQA ques- tions, VisualCOMMET (Park et al., 2020) where we convert the before/after/intent into questions by converting the input regions into location tokens, SNLI-VE (Xie et al., 2019) where integrate the entailed text into an input question, and VCR (Zellers et al., 2019a) where we again integrate the input regions into the prompt by encoding them with location tokens and integrate the rationales into the target text for the answer justification task.
VISUAL QUESTION ANSWERING
PREDICTION
How many
ANZA
TRUTH
Grrundd Visual Question Answering. This task requires both answering a question and returning binary mask specifying the region of the image used answer the question. The format for this task follows the one for VQA except that a binary mask is also used as an additional output. Training data comes from VizWiz-VQA (Chen al., 2022a), dataset designed to train models that could benefit people with visual impairments.
PREDICTION
What color this band
TRUTH
gold
gold
Relationship Detection. This task requires predicating a relationship between a pair of objects which are grounded by bounding boxes. The prompt contains both the object regions, and the output is the predicted predicate. There are 2 datasets in this tasks: Visual Genome (Krishna et al., 2017) and Open Images (Kuznetsova al., 2020).
PRED PREDICTION CTIION
RE AATTON
What is the ralationship between loc245 1oc291 loc351 loc379 flower and" loc379 loc223 loc1059 foc5.5
PRED IITIIOIO
flower
TRUTH
flower
A.1.7 NATURAL LANGUAGE PROCESSING TASKS
Question Answering. Following prior work in natural language processing (Raffel et al., 2020), QA tasks are formatted by placing both the question and any text context (e.g., paragraph containing the answer) into the prompt and training the model to generate the text answer. UNIFIED-IO IO is trained several QA datasets including SQuAD 2.0 (Rajpurkar al., 2016), other training datasets from the MRQA (Fisch et al., 2019) shared tasks (Trischler et al., 2017; Joshi et al., 2017; Dunn al., 2017; Yang al., 201 8; Kwiatkowski al., 2019), QA datasets from SuperGI (Wang
21

---

Published as a conference paper at ICLR 2023
et al., 2019; Clark et al., 2019; Khashabi et al., 2018; Roemmele et al., 2011), Cosmos QA (Huang et al., 2019b), OpenBookQA (Mihaylov et al., 2018), and HellaSwag (Zellers et al., 2019b). the text context is longer then maximum sequence length following Devlin et al. (2019) which exposes the model to different windows of text from the context and returns the highest-confidence answer.
blddng-window approach
ANSWERING MOSSSSS NO
context: Uptake of from the air the esseruial purpose respiration, oxygen suoplementation is used medicine.
PREDICTION TION
oxygen supplementation
oxygen supplementation
Treatmant not only incresses oxygan lovals the patient's blood....
question: What medical treaément is used increase oxygen uptake in patient?
Text Classification. Also following past work (Raffel al., 2020), text classification tasks formatted by placing the input sentences and query in the prompt and training the model to generate the target class. Datasets include tasks from GLUE and SuperGLUE (Wang et al., 2018; 2019; Warstadt et al., 2018; Socher et al., 2013; Dolan & Brockett, 2005; Iyer et al., 2017; Cer et al., 2017;
Williams et al., 2018; Dagan al., 2005; Bar-Haine al., 2006; Giampiccolo al., 2007; Bentivogli et al., 2009; Levesque et al., 2012; Williams et al., 2018; De Marneff et al., 2019; Pilehvar & os 2018), as well as SNLI (Bowman et al., 2015), SciTail (Khot al., 2018), IMDB Reviews (Maas et al., 201 1), and PAWS (Zhang et al., 2019).
TEXT CLASSIFICATION
context: Swanses striker Lee Trundie has negocisted
PREDICTION TION
imeege-rights the League One club. Trundlee in business with the League One club. the
TRUTH
entails
ennaill
this following sentence?
Text Summerization. Text summarization is done again by providing the input paragraph and prompt as input and generating summary output. We use the Gigaword dataset (Graff al., 2003; Rush et al., 2015) for training data.
SUMMERIZATION
context: Lankan president Chandhika Kumaratunga called on the countrys private the establishment towrehips while pledging to provide govermment essistance for such ventures, official redio Seturday.. question: What short summary this document?
P PREDICTION
TE
Lankan president calls for privete sector investment
TRUTH
Lankan president cails for privete sector investment in townships
A. 1.8 LANGUAGE MODELING TASKS
Mask Language Modeling, Following T5 (Raffel et al., 2020), the mask language modelling ob- jective randomly samples and then drops out 15% tokens in the input sequence. consecutive spans of dropped-out tokens replaced by single sentinel token. The target is to recover the dropped tokens given the sentinel token. We use C4 (Raffel et al., 2020) and Wikipedia (Founda- tion) datasets.
Buy dlddel BBenBusn ygaey)
PREDICTION
Reginners <axtra, Jd. 1000> Taking Place in Mussolaf you get better
<extra _d__ 1000> BBQ
TRUTH
id 1000> BBQ
making delicious BBQ? You have jdltro_dd_1100 1001> this on your calendar now
cextra id 1001> the opportumit,
<extra id 1001>
date
22

---

Published as a conference paper at ICLR 2023
pikiadia
RadCapa
G4
Image Caption
CCON
Text
Emagenst
Text Deneising
Image Caas
Imaggnnd
Image
Flaces265
Class
VFCC
Castion
Imagenet 21k
RedCaps
CC12M
Pre-training objectives (inner circle), annotation types (middle circle), and datasets (outer cirele) used in pre- pre-training of UNIFIED-IO. Sizes correspond to the sampling rate in the training distribution. Best viewed color.
Figure 3:
A.2 PRE- TRE-TRAINING DATA DISTRIBUTION
Figure shows visualization of pre-training data distribution used by UNIFIED-IO. As discussed Section 3.3, we equally sample data with the text denoising and image denoising objective (in- circle of Figure 3). For text denoising, half of the samples Wikipedia. The other half is constructed from image and class, such as Imagenet21k (Ridnik et al., 2021) or image and caption, such as YFCC15M (Radford et al., 2021). For image denoising, the text information when class and caption are present in the data source and sample the dataset pro- portional to the dataset size. For both text and image denoising, randomly drop both modalities 10% of the time if both text and image as inputs.
from pure text data, i.e. C4 and
23

---

Published as a conference paper at ICLR 2023
CSCO cijene
Open prsapee Cloodss
Coeee 000000000
On honae coppre
MC Olsocs
Grean leagees Sergnemsation
COOD Seggommution
emageeee 2012
LSFS Singiiii Siggmmmttoon
CCAN
Cewning--eeee Exprevaioo
Imauge topaering
ResiCren
Reewinn
Oileect sigmertattoo
ReCiogs
emagieet 22).
Imsge Spediaai som Text
Witely Rupernied
Letating
Capporong
Goagee Synthusii
Sparde
VS Regin
YROC LBM
Image Captoning
Regoon Capaloving
Superifoos Caplamiing
NLP Unsuptreeed
image claysdicppo
Wapela
Viem Larguag
image Claselicaiin
Ninganat 21A
Masied Langiagh Maddingg
Olypcd Gair gregraaann
Phanectes
Maggaed 2012
Foe Summmartatiin
CA
Toxt clesniccaiin
Can snaper Olperzs
Aspurate
Dijenns
Orgeoodd
Doen aane
VON
Nerin Gonann
Seweth 08
DJ
Naw 08
Pive
Figure 4: Task groups (inner circle), tasks (middle circle), and datasets (outer circle) used multi-task training of UNIFIED-IO. Sizes correspond to the sampling rate in the training distribution. Best viewed in color.
A.3 MULTI-TASKING DATA DISTRIBUTION
Figure 4 shows a visualization of the multi-task training distribution used by UNIFIED-IO from Table 1. As discussed in Section 3.3, we equally sample each group (1/8) except image synthesis (3/16) and dense labeling (1/16) since dense labeling has a much smaller sample size compared to image synthesis. We sample tasks and datasets (middle and outer circle) with temperature-sealed mixing strategy to make sure the model is sufficiently exposed to underrepresented tasks. We raise each task's 's mixing rate to the power 1/T and then renormalize the rates so that they sum 1. Following Raffel et al. (2020), we use T = in our experiments.
Due to the large variance in dataset size, some of the tasks are rarely sampled. For example, the depth estimation task only has the NYU Depth dataset source (Nathan Silberman & Fergus, 201 2) and thus the sampling rate is only 0.43%. However, the model still works well for depth estimation tasks, even outperforming concurrent work (Kolesnikov et al., 2022) (0.385 0.467 RMSE). We suspect the large model capacity and masked image denoising pre- training improve the performance. Similarly, Grounding VQA (Chen et al., 2022a) has 0.15% sample rate, but the model can still achieve state- of-the-art performance on this task partly because is trained on many related datasets for VQA and segmentation.
24

---

Published as a conference paper at ICLR 2023
A.4 IMPLEMENTATION DETAILS
The total vocabulary size is 49536, with 32152 language tokens, 1000 location tokens, and 16384 vision tokens. We use the imagenet pretrained VQ-GAN checkpoints with 16384 tokens and s = 16'. Please refer (Esser et al., 2021) During training, we random sub-sample 128 image patches for pre-training state and 256 image patches (out of 576) for multi-task stage. We do not use dropout. Adafactor (Shazeer & Stern, 2018) optimizer is used to save memory. We use learning rate of 10-2 for the first 10,000 steps and then decay rate of 1/vk. We train with 81 0.9 and 82 1.0 - A 0-0.8_ where & is the step number. We use global norm gradient clipping with 1.0 and find this is crucial batch size of 204 8 and XI. with batch size of 1024 due to memory consideration. 4-way in-layer parallelism and 128-way data parallelism were used scale the 3B model training. For all models, we train 1000k steps 500k for pre- pre-training and multi-task training respectively.
stabilized XL training. We train the Small 1, Base and La agre with
A.5 EVALUATION ON THE SAME CONCEPT AND NEW CONCEPT
Categorization
Localization
VQA
Refexp
Segmentation
Keypoint
Normal
restricted
perams (M)
same
new
same
new
same
BEW
same
DOW
samao
nEw
same
DOW
same
now
C ALg-
72
50.7
Mask R-CNN
58
51.9
40.8
44,9
0.3
70.9
2 GPV-1
236
58.7
0.8
48.3
37.8
58.4
74.0
29.7
23,1
CLIP
302
49.1
46.7
OFALARSE
473
28.9
15.8
74.9
88
63.4
58.5
GPV-2
370
85.0
13.5
54.6
54.2
69.8
81.7
57.8
48.3
UNIFIDD-IO, 5
71
52.9
31,9
47.5
59.0
72.5
54.2
37.4
48.5
46.6
33.6
UNIFEDDIOO 1O
241
60.3
47.5
57.9
68.4
68.0
81.8
72.5
62.2
45.8
57.2
60.2
37,7
UNIFIED-IO LOan
776
63.0
52.7
79.2
62.2
UNIFIED-IO.:
2925
90.2
64.2
45.1
Table 6: Generalization new concepts on the GRIT ablation
GRIT provides a breakdown of metrics into two groups: same for samples that only contain concepts seen the primary training data (a set of common datasets like COCO, ImageNet, and Visual Genome), and new for samples containing at least one concept unseen in primary training data. Table shows results for UNIFIED-IO and other leaderboard entries for the ablation set, divided into the same and new concepts.
UNIFLED-IOxt. shows little degradation in performance between same and new, compared to com- peting entries. On some tasks UNIFIED-IO is even able to outperform the new split compared to the same. This indicates that the volume of training data used to train UNIFIED-IO has a broad coverage of concepts, and provides almost effective level of supervision as provided by large standard vision datasets like COCO. Furthermore, since UNIFIED-IO is a uniquely unified architec- ture with task-specific parameters, it is very likely able to effectively transfer knowledge across different tasks.
In comparison Mask-RCNN (row 1), GRIT metries show UNIFIED-IO (row 14) is better by large margin on new concepts, i.e., non-COCO examples (74.4 vs 40.8 for localization and 64.2 vs 0.3 on segmentation), but is still superior the COCO-like examples (65.6 vs 51.9 for localization and 53.0 vs 44.9 on segmentation). UNIFIED-IO is also able to beat GPV-2 (row 5) on new concepts by large margins across all 4 tasks supported by GPV-2 even though GPV-2 is exposed these concepts via webly supervised data and is designed to transfer concept knowledge across skills.
A.6 PROMPT GENERALIZATION CASE STUDY
To better understand how different prompts affect UNIFIED-IO, we do a case study on referring expressions. In particular, we re-evaluate UNIFIED-IO the GRIT referring expression ablation set while replacing the prompt used during training (first row the table) with paraphrase (following rows). Results are shown in Table 7.
25

---

Published as a conference paper at ICLR 2023
Prompt
Refexp Score
0 Which region does the text REE EXP descrriee ?
78,9
1
Which region does the text "REFEXP" describe?
76.7
2 Which region matches the text REFEXP "
77.4
3 Locate the REFEXE EX2
65.6
Which region be described REFEXP "?
64.8
5 Locate the region described by "RFEXP"" ",
43.2
6 Where is the REFEXP ?
41.5
7 Where is the "REFEXP"?
Table 7: Case study on GRIT referring expressions using different prompts. The first prompt is the one used during training, the others are paraphrases. REFEXP is replaced by the referring expression text of individual examples during evaluation.
Overall, we find that the model has some capacity to generalize to paraphrases of the prompt (e.g., row 3 works reasonably well despite using completely different words), but there are paraphrases that result in a very significant performance decrease (e.g. rows 5, 6, and 8). We also find removing the spaces around the punetuation sometimes results minor regressions (row vs row 1) and sometimes in sharply reduced performance (row 6 vs row 7), showing UNIFIED-IO IO can be sensitive to formatting details. We hypothesize that this is caused by the SentencePicce tokenizer changing the tokenization of the referring expression if the quotes multi-task models that can generalize to different prompts, and ideally to prompts for completely new tasks, an exciting avenue for future work.
separated from it by spaces. Building
A.7 CROSS TASK GENERALIZATION CASE STUDY
Qualitative examples for UNIFIED-IO when applied to two out-of-domain settings, surface normal detection on COCO images and animal pose estimation, are included in Figure 5 with the other qual- itative examples. For surface normal detection, we find that the model produces plausible images even for objects like cats or humans that are not in the surface normal training data. However, other scenes, such outdoor scenes (Figure 5 bottom right), are less coherent.
Despite not being trained on animal pose estimation, UNIFIED-IO is able to sometimes find animal keypoints. For animals standing crouching on two legs, the keypoints are reasonably accurate (first two images), however for animals standing on four legs the model will find leg and eye points but then guess arm positions that would make sense for person instead of attaching points to the other legs. While this hints that the model was able to combine skills learned from human pose estimation data with the knowledge of animals learned from other tasks, also shows that more work is needed to fully realize this potential.
A.8 NYUv2 RESULTS
The first version of UNIFED-IO multi-tasking data distribution contains two sources of depth dataset. nyu.depth.v2 from Tensorflow Dataset and a pre-processed version from sparse-to-dense pytorch'" Since the original NYUv2 dataset has lot of O-distance re- gions (holes) which can be problematic for sequence training, we included the latter source because replaces the O-distance holes with approximations. The second version of UNIFIED-IO model trained for code release is trained on an updated multi-tasking data distribution that only con- tains the Tensorflow source of the NYUv2 dataset, and we find significant drop in performance for XL model 0.475 vs. 0.385 while other tasks maintain similar performance. We suspect the rea- son is aparse-to-dent NYUv2 evaluation. Our final result is little worse compared to UViM 0.475 vs. 0.467.
pyt mytorco orch has a different split and contaminates the training data for
depth v2
http://aaaaeets. lids. mit
26

---

Published as a conference paper at ICLR 2023
ANIMAL POSE ESTIMATION
Find the animaa joints this regian.
MPPIT
PRED.CTIO
COCO SURFACE NORMAL ESTIMATION
What is the surface normal of the image?
Qualitative examples on out-of-domain inputs. We directly evaluate UNIFIED-1Oxa on animal pose estimation and surface normal of human or outdoor seenes.
Figure 5:
A.9 QUALITATIVE EXAMPLES
Here we present qualitative examples of predictions from UNIFIED-IO for all training tasks. For brevity, if prompts are identical for each example we only show the prompt once, and the prompt follow the same template for each example we show the template with parts that would be substi- tuted with different words or location tokens underlined, and then show just the substitution with individual examples.
A.10 OTHER RELATED WORK
Other Modalities. Multi-modal models for video (Li et al., 2022c;a; Wang et al., 2022a; Alayrac al., 2022; Zellers et al., 2021; Yu al., 2022a), audio (Zellers al., 2022: Jaegle et al., 2022), and other modalities including game-playing and robot controlling (Reed et al., 2022; Jaegle et al., 2022: Liang al., 2022) have also been studied. Integrating these modalities is an important line of research, however existing models often do not even support sparse structured output and do not support dense structured outputs, so they not meet our objective of supporting classic vision tasks.
Vision & Language Pre-Traine Training. Vision and language pre-training has become standard prac- tice for multi-modal models, including both unified models and non-unified models that require task-specific heads to be trained from scratch during fine-tuning. Many initial pre-training strate- gies were inspired by BERT (Devlin et al., 2019) and included laaked-language- language-modeling modeling. image- text-matching, mask-region modeling objectives, often supplemented with objectives using the predictions of a strong object detector model (e.g. VILBERT (Lu et al., 2019), LXMERT (Tan & Bansal, 2019), VisualBERT (Li et al., 2019)). More recently contrastive-image-eet losses (Radford
27

---

Published as a conference paper at ICLR 2023
IMAGE SYNTHESIS
What complete image? Text
INPUT
small personal pizza with bacon and spinach
many largo kitos flying the sky
the train on the tracks the station
beach acea with black birds Rying over 2t
NO PRED
IMAGE INPAINTING
Fill in the blank ragion with this
IPPUT
PREDICTION
IMAGE GENERATION FROM SEGMENTATION
What is the complete image? Segmentation color
tree, yallow: aqua: baseball...
Time: building, navy: wall...
white: tree, red: building...
PREDICTION
Figure Image synthesis qualitative examples.
al., 2021: Li et al., 2022b; 2021) or augo-segressive generation losses (Wang al., 2022d;a; Yu et al., 2022a), have become common. Several works have also directly used object detection or segmentation datasets for pre- pre-training Yuan et al. (2021); Wang The generalized masked-data-modellng objective used in UNIFIED-IO is similar to the ones used in several recent works (Wang et al., 2022c; Peng et al., 2022; Singh et al., 2022).
(2022b); Sun al. (2022).
28

---

Published as a conference paper at ICLR 2023
OBJECT DETECTION
Locate all objects image.
glown
person
OBJECT LOCALIZATION
Whet region does this describe?
parking meter
SPAM
PREDICTION
SPAM
REFERRING EXPRESSIONS
What region does this describe?
gray-Bie man
dark
MPPUT
PREDICTIO
KEYPOINT ESTIMATION
Find the human joints in this ragion.
UnPNI
PREDICTION
Figure 7: Sparse labelling qualitative examples.
29

---

Published as a conference paper at ICLR 2023
DEPTH ESTIMATION
What is the depth map of the image?
INPUT
PREDICTION
SURFACE NORMAL ESTIMATION
What is the surface normal of the image?
OBJECT SEGMENTATION
What the segmentation of this?
pizza
bed
apple
FREDICTION
Figure 8: Dense labelling qualitative examples.
30

---

Published as a conference paper at ICLR 2023
IMAGE CLASSIFICATION
What this image?
IPPUT
whippel
NON
REGION CLASSIFICATION
What is the category of this region?
PREDICTION
bottle
labtop
Figure 9: Image classification qualitative examples.
31

---

Published as a conference paper at ICLR 2023
IMAGE CAPTIONING
What does this image describe?
IePIT
PREEICCION
large clock lange room with glass ceiling.
A flock birds cloudy sky
with this slice of pizza
black and white waaring purple
REGION CAPTIONING
What does the region describe?
INPUT
PREDICTION
grass along
women holding
grean mannaquin
banging
Figure 10: Image captioning qualitative examples.
32

---

Published as a conference paper at ICLR 2023
VISUAL QUESTION ANSWERING
food person usually
What are cats iaying next
Which vegetable in soup shaped?
hyorant?
IPPUT
PRED CTION
black and yellow
GROUNDED VQA
Is there anything writenn on the screen?
What color the wal??
What is this?
What's this can?
B1
IPPUT
BEGER
pbell'.
TESS
6lue
PREDICCION
RELATIONSHIP DETECTION
What the relationship between region and region?
FRED PREDICTION CTION
writing sign
cow esting grass
window building
hand man
Figure 11: Vision and language qualitative examples.
33

---

Published as a conference paper at ICLR 2023
QUESTION ANSWERING
residuai of force observed hadkone known axample being the force that acts belteen nuclwors in atorvic nuchei)...
context: The toumament book of the orvton 1883
contect Terna oveta (black earth), is aistributed over large arcos the Ameson forest. The development of this fertile soil allowed agricuiture and siviculture the previsusly hostle envtonment.
contest New someore stards
alone grond formal
international chess ioumament requires that: "A
haiway, Head bowed, he...
(5) holds his hanas his pockets joins people (2) jegs down halloay. (3) peers into microptine.
Paen reaching the eightni square must named as Gueen piece
question: What force betweon nucleons?
questtonn chess, can you promotts a pawn to pasoo?
quession: The devolopment Terra Preta allowed for what?
question: Which option is the most lokey continutation of this paragraph?
PRED CT CO
nuclear force
agnivuture and sivicuiture
(0) haids his hards his pockets
SENTENCE CLASSIFICATION
sentence Everyone really lovad
sentence Ahern, who travelling to Tokyo FU- consuit with yesterday,, will leaderss by telephone later this week, sentence2: summit between Furope andi Jepan taking place the Japanese capital. question: is the relation the sentences entailment, neutral contradicton??
sentence rectined soft chair woolly the olackened capsula.
The Rhina Gorge between Rüderhoim fineln
cookies only a few people liked tha chocolate chip cookies. Next should them
Koblonz listed
UNESCO World Hlantaga The Rhina Gorge
of question:: Does the word "tham" reder to the chocolate chip cookies?
back home, said Lu, dressed the
sentence2::
between Kablenz and what other
woolly ölackened capsule. question; Are these paraphrases? ?
Does the first
contain the second sentence?
entaimmet
AO
NO
TEXT SUMMERIZATION
context: European stocks finished on a mixed nate
Buigana, which both share borders with Serbia and Croate, want
contaut: Hungary and
chairiaan Ben Bernanka
Federal
contest: Toyota
Tuesday fuxary
Tuesday, as continental markets recouped eanier positvv treaing in U.S. equity mankets, quastion: What one sentence summary of this document?
sought to asaure Wall Street and congress Tuessigy the abil
INPUT
longar than cureently planned in disnis to keep the peace in Yugoslavia.
internaiional
reel its extraordinary economic stimulus and prevent flare inflation. question: What one summary document?
pushing Increasingly popular green technology.
question:
sentence of document?
summary document?
PREDICTION
Furopean stocks and mixee!
Hungaary Bulgario went longer U.S. presence Roania
Berankk says led ready to act economy
Toyote rolls out first UNK Lexus Nunury model
Figure Natural language processing qualitative examples.
34


Reviews:
[{"review_id": "lbTvfNeZBN", "reviewer": "Reviewer_iS57", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Unified-IO, a single Seq2Seq transformer model that can perform a wide range of tasks in vision, NLP, and vision-language domains. The unification of tasks is achieved by converting the input&output of each task into a sequence of discrete tokens. The model is trained in two stages: pre-training with masked language/image denoising and multi-task learning. The resulting model achieves competitive performance on downstream tasks without per-task finetuning.", "summary_of_the_review": "Unified-IO demonstrates the possibility of unifying a wide range of different tasks into a single model with competitive performance. This contribution alone can justify acceptance in my opinion. The paper is a bit lacking in terms of more in-depth anaylsis of pre-training and multi-task learning. The work will also be much more valuable if the code and model can be open-sourced.", "strength_and_weaknesses": "Strength:\n- The number of tasks supported by Unified-IO is impressive. It would require an enormous amount of engineering effort to unify the data format for tasks of diverse nature, not to mention setting up the multi-task training. The community would benefit a lot if the authors could open-source their code for data preprocessing and multi-task training.\n- It is interesting to see that a unified model can achieve competitive performance on multiple tasks without per-task finetuning. In particular, Table 4 sheds some light on how various tasks may benefit or conflict each other. \n- The paper is very well-written with a clear structure and flow.\n\n\nWeaknesses:\n- In NLP, task unification enables **zero-shot** generalization to novel tasks unseen during training. Papers such as Flamingo have also shown that models trained on image captioning can solve zero-shot VQA. Unified-IO shows the capability to generalize to novel concepts, but concept-generalization is much easier than task-generalization. Have the authors observed any capability of Unified-IO to generalize across tasks? This might be the most important advantage that is expected from task unification.\n- Besides reducing multiple models to a single one, what other advantage does multi-task learning bring? Table 4 shows that different tasks may benefit or harm each other. It would be good to see a more principled analysis of task relationships. If one is interested in a particular task, what is the optimal multi-task selection strategy?\n- How important is the pre-training stage? It would be good to see some ablation experiments to study the effect of pre-training and analyze each proposed pre-training objective.\n- The paper claims the pre-training stage to be ``webly supervised''. However, Imagenet21k has human-annotated class labels.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bx7PTEPc_4", "reviewer": "Reviewer_6qFN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes UNIFIED-IO, a unified model for a large variety of vision, language and V+L tasks. By formulating all inputs as sequences of embeddings, and all outputs as sequences of discrete tokens, UNIFIED-IO is able to use a simple Seq2Seq model to handle most V&L tasks such as image synthesis, depth estimation, object detection, segmentation, VQA, and more. It uses the T5 architecture as the Seq2Seq model, the SentencePiece tokenizer for encoding/decoding text tokens, a pretrained VQ-GAN for decoding image tokens into images. The model is trained with two stages: firstly pretrained unsupervised with masked token prediction objective on image, text and image-text pair data, then trained with supervision on a large number of tasks and datasets. The model is then evaluated on the recently proposed GRIT benchmark and 16 additional tasks to show competitive results on most of these tasks without finetuning for any specific task.\n\n**Major contributions of this paper:**\n- UNIFIED-IO is the first model to combine a large number of V&L tasks.\n- It shows the capability of a transformer Seq2Seq model to learn and perform a large variety of V&L tasks.\n", "summary_of_the_review": "This paper proposes a simple and generic framework that can perform on a large variety of vision, language and V+L tasks. Even though it is not topping in technical novelty, it is a good proof of concept, and the amount of tasks and datasets combined is impressive. In addition it is not trivial to combine so many datasets and tasks altogether in terms of engineering. I would recommend it to be accepted, with some reservation, mainly in terms of evaluation of the model. I am happy to change my score if the evaluation of the model is more rounded (on a more balanced set of tasks, on salient benchmarks, for easier comparison with existing methods).", "strength_and_weaknesses": "**Strengths:**\n1. The method unifies input and output of V&L tasks to sequences in a clean and relatively simple way.\n2. The main architecture that follows T5 is also conceptually simple and clean, consisting of a transformer encoder and decoder without much tweaking/modification.\n3. No task-specific augmentations are used.\n4. It is non-trivial in terms of engineering to combine so many datasets and tasks together, as well as evaluation on a large variety of tasks.\n5. This work is a demonstration of clever combination of existing techniques into a generic framework. The framework only consists of a few well-established components (T5, Seq2Seq, VQ-VAE as used in UViM, masked token prediction, coordinates tokenization as proposed by Pix2Seq) while obtaining decent performance.\n\n**Weaknesses:**\n1. The need for VQ-VAE may limit the quality of image synthesis (the model is only trained on image synthesis but not evaluated on such tasks).\n2. The model is only evaluated on two dense output tasks: depth estimation and segmentation (in GRIT). \n  - For depth estimation, it is compared with UViM but there is no comparison of model size.\n  - For segmentation, there is only a mask R-CNN baseline for GRIT (without model size comparison), but there are better-established benchmarks (such as COCO, Cityscape) where more methods have reported on.\n  - Results on more dense output tasks such as panoptic segmentation would be interesting.\n3. For tasks that require coordinates generation, the model seems to do better in short-length or fixed-length outputs (e.g. keypoint), but not in free-length outputs such as object detection. There is also no evaluation on certain core vision tasks such as object detection, even if the model is trained on it.\n4. Overall, the benchmarks could provide more comparison to be more convincing. Right now the results shows that UNIFIED-IO can do a large variety of tasks, but only some of them competitively.\n  - GRIT is relatively new, so it would be nice to evaluate on salient benchmarks on some of the tasks without much comparison in GRIT (e.g. segmentation).\n  - Additional tasks are mostly with text outputs or image classification, except one task for depth estimation. More tasks with coordinates generation and dense output would be nice.\n  - No model comparison in Table 3 and Table 5, so it is hard to separate gains from model scaling and methodology. \n5. Known concern for speed of autoregressive generation.\n6. Strictly speaking, all components and techniques have been proposed in previous or concurrent work, so it is a bit weak on novelty.\n\n**Actionable items:**\n- More baseline on tasks with coordinates output (object detection) or dense output (segmentation) on popular benchmarks such as COCO. I’d suggest that have object detection, semantic and/or panoptic segmentation, keypoint detection on COCO validation/test set for easier comparison with existing methods and concurrent work.\n- Add model size comparison in Table 3 and 5 to help explain how much gain comes from model scaling vs methodology. The XL version with 2.9B parameters is likely much larger than most of the methods in comparison.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bB92X96vcc", "reviewer": "Reviewer_gX6J", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a unified learner for various vision and language tasks that involve dense and sparse prediction. To be able to represent languages and various vision tasks in a homogeneous way, the authors employed the tokenized representation where the discrete codebooks for words and visual patches are learned by the VQ-GAN framework and used in the sequence-to-sequence model. When learned with a gigantic dataset of visual and linguistic tasks, the proposed model demonstrated outstanding performance on various downstream tasks without task-specific retraining.  ", "summary_of_the_review": "In general, I am positive about this paper due to the strong and convincing evaluation results. Although the technical contribution of this work is not ground-breaking, the authors executed the idea of building a unified model for vision and language based on a powerful sequence-to-sequence model well and achieved encouraging results. ", "strength_and_weaknesses": "Strength:\nThe idea of building a unified model for vision and language based on a powerful sequence-to-sequence model has been around recently, but the paper greatly scaled it up to a large number of tasks and datasets, nicely demonstrating that it can achieve decent performance on general vision tasks without task-specific inductive biases. The implication of this result is huge in my opinion.\n\nWeakness:\nI do not have a major concern in general since the experiments and results are strong and convincing. One weakness observed from Table 4 is that the multi-task pre-training often hinders the performance of individual downstream tasks, which implies that improving the performance of the model is not simply about scaling up the pre-training data but requires more careful considerations in choosing the pre-training tasks and data. However, I believe that this is not the main scope of this work hence do not have a major concern. Other than that, I would like to suggest authors include some discussions and results that might help readers to understand the limitation/capability of this work better, which are summarized below.\n\n1. Impact of multi-task learning\n\nOther than efficiency, one of the benefits of having a single parameterization of multiple tasks is that it might learn to understand tasks across domains. For instance, some tasks used in the pre-training have strong biases in their domains (e.g., pose is mostly labeled for humans and the surface normal is labeled mostly for rigid indoor objects). Yet, an ideal multi-task learner might be able to disentangle the task and domain, and perhaps generalize the task across domains in other tasks. Many works on the vision-language model often demonstrate this out-of-distribution generalization in a limited context (e.g., in the context of open-vocabulary learning where the pre-trained model is applied to downstream tasks of unseen object categories), but it would be interesting to see how much a massive multi-task learner such as the proposed model can extend it to more difficult cases (e.g., estimating a pose of animal or surface normal of human).  \n\n\n2. Performance gap in zero-shot generalization\n\nThe authors demonstrated that the proposed method exhibits compelling performance to the SOTA models specialized in individual tasks. Although it is quite encouraging, it does not directly show the upper-bound performance of the proposed model on individual tasks (i.e., how powerful the architecture and pre-training strategy for each task and how much performance we need to compromise by eliminating the test-time finetuning). Ablation studies with and without fine-tuning in downstream tasks might be helpful in understanding this gap.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MzDaIUghqx", "reviewer": "Reviewer_csLY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a unified framework for vision, language and multi-modal tasks. Instead of directly predicting labels with different formats, a unified task representation is designed to enable joint training of multiple tasks. The method exhibits state-of-the-art performance on the GRIT benchmarks and comparable results with specialized state-of-the-art methods.", "summary_of_the_review": "The paper presents one of the early attempts to develop one single unified model for almost all major vision tasks and some NLP tasks. The method clearly outperforms previous frameworks on the GRIT benchmark. Although there is still a notable performance margin between unified models and specialized models, this paper presents a solid attempt toward unified visual perception. Therefore, I recommend accepting this paper.", "strength_and_weaknesses": "Strengths:\n\n- The paper presents one of the early attempts to develop one single unified model for major vision tasks including classification, dense predictions, locations, and vision-language tasks and some NLP tasks. The method also achieves top performance on GRIT benchmark and comparable results with specialized state-of-the-art methods.\n\n- The paper provides a practical solution to combine multiple datasets and supervisory signals with discrete encoding and decoding.  \n\n- The method shows good scaling ability for very large models.\n\nWeaknesses:\n\n- The technical novelty of the proposed method is somehow limited although the whole framework is new and the results are significant.  The idea of encoding labels to discrete tokens is similar to earlier work UViM. [r1] also uses the VQ-GAN encoder to convert dense labels to discrete tokens. \n\n[r1] Visual Prompting via Image Inpainting, NeurIPS 2022.\n\n- Some implementation details are missing. For example, the training details of the VQ model are not mentioned in both the main paper and the supplementary materials. \n\nMinor issues:\n\n- In Sec. 1 and Fig. 1, it is mentioned a \"VQ-VAE\" model is used for image serialization. But in Sec. 3, only \"VQ-GAN\" is mentioned. I understand it seems the VQ-GAN architecture is used but the adversarial loss is not added during encoding images. But different terms in different sections may make the readers hard to follow. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%