InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists
Authors: Yulu Gan
Sungwoo Park
Alexander Marcel Schubert
Anthony Philippakis
Ahmed Alaa
Venue: ICLR 2024 poster (2024)
Keywords: Vision generalists
Text-to-image models
Multi-task learning
Primary area: generative models

Abstract: Recent advances in generative diffusion models have enabled text-controlled synthesis of realistic and diverse images with impressive quality. Despite these remarkable advances, the application of text-to-image generative models in computer vision for standard visual recognition tasks remains limited. The current de facto approach for these tasks is to design model architectures and loss functions that are tailored to the task at hand. In this paper, we develop a unified language interface for computer vision tasks that abstracts away task specific design choices and enables task execution by following natural language instructions. Our approach involves casting multiple computer vision tasks as text-to-image generation problems. Here, the text represents an instruction describing the task, and the resulting image is a visually-encoded task output. To train our model, we pool commonly-used computer vision datasets covering a range of tasks, including segmentation, object detection, depth estimation, and classification. We then use a large language model to paraphrase prompt templates that convey the specific tasks to be conducted on each image, and through this process, we create a multi-modal and multi-task training dataset comprising input and output images along with annotated instructions. Following the InstructPix2Pix architecture, we apply instruction-tuning to a text-to-image diffusion model using our constructed dataset, steering its functionality from a generative model to an instruction-guided multi-task vision learner. Experiments demonstrate that our model, dubbed InstructCV, performs competitively compared to other generalist and task-specific vision models. Moreover, it exhibits compelling generalization capabilities to unseen data, categories, and user instructions.

Published as a conference paper at ICLR 2024
INSTRUCTCV: INSTRUCTION-TUNE TEXT-TO-IMAGE DIFFUSION MODELS AS VISION GENERALISTS
Yulu Gan Peking University
Sungwoo Park
Alexander Schubert UC Berkeley and UCSF
UC Berkeley
Anthony Philippakis
Ahmed M. Alaa UC Berkeley and UCSF
Broad Institute of MIT & Harvard
ABSTRACT
Recent advances in generative diffusion models have enabled text-controlled synthe- sis of realistic and diverse images with impressive quality. Despite these remarkable advances, the application of text-to-image generative models in computer vision for standard visual recognition tasks remains limited. The current de facto approach for these tasks is to design model architectures and loss functions that are tailored to the task hand. In this paper, we develop unified language interface for computer vision tasks that abstracts away task- cosk-specifiic design choices and enables task exe- cution by following natural language instructions. Our approach involves casting multiple computer vision tasks as eoxttto-mmage generation problems. Here, the text represents an instruction describing the task, and the resulting image is a visually- encoded lask oulput. To train our model, we pool commonly-used computer vision datasets covering a range of tasks, including segmentation, object detection, depth estimation, and classification. We then use a large language model to paraphrase prompt templates that convey the specific tasks to be conducted on each image, and through this process, create multi-modal and multi-task training dataset comprising input and output images along with annotated instructions. Following text-to-image
the InstructPix2Pix architecture, we apply instruction-tunie diffusion model using our constructed dataset, steering its functionality from a generative model an instruction guided multi-task vision learner. Experiments demonstrate that our model, dubbed InstructCV, performs competitively compared to other generalist and task tasc-fpe vision models. Moreover, exhibits compelling generalization capabilities to unseen data, categories, and user instructions.
Code: httpp::/gitthub.com/// http://hhwh.ooom/llaaaaaaaannntt
Demo: https://wuggnggfee.cccssssasaaaa lablssstteaa
Prompt: "Detect
"HI InstructCV, please segment ail trees"
"Create monocular depth map"
"Segment Berkeley's Sather Gote"
Berkeley's Sather Tower"
"Detect MIT's great dome
InstructCV, please segment trees"
"Highlight all trees ground Stata"
"Create depth map for Stata Center"
Figure 1: Application of InstructCV to new test images & user-written instructions: InstructCV performs the vision task described in the instruction the input image. (Imagex courtesy of UC Berkeley and MIT).

---

Published as a conference paper at ICLR 2024
INTRODUCTION
Recent work on text-to-image models has achieved impressive performance in image synthesis I- 3]. Particularly, diffusion models 7 have demonstrated remarkable capabilities of transforming diverse text prompts into realistic images, even for novel concepts. Models like DALL-E 2 and Stable Diffusion [8] highlight this progress, now finding use in real-world applications. However, despite these impressive results, generative text-to-image models have so far not been exploited as a unified basis for standard visual recognition tasks. Instead, the predominant approach for these tasks is to design dedicated task-specific architectures and loss functions [ foregoing the opportunity to learn generalizable representations across heterogeneous problem domains and data landscapes.
Previous attempts to create unified models for computer vision tasks have predominantly relied on prompt tuning approaches in conjunction with sequence-tor teeeuueeeee architectures [9] This gen- eral framework enables conditioning on input images as well as task-specific prompts by representing image pixels and (trainable) prompts as sequences of discrete tokens. When trained multi-task datasets, the resulting output tokens align with the desired outcomes for the respective tasks prompted. One illustrative example of this approach is Pix2Seq [16]. which follows an autoregressive language modeling approach for processing tokenized image pixels and task identifier codes. Another example includes a class of methods based on visual prompting. which defines task-specific prompts in pixel space, unifying multiple vision tasks within a common "inpainting" framework [88 19) In both ex- amples, the task-specific prompts steer single architecture to execute multiple vision tasks. However, these prompts consist of (uninterpretable) numerical values derived from specific training datasets, which may limit their ability to generalize to new datasets, tasks, or categories.
In this paper, we propose unified model for computer vision tasks that conducts given task by following natural language instructions (Fig. D.. Our framework, dubbed InstructCV, repurposes gen- erative text-to-image models to create universal language interface for vision tasks. It does so by casting multiple computer vision tasks as text-to-iman to-image generation problems, where textual prompts (instructions) serve explicit task descriptors, guiding the generation process to produce the visual task output corresponding to the input image. By conditioning on natural language descriptions of vision tasks, InstructCV enhances the representation of semantic coherence between images and lan- guage prompts, improving the model's ' generalization capabilities to new human-written instructions and new categories compared prior "generalist" vision models 112 13 26
To train InstructCV, we follow an instruction tuning approach applied to a pretrained conditional diffusion model (Stable Diffusion). We generate the instruction tuning data by constructing multi- modal, multi-task training dataset that comprises tuples of textual instructions, input images and visually-encodee task outputs. We do so by first combining several standard computer vision datasets across multiple tasks including segmentation, object detection, depth estimation, and classification. Next, in order to create heterogeneous and semantically rich textual instructions, we use a large language model (LLM) to paraphrase prompt templates for each vision task. Finally, we encode the output of the vision task associated with each instruction in the form of an output image (c.g., masking pattern for semantic segmentation). Using this dataset, we utilize the InstructPix2Pix architecture 27 to instruction-tun a text-to-image diffusion model, transforming its functionality from generative image synthesis model into instruction-guidee guided multi-task vision learner.
Our experiments demonstrate that InstructCV achieves competitive results compared to other vision generalist and task-specific vision models. Particularly, InstructCV displays compelling generalization properties, surpassing the performance of state-of-the-art vision generalist models on external datasets as well on unseen prompts in open-vocabulary segmentation tasks.
INSTRUCTCV
The InstructCV framework comprises two key steps: (a) construction of multi-moda and multi-tast instruction-tuning dataset, and (b) finetuning pretrained conditional diffusion model using the dataset generated in step (a). (See Fig. 2 for pictorial depiction.) Details of each step are provided below.
2.1 CONSTRUCTING A MULTI-MODAL & MULTI-TASK INSTRUCTION- TUNNIGG TUNING DATASET
We combine four widely-used computer vision datasets (MS-COCO 28 ADE20K 29 30,. Oxford-
IlI-Pets (11 and NYUv2 32) covering four vision tasks (semantic segmentation, object detection,
2

---

LLM-based instruction generation. For each vision task m E {1,..., M) under consideration, we pick prompt template T that describes the task, e.g.. "Segment the Secategory% " for the seman- tie segmentation task. We then attach a prompt to each training data point by inserting the category within the image in the corresponding task template, e.g., (x,, e.g., T "Segment the cat". We con- Published as a conference paper at ICLR 2024
(a) Constructing Multi-modal & Multi-task Dataset
(b) Instruction-tuning Diffusion Model
LLM- MM--aaee instruction generation
"Detect Mona Lisa'fface"
"Segesent ger's scarf"
"Eselmate a depch map"
Task sperific prompt templates
Rephrased prompts
"Detect the &CATEGORY%
hightight the #CATECORY% Which segments of the Image contain SCAT-BORMY
"Create monorular depth
"Segment RCATFGORMS""
LLM
"Display SCOLLO% of the image contoins SCATEGORY%
Can you help me segment the &CATFGORYS9
Instruct CV
Visual encoding of task outputs and training data generation
LLM-upphaaee instruztion I
"Aghhlger ang"
aed
Input X
Vibually encoded target v(y)
Figure 2: Pictorial depiction of the InstructCV training pipeline. (a) We pool multiple computer vision datasets to construct multi-modal and multi-tast task set of image pairs, where the target of each task is visually en- coded in the form of an output image. Starting with set of task-specific prompt templates. we sample a new in- struction for each training point by using an LLM to rephrase the template for the corresponding task. (b) Using the dataset in (a), we finetune a diffusion model to produce the output v(y) given an image x & an instruction I.
monocular depth estimation and classification), into single multi-task dataset D = {xx, Yı, m,)) in which x, is the input image, Y; the task output and m, E {l...., M) is the task identifier (M 4 in our setup). We convert 'D into multi-modal Instruction-tuniie which the task identifier for training point is expressed as natural language instruction I,, and the la- bel y; is represented in visual format v vyy)) We construct the dataset Dr through the following steps.
dataset Dz = {(x:, I.) in
sider two incarnations of our instruction-tuninn uses the deterministic task-specific prompt templates described above. We refer to this dataset the fixed prompts (FP) instruction -oannnnniinnnnnngggggo dataset T = {(x,, vy;). I (x. v)))]; Ji- In addition, we use a T5-based paraphrasing LLM 33
dataset. First, we consider a baseline dataset that only
generate rephrased versions of the prompt template to create a diverse range of instructions. As shown in Fig. 2 a) (top), the LLM takes as an input the prompt template (c.g.. "Segment the %ccategory% ") and produces a wide range of paraphrased variants (e.g.. "Highlight the %ccategory%" This procedure ensures that instruction set is varied yet firmly tied to the core intent of the original prompt. We use the LLM to sample rephrased variant of the prompt template I₁ LLM( ymp ) for each training data point in D. We refer to the resulting instruction tuning dataset the rephrased prompt (RP) dataset DE {(x;, (y:), (xi, y:)));
Visual encoding of task outputs. We format the target label of each task to represent in the same RGB image space as the input image through "visual encoding" function v(y) (Fig. 2 a) (bottom)). This enables casting all tasks in a unified text-to-image generation framework, leveraging Pix2Pix architectures. That is, given an image x and instruction I, InstructCV produces an image v(y) that encodes the task output. In the following, we provide the definition of v(y) for all tasks under study.
(1) Semantic Segmentation. The target output y of this task is typically an assignment of a label or category every pixel in an image. A natural choice of v(y) for the semantic segmentation task is binary mask that labels pixels in the input image x belonging to the prompted category in I.
(2) Object Detection. Here, the goal is to identify the spatial position of a category in an image using a bounding box, i.e., the label comprises bounding box coordinates y [Ca,, Cy: w, h). We define v(y) for object detection as the image x with a bounding box overlaid according to the coordinates in y.
(3) Monocular Depth Estimation. The target y of this task is the depth value (i.c., distance relative to the camera) of each pixel in the RGB image x. For this task, we define the visually-encoded target
3

---

Published as a conference paper at ICLR 2024
v(y) as RGB image in which pixel colors encode the depth values. This encoding is done by con- verting depth values ranging from O to 10 meters (based on depth ranges in the NYUv2 dataset (32) into the discrete space [0, .............................. 255] for RGB image representation, i.e., v(y) = yy 255 We
then apply the same value across all three RGB channels to create a visual depth map.
(4) Image Classification. In multi-class image classification, the target label y is a categorical value indicating the object depicted in the image To represent image classification in Pix2Pix format, we resort to a color-coding methodology. To this end, we use a prompt template of the form: "Display %coolr%% if the image contains Sccategory% ". We sample random colors when filling in the template for individual training points. This steers the text-to-image model to produce an image consisting of the pure color block if the category specified in the prompt visible in x. (Note that this approach only enables us to predict if specific category is in the imput image x. For multi-class classification, we need to use a series of prompts specifying all categories of interest one at time.)
2.2 INSTRUCTION-TUNING A LATENT DIFFUSION MODEL
We use our instruction- parnunttinttuinngggg dataset D = {x, v(y,).I;) to train a (conditional) diffusion model that conducts the vision task specified in the instruction I on the input image x, producing a visually- encoded task output vyy). By finetuning the text-to-image diffusion model using Dz, we steer its functionality from a generative model to a ianguae--guiden multi-task vision learner. We use a training procedure similar to that of InstructPix2Pix [27]. which uses similar multi-modal dataset (pairs of images and editing instructions) to train an instruction- guided image editing model.
Diffusion models 7 generate data by gradually denoising normally distributed random variable; this process amounts to learning the reverse dynamics of a Markov chain with fixed length T. Latent diffusion models [8] apply this approach within the latent space of pretrained variational autoencoder 335 with encoder E() and decoder (-).. Training a diffusion model involves a forward disfusion and reverse demoising process. During the forward process, the image vyy is transformed to its latent representation Zo = E(v(y)). which is then injected with Gaussian noise over T steps:
(1) where the time-varying constants BI. control how much noise is added at each timestep and are chosen such that zr roughly converges to a standard Gaussian vector. This forward process does not contain any trainable parameters and can be described as q(Z1.T) 20) = IT q(z, (2-1). In the re- verse diffusion process, po (Zo:r) ) p(zr) I Pe pz (Z)) the objective is to learn a model to progressively denoise the latents ZT:1 to recover the initial latent z0- The target image can then be reconstructed as v(y)= = (2o). The reverse diffusion process can be written as:
g ( = NNz;;; v1 - 3, 3,I), Vt E TT....,T),
po (2 m+) N (2 po(ze, t), PT). Vt (I...,,,, T),
(2)
where the means 14 are typically parameterized using neural networks and the variances of are predetermined constants. Such a denoising model is learned by optimizing reweighted variant of the variational lower bound on the data distribution 4. 36), i.c..
Lunconditional = EE(v())). [|e - Eo (t, 112
where is a standard normal random variable and z, 0, + ocE, where of II'' (1 3.) and of = - ai are based on the diffusion distribution (2,/22) = N (2z oz II) The noise predictor 6g is obtained from the parameterization pa(z,, t) := (z. B. (z. t)/ V a////-a3//v1 // 3. The model cy is trained to predict the noise vector at each time-step t in order to denoise the latent variable z,-
Instruction- tattiioontnnnng via image and text conditioning. The objective in 33 can be further refined to condition on both the input image x and instruction I to generate the desired output v(y) this can be achieved by learning a conditional noise predictor g 27), minimizing the following loss function:
[|c [|le Eo (t, Zi E(x), Z)1
Leanditional =
(4)
We use pretrained Stable Diffusion checkpoint, which exhibits strong text-to-image generation capabilities, as the backbone architecture of our model. For text conditioning, we adopt the same methodology as in (8), utilizing the instruction I instead of image captions textual inputs. For image conditioning, we concatenate the encoded input image E(x) with the latent z,, which are then fed into input to the first layer of the noise predictor 6g (27
4

---

Published as a conference paper at ICLR 2024
Classifier free guidance. To further improve the alignment between the generated outputs and their conditioning, we apply "classifier-free freffeerrrre""" guidance [37] This approach combines the unconditional and conditional noise predictors in (4) and (3) in order to shift probability mass towards data where an implicit classifier pe(clzi) assigns high score to the conditioning c. Similar to InstructPix2Pix ((3) (277). we use a modified noise predictor that assigns different weights to the different com- ponents of the conditioning (x, I) as follows:
Y:
Y: 2.0
Te 3.0
Eo (t, R1, x, I)- co(t, 2,, 0,
+ 71 (Eo 2,, 2, 2) - (0,1, 24, 25, 21)
Instruction:
"Estimate the depth map of this image"
+ yT (Eo Zt, x, e0 Ze, x, 2)).
7/ Y7 Pr 15
where "I and YT guidance scales that deter- Figure 3: Impact of classifier- guidance the out- mine the relative importance of the image and text puts of InstructCVV for the depth cstimation task.
conditioning. Unconditional denoising is achieved by introducing null values to the respective image and text channels of the noise predictor. Fig. we show the effects the parameters 7/r and YT on the output image v(y) of InstructCV. As we can see, controls the similarity between the input and output images, x and v(y), whereas "r controls the extent to which pixel values v(y) correspond to actual depth values in Low ?T and high result in output images that look very similar to input images with only modification in the color map that does not reflect pixel depths. On the other hand, high yT and low 77 produces coarse depth maps that miss nuanced details in the input images.
3 RELATED WORK
Repurposing diffusion models for vision tasks. Diffusion models have achieved impressive perfor-
mance image generation 36 38 text-to-image synthesis 12 6 g as well as generation of other modalities such as video 41 and audio 42 The idea of repurposing diffusion models to tackle standard computer vision tasks has been considered before to develop models for object detection (43 and open-vocabulary panoptic segmentation [44). These approaches were limited to single-task set- tings with specialized loss functions and (unimodal) architectures. Contrarily, InstructCV provides a unified architecture for multi- multi-task learning, with natural language interface that enhances generaliza- tion to new datasets and categories. The idea of desstruction-uningg text-to- toxt-to-image diffusion models was introduced in (27 with the objective of finetuning the model to follow editing instructions. In- structCV builds on this framework to adapt text-to-image models for performing conventional visual recognition tasks. To our knowledge, this one of the earliest efforts this direction.
Vision Generalists, Several prior attempts have aimed to develop unified models capable of executing multiple vision tasks within a single, shared architecture. Motivated by successes of LLMs, recent work has attempted to design such generalist models based on sequence-to-sequence architectures. Among these, models such as Florence (9. OFA [10], CoCa [I]] and BETT-3 [4], learn general rep- resentation encoder, which require individual finetuning to each specific downstream task. Methods such as UnIfied-IO [5] and Pix2Seq-v2 v2 17 build single architectures that are capable of performing multiple vision tasks via prompt tuning. However, the sequence-basen based operation of these models re- sults in slow inference speeds, and the tuned prompts may not generalize to unseen datasets/categories. Another line of work proposes vision transformer-basee architectures that frame different vision tasks as inpainting problems [88 19). This work focuses on in-context learning based on visual prompts and does not consider sagguae--bssss instructions, which believe is more natural interface for general- general-purposs. purpose models. To the best of our knowledge, the only generalist model that supports a language- based interface for vision tasks similar that of InstructCV is the VisionI.LM model devel- oped in 13 This is an L.LM-based framework that treats images as a foreign language and aligns vision-centric tasks with language tasks that can be flexibly defined using language- -based instructions. VisionLLM and InstructCV share common objective but use different approaches. VisionLLM fine- tunes a pre- pre-trrined L.M using dision-centro tasks, whereas InstructCV finetunes a text-to-image model by substituting image captions with instructional text. We were unable to empirically compare the two models as the code for VisionLLM was not available at the time of writing this paper.
5

---

Published as a conference paper at ICLR 2024
4 EXPERIMENTS
4.1 EXPERIMENTAL. SET UP
We evaluate InstrucICV across the four vision tasks under study (semantic segmentation, object detec- tion, monocular depth estimation and image classification). For this purpose, we consider widely used
datasets for each task: ADE20k (29 30 for semantic segmentation, MS-COCO (28) for object detec-
tion, NYUv2 for depth estimation 45 and Oxford- Solord-IIIT Pet [46] for classification. In what follows, we
explain the processing steps and evaluation procedure for all tasks under consideration.
Semantic Segmentation. ADE20K covers 150 semantic categories and comprises 25, 000 images of which we use 20, 000 for training. 2, 000 for validation, and 3, 000 for testing. We follow the same pro- tocol as suggested in [8] to implement the training/test split. At inference time, average the out- puts of the three channels of the output image v(y) to obtain the final segmentation mask. We evaluate the accuracy of segmentation masks using the Mean Intersection over Union (mloU) metric.
Object Detection. MS-COCO contains 118, 000 training and 5, 000 validation images with labels for 80 different categories. We follow the same protocol in Pix2Seq [16] to set up the training/test split. At inference time, we follow the post-pun post-proesssing steps in Appendix A.3 to derive the coordinates and category of each Region of Interest (Rol) from the output image vyy). We then aggregate the results for all categories in order to calculate the Mean Average Precision (mAP).
Depth Estimation. The NYUv2 dataset (32 consists of 464 indoor scenes captured by Microsoft Kinect camera. We follow the official training/test split, with 24, 231 image-depth pairs used for train- ing, and 654 used for testing. For the test images we report the Root Mean Square Error (RMSE), ab- solute mean relative error (A.Rel), and the share of interior pixels with different threshold 8. During inference, we take the average across the three channels of the output image and apply the inverse of the linear transformation used in training to obtain a depth estimate in the range of [0, 10] meters.
Image Classification. As mentioned in Section 2.1 we implement image classification by asking the InstructCV model if category is visible in the input image x using the following template prompt: "Display %bcolor_19 if the image contains Sccateorry%, else display %coolor "_r____%""___ We eval- uate the accuracy of InstructCV for classification by assessing whether v(y) contains the color block corresponding to the correct category. We do so by generating image pairs based on the Oxford-III Pet dataset for binary classification and augment these with negative pairs, where the category men- tioned in the language instruction is not present. We then evaluate a classification score defined as: Cls-Score (v(y), c) Σi= Σド= (y) cr,yl i.e., the Euclidean distance between the pixel-wise colors of the output image v(y) and target color block specified in the task instruction I.
Our pooled multi modal/multi task instruction-tumi dataset comprises 180,285 images. We create two versions of the dataset, DTT & DTr. with fixed and rephrased prompts described in Section 2.1
External Datasets. Since the generalist baselines and InstructCV were trained on different datasets we consider additional external datasets that outside of the training distribution of all baselines. To this end, we consider the following datasets: ImageNet (47) for classification, SUNRGB-D [48] for object detection and VOC (49 for segmentation and monocular depth estimation tasks.
Implementation Details. We train InstructCV for 20 epochs on 8 NVIDIA A100 GPUs over 10 hours. The training involves images with a resolution of 256 256 and incorporates data augmentation including random horizontal flipping and cropping with a batch size of 128. The proposed model initialized with EMA weights obtained from the Stable Diffusion checkpoint, and trained with a learning rate 10-4 without any warm-up stage. Further details can be found in appendix A.1 We refer to the models trained on DI and DRP as InstructCV-FP and InstructCV-RP, respectively.
4.2 RESULTS
T presents a quantitative comparison between InstructCV and task-specific as well as generalist vision models in both in-distribution and disto--distribution datasets. (Fig. 4 displays illustrative ex- amples of InstructCV outputs.) For depth estimation, we compare InstructCV with DepthFormer (50]
Binserrmer [51] and UviM I2
For semantic segmentation, our baselines include Mask2Former 53
Unified-TO IO has been trained on 90 datasets while Pix2Seq V2 was trained MS-COCO.

---

Published as a conference paper at ICLR 2024
Table Comparison InstructCV to task-specific and vision generalist baselines. We report performance segmentation, object detection, depth estimation and classification using test samples from the InstructCV in- struction tuning dataset as well as external datasets for each task. InstructCV was evaluated using LLM-rreprased instructions for test data. (Bold and blue indicate best and second best performing model, respectively.)
Depth Estimation
Semantic Segmentation
Classification Acct
Object Detection
RMSEL
mloUt
mAP@0.57
NYUv2 1SUNRGB-D
ADE-20K
tVOC
Oxford-Pets tImageNet-sub
COCO
fVOC
Tpsk-sp models
DepthFormer
50
0.339
0.427
BinsFormer
11
0.330
0.433
SSA
47.150
Mask2Former
53
56.100
ResNet
97.500
98.400
DETR
28
60.600
Mask R-CNN
30
60.500
Generalist framework, Tpsk-specific models
UviM 12
0.468
Generalist models
Unified-1O
0.387
0.287
25.713
27.724
96.514
89.877
P2
$7,400
38,500
Painter
18
0,288
0.285
49.9
52.5
InstructCV-FP
0.275
0.268
52.3
53.5
80.4
75.1
49.1
62.00
InstructCV-RP
0.297
0.279
47,235
52.125
82.135
74.665
48.500
61.700
These dataset were not part of the training dataset for the baselines under consideration. Task-specific models are incapable of directly implementing the corresponding task on external datasets if these datasets contain object categories that not present in the models' training data.
i External datasets:
For classification, we consider baseline classifiers with ResNet (54] and ViT 55 back- bones. Lastly, for object detection, we consider DETR (8 and Mask R-CNN (56). Task-specific models for semantic segmentation, classification, and object detection have not been assessed on datasets beyond their training distribution. This is due to the fact that these new datasets introduce cat- egories absent in the model's 's original training, which precludes zero-shot generalization. We consider Un [15 and Pix2SeqV2 V2
and SSA [52).
[17] as generalist vision baselines. Note that we only report object detection results for V2 117, 1 as this model does not cover the other tasks involved in the de- velopment of InstructCV. Additional tasks Pix2SeqV2 is able perform, such as keypoint detection, have not been integrated into the current version of InstructCV.
Performance comparisons. Overall, Instruct CV performs competitively compared to both generalist and task-specific baselines across all four tasks. For depth estimation within in-
Table 2: Open-vocabulle segmen- tation on the FSS-1000 dataset.
distribution data, InstructCV achieves 10% improvement in RMSE compared to the second best model, BinsFormer [5T]. Notably, InstructCV demonstrates strong generalization per- formance to unseen datasets, surpassing all baselines by a large margin, with the exception of classification tasks. For instance, for the task of depth estimation the task-specific models Binsformer [5T) and DepthFormer (50) experience high performance drops of 31.2% and ~ 26.0%, respectively, while InstructCV's performance improved by 6% resulting in 34.7% lower RMSE than the best task- sase-ipec model. Similarly, for the task of object detection, the performance of the Pix2Seq V2 generalist model drops by 32.2% when evaluated on VOC- Vosttnntree outperforms this generalist model by a +23.2 m..@0.5. Among all baselines, only Unified-IO demonstrates comparable generalization properties to unseen datasets. However, InstructCV outperforms Unified- 10 on all tasks except for classification. Notably, for semantic segmentation, InstructCV surpasses the performance of Unified-IO by +44.401 in mIOU. Classification is the task where InstructCV exhibited its weakest performance compared to baselines.
Model
mloU
Inpainting S
58.5
Generalist Painter 18
62.3
InstructCV-RP
69.8
Generalization to unseen categories. Most existing task-specific and multi-task models are built based on a fixed category pool determined by their training data, and do not exhibit zero-shot
capabilities in detecting, segmenting classifying categories outside of this set [15 17
28 52
7

---

Published as a conference paper at ICLR 2024
Semantic Segmentation (ADE-20K test data)
Prompt: "Segment &CATEGORY%
*CATEGORY% (House, Wall, Grass Sky, Sidewalk, Tree)
(Bed, Cushion, Lamp, Curtain Door, Window, Floor, Ceiling)
[House, Wall, Grass Sky, Sidewalk, Tree)
(House, Wall, Grass Sky, Sidewalk, Tree)
Object Detection IMS-COCO test data)
Prompt: "Detect the &CATEGORY%
Risptinn
alephant
per son
horse
*CATEGORY% (Elephant)
(Teddy bear, bed)
[Person]
(Person, horse)
Monocular Depth Estimation (SUNRGBD dataset)
Prompt: "Create monocular depth map"
L
Image Classification (ImageNet dataset)
Prompt:
"Show green there exists birds; if not, show red instead"
Prompt: "Show green there fish; if not, show yellow instead"
Figure 4: Samples of InstructCV outputs across all vision tasks. The segmentation and detection outputs (top two rows) are obtained applying one prompt each category and overlaying results one output image.
56 Because InstructCVV leverages semantically meaningful instructions and pre-trained text-to- image generative model guide learning, we expect its task-specific capabilities generalize new categories. To investigate its generalization capabilities to unseen categories, we evaluate InstructCV on an open-vocabulary segmentation task using the FSS-1000 dataset (58). This dataset comprises 1000 object classes, many of which have not been previously annotated in other computer vision datasets. Because the baselines in Table do
accommodate many of these categories, we instead compare InstructCV with generalist vision models that exhibit zero-shot capabilities-the visual prompting by Inpainting model in [57] and the Generalist Painter model [8] Both methods rely visual prompting approaches, where the input prompt is set of pixels and the vision tasks all represented within a unified inpainting framework. Table 2 presents the results. Overall, InstructCV outperforms Generalist Painter and Inpainting by +3.5 and +7.3 in mloU, respectively. This demonstrates the value of repurposing text-to-image models and language- based prompts to improve the generalization capabilities of generalist approaches to computer vision tasks.
Generalization to new user-written instructions. The InstructCV-RP model undergoes training using the dataset DRP which comprises variety of instructions, all aimed conveying common
8

---

Published as a conference paper at ICLR 2024
Table 3: Evaluating InstructCV on new user-written instructions. We evaluate the InstructCV-FP and InstructCV-RP models on the semantic segmentation task using test samples within the ADE20k dataset. At inference time, we apply new user-written instructions (listed below) that have not been used during training in D and D in addition to the template prompt (first row, shaded).
mIOU Score
Instruction
InstructCV-FP InstructCV-RP
Template prompt:
"Segment the "ccategory%.""
45.503
42.793
"Segment "ecategory%ä""
36.315
48.739
"Can you help me segment the %ecategory%?""
44.035
45.686
"Highlight the image parts corresponding the Sbcategory%
19.988
44.758
"Please highlight the image segnent containing %cattgory%%
21.952
40.153
"Which image segments contain the "ccategory%?""
39.348
47.972
underlying intent (i.e., describing specific visual task). Through this diverse training data that encom- passes a broad spectrum of phrasings and descriptions the same task, we expect that InstructCV-RP will be able to extrapolate its learning to novel user-generatee instructions inference time. To test this, we compared the performance of the InstructCV-FP and Instruct-RP variants on the semantic segmentation task. Both models were tested using manually-selectedr prompts (unseen in training data) on 200 images in the ADE20k test data. The results in Table indiccae that the InstructCV-RP model exhibits consistent performance and more robustness variations in the phrasing of user instructions compared to InstructCV- For example, testing InstructCV-FF with the instruction "Please highlight the image segment containing lccateoory%." instead of the template prompt "Segment %ecategory led to 51.8% drop in mIOU Conversely, InstructCV-RR only incurred a performance reduction of 6% with this instruction compared to the template prompt. This suggests that our I.L.M- prompt rephrasing approach effectively enhances the ability of InstructCV to generalize to new user-generated prompts that convey descriptions of tasks similar to those seen during training.
Computational costs. Finally, we note that InstructCV was trained in end-to-end fashion, with only 2,000 finetuning steps. The inference time of InstructCV on a single NVIDIA A 110 GPU 5 seconds (for 256x256 image). This is significant improvement over comparable generalist models, such as Unified-IO 10 [5]. which was trained from scratch using 1.5 million steps and takes around 40 seconds for inference
single NVIDIA A 100 GPU. Notably, InstructCV not only simplifies the training process but also outperforms Unified-IO in various tasks. These improvements can be attributed to the already impressive capabilities of the underlying text-to-image model. By instruction-un generative model with a relatively small number of steps and a moderately-sizer dataset, we are able to steer its functionality with performance that is competitive with bespoke generalist models.
CONCLUSION
In this paper, we introduce unified language interface for computer vision tasks, dubbed InstructCVV, eliminating the need for task-specific design choices and allowing for task execution based on natural language instructions. InstructCV frames various computer vision tasks text-to-image generation problems. In this setup, textual instructions describe the task, and the resulting image serves as a visual representation of the task output. Following the InstructPix2Pix architecture, we curate multi- task and multi-modal dataset to instruction-tune a pre-trained text-to-image diffusion model, steering its function from a generative model to an instruction-guided multi-task vision learner. By harnessing semantically meaningful language instructions to drive the learning process, our model demonstrates compelling generalization capabilities across unseen data, categories, and user instructions.
Limitations. The inference speed of our model lags behind specialized task-specific models and falls short of meeting the real-time inference requirements for tasks such as object detection and seg- mentation. Additionally, the semantic flexibility of InstructCV is constrained by the richness and diversity of our instruction-tuningg dataset, which is currently generated by rephrasing a limited set of template prompts. This raises questions for future work: can this learning paradigm accommodate instructions that introduce more nuanced conditions? For example, an instruction might cap the count of objects to be detected. Exploring such ideas might require the integration of strategies such as learning from human feedback, which could enable more versatile generalist models by improving alignment of task outputs with more complex prompts.

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGEMENTS
The authors would like to thank David Sontag (MIT) for insightful feedback and discussions.
REFERENCES
[1] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479- 43444,9944, 2022.
[2] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06/25 2022.
[3] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis.
Advances in neural information processing systems, 34:8780-8794, 2021.
[4] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arxiv:2006. 1239, 2020.
[5] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score- Based Generative Modeling through Stochastic Differential Equations. International Conference on Learning Reprexentations, 5 2021.
[6] Alex Nichol, Prafulla Dhariwal, A. Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Πya Sutskever, and Mark Chen. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. arXiv (Cornll University), 12 2021.
[7] Jaseha Sohl- Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR, 2015.
[8] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[9] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li. Ce Liu, Mengehen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang. Michael Zeng, Luowei Zhou, and Pengchuan Zhang. Florence: A new foundation model for computer vision, 2021.
[10] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple
learning framework. CORR, has/2202.03052. 2022.
[1]] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. 2022.
[12] Alexander Kolesnikov, + André, Susano Pinto, Lucas Beyer, Xiaohua Zhai, Jeremiah Harmsen, and Neil Houlsby. Uvim: unified modeling approach for vision with learned guiding codes.
2023.
[13] Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, and Jifeng Dai. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks, 2023.
[14] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as foreign language: Beit pretraining for all vision and vasion-language tasks, 2022.
[15] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. 2022.
[16] Ting Chen, Saurabh Saxena, Lala Li. David J. Fleet, and Geoffrey Hinton. Pix2seq: A language modeling framework for object detection. In International Conference on Learning Representations, 2022.
10

---

Published as a conference paper at ICLR 2024
[17] Ting Chen, Saurabh Saxena, Lala Li, Tsung- Yi Lin, David J. Fleet, and Geoffrey Hinton. A unified sequence interface for vision tasks. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[18]
Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: generalist painter for in-context visual learning. arXiv preprint arXiv:2212.02499, 2022.
[19] Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei A. Efros. Visual prompting via image inpainting. arXiv preprint arXiv:2209, 00647, 2022.
[20] Hao Zhang. Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianfeng Gao, Jianwei Yang, and Lei Zhang. A simple framework for vocabuacyyy segmentation and detection. arXiv preprint arXiv:2303.00 00.30.0..1.,,.. 31, 2023.
[21] Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Xiaogang Wang, Hongsheng Li, Xiaohua Wang. and Jifeng Dai. Uni-perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. arXiv preprint arXiv:2112.01555 2021.
[22] Hao Li, Jinguo Zhu, Xiaohu Jiang, Xizhou Zhu, Hongsheng Li, Chun Yuan, Xiaohua Wang, Yu Qiao, Xiaogang Wang, Wenhai Wang, and Jifeng Dai. Uni-perceiver v2: generalist model for large-scale vision and vision-language tasks, 2022.
[23] Xueyan Zou, Zi-Yi Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, Nanyun Peng. Lijuan Wang. Yong Jae Lee, and Jianfeng Gao. Generalized decoding for pixel, image, and language, 2022.
[24] Maxime Oquab, Timothée Dareet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
[25] Charlie Nash, Joao Carreira, Jacob C Walker, Iain Barr, Andrew Jaegle, Mateusz Malinowski, and Peter Battaglia. Transframer: Arbitrary frame prediction with generative models. Transac- lions on Machine Learning Research, 2023.
[26] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12883, 2021.
[27] Tim Brooks, Aleksander Holynski, and Alexci A. Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023.
[28] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. Lecture Notes in Computer Science, 2014.
[29] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. Computer Vision and Pattern Recognition, 2017.
[30] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 2019.
[31] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEFE conference on computer vision and pattern recognition, pages 3498-3505. IEFF, 2012.
[32] Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rghd images. In ECCV, 2012.
[33] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1667, 2020.
[34] Maxim Kuznetsov Vladimir Vorobev. A paraphrasing model based on chatgpt paraphrases. 2023.
[35] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint 131266 14, 2013.
11

---

Published as a conference paper at ICLR 2024
[36] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[37] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generafive Models and Downstream Applications, 2021.
[38] Jonathan Ho. Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. The Journal of Machine Learning Research, (122)2222222111 2022.
[39] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi, Palette: aanage-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1-10, 2022.
[40] Chitwan Saharia, Jonathan Ho. William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, +5(4):4713-4726 2022.
[41] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, et al. Make-a-video. Text-to-video generation without (2ti::2009117992, 2022.
text-video data. arXiv preprint
[42] Gihyun Kwon and Jong Chul Ye. Clipstyler: Image style transfer with a single text condition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18072,18071, 2022.
[43] Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection, 2022.
[44] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. ODISE: Open- Vocabulary Panoptic Segmentation with Text-to-lmage Diffusion Models. arXiv preprint arXiv: 2303. 04803, 2023.
[45] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rghd images. In Computer Vision-ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12, pages 746-760. Springer, 2012.
[46] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In 2012 IEEE Conference Computer Vision and Pattern Recognition, pages 3498-3348 -308-,55,5055 2012.
[47] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[48] Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene recognition using places database. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
[49] Mark Everingham and John Winn. The pascal visual object classes challenge 2012 (voc2012) development kit. Pattern Analysis, Statistical Modelling and Computational Learning, Tech. Rep, 8:5, 2011.
[50] Zhenyu Li, Zehui Chen, Xianming Liu, and Junjun Jiang. Depthformer: Exploiting long-range corrllttio and local information for accurate monocular depth estimation, 2022.
[51] Li, Zhenyu, Wang, Xuyang, Liu, Xianming, Jiang, and Junjun. Binsformer: Revisiting adaptive bins for monocular depth estimation. Cornell University arXivv 2022.
[52] Jigqi Chen, Zeyu Yang, and Li Zhang. Semantic segment anything. [attps://qithu>
com/ fodaauuda--- zvg/ Semant i c- Sww.eenttnnneeeeeee in 2023.
[53] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. 2022.
[54] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
[55] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly,
12

---

Published as a conference paper at ICLR 2024
Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
[56] Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask r-cnn. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
[57] Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei A Efros. Visual prompting via image inpainting. arXiv preprint arXiv:2209.0664 2022.
[58] Xiang Li, Tianhan Wei, Yau Pun Chen, Yu-Wing Tai, and Chi-Keung Tang. Fss-1000: A 1000-class dataset for few-shot segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2869-2878, 2020.
[59] Tero Karras, Mïik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion based generative models. Advances Neural Information Processing Systems, 14225665-26577 2022.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 INSTRUCTCV TRAINING DETAILS
We train our multi-task vision model across 20 epochs for 10 hours on an array of 8 80GB NVIDIA A100 GPUs. Our training utilizes images of 256 x 256 resolution and batch size of 128. Augmen- tation techniques applied include random horizontal flipping and crop augmentation. For the latter, images are first subjected to random resizing between 256 and 288 pixels before being cropped to a 256-pixel size. We set our learning rate at 10e-4 without incorporating learning rate warm-up phase. Model initialization is performed using the EMA weights from the Stable Diffusion v1.5 checkpoint, and we adopt other training settings from the public Stable Diffusion repository. For the results presented in this paper, we operate 256-pixel resolution with 100 denoising steps. We employ an Euler ancestral sampler with a denoising variance schedule as proposed by [59]. On an NVIDIA A100 GPU, our model takes approximately 10 seconds to solve given vision task.
A.2 EXAMPLES OF THE REPHRASED PROMPTS.
Figure (a)/(b) qualitatively compares the robustness of InstructCV to changes in instruction wording when trained using the fixed prompt or the more diverse rephrased prompt dataset. The model trained using the fixed prompt data shows reasonable performance segmentation tasks, even for prompts that slightly deviate from the standard instruction wording. However, as these deviations increase misclassfications become more common. Instead, the model trained on the rephrased prompt data appears more robust to such changes in task formulation. Notably, the model appears to show basic semantic understanding as it is in the last prompt example able to infer the correct intent despite the simultaneous occurrence of the potential object detection targets 'spider man' and 'face".
A.3 POST-PROCESSING STEPS FOR OBJECT DETECTION TASKS
We employ image processing techniques derive the bounding box coordinates from the output image. First, we apply median and bilateral filters to the image in order to mitigate noise and enhancing features. Following, we convert the image from RGB to HSV space isolate the red region within the target object, which is then extracted from the original image. Next, identify closed contours in the image by converting it to grayscale map, performing threshold segmentation based grayscale, and ultimately, binarizing the image. However, naively applying this approach could potentially result in the removal of some accurate bounding boxes due to the disruptions induced by complex image backgrounds. To circumvent this issue, pross-reference the bounding boxes with the dataset annotations and retain any bounding box predicted by the model that exhibits an Intersection over Union (IOU) greater than 0.5. We exclude disturbances that not rectangular or that contain numerous red dots within the contour. The coordinates of the remaining contours are subsequently added to the prediction list.
14

---

Published as a conference paper at ICLR 2024
Table 4: Rephrased prompts. We automatically generate diverse rephrased prompts generated a diverse rephrased prompts (RP) instruction dataset. Here we display some examples of rephrased task templates.
Estimate the depth of this image.
Make an estimation of how deep the this image is.
Make an estimation of how deep the image goes.
Approximate the depth of this image.
Provide a rough calculation of the image' 's depth.
Make an informed guess of the depth of the image.
Give approximate measurement of the image's 's depth.
Depth estimation
Show * ifere exists % in the figure.
Display an * a % is present in the figure.
If there is a % in the figure, show it as
Show an * if there is a within the figure.
In case a % exists in the figure, display an *
Show th figure contains a %..
a % present in figure,
indicate it with *
Classification
Utilize bounding boxes in order to identify the presence of the %.
Can you help me detect the %
Please employ bounding boxes for the purpose % detection.
Please use bounding boxes to identify the presence of a %..
Detect and identify the 's location.
Locate the % presence.
Detect the %
Delection
Please divide the % into smaller parts.
Please perform image segmentation to isolate the % in
Would you be willing segment the %6?
Please break down the % into individual parts.
Can you provide me with segment of the %
Help me segment the %..
Segment the %
Segmentation
this image.
Prompts
15

---

Published as a conference paper at ICLR 2024
(a) FP model - Different Prompts for Semantic Segmentation Task
Segment the
Could you help the grass?
Please provide segmentation of the grass.
Please highlight the segment.
(b) RP model Different Prompts Object Detection Task
Detect the Spider Man.
Please locate Spider
Detect the face the Spider Man.
Detect the Spider Man's face.
Figure 5: Semantic segmentation performance for different prompt formulations (a) FP model. Results for model trained on fixed prompt (FP) instruction dataset (b) RP model. Results for model trained on diverse rephrased prompt (RP) instruction dataset. Overall the RP model shows greater robustness variations in wording.
Single Category Segmentation Classes
king
Pinecone
Golden Plover
Jackfruit
Single Category Segmentation the External Dataset VOC
Horse
Train
Shoep
Airplane
Multi Category Segmentation the External Dataset VOC
Horse, Person
Monitor, Bottle
Chatc Penson, lable
Car, Blcycle
prompt: Sepment the %, %: the specific categories.
Figure 6: Visualization of semantic segmentation examples. Segmentation categorics provided below each image.
16

---

Published as a conference paper at ICLR 2024
Object Detection on Novel Classes
Tom, Jerry
Dug. Carl
Agnes
Judy
Object Detection on the External Dataset VOC
ap1a
train
cot
sheep
Bird
Shesp
Train, car
Cat, sofa
Object Detection on the Oxford Pets
beugle
shibo inu
shorthaie
Boagle
Birman
Shiba inu
Shorthair
Input prompt: Detect the %.
%: the specific categories.
Figure 7: Visualization of object detection examples. Segmentation categories provided helow the image.
Monocular Depth Estimation on Out-of-Domain Data
Input prompt: Create monocular depth map.
Figure 8: Visualization of monocular depth estimation examples. All depth maps presented here are produced for datasets not included during model training.
17


Reviews:
[{"review_id": "s7IitD0nqd", "reviewer": "Reviewer_wRB5", "summary": "This paper proposes a vision generalist, InstructCV, which casts various computer vision tasks as text-guided image generation. Based on a text-guided image editing model, InstructCV learns to generate visually encoded outputs of different vision tasks. Experimental results on several visual recognition tasks showcase the effectiveness of InstructCV.", "questions": "See weaknesses.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Different from previous attempts to build vision generalists, this paper provides a solution built upon text-to-image diffusion models. Task prompting is achieved by text instructions, which are human-intuitive and general.\n\n2. The experimental results indicate the validity of InstructCV in handling a variety of visual recognition tasks.", "weaknesses": "1. After reading the paper, I am a bit confused about the visually encoded outputs for different tasks.\na) Semantic segmentation: How to derive the mask of all semantic classes for an input image? If predicting them one by one and each class is manually indicated, I would say it is more like a task of referring segmentation rather than semantic segmentation. It would be helpful if the authors could provide more elaboration on this.\nb) Object detection: What do the authors mean by saying “cross-reference the bounding boxes with the dataset annotations” in Appendix A.3?\n\n2. What type of instructions is used during inference? Fixed templates or random rephrased ones? Does it remain the same for InstructCV-FP and InstructCV-RP during inference? As reported in Table 3, InstructCV-FP outperforms InstructCV-RP if both use template prompts during inference, so why bother training InstructCV using rephrased prompts? I understand this could improve the robustness of variations in wording, but one can just adopt a fixed template prompt to address a standard computer vision task. In addition, the rephrased prompts for each task are not that diverse (only 7 variations according to Table 4), why do we need to leverage LLMs as this could be readily done by humans? I would like to see more discussions about the benefits brought by utilizing rephrased prompts.\n\n3. More ablations should be conducted to compare InstructCV-FP with InstructCV-RP. For example, the authors could report the results of InstructCV-FP in Table 1.\n\n4. For the organization of this paper, I suggest the authors move the introduction of diffusion models in Section 2.2 (which takes up most of this section) to a “Prerequisites” subsection or Appendix.\n\n5. The proposed method claims to be a vision generalist but it is validated only in some visual recognition tasks. What about other vision tasks such as low-level tasks (e.g., denoise, deblur, and derain) and generation (as done in Prompt Diffusion [1*])?\n\n6. Further comparisons with other SOTA unified models such as Painter [2*] should be conducted. Painter is currently performing the best in prediction tasks as a vision generalist.\n\n7. In terms of the computational cost, the authors claim that InstructCV exhibits much faster inference speed than previous methods such as Unified-IO. However, InstructCV needs multiple inferences for some tasks (e.g., semantic segmentation and classification), which could consume N×times where N is the number of semantic classes. Besides, the reported inference time of InstructCV on a single NVIDIA A100 GPU for an image is 5 seconds in the main paper but 10 seconds in the Appendix.\n\n8. In the second paragraph of Section 1, the authors claim that “However, these prompts consist of (uninterpretable) numerical values derived from specific training datasets, which may limit their ability to generalize to new datasets, tasks, or categories”. Why do the prompts used in inpainting-based methods (which are visual examples) contain uninterpretable numerical values?\n\n9. The format of references is messy and inconsistent. For example, a bunch of references do not list the conference/journal title.\n\n10. Just a reminder that there is a very similar work, InstructDiffusion [3*], to this submission. The authors could consider citing that paper and provide comparisons and discussions.\n\n[1*] In-context learning unlocked for diffusion models. arXiv, 2023.\n\n[2*] Images speak in images: A generalist painter for in-context visual learning. In CVPR, 2023.\n\n[3*] InstructDiffusion: A Generalist Modeling Interface for Vision Tasks. arXiv, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wP9sN6ZEJM", "reviewer": "Reviewer_ERkX", "summary": "This paper introduces InstructCV, a unified language interface for computer vision tasks that enables task execution through natural language instructions. It leverages text-to-image generation by casting various computer vision tasks as instruction-based image synthesis problems. By creating a multi-modal and multi-task training dataset and applying instruction-tuning to a text-to-image diffusion model, InstructCV achieves competitive performance and compelling generalization capabilities compared to other vision models.", "questions": "1. The advantage of instruction tuning lies in its ability to enable the model to generalize to unseen tasks, which sets it apart from simple multi-task learning. However, in the paper titled \"InstructCV,\" it seems that the approach primarily identifies instructions as task indicators within a predefined set of tasks. This may not demonstrate an advantage over traditional multitask learning. Could the author provide further explanation regarding the unique advantages of this approach?\n2. I believe the good generation ability to new datasets is mostly because the comparison baselines lack the ability to handle open vocabulary. On the other hand, T2I  models are trained on a large amount of text-to-image pairs, which naturally gives them a good open vocabulary capability. Additionally, some unified models like x-decoder also demonstrate excellent generalization ability to new categories.\n\nIn summary, one aspect that concerns me is the lack of clear advantages in unifying dense prediction as an image-to-image generation task. The article mentions instruction tuning, but the experimental results do not generalize to new tasks. The open vocabulary capability is primarily attributed to the T2I models, and I don't believe this alone can be considered unique because The open vocabulary capability has also demonstrated impressive performance in previous works, such as xdecoder[1].\n\n[1] X-Decoder: Generalized Decoding for Pixel, Image and Language", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A valuable contribution of the paper is the introduction of a novel approach to integrate dense prediction tasks with a T2I model. Specifically, the paper suggests utilizing a generative model to perform a task similar to editing, which seems reasonable for tasks like segmentation, and depth prediction compared to sequence to sequence model.\n2. With the help of T2I models, the algorithm has demonstrated good generalization ability in dealing with unseen categories.", "weaknesses": "1. The unique advantage of instruction tuning lies in its ability to generalize to unseen tasks[1], thereby enabling the model to become a general-purpose model. In the article, I believe it resembles more of a multi-task learning approach since the instructions are mapped to several tasks mentioned in the article. In my opinion, this approach does not have clear advantages compared to simply performing these tasks in a multitask learning setting on T2I model.\n2. For certain tasks that do not require dense output, such as classification and detection, it may not be necessary to forcefully address them as image generation tasks. Sequence models are generally more suitable for modeling such tasks, and the approach described in the article seems highly inappropriate in such cases.\n\n[1] InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zV5Y0k2xZp", "reviewer": "Reviewer_QhMY", "summary": "In this paper, the authors focus on the problem of a generalist visual model. To this end, they train a text-to-diffusion model on multiple tasks: Object Detection, Image Classification, Semantic Segmentation, and Depth Estimation. They use standard template phrases for each of these tasks and leverage a LLM to generate paraphrased variations of these questions and commands. They compare their visual model against other generalist models as well as task-specific models on each task. They show a level of performance which is competitive on most tasks.", "questions": "1. Please add more justification in the paper specifically for the use of a single diffusion model versus an ensemble of experts.\n\n2. Minor suggestion: Moving Section 3, Related Work, to come right after the introduction might help the paper flow better.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is very well organized and presented. Figures are very polished and simple to understand.\n\n2. Given the strong performance of task-specific models and LLMs in recent years, the problem in question-generalist visual models-is a timely and relevant topic. \n\n3. Quantitative performance (except for Image Classification) is impressive.", "weaknesses": "1. Performance on image classification is weak versus baselines. It is unclear if diffusion models are well suited to object classification versus the other tasks which are spatially grounded. \n\n2. While generalist visual models are the frontier in Computer Vision, there should be more justification why such a system should rely on a single backend visual model versus a ensemble of experts (i.e. prompts to use Mask-RCNN for segmentation, ResNet for classification, etc.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CXzEeEftf3", "reviewer": "Reviewer_cJRy", "summary": "This paper slightly modifies the pipeline of InstructPix2Pix to use the diffusion model to achieve several compute vision tasks including segmentation, object detection, etc. The experimental results show that the proposed framework can be used to multiple tasks, but the performance is not very impressive.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper first uses instruct-tuning to adapt a pre-trained text-to-image diffusion model to several computer vision tasks.\n2. This paper is well-written and easy to follow.", "weaknesses": "1. The novelty is limited. Formulating computer vision tasks as generative tasks is not novel and has been used in previous works[18]. The overall pipeline highly resembles InstructPix2Pix\n2. The performance is not impressive at all. The results lag far from the SOTA methods in Table 1. Note that SOTA methods of depth estimation are not listed for comparisons and other bold results (VOC) are computed on very old benchmarks. The authors might argue that their method is not task-specific, but I think this is not the reason for such poor results.\n3. The inference pipeline of image classification (Figure 4) seems really weird. I think it would be really hard to perform evaluation on 1000-class ImageNet.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)