Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Authors: Yang Shen
Xiu-Shen Wei
Yifan Sun
YuXin Song
Tao Yuan
Jian Jin
He-Yang Xu
Yazhou Yao
Errui Ding
Venue: ICML 2025 poster (2025)
Keywords: Vision Tasks Understanding
Task-level Zero-shot Generalization
Primary area: deep_learning
TL;DR: Vision task-level zero-shot generalization and instruction-level zero-shot generalization

Abstract: Computer Vision (CV) has yet to fully achieve the zero-shot task generalization observed in Natural Language Processing (NLP), despite following many of the milestones established in NLP, such as large transformer models, extensive pre-training, and the auto-regression paradigm, among others. In this paper, we rethink the reality that CV adopts discrete and terminological task definitions (e.g., "image segmentation"), and conjecture it is a key barrier that hampers zero-shot task generalization. Our hypothesis is that without truly understanding previously-seen tasks—due to these terminological definitions—deep models struggle to generalize to novel tasks. To verify this, we introduce Explanatory Instructions, which provide an intuitive way to define CV task objectives through detailed linguistic transformations from input images to outputs. We create a large-scale dataset comprising 12 million "image input $\to$ explanatory instruction $\to$ output" triplets, and train an auto-regressive-based vision-language model (AR-based VLM) that takes both images and explanatory instructions as input. By learning to follow these instructions, the AR-based VLM achieves instruction-level zero-shot capabilities for previously-seen tasks and demonstrates strong zero-shot generalization for unseen CV tasks.  Code and dataset will be open-sourced.

Explanatory Instructions:
Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Yang Shen Xiu-Shen Wei
Yifan Sun 3t Yuxin Song? Errui Ding²
Tao Yuan Jian Jin Heyang Xu
Yazhou Yao 1
Abstract
learning to follow these instructions, the AR- based VLM achieves instruction-level zero-shot capabilities for previously seeen tasks and demon- strates strong zero-shot generalization for unseen CV tasks. Code and dataset have been openly available on our GitHub repository.
Computer Vision (CV) has yet to fully achieve the zero-shot task generalization observed in Nat- ural Language Processing (NLP), despite fol- lowing many of the milestones established in NLP, such as large transformer models, extensive pre-training, and the auto-regression paradigm, among others. In this paper, we rethink the re- ality that CV adopts discrete and terminological task definitions (e.g., "image segmentation"), '), and conjecture is a key barrier that hampers zero- shot task generalization. Our hypothesis is that without truly understanding previously-seen tasks- due these terminological definitions-deey mod- els struggle to generalize to novel tasks. To ver- ify this, we introduce Explanatory Instructions, which provide an intuitive way to define CV task objectives through detailed linguistic transforma- tions from input images to outputs. We create a large-scale dataset comprising 12 million "im- age input + explanatory instruction + output" triplets, and train an auto-regressive-based vision- language model (AR-based VLM) that takes both images and explanatory instructions as input. By
1. Introduction
Natural Language Processing (NLP) has achieved sig- nificant advancements in task-level zero-shot general- ization, driven by key milestones such as transformer models, large-scale pretraining, and the auto-regression paradigm (Vaswani et al., 2017; Brown et al., 2020; Touvron et al., 2023a). This impressive adaptability stems from the language models' capacity to interpret flexible, open-ended but semantically rich instructions, enabling them to gener- alize across language tasks without additional task-specific fine-tuning.
In contrast, Computer Vision (CV) has not yet unlocked the task-level zero-shot generalization capabilities. Early CV models largely depend on discrete vision task boundaries (ef. Fig. (a)), which restricts their flcxibility and generalization capabilities. While recent efforts, e.g., Lumina- mGPT (Liu et al., 2024), OmniGen (Xiao et al., 2024), and Pix ard (Lin et al., 2024), have attempted to incorporate a wide range of vision tasks via fixed or open-style terminological instructions (cf. Fig. (b)), their zero-shot capabilities re- main confined within seen vision tasks or rely on few-shot prompting for guidance. Our intuition strongly implies that this limitation may stem from the constrained scope of in- structions and existing dision-languee models (VLMs) do not truly understand the previously-seen vision tasks. To be specific, these terminological instructions are consisted of discrete and symbolic task definitions that do not convey the objective behind each task, which can be replaced with any other symbols (e.g., changing "Semantic segmentation" within these instructions into "Task X" or "Task djijvau") without affecting the vision task results.
'School of Computer Science and Engineering. Nanjing Uni- versity Science and Technology, China. "School of Computer Science and Engineering, and Key Laboratory of New Genera- tion Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University. China. 'Baidu VIS. "School of Instrument Science and Engincering, and State Key Labora- tory of Digital Medical Engineering. Southeast University, China. 'ttate Key Laboratory of Intelligent Manufacturing of Advanced Construction Machinery. China. This work was supported by Na- tional Key R&D Program of China (2021 YFA1 100), National Natural Science Foundation of China under Grant (62272231, 62472222), CIE-Tencent Robotics X Rhino-Bird Focused Re- search Program, the Fundamental Research Funds for the Cen- tral Universities (4009002401).. Natural Science Foundation of Jiangsu Province (BK20240080), and the Big Data Computing Center of Southcast University. Correspondence to: Xiu-Shen Wei <weixs gm @ gem@ggail.com comm, Yifan Sun <sunyf15 rrsiinhua.org.cm
To enable deep models to understand vision tasks genuinely, we introduce Explanatory Instructions that characterize vi- sion tasks by providing linguistic descriptions of the task
Proceedings of the nnnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Sementic Segpeensiton
Desaining
Pose to Image
Sementic segmentttionn -imgge
Demining <imgge
Animal pose mep to (caption), <imgge
Gevende thegh for this imagee <magee
Depth Estimatiion
Object Detectton
...
Detect: The zelra. <imege>
(a) Discrete V Vision Tasks
(b) T Terminological Instructions
Explenetory Instuutions: "Overlay bright color. Highlight the strean and around it with vivid red Explanstory Instructions: revealing the the natural appearance the strean
with
Expleratory Instuctions "Renove the droplets on surface, reducing overall diffusion of light to increase the clarity
color overleys, colors the scene. Restore
Expartoy Instuctions "Introduce raindrops the surface, creating blurred effect with diffused reflections through these droplets
Explaretory Instuudions: "Rotate the face
center
Explarstry Irstructions "Shift perspective so that the spoon is angled. Reduce the quantity of the of
the left, shifting the perspactive to a side prafile vivid
maintaining
shredded topping,
Explaatory Instrctions: "Orient the head to frontal perspective, oringing the mouth and eyes to face forward, while adjusting
Exlarday Instuctions: "ndjusting spoon so that it is viewed directly fror above. Next, increase the arount of shredded topping
(c) Ours: Explanatory Instructions and Training via the Constructed Dataset of Explanatory CV Tasks
Explanatory Instruction: "Enhance the segnented and simplified scene by introducing detailed textures and sky with
Zero-shot Task: Caug-lo-Inage
Explamatory Instruction: "Fill in enpty outlines
colors. Roplace the dark blue sky. Transforn
colors layers of depth to the flat contours by Lay
natural
nediut blue ocean into deep blue hue with cain surface. Renove the light green shade from the beach and depict the sand with texture
enhancing brightness gradi-
ents In the sky, shadowling in the nountains, intricate shades among
natural
the flavers
Zero-shot Depti-to-lmage
Explanatory Instruction: "Adjust Lighting to create a nighttine setting with arbient city lights... Add reflections. Rotate the car's the car's
Explanatory Instruction: "Create based
coastal image. Brgin by interpreting gradient calors to identify the relative distance of tandscape ele- ments. Use this information to construct
surface to orientation
face towards
the canera."
coastline with proninent headland and ..
(d) Ours Inference on Unseen Instructions
(c) Ours: Inference on Unseen Vision Tasks
Figure 1. (a) Early CV models designed to handle discrete vision tasks. (b) Recent VLMs terminological instructions (i.e.,
terminological task definitions), e.g., "semantic segmentation" and "pose map". (c) We propose Explanatory Instructions to explain CV tasks' objective and construct the dataset of Explanatory CV Tasks. We train the model via this dataset.
(d) The trained model showcases instruction-leved zero-shot capabilities. (c) By omitting certain human-defined vision tasks in the training dataset (cf. Sec. 4.2), we demonstrate promising vision task-level zero-shot capabilities.
objective, i.e., the detailed transformations from input im- ages outputs, ef. Fig. 1 (c). To provide context, first revisit the concept of vision tasks. Generally, vision task refers to a specific objective or operation within the visual domain that humans or machines need to accomplish (Marr, 2010). However, current vision tasks are constrained by rigid, human-defined boundaries, e.g., controllable genera- tion tasks such as Depth-to to to-Image, Canny-to-Image, Pose-to- Image, dense image prediction tasks such as Segmentation, Pose Estimation, Surface Normal Estimation. In practice, re- searchers present vision tasks to models via discrete and ter- minological definitions, limiting both the articulation of task objectives and the expressive flexibility task descriptions. For instance, when describing a segmentation task, non- experts often use natural, descriptive phrases that, more im- portantly, carry clear and specific meanings, e.g., "overlay the stream with bright blue color" or
"paint the stream in blue and mark the rocks within it in red", rather than narrowly defined termi- nological instructions such as "semantic segmentation" or close variants.
Through this form of instruction, we clarify the true objec- tives of various vision tasks, moving them beyond the con- straints of terminological task categories. We further help enhance the diversity of task expressions within the vision domain, while also strengthening the alignment between text and vision modalities at the task level. To operationalize this idea, we construct the Dataset of Explanatory Computer Vison Tasks (DECVT) that contains 12 million "image input + explanatory instruction + output" triplets. No- tably, for the same image inputs, DECVT provides different explanatory instructions and corresponding output images for varying task objectives. Furthermore, DECVT adopts
2

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
explanatory instructions represent transformations in both directions, e.g., from original images to segmentation out- puts and from segmentation outputs back to original images (cf., the segmentation example in Fig.1 (c)), capturing a richer diversity of task expressions. By omitting certain terminological-baseee vision tasks, e.g., Camaytt--magge and Image-to- Canony hepth-to-Image and Image-to- Deeth,, we conduct iastructionddriver supervised fine-tuning on an auto- regressive- b-ade vision-language model (AR-based VLM), utilizing approximately 1.5 million bidirectional pair of "im- age ++ explanatory instructions ++ image" triplets (i.e,, input image, oulput image, explanatory instructions from input to output and from output to input). Results show that the fine-tuned model exhibits both instruction-level zero-shot capabilities (cf. Fig. 1 (d)) and promising task-level zero- shot capabilities (cf. Fig. I (e)), demonstrating its potential to generalize effectively across diverse vision tasks.
Caption of image A:
A close-up of traditional moon- cake with an intricate, embossed design on its golden-brown crust, placed on decorative white doily.
Caption of image B:
A simplified version of the moon- cake where only the main shapes and contours of the design are vis- ible. showing stark contrast be- tween the outlines and the back- ground, with detailed textures re- moved.
Explanatory instruction from A to B:
"Reduce the detailed visual elements, focusing only on the essential shapes and contours of the design. Strip away color and texture, retaining only the prominent outlines to emphasize the primary form and structure againat a plain background."
The main contributions of this work are as follows:
We propose Explanatory Instructions that intuitively ex- plain task objectives by characterizing diverse vision tasks through linguistic descriptions of the transformations be- tween paired images. We further construct Dataset of Explanatory CV Tasks, which, to the best of our knowledge, is the first dataset to integrate multiple vision tasks while providing instructions that authentically describe the task objectives.
Explanatory instruction from to
"Add layers of color, texture, and depth to the outlined shapes, recreating the intricate details and embossed patterns. Introduce a warm, golden-brown hue to bring aut the richness of the surface, restoring the complete and detailed appearance of the object with its decerative base."
We conduct experiments to demonstrate that, after learn- ing to follow the proposed Explanatory Instructions, the AR-based VLM model exhibits zero-shot generalization not only at the instruction level but also at the vision task level. Besides, fine-tuning on the constructed dataset further enhances the model's versatility, enabling it to handle any combination of tasks, rather than being restricted to a single task that relies on specific terminological instructions. This advancement represents a significant step toward more flexible and unified understanding of vision tasks.
Figure 2. Examples of terminological based vision tasks, e.g., holistically nested edge detection.
two components contain approximately 6 million bidirec- tional pairs of "image ++ explanatory instructions + image" triplets (i.e., around 12 million individual "input image + explanatory instruction - output" triplets). Below, we pro- vide an introduction along with example entries for each component. Please refer to Appendix A for specifics on the dataset construction.
2. Dataset of Explanatory CV Tasks
2.1. Terminological-basee Vision Tasks
We construct Dataset of Explanatory CV Tasks (DECVT) based on our proposed Explanatory Instructions fashion, where vision tasks are represented exclusively through tex- tual descriptions of transformations between images, ef- fectively clarifying each task's objective without relying on predefined terminological categories. During the con- struction of DECVT, we consider the discrete nature of the established concept of computer vision tasks and divide the dataset into two components, including "Terminological- based Vision Tasks" and "Explanatory-based Vision Tasks". The first component primarily includes tasks whose objec- tives have been abstracted into standardized terminologies, while the second component encompasses tasks that re- quire explanatory deseriptions differentiate them. These
This component primarily encompasses low-level vision tasks, controllable generation tasks, dense image predic- tion tasks, image grounding tasks and their inverse coun- terparts. While these tasks have been refined and suc- cinctly abstracted through human summarization, their in- herent discreteness limits the model's ability to compre- hend these vision tasks fully. Therefore, we aim to reduce the rigidity by providing detailed explanatory instructions that explicitly articulate the underlying objectives of each task. To further explore task-level zero-shot generaliza- tion capabilities, this component includes only a subset of terminological-
vision tasks. Specifically, we incor- porate low-level data from open-soure datasets including AllWeather (Valanarasu et al., 2022), NYU-Rain (Li et al.,
3

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
2019b), D-HAZE (Ancuti al., 2016), NH-HAZE (An- cuti al., 2020), UIEB (Li et al., 2019a), Adobe-5k (By- chkovsky et al., 201 covering restoration tasks of Im- age Restoration, Deraining. Dehazing and Desnowing. We also collect Object Detection data from the LVIS (Gupta et al., 2019) dataset, Style Transfer data from CSGO (Xing et al., 2024) and prepare data for dense image prediction tasks including Depth Estimation, Surface Normal Estima- tion, Pose Estimation, and Semantic Segmentation from ADE20K (Zhou et al., 2017b), Depth in the Wild (Chen et al., 2016) and randomly selected data from the next two components, Additionally, for controllable generation tasks, we incorporate Holistically-Nested Edge (HED) Boundary to Image and inverse dense image prediction tasks (e.g., Pose-to-Image,
Caption of image A:
A glass of iced latte with a light creamy color sits on a wooden sur- face. The drink has visible ice cubes floating at the top, creating a refreshing appearance, The back- ground is softly blurred, focusing the attention on the drink and en- hancing its warm, cozy aesthetic.
Caption of image B:
A glass of iced coffee with distinct layers, where dark espresso swirls into the lighter creamy base. The drink is set on a hexagonal coaster placed textured glass surface, with a blurred, minimalistic back- ground. The presentation empha- sizes the richness of the espresso and gives modern, refined feel to the scene.
tiitenttattontt--maag)) Examples for this component are shown in Fig. 1 (c) and Fig. 2. Details and additional examples provided in Appendix A.2 and Appendix A.3.2.
marth-to-Image
2.2. Explanatory-based Vision Tasks
Explanatory instruction from A to B:
Compared to language tasks, vision tasks inherently re- quire more intricate forms of expression. While leveraging terminological-base vision tasks offers solid foundation, their predefined boundaries can limit the diversity and com- plexity of task objectives. To address this, we introduce explanatory-basse vision tasks, which utilize descriptive in- structions to articulate the transformations between images, enabling broader and more diverse range of vision task objectives.
"Change the wooden surface beneath a chilled drink presentation to glossy, rextured platform that refracts light. Alter the arrangement and appearance of the beverage to have a swirl of ingredients visible from the side, creating smoother gradient effect. Adjust the lighting to soften shadows and Introduce more ambient illumination across the scene, highlighting the layers and making the glass's 's ridges appear more prominent." Explanatory instruction from B to "Replace the sleek, reflective surface beneath the clear beverage vessel with a warm, rustic wooden texture that enhances the natural, earthy tone of the scene. Modify the beverage's visual layers to achieve a more blended appearance with a focus on a single color. Adjust lighting conditions to create sharper contrasts and distinct edges."
We first collect data related to instruction-based image editing, where diverse editing instructions enable users to perform straightforward edits using natural language commands. However, each editing instruction typically involves only a single operation of modification, such as adding, removing, or modifying the background or spe- cific objects within an image, while preserving the overall structure and contextual integrity of the unaltered regions. Specifically, we directly incorporate publicly available im- age editing datasets including MagicBrush (only involy- ing the real world and multi-turn subsets) (Zhang et al., 2023a), SEED (Ge et al., 2024), HQ-Edit (Hui et al., 2024), HIVE (Zhang et al., 2024), InstructPix2Pix (Brooks et al., 2023) and PromptFix (Yu et al., 2024).
Figure 3. Examples of explanatory-basee vision tasks. cate lighting compositions, nuanced adjustments to scene composition and visual ambiance, or macro-level visual content changes but with similar features. These complex dynamics extend beyond the capabilities of conventional terminological-basee based vision tasks and require more intricate language instructions. We collect images to expand this component from search engines, with examples shown in Fig. (c) and Fig. 3. Details on constructing these explana- tory instructions and additional examples are provided in Appendix A.1 and Appendix A.3.1.
Afterward, we seek transcend established vision task paradigms by incorporating more visual-related image pairs, thereby expanding the scope of task objectives. The trans- formations between these image pairs involve sophisti- cated variations which cannot be adequately described with straightforward instructions that typically focus on individ- ual changes, e.g., combinations of camera angles with intri-
3. Methodology
As CSGO has not been open-sourced, we only collect data from the paper.
The primary objective of this work is to demonstrate that VLMMs can achieve both instruction-level and vision task-

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Descriptions from soutc to barptt Overlay the stroom amoo with a bright blue color. Highlight the rocks within the stream and around it with vivid red color. Divide the forested boclgpourd into two soctiona: the night side with a light green oveday and the left adde.. Ensure the color overlys are transtucent enoggh reveel the uncerlying bees follage...
Explanatory Instructions
Source Images
S Text. Tokenizer
S VQ-VAE Decoder
G VQ-VAE Encoder
Rinetanddd
Fromm
Sourre turge
Large I. wwwguagggeeaaaoooooo Models
Target souace
VQ-VAR Encoder
VQ-VAE Decoder
Text Tokenizer
Target Images
Explanatory Instructions
Desciptions from larget to soure: Restore the nabural appeerance of stresmm rodks and forested
aness Highlight the vibrent fall colors of the follage, inchuding shades of green, yellow, and
orane Maintan gantle flow water over rocky stramm bed and the sumlight filtering, throuh dense tre canogy. Pasure the groudd remmins covared with lealen and rocks....
Figure 4. Framework of our vanilla token-based VLM method.
level zero-shot capabilities using the proposed Explanatory Instructions, which characterize vision tasks through lin- guistic descriptions of their objectives, rather than to pro- vide a oftt--of-the-ar solution. Consequently, we adopt a straightforward token-based VLM that integrates an im- age lokenizer/deetooeniiee with a general mixed-modal AR model. The overall framework is illustrated in Fig. 4.
This approach eliminates the need for random initialization or language-only model initialization, conserving substantial computational resources otherwise required for establishing foundational image generation ca- pabilities. Consequently, we can concentrate on exploring zero-shot generalization at both the instruction level and the vision task level within a vanilla token-based VLM.
ation capabilities.
Architecture To create unified token sequence, both text and images should be tokenized into discrete spaces. Fol- lowing Chameleon (Team, 2024) and Lumina-mGPT (Liu et al., 2024), we employ a byte pair encoding tokenizer (Sen- nrich, 2015) for the text modality and use quantization- based tokenization method to convert continuous image patches into discrete tokens for the image modality (van den Oord et al., 2017; Razavi et al., 2019; Esser et al., 2021; Ramesh et al., 2021). After projecting text and image inputs into a unified sequence by concatenating the text tokens and the flattened ID image tokens, we employ decoder-only autoregressive transformer utilizing the stan- dard dense transformer architecture to simplify the genera- tive modeling. Following I. 2.LaM-22 (Touvron et al., 2023b), we adopt adaptations including RMSNorm (Zhang & Sen- nrich, 2019) for normalization, SwiGLU (Shazeer, 2020) for activation and rotary positional embeddings (RoPE) (Su et al., 2024) for positional encoding. To enhance train- ing stability, Pre-LayerNorm (Xiong et al., 2020), Post- LayerNorm (Ding et al., 2021) and Query-Key Normaliza- tion (QK-Norm) (Henry et al., 2020) are incorporated into each transformer block. Unambiguous Image Representa- tion (Uni-Rep) (Liu et al., 2024) are added to enable image generation at flexible resolution and aspect ratio.
Supervised Fine-tuning During training, the decoder- only auto-regressive transformer models the conditional probability p( [21, using the standard next-token prediction objective. Follow- ing previous works (Chowdhery et al., 2023; Wortsman et al., 2023), we apply 2-loss to prevent the model's output logits from becoming overly confident, thereby enhancing generalization. To streamline the supervised fine-tuning pro- cess, we organize all the data into single-dialog formats (e.g.. for multi turn editing data, if image IA is edited to image Ig by instruction a and then to image Ic by instruction B, we treat this as a single sequence where image IA is edited into image Ic by instruction a + B8), applying the loss func- tion exclusively to output tokens. For all the experiments, we employ the Adam (Loshchilov, 2017) optimizer with a weight decay of 0.01 and betas set to (0.9, 0.95). The learn- ing rate is configured at 4 x 10-5, and the z-loss is applied with a weight of 10 5 To increase training throughput, all the data are pre-tokenized before training and clustered based on the number of tokens.
of multimodal sequences
Inference During inference, any description associated with the input image can be provided as an explanatory instruction, guiding the fine-tuned token-based VLM to generate the corresponding output image based on both the image and the instruction. Furthermore, the sampling strategy for auto-regressive models involves various hyper- parameters that significantly affect the results. For example, while the top-k value is typically set to 5 for text generation in LLMs, we recommend increasing the top-k to 2048 for the image generation stage.
The core contribution of this work lies in redefining vision tasks through Explanatory Instructions and investigating their potential for generalization, shifting the focus away from pretraining a decoder-only autoregres- sive transformer from scratch. To achieve this, we leverage a multimodal generative model pretrained on large-scale scale image-text datasets, which provides flexible image gener-
Initialization
MMore discussions are provided Appendix
5

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Explanatory Instruction: "Transform the water elements into fire elements, using a burning flame as the main theme. The hair becomes flame-like, in orange and red tones, rising upwards with subtle sparks. The dress is redesigned with floving flame textures, resembling burning silk, creating dynamic contrasts between light and shadow. The background shifts to resemble molten lava, with lively fire flicker."
Input Image
Explanatory Instruction: "Dim the scene's a brightness, reduce the intensity of sunlight, and adjust the environment to a rainy, overcast setting, Add sense of light drizzle and mist to the scene, creating a soft yet slightly melancholic atmosphere. Gradually increase the vibrancy of the pink tones around the petals, but give them a faintly faded and blurred appearance, and introduce sense of decay."
Output Image
Input Image
Output Image
Explanatory Instruction: "Change the color of the car red while try to keep all other objects in the scene unchanged. Add a layer of anow to the surfaces of the road, the caz, and the trees to create the atmesphere of a winter morning. Pay close attention to details auch as shadows and lighting adjustentss."
Input Image
Output Image
Figure 5. Examples of instruction-level zero-shot capabilities.
4. Experiments
potential to become a versatile vision generalist. Note that, in addition to image editing examples, our proposal also performs well on traditional discrete vision tasks, such as Semantic Segmentation, Surface Normal Estimation, and more, detailed in Appendix C.
In this section, we evaluate the model's zero-shot capabili- ties on unseen instructions and unseen vision tasks through different experimental settings.
4.1. Zero-shot Capabilities on Unseen Instructions
4.2. Zero-shot Capabilities on Unseen Vision Tasks
Settings We fine-tune a 7B AR-based VLM by utilizing all the DECVT training data. The pre-trained model used to initialize is Lumina-mGPT-7B--- 108-Omni".. Training is conducted on 64 A100 GPUs, with batch size for each GPU set as 8 over 2 epochs (around 47k iterations), totaling 5,400 GPU hours. Image resolution equals 448 x 448.
Settings To directly assess the task-level zero-shot capa- bilities of the decoder-only autoregressive transformer fine- tuned on the constructed DECVT, we exclude certain tasks from the DECVT training data. Specifically, we remove data corresponding to Image Restoration, Depth Estima- tion, Surface Normal Estimation, Hepth-to-Image Sur- face Normal-to-Image, JHEDDttolmmage and HED Bound- ary Detection tasks from the "Terminological-basee based Vi- sion Tasks" component. For quick validation, we utilize a subset of the DECVT training data: 30% data from the "Explanatory-basee Vision Tasks" component (50% image editing data while the remaining 50% data from more visual- related image pairs) and 20% data from the remaining por- tion of the "Terminological-baie with each subset containing approximately 0.5 million bidi- rectional pair i ++ explanatory instruction ++ image" triplets (totally 1.5 million bidirectional pair of triplets). The pre-trained model used to initialize is "Lumina-mGPP: 7B-768". Training for the 7B AR-based VLM model is conducted on 8 A100 GPUs with batch size set as 128 over epochs (around 46k iterations), totaling 1,340 GPU hours. Image resolution equals 448 x 448.
Instruction-level Zero-shot Capabilities Certain terminological-basse tasks in NLP, egg., Machine Transla- tion, Emotion Detection, Named Entity Recognition, and Relation Extraction, have clearly defined task boundaries (i.e., their objectives are specific). However, many other language tasks, such as Dialogue and Question Answering, lack such clear task objective boundaries. Similarly, we vision tasks
argue that the set of terminological- represents only small subset of possible vision-related tasks. Many vision-related tasks are likely analogous to Dialogue in NLP, lacking clear task definitions or boundaries, As result, it is difficult to determine whether certain vision tasks are explicitly included during dataset construction (just as dataset for LLMs often include Dialogue or Question Answering data). However, it is easy calculate textual similarity to verify whether the instructions associated with given image part of the training set. Therefore, we refer to the model's ability to generalize on such instructions and images as instruction- zero-shot capabilities. Examples in Fig. 5 provide evidence that the bR-based VLM fine-tuned with the proposed explanatory instructions exhibits promising
Vision Tasks" component,
Task-level Zero-shot Capabilities We evaluate the fine- tuned model's task-level zero-shot generalization capabili- ties on three previously unseen terminological- bbased vision tasks, i.e., -mmaee, Canny-to- -mmgee and Depth-to- Image, as directly testing on these unseen vision tasks pro-
6

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Input Image
Output Image
Figure Examples of task-level zero-shot capabilities (HED-to-Image) Resolution: 448x Explanatory Instruction: "Gradually restore the scene's natural colors, filling in each ragion with real istic gradients and textures that represent natural elements. Reintroduce datail ils such as color gradients in the sky, reflect ions in the water, and varied shades across fields and trees. Add depth by applying soft shading to convey lighting and shadow, ensuring the scene captures the natural flow of colors, reflections, and atmospheric lighting typical of landscape under daylight."
Input Image
Output Image
Figure 7. Examples of task-level zero-shot capabilities (Canny-to-Image) Resolution: 448x448. Explanatory Instruction: "Fill in the empty out ines rich colors that refl ect vibrart tones, smooth textures. Add layers of depth to the flat contours by enhancing brightness gradi ents in the sky, shadowing the mountains, and intricate shades among the flowers. Reintroduce the sensation of space and dimension by contrasting sharp objects with nuted backgrounds and crisp details in the foreground."
redefining shapes with
Input Image
Output Image
Figure 8. Examples of task-level zero-shot capabilities (Depth-to-Image). Resolution: 448x448. Explanatory Instruction: "Create realistic coastal scene based on the image. Begin by interpreting the gradiens colors to identify the relative distance of landscape elements. Use this information to construet a rocky coastline with prominent headland and isolated rock. Add natural colors for sea and sky, and introduce texture and shading to the rocky surfaces, alming to replicate dask setting with soft lighting and gentle waves."
vides the most intuitive demonstration of the model's task- level zero-shot potential. For each task, we provided lin- guistic descriptions, termed as "Explanatory Instructions". which detailed the desired transformations for the input im- ages. These instructions, input along with the source images, guided the model in generating outputs that closely aligned with the specified transformations, as shown in Fig. 6, Fig. 7 and Fig. 8. These successful interpretation and generation processes, without prior exposure to these tasks, highlights potential and feasibility of the proposed explanatory in-
structions in promoting task-level zero-shot learning within the vision domain. For additional task-level zero-shot exam- ples, please refer to Appendix C.
4.3. Quantitative Results
In this section, we evaluate the fine-tuned AR-based VLM on generation tasks including moany-to-Image. HED-to- Image, Inpainting and Outpainting, dense image prediction tasks including Semantic Segmentation, Depth Estimation

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Table 1. Comparisons with task-ipe vision generalist baselines across four representative tasks. Our model adopts unseen explanatory instructions during the inference stage. We use GPT-4o (OpenAI, 2024) to recaption images in the MultiGen-20M (Qin et al., 2023) validation set the original dataset do not provide captions in general nature language format. Scores in the format of "a /b" indicate CLIP-Score for "previous/ / recaption" images and texts. "x" indicates that the method is incapable of performing the task. "-" indicates that the methed does report the corresponding results. "T. 7..s." denotes task-level zero-shot, i.e., hoth the task and instructions seen during training. "1. Z.".." denotes instruction level zero-shot, i.c. the instructions are not seen during training.
IT. Z.-s.] Canny-to- [I. Z.-s.] HED-to-Image IT. Z.-s.] Inpainting
IT. Z.-s.] Outpainting
Methods
FI FIDL CLIP-ST
SSIMT FIDL CLIP-ST FID LPIPS
FIDI
1ST
MultiGen-20M
MultiGen-20M
Places
Places
Original Images
100.00 0.00
31.99/33.13
1.0000 0.00 31.99/33.13
0.00
0.00
0.00
27.70
ControlNet-SDL.5 (2023b)
34.65
14.73
32.15
0.7621
15.41 32.33
X
X
X
x
121-Adapter-SDD (2024)
23.65
15.96
31.71
X
x
x
IDMM4 (2022)
X
X
X
9.39
0.25
LaMa (2022)
X
X
X
X
X
X
12.00
0.24
DeepFill v2 (2019)
X
11.51
17.70
MaskGIT (2022)
X
X
X
X
7.80
22.95
OmniGen (2024)
35.54
0.8237
PixWiz (2024)
35.46 15.76
32.01
9.27
0.25
7.54
22.18
Lum (2024)
10.09
61.65
25.33
0. 1818 69.49
25.97
42.69
0.77
42.66
11.28
Ours
20.69
26.93
27.16
0.6561 15.91
29.59
15.16
0.48
15.15
17.20
Table Comparison with task spocific and vision generalist baselines across seven dense image prediction or low-level tasks. Our model
adopts unseen explanatory instructions during the inference stage. *** indicates the result is higher than typically expected under standard
evaluation method and is provided for reference only. "x" indicates that the method is incapable of performing the task. "_" indicates that the method does not report the corresponding results. "T. Z.-s." denotes task-level zero-shot, i.e., both the task and instructions are not
seen during training. "I. Z.-.." denotes instruction-levs zero-shot, i.e. the instructions are not seen during training.
[I. Z..s.] Depth Est. Semantic Seg. [T. Z..s.] Surf. Norm. Est. [T. Z..s.] Derain [I. 7.-s.] Dehazing [T. Z.-s.] (UDC)IR
[T. Z. Low-light Enh.
Methods
RMSEL
mloUt
Mean Angle Urroe)
PSNRT SSIM PSNRT SSIM
PSNRT SSIM
PSNR
SSIM
NYU-Depth V2
ADE20K
NYU-Depth V2
Rain100L
SOTS
UDC(T-OLED)
LOLV2
DepthAnything (2024)
0,206
Marigold (20244
0.224
X
Mask DINO (2023a)
60.80
x
Mak2Former (2022)
56,10
Bac et al. (2021)
X
14,90
X
X
X
X
X
X
In<PT (2022)
19.04
AirNet (2022)
32.98
0
21.04
0,884
26,76
0.799
19.69
0.821
PromptR (2023)
X
36.37
0.972
30.58
0.974
21.23
0.860
Unifled-10 (2022)
(2022)
0.387
25.71
X
X
X
Painter (2023)
0.288
49.90
X
29.87
0.882
InstructCV (2024)
0.297
47,25
X
x
InstmitDiffusionn (2024)
x
x
x
19.82
0.741
WixWizard (2024)
0,287
36.76
19.65
31.43
0.917
0.937
27.22
0.807
Lamina-mGPT (2024)
X
28.49
x
x
X
Ours
0.553
42.12
27.21
16.72
0.462
16.90
0.542
14.54
0.332
14.48
0.371
and Surface Normal Estimation, low-level tasks including Deraining, Dehazing, Image Restoration and Low-light En- hancement. Training settings follow Sec. 4.1 and results are shown in Table and Table 2. Details for these datasets, evaluation settings and more quantitative results please refer to Appendix B.1.2 and 1.1.3.
constructed with the proposed explanatory instructions ex- hibit task-level zero-shot capabilities, it is important to note that these capabilities are observed primarily in genera- tion tasks (e.g., HE.--to-lmage, Canny-to- -m-age Depth- lo-Image, Outpainting and Inpainting) and low-level vision tasks (e.g., Low-light Enhancement and Deblurring). How- ever, it fails to exhibit task-level zero-shot capabilities in vision tasks like Image-to- ta--nyy or spite these limitations, it is important to highlight that the task-level zero-shot capabilities observe are fundamen- tally different from the exed-instruction forms commonly used in controllable generation tasks. In fixed-instruction scenarios, models not only know the task objectives but are also given the captions for the images to generated. In contrast, by providing varied explanatory instructions, we
5. Limitations and Discussions
De-
In this section, we primarily focus on the most critical aspect-task-level zero-shot capabilities. For a more com- prehensive discussion on limitations and additional insights (e.g., dataset constraints and model stability), please refer to Appendix D. While the experiments in Sec. 4.2 and Ap- pendix C demonstrate that models fine-tuned on datasets
8

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
guide the model to understand the specific task objective, thereby expanding its ability to handle a diverse range of tasks.
In the ongoing exploration find a unified approach for handling diverse vision tasks and enabling vision task-level generalization, Chameleon (Team, 2024) is among the first to apply the same transformer architecture to sequences of both image and text tokens, enabling the generation of both text descriptions and image representations within the LLM paradigm. Show-O (Xie et al., 2024) also demon- strate unified models by combining text and image genera- tion capabilities, while these models are limited to specific generation tasks. Lumina-mGPT (Liu et al., 2024), Omni- (Lin et al., 2024)
Regarding the scope of task-level zero-shot capabilities, we hypothesize that the primary reason for this limitation is the lack of alignment between the image tokenizer (e.g., VQ-VAE or VQ-GAN) and the text modality during pre- training. If the pretrained model used for initialization lacks the ability to generate images resembling Canny edges or depth maps, the fine-tuned model struggles to generalize to these tasks based solely on linguistic descriptions. In contrast, for controllable generation tasks such as Canmy- to-Image, the text concepts (e.g.. "sky") have already been aligned with image tokens in the pre-trained model through paired text-image data. As a result, it is less affected by the lack of alignment between the image tokenizer and text modality, enabling task-level zero-shot capabilities through our proposed explanatory instructions.
Gen (Xiao et al., 2024), and then leverage fixed or open-style terminological instructions, employing either LLMs or flow-based Diffusion Transform- ers (DiT) (Ma et al., 2024) as foundational architectures to integrate a wide range of vision tasks. Despite these ad- vancements, these so-called vision generalist models still fail to exhibit vision task-level generalization. We infer the primary reason is that these models still rely on terminolog- ical instructions such as "Semantic segmentation", which confine the model to perform only the specific tasks de- fined by these terms or closely related tasks (e.g., different forms of Image Restoration). In addition, the scope of these terminological instructions remain constrained and do not explain the true objective of vision tasks, far from the flexi- bility and richness of task expressions and objectives in the field of NLP. In this work, we move beyond the traditional notion of computer vision tasks and introduce explanatory instructions to explicitly articulate the objectives underlying different visual changes, forming an expansive set of uncon- strained vision tasks that enable the model achieve vision task-level generalization.
6. Related Work
Vision Task Understanding With the groundbreaking success of LLMs (Brown et al., 2020; Touvron et al., 2023a) in the field of NLP which have unified language tasks, re- searchers are no longer satisfied with earlier proprietary vision models (where each model handles a specific vision task). Instead, they seek to leverage the language modality to better understand the vision modality (Lian et al., 2025; Fu et al., 2025) and associated vision tasks. Advancements in these VLMS (Liu et al., 2023a; Li et al., 2023b; Alayrac et al., 2022; Dai et al., 2023; Zhu et al., 2023; Chen et al., 2024) have enabled substantial progress vision-language tasks, achieving outstanding performance across diverse ap- plications. However, early VL.Ss primarily produce textual outputs, which constrains their capacity to represent and manipulate visual information. To push beyond these limi- tations, researchers have explored integrating task-specific extensions such as expert models and downstream tools to enhance VLMs' ability to handle diverse downstream vision tasks (Liu et al., 2023c: Wu et al., 2023; Liu et al., 2023b; Fei et al., 2024; Huang et al., 2024; Gan et al., 2024; Geng et al., 2024; Wu et al., 2024). However, such simple exten- sions can not enable models to generalize across computer vision tasks. As the pursuit of Artificial General Intelligence (AGI) continues to grow, there is an increasing demand for unified foundational models that can tackle diverse vision tasks and achieve zero-shot generalization at vision task level.
7. Conclusion
A significant body of research (Brown et al., 2020; Touvron et al., 2023a; Radford et al., 2021; Wei et al., 2023; Zhang et al., 2025) has demonstrated that, with sufficient training data, models can exhibit zero-shot generalization capabili- ties. Building on these findings, this work goes beyond the traditional notion of "vision tasks" by introducing explana- tory instructions to reveal the true objectives underlying var- ious vision tasks. In constructing the Dataset of Explanatory CV Tasks, we effectively captured and described a diverse range of non-terminological vision tasks. Through straight- forward fine-tuning experiments on a vanilla token-basee VLM, we showed that this approach enables the model to achieve both instruction-level and vision task-level zero- shot capabilities. Although the model still faces certain limitations, we believe this work represents a step toward achieving broader zero-shot generalization in computer vi- sion by leveraging explanatory instructions. This method overcomes the constraints of conventional vision task defi- nitions and paves the way for more flexible and universally applicable unified generative models.
Vision Task Generalization Although carly VLMs have demonstrated some degree of generalization ability, it largely limited to visual question answering, where the ques- tion answering capability inherently possessed by L.MSs.
9

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Acknowledgements
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
The authors would like to extend their sincere thanks to Xuhao Sun for his valuable discussions during the early stages of this work. Special thanks are also due to Mengxi Zhang and Chuyang Zhao for their contributions in review- ing and verifying the code. Additionally, we express our gratitude to the Chameleon team and the Lumina-mGPT team for their contributions to the open-source community.
Askell, A., Agarwal, S., Herbert- A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E.,, Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Advances in Neural Inf. Process. Syst., pp. 1877-1901, 2020. 1,9
Impact Statement
Bychkovsky, V., Paris, S., Chan, E.,, and Durand, F. Learning photographic global tonal adjustment with database of input/output image pairs. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 97-104, 2011. 4
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Chang, H., Zhang, H., Jiang. L.,, Liu, C.,, and Freeman, W. T. MaskGIT: Masked generative image transformer. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 11315-11325, 2022. 8, 8,28
References
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I, Hasson, Y., Lenc, K., Mensch, A., Millican, K.,
Chen, W., Fu, Z., Yang, D., and Deng, J. Single-image depth perception in the wild. In Advances in Neural Inf. Process. Syst. pp. 730-738, 2016. 4
Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei. S., Monteiro, M., Menick, J. L., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M. a., Barreira, R., Vinyals, O., Zisser- man, A., and Simonyan, K. Flamingo: A visual language model for sho-shot learning. In Advances in Neural Inf.
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., Li, B., Luo, P., Lu, T., Yu, Q.. and Jifeng, D. InternV:: Scaling up vision foundation models and aligning for generic visual- linguistic tasks. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 24185-24198, 2024. 9, 16
Process. Syst., pp. 23716-23736, 2022. 9
Ancuti, C., Ancuti, C. O., and De Vleeschouwer, C. D- HAZE: A dataset to evaluate quantitatively dehazing al- gorithms. In Proc. IEEE Int. Conf. Image Process. pp. 2226-2230, 2016. 4
Cheng, B., Misra, I., Schwing, A. G., Kirillov, A., and Gird- har, R. Masked-attentinn mask transformer for universal image segmentation. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 1290-12999 2022. 8
Ancuti, C. O., Ancuti, C., and Timofte, R. NH-HAZE:
An image dehazing benchmark with non-homogeneous hazy and haze-free images. In JEEE Conf. Comput. Vis. Puttern Recog. Worksh., pp. 444-445, 2020. 4
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levskaya, A., Ghemawat, S., Dev. S., Michalewski, H., Garcia, X., Misra, V., Robinson, K., Fe- dus, L.. Zhou, D., Ippolito, D., Luan, D., Lim, H., Zoph, B., Spiridonov, A., Sepassi, R., Dohan, D., Agrawal, S., Omernick, M., Dai, A. M., Pillai, T. S., Pellat, M., Lewkowycz, A., Moreira, E.,, Child, R., Polozov, O., Lee, K., Zhou, Z., Wang, X., Saeta, B., Diaz, M., Firat, O., Catasta, M., Wei, J.,
Anthopicc. Claude 3.5 sonnet, 2024. URL https: / www.
anthropic. c.c. com / hww////www///e/eeeeeeee//e httww//clae cww.eeaaaaeeeeeeeeee https: www.://wwwhopp...c...o.. www.c/o/////./eee /
claude- 3- 5--5-oone sonnet. 16
Bae. G. and Davison, A. J. Rethinking inductive biases for surface normal estimation. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 9535-9545, 2024. 19
Bae, G., Budvytis, I., and Cipolla, R. Estimating and exploit- ing the aleatoric uncertainty in surface normal estimation. In Proc. IEEE Int. Conf. Comp. Vis., pp. 13137-13146, 2021. 8
K., Eck, D., Dean, J., Petrov, S., and Fiedel, N. PALM: Scaling language modeling with pathways. J. Mach. Learn. Res., 24(240): 1-113, 2023. 5
Brooks, T., Holynski, A., and Efros, A. A. InstructPix2Pix: Learning to follow image editing instructions. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 18392-18400 18402, 2023. 4, 27
Dai, W., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. InstructBI.IP, Towards
10

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
general- vision-language models with instruction tuning. arXiv preprint arXiv:2305.065500 2023. 9
Hui, M., Yang, S., Zhao, B., Shi, Y., Wang, H., Wang, P. Zhou, Y., and Xie, C. HQ-Edit: A high-quality dataset for instruction-based image editing. arXiv preprint arXi::2404.0900 2024. 4
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for lan- guage understanding. arXiv preprint 180.000800 2019. 42
Ke, B., Obukhov, A., Huang, S., Metzger, N., Daudt, R. C., and Schindler, K. Repurposing diffusion -based image generators for monocular depth estimation. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp- 9492-9502, 2024.
Ding, M., Yang, Z., Hong, W. Zheng, W., Zhou, C., Yin, D., Lin, J., Zou, X., Shao, Z., Yang, H., and Tang, J. Cogview: Mastering text-to-image generation via transformers. In Advances in Neural Inf. Process. Syst. pp. 19822-19835,
Li, B., Ren, W., Fu, D., Tao, D., Feng, D., Zeng, W., and Wang, Z. Benchmarking single-image dehazing and be- yond. IEEE Trans. Image Process., 28(1):492-505, 2018. 28
2021. 5
Esser, P., Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn.. pp. 12873-12883, 2021.
Li, B.,, Liu, X., Hu, P., Wu, Z., Lv, J., and Peng, X. All-in- one image restoration for unknown corruption. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 17452-17462, 2022. 8
Fei, H., Wu, S., Zhang, H., Chua, T.-S., and YAN, S. VIT- RON: A unified pixel-level vision LLM for understand- ing, generating, segmenting, editing. In Advances in Neu- ral Inf. Process. Syst, 2024. https:: http///ppeeeeeeeeeeeeeeeeeeeeeee net/fofee forum? id- KPs.ddh5mmmssMss.... 9
Li, C., Guo, C., Ren, W. Cong, R., Hou, J., Kwong, S., and Tao, D. An underwater image enhancement benchmark dataset and beyond. IEEE Trans. Image Process., 29:
4376-4389, 2019a. 4
Fu. T., Yang. Z., Ye, Z., Ma, C., Han, Y., Luo, Y., Wang, X., and Wang, Z. A survey on the scheduling of DL and LLM training jobs in GPU clusters. Chinese Journal of Electronics, 25, 2025. 9
Li, F., Zhang, H., Xu, H., Liu, S., Zhang, L., Ni, L. M., and Shum, H.-Y. Mask DINO: Towards a unified transformer- based framework for object detection and segmentation. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 3041 3050, 2023a. 8
Gan, Y., Park, S., Schubert, A. M., Philippakis, A., and Alaa, A. InstructCV: Instruction -unnodd--ttdddddd text-to-image diffusion models as vision generalists. In Proc. Int. Conf. Learn. Representations, 2024. 8,
Li, J., Li, D., Savarese, S., and Hoi, S. BLIP-2: Boot- strapping language-image pre-training with frozen image encoders and large language models. In Proc. Int. Conf. Mach. Leurn., pp. 19730-19742, 2023b. 9
Ge, Y., Zhao, S., Li, C., Ge, Y., and Shan, Y. SEED-data edit technical report: A hybrid dataset for instructional image editing. arXiv preprint arXiv:2405.04007, 2024. 4
Li, M., Yang, T., Kuang, H., Wu, J., Wang, Z., Xiao, X., and Chen, C. ControlNet++: Improving conditional controls with efficient consistency feedback. In Proc. Eur. Conf.
Geng, Z., Yang, B., Hang, T., Li, C., Gu, S. Zhang, T., Bao, J., Zhang, Z., Li, H., Hu, H., Chen, D., and Guo, B. InstructDiffusion: A generalist modeling interface for vision tasks. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 17709-12720. 2024. 8,9
Comp. Vis., pp. 129-147, 2024. 27
Li, R. Cheong, L.-F, and Tan, R. T. Heavy rain image restoration: Integrating physics model and conditional adversarial learning. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp.
Gupta, A., Dollar, P., and Girshick, R. LVIS: A dataset for large vocabulary instance segmentation. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 5356-5364, 2019. 4
2019b. 3
Lian, W., Zhang, C., Zhang, H., Jia, B., and Liu, B. Rule- Master+: L.LM-based automated rule generation frame- work for intrusion detection systems. Chinese Journal of Electronics, pp. 1 1-14, 2025. doi: 10.23919Hcje.2024..00 342. 9
Henry, A., Dachapally, P. R., Pawar, S., and Chen, Query- Key normalization for transformers. arXiv preprint arXiv:2010.04245. 2020. 5
Huang, M., Long, Y., Deng, X., Chu, R., Xiong, J., Liang, X., Cheng, H., Lu, Q., and Liu, W. DialogGen: Multi- modal interactive dialogue system for multi-turn text- to-image generation. arXiv preprint 0r8iv:2403.08857, 2024. 9
Lin, W., Wei, X., Zhang, R., Zhuo, I.., Zhao, S., Huang, S., Xie, J., Qiao, Y., Gao, P., and Li, H. Pix Versatile image-to-image visual assistant with open-language in- structions. arXiv preprint arXiv:2409.1522 2024. 1, 8, 9, 27

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Liu, D., Zhao, S., Zhuo, L.,, Lin, W., Qiao, Y., Li, H., and Gao, P. Lumina-mGPT: Illuminate flexible photorealistic text-to-image generation with multimodal generative pre- training. arXiv preprint arXivv2408.026657 2024. 1, 5, 9, 27, 30, 42
Potlapalli, V., Zamir, S. W. Khan, S. H., and Shahbaz Khan, F. PromptiR: Prompting for all-in-one image restoration. In Advances in Neural Inf. Process. Syst., pp. 7 71275- 71293, 2023. 8
Qin, C.., Zhang, S., Yu, N., Feng, Y., Yang, X., Zhou, Y., Wang, H., Niebles, J. C., Xiong, C., Savarese, S., Ermon, S., Fu, Y., and Ran, X. Unicontrol: A unified diffusion model for controllable visual generation in the wild. arXiv preprint arXiv:2305. 2023. 8, 27
Liu, H., Li, C., Wu, Q. and Lec, Y. J. Visual instruction tuning. arXiv preprint (rXiv:2304008483, 2023a. 9
Liu, S., Cheng, H., Liu, H., Zhang. H.,, Li, F., Ren, T., Zou, X., Yang, J., Su, H., Zhu, J., Zhang, L., Gao, J.. and Li, C. Llava-plus: Learning to use tools for creating multimodal agents. arXiv preprint vrXiv:2311.05444
Qwen Team. Qwen2.5: A party of foundation
2023b. 9
models, September 2024.
URL https://
swenim.githee ht/blog///e/nn////oooo 5/. https:
Liu, Z., He, Y., Wang, W., Wang, W., Wang, Y., Chen, S., Zhang, Q., Lai, Z., Yang, Y., Li, Q., Yu, J., Li, K., Chen, Z., Yang. X., Zhu, X., Wang. Y.. Wang, L., Luo, P., Dai, J., and Qiao, Y. InterNGPT: Solving vision-centric tasks by interacting with chatgpt beyond language. arXiv preprinl arXiv:2305.05665 2023c. 9
/ wowenlm.githiin http://oooooooogggeeooeeeeoe 5/, 16
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. In Proc. Int. Conf. Mach. Learn., pp. 8748-8763, 2021. 9
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv: 2017. 5
Ramesh, A., Paviov, M., Goh, G., Gray, S., Voss, C., Rad- ford, A., Chen, M., and Sutskever, I. Zero-shot text-to- image generation. In Proc. Int. Conf. Mach. Learn., pp.
Lu, J., Clark, C., Zellers, R., Mottaghi, R., and Kembhavi, A. Unified-IO: unified model for vision, language, and multi-modal tasks. In Proc. Int. Conf. Learn. Representa- tions, 2022. 8
8821-8831, 2021.
Razavi, A., Van den Oord, A., and Vinyals, O. Generating di- verse high-fidelity images with VQ-VAE-2. In Advances Neural Inf. Process. Syst., pp. 14866-1 14876, 2019. 5
Ma, N., Goldstein, M., Albergo, M. S., Boffi, N. M., Vanden- Eijnden, E., and Xie, S. Sit: Exploring flow and diffusion- based generative models with scalable interpolant trans- formers. arXiv preprint arXiv:2401.0874 2024. 27
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 10684-10695, 2022.
Marr, D. Vision: computational investigation into the hu- man representation and processing of visual information. MIT press, 2010. 2
Sennrich, R. Neural machine translation of rare words with subword units. arXiv preprint arXiv:150..07900 077009 2015. 5
Mokady, R., Hertz, A., Aberman, K., Pritch, Y., and Cohen- Or. D. Null-text inversion for editing real images using guided diffusion models. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 6038 6047, 2023. 27
Shazeer, N. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020. 5
Mou, C., Wang. X., Xie, L., Wu, Y., Zhang, J., Qi, Z., and Shan, Y. T2I-Adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proc. Conf. AAAI, pp. 4304-4304, 2024. 8
Sheynin, S., Polyak, A., Singer, U., Kirstain, Y., Zohar, A.,
Ashual, O., Parikh, D., and Taigman, Y. Emu Edit: Pre- cise image editing via recognition and generation tasks. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 8871- 8879, 2024. 27
OpenAI. Hello GPT-4o, May 2024.
https: /ttp///pppaaaaa/// com/index/hello gpt 40/. 15, 27
Silberman, N., Hoiem, D., Kohli, P., and Fergus, R. Indoor segmentation and support inference from rghd images. In Proc. Eur. Conf. Comp. Vis., pp. 746-760, 2012. 28
OpenGVLab. InternVL family: Closing the gap to commercial multimodal models with open-source suites a pioncering open-source alternative to GPT-40, 2024. URL https://githi/
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
com./openGvi.ab/neteee Interrnn.. 16
12

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Suvorov, R., Logacheva, E., Mashikhin, A., Remizova, A., Ashukha, A., Silvestrov, A., Kong, N., Goka, H., Park, K., and Lempitsky, V. Resolution-robust large mask in- painting with fourier convolutions. In Proc. Winter Conf. Applications of Comp. Vis., pp. 2149-2159, 2022. 8
Wei, X.-S., Xu, Y.-Y,, Zhang, C.-L.,, Xia, G.-S., and Peng, Y.-X. CAT: A coarse-to-fine attention tree for semantic change detection. Vis. Intell., 1:1--12 12, 2023. 9
Wortsman, M., Liu, J., Xiao, L., Everett. K., Alemi, A.. Adlam, B., Co-Reyes, J. D., Gur, I., Kumar, A., Novak, R., Pennington. J., Sohl-dickstein, J., and Xu, K. Small-scale proxies for large-scale transformer training instabilities. arXiv preprint arXiv:2309. 14322, 2023.
Team, C. Chameleon: Mixed-modal early-fusion foundation models, 2024, arXiv preprint arXiv:2405.0900 09818, 2024. 5,
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lam- ple, G. Llama: Open and efficient foundation language models. arXiv preprint vrXiv:23302.13977 2023a. 1,9
Wu, C., Yin, S., Qi, W., Wang, X., Tang, Z., and Duan, N. Visual ChatGPT: Talking, drawing and editing with visual foundation models. arXiv preprint Stdi:2333.046777 2023. 9
Wu, J.. Zhong. M., Xing. S., Lai, Z., Liu, Z., Wang, W., Chen, Z., Zhu, X., Lu, L., Lu, T., Luo, P., Qiao, Y., and Dai, J. VisionLLM v2: An end-to-end generalist multimodal large language model for hundreds of vision- language tasks. arXiv preprint wrXiv:2406.08394, 2024.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, I.., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshom, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V. Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J.. Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog. I. Nic, Y. Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnie, R., Edunov, S., and Scialom, T. LLAMA Open foundation and fine- tuned chat models. arXiv preprint 200.0230,,,,9288, 2023b. 5
9
Xiao, S., Wang, Y., Zhou, J., Yuan, H., Xing, X., Yan, R., Wang. S., Huang, T., and Liu, Z. OmniGen: Unified im- age generation. arXiv preprint arXiv:2409.11344 2024. 1, 2, 27
Xie, J., Mao, W., Bai, Z., Zhang, D. J., Wang. W., Lin, K. Q., Gu, Y.. Chen, Z., Yang. Z., and Shou, M. Z. Show-O: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. 12528, 2024. 9
Xing. P., Wang. H., Sun, Y., Wang. Q. Bai, X., Ai, H., Huang, R., and Li, Z. Csgo: Content-style com- position in text-to-image generation. arXiv preprint arXiv:2408 16766, 2024. 4
Tumanyan, N., Geyer, M., Bagon, S., and Dekel, T. Plug- and-play diffusion features for text-driven image-to- image translation. In Proc. IEFE Conf. Comp. Vis. Patl. Recogn., pp. 1921-1930, 2023. 27
Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L.,, and Liu, T. On layer normalization in the transformer architecture. In Proc. Int. Conf. Mach. Learn pp. 10524-10533 2020.
Valanarasu, J. M. J., Yasarla, R., and Patel, V. M. Tran- sWeather: Transformer- based restoration of images de- graded by adverse weather conditions. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 2353-2363, 2022.
Yang, L.,, Kang, B., Huang, Z., Zhao Z.,, Xu, X., Feng, J., and Zhao, H. Depth anything v2. arXiv preprint trXiv:2406.09411 2024. 8, 19
van den Oord, A., Vinyals, O., and kavukcuoglu, k. Neural discrete representation learning. In Advances in Neural Inf. Process. Syst., pp. 6306 6315, 2017. 5
Yang, W., Tan, R. T., Feng, J., Liu, J., Guo, Z., and Yan, S. Deep joint rain detection and removal from a single image. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 1357-1366, 2017. 28
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Advances in Neural Inf. Process. Syst., pp. 6000-6016 2017.
Yang, W., Wang, W., Huang, H., Wang, S., and Liu, J. Sparse gradient regularized deep retinex network for ro- bust low-light image enhancement. IEEE Trans. Image Process., 0102072-20866 2021. 28
Wang, X., Wang, W. Cao, Y., Shen, C., and Huang, T. Im- ages speak in images: A generalist painter for in-context visual learning. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 6830-6839, 2023. 8
Ye, H. and Xu, D. Inverted pyramid multi-tase transformer for dense scene understanding. In Proc. Eur. Conf. Comp. Vis., pp. 514-530, 2022. 8
13

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., and Huang, S. Free-form image inpainting with gated convolution. In Proc. IEEE Int. Conf. Comp. Vis., pp. 4471-4480, 2019.
Yu, Y., Zeng, Z., Hua, H., Fu, J., and Luo, J. Prompt- Fix: You prompt and fix the photo. arXiv preprint arXiv:2405.16785, 2024. 4
Zhang. B. and Sennrich, Root mean square layer nor- malization. In Advances in Neural Inf. Process. Syst., pp.
12381-12392, 2019. 5
Zhang, F., Xu, Q.. Hu, W., and Ma, F. EASA: Fine-tuning SAM with edge attention and adapters for image manip- ulation localization. Chinese Journal of Electronics, 34: 980-988, 2025,
Zhang. K., Mo, L.., Chen, W., Sun, H., and Su, Y. Mag- icBrush: manually annotated dataset for instruction- guided image editing. In Advances Neural Inf. Process.
Syst. pp. 31428-31449, 2023a. 4, 27
Zhang, L.., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. Proc. IEEE Int. Conf. Comp. Vis., pp. 3836-3847, 2023b. 8
Zhang, S., Yang, X., Feng, Y., Qin, C., Chen, C.-C,, Yu, N., Chen, Z., Wang, H., Savarese, S., Ermon, S., Xiong, C.. and Xu, R. HIVE: Harnessing human feedback for instructional visual editing. In Proc. IEEE Conf. Comp.
Vis. Patt. Recogn., pp. 9026 9026-9036. 2024. 4
Zhao, H., Ma, X., Chen, L., Si, S., Wu, R., An, K., Yu, P., Zhang, M., Li, Q., and Chang, B. UltraEdit: Instruction- based fine-grained image editing scale. arXiv preprint vxXiv:2407.05282 2024. 27
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., and Torralba, A. Places: A 10 million image database for scene recog- nition. IEEE Trans. Pattern Anal. Mach. Intell., 40(6): 1452-1464, 2017a. 28
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., and
Torralba, A. Scene parsing through ADE20K dataset. Proc. IEEE Conf. Comp. Vis. Patl. Recogn., pp. 633-641,
2017b. 4, 16, 28
Zhou, Y., Ren, D., Emerton, N., Lim, S., and Large, T. Image restoration for under-display camera. In Proc. IEEE Conf. Comp. Vis. Patl. Recogn., pp. 9179-9188, 2021. 28
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. M4n:GPT-4: Enhancing vasion-language understanding with advanced large language models. arXiv preprint arXiv:2304.100 10592, 2023. 9
14

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
A. Details for the Dataset of Explanatory CV Tasks
A.1. Describing or Generating Explanatory Instructions
For certain terminological-base vision tasks (e.g., Detection and Segmentation), we selected subset of data and manually crafted descriptions based on the category and other label information. For other data, explanatory instructions were generated by constructing prompts and leveraging GPT-4o (OpenAI, 2024) for automated description generation.
For the manually crafted descriptions, we utilized rule-based combinations to produce over 5,000 unique forms of expression. Specifically, the entire Object Delection data, as well as the majority of data for Segmentation, HED Boundary to Image. Surface Normal Estimation, Depth Estimation, and their inverse tasks, were constructed through this approach.
-writen expressions alongside GPT-4o-generater outputs, applying
For the GPT-4o-generated descriptions, early experiments utilized "3pt-4o-2024-5--333 to construct explanatory instructions. Later, for simpler sections of terminological-busee vision tasks (e.g., HED Boundary Image, Deraining, Dehazing, Desnowing), we employed 0001-44-2024-8--000
For all other tasks, used "chatgpt-4o-lates" generate the instructions. The prompt design is summarized as follows (contents within "[**]" and "(==)" are optional while content in "[**]* y is activated only when human evaluation deems GPT-4o's descriptions inaccurate):
response client. chat cwwplttionsnnoooooooooccooocoo.t creater
model "chatgpt-4o-latee" Aatpeesttrppttt---22000 4gpt-40-2024-55-1 got-4o-2024-08-08 response_forme format ( ("type": "jscn_objectt") messages
"role": "system",
"content": "You are an expert in computer vision and describe vison tasks, with exceptional attention to detail. Your task is to provide detailed descriptions of the transformations between the images, explaining how elements appear, disappear, or change. Your analysis is thorough, accurate, and insightful."),
"role": "user", "content": [
10
11
12
"type": "text",
13
"text": "(We are currently working on tasks related ...*)
Define the first image as A, the second image B. Task (-1) Describe these images.-) 2) Describe 2 scenarios: how to transform image A into image B, image B into image A without referencing the contents of the other image directly. Output format JSON format with keys only contain (=Image Task pass__eecrrjptiioss_frmm..... Fast__ascciipttioo___rooo_LLLLL without any nested JSON structures. The descriptions of transformations should be as diverse as possible, either as a single paragraph or as a step-by-step description. Constraints: 1) The task descriptions should not use any additional tools or references, such as image editing tools. 2) Do not use terms like 'image A', 'image B', 'to transform into *', or similar phrases."
Image_B_Caption,
15
16.
"type": "image_ur1", "image_url" url":
17
18
"url": imdata::mage//peg; base64, (base64 [base64_image_ "detail"; "high"
19
15

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
20
23
"type": "image_url", "image_ur1"; "url": E"data:image/ jpeg; base64, (base64 "aae66__maae__""" "detail"; "high"
24
25
26
27
31
1, max tax_ns 4096
32
33
In addition, we experimented with several other models (e.g., Intern VL2-8B (Chen al., 2024; OpenGVLab. 2024), QWen2.5 (Qwen Team, 2024), Meaude33.5-Someg (Anthropic, 2024), etc.) generate explanatory instructions. However, minimize potential biases in the generated explanatory instructions that could affect validation experiments, we ultimately selected GPT-4o. A comparison of selected models using the first image from the ADE20K Validation Set (Zhou et al., 2017b) is provided below (Table 3 ~ Table 6). Content that does not meet the description requirements is highlighted in red.
Caption of image A:
Traditional stone building with steeply pitched red-tiled roof.
2. Multiple chimmeys and dormer windows.
3. Large white-framed windows.
4. Well-kept lawn in front.
Stone wall in front of the building.
Clear blue sky indicating a sunny day.
Caption of image B:
The same building with specific modifications.
Building and extension shaded with semi-trane transtrrnsprrent reddish huc.
Sky colored a bright blue.
Lawn colored a vivid green.
5. Stone wall highlighted with a grayish color.
Bush the right of the stone wall colored yellow.
Explanatory instruction from A to B:
1. Color the Sky: 1) Select the sky area. 2) Fill the selected area with a bright, distinct color to make the sky stand out. Select the entire building and its extension, including the roof and chimneys. 2) Apply semi- transparent hue over the selected area give shaded appearance while still allowing the underlying details visible. Color Lawn: 1) Select the lawn area in front the building. 2) Fill this area with a bright, vibrant color to emphasize the grassy area.
2. Shade the Building:
4. Highlight the Stone Wall: 1) Select the stone wall in front of the building. 2) Apply distinct, muted color to the wall to make stand from the lawn and building.
Color the Bush: Identify and select the bush shrub the right the stone wall. 2) Fill this area with a bright, distinct color to differentiate it from the surrounding elements.
Explanatory instruction from B to A:
1. Restore the Sky: J)Select the area of the sky that has been filled with bright, distinct color. 2) Replace the bright color with natural sky color, making it appear more realistic.
extension shaded with semi-transparent hue. 2) Remove the shading to restore the building's 's natural appearance, showing its original material and color.
Remove Building Shading: 1) Select
area of building
3. Restore the Lawn Color: 1) Select the area of the lawn that has been filled with a bright, vibrant color. 2) Replace the bright color with natural grass color to make look realistic.
Remove the Highlight from the Stone Wall: 1) Select the stone wall that has been highlighted with distinct, muted color,
2) Remove the highlighting to return the stone wall to its natural appearance.
5. Restore the Bush Color: 1) Select the bush or shrub to the right of the stone wall that has been colored with a bright, distinct color. 2) Replace the bright color with natural color to make it look realistic and blend in with the surrounding environment.
Table 3, Examples of outputs from "gpt-4o-2024-05-12
16

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Caption of image A:
The image shows large, two-story stone house with red-tiled roof. The house has multiple chimneys and windows, some of which dormer windows protruding from the roof. The building surrounded by a rell-maintained green lawn, and there a stone wall in foreground. The sky is clear and blue.
Caption image B:
The image shows large, two-story stone house with a red-tiled roof. The house has multiple chimmeys and windows. some of which are dormer windows protruding from the roof. The building is surrounded by well-maintained green lawn, and there is stone wall in the foreground. The sky is clear and blue, but there noticeable green overlay the right side of the house and lawn.
Explanatory instruction from A B:
Change the sky color to solid bright blue.
2: Add a green overlay to the grass area.
3: Add a green overlay to the ivy on the right side of the building.
4: Add green overlay to the bushes on the left side of the huilding Explanatory instruction from B A:
1: Change the sky color back to natural blue with some cloud details.
2: Remove the green overlay from the grass area.
3: Remove the green overlay from the ivy on the right side of the building.
4: Remove the green overlay from the bushes on the left side of the building.
Table 4, Examples of outputs from VL2-76.
Caption of image A:
This image shows large stone farmhouse manor house in rural setting. The building has multiple stories with dormer windows on roof. features brick chimneys and a mix of stone and brick construction. There's a low stone wall in the foreground, and a well-maintained lawn area. The sky is clear and blue, indicaini sunny day. Some trees or vegetation can be seen in the background.
Caption of image B:
This image appears to be digitally altered version of Image A. The basic structure and layout of the building remain the same, but the colors have been dramatically changed. The sky is a vibrant, unnatural turquoise blue. The grass in the foreground green. There's s a yellow-green patch on the right side of the building that looks like it could be meant to represent ivy or some otherclimbing plant, but in a highly stylized way. The overall effect is surreal and cartoon-like
bright, artificial-looking
Explanatory instruction from A to B:
To transform Image A into something resembling Image B without referencing B content:
1. Increase the color saturation of the entire image dramatically.
Change the sky to bright, unnatural shade of blue.
3. Alter the grass color to vivid, artificial-looking green.
4. Add a bright, contrasting color to one side of the building, perhaps representing vegetation in an exaggerated way.
5. Simplify the textures and details to create more flat, cartoon-like like appearance.
Explanatory instruction from A:
To transform Image B into something resembling Image without referencing content:
1. Reduce the overall color saturation of the image significantly.
2. Adjust the sky color to more natural, pale blue tone.
3. Change the grass
muted, realistic shade of green.
Remove any unnaturally bright or contrasting colors from the building surroundings.
5. Enhance the texture and details of the building to create more realistic, photographic appearance.
Table 5. Examples of outputs from "claude-3-5-soomee-22222000
17

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Caption of image A:
The image captures a serene scene of a traditional English country house, nestled amidst lush green lawn. The house, painted in pristine white, stands against the clear blue sky. It features a prominent red brick chimney, adding touch rustic charm The house is adorned with multiple windows, each framed by white shutters, allowing ample natural light to filter into the rooms within. A large wooden door, painted in contrasting dark brown color, serves as the main entrance the house.
appearance.
stone wall encircles the property, providing privacy and security. The wall punctuated by a gate, inviting visitors into the tranquil haven beyond. The house is surrounded by an expanse of well-maintained grass, creating sense of openness and tranquility.
Despite the detailed description, there's no text visible in the image, keeping the focus solely on the visual elements. The relative positions of these objects suggest well-planned layout, with the house at the center, the stone wall forming boundary around it, and the gate marking the entry point. Overall, the image presents a picturesque view of a typical English countryside setting, where nature and architecture coexist harmoniously.
Caption of image B:
This image showcases a traditional English country house, set against a backdrop a clear blue sky. The house, construceed from red brick, boasts a prominent chimney and a large wooden door. The roof, coveredin tiles, adds a touch of rustic charm to the overall structure. The house is situated provides a boundary but also enhances the aesthetic appeal the property. A gate can be seen in the wall, possibly serving as the entrance to the property.
lush green lawn, which is enclosed by a wall. This wall not only
In terms of object count, there's one house, one chimney, one door, and one gate. The relative position of these objects suggests well-planned layout - the house is centrally located, the chimney is attached to the house, the door is part of the house, and the gate is integrated into the stone wall surrounding the property.
There's 's no text present in the image, keeping the focus solely on the visual elements. The image does not provide any information about the actions or movements of the objects, as they are stationary structures. Overall, the image presents a picturesque view of a typical English countryside setting, where nature and architecture coexist harmoniously.
Explanatory instruction from A to B:
Adjust Color Scheme: Begin by altering the color palette of the main building. You could introduce warmer hues to the exterior, perhaps changing the white walls to soft cream adding accents of natural stone colors to give it different character.
Modify Architectural Features:Modify architectural elements such as chimney and windows. For instance, you might extend the chimney height, add decorative brickwork, or replace the window shutters with functional window boxes filled with blooming flowers.
Landscape Transformation: Redesign the surroundings by introducing garden path leading up the house or adding small pond the entrance. Change the lawn incorporate more diverse flora, like flower beds shrubheries. Add or Remove Structures: Introduce detached garage or garden gazebo in the background, or conversely, remove existing elements like the stone wall to open up the view and create a more expansive landscape.
Lighting and Atmospheric Effects: Adjust the lighting to simulate different time of day, such as sunset or twilight. and add atmospheric effects like mist rain to alter the mood of the scene. Explanatory instruction from
A:
Reverse Color Alterations: Restore the original color palette of the building, possibly reverting any warm tones added the previous transformation back to cooler shades, like the classic white or a more subtle hue that contrasts with the greenery. Revert Architectural Modifications: Remove any additional architectural features introduced, such as decorative chimney work or new window features, restoring the building to its simpler, more traditional form.
Revive Original Landscape: Return the landscaping structures like gazebos that were added. Reinstate the uniform green lawn and, if removed, rebuild the enclosing stone wall, Alter Atmospheric Conditions: Adjust the lighting and weather conditions to their prior state, perhaps brightening the scene reflect midday sunlight or clearing away any atmospheric effects like fog or rain.
initial simplicity, eradicating any new paths, ponds, or additional
Restore Original Scene Balance: Ensure changes collectively bring the scene back to state reminiscent the original, while not precisely replicating Image A, focusing on balancing the composition and elements to achieve similar ambiance.
Table 6. Examples of outputs from Qwen2.5.
18

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
A.2. Generating Data Samples
For subset of terminological-basse vision tasks, we employed existing methods to generate image pairs.
HED Boundary Detection & HED Boundary to Image. We use HED boundary detection to generate approximately 10k edge-image pairs from randomly selected images in other datasets.
Image Depth Estimation & Depth Map to Image. We used the Depth Anything V2 model (Yang et al., 2024) and obtain about 250k depth-image pairs. Original images are randomly selected from other datasets.
Surface Normal Estimation & Normal Map to Image. We generated the surface normal maps by using DSINE (Bae & Davison, 2024). We obtain around 250k normal-image pairs. Original images are randomly selected from other datasets.
A.3. More Data Samples for the Dataset of Explanatory CV Tasks
In this section, we provide more data samples for the Dataset of Explanatory CV Tasks.
A.3.1. MORE DATA SAMPLES FOR EXPLANATORY-BASED VISION TASKS
More data samples for "Explanatory- -base Vision Tasks" are provided in Fig. 9 ~ 16.
Explanatory instruction from A to B:
The dress changes from having long sleeves with ribbons to having off-the-shoulde sleeves. The color of the dress shifts from deep purple to lighter shade with satin finish. The hairstyle transitions from braided and tied back to hair let down with a tiara. Gloved hands shift to being ungloved with a visible change in the hand accessory. Socks become part of the footwear, altering from lace-up shoes to white socks. The expression shifts from side-facing a front-facing stance with visible smile.
Explanatory instruction from B to A:
Adjust the dress design from off-the-shoulder with satin to one featuring long sleeves with ribbon details. Modify the dress color from a light, shiny appearance to deep purple tone with more textured fabric. Change the hairstyle from loose hair with a tiara a braided style pulled back. Introduce gloves the hands, replacing the bare hand appearance. Transition the socks dark shoes, altering the footwear style. Shift the facial expression from smiling, forward-facing look to neutral, side-facing demeanor.
Figure 9, Data sample for explanatory based vision tasks.
Explanatory instruction from A to B:
Shake up the perspective to bring the animal slightly closer, reposition upright with a slight turn of the head. Reduction of debris gives the ground a smoother, even appearance.
Explanatory instruction from B to A:
Adjust the animal's posture to upright sitting position, shifting the angle reveal more surroundings. Enhance the scene with additional greenery and scattered natural elements.
Figure 10. sample explanatory- vision tasks.
19

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Explanatory instruction from A to B:
Reposition the vase that it is more upright, causing the flowers to shift slightly to the left. Adjust the angle of the sunlight so that shadows fall more prominently to the bottom right, emphasizing a more elongated shadow the floral arrangement. Rotate the floral elements along their central axis, making each petal and leaf face slightly different directions while maintaining their original colors and textures.
Explanatory instruction from B to A:
Tilt the vase towards the viewer, altering the flowers' positions to the right. Modify the lighting to change the direction of the shadows, creating a more compact and less defined shadow below the arrangement. Slightly adjust the orientation of the petals and leaves, ensuring they align in a different configuration yet retain their original forms and hues.
Figure 11. Data sample for explamatory- hased vision tasks.
Explanatory instruction from A to B:
Brighten the overall scene, adding more sunlit ambiance. Modify the gripping object to reflect natural, green element while altering the environment more elevated, branch-like setting.
Explanatory instruction from B to A:
Soften the vivid warmth by introducing cooler tones. Exchange the greenery for a different object hold, and replace the leafy background with consistent, textured surface.
Figure 12. Data sample for explanatory- based vision tasks.
Explanatory instruction from A to B:
Replace the white square base with a large wooden board. Display the spheres in a less strict arrangement. Incorporate garlic bulbs and scattering of spices in the background, complemented by soft cloth textures.
Explanatory instruction from B to A:
Shift the spheres onto a neat, white square template. Simplify the herb decoration to single leaf per sphere and clean up the overall setting for a minimalist and fresh appearance.
Figure 13. Data sample for explanatory- vision tasks.
20

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Explanatory instruction from A to B:
Add crowd at the bottom with open hands raised towards the sky, and increase the number of lanterns, making them appear smaller as they move upwards. Darken the background slightly and include a distant bright light at the horizon.
Explanatory instruction from B to A:
Remove the crowd and their raised hands from the bottom, leaving only the lanterns. Brighten individual lanterns and make the scene more clustered with fewer visible lanterns.
Figure 14. Data sample explanatory based vision tasks.
Explanatory instruction from A to B:
Introduce background featuring a variety green leaves and stems, while adding another flower similar to the first one. Ensure the new flower has a more fully opened appearance and a slightly different arrangement of petals.
Explanatory instruction from B to A:
Focus single flower, removing the secondary bloom and simplifying the background to emphasize the central floral subject. Strip away diverse leaves and stems, leaving plain backdrop to highlight the singular floral element.
Figure 15. Data sample for explanatory-based vision tasks.
Explanatory instruction from A B:
Adjust the color scheme from blue-green light spectrum combination of red and green, creating shift in the color dynamics. Alter the central projection, ensuring that the beam patterns reflect less lincarity and more concentric circular formations on the surface. Intensify the red hues across the visible light paths and enhance the circular designs with stronger emphasis on the differences color, resulting in more elaborate geometric patterns.
Explanatory instruction from B to A:
Change the predominant color from red-green blue-green spectrum, modifying the overall illumination convey cooler tones. Modify the light projections the surface, transitioning from circular formations directed from the center point outward. Reduce the complexity of the geometric patterns, focusing on the linear symmetry, and diminish the red hues, emphasizing blues with a vibrant green accent.
design where the beams more linear and
Figure 16. Data sample explanatory- based vision tasks.
21

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
A.3.2. MORE DATA SAMPLES FOR TERMINOLOGIIAL--DASED VISION TASKS
More data samples for "Terminological-basse Vision Tasks" are provided in Fig. 17 31.
Explanatory instruction from A to B:
Enhance the overall saturation and vibrancy of the image by increasing color intensity, bringing out vivid blue sky and deepening the reds in the earth tones. Adjust the contrast to sharpen the details of the rocky formations.
Explanatory instruction from B to A:
Reduce the saturation and brightness to subdue the colors, reverting the image to a more natural state. Soften the contrast to give the image more balanced and moderate appear- ance.
Figure 17. Data sample for terminological-basee -ased vision tasks (Image Restoration).
Explanatory instruction from A to B:
Enhance color saturation while maintaining the current brightness, Emphasize cooler tones for a more vibrant and lively appearance.
Explanatory instruction from B to A: Reduce color saturation to achieve a more neutral tone. Neutralize cooler tones to recreate the initial natural look.
Figure 18. Data sample for terminological-base vision tasks (Image Restoration).
Explanatory instruction from A to B:
Drain the image of all bright colors, shifting the mood into sepia-like monochrome. The dragon figure remains, but now weathered and eroded, appearing as if composed of twisted, dried-out textures. The background morphs dramatically as the lush flora converts into barren, tangled form entirely overtaken by shadows and creeping vines. The resulting atmosphere becomes dark and decayed, contrasting the lively, colorful origin.
Explanatory instruction from B to A:
Reimagine the desolate, worn textures by replacing them with lively, bright tones. Breathe life and energy back into the figure by restoring facial features, sharpening its horns, and returning its surroundings distinctly separated flora. Soften the twisted appearance of the dragon, allowing its scales to return to more uniform, sleck form, where bold colors and vivid contrasts dominate the atmosphere.
thriving, colorful forest, full of sharp contrasts and
Figure 19. Data sample for terminological-aasee vision tasks (Style Transfer).
22

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Explanatory instruction from
Maintain the layout but adjust the overall structure by shifting the visual style more artistic, painted form. The material should evolve into representation with stronger outlines and exaggerated contrasts between shadows and highlights. All elements transition to bold blocks of color and simplified features, giving the appearance of something more abstract, yet still retaining the core elements of the original composition.
Explanatory instruction from B to A:
Soften the rough, exaggerated lines and deep hues present in the scene. Gradually smooth out all blocks of visual complexity forming natural color transitions between shadows and bright areas. Pull the bright elements into more neutral, natural tone, eliminating hard contrasts. Return the visuals lifelike depiction, but keep the core layout intact.
Figure 20. Data sample for terminological-haste vision tasks (Style Transfer).
Explanatory instruction from A B:
For the following object categories, apply the corresponding solid color overlays fully them:
Color every wineglass object with aqua solid layer.
Paint each fork with linen solid color fill.
Color each banana object with darkolivegreen solid overlay.
Paint each spectacles with a purple solid color fill.
Apply solid lawngreen color overlay to fully cover searf objects.
Explanatory instruction from B to A:
Delete the solid color covering the objects, restoring their original look.
Figure 21. Data sample for terminological-basce vision tasks (Segmentation).
Explanatory instruction from A to B:
Paint over every clock, streetlight object with dodgerblue, covering them entirely.
Explanatory instruction from B to A: Erase the dodgerblue overlay on the clock, streetlight objects, restoring the original appearance.
Figure 22, Data sample for terminological-honn based vision tasks (Segmentation).
23

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Explanatory instruction from
Draw floralwhite bounding box around all apron objects.
Explanatory instruction from B to A: Remove all bounding boxes from the objects, restoring the original image.
Figure 23. Data sample for terminological-basee vision tasks (Object Detection).
Explanatory instruction from A B:
Remove the raindrop patterns that obscure sections of the scene to reveal clear, unob- structed view. Enhance the visibility of details by eliminating blurred spots. Increase the sharpness and contrast to accentuate the overall clarity, ensuring all objects and surfaces are distinctly visible.
Explanatory instruction from B to A:
Introduce raindrop patterns on the lens to create areas of distortion and blur across the scene, simulating a rainy effect. Soften certain details by incorporating these smudged patterns, which partially obscure parts of the view. Decrease the contrast and sharpness to mimic the presence of rain on the camera lens.
Figure 24, Data sample for terminological-basee vision tasks (Deraining).
Explanatory instruction from Introduce a layer of fog that softly envelops the scenery, generating slightly diffused appearance with reduced contrast. Increase the density of mist, particularly around the lower parts of the tree and the ground, to create moody and mysterious ambiance. Soften the outlines of distant elements and elements in the foreground, diminishing the clarity to simulate a hazy atmosphere. This results in a muted color palette, lending an ethereal quality to the overall visual experience.
Explanatory instruction from B to A:
Reduce the visibility of mist around the scene by gradually decreasing the density of fog, improving the clarity and sharpness of the landscape. Carefully remove any diffused light scattering, allowing the intricate details of the surrounding environment to emerge more distinctly. Ensure that the distant elements become more defined and that the colors appear more saturated and less muted, providing unobstructed view of the complete scene.
Figure 25. Data sample for terminological-basee vision tasks (Dehazing).
24

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Explanatory instruction from A
Reduce the visibility of falling particles obscuring the background by removing white speckles scattered across the scene. Clarify the view by eliminating opaque overlays, focusing enhancing the details of the distant objects and sky. Enhance the overall brightness and contrast to highlight the serene atmosphere with unobstructed sightlines.
Explanatory instruction from B to A:
Introduce layer of translucent white specks distributed unevenly across the scene simu- late snowy condition. Gradually decrease the clarity of distant elements by overlaying semi-transparent textures while maintaining the overall composition. Weave sense of dynamic motion through the addition of irregular shapes and lighter shades throughout the field, mimicking a flurry of snowfall.
Figure Data sample for terminological-be -assd vision tasks (Desnowing).
Explanatory instruction from A to B:
Simplify the scene into grayscale silhouette, emphasizing the outlines and contours of prominent subjects while eliminating detailed features and textures. Fade out background patterns into smooth gradient to suggest depth. Focus on transitioning the entire scene into varying shades of gray, concentrating on the key shapes and forms.
Explanatory instruction from B to A:
Introduce complex patterns and vibrant colors to the scene, adding texture and intricate details to key subjects. Enhance foreground subject clarity, incorporating color variations and shading. Overlay the background with rich details and vivid patterns to create a more detailed and nuanced appearance, emphasizing depth with contrasting hues.
Figure 27. Data sample for terminological based vision tasks (Depth Estimation).
Explanatory instruction from A to B:
Introduce a series of vividly colored dots and lines across the canine figure. Each line should connect to form an intricate pattern that traces the outline and features of the animal, adding a whimsical and abstract design element.
Explanatory instruction from B to A:
Remove all the colored dots and lines overlaying the animal. Preserve the misty back- ground along with the tree and the canine, ensuring the environment exudes quietude and mysticism.
Figure 28. Data sample for terminological based vision tasks (Pose Estimation).
25

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Explanatory instruction from
Focus solely on the anatomical framework by removing all environmental and bodily details, leaving only colorful joint lines dark canvas abstractly outline movement.
Explanatory instruction from B to A:
Reintroduce the snowy environment, captured with deer in motion, and omit the line structure display natural scene of winter and untouched wildlife activity.
Figure 29. Data sample for terminological- based vision tasks (Pose Estimation).
Explanatory instruction from A B:
Apply transformation that generalizes the depiction of depth and orientation using colorful gradient representing angles across the surface. Structures become abstract with smooth transitions between colors, translating the detailed textural elements into flat gradients indicating spatial variations. The skyline should shift into simplified forms with no explicit textures, instead presenting undulating colors to denote geometric relationships. Emphasize prominent architectural features by focusing on their orientation, using color to suggest their form without explicit details. The luminance of the scene should be replaced with a vibrant, multi-hued representation that suggests the directionality and depth of each element.
Explanatory instruction from B to A:
Introduce detailed textural elements and a realistic color scheme over the abstract tonal surfaces, replacing the gradient with distinct textures that convey material properties and illumination. Reconstruct the background to include various light sources and urban details, accentuating the contrast and natural colors found on physical architectures. Re-establish the shadows and highlights to convey depth using realistic lighting that defines the shapes explicitly and clearly. Transition the simplified forms into detailed and distinct buildings, featuring precise windows, lighting, and signs that illustrate an urban setting at night, capturing the variations in illumination and environment.
Figure 30. Data sample for terminological-basee based vision tasks (Surface Normal Estimation).
Explanatory instruction from A to B:
Highlight only major edges, transforming the image into a boundary map.
Explanatory instruction from B to A:
Convert boundary outlines into a realistic image, applying textures and colors.
Figure 31. Data sample for terminological- based vision tasks (HED Boundaries).
26

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
B. Results, Details and Samples for Quantitative Experiments
B.1. Quantitative Experiments
Training settings follow Sec. 1 the paper.
Table 7. Comparisons with image-editing baselines evaluated on the Emu Edit and MagicBrush test set. report the corresponding results.
indicates that the method does
Emu Edit test set
MagicBrush test set
Methods
CLIP sm f
CLIP out"
4
DINOt
CLIP im f
CLIP out
DINO
InstructPix2Pix (2023)
0.834
0.219
0.121
0.762
0.837
0.245
0.093
0.767
MagicBrush (2023a)
0.838
0.222
0.100
0.776
0.883
0.261
0.058
0,871
PnP (2023)
0.521
0.089
0.304
0.153
0.568
0.101
0.289
0,220
Null-Text Inv. (2023)
0.761
0,236
0.075
0.678
0.752
0.263
0.077
0.664
Emu Edit (2024)
0.859
0,231
0.094
0.819
0.897
0.261
0.052
0.879
UltraEdit (2024)
0.844
0.283
0.071
0.793
0.868
0.088
0.792
OmniGen (2024)
0.836
0.233
0.804
PixWizard (2024)
0.845
0.248
0.069
0.798
0.884
0.265
0.063
0.876
Lumina-mGPT (2024)
0.815
0.283
0.149
0.735
0.855
0.282
0.114
0.786
Ours
0.821
0.286
0.132
0.768
0.875
0.292
0.093
0.831
B.I.I. IMAGE EDITING RESULTS
We evaluate the fine-tuned AR-based VLM on two image editing benchmarks, i.e., the MagicBrush test set (Zhang et al., 2023a) and the Emu Edit test set (Sheynin al., 2024), which include several human-defined operations, e-g., background alteration, object removal, object addition, localized modifications, etc. In alignment with the evaluation metrics used both by Emu Edit and MagicBrush, we measure four metrics: 1) CLIP CLIP image similarity between source image and output image; 2) CLIP o CLIP text-image similarity between edited image and target caption; 3) & l distance between source image and output image; 4) DINO: DINO similarity between source image and output image.
As shown in Table 7, compared to instruction-guided image editing methods such as InstructPix2Pix (Brooks et al., 2023), PnP (Tumanyan et al., 2023), and Null-Text Inv. (Mooady et al., 2023), the straightforward fine-tuned AR-based VLM demonstrates certain advantages. However, when compared to more advanced methods such Emu Edit(Sheynin et al., 2024), UltraEdit (Zhao et al., 2024), and recent VLMs employing flow-based Diffusion Transformers (DiT) (Ma et al., 2024) backbones (e.g., OmniGen (Xiao et al., 2024) and Pix Wizard (Lin et al., 2024)), the performance of model still shows noticeable gap. Nevertheless, compared to vanilla token-based VLM (i.e., Lumina-mGPT (Liu et al., 2024)), the fine-tuned model achieves significant performance improvements (Chameleon is excluded due to the lack of image generation capability).
1.1.2. IMAGE GENERATION RESULTS
We then evaluate the effectiveness of generation capabilities. We assess its performance across three tasks: controllable image generation (including Canny-to-Image and HED-to-Image), Inpainting, and Outpainting. Results are shown Table I in the main paper.
Controllable Image Generation We adopt the evaluation split of MultiGen- 20M for Canny-to-Image and HED-to-Image condition. As MultiGen-200 (Qin et al., 2023) do not provide image captions in general nature language format and the dataset is not used during training, we use GPT-4o (OpenAI, 2024) recaption them. We also provide unseen explanatory instructions for each controllable generation data pair (the construction method for explanatory instructions refer to Appendix A. to evaluate our model's 's instruction-level zero-shot capability. Following ControlNet++ (Li et al., 2024), we assess controllability by measuring the similarity between the input conditions and the extracted conditions from the generated images. We use F1-Score for Canny-to-Image and SSIM for HED-to-lmage. We adopt FID and CLIP-Score to evaluate the quality the generated images and their alignment with caption. The resolution of generated images 448 448 but are resized to 512 512 for evaluation (output resolution for Lumina-mGPT 512 512). Results
27

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
are shown in Table 1. As both Canny Edge Detection and Canny-to-lmage tasks are not seen during training, results for Canny-to-lmage represent the more challenging task-level zero-shot setting. Instructions for Lumina-mGPT used fixed format (Lumina-mGPT does not work if change the instruction format, details can refer to Appendix B.2). However, evaluate instructions for ours are unseen during training. While the results still fall short of those achieved by task-specifie or so-called vision generalist models, it demonstrates significant progress compared to previous models, which were largely incapable of zero-shot generalization at either the instruction level or the task level.
Inpainting and Outpainting For image inpainting, we use random rectangle or circle to mask 40%-50% of the image area and measure FID and LPIPS assess the quality of the generated images. For image extrapolation (outpainting). we follow MaskGIT (Chang et al., 2022) settings, i.e., extending the image by 50% to the right and use FID and Inception Score (IS) to assess the quality of the generated images. We do not use image captions that directly describe the output images but construct unseen explanatory instructions during inference stage for our model (details can refer to Appendix A. Both the inpainting task and the outpainting task are not seen during training (but inpainting may have task-level overlap with Segmentation-to -omagnee)))... We evaluated on 10,000 image crops from the Places dataset (Zhou et al., 2017a) (training set for the Places dataset are not used during our training stage). The size of generated images is 448 448 but we resize images to 512 x 512 for evaluation (output resolution for Aumina-mGPT is 512 x 512). As shown in Table 1, although the results of our generated images still show a gap compared other task-specific or vision generalist models, we represent significant breakthrough in zero-shot capability.
B. 1.3. RESULTS FOR DENSE IMAGE PREDICTION AND LOW-LEVEL. TASKS
We provide more details for dense image prediction tasks and low-level tasks in this section. Results are shown in Table 2 in the main paper.
Dense Image Prediction We conduct Semantic Segmentation experiments on the ADE20K validation set (Zhou et al., 2017b), while Depth estimation and Surface Normal Estimation experiments on the NYU-Depth V2 dataset (Silherman et al., 2012). For Semantic Segmentation, since we use color to control category generation and managing all categories simultaneously leads to higher color inaccuracies, we test each category individually during the evaluation stage. This approach yields results higher than conventional evaluation methods, and the result is provided as reference only. Accuracy is evaluated using the Mean Intersection over Union (mloU) metric. For the monocular Depth Estimation task, we adjust the depth values of the generated image to fall within the range of zero to ten meters. Accuracy is measured using the Root Mean Square Error (RMSE). For the Surface Normal Estimation task, we recover the corresponding normal vectors from the output image and assess accuracy using the Mean Angle Error metric. For both Depth Estimation and Surface Normal Estimation tasks, we employ unseen explanatory instructions during evaluation to guide the model toward the target objective.
Low-level Tasks We conduct Deraining experiments on the Rain 100L (Yang et al., 2017) dataset and Dehazing experiments on the SOTS (Li al., 2018) dataset, both using unseen explanatory instructions during the evaluation stage. Subsequently, we perform task-level zero-shot evaluations on under-display camera (UDC) Image Restoration task using the UDC (T- OLED) dataset (Zhou et al., 2021) and on Low-light Enhancement task using the LOLv2 dataset (Yang et al., 2021). Similarly, the explanatory instructions used during evaluation are unseen during the training phase. We evaluate performance using PSNR and SSIM as distortion metrics.
Although the results of the generated images still show a gap compared to task specific or existing vision generalist models, our model successfully understand the task objectives conveyed by the unseen explanatory instructions and effectively complete the corresponding tasks. Unseen explanatory instruction samples for these tasks can be found in Section B.2 (cf. Fig. 33 ~ 43). While examples of the generated images for these tasks can be found in Section C.
1.1.4. MORE MERITS OF EXPLANATORY INSTRUCTIONS
For vision tasks have already been included in the training set, explanatory instructions can also contribute to the understand- ing of zero-shot samples. As shown in Fig. 32, for categories present in the training set (e.g., turtle), both direct labels like "turtle" and descriptive phrases such as "the creature swimming in the water" effectively guide the model in completing tasks. However, for categories not included in the training set (e.g., broad-winged damselfly), the model struggles to interpret the category name alone but can be assisted by descriptive expressions like "the creature on the leaf"
28

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Explanatory Instruction: "Apply red ca overlay to the turtle."
Output Image
Explanatory Instruction: "Apply red color overlay to the creature sw Liminng in the water in the image."
Input Image
Output Image
Explanatory Instruction: damsel Flses."
"Apply red color overlay to Broad-winged
Output Image
Explanatory Instruction: "Apply red overlay to the creature on the leaf
Input Image
in the image."
Output Image
Explanatory Instruction:" Change the color of aegopodius um podagraria in the picture to purple."
Output Image
Explanatory Instruction: Change the color of the white flower-like plants in the picture purple."
Input Image
Output Image
Explanatory Instruction: Cover elaeagrus unbellata with blue."
Output Image
Explanatory Instruction: Cover the red fruit in the picture with blue."
Input Image
Output Image
Figure 32, Explanatory instructions assist model in understanding task objectives. (Segmentationn.
29

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
B.2. Details and Unseen Instruction Samples for Quantitative Experiments
In this section, we provide a more detailed explanation for quantitative experiments. As described in Sec. B.1.2, do not rely image captions but construct entirely unseen instructions during the evaluation stage. In contrast, other vision generalist models rely instructions during training and incorporate image captions in their evaluation process. For clarification, we use Lumina-mGPT (Liu et al., 2024) an example to illustrate this difference.
In the evaluation process for Lumina-mGPT, as shown in Table all instructions follow this format: "Generate an image according to the provided image, and according to the following caption: {Image (optiinn)<limagclh For the experiments in Appendix B.I, the instructions are "Depth estimation. <limagel>" for the Depth Estimation task, "Semantic segmentation. <limagel>" for the Semantic Segmentation task and "Surface normal estimation. <limagel>" for the Surface Normal Estimation task. Any alteration to the format of these instructions leads to model failure significantly degraded its performance.
In the following, we directly illustrate the variety of unseen instructions we constructed for each evaluation sample through examples from quantitative experiments (cf. Fig. 33 ~ 43).
Unseen Explanatory Instruction: "The scene shi fts. from wet and rainy ronment calm and dry one by gradually reducing the intensity of the rain until it stops complecely."
Input Image
Ground Truth
Unseen Explanatory Instruction: "Tnagine the sky clearing up, the rain stopping entirely, all moi moisture being removed from scene, leaving the surface dry and the sky clear."
Input Image
Ground Trruth
Unseen Explanatory Instruction: "To reach scene where the weather turns fair, imagine the clearing the ceasing, bringing out the sun. The ground dries up quickly as the rain evaporates, giving way to a brighter ambiance."
Input Image
Ground Truth
Figure 33. Instruction-less zero-shot samples for quantitative experiments (Deraining).
Unseen Explanatory Instruction: "The vibrancy intensifies, revealing spectrun of colors that emphasize the angles and orientations of each surface in the scene."
Input Image
Ground Truth
Unseen Explanatory Instruction: "Tranalate the visible atructures into range of bright colors reflecting orientation angles, enhancing variations surfaces."
Input Image
Ground Truth
Unseen Explanatory Instruction: "Transform the visible elements into a vibrant array where distinct colors signify the orientation of surfaces."
Input Image
Ground Truth
Figure 34, Instruction-le level zero-shot samples for quantitative experiments (Surface Normal Estimation).
30

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Unseen Explanatory Instruction: "Reduce the color details and enhance the gradient of Light to dazk, emphasizing depth transitions, Apply a uni form grayscale filter across the entire scene, melding textures and patterns into smooth surfaces. Simplify distinct forms, softening and uni fying borders accentuate spatial di fifeenncce.."
Input Image
Ground Truuh
Unseen Explanatory Instruction: "Introduce gradient effect to the space, ensuring that the light textures and colors to various shades gray, emphasizing the depth and contours of objects. Pocus on the relat positioning and shapes, creating i nq a smooth transition from light to dark areas, hight lighing the spat al arrangenent.""
becomes the cominan: feature. Simplify
Input Image
Ground Truth
Unseen Explanatory Instruction: "Convert al1 color data irto grayscale, ad instting the brightness based on perceived depth, with el closer objects appearing darker and farther ones lighter. Gradually diminish surface textures and fine detaila, focusing solely relative positions of objests. Preserve the edgee and outlines to enhance depth perception, emphasizing structural differences. Amplify the contrast betweer. closer and more distant objects to create clear spatial arrangement."
the contours and
Input Image
Ground Truth
Figure 35. Instmuction-level zero-shot samples for quantitative experiments (Depth Estimation).
Unseen Explanatory Instruction: "Enhance the clarity by zeducing the haze, adjusting cantrast and brightness levels for a sharper and more detailed view."
Input Image
Ground Truth
Unseen Explanatory Instruction: "Increase the vibrancy and contrast while reducing atmospheric haze to reveal details, making sky and structures more defined."
Input Image
Ground Truth
Unseen Explanatory Instruction: "To enhance clarity and vibrancy, imaqine each object gaining sharpness and colors beconing nore pronounced. Gradually zemove the grayish tone, revealing the street with increased contzast and brightness."
Input Image
Ground Truth
Figure 36. Instruction-levee zero-shot samples for quantitative experiments (Dehazing).
31

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Unseen Explanatory Instruction: "Simplify the intricate details, focusing on large areas of color. Reduce the textures the walls, roof, and garden to single, solid colors, ensuring clear separation between different elements."
Input Image
Ground Truth
Unseen Explanatory Instruction: "simplify the scene by Ldentifying distinet regions such the sky, structures, and foliage, assigning each distinct color."
Input Image
Ground Truth
Unseen Explanatory Instruction: "Begin by abstracting detailed features into flat, vivid colors, transforming complex textures inte simple hues. Convert the varied Landscape elements into salid bands of color, maintaining only their fundamental positions and shapes. The sky turns uniform light blue, and the plane becomes defined mint green silbouette. The mower simplified into blue shape, with the seat becoming highlight in red."
Input Image
Ground Truth
Figure 37. Instruction-level zero-shot samples for quantitative experiments (Semantic Segmentation).
Unseen Explanatory Instruction: "Adding rich layers of color and texture return the original setting's
to the outlines would allow vibrancy. Explosive growth of texture from flat lines and edges would populate the space with marine 1ife and details, enhancing the divers' surroandings and restering their environment. Subtle gradlents and vivid aquatic elements, like fish and the deep blue hues, re-emerge, filling the outlines with life."
Input Image
Ground Truth
Unseen Explanatory Instruction: "start by filling the abstract lines with rich details, deepening the creases of the outline with fur textures end shadowed areas. Build up the dimensionality by adding layers of calor, texture, and shading Saften the stark contrasts while adding reflective light in the eyes and the nuanced light scattering of fur. Bring back the ambient background details, incorporating stars and subtle sky gradients to restore fullness to the scene."
reconstruct the depth in every feature.
details, such
Input Image
Ground Truth
Unseen Explanatory Instruction: "Refill the outlines with authentic textures, colors, and shading. Introduce the interplay of natural lighting, allowing shadows and highlights to emerge once again over every visible surface, Rebuild the snowy forest backdrop, enhancing the dimensionality
the nountain scene with vibrant snox-covered el elements under warm, diffuse sunlight. The cabin should regain the illasion. of physicality, reflecting the interaction between the stracture and the surrounding environment including intricate snow details the roof and ground."
Input Image
Ground Truth
Figure 38. Instruction-level zero-shot samples for quantitative experiments (HED-to-Image).
32

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Unseen Explanatory Instruction: "Add colors, lighting, and fur texture gradually to 1ime-based outline, introducing gradients that replicate the realism of fur. Ensure soft lighting is integrated into the scene with attention to shading and depth on the dog's s face and body."
Input Image
Ground Truth
Unseen Explanatory Instruction: "Rebuild the scene by filling all edges with rich color, shading, and texture, Reconstruct the orange clouds in the sky and provide depth to the environment by reintroducing lighting, perspective, and atmospherie effects, along with adding detail individual objects like the child and bicycle."
Input Image
Ground Truth
Unseen Explanatory Instruction: "starting from abstract outlines, vibrant colors would need to emerge, #illing in the spaces between the lines with rich textures, colors, and delicate lighting. Details like the lushness surroundings and the intricate clothing details would have to grow back into the scene, building sense of volume, dimensionality, and narrative where before there was only flat abstraction."
Input Image
Ground Truth
Figure 39, Task level & instruction-lewe zero-shot samples for quantitative experiments (Canny-to-lmage),
Unseen Explanatory Instruction: "To improve visibility and clarity, increase the brightress uni unrmorll across the entire area. Next, adjust the contrast to differentiate objects and their surroundings, ensuring the whitea and highlights stand color correction to restore the vibrancy of dull tones, balancing any tints to make them appear natural, Enhance sharpness to recover fine details in textures, like the targets, arrows, and floor. Smooth out any noise that might emerge from the exposure increase, ensuring clean and crisp finish while retaining natural lighting properties."
without losing shadows. Apply
Input Image
Ground Truth
Unseen Explanatory Instruction: "Begin by enhancing the overall brightness of the scene to bring out hidden details. Gradually increase the cantrast di ffeeenttiate the elements correction to amplify the muted tones, ensuring that the hues are vivid and lifelike. Finally, atilize a sharpening techrique to accentuate the edges and textures, providing more defined appearance."
clearly. Apply calor
Input Image
Ground Truth
Unseen Explanatory Instruction: "Begin adjusting the overall brightness the scene to allow more light to penetrate the darker areas. Gradually increase the contrast to ensure that the highlights pop while maintaining some depth in the shadows. Employ techniques to amplify cclor saturation, bringing out the richness of hues that were previously subdued. Finally, apply sharpening filters to enhance the clarity fine details that were not discernible in the original."
Input Image
Ground Truth
Figure 40. Task level & instruction-level zero-shot samples for quantitative experiments (Low-light Enhancement).
33

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Unseen Explanatory Instruction: "Remove the sircular obstruction reveal the previously hidden water and foliage, restoring the full view of the landacape."
Input Image
Ground Truth
Unseen Explanatory Instruction: "Reveal the hidden elements by removing the black circle, restoring the visibility of the entire kitchen setup."
Input Image
Ground Truth
Unseen Explanatory Instruction: "Remove the central black square to reveal the previously obscured parts of the architectural structure landscape, restoring the full scene."
Input Tmage
Ground Truth
Figure 41. Task level & instruction-level zero-shot samples for quantitative experiments (Inpainting).
Unseen Explanatory Instruction: "Expand vertical Floral display ly & broader composition ensuring the bloans are evenly spread across, with minimal visible soil."
Input Image
Ground Truth
Unseen Explanatory Instruction: "Expand the view horizontally to capture a wider perspective, incorporating additional elements from the periphery to create broader context."
Input Image
Ground Truth
Unseen Explanatory Instruction: "Expand the view by extending the horizonta dimensians both sides to reveal more of the surrounding
This creates wider scene, repl inpticainn the perspective broad corridor lined with books extending into the distance." "
Input Image
Ground Truth
Figure 42, Task level & instruction-level zero-shot samples for quantitative experiments (Ouspainting).
34

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Unseen Explanatory Instruction: "Enhancing the details by reducing the blurriness, increasing sharpness, and adjusting the contrast to bring clarity to scene."
Input Image
Ground Truth
Unseen Explanatory Instruction: "Increase the sharpness and enhance the reveal clear details make the colors more vibrant."
Input Image
Ground Truth
Unseen Explanatory Instruction: "Gradually refine the clarity by reducing any soft blur and enhancing ng details, while increasi ng contrast gently to make textures more pronounced."
Input Image
Ground Truth
Figure 43. Task level instruction-level zero-shot samples for quantitative experiments (Under-display Camera Image Restoration).
35

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
C. More Examples for Zero-shot Capabilities on Vision Tasks
In this section, we provide additional examples to demonstrate instruction-level and task-level zero-shot capabilities. The examples are generated using the model trained on the full DECVT. Note that all task-level zero-shot examples inherently fall under the category of instruction-level zero-shot examples as well. Additionally, we highlight the model's shortcomings based these examples. For more detailed exploration of limitations, please refer to Appendix D.
Input Image
Output Image
Figure 44. Examples of both task- and instruction-level zero-shol capabilities (Low-light Enhancement). Resolution: 448x448. Ex- planatory Instruction: "Increase the overall brightness to reveal details in dark areas while preserving highlights Adjust the contrast to enhance the brightness differances between regions, making the structures and textures more distinct. Optimize color saturation to make previously dull colors more vibrant, such as the blue the floor becoming commonly found in low-light images, improving the overall quality. Ensure the final image appears natural while retaining autheatic style of scene." " Limitations: Controlling the intensity of lighting en- hancement through language instructions is challenging, often resulting in significant deviations in the output.
prominent. Apply denoising reduce noise
BELL
Input Image
Output Image
Figure 45. Examples of both task- and instruction-level zero-shot capabilities (Map-to-lmage). Resolution: 448x Explana- tory Instruction: "Apply process
introduces intricate details by simulating realistic textures, adding natural elements like trees and vegetation, and enhancing geometric shapes into more organic forms. Convert abstract layouts into visually accurate structures, enhancing color depth and including real-werld spatial elements such reof patterns and environmental surzoundings." Limitations: The results short when compared to worldworld scenes.
Input Image
Output Image
Figure 46. Examples of instruction-level zero-shot capabilities (Deraining). Resolution: 448x448. Explanatory Instruction: "slowly remove the rain Fall lalling from the sky in the image, still maintain the state of night, and the girl on the bridge is also still helding umbrella, but readjust the light in the distance." Limitations: The model struggles to preserve smaller objects and environmental details.
36

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Input Image
Output Image
Figure 47. Examples of instruction-level zero-shot capabilities (Desnowing). Resolution: 448 448. Explanatory Instruction: "Remove the falling snow from the aky in the inage, keep the other objects and stow in the image, still keep the adjustment of light behind the tree." Limitations: The second generated
it dark, but pay attention image struggles to retain nighttime details, while the third and fourth images exhibit poor performance in removing snow from the sky. Additionally, attempting remove from the ground simultaneously result in significant distortions.
Input Image
Output Image
Figure 48. Examples of both task and lnstruction-level zero-shot capabilities (Deblurring). Resolution: 448x Explanatory Instruction: "The image shows noticeable multiple visual averlaps of trees and buildings. would like to remove visual overlaps and restore clear, sharp image without blurring. Do not alter the main content and pay attention to adjusting the light." Limitations: The success rate guiding the model's task-level zero-shot capability through language instructions is relatively low.
Input Image
Output Image
Figure 49, Examples instruction-level zero-shot capabilities (Dehazing). Resolution: 448x Explanatory Instruction: "Retain the distant el ouds in the image while removing much possible. Attempt to restore the faintly visible sun in the distance, but enaure there is no strong sunlight. Focua on zecovering the mountains and the nearby trees much possible." LLimitations: will cause distortions in certain objects.
37

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
In the following, provide few simple examples used for evaluation.
Unseen Explanatory Instruction: "Acknowledge the spatial structure and identify variations in light intensity, translating these into gradient scale zepresenting distances. Acsentuate regions where light diminishes gradually, enhancing the
perception of depth by dimming peripheral areas, Adjust the
Input Image
distribution of luminance to highlight the central vanishing transitions
Output Image
Ground Truth
point, corverting detailed textures into of grayscale."
Unseen Explanatory Instruction: "Start by analyzing the spatial layout identify key structural elements. Gradually obscure less relevans details in the periphezy to focus primarily on central depth. Increase contrast between light and dark areas to enhance perception of distance. Transition the textures Input Image into smooth gradi ents to reflect variations in depth, with a focus enhanced luminosity regions that are away." "
Output Image
Ground Truth
Unseen Explanatory Instruction: "Convert each region's color intensity to grayscale distance from the viewer, with nearer objects appearing lighter and those farther away darker. Gradually smooth transitions between these regions to reflect continuous depth variation. Remove textural details that do not affect perceived depth create uniformity based on object preximity. Adjust overall brightness te highlight the spatial configuration ion without explicit texture representation."
correaponding
relative
Input Image
Output Image
Ground Truth
Figure 50. Examples of instruction-level zero-shot capabilities (Depth Extimation). Resolution: 448 x448.
Unseen Explanatory Instruction: "Translate the visible struerures into a range of bright colors reflecting oriantation angles, enhancing variations across surfaces."
rqua
Input Image
Output Image
Truth
Unseen Explanatory Instruction: "Cenvert visual elements into spectrum of colors that represent the directionality surfaces, capturing the angles orientations vividly."
Input Image
Output Image
Ground Truth
Unseen Explanatory Instruction: "Translate the scene into colerful array to indicate surface orientations and engles."
Input Tmage
Output Image
Ground Truth
Figure 51. Examples of instruction-leww zero-shot capabilities (Surface Normal Estimation). Resolution: 448x448.
38

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Unseen Explanatory Instruction: "Apply pink color overlay bicycles, completely matching their shapea."
Input Image
Output Image
Ground Truth
Unseen Explanatory Instruction: "Apply solid iolid grey color tint to fully banana instance. Paint over each stove with powderblue color."
Input Image
Output Image
Ground Truth
Unseen Explanatory Instruction: "Spectral_" is the reversed version of Spectral, transitioning through rea, yellow, green, and blue. Based the previously defined colors, help me complete the segmentation task below. Color all instances bucket, toilet using Spectral_r colors, following their contours precisely."
Input Image
Output Image
Ground Truth
Figure 52, Examples instruction- level zero-shot capabilities (Semantic Segmentation). Resolution: 448 x488.
Unseen Explanatory Instruction: "Mark the cat-like creature the picture with yellow box."
Input Image
Output Image
Ground Truth
Unseen Explanatory Instruction: "Mark trees with snow in the picture with red boxes."
Input Image
Output Image
Ground Truth
Unseen Explanatory Instruction: "For the following object categories, apply the corresponding bounding box colors: Rnci rale al1 toilet objects with gold color border. Drew
thí bourding fan and apply crimson bounding box azound them. Detect
doorknob objects Detect
air sir_conditioner, box color: palevioletred color. Detect
Input Image
all mirror and draw darksalmen color bounding box.Detect foilet_tissee and apply darkturquoise bounding around them."
Output Image
Ground Truth
Figure 53. Examples of instruction-level zero-shot capabilities (Object Detection). Resolution: 448
39

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Unseen Explanatory Instruction: "Capture the out ine and prominent edges of the cylindrical object and its surroundings, simplify everything by removing textures and detailed surfaces, and emphasize only the contours and distinct features while rendering a higher contrast between light and dark regions with sharp shifts in tones."
Input Image
Output Image
Ground Truth
Unseen Explanatory Instruction: "The vibrant multiple colors and details could be simplified into
with
monochrome representation. First, focus on defining the high-contrast areas between light and dark in much starker,
black-and way. Then, it's important to emphasize contours significant edges, such as lines around the face, the dress' folds, and the furniture's s details, while downplaying aofter gradients. Removing extraneous colors and textures leaves behind only the essential structural features that provide abstract, but recogni zable zable si houette and objects."
Input Image
Output Image
Ground Truth
Unseen Explanatory Instruction: "Begin by el elininating most of the intricate details and colors, transforming the vibrant elements into simplified outlines. Keep only the borders and
defined structures, ensuring that the environment and figure take on an abstract form. Remove all texture, redacing the entire composition to minimal contrasting edges that define the shapes than the details."
Input Image
Output Image
Ground Truth
Figure 54, Examples of instruction-level zero-shot capabilities (HED Boundary Detection). Resolution: 448 448x
Unseen Explanatory Instruction: "Gradually reduce atmospheric interference, allowing clearer visibility of buildings and sharpening the outlines. Rnhance clarity and brightness bring out the details within the cityscape, providing crisper view."
Input Image
Output Image
Ground Truth
Unseen Explanatory Instruction: "Increasing the clarity by reducing haze, enhancing centrast, and deepening colors give sharper and more vibrant appearance to the scene."
Input Image
Output Image
Ground Truth
Unseen Explanatory Instruction: "To achieve clarity and vibrancy, adjust the brightness and reduce the foggy effect. Enhance the sharpness of the trees and structures, allowing their details stand out against the clear blue sky."
Input Image
Output Image
Ground Truth
Figure 55. Examples of instruction-level zero-shot capabilities (Dehazing). Resolution: 448 x 448.
40

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Unseen Explanatory Instruction: "Imagine a acenario where rainfall auddenly stope and the water settles, clearing up the acene to enhance visibility and eliminate rain streaks."
Input Image
Output Image
Ground Truth
Unseen Explanatory Instruction: "Renove the raindrops and stakks,, focusing enhancing clarity and brightness achleve crisp and rain-free appearance in the environment."
Input Image
Output Image
Ground Truth
Unseen Explanatory Instruction: "Imagine the rainfall gradually lessening until the sky clears completely, leaving only the vibrant greenery and the birds in focus."
Input Image
Output Image
Ground Truth
Figure 56. Examples of instruction-levdd zero-shot capabilities (Deraining). Resolution: 448x448.

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
D. Limitations and More Discussions
In this section, we provide more limitations and discussions of this work. Due to GPU resource constraints, we were unable to conduct additional validation experiments to address these issues. Therefore, we present only our reflections and insights based on the current results and our experiences.
1) As we have mentioned in Sec. 3 of the paper: "While the top-k value is typically set to 5 for text generation in large language models (1 (I..Ms), we recommend setting the top-k to 2048 during the image generation stage." This recommendation is also consistent with findings in Lumina-m Minni (Liu et al., 2024), where larger top-k value lead to improved visual details. However, this phenomenon should not necessarily apply to all vision tasks, i.e., in some vision tasks, larger top-k value should not improve the generate results. For instance, in standard segmentation tasks controlled by colors and classes, the answer should be unique for most of the image, with only a few pixels potentially subject to ambiguity. While in such cases, lowering the top-k value still can not improve the model's ability to follow these instructions. Our intuition is that the current VQ-based training methods and the straightforward fine-ture fin-ttning approach may have significant impact on this issue.
Dud
HD
Anowery
Dectby Acarrtn
Task-level recv-shot (Trgainting)
Task-leve zecv-shot (Oulgsinting)
Task-level coro-shot (Canng-brlmmeg
Figure 57. Latent space visualization of explanatory instructions.
2) When visualized in a reduced feature space, fixed task-specific instructions (e.g., Semantic Segmentation) exhibit discrete clustering- property that inherently ties task execution to rigid syntactic forms, potentially limiting generalization. In contrast, explanatory instructions span a continuous spectrum across tasks, which we identify as the primary driver of zero-shot generalization capability. To validate this, we extracted the text features of explanatory instructions for each task using the BERT (Devlin al., 2019) model, and then applied PCA reduce the high-dimensional features two dimensions. For better visualization, we randomly sampled 100 features per task and plotted them in a scatter plot. As shown in Fig. 57, explanatory instructions avoid the tight clustering seen with task-specific instructions, instead forming overlapping distributions even across distinct tasks. This continuity-wiie task boundaries blur in the latent space- enables the model to generalize by exploiting semantic relationships between instruction formulations rather than relying on pre-defined task categories.
3) Experiments in this work are limited to image pairs, and we do not explore more complex data such as video or 3D data. Actually, in the early stage of our experiments, we attempted construct explanatory instructions between different frames in videos. Unfortunately, for complex videos, even models like GPT-40 still produced significant errors within the descriptions (cf. Table 8). While for simpler videos (e.g., those depicting weather changes), although the descriptions tend to be accurate, we feel that adding data with minimal changes between frames does not significantly contribute to testing the zero-shot capability of the model, which is the main goal of this work (we acknowledge that such data, with minimal changes, could potentially enhance the model's ability to control generation and follow instructions, but this is not the focus of this study). Nevertheless, we believe that collecting more complex video data could be beneficial for advancing vision task-level generalization. Such data would not only improve the model's 's scene control capabilities but also increase the diversity of task objectives that the model can understand. While these tasks could be treated as form of editing, they surpass the capabilities of current editing models (e.g.. transformation from figure (a) (d) in Table 8).
4) In constructing the Dataset of Explanatory CV Tasks, we adhered as closely as possible to the principle that the provided instructions should avoid obvious inaccuracies. While GPT-4o exhibits one of the most advanced descriptive capabilities among existing models, it still faces challenges such as incomplete descriptions and occasional deviations. Although these data significantly enhance the model's 's ability to interpret a wide range of instructions and facilitate task- generalization,
42

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Descriptions of image flow A:
A closed structure with no visible contents.
2. The structure begins to reveal an internal object.
3. The internal object becomes partially visible.
Descriptions of image flow B:
(d)
partially revealed internal object within a structure.
2. The internal object becomes more visible.
3. The structure closes, hiding the internal object.
Explanatory instruction from to B:
1. Start with a closed structure.
2. Gradually reveal an internal object.
Continue revealing more of the internal object until is partially visible.
Progress showing internal object more clearly.
5. Finally, close the structure, hiding the internal object again. Explanatory instruction from B to A:
1. Begin with partially revealed internal object within structure. Gradually hide the internal object until it longer visible.
Ensure the structure completely closed with no visible contents.
4. Open the structure slightly to start revealing an internal object.
5. Continue to reveal more of the internal object until it is partially visible.
Table Video output sample from GPT-4o. Content that not meet description requirements highlighted in red.
they also lead to certain trade-off'. Specifically, the inclusion of these data can reduce the stability of the model's outputs and compromise fidelity to the original image content. Furthermore, these data issues, including the presence of some low-quality datasets for the image editing tasks, may adversely affect the model's instruction-followe capabilities.
5) Although the dataset construction and training approach described above have several limitations, believe that the of explanatory instructions can enhance the model's adaptability complex instructions and objectives. We hypothesize that there exists an optimal level of instruction complexity for models. Up to this threshold, more detailed instructions may improve the model's understanding and performance. However, beyond this point, the model's ability to grasp the intended objective may begin to decline. Moreover, when instructions encompass overly broad objectives while the model lacks sufficient capacity to process and generalize this information, it may lead to performance degradation.
6) Due to budget constraints during the construction of the Dataset of Explanatory CV Tasks, significant portion of the data was generated by directly instructing the model output explanatory instructions, bypassing the generation of image captions. However, based empirical observations, recommend that the model generate image captions before producing explanatory instructions. As for GPT-4o, if the generated image captions exhibit substantial errors, the explanatory instructions are also likely to be inaccurate. Conversely, when the image captions are accurate, the explanatory instructions tend to be more precise. This approach helps to significantly reduce noise within the dataset.
7) Due to resource limitations, we only conducted our experiments on the vanilla token- based VLM with 7B parameter, and
43

---

Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
image-caption-base data used for image generation during training. While this work demonstrates that explanatory instructions enable zero-shot generalization at the vision task level, this ability remains unstable. Furthermore, benchmark datasets, the performance of our model under zero-shot settings still lags behind task-specific models and some vision generalist models. This performance gap remains substantial compared to our expectations: as humans possess the ability to generalize across different tasks during the learning process and, based on these generalizations, can better accomplish existing tasks. We guess the primary reasons for the current limitations lie in the architecture of the model and data noise the Dataset of Explanatory CV Tasks.
8) Just as with language understanding, visual cognition also vary from person to person. For various vision tasks, even when dealing with vision tasks related single image pair, different people may interpret and describe the images and tasks differently. Therefore, we think that the simple next-token prediction approach may be well-suited to those vision tasks that have already been generalized and defined by humans (e.g., Depth Estimation and Segmentation). However, for more complex vision tasks, directly training through next-token prediction may not be the most appropriate approach, as task descriptions can vary widely and multiple valid answers may exist.
44


Reviews:
[{"review_id": "iKWfpWLToJ", "reviewer": "Reviewer_v7hm", "summary": "This work proposes a new method for fine-tuning visual tasks, explanatory Instructions. Inspired by the work on text instruction fine-tuning, the authors aim to explore whether there is a generalization phenomenon in instruction fine-tuning for visual tasks. Therefore, they construct pure-text instructions for different visual tasks and use a large vision model, specifically Lumina-mGPT-7B-768-Omni, to verify the effectiveness of the constructed instruction dataset. The authors observe both intra-task generalization ability and cross-task generalization ability. Meanwhile, this article proposes an Explanatory Instructions dataset.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your recognition of the idea presented in our paper. Below are our responses to the concerns you raised.\n\n**Question 1 & 4 / Weakness 3**: *I am curious about the cross-task generalization ability of the initial VLM model, Lumina-mGPT-7B-768-Omni, without any training. Does task generalization come from the base model or from instruction fine-tuning? It is necessary to further demonstrate whether this cross-task generalization ability is inherent in the base model itself or is stimulated by the task fine-tuning method. It is recommended to use multiple base models for proof.*\n\n**Response**: Thank you for your valuable suggestions. We would like to clarify that, in our experiments on analysing the **zero-shot capabilities on unseen vision tasks**, the base model used for initialization was **Lumina-mGPT-7B-768** (cf. 300~301 of the paper, it is a text-to-image generator). This base model **has no multi-task capabilities beyond text to image generation**, so it cannot perform other vision tasks on its own. \n\nThus, any cross-task generalization observed comes from our instruction fine-tuning, not from an inherent ability of the pre-trained model. We explicitly discuss this in Appendix B.2 (lines 1546–1551), confirming that Lumina-mGPT-7B-768-Omni by itself does not generalize to unseen tasks – the generalization emerges only after training with our explanatory instructions.\n\n\\-------------------------------------------------------------------------------------------------------------------------\n\n**Question 2 / Weakness 2**: *What if the base model is a text + image-to-image model, such as Stable Diffusion (SD)?*\n\n**Response**: Thank you for your suggestions. Using a stronger or more flexible foundation model (e.g. more advanced DiT model) could indeed further improve the overall performance. Our primary goal, however, was to validate the methodology of unified vision task-level generalization rather than to chase state-of-the-art results on any single base model. Furthermore, traditional stable diffusion model can only generate images but can not generate text. Although directly using stable diffusion (text + image-to-image) can also adopt such vision task-level zero-shot capability (following settings in Section 4.2 of the paper), to enhance the scalability of the model, i.e., enabling the model to simultaneously generate text, images, and other multimodal data, we therefore adopted the vanilla AR-based VLM introduced in Chameleon and Lumina-mGPT.\n\n\\-------------------------------------------------------------------------------------------------------------------------\n\n**Question 3 / Weakness 1**: *Are there any quantitative results? Case-level arguments are hardly convincing enough for me to determine whether the theory you proposed is truly reliable.*\n\n**Response**: Thank you for your suggestions. Actually, we have provided **quantitative results in Appendix B (Ln. 1375\\~1510)** of the paper. In particular, Table 6 and Table 7 in Appendix B reports results on 5 instruction-level zero-shot tasks (the instructions are not seen during training) and 5 task-level zero-shot tasks (both the task and instructions are not seen during training). **Notably, all methods used for comparison do not use instruction-level or task-level zero-shot settings except Lumina-mGPT.** However, Lumina-mGPT requires rigid and fixed format prompts as input, as we had discussed in Ln. 1546\\~1551 of the paper: \nIn the evaluation process for Lumina-mGPT in Table 6, all instructions follow this format: \"``Generate an image according to the provided image, and according to the following caption: {Image Caption},
``\". For the experiments in Appendix B.1, the instructions are \"``Depth estimation. 
``'' for the *Depth Estimation* task, \"``Semantic segmentation. 
``'' for the *Semantic Segmentation* task and \"``Surface normal estimation. 
``'' for the *Surface Normal Estimation* task. Any alteration to the format of these instructions leads to model failure or significantly degraded its performance.\n\nIn addition, following the suggestions from Reviewer #yk8P (Question 1), we further demonstrate that under both the instruction-level and task-level zero-shot settings, there is a significant divergence between the training and test samples (visualizations for instruction-level zero-shot can be found in the following anonymous link: https://anonymous.4open.science/r/ICML_Re-C752/Instruction_level_zero_shot_bert_pca_3_samples.pdf , visualizations for task-level zero-shot can be found in the following anonymous link: https://anonymous.4open.science/r/ICML_Re-C752/task_level_zero_shot_bert_pca_3_samples.pdf). This provides additional evidence for the model’s generalization capability.", "claims_and_evidence": "NA", "methods_and_evaluation_criteria": "NA", "theoretical_claims": "NA", "experimental_designs_or_analyses": "NA", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "I really really like the idea proposed in this paper. However, the discussion on zero-shot task generalization in this paper is clearly insufficient. If my concerns can be addressed and the paper can be further improved, I will raise my score. Otherwise, I will lower my score after the rebuttal stage.\n\nStrengths:\nThe idea put forward in this paper, namely that zero-shot generalization can be observed across different vision tasks, is an important finding. The publicly available dataset Explanatory Instructions will also make a significant contribution to the research community and accelerate the process of achieving true unification in visual tasks. \n\n\nWeaknesses:\n1. There are only relatively subjective case analyses as experimental results. It is unclear whether they have been cherry-picked, and there is no quantitative measurement of the true capability of solving tasks generalized in the zero-shot setting.\n2. I am not sure if a well-trained stable diffusion model for image generation can also solve these problems.\n3. I am curious about the cross-task generalization ability of the initial VLM model, Lumina-mGPT-7B-768-Omni, without any training. Does task generalization come from the base model or from instruction fine-tuning?", "other_comments_or_suggestions": "NA", "questions_for_authors": "1. I am curious about the cross-task generalization ability of the initial VLM model, Lumina-mGPT-7B-768-Omni, without any training. Does task generalization come from the base model or from instruction fine-tuning?\n\n2. What if the base model is a text + image-to-image model, such as Stable Diffusion (SD)?\n\n3. Are there any quantitative results? Case-level arguments are hardly convincing enough for me to determine whether the theory you proposed is truly reliable.\n\n4. It is necessary to further demonstrate whether this cross-task generalization ability is inherent in the base model itself or is stimulated by the task fine-tuning method. It is recommended to use multiple base models for proof.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xp1Al7dqXu", "reviewer": "Reviewer_Mwwt", "summary": "This paper proposes Explanatory Instructions to address the challenge of task-level zero-shot generalization in computer vision, inspired by the success of instruction-driven models in NLP. The authors hypothesize that conventional terminological task definitions (e.g., \"semantic segmentation\") limit models' ability to generalize, as they fail to convey task objectives intuitively. To overcome this, they introduce Explanatory Instructions—natural language descriptions of image-to-output transformations—and construct a Dataset of Explanatory CV Tasks (DECVT) with 12M \"image-input → instruction → output\" triplets. A token-based autoregressive vision-language model is fine-tuned on DECVT, demonstrating instruction-level (unseen instructions for seen tasks) and task-level (unseen tasks like Canny-to-Image) zero-shot capabilities. Experiments show improvements over baselines in image editing, generation, and low-level tasks, though performance gaps remain compared to task-specific models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback on our paper. In the following responses, we have addressed the concerns you raised during the review process, and we hope these answers will resolve your questions.\n\n**Question 1 / Weakness 1**: *The authors used GPT-4o for training data generation, especially the instructions. From my experience, although GPT-4o seems better than other VLMs as discussed in Appendix A, there still remains potential noise and biases in the generated instructions. How the authors deal with these noises, biases, and even inaccuracies?*\n\n**Response**: We appreciate the reviewer highlighting this important issue. Although GPT-4o indeed provides superior capability compared to other VLMs (as detailed in Appendix A), the instructions it generates still contain noise, biases, or inaccuracies. To address these potential issues, we have implemented a data quality control procedure.\nTo be specific, we noticed that GPT-4o can occasionally produce unstable or inaccurate explanatory instructions if it misunderstood the input images. To mitigate this, we simultaneously asked GPT-4o to generate captions corresponding to the images. We then employed CLIP to measure the semantic similarity between these captions and their associated images. Any generated data pair with a similarity score below 0.5 was automatically filtered out from the final dataset. Otherwise, we maintained the explanatory instructions. In addition, a portion of our dataset was manually annotated as described in Section A.1 (Ln, 718\\~719), ensuring high-quality and bias-free explanatory instructions.\n\n\\-------------------------------------------------------------------------------------------------------------------------\n\n**Question 2 / Weakness 2**: *Section 2 provides a detailed account of the dataset composition, and the supplementary materials explain how the dataset was constructed. However, certain details remain unspecified. For instance, in the task-level zero-shot experiments, what proportion of the data is allocated to each task? And in the complete dataset, what is the ratio of each task pair?*\n\n**Response**: Thank you for your suggestions. We clarify these details as follows:\n\n**Complete Dataset Composition**: Our full dataset consists of two components: Terminological-based Vision Tasks and Explanatory-based Vision Tasks. Specifically, the Terminological-based Vision Tasks component includes approximately 4M individual “input image → explanatory instruction → output” triplets for image editing tasks. For the other vision tasks within this component, each task has a varying number of triplets, ranging from a minimum of 0.05M to a maximum of 2M. In contrast, the Explanatory-based Vision Tasks component contains approximately 2M individual “input image → explanatory instruction → output” triplets.\n\n**Task-Level Zero-Shot Experiment Allocation**: As detailed in Section 4.2 (Ln. 320\\~329), to **rigorously test task-level zero-shot generalization**, we deliberately removed data corresponding to the following tasks from the Terminological-based Vision Tasks component: *Image Restoration*, *Depth Estimation*, *Depth-to-Image*, *Surface Normal Estimation*, *Surface Normal-to-Image*, *HED Boundary Detection* and *HED-to-Image*. From the remaining tasks, we then constructed the training dataset for zero-shot experiments by randomly selecting 1M triplets from the Explanatory-based Vision Tasks component, 1M triplets specifically for image editing tasks, and another 1M triplets from other remaining vision tasks.\n\n\\-------------------------------------------------------------------------------------------------------------------------\n\n**Question 3 / Weakness 3**: *The examples in Figure 32 are quite helpful in illustrating the effectiveness of explanatory instructions. However, the authors only provided pair of examples, and it would be beneficial if they included more convincing samples.*\n\n**Response**: Thank you for your suggestions. We have provided additional examples at the following anonymous link: https://anonymous.4open.science/r/ICML_Re-C752/More_Examples.pdf .", "claims_and_evidence": "Most claims made in the submission are supported by clear and convincing evidence, e.g.,\n1) Explanatory Instructions improve instruction-level and task-level zero-shot generalization.\n2) DECVT enables models to handle diverse vision tasks via linguistic instructions.\n3) The AR-VLM achieves competitive results on unseen tasks (e.g., Depth-to-Image) compared to vision generalist models.", "methods_and_evaluation_criteria": "As the unified token-based AR-VLM architecture aligns with recent trends in multimodal modeling, this paper further constructed a systematic dataset of explanatory CV tasks, combining manual annotation and GPT-4o-generated instructions. Although the results in Table 5 ~ 7 still show significant gaps compared to state-of-the-art task-specific models, both the task-level and instruction-level zero-shot generalization for CV tasks shown in this paper are exciting.", "theoretical_claims": "The paper does not introduce formal theorems or rigorous mathematical proofs, so there is no specific need for verification of theoretical correctness. Its principal contribution lies more in method design and dataset construction than in theoretical innovation.", "experimental_designs_or_analyses": "The authors make comprehensive experiments across 10+ tasks in Appendix B. During the evaluation for task-level zero-shot tasks, it seems that authors have excluded specific tasks during training, which make the zero-shot scores convincing. However, I think there need some discussions for failure cases.", "relation_to_broader_scientific_literature": "The paper clearly situates its contributions within the broader scientific literature. It identifies a clear gap between NLP and CV regarding zero-shot generalization capabilities, acknowledging previous approaches such as Lumina-mGPT, OmniGen, and PixWizard. The paper builds convincingly upon prior NLP-inspired vision-language paradigms and emphasizes the uniqueness of explanatory-based instruction as a conceptual advance beyond existing terminological frameworks.", "essential_references_not_discussed": "Overall, the authors cite most of the major and emerging literature on multi-task vision models and multimodal pretraining.", "other_strengths_and_weaknesses": "I just brief summarize some other strengths and weakness.\n\nStrengths:\n1)\tNovel idea of Explanatory Instructions for both task-level and instruction-level generalization in computer vision.\n2)\tThe proposed large-scale dataset is meaningful to the community, the dataset also encompasses various transformations and task types.\n3)\tAuthors provide various discussions in the appendix.\n4)\tIf validated further, this approach could be an important training paradigm for general-purpose multimodal models.\n\nWeaknesses:\n1)\tOverreliance on GPT-4o for instruction generation introduces potential noise and biases.\n2)\tThere lacks an explanation regarding the distribution of data across different tasks in the dataset.\n3)\tZero-shot experiments can address a subset of unseen tasks, while the authors have also discussed in Section 5. More diverse or challenging tasks need exploration to confirm broader scalability.", "other_comments_or_suggestions": "None", "questions_for_authors": "1)\tThe authors used GPT-4o for training data generation, especially the instructions. From my experience, although GPT-4o seems better than other VLMs as discussed in Appendix A, there still remains potential noise and biases in the generated instructions. How the authors deal with these noises, biases, and even inaccuracies? \n2)\tSection 2 provides a detailed account of the dataset composition, and the supplementary materials explain how the dataset was constructed. However, certain details remain unspecified. For instance, in the task-level zero-shot experiments, what proportion of the data is allocated to each task? And in the complete dataset, what is the ratio of each task pair?\n3)\tThe examples in Figure 32 are quite helpful in illustrating the effectiveness of explanatory instructions. However, the authors only provided pair of examples, and it would be beneficial if they included more convincing samples.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SXQMwLwXeA", "reviewer": "Reviewer_yk8P", "summary": "This paper proposes a concept called “Explanatory Instructions” to move beyond the conventional limitations of computer vision (CV) tasks. The authors argue that the currently common terminological definitions (e.g., “semantic segmentation”) oversimplify the expression of CV objectives, limiting the model’s ability to achieve generalized zero-shot performance across tasks. To address this, they propose using detailed natural language instructions to describe transformations between input and output images in a large-scale dataset. The idea is that by exposing the model to these more expressive and fine-grained task descriptions, it learns to “understand” the underlying objectives and changes for each visual task. Built on an autoregressive vision-language model their experiments show that the method not only generalizes in zero-shot fashion to previously seen tasks but also demonstrates some ability to handle unseen tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive evaluation of our paper. Below are our responses to the concerns you raised.\n\n**Essential References Not Discussed**: *While the paper covers recent VLM-related literature well, it would benefit from discussing more explicitly related previous work on task-level generalization in vision-language domains.*\n\n**Response**: Thank you for your suggestions. We will discuss these works in our paper.\n\n\\-------------------------------------------------------------------------------------------\n\n**Question 1**: *Could Explanatory Instructions potentially exhibit clustering patterns?*\n\n**Response**: Thank you for your insightful comment. Based on your suggestion, we analyzed 14 different **Terminological-based Vision Tasks** (including *Image Restoration*, *Deraining*, *Dehazing*, *Desnowing*, *Object Detection*, *Style Transfer*, *Depth Estimation*, *Surface Normal Estimation*, *Pose Estimation*, *Semantic Segmentation*, *HED Boundary to Image*, *Pose-to-Image*, *Depth-to-Image*, *Segmentation-to-Image*) to explore whether Explanatory Instructions exhibit clustering patterns.\n\nWe extracted the text features of Explanatory Instructions for each task using the BERT model, and then applied PCA to reduce the high-dimensional features to two dimensions. For better visualization, we randomly sampled 100 features per task and plotted them in a scatter plot.\n\nResults can be found in the following anonymous link: https://anonymous.4open.science/r/ICML_Re-C752/instruction_features_pca_bert_all_tasks.png .Our findings show that **the Explanatory Instructions, when used to express vision tasks, do not exhibit the same clustering patterns as fixed task-specific instructions** (e.g., \"Semantic Segmentation\"). Fixed task-specific instructions tend to be highly discrete in nature, which may hinder the model’s generalization capability. On the other hand, Explanatory Instructions display a greater degree of continuity, which enhances the model’s ability to generalize across different vision tasks.\n\nAdditionally, we further investigated the zero-shot performance of specific vision tasks during testing. Specifically, we plotted the Explanatory Instructions used during both the training and testing phases in the same manner. For instruction-level zero-shot, we focused on the Explanatory Instructions for the same task in both the training and testing phases. We provide 3 examples in the following anonymous link: https://anonymous.4open.science/r/ICML_Re-C752/Instruction_level_zero_shot_bert_pca_3_samples.pdf . For task-level zero-shot, we visualized all the Explanatory Instructions across the training and testing tasks. For the selected zero-shot vision task, we random sample 500 features. We also provide 3 examples in the following anonymous link: https://anonymous.4open.science/r/ICML_Re-C752/task_level_zero_shot_bert_pca_3_samples.pdf . The above results demonstrate that in both instruction-level zero-shot and task-level zero-shot scenarios, there is almost no overlap between the training set and the test set.\n\nWe hope this analysis helps clarify the potential of Explanatory Instructions for enhancing task-level zero-shot generalization, and we appreciate your suggestion for further exploring clustering patterns.\n\n\\-------------------------------------------------------------------------------------------\n\n**Question 2**: *As real-world instructions can be vague or ambiguous. So if the instruction phrasing, length, style, or ordering changes, does this result in notable performance differences for image generation or understanding tasks?*\n\n**Response**: We thank the reviewer for raising this important question. Indeed, real-world instructions can vary significantly in phrasing, length, style, and ordering, and such variations can impact model performance. To thoroughly address this concern, we have provided numerous examples of real-world instructions in Appendix C, explicitly illustrating a wide diversity in phrasing, length, style, and structure.\n\nIn our validation experiments, the unseen explanatory instructions indeed include substantial variations in linguistic expression. We acknowledge that different descriptive language choices can lead to noticeable differences in the model's generation or interpretation outcomes. However, it is precisely this linguistic diversity that facilitates stronger generalization capabilities of our model. For instance, as discussed explicitly in Appendix B.1.4 (Fig. 32), when encountering categories not included in the training set (e.g., \"broad-winged damselfly\"), the model struggles to recognize the category name alone. Yet, it significantly benefits from alternative descriptive expressions (e.g., \"the creature on the leaf\"), highlighting the model’s improved interpretability under instruction diversity.\n\nTo further illustrate this behavior, we provide additional examples at the following anonymous link: https://anonymous.4open.science/r/ICML_Re-C752/More_Examples.pdf .", "claims_and_evidence": "The primary claim of the paper is, by introducing Explanatory Instructions and constructing a corresponding large-scale dataset, the authors claim to achieve zero-shot generalization for both seen and unseen vision tasks.\nThe authors have presented numerous qualitative examples—such as generating reasonable outputs for previously unseen image-to-image transformation tasks—and some initial tests to illustrate zero-shot performance.\nWhile the qualitative results are intuitive, more thorough quantitative evaluations or systematic comparisons with existing baselines would bolster the credibility of their claims about task-level zero-shot generalization.", "methods_and_evaluation_criteria": "I think the proposed methods and evaluation criteria in this paper are appropriate for addressing zero-shot generalization in vision tasks. The intuitive that define CV task objectives through detailed linguistic transformations from input images to outputs also sounds great.\n-Methods: The authors adopt a standard autoregressive Transformer as the backbone for their vision-language generative model, converting both image and text inputs into discrete token sequences, and training via next-token prediction.\n-Datasets and Evaluations: The authors constructed a large dataset, separating some tasks to remain unseen for zero-shot testing. This design helps assess whether the model can generalize to tasks not included in its training set.\nFrom an application perspective, replacing fixed “task-name” categories with more flexible, instructive natural language is a promising approach. It also aligns with current instruction tuning trends in NLP. However, it seems that the evaluations make sense are primarily qualitative visualizations.", "theoretical_claims": "The paper does not include formal theoretical claims or proofs, so there was nothing to verify in this regard.", "experimental_designs_or_analyses": "The key experimental setup is distinguishing between tasks included in the training set and tasks that remain unseen, testing for zero-shot performance on the latter. Most analyses focus on sample outputs and visual demonstrations of how the model handles various image editing and transformation tasks. While these examples highlight the model’s adaptability, a more fine-grained quantitative analysis, such as performance under different instruction styles or varied input modalities would provide additional insight into robustness and failure modes.", "relation_to_broader_scientific_literature": "This work is building on vision-language models and closely tied to the current trend of creating generalist models for vision tasks. Recent methods, such as Lumina-mGPT and OmniGen, rely on the notion of “task tag + input-output” to unify tasks such as image generation, segmentation, depth estimation, etc. In contrast, this paper introduces more descriptive natural language explanations to capture the goal behind each task, potentially enlarging the task space and improving zero-shot feasibility.", "essential_references_not_discussed": "While the paper covers recent VLM-related literature well, it would benefit from discussing more explicitly related previous work on task-level generalization in vision-language domains, such as [1] and [2].\n\n[1] Bachmann R, Kar O F, Mizrahi D, et al. 4M-21: An any-to-any vision model for tens of tasks and modalities. NeurIPS, 2024: 61872-61911.\n[2] Xiao B, Wu H, Xu W, et al. Florence-2: Advancing a unified representation for a variety of vision tasks. CVPR, 2024: 4818-4829.", "other_strengths_and_weaknesses": "Other Strengths:\n1.Substantial scale of dataset (12 million triplets) facilitating robust training.\n2.Promising demonstration of task-level zero-shot generalization beyond conventional terminological boundaries.\n3.Clearly articulated ideas and methodological innovations.\n\nOther Weaknesses:\n1.It seems that this paper could benefit from stronger baselines or comparative studies with existing methods.\n2.This paper also lacks some in-depth analysis for the proposed Explanatory Instructions, details can refer to “Questions for authors” part.", "other_comments_or_suggestions": "Authors can consider providing quantitative evaluation metrics (FID, CLIP scores, etc.) in the main paper.", "questions_for_authors": "As mentioned in the weaknesses, I am more focused on some analytical issues regarding Explanatory Instructions.\n\n1. Could Explanatory Instructions potentially exhibit clustering patterns? For instance, short editing instructions and fixed task-specific instructions (e.g., “Semantic Segmentation”) used in some VLMs might cluster around a few points when their features are extracted and visualized. I wonder whether different descriptions of the same task, when expressed via Explanatory Instructions, would demonstrate similar properties. If even for the same task, the representations via Explanatory Instructions in the evaluation dataset show significant bias. For example, wide dispersion in feature space. This could provide stronger evidence for the feasibility of task-level zero-shot generalization.\n\n2. As real-world instructions can be vague or ambiguous. So if the instruction phrasing, length, style, or ordering changes, does this result in notable performance differences for image generation or understanding tasks? \n\nIf Explanatory Instructions show higher intra-task diversity and clear inter-task separation, it would empirically validate their ability to capture task objectives beyond rigid terminological definitions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LaZAJ3Eurt", "reviewer": "Reviewer_7Dut", "summary": "This work proposed a large-scale explanatory instruction dataset to unify multiple CV tasks for AR VLM understanding and then generation. It uses VQ-VAE style tokens for vision and then an AR model to merge the 2 modality. It provides qualitative results on various tasks, showing the zero-shot capabilities.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We realize there may be some misunderstandings regarding both the dataset and idea presented in our paper. We hope the following responses will help clarify these points.\n\n**Experimental Designs**: *The model is trained for 2 epcohs.*\n\n**Response**: Thank you for your comments. Most large-scale VLM or AR models typically adopt only 1 epoch during pre-training, primarily due to the sheer volume of data available, which is usually sufficient for the model to generalize effectively. However, for SFT, it is reasonable to employ more than 1 epoch, since the dataset size is typically smaller and involves more complex tasks. Specifically, our dataset includes rich explanatory instructions covering diverse vision tasks, necessitating multiple exposures for the model to effectively learn and generalize. Empirically, we observed improved performance and better instruction comprehension when training for 2 epochs compared to just 1. We will clarify this reasoning to avoid confusion.\n\n**Weakness 1**: *It is common knowledge that an instruction method would be helpful to in the proposed task.*\n\n**Response**: We have to clarify that the vision task-level zero-shot generalization has not been clearly addressed by any prior work. Indeed, prior sota vision-language models do not demonstrate this capability to generalize across multiple distinct vision tasks. The idea is also considered highly innovative by Reviewers #v7hm, #Mwwt, and #yk8P. We notice that the reviewer might misunderstand the type of generalization we target. Specifically, we distinguish clearly between **task-level zero-shot generalization** and **data-level zero-shot generalization**, the latter of which has indeed been extensively studied in prior works.\n\nData-level zero-shot generalization, exemplified by models like VideoPoet that the reviewer mentioned, refers to generating diverse outputs (e.g., images or videos) conditioned on various textual prompts within a single predefined vision task, such as text-to-image or text-to-video generation. Although VideoPoet achieves impressive zero-shot performance in video generation tasks, it still remains constrained to the single task of video generation and does not exhibit cross-task generalization capabilities. In contrast, our paper explicitly addresses the fundamentally different challenge of task-level zero-shot generalization, where the model is expected to understand and execute completely novel and diverse vision tasks solely based on explanatory instructions, without task-specific fine-tuning.\n\nMoreover, to facilitate and validate this task-level generalization, we have constructed a dataset consisting of approximately 12 million individual \"input image → explanatory instruction → output\" triplets. Unlike existing datasets, which typically contain predefined task-specific annotations, our dataset uniquely provides rich explanatory instructions that describe the objectives of diverse vision tasks explicitly. To the best of our knowledge, this dataset is the first and currently the only dataset specifically designed for studying and enabling unified task-level zero-shot generalization in vision tasks.\n\n**Weakness 2**: *There are not many quantitative results.*\n\n**Response**: We appreciate the reviewer raising this concern. To clarify, we have indeed provided extensive quantitative evaluations in the supplementary materials, covering 12 different tasks in total. For each task, representative visual examples are also included in Appendix Section C for further transparency.\n\nIt is important to note that all baseline methods we compared against, with the exception of Lumina-mGPT, do not utilize instruction-level or task-level zero-shot settings. While Lumina-mGPT is closer to our method in principle, it still requires rigid and fixed-format prompts for each specific task, as explicitly discussed in lines 1546\\~1551 of the paper, and it did not provide any quantitative results for both instruction-level or task-level zero-shot experiments. Thus, it does not truly operate in a fully zero-shot, instruction-driven manner as proposed by our method. Additionally, to the best of our knowledge, this paper is the first work to provide quantitative analyses under both instruction-level and task-level zero-shot settings.\n\n**Question**: *The dataset is like distilling ability from GPT?*\n\n**Response**: The reviewer seems misunderstood the construction of the dataset. Before the submission of this paper, the three GPT-4o models officially released by OpenAI, i.e., gpt-4o-2024-05-13, gpt-4o-2024-08-06, and chatgpt-4o-latest, did not possess image generation or image editing capabilities, let alone the vision task-level generalization capability to be addressed by our work. In the construction of this dataset, we only adopt gpt-4o to generate part of the explanatory instructions, while other explanatory instructions for terminological-based vision tasks are manually annotated.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "None in this work.", "experimental_designs_or_analyses": "1. The model is trained for 2 epcohs. My impression is that most VLM or AR model either during pre-training or fine-tuning train for 1 epoch. Does the author justify the setup?", "relation_to_broader_scientific_literature": "This work contributes to the areas of VLM understanding and generation, more specifically, showing a unified AR model can do multiple CV tasks and zero-shot generalization.", "essential_references_not_discussed": "I think the VideoPoet work bears resemblance to what the authors want to achieve, using AR VLM to do some zero-shot tasks, but there was no mention of it.", "other_strengths_and_weaknesses": "Weakness:\n\n1. It is common knowledge that an instruction method such as this paper proposed would be helpful to in the proposed task. So, I'm not sure if the novelty is not already understood or practiced.\n\n2. There are not many quantitative results to objectively compare with others as a whole. (I know there are a few in the appendix.) \n\n\nStrength:\n1. The paper provided a large-scale synthetic dataset that integrates multiple CV tasks.", "other_comments_or_suggestions": "I think the captions of the images can be more explanatory for reading flow, but that is optional.", "questions_for_authors": "So given the dataset is a bit like generated from GPT-4o, wouldn't it be like distilling ability from GPT? and resulting in a chicken-n-egg problem? What if there is no ChatGPT?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes Explanatory Instructions to address the challenge of task-level zero-shot generalization in computer vision. All reviewers unanimously acknowledged that the methodology proposed by the authors exhibits a high degree of novelty, supported by comprehensive experimentation and substantiated content. Following extensive deliberation, the concerns raised by the reviewers have been adequately addressed by the authors.  Therefore, I recommend accepting this submission.
Decision: Accept (poster)