In-Context Learning Unlocked for Diffusion Models
Authors: Zhendong Wang
Yifan Jiang
Yadong Lu
yelong shen
Pengcheng He
Weizhu Chen
Zhangyang Wang
Mingyuan Zhou
Venue: NeurIPS 2023 spotlight (2023)
Keywords: diffusion models
in-context learning
TL;DR: Unlock the in-context learning ability of diffusion models with vision-language prompt.

Abstract: We present Prompt Diffusion, a framework for enabling in-context learning in diffusion-based generative models. Given a pair of task-specific example images, such as depth from/to image and scribble from/to image, and a text guidance, our model automatically understands the underlying task and performs the same task on a new query image following the text guidance. To achieve this, we propose a vision-language prompt that can model a wide range of vision-language tasks and a diffusion model that takes it as input. The diffusion model is trained jointly on six different tasks using these prompts. The resulting Prompt Diffusion model becomes the first diffusion-based vision-language foundation model capable of in-context learning. It demonstrates high-quality in-context generation for the trained tasks and effectively generalizes to new, unseen vision tasks using their respective prompts. Our model also shows compelling text-guided image editing results. Our framework aims to facilitate research into in-context learning for computer vision. We share our code and pre-trained models at https://github.com/Zhendong-Wang/Prompt-Diffusion.

In-Context Learning Unlocked for Diffusion Models
Zhendong Wang : Yifan Jiang', Yadong Lu², Yelong Shen", Pengcheng He? Weizhu Chen?, Zhangyang Wang's and Mingyuan Zhou' 'The University of Texas at Austin, "Microsoft Azure Al
Complh Det wagee
Dam mane
Quiset
Camy: Areth
"Syy Porker Gecrpes hoomio
for dpocce, with
post shesa prd
Papecc 0S5
Seepl Gallery
BUTTERWARE AACENNAT
tartlestands te forer
Figure 1: Illustration of the in-context learning ability enabled by proposed Prompt Diffusion for conditioned image generation tasks: With a prompt consisting task-specific example pair of images and guidance, given a new query image that aligns type with the source image in the example pair, Prompt Diffusion can comprehend the desired task and generate the corresponding output image on both seen (trained) and unseen (new) task types.
Abstract
We present Prompt Diffusion, a framework for enabling in-context learning in diffusion-based generative models. Given pair of task-specific example images, such as depth from/to image and scribble from/to image, and a text guidance, our model automatically understands the underlying task and performs the on a new query image following the text guidance. To achieve this, we propose vision-language prompt that model wide range of vision-language tasks and a diffusion model that takes it as input. The diffusion model is trained jointly six different tasks using these prompts. The resulting Prompt Diffusion model becomes the first diffusion-based vision-language foundation model capable of in-context learning. It demonstrates high-quality in-context generation for the trained tasks and effectively generalizes to new, unseen vision tasks using their respective prompts. Our model also shows compelling text-guided image editing results. Our framework aims to facilitate research into in-context learning for computer vision. We share our code and pre-trained models https:::s :/ttps://githtl com/ hetp/hennonn--Wageee
task
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
Recent advancements in machine learning, particularly in the field of natural language processing (NLP) 154) have led the development of several state-of-the-art large language models (LLMs), such as BERT (9), GPT-2 [36. BART 220, T3
38 GPT-3 [5]. and GPT-4 33 These models have been successfully applied to a wide range of tasks, including sentiment analysis, question answering. machine translation, and text generation, to name a few. An emergent behavior of these LLMs is their ability Iearn from context, a phenomenon often referred to as in-context learning. In LLMs such as GPT-3 [ the in-context learning ability allows them to perform a task just by conditioning on the combination of input-output examples and a new query input, a.k.a. prompts, without optimizing any model parameters. With proper design of the prompt structure and in-context learning, LLMs can unite the pre-training of multiple language tasks and generalize well to previously unseen tasks.
While in-context learning has been extensively studied in NLP, its applications in the field of computer vision are still limited. To showcase the viability and potential of in-context learning as a standard approach for large-scale vision applications, two significant challenges need to be addressed: I) Designing an effective vision prompt, which incorporates both domain-specific input-output pairs as examples and image queries as conditions, is more challenging than designing prompts for language tasks. 2) In computer vision, large models are typically trained for specific tasks such segmentation 22]. detection 66, classification 220), self-supervised representation learning [37], class plass-conditional generation [10, and text-to-image generation [41 67 (1]. As a result, these large vision models are not designed for in-context learning and lack the flexibility to adapt to new tasks. Several recent
attempts [3 56 57 tackle these difficulties by following the solutions in NLP. More specifically, simple visual prompt is built by stitching example images, query images, and output images into one large image, and then a Transformer-basee image inpainting model is trained to predict the masked output images 33 (56). Stitching to large images, however, will dramatically increase the computational cost, especially in high-resolution cases.
Tackling these two challenges, this paper aims to unlock the in-context learning ability of text-guided diffusion-based generative models. We introduce a new model architecture, Prompt Diffusion, to perform in-context learning under a vision-language prompt that could accommodate a wide variety of vision- language tasks. We train Prompt Diffusion jointly on six different vision-langued tasks. Specifically, we initially establish a general vision-language task by employing our vision- language prompt:
prompt: (lext-guidance, example: (imagel - image2), image-query: image3) - target: image4,
where (imagel + image2) consists of a pair of vision task examples, e.g., (depth map + image), lext-guidance provides language instructions for specific tasks, and image3 is the input image query that aligns type with imagel and hence could be a real image or an image condition (e.g., depth or hed map). We subsequently develop Prompt Diffusion, drawing inspiration from the design of Stable Diffusion [69] and ControlNet [41), which can take our Dision-language prompt as the input. We finetune Prompt Diffusson from Stable Diffusion [69] checkpoints six different vision-language tasks, concurrently and uniformly, including three forward tasks: image segmentation/hed//a map generation, and three inverse tasks: image generation given segmenntaioon/hedddephh maps. We illustrate Prompt Diffusion through examples in Figure 2
Prompt Diffusion successfully integrates the learning of six different tasks into one vision-language foundation model. Through prompt-based learning, the model can effectively grasp the underlying relationship between the input example pairs. The model can then use this understanding to generate the output image by re-mapping the relationship onto the query image and incorporating the language instructions, as illustrated in Figures 2 and 4 More importantly, the model acquires in-context learning ability by learning across multiple tasks. Remarkably, we observe that Prompt Diffusion not only performs well on the six tasks it has seen during training, but also generalizes effectively across several new unseen tasks, as shown in Figure
Overall, we propose Prompt Diffusion as an initial attempt to unlock the in-context learning ability of text-guided diffusion models. Empirically, we observe that Prompt Diffusion has a promising in-context learning ability on both trained and new, unseen tasks. The success achieved by Prompt Diffusion is to inspire and spur additional research in the field of diffusion- in-context visual learning. We summarize our main contributions as follows:

---

surple
Aepii
Diffuiion
Mull-Tra Tuaed
Inverse Tasks
Forward Tasks
Figure 2: Illustration of Prompt Diffusion trained jointly on six different vision-langues tasks. Each gray- dashed box represents task example: prompt--outpul. The output random generation from trained model given a rision-languaa prompt, where the query image aligns in type with the source image from example.
A novel vision-language prompt design that well supports the integration of various vision- language tasks.
The Prompt Diffusion model, the first diffusion- based versatile vision-language foundation model capable of in-context learning.
Demonstration of high-quality in-context generation both trained tasks and new, unseen tasks.
2
Related Work
2.1 Diffusion Models
Recent advances in diffusson models have exhibited overwhelming success in generative applications. Diffusion models [47 49 17], serving as sophisticated generative models, generate intriguing examples through step-wise denoising process. These models utilize forward process that incorporates noise into data distributions and reverses this process to reconstruct the original data. are focused on scaling the diffusion models up
Subsequently, numerous studies [10] 31 39
60 71 to achieve better performance. In particular, LDM [1 is prominent model that reduces computational costs by applying the diffusion process to low-resolution latent space, successfully scaling text-to-image generation to web-scale data. Lately, Versatile Diffusion [67] expands the existing single-flow diffusion pipeline into a multi-task multimodal network that handles multiple flows text-to-image, image-to-tees and variations in unified model. Many other pieces of rescarch also extend the diffusion framework to various domains, including stable GAN training (5),, music generation [18), text-to-31) (34] 2
or improving the training and sampling efficiency [48
26
language generation (24), novel-view synthesis [66], uncertainty quantification (13), reinforcement learning [58), etc. Besides generation, recent studies [8 69 13 30
12 19 also explore the potential of diffusion models in image and video editing-relane applications.
2.2 In-Context Learning
Recent advances in large language models (LLMs) have made substantial impact on multi-modal modeling. Despite this, many of these models 88 23 25 27 46, 50
55 53 still necessitate
fine-tuning for specific downstream tasks. By contrast, in-context learning, introduced in generative pretrained transformers (GPT-1/2/3/4 B5 rapidly adapt to new tasks using only a handful of prompts. Rather than finetuning on downstream tasks, GPT formulates various language tasks predicting the future tokens given the context, exhibiting remarkable emergent abilities [62] on unseen tasks without explicit training 22 42 63 77)
novel paradigm 33 ) enables models to
36 5
Lately, researchers have been working on in-context learning frameworks that encompass computer vision tasks as well. For instance, Alayrac et.l. propose Flamingo that combines pretrained vision- only and language-only models and employs paired image-text examples prompts for few-shot

---

learning, Bar et.al. [3] introduce universal inpainting model to tackle multiple vision tasks by taking the desired output as a masked region and predicting based on given examples. Inspired by this approach, Painter 56 further incorporates more tasks to construct universal task solver, observing emergent abilities on unseen categories if evaluated on in-domain tasks. Subsequently, SegGPT [57] unifies multiple segmentation tasks into generalist framework and investigates both spatial ensemble and feature ensemble methods for creating prompt examples. Different from previous works, our work is primarily focused on exploring the in-context learning ability of diffusion models.
2.3 Prompt Design for Visual Language Models
The idea [ 36] of transforming various language tasks into a generative problem has motivated a lot of following works to explore how to formulate appropriate generative conditions, which is also known as prompt engineering. Designing different prompts is observed to result in a huge performance gap in many language tasks [32 63]. Prompt designing also raises much attention in vision areas after the breakthroughs of vision- language models (e.g., CLIP B7). Instead of directly predicting one-hot label-assignment vector using classifier, CLIP can be used as classifier by using a correct prompt template, such as "A photo of a [label]". following work [72] observes that changing the text template itself matters for the performance of different downstream vision tasks. Other works further propose improving strategies by adding red circle on the interesting areas of given images [45]. coloring the interesting image patches [68]. While proposed method also explores vision-language prompt engineering, we further introduce a new interface in the vision backbone that helps take and understand the image prompts.
3 Prompt Diffusion and In-Context Learning
In this section, we describe how to perform vision-language prompting on diffusion-basee text-to- image generative models. First, we introduce novel vision-language prompt design in Section 3.1 Then, we show our model design of Prompt Diffusion in Section 3.2 Finally, we demonstrate how we represent different vision-language tasks via our prompting and how we jointly train Prompt Diffusion on multiple tasks in Section 3.3
3.1 Vision-Language Prompt Design
In LLMs, common way of mask-prompting for a specific language understanding task is provide the trained model with pairs of examples the target task and the input query [5]. For example, in machine translation, Englishhto-Frenchh Brown al. [5] construct the prompt /example: otter + loutre de mer, query: cheese >> J. and the model will learn from the example in the prompt to translate the query 'cheese' into 'fromage'.
Following the convention, we design the vision-language prompt by replacing the text examples with paired image examples and the text query with image query. We further add one more text input to allow text-gun eot--giddd image generation. By this design, we propose a new input-output pair format that could generalize the input-output configuration of most vision-langue tasks. We show it as follows together with one example:
prompt: fexi-nnid example: (imagel image2), image-query: image3)
target: image4,
Prompt
test guidonce
Imnoged
image2
Imogee3
imaged
Tonderape Art Prine
style of comic book
fcotwing the pointing Forgorten Rood by Front Whboo"
where the exmaple: (imagel > image2) informs the model what the target task is and how it could be done through examples, c.g., image generation from the hed map, rext-guidance guides the model to generate an image conditioning on the given text, and the image-query: image3, which aligns in type with imagel, represents the input to the specific task. The exumple pair could accommodate any image domain transformation tasks, e.g., forward tasks: image + segmentaaioon/deethhhee map, and inverse tasks: segmentation/deepthhhee map - image, while the involvement of text-guidance provides additional context to generate the target images. Next, we present Prompt Diffusion, which

---

Lakeni Noae
Prompt
ExErppp
Acarrpa Tirpet
Lo Ouayy
Text:
Text:
'LandioppAAr
"arddcass
Stable biftesinn
Stable Difusionn
Encoder
Encoder
ControlNet
Text Encoder
Encoder
Conv
Cory
ControlNet
Decader
Decoder
Densking Cydle
Donwiing Cydle
Decoder
Controlieet
Prompt Diffusion
Figure 3:
Comparison of the model architectures of ControlNet (Left) and Prompt Diffusson (Right). More details on the network structures of Prompt Diffusion-induced modules can found Appendix
diffusion felfussion-based generative model that takes the vision-language prompts inputs and outputs the target images.
3.2 Prompt Diffusion
Motivated by the success of Stable Diffusion 41 and ControlNet [69] on text-to-image and image conditional generation, we build Prompt Diffusion based on the design of ControlNet. Our vision- language prompt is multimodal, incorporating both text and image inputs. For image inputs, concatenate the example pair of images in the channel dimension, and then project the concatenated example pair and image query into equivalent dimensional embeddings via independent stacked convolutional layers. We compute the sum of the two embeddings and feed it into the ControlNet branch. For the text input, we follow the convention of Stable Diffusion 41 and encode the text input through the pre-trained CLIP 37 text-encoder. The resulting CLIP text embedding is then fed into the Stable Diffusion branch via cross-attention layers. We illustrate Prompt Diffusion Figure
We keep the Stable Diffusion branch and ControlNet branch same as their original versions, so that could finetune Prompt Diffusion from existing checkpoint without the need to train it from scratch. We finetune Prompt Diffusion from the Stable Diffusion "v1.5' checkpoint.
3.3 In-Context Learning
We consider six different vision tasks in joint training. We call them forward tasks when the task inputs are clean images and inverse tasks when the task inputs are image conditions, e.g., segmentation//lephee maps, depicted in Figure 2
Datasets. We use the public dataset proposed by Brooks et al. as our base dataset, which consists of around 310k image-caption pairs. Furthermore, apply the Midas [40] to obtain the depth and normal maps of all images in our dataset. We obtain the image segmentation maps by applying Uniformer [22] We collect the canny edge maps by the Canny edge detector (6) and hed
maps by the HED boundary detector [64]. Note the depth, hed, and segmentation maps are all used for our model training, while the canny edge and normal maps are only used in the testing time for
evaluating the in-context learning ability of Prompt Diffusion on new, unseen tasks.
Inverse Tasks. We consider three inverse tasks: depth maps to images, hed maps to images, and segmentation maps to images. For any image-caption pair in the dataset, (I,. C1). we first sample random task and another random image /2 create the example pair, e.g., (HED(I2), I2). Then, we build the vision-language prompt with the example pair, the caption, and the image condition that is consistent with the task specified by the example pair. The image 7 the denoising target of Prompt Diffusion. One complete example of the inverse-hed task is shown as follows:
5

---

prompt: flex-guuddacc: C₁ exumple puir: (HED((s) - Iz], image-query: H) - target: Zi.
By replacing the hed maps with the other two image conditions, we could obtain the vision- vision-language prompts for all the three inverse tasks.
Forward Tasks. We also consider three forward tasks (image processing tasks): images to depth maps, images hed maps, and images to segmentation maps. We follow the same rule of inverse tasks to create the vision-language prompts. Note here the tasks are flipped, so we flip the order inside both the example pair and the query-target pair. The captions of images are not necessary for image processing tasks, so we use task-specific fixed text label for each task, such as 'hed maps'. We show one complete example of the forward-hed task as follows:
prompt: /text-guidance: 'hed maps', example pair: [12 - HED(Ia)], image-query: 1/ target: H
Joint Training. We train Prompt Diffusion jointly on these six different vision-language tasks. For simplicity, train model on these six tasks uniformly at random. Specifically, each minibatch data contains randomly sampled vision-language prompts and the corresponding target images from randomly selected tasks. Joint training over these different tasks unlocks the in-context learning ability of diffusion models, as shown in Figures 3 and [S] In order to apply classifier-fre guidance (CFG) [6], randomly drop 10% text guidance during training.
Experiments
In this section, conduct extensive experiments to illustrate the power of Prompt Diffusion strong versatile vision-language foundation model that is capable of in-context learning. We first show that Prompt Diffusion performs well with multi-task training in Section 4.1 Then, we show that Prompt Diffusion has promising in-context learning ability and could generalize well to new, unseen tasks in Section
4.2 We show that Prompt Diffusion supports controllable and text-guided Finally, we conduct an ablation study on the three main components of our vision- language prompt in Appendix A
image editing in Section 4.3
Implementations. We implement Prompt Diffusion upon the codebase of ControlNet [69] As illustrated in Figure 3 we implement both the visual-example caira encoder and image-query encoder via stacked convolution layers. See the details of these layers in Appendix C We load the weights from Stable Diffusion checkpoint 'v1.5' for finetuning. We fix the learning rate at 1 x 10 and used and how augment the original datasets are described in detail in Section 33.3 We train our model on 8x A 100 Nvidia GPUs for 5000-20000 steps. Longer training helps the model perform better the finetuned dataset while may perform worse on out-of of-distribttion images. The results shown in the paper are based on the 5000-step finetuned checkpoint.
accumulate gradients every 4 mini-batches with batch size 512. The datasets
Evaluation. We conduct both qualitative and quantitative evaluations to assess the performance of Prompt Diffuusion We provide comprehensive qualitative evaluations to measure Prompt Diffusion's in-context learning capabilities. To assess generation quality, we provide quantitative results measured using the zero-shot Fr' echet Inception Distance (FID) for inverse tasks and Root Mean Square Error (RMSE) for forward tasks, detailed in Section 4.4 The test dataset is composed of random images from the ControlNet [69] example images, DreamBooth (43) data benchmark, and the test split of our base dataset 4.. During the inference stage, one additional random image from our dataset is used to construct the example pair for each specific task.
4.1 Multi-Task& Learning
We jointly train Prompt Diffusion on three inverse tasks, including inverting deptth/heddseemmentttion and three forward tasks, including extracting depthhhed/segmmentttoi maps. Note ControlNet [69] aims to enhance pretrained image diffusion models with task-specific conditions (e.g., depth maps). We finetune a ControlNet, from the same Stable Diffusion checkpoint, our dataset for each inverse task independently as our baseline, and we refer to it as CN(FT). We qualitatively evaluate Prompt Diffusion on the six trained tasks in Figure 4

---

Vegil mnd Trogroe
"aag
Onpthvlto-magee
Imageelodepph
Hed perl-to-lmage
Image to Hod
Sog to Image
Image to Seg
Inverse Tasks
Forward Tasks
Figure 4: Qualitative Results of forward tasks and inverse tasks. We show examples of applying our Prompt
Diffusion on the six trained tasks finy-eepthhhcd/seg and forward wewardddepthhhed/seg).
We observe that Prompt Diffusion could not only generate high-fidelity images based different task-specific conditions for the inverse tasks, but also generate detailed and rational image conditions if it is given the real images for the forward tasks. For inverse task, when the image conditions already sketch the main components of the output images, such as depth and hed maps, Prompt Diffusion could successfully maintain the shape consistency in the output images and generate diverse images with different random seeds and text guidance. When the image conditions are coarse like segmentation maps, Prompt Diffusion could generate personalized hair style in the third block). For forward tasks, we empirically find Prompt Diffusion generally produces even more detailed depth and hed maps compared to these maps in the original dataset. The produced segmentation maps are comparable to the ones used as the targets during finetuning.
imaginary images (e.g-, the fairy with
Note ControlNet 699 is designed for image generation with image conditions. Hence, we compare Prompt Diffusion with CN(FT) on the three inverse tasks in Figure [f the Appendix. The results show that Prompt Diffusion performs comparably well to CN(FT) which is trained specifically for each individual task. We further evaluate the generation ability of CN(FT) in Figure T1 of the Appendix. We directly infer CN(FT) on new tasks and observe a large number of failures. This validates the necessity of multi-task training for Prompt Diffusion achieve the capability to generalize across a wide variety of tasks.
4.2 In-Context Learning Ability
We evaluate Prompt Diffusion on tasks that are outside of our supervised training dataset further study its generalization ability. We first select three unseen tasks, inv-scribble (image generation from personalized scribbles), inv-edge (image generation from canny-edge maps), and inv-normal (image generation from normal maps). The model is given pair of image examples that illustrates how to perform a desired task, and asked to re-conduct the task onto new image query inputs with text guidance.
7

---

Ino Scribbie
Inu Edge
Inv Narmal
Figure 5: Generalization to New Tasks. We show Prompt Diffusion has promising generalization ability to new, tasks, such Inv-Seribble, Inv-Edge Edgen) and Inv-Normal (Normal map).
Example Pair: Style Transfer
Image Query
Text
Generated (two examples)
"A high quality Image"
Image"
Example Pair: Misalignment
Image Query
Text
Generated (two examples)
"henry_ john ye end _351 9_the old. mill _ssmal"
"henry, John, ye end Kingkkig__l 9. t_ old, mill $smil"
Figure 6: We perform Prompt Diffusion on two completely tasks: style transfer and misaligned input-output example pairs. The first two rows show the style transfer results while the last show the misaligment results.
We show the qualitative results in Figure 5 Prompt Diffusion successfully learns the underlying correlation between the example pairs and re-maps the relationship onto query and output pairs, without the need training or finetuning on the three new domains. The image generation is also well controlled by text guidance as shown in the results.
In Figure we further evaluate Prompt Diffusion on two distinct tasks: style transfer and misaligned input-output example pairs. For the first task, as shown in the initial two rows, Prompt Diffusion is given a pair of image style transfer examples. new target image, accompanied by general text description, such as "A high quality image." The findings indicate that Prompt Diffusion not only capable of discerning the underlying task from the visual example pairs but can also adeptly carry it out fresh queries. For the second task, detailed in
then tasked with applying this style transfer to a

---

the final two rows, we investigate the impact of pixel alignment in the example pairs. Even when various regions in the example pair are concealed, resulting in pixel misalignment, Prompt Diffusion manages to execute the intended task proficiently without any significant disruption.
4.3 Image Editing
Depth image
Sng Image
Ploeggar Jan
Oigil Weur
One. One.Step
[Image Depth)+ Image
jinage Sepmentation] Image
[Image Hed]
Image
Two-Step
Thas Ait
Figure 7: Image Editing Results. We demonstrate that Prompt Diffusion excels in image editing tasks through two strategies: one-step and two-step. One-step: when only the image condition, e.g., depth/seg/hed maps, given, our model would edit the condition for generation according to text descriptions. Two-Step: when the original image is given, we first do forward sampling to sample image conditions and then conduct inverse tasks to edit the condition with text guidance. Two-Step provides more controllable editing.
We show that Prompt Diffusion has strong image editing ability in two ways. First, only the conditions of images are given, e.g., depth/segmmenatiionneed maps, Prompt Diffusion is able to synthesize images based on the input image conditions and additional text descriptions. Second, one base image is given, we could apply a two-step strategy to achieve controllable image editing: We first generate the image conditions and then generate edited images based on both the generated conditions and text descriptions, all within one model: Prompt Diffusion. As shown in Figure 7 with the help of image conditions, Prompt Diffusion capable of performing consistent modifications specified by the text guidance input without worrying about undesired excessive changes.
4.4 Quantitative Results
We compare Prompt Diffusion with CN(FT) quantitatively. For inverse tasks, measure the shot FID on the test split of our base dataset 4 We use the corresponding image condition, text input, and example pair as the input for either Prompt Diffusion or CN(FT), and generate 10k random images where each input produces one image. We then measure the FID using the 10k generated samples with the corresponding 10k test images used reference. For forward tasks, we measure the zero-shot RMSE between the generated image conditions (e.g., depth/hhedsegmmentaiee the corresponding ones in the test set, where all image conditions are rescaled to [0, 1] for computing the RMSE. Note for both Prompt Diffusion and CN(FT), we finetune them with the same number of 5000 steps. We show the quantitative comparison in Table T We observe that Prompt Diffusion has comparable or even better performance compared to CN(FT).
maps) and
5 Conclusion and Discussion
In this study, we introduce Prompt Diffusion, novel in-context visual foundation model that leverages diffusion-based techniques to generate images from vision-language prompts. Our model is designed be flexible and compatible with various vision-language tasks, allowing it to accommodate multiple use cases. We train Prompt Diffusion six different vision- laiiual tasks and conduct

---

Table I: Zero-Shot Quantitative Results. for forward tasks on our test dataset.
We provide the FID comparison for inverse tasks and RMSE comparison
FID (Inverse Tasks)
RMSE J. (Forward Tasks)
Methods
Depth-to-Imag -mage
Hed-to-Image
Seg-to-Image
Image-ta-Depth
Imuge-to-Hed
Image-to-Seg
CN((T) 69
0.20
Prompt Diffusion (ours)
18.60
13.35
19.46
0.21
0.14
0.31
Image query
supp
We present some failure cases of Prompt Diffusion when generalizing to new, unseen tasks under ambiguous text guidance.
Figure 8:
extensive experiments to evaluate its performance. Our results demonstrate that Prompt Diffusion has promising generalization ability, not only on trained tasks but also unseen ones. Additionally, Prompt Diffusion enables controllable image editing, including style, artistic medium, and contextual changes, without introducing excessive undesired modifications.
While our model has demonstrated notable successes in in-context learning, it important to acknowledge its limitations. Currently, the visual scope of our model is restricted by the limited number of real-life images in the training dataset used for finetuning. As our base dataset ( is synthesized by Stable Diffusion with proper filtering, it is difficult for the model to generate high- fidelity, real-life images. Moreover, our model is only trained jointly across six pre-defined tasks, and we believe that expanding the range of joint tasks would enhance Prompt Diffusion's 's in-context learning capability. As shown in the last row of Figure 9 it is difficult to predict the behavior of model when the query image is misaligned in type with the source image in the example pair, and as shown in Figure 8 there are instances where our model struggles with new tasks. Prompt Diffusion is also limited when it comes executing physical transformation tasks like rotation and flipping without additional fine-tuning specific to these tasks. To unlock more of Prompt Diffusion's 's in-context learning ability, it is possible to change the finetuning setting to training from scratch, but this would require significant computational resources, which presently limits its feasibility for us.
Prompt Diffusion represents pioneering effort in unlocking the in-context learning capacity of diffusion- brauud models. We hope that this work will encourage more researchers to explore the po- tential of eeffusion-based in-context learning and contribute to the advancement of this exciting field.
Acknowledgments
Z. Wang and M. Zhou acknowledge the support of NSF-II 2212418, NIH-R37 CA271186, and the NSF Al Institute for Foundations of Machine Learning (IFML).
References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Lue, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35: 23716-23736, 2022.
[2] Mohammadreza Armandpour, Huangjie Zheng, Ali Sadeghian, Amir Sadeghian, and Mingyuan Zhou. Re-imagine the negative prompt algorithm: Transform 2d diffusion into 3d, alleviate janus problem and beyond. arXiv preprint arXiv:2304.04996 2023.
10

---

[3] Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image impainting. Advances Neural Information Processing Systems, 35: 25005-25017, 2022.
[4] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:2211.098000 2022.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[6] John Canny. A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelligence, (6):679-69)) 1986.
[7] Ting Chen, Saurabh Saxena, Lala Li, Tsung- Yi Lin, David J Fleet, and Geoffrey Hinton. unified sequence interface for vision tasks. Advances in Neural Information Processing Systems, 2022.
[8] Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX, pages 104-124 120. Springer, 2020.
[9] Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810004805. 2018.
[10] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
[11] Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. Violet: End-to-end -anguage transformers with masked visual-token modeling. arXiv preprint arXiv:2111 12681, 2021.
[12] Vidit Goel, Elia Peruzzo, Yifan Jiang, Dejia Xu, Nicu Sebe, Trevor Darrell, Zhangyang Wang, and Humphrey Shi. Pair-diffusion: Object-level image editing with structure-andaappen paired diffusion models. arXiv preprint arXiv:2303. 1756,, 2023.
[13] Xizewen Han, Huangjie Zheng, and Mingyuan Zhou, CARD: Classification and regression diffusion models. In Advances in Neural Information Processing Systems, 2022.
[14] Lisa Anne Hendricks, John Mellor, Rosalia Schneider, Jean- Caan-Beptiste Alayrac, and Aida Ne- matzadeh. Decoupling the role of data, attention, and losses in multimodal transformers. Transactions of the Association for Computational Linguistics, 9:570-585 2021.
[15] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to toopp-tt-promptt image editing with cross attention control. arXiv preprint 228.0:2208.01666, 2022.
[16] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022.
[17] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[18) Qingqing Huang, Daniel S Park, Tao Wang, Timo Denk, Andy Ly, Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Frank, et al. Noise2music: Text-conditioned music generation with diffusion models. arXiv preprint a02hh:2302..33117 2023.
[19] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel. Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. arXiv preprint arXiv:2303. /339,, 2023.
[20] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6): 2017.
11

---

[21] Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. BART: Denoising pauuuence-t--sqqenee pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910. 1 13461, 2019.
[22] Kunchang Li, Yali Wang, Junhao Zhang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unifying convolution and self-attention for visual recognition. arXiv preprint arXiv: 094400 2022.
[23] Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. Hero: Hi- erarchical encoder for video+ language omni-representation pre-training. arXiv preprint arXiv:2005.002000 2020.
[24] Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B Hashimoto. Diffusion-Im improves controllable text generation. arXiv preprint arXiv:2205. 14217, 2022.
[25] Xiujun Li, Xi Yin. Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang. Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer RsiionECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16, pages 121-137. Springer, 2020.
[26] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206. 009277 2022.
[27] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for siiion-ndd-anguuee tasks. Advances in neural information processing systems, 32, 2019.
[28] Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. CoRR, abs/2108.01073, 2021. URI. /arxivv.vvvv wwtpeiivv.rrgoooooooeeoo abs/2b/s 2208 01073
[29] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202./28877 2022.
[30] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or Or. Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv:2211.09790 2022.
[31] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2 10741, 2021.
[32] Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bicber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021.
[33] OpenAl. OpenAI GPT-4. httts:://ppee:ttt www.iaaaaaoommmaaaaaaaaaeeeeeeee www/gtt.tt.... 4/ 2022.
[34] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14990 2022.
[35] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training, 2018
[36] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenA/ blog, 18):9, 2019.
[37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages PMLR, 2021.
12

---

[38]
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-61 2020. URL
http://jmlr... http://jmlorrrrpppaaee pttppapprs/v21/2000
html
[39] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint (xXiv:2204.06/22. 2022.
[40] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions pattern analysis and machine intelligence, 44(3):1623 1637, 2020.
[41] Robin Rombach, Andreas Blatmmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[42] Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. arXiv preprint arXiv:2112. 08633, 2021.
[43] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208. 12242, 2022.
[44] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36449 2022.
[45] Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about a red circle? visual prompt engineering for vims. arXiv preprint 2:X1v:230..06712, 2023.
[46] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- lion, pages 15638-15650, 2022.
[47] Jascha vised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.
Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper-
[48] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500. 2020.
[49] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[50] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre-training of generic visual- -inguisin representations. arXiv preprint 1908.0998..08. 2019.
[51] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7464-7473, 2019.
[52] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908. 07490,, 2019.
[53] Korawat Tanwisuth, Shujian Zhang, Huangjie Zheng, Pengcheng He, and Mingyuan Zhou. POUF: Prompt-oriented unsupervised fine-tuning for large pre-trained models. arXiv preprint 1rXhv:2305.00350, 2023.
[54] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
13

---

[55] Jianfeng Wang, Xiaowei Hu, Zhe Gan, Zhengyuan Yang, Xiyang Dai, Zicheng Liu, Yumao Lu, and Lijuan Wang. Ufo: A unified transformer for vision-language representation learning. arXiv preprini arXiv:2111.1002. 2021.
[56] Xinlong Wang, Wen Wang. Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. arXiv preprint arXiv:2212.02499, 2022.
[57] Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting everything in context. arXiv preprint arXiv:2304.03224, 2023.
[58] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint S2Kiv:2208.061993 2022.
[59] Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. DiffusionGAAN: Training GANs with diffusion. arXiv preprint arXiv:2206.02262 2022.
[60] Zhendong Wang. Yifan Jiang, Huangjie Zheng. Peihao Wang, Pengcheng He, Zhangyang Wang, Weizhu Chen, and Mingyuan Zhou. Patch diffusion: Faster and more data-efficient training of diffusion models. arXiv preprint
2023.
[61] Zhixin Wang, Ziying Zhang, Xiaoyun Zhang, Huangjje Zheng, Mingyuan Zhou, Ya Zhang, and Yanfeng Wang. DR2: Diffusion-based robust degradation remover for blind face restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 1713-1713, 2023.
[62] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. arXiv preprini arXiv::2000 07682, 2022.
[63] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou, Chain of thought prompting elicits reasoning large language models. arXiv preprint 2rXIv:2201.11903, 2022.
[64] Saining Xie and Zhuowen Tu. Holistically-nester edge detection. In Proceedings of the IEEE international conference on computer vision, pages 1395-1403, 2015.
[65] Sang Michael Xie, Aditi Raghunathan, Percy Liang and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02088 2021.
[66] Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Yi Wang, and Zhangyang Wang. Neurallift- 360: Lifting an in-the-wild 2d photo to 3d object with 360" views. arXiv e-prints, pages arXiv-2211, 2022.
[67] Xingqian Xu, Zhangyang Wang, Eric Zhang, Kai Wang, and Humphrey Shi. Versatile diffusion: Text, images and variations all in one diffusion model. arXiv preprint arXiv:2211.08332,, 2022.
[68] Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Cpt: Colorful prompt tuning for pre-trained vision-language models. arXiv preprint arXiv:2109.1177 2021.
[69] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.055443 2023.
[70] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu, What makes good examples for visual in-context learning? arXiv preprint arXiv:2301.13677 2023.
[71] Huangjic Zheng, Zhendong Wang, Jianbo Yuan, Guanghan Ning, Pengcheng He, Quanzeng You, Hongxia Yang, and Mingyuan Zhou. Learning stackable and skippable lego bricks for efficient, reconfigurable, and mariable-rsoouution diffusion modeling. arXiv preprint arXiv:2310.06388, 2023.
[72] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for lasione models. International Journal of Computer Vision, 233-9223377- 2022.
14

---

Appendix
A
Ablation Study
We conduct ablation study to investigate the effects of vision-language prompt, which consists of the example pair, text guidance, and image query. We show the generation results Figure In the first row, keep the text guidance and image query fixed, while we use different random images to construct the example pairs. The results suggest that Prompt Diffusion is not sensitive to what images used to construct the example pairs, while it cares about the underlying correlation between the images inside the example pairs. In the second row, compared to the first row, we only modify the image query part, and we could see the output images are dramatically changed but consistent with the new image queries. In the third row, apart from the first row, we modify the text guidance and observe the output images are modified accordingly. In the last row, we investigate the impact of using an image query that is intentionally set to be misaligned in type with the source image of the example image pair. Our results show that such cases, the model's behavior often becomes difficult to predict. Overall, with our vision-las languae prompt where the image query is aligned with the source image in the example image pair, the example pair tells the model what the specific task is, the image query provides the image domain condition for the generation, and the text guidance allows diverse semantic modifications.
THE
Migg Cars
Odell
kag Dar
Sepd
boc tocksoc the bwcn
the beach
barkpeck the Ansch
basipack on the berch
beckseot the
backpack on the deach
boach
logo dockpack
Dasipack on the beach
Figure 9: Ablation study for different variations of the vision-language prompt in Prompt Diffusion. The last row shows when intentionally mismatching the type of the query image and that of the source image in the example pair, the hehavior of Prompt Diffusion becomes difficltt predict.
B Comparison to ControlNet
We qualitatively compare Prompt Diffusion with ControlNet 169 We follow the guidance of ControlNet, finetune a ControlNet, from the same Stable Diffusion checkpoint, on our dataset for each inverse task independently
our baseline, and we call it CN(FT). We show the comparison results in Figure 10 We observe the jointly trained Prompt Diffusion performs comparably well to the independently trained ControlNet, which implies significant performance change with multi-task learning applied.
We directly apply the independently finetuned ControlNet on one new during inference. We could observe large number of failures when directly applying ControlNet to new, unseen tasks.
We further evaluate the generation ability of the task-specific ControlNet in Figure II
15

---

meplas Paay souing Ae belcoss
Ohrs
Iie Degih Map
O2
Red
2
Oui
Inv Seg
Figure 10: We show the comparision of Prompt Diffusion with CN(FT), ControlNet that finetuned specifically for each individual task, on the three inverse tasks.
C Model Architecture
We build Prompt Diffusion based ControlNet [69]. We show the detailed model architecture in Figure 12 Note we lock the parameters of Stable Diffusion encoder blocks for finetuning to inherit the encoder capability that Stable Diffusion learns from large-scale pre-training. All the other parameters are open for finetuning. The Stable Diffusion branch takes images in the latent space inputs, while the ControlNet branch takes the original images inputs. The pretrained image encoder and decoder, which map images to latent space and re-map latent images to original images, not shown in the illustration. The latent images are in four channels and eight times smaller in both height and width.
As mentioned in the main paper, ControlNet branch takes three images as inputs. Two of them form the example pair, which is first concatenated the RGB channel and then encoded by stacked convolution layers. The third image represents the image query and is encoded with independent stacked convolution layers. We show the architecture of the stacked convolution layers in Figure T3 We encode the example pair and the image query the same dimensional latent embeddings and then sum them up as the inputs for the ControlNet branch.
D Potential Social Implications
The development of image generation models has the potential to have negative social impacts. These models can create realistic images that are difficult to distinguish from real photographs, which could lead to proliferation of fake images used for malicious purposes. For example, these models could be used create fake images of people committing crimes or engaging in immoral behavior, which could damage reputations and lead to false accusations. Additionally. the widespread use of image generation models could contribute the erosion of trust in visual evidence, which could have implications for legal proceedings, journalism, and other fields that rely on visual evidence. There also a risk that these models could perpetuate biases and stereotypes, as they may learn from biased datasets and generate images that reinforce those biases. It is therefore important to carefully consider the ethical implications of image generation models and take steps to mitigate their potential negative impacts.
16

---

Text
Capar
Diggrt
Seep Imagee
Inferesie
Figure We show that ControlNet finetuned for individual task on the dataset fails generate high-quality images for another task, which validates the necessity of multi-task training.
E More Examples
We show more exmaples of applying Prompt Diffusion here.
17

---

Prompt
Guidance
Noise
4
concat
Stadied Convolition
Starked Comelation
Encoder
Time
Encoder
Encoder Block 16xl6
16x16
50 Encoder Block-4 budi
90 Enooder
Block-4
5D Midale Block acl
SD Middle
Block Bull
30 Decader Block4 Bull
zoro vorvocciio X3
Decoder Block33
zero COTvoUciOn
Block
32x32
2e10
Stable
ControlNet Branch
Figure 12: We show the detailed model architecture of Prompt Diffusion.
Corv2dji_channete nout channels out channels-16, ternel stbe3, padding-1)
Activation Shil)
Conv2olinn charneis-6,, but cha kernel stee33, prene__nnee.. padding:1]
Activation Stüll
Convzolin charneis:16, out charnels:32 #20+3, paddingel, 2ride=2)
Activation S(UU)
Conv2d0nn charnes-32, charnes-32, bernel sime3, padsing=1)
SitlI)
Comzziinn charneis332 charnes--6.
SiUl)
Conzddjn charnas--fE, out charnel-066 kernel steedd, paddings))
Conv2din channes=66, out, c5an lermel siwe3, paddinget, stridee))
Stacked Convelution Layers
Figure 13: Stacked Convolution Layers.
18

---

"eater tee Dooil', chatopmeent by 2hshes
Suzy Apttpoattdd by Genger Gambarr
"say foverr stopopeed doreggs Dayek
imput
Depth inpux
Depch Ipput
"ANTO FATET Smacs AVEEAAg Aaleys!
"Tees Sot WoarAN anAAVAA
Teresr Angleeitteeei Siight
Depth oppu
Aame erthi 2000
Stew chiikin
Figure 14: More Depth-to-Image Examples of Prompt Diffusion.
19

---

Apesi nan waily whtx ootter weavae
Sepett pivtt curtans, parkg vaa
poek eooring while derterpeiiy tra
Hed input
Hed Ioput
Toa Kang
Sakayy
"On Freiig Hemoroonto with buttertie
Med
Aright Mininns kits Aftencce Gomsshaer
"Cesit of lone, with bage
Thary, Job WAw._HAK_JEEEAA. (ww.jNEAAAaloiiioooooooo Ph, rrec
MNL. mook aod rage
TWALJAN Arg. PALAN eal HND
Figure 15: More Hed-to- Examples of Prompt Diffusion.
20

---

Sog
Parsage on countaye
Seg
Rusiie man pao.g bocstt Alapp."
Me old Mors"
Sog
"lesshoo eosucol and coor taiend Micie
Hage tobbrg
Sog imput
TUTTONEN MOAWCC
PUUTHAVEES AANNON SYAWAN??
BOUTENNAEE NORWING COVEREO SNOW"
Figure 16: More Segate-Image Examples of Prompt Diffusion.
21


Reviews:
[{"review_id": "jhesnw2m7r", "reviewer": "Reviewer_foSi", "summary": "This work introduces Prompt Diffusion. It is a framework designed to facilitate in-context learning within diffusion-based generative models. \nBy providing a pair of task-specific example images, such as depth from/to image and scribble from/to image, along with textual guidance,  the designed solution can automatically comprehend the underlying task and replicate the same task on a new query image following the provided text instructions. \nSpecifically, the authors propose a vision-language prompt that can effectively model various vision-language tasks. The diffusion model, which takes the prompt as input, is trained in a joint manner on six distinct tasks utilizing these prompts.", "questions": "N.A.", "limitations": "Overall, this paper presents an interesting extension to previous work in the field. The clarity of the writing and the strong motivation make it a valuable contribution. However, a more detailed comparison with the prior work could strengthen the paper and provide more insights into the novelty and significance of the proposed approach.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "--The paper is well-written and easy to follow. \n--The authors have done an excellent job of clearly articulating the motivation behind their work. \n--The problem statement is well-defined and the objectives are clearly outlined.\n", "weaknesses": "--  The work appears to be an extension of a previous work titled \"A Generalist Painter for In-Context Visual Learning\". The design principles and motivations are similar, while there are notable differences that distinguish this work from the previous one.\n-- Differences from Prior Work: The authors have introduced text commands into prompts, which adds a new dimension to the model and potentially increases its versatility and applicability. Additionally, the foundation model used in this work is different from the one used in the prior work. This change could potentially lead to different performance characteristics and should be explored in more detail.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer foSi for providing positive feedback. We hereby provide a more detailed comparison between our work and Painter. \n\n- **Task.** Painter follows [1] and focuses on solving high-level discriminative and low-level processing tasks, such as semantic segmentation, instance segmentation, depth estimation, keypoint detection, denoising, deraining, and light enhancement. Our Prompt Diffusion model, instead, further explores condition-guided image generations. \n- **Prompting.** Painter follows [1] and stitches example images, query images, and output images into one large image as the input prompt, which will dramatically increase the computational cost, especially in high-resolution cases. However, we use concatenation in the channel dimension and split the first stage encoding of example images and query images via two independent stacked convolutional layers. Note our model could generate high-resolution, photorealistic images, such as the 512x512 images shown in the paper. \n- **Model Architecture.** Painter follows [1] to formulate the problem as a demasking problem and a Transformer-based image inpainting model (ViT) is trained to predict the masked output tokens. Our work formulated the problem as an image generation problem and the state-of-the-art diffusion model, Stable Diffusion, is our backbone model for finetuning.  \n- **Modality.** Painter only allows image-visual prompts, while our work supports multi-modal vision-language prompts. Quoting your comment, the external dimension brings more versatility and applicability to our model. \n\nDrawing upon the aforementioned statements, we point out that Prompt Diffusion explores a different direction of in-context learning in visual models, instead of \"an extension of a previous work, Painter.\" We believe that both works offer distinct and promising viewpoints on in-context visual learning.\n\nReferences:\n\n[1] Bar, Amir, et al. \"Visual prompting via image inpainting.\" Advances in Neural Information Processing Systems 35 (2022): 25005-25017.\n"}, {"review_id": "MHH3LcaYnG", "reviewer": "Reviewer_7sFz", "summary": "Prompt Diffusion is a novel framework that enables in-context learning in diffusion-based generative models. It consists of a vision-language prompt and a diffusion backbone, which is trained jointly on six different tasks using their respective prompts. The resulting Prompt Diffusion model becomes the first diffusion-based vision-language model capable of in-context learning, demonstrating high-quality in-context generation for the trained tasks while generalizing to unseen vision tasks.", "questions": "Please refer to the weaknesses.", "limitations": "Yes.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1) The proposed vision-language prompt design is novel, which replaces texts with paired images and replaces the text query with an image query. This design allows for a new input-output pair format that is able to generalize the input-output configuration for most vision-language tasks. \n\n2) The Prompt Diffusion model is the first diffusion-based versatile vision-language foundation model capable of in-context learning. The model acquires in-context learning ability by multi-task learning pre-training, so as to generalize “zero-shot” to additional new tasks with no further adaptation\n\n3) The authors conducted extensive experiments to illustrate their capablity of in-context learning. An ablation study is conducted to investigate the effects of vision-language prompt, which further supports the effectiveness of the proposed design.\n\n4) The model also yields compelling text-guided image editing results, which demonstrate the controllability of the model in generating images based on text guidance.", "weaknesses": "1) I would be cautious to call the current model to have \" strong in-context learning ability\", since the unseen tasks tested in this paper are still similar to the known ones. For example, \"HED2Image\" task versus \"Canny2Image\" and \"Scribble2Image\" tasks. The paper would be strengthened greatly if more diverse unseen tasks could be learned in context.\n\n2) For text encoding, the authors used a pre-trained CLIP. I wonder that if replace CLIP with an LLM encoder module, some of which have its own in-context learning ability in language domain, will enhancethe in-context learning ability?  This is explored by a rencent work: \"LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models\". \n\n3) The current framework follows instructive pix2pix in most cases and the implementation leverages ControlNet straightforwardly. However, this is not my major concern due to the unconventionality and novelty of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 7sFz for evaluating the significance of our work and providing encouraging feedback. We provide more clarifications below. \n\n- We commit to exercising greater prudence in claiming the ability of in-context learning in our revised manuscript.\n\n- Appreciate your insight. Indeed, integrating stronger text encoding models, such as LLMs, might potentially enhance the in-context learning ability of our model. Currently, we are actively investigating the application of LLMs within our framework. Thanks for pointing out the recent work. We acknowledge this noteworthy work and intend to cite it while discussing its potential future implementation in our next revision.\n\n- Our data originates from the IP2P dataset, and we acknowledge that employing a larger and more diverse in-context learning dataset could further augment the capabilities of Prompt-Diffusion.\n\n"}, {"review_id": "zIKAvox3Mm", "reviewer": "Reviewer_vt4T", "summary": "The paper proposes a novel prompt design and model called Prompt Diffusion, for in-context learning in vision-language tasks. The vision-language prompt is designed by replacing text examples with paired image examples and the text query with an image query. The paper conducts extensive experiments to demonstrate Prompt Diffusion as a strong versatile vision-language foundation model that is capable of in-context learning. ", "questions": "See above ", "limitations": "The authors have thoroughly discussed limitations in Section 5. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "This paper pioneered a new and timely problem setting: extending in-context learning beyond LLMs, to text2image diffusion models. The proposed model, Prompt Diffusion, integrates the learning of multiple tasks into one vision-language foundation model, and it acquires in-context learning ability by learning across multiple tasks and generalizes effectively across new, unseen tasks. \n\nThe framework consists of two main components: a vision-language prompt and a diffusion model. The vision-language prompt replaces text examples with paired image examples and the text query with an image query, allowing for a new input-output pair format that could generalize the input-output configuration of most vision-language tasks. The implementation borrowed ideas from ControlNet. The diffusion model then takes the vision-language prompt as input and is trained jointly on six different tasks using these prompts. The resulting Prompt Diffusion model becomes the first diffusion-based foundation model capable of in-context learning. \n\nThe model demonstrates high-quality in-context generation for the trained tasks and effectively generalizes to new, unseen vision tasks using their respective prompts. It also demonstrates strong image editing ability. The proposed framework aims to facilitate research into in-context learning for computer vision.\n", "weaknesses": "While this pilot study is highly interesting, it is hard to assess if the diffusion model has the potential to achieve truly generalizable “in-context learning” as LLMs do. Currently, the model is only trained jointly across six pre-defined tasks, and those tasks are similar dense prediction type. The limited diversity of tasks evaluated in this paper constitutes an important hurdle to assess the work’s true value. \n\nIt is also hard to predict whether expanding the range of joint tasks would scale up Prompt Diffusion’s in-context learning capability.  Additionally, the model has poor robustness in handling misaligned query images (Figure 7). I am curious how the authors think this can be fixed: shall we expect to encompass more diverse training tasks, training data, etc?\n\nThe current generation results of high-fidelity, real-life images look underwhelming. It is noted that the authors used SD v1.5 backbone. I’ll be curious to see if in-context learning can work better if starting from SD v2.1.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer vt4T for acknowledging the innovativeness of our approach to the problem and appreciating the results of our experiments. Below, we address your questions and concerns:\n\n- We acknowledge the potential benefits of utilizing a larger and more diverse in-context learning dataset to strengthen Prompt-Diffusion. At present, our version is constrained by the available data, which incorporates as few as 6 tasks. \n\n- In Figure 7, we demonstrated that the generation quality is limited when the task is misaligned with our training tasks, and the input text lacks sufficient information. We believe that incorporating more tasks will render the model more adaptable to out-domain tasks.\n\n- As stated in section 5, \"As our base dataset is synthesized by Stable Diffusion with proper filtering, it is difficult for the model to generate high fidelity, real-life images.\" We attribute the underwhelming real-life image generation partly to the base dataset from Instruct Pix2Pix, which is a critical factor since the finetuning dataset is entirely generated by Stable Diffusion and then filtered by CLIP. We firmly believe that a more diverse and high-quality dataset containing real-life images would significantly address this issue. In our preliminary experiments, we did not observe substantial differences between using SD v1.5 and v2.0.\n"}, {"review_id": "rd3W7dnRho", "reviewer": "Reviewer_Gb8x", "summary": "The authors propose an image generation framework. Given the textual description of a task and an input-output example pair, Prompt Diffusion can inpaint the missing output in a consistent way like in the input-output example and the textual guidance instruction. Prompt Diffusion is trained (in a supervised manner) over 6 different tasks and shows some generalization to new tasks and demonstrates emerging image editing capabilities.", "questions": "1. The model is \"capable of in-context learning\" and \"generalization to new tasks\". \n* Can the model extend to different input-output structures in the test time? For example, by utilizing more than one visual example? \n* Given that the input-output examples are concatenated in channels dimension, can the model extend to input-outputs that are *not* aligned pixel-wise?\n* Considering edge prediction as in-context learning after supervised training for depth/hed is a stretch. These tasks are super similar. Similarly, with normals given depth/segmentation supervision.\n* Given the previous points, is it possible that the model performs in-context learning of style (Pix2Pix prompting)? I find this claim more accurate, and I think this is still novel.\n\n2. The model is trained on a discrete number of tasks\n* Are visual examples actually needed? How does the model perform with standard image input image output with text guidance? (similar to InstructPix2Pix).\n\n3. In general, I think more results are necessary to establish the motivation for the new proposed framework. For example:\n* Which modality instructions are more important better? Is it text? image? \n* What are the computation tradeoffs? I assume the text is more efficient than images and easier to obtain. Textual prompts are faster, easier to obtain, and have less memory overhead.", "limitations": "I think the authors should consider emphasizing the following limitations which were not mentioned: \n* the current framework is not very flexible and it assumes a fixed structure of the input-output and query image. \n* is the in-context learning limited to style interpolation? (e.g normals based on depth/segmentation, edges instead of depth/hed)", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Overall, I think the problem setting of visual prompting conditioned on both visual examples and text is novel and interesting.\n- The paper is well written and the presentation is clear\n- By starting from pretrained diffusion models, the authors bring the advantages of visual quality and text guidance to the prompt diffusion framework.", "weaknesses": "My main concern is with the framing of the paper, claiming in-context learning and generalization to new tasks. I think this is a very broad claim which is not backed up by the results. Please see the questions box below for more elaborate feedback/questions.\n\n1. The authors claim the model is \"capable of in-context learning\" and \"generalization to new tasks\". Given the empirical results here I find this is hardly convincing. \n2. The model is trained on a discrete number of tasks, so it is unclear whether visual examples are actually needed or if text-only guidance is enough. \n3. In general, I think more results are necessary to establish the motivation for the new proposed framework.\n\n---\n\nPost rebuttal: given that the authors decided to acknowledge the limitation we discussed, I've decided to update my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer Gb8x for acknowledging the novelty of our proposed framework, appreciating our presentation, and proposing constructive feedback. Below, we address the concerns raised in your review point by point.\n\n> Claiming in-context learning and generalization to new tasks is very broad.\n\n  We acknowledge that in-context learning within the realm of vision is still a step behind the impressive advancement in large language models. These language models benefit from extensive datasets and the capacity to tackle a wide range of tasks with remarkable flexibility. Considering that the datasets and designed tasks are still in the exploratory process, we believe it will be good for the community to not set the bar unrealistically high by comparing the accomplishments of in-context learning progress in language models against those in vision models. \n\n  The pursuit of in-context visual learning with the ability of generalizing to new tasks has attracted increasing popularity within the vision research community. Our innovation, Prompt Diffusion, distinguishes itself from prior works [1,2] by incorporating this pivotal attribute into the state-of-the-art generative framework, Diffusion Models. This amalgamation enables a robust capability to effectively process diverse input samples and manifests a powerful capacity for comprehending distinct modalities encompassing both images and text. Consequently, in contrast to solely relying on image-visual prompts, Prompt Diffusion accommodates multi-modal vision-language prompts, culminating in the generation of high-resolution and photo-realistic images. Quoting Reviewer foSi, the external dimension brings more versatility and applicability to our model. \n\n  Prompt-Diffusion starts from a diffusion-based approach to solve the in-context visual learning problem. We are optimistic that through meticulous architecture design and the utilization of a diverse and comprehensive dataset, encompassing a wide range of tasks, it could significantly enhance the performance of Prompt-Diffusion in the context of visual learning. \n\n  Following the reviewer's suggestion, we plan to add a limitation discussion in our revised manuscript, as shown in the following. “It is essential to acknowledge that the field of in-context learning in vision is currently in a developmental phase and encounters specific challenges in aligning with the accomplishments attained by large language models. Despite the progress achieved through Prompt-Diffusion, it is conceivable that constraints may exist concerning the extent and adaptability of tasks that can be proficiently accommodated. Going forward, further exploration and improvements in architecture and dataset diversity are needed to overcome these limitations and advance the progress of in-context visual learning.”\n\nQuestion 1\n- > Q1.a Can the model extend to different input-output structures in the test time? For example, by utilizing more than one visual example?\n\n  Our current design has a fixed number of stacked convolutional layers to encode example and query images, so the use of CNN based encoder limits the number of visual examples that can be plugged into. However, modifying the encoder to a ViT based model would potentially bring more flexibility in test time for encoding visual examples.\n\n- > Q1.b Can the model extend to input-outputs that are not aligned pixel-wise?\n\n\n  For the current designed tasks, the input-outputs are aligned pixel-wise, which is determined by the dataset that we use. If we have a dataset that supports unaligned input-output pairs, then training on the data would make it work.\n\n\n- > Q1.c The conducted tasks are super similar.\n\n\n  Note the tasks look similar from human knowledge but that doesn’t mean the model trained on one task could easily transfer to the others. We conducted one ablation study in Figure 10. We show that a ControlNet finetuned for an individual task on the dataset easily fails to generate high-quality images for another task.\n\n- > Q1.d Is it possible that the model performs in-context learning of style (Pix2Pix prompting)?\n\n  We follow the convention in the literature in claiming in-context image generation.\n\nQuestion 2\n\n- > Q2.a The discrete number of tasks is limited by the dataset that we have. \n\n  We believe utilizing a larger and more diverse in-context learning dataset could significantly strengthen Prompt-Diffusion.\n\n\n- > Q2.b Are visual examples actually needed? \n\n  Yes, see our ablation study in Figure 8, where we investigate the effects of our vision-language prompt, which consists of the example pair, text guidance, and image query.\n\n- > Q2.c How does the model perform with standard image input image output with text guidance? (similar to InstructPix2Pix).\n\n\n  Our model can take standard images as inputs for forward tasks, such as depth/hed/segmentation map generation. For image editing (similar to Instruct Pix2Pix), we show examples in Figure 6, and the two-step strategy uses standard images as inputs.\n\nQuestion 3\n\n- > Q3.a Which modality instructions are more important better? Is it text? Image?\n\n  We provide ablation study in Figure 8, which validates that both texts and images are important during image generation.\n\n- > Q3.b What are the computation tradeoffs?\n\n  Yes, text is more efficient than images and easier to obtain.\n\nLimitations\n\n- > The current framework is not very flexible\n\n  The current structure of the input-output and query image is fixed, but we think it could be extended to a more flexible version and we left that for future work.\n\n- > Is the in-context learning limited to style interpolation? \n\n  We follow the convention in the literature in claiming in-context image generation.\n\nReferences:\n\n[1] Bar, Amir, et al. \"Visual prompting via image inpainting.\" NeurIPS 2022.\n\n[2] Wang, Xinlong, et al. \"Images speak in images: A generalist painter for in-context visual learning.\" CVPR 2023.\n"}]

Author rebuttal:

Meta-review: The reviewers acknowledge the paper's novel and intriguing problem setting involving visual prompting conditioned on both images and text. The introduction of pretrained diffusion models to incorporate visual and text cues is highlighted as a strong point, enabling improved visual quality and text-guided diffusion. The novel vision-language prompt design, which replaces text with image pairs and queries, is emphasized for its potential to generalize across various vision-language tasks. The reviewer appreciates the model's unique in-context learning ability achieved through multi-task learning pre-training, allowing it to adapt to new tasks seamlessly. The extensive experiments and ablation study conducted to validate the proposed approach's effectiveness further contribute to the paper's strength.
Decision: Accept (spotlight)