Title: In-Context Learning Unlocked for Diffusion Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: diffusion models

in-context learning
Abstract: We present Prompt Diffusion, a framework for enabling in-context learning in diffusion-based generative models. Given a pair of task-specific example images, such as depth from/to image and scribble from/to image, and a text guidance, our model automatically understands the underlying task and performs the same task on a new query image following the text guidance. To achieve this, we propose a vision-language prompt that can model a wide range of vision-language tasks and a diffusion model that takes it as input. The diffusion model is trained jointly on six different tasks using these prompts. The resulting Prompt Diffusion model becomes the first diffusion-based vision-language foundation model capable of in-context learning. It demonstrates high-quality in-context generation for the trained tasks and effectively generalizes to new, unseen vision tasks using their respective prompts. Our model also shows compelling text-guided image editing results. Our framework aims to facilitate research into in-context learning for computer vision. We share our code and pre-trained models at https://github.com/Zhendong-Wang/Prompt-Diffusion.

In-Context Learning Unlocked for Diffusion Models

Abstract

We present Prompt Diffusion, a framework for enabling in-context learning in diffusion-based generative models. Given a pair of task-specific example images, such as depth from/to image and scribble from/to image, and a text guidance, our model automatically understands the underlying task and performs the same task on a new query image following the text guidance. To achieve this, we propose a vision-language prompt that can model a wide range of vision-language tasks and a diffusion model that takes it as input. The diffusion model is trained jointly on six different tasks using these prompts. The resulting Prompt Diffusion model becomes the first diffusion-based vision-language foundation model capable of in-context learning. It demonstrates high-quality in-context generation for the trained tasks and effectively generalizes to new, unseen vision tasks using their respective prompts. Our model also shows compelling text-guided image editing results. Our framework aims to facilitate research into in-context learning for computer vision. We share our code and pre-trained models at https://github. com/Zhendong-Wang/Prompt-Diffusion .

1 Introduction

Recent advancements in machine learning, particularly in the field of natural language processing (NLP) [54], have led to the development of several state-of-the-art large language models (LLMs), such as BERT [9], GPT-2 [36], BART [21], T5 [38], GPT-3 [5], and GPT-4 [33]. These models have been successfully applied to a wide range of tasks, including sentiment analysis, question answering, machine translation, and text generation, to name a few. An emergent behavior of these LLMs is their ability to learn from context, a phenomenon often referred to as in-context learning. In LLMs such as GPT-3 [5], the in-context learning ability allows them to perform a task just by conditioning on the combination of input-output examples and a new query input, a.k.a. prompts, without optimizing any model parameters. With a proper design of the prompt structure and in-context learning, LLMs can unite the pre-training of multiple language tasks and generalize well to previously unseen tasks.

While in-context learning has been extensively studied in NLP, its applications in the field of computer vision are still limited. To showcase the viability and potential of in-context learning as a standard approach for large-scale vision applications, two significant challenges need to be addressed: 1) Designing an effective vision prompt, which incorporates both domain-specific input-output pairs as examples and image queries as conditions, is more challenging than designing prompts for language tasks. 2) In computer vision, large models are typically trained for specific tasks such as segmentation [22], detection [6], classification [20], self-supervised representation learning [37], class-conditional generation [10], and text-to-image generation [41, 44, 67, 31]. As a result, these large vision models are not designed for in-context learning and lack the flexibility to adapt to new tasks. Several recent attempts [3, 7, 56, 57] tackle these difficulties by following the solutions in NLP. More specifically, a simple visual prompt is built by stitching example images, query images, and output images into one large image, and then a Transformer-based image inpainting model is trained to predict the masked output images [3, 56]. Stitching to large images, however, will dramatically increase the computational cost, especially in high-resolution cases.

Tackling these two challenges, this paper aims to unlock the in-context learning ability of text-guided diffusion-based generative models. We introduce a new model architecture, Prompt Diffusion, to perform in-context learning under a vision-language prompt that could accommodate a wide variety of vision-language tasks. We train Prompt Diffusion jointly on six different vision-language tasks. Specifically, we initially establish a general vision-language task by employing our visionlanguage prompt:

prompt: { text-guidance, example: (image1 → image2), image-query: image3} → target: image4 , where (image1 → image2) consists of a pair of vision task examples, e.g. , (depth map → image) , text-guidance provides language instructions for specific tasks, and image3 is the input image query that aligns in type with image1 and hence could be a real image or an image condition ( e.g. , depth or hed map). We subsequently develop Prompt Diffusion, drawing inspiration from the design of Stable Diffusion [69] and ControlNet [41], which can take our vision-language prompt as the input. We finetune Prompt Diffusion from Stable Diffusion [69] checkpoints on six different vision-language tasks, concurrently and uniformly, including three forward tasks: image segmentation/hed/depth map generation, and three inverse tasks: image generation given segmentation/hed/depth maps. We illustrate Prompt Diffusion through examples in Figure 2.

Prompt Diffusion successfully integrates the learning of six different tasks into one vision-language foundation model. Through prompt-based learning, the model can effectively grasp the underlying relationship between the input example pairs. The model can then use this understanding to generate the output image by re-mapping the relationship onto the query image and incorporating the language instructions, as illustrated in Figures 2 and 4. More importantly, the model acquires in-context learning ability by learning across multiple tasks. Remarkably, we observe that Prompt Diffusion not only performs well on the six tasks it has seen during training, but also generalizes effectively across several new unseen tasks, as shown in Figure 5.

Overall, we propose Prompt Diffusion as an initial attempt to unlock the in-context learning ability of text-guided diffusion models. Empirically, we observe that Prompt Diffusion has a promising in-context learning ability on both trained and new, unseen tasks. The success achieved by Prompt Diffusion is to inspire and spur additional research in the field of diffusion-based in-context visual learning. We summarize our main contributions as follows:

Painting of

man

wingshooting

in snow

text

example source

example target

Image query

output

depth map

text

example source

example target

Image query

output

Painting of

man

wingshooting

in snow

Painting of

man

wingshooting

in snow

hed map

segmentation

map

Prompt

Diffusion

Multi-Task Trained

Inverse Tasks

Forward Tasks

Inv-depth

Inv-hed

Inv-seg

depth

hed

seg

CP

A novel vision-language prompt design that well supports the integration of various visionlanguage tasks.

The Prompt Diffusion model, the first diffusion-based versatile vision-language foundation model capable of in-context learning.

Demonstration of high-quality in-context generation on both trained tasks and new, unseen tasks.

2 Related Work

2.1 Diffusion Models

Recent advances in diffusion models have exhibited overwhelming success in generative applications. Diffusion models [47, 49, 17], serving as sophisticated generative models, generate intriguing examples through a step-wise denoising process. These models utilize a forward process that incorporates noise into data distributions and reverses this process to reconstruct the original data. Subsequently, numerous studies [10, 31, 39, 41, 44] are focused on scaling the diffusion models up or improving the training and sampling efficiency [48, 26, 60, 71] to achieve better performance. In particular, LDM [41] is a prominent model that reduces computational costs by applying the diffusion process to a low-resolution latent space, successfully scaling text-to-image generation to web-scale data. Lately, Versatile Diffusion [67] expands the existing single-flow diffusion pipeline into a multi-task multimodal network that handles multiple flows of text-to-image, image-to-text, and variations in one unified model. Many other pieces of research also extend the diffusion framework to various domains, including stable GAN training[59], music generation [18], text-to-3D [34, 2], language generation [24], novel-view synthesis [66], uncertainty quantification [13], reinforcement learning [58], etc. Besides generation, recent studies [28, 69, 15, 30, 4, 61, 12, 19] also explore the potential of diffusion models in image and video editing-related applications.

2.2 In-Context Learning

Recent advances in large language models (LLMs) have made a substantial impact on multi-modal modeling. Despite this, many of these models [8, 11, 14, 23, 25, 27, 46, 50-52, 55, 53] still necessitate fine-tuning for specific downstream tasks. By contrast, in-context learning, a novel paradigm introduced in generative pretrained transformers (GPT-1/2/3/4 [35, 36, 5, 33]), enables models to rapidly adapt to new tasks using only a handful of prompts. Rather than finetuning on downstream tasks, GPT formulates various language tasks as predicting the future tokens given the context, exhibiting remarkable emergent abilities [62] on unseen tasks without explicit training [29, 42, 65, 70].

Lately, researchers have been working on in-context learning frameworks that encompass computer vision tasks as well. For instance, Alayrac et.al. [1] propose Flamingo that combines pretrained visiononly and language-only models and employs paired image-text examples as prompts for few-shot

learning. Bar et.al. [3] introduce a universal inpainting model to tackle multiple vision tasks by taking the desired output as a masked region and predicting it based on given examples. Inspired by this approach, Painter [56] further incorporates more tasks to construct a universal task solver, observing emergent abilities on unseen categories if evaluated on in-domain tasks. Subsequently, SegGPT [57] unifies multiple segmentation tasks into a generalist framework and investigates both spatial ensemble and feature ensemble methods for creating prompt examples. Different from previous works, our work is primarily focused on exploring the in-context learning ability of diffusion models.

2.3 Prompt Design for Visual Language Models

The idea [5, 36] of transforming various language tasks into a generative problem has motivated a lot of following works to explore how to formulate appropriate generative conditions, which is also known as prompt engineering. Designing different prompts is observed to result in a huge performance gap in many language tasks [32, 63]. Prompt designing also raises much attention in vision areas after the breakthroughs of vision-language models ( e.g. , CLIP [37]). Instead of directly predicting a one-hot label-assignment vector using a classifier, CLIP can be used as a classifier by using a correct prompt template, such as 'A photo of a [ label ]'. A following work [72] observes that changing the text template itself matters for the performance of different downstream vision tasks. Other works further propose improving strategies by adding a red circle on the interesting areas of given images [45], or coloring the interesting image patches [68]. While our proposed method also explores vision-language prompt engineering, we further introduce a new interface in the vision backbone that helps take and understand the image prompts.

3 Prompt Diffusion and In-Context Learning

In this section, we describe how to perform vision-language prompting on diffusion-based text-toimage generative models. First, we introduce our novel vision-language prompt design in Section 3.1. Then, we show our model design of Prompt Diffusion in Section 3.2. Finally, we demonstrate how we represent different vision-language tasks via our prompting and how we jointly train Prompt Diffusion on multiple tasks in Section 3.3.

3.1 Vision-Language Prompt Design

In LLMs, a common way of task-prompting for a specific language understanding task is to provide the trained model with pairs of examples of the target task and the input query [5]. For example, in machine translation, English-to-French, Brown et al. [5] construct the prompt as {example: sea otter → loutre de mer, query: cheese → } , and the model will learn from the example in the prompt to translate the query 'cheese' into 'fromage' .

Following the convention, we design the vision-language prompt by replacing the text examples with paired image examples and the text query with an image query. We further add one more text input to allow text-guided image generation. By this design, we propose a new input-output pair format that could generalize the input-output configuration of most vision-language tasks. We show it as follows together with one example:

prompt:

{

text-guidance, example: (image1

→

image2), image-query: image3}

→

target:

image4

,

'Landscape Art Print in the

style of a comic book

featuring the painting

Wilson'

Prompt

image1

image2

image3

text-guidance

image4

where the exmaple: (image1 → image2) informs the model what the target task is and how it could be done through examples, e.g. , image generation from the hed map, text-guidance guides the model to generate an image conditioning on the given text, and the image-query: image3 , which aligns in type with image1 , represents the input to the specific task. The example pair could accommodate any image domain transformation tasks, e.g. , forward tasks: image → segmentation/depth/hed map, and inverse tasks: segmentation/depth/hed map → image, while the involvement of text-guidance provides additional context to generate the target images. Next, we present Prompt Diffusion, which

Prompt Diffusion

SD

Encoder

SD

Decoder

Text Encoder

Time Encoder

SD Middle

ControlNet

𝐼 2

:Example Target

𝐼

1

: Example Source

𝐼 3

:Image Query

Conv

Conv

Text:

'Landscape Art

Print … '

Prompt

ControlNet

Concat

+

Stable Diffusion

SD

Encoder

SD

Decoder

Text Encoder

Time Encoder

SD Middle

ControlNet

Control

Stable Diffusion

Denoised Latent

Denoising Cycle

Decoder

𝐼 4

:Output Image

Text:

'Landscape Art

Print … '

Denoised Latent

Denoising Cycle

Decode …

一

一

一

is a diffusion-based generative model that takes the vision-language prompts as inputs and outputs the target images.

3.2 Prompt Diffusion

Motivated by the success of Stable Diffusion [41] and ControlNet [69] on text-to-image and image conditional generation, we build Prompt Diffusion based on the design of ControlNet. Our visionlanguage prompt is multimodal, incorporating both text and image inputs. For image inputs, we concatenate the example pair of images in the channel dimension, and then project the concatenated example pair and image query into equivalent dimensional embeddings via independent stacked convolutional layers. We compute the sum of the two embeddings and feed it into the ControlNet branch. For the text input, we follow the convention of Stable Diffusion [41] and encode the text input through the pre-trained CLIP [37] text-encoder. The resulting CLIP text embedding is then fed into the Stable Diffusion branch via cross-attention layers. We illustrate Prompt Diffusion in Figure 3.

We keep the Stable Diffusion branch and ControlNet branch the same as their original versions, so that we could finetune Prompt Diffusion from an existing checkpoint without the need to train it from scratch. We finetune Prompt Diffusion from the Stable Diffusion 'v1.5' checkpoint.

3.3 In-Context Learning

We consider six different vision tasks in our joint training. We call them forward tasks when the task inputs are clean images and inverse tasks when the task inputs are image conditions, e.g. , segmentation/dep/hed maps, as depicted in Figure 2.

Datasets. We use the public dataset proposed by Brooks et al. [4] as our base dataset, which consists of around 310k image-caption pairs. Furthermore, we apply the Midas [40] to obtain the depth and normal maps of all images in our dataset. We obtain the image segmentation maps by applying Uniformer [22]. We collect the canny edge maps by the Canny edge detector [6] and hed maps by the HED boundary detector [64]. Note the depth, hed, and segmentation maps are all used for our model training, while the canny edge and normal maps are only used in the testing time for evaluating the in-context learning ability of Prompt Diffusion on new, unseen tasks.

Inverse Tasks. We consider three inverse tasks: depth maps to images, hed maps to images, and segmentation maps to images. For any image-caption pair in the dataset, ( I 1 , C 1 ) , we first sample a random task and another random image I 2 to create the example pair, e.g. , ( HED ( I 2 ) , I 2 ) . Then, we build the vision-language prompt with the example pair, the caption, and the image condition that is consistent with the task specified by the example pair. The image I 1 is the denoising target of Prompt Diffusion. One complete example of the inverse-hed task is shown as follows:

prompt: {text-guidance: C 1 , example pair: [HED( I 2 ) → I 2 ], image-query: HED( I 1 )} → target: I 1 .

By replacing the hed maps with the other two image conditions, we could obtain the vision-language prompts for all the three inverse tasks.

Forward Tasks. We also consider three forward tasks (image processing tasks): images to depth maps, images to hed maps, and images to segmentation maps. We follow the same rule of inverse tasks to create the vision-language prompts. Note here the tasks are flipped, so we flip the order inside both the example pair and the query-target pair. The captions of images are not necessary for image processing tasks, so we use a task-specific fixed text label for each task, such as 'hed maps'. We show one complete example of the forward-hed task as follows:

prompt: {text-guidance: 'hed maps', example pair: [ I 2 → HED( I 2 )], image-query: I 1 } → target: HED( I 1 ) .

Joint Training. We train Prompt Diffusion jointly on these six different vision-language tasks. For simplicity, we train our model on these six tasks uniformly at random. Specifically, each minibatch data contains randomly sampled vision-language prompts and the corresponding target images from randomly selected tasks. Joint training over these different tasks unlocks the in-context learning ability of diffusion models, as shown in Figures 4 and 5. In order to apply classifier-free guidance (CFG) [16], we randomly drop 10% text guidance during training.

4 Experiments

In this section, we conduct extensive experiments to illustrate the power of Prompt Diffusion as a strong versatile vision-language foundation model that is capable of in-context learning. We first show that Prompt Diffusion performs well with multi-task training in Section 4.1. Then, we show that Prompt Diffusion has promising in-context learning ability and could generalize well to new, unseen tasks in Section 4.2. We show that Prompt Diffusion supports controllable and text-guided image editing in Section 4.3. Finally, we conduct an ablation study on the three main components of our vision-language prompt in Appendix A.

Implementations. We implement Prompt Diffusion upon the codebase of ControlNet [69]. As illustrated in Figure 3, we implement both the visual-example-pair encoder and image-query encoder via stacked convolution layers. See the details of these layers in Appendix C. We load the weights from Stable Diffusion checkpoint 'v1.5' for finetuning. We fix the learning rate at 1 × 10 -4 and accumulate gradients every 4 mini-batches with batch size 512. The datasets we used and how we augment the original datasets are described in detail in Section 3.3. We train our model on 8 × A100 Nvidia GPUs for 5000-20000 steps. Longer training helps the model perform better on the finetuned dataset while may perform worse on out-of-distribution images. The results shown in the paper are based on the 5000-step finetuned checkpoint.

Evaluation. We conduct both qualitative and quantitative evaluations to assess the performance of Prompt Diffusion. We provide comprehensive qualitative evaluations to measure Prompt Diffusion's in-context learning capabilities. To assess generation quality, we provide quantitative results measured using the zero-shot Fr'echet Inception Distance (FID) for inverse tasks and Root Mean Square Error (RMSE) for forward tasks, as detailed in Section 4.4. The test dataset is composed of random images from the ControlNet [69] example images, DreamBooth [43] data benchmark, and the test split of our base dataset [4]. During the inference stage, one additional random image from our dataset is used to construct the example pair for each specific task.

4.1 Multi-Task Learning

We jointly train Prompt Diffusion on three inverse tasks, including inverting depth/hed/segmentation, and three forward tasks, including extracting depth/hed/segmentation maps. Note ControlNet [69] aims to enhance pretrained image diffusion models with task-specific conditions ( e.g. , depth maps). We finetune a ControlNet, from the same Stable Diffusion checkpoint, on our dataset for each inverse task independently as our baseline, and we refer to it as CN(FT). We qualitatively evaluate Prompt Diffusion on the six trained tasks in Figure 4.

'African elephant,

foggy morning, hot air

balloons landing on

background …

'

-

Village Autumn

-

Digital Prints

'

'

cammello in Assassin's

Creed Origins

'

'venice

-balcomy-

By-

'

'

photogasm_zena_

holloway_

beautifulbizarre

'

Depth-to-Image

Hed-to-Image

Seg-to-Image

[Task Name]

'depth map'

'

hed

map'

'segmentation map'

[Task Name]

'depth map'

'

hed

map'

'segmentation map'

[Task Name]

'depth map'

'

hed

map'

'segmentation map'

[Task Name]

'depth map'

'

hed

map'

'segmentation map'

[Task Name]

'depth map'

'

hed

map'

'segmentation map'

Image-to-Depth

Image-to-Hed

Image-to-Seg

Inverse Tasks

Forward Tasks

We observe that Prompt Diffusion could not only generate high-fidelity images based on different task-specific conditions for the inverse tasks, but also generate detailed and rational image conditions if it is given the real images for the forward tasks. For an inverse task, when the image conditions already sketch the main components of the output images, such as depth and hed maps, Prompt Diffusion could successfully maintain the shape consistency in the output images and generate diverse images with different random seeds and text guidance. When the image conditions are coarse like segmentation maps, Prompt Diffusion could generate more imaginary images ( e.g. , the fairy with a personalized hair style in the third block). For forward tasks, we empirically find Prompt Diffusion generally produces even more detailed depth and hed maps compared to these maps in the original dataset. The produced segmentation maps are comparable to the ones used as the targets during finetuning.

Note ControlNet [69] is designed for image generation with image conditions. Hence, we compare Prompt Diffusion with CN(FT) on the three inverse tasks in Figure 10 of the Appendix. The results show that Prompt Diffusion performs comparably well to CN(FT) which is trained specifically for each individual task. We further evaluate the generation ability of CN(FT) in Figure 11 of the Appendix. We directly infer CN(FT) on new tasks and observe a large number of failures. This validates the necessity of multi-task training for Prompt Diffusion to achieve the capability to generalize across a wide variety of tasks.

4.2 In-Context Learning Ability

We evaluate Prompt Diffusion on tasks that are outside of our supervised training dataset to further study its generalization ability. We first select three unseen tasks, inv-scribble (image generation from personalized scribbles), inv-edge (image generation from canny-edge maps), and inv-normal (image generation from normal maps). The model is given a pair of image examples that illustrates how to perform a desired task, and is asked to re-conduct the task onto new image query inputs with text guidance.

Inv Edge

'a blue backpack on the beach

'

- Village Autumn

'

'a shoe on the floor

'

'

'

Inv Scribble

'a turtle stands on the beach

'

'hot air balloon

'

'a fluffy golden retriever stands on the ground

'

'

a colorful backpack

'

Inv Normal

'a nice

-looking clock

'

'a nice

-looking shoe

'

'a duck toy in the water

'

'

a bowl of strawberries

'

Example Pair: Scribble -> Image

Example Pair: Edge -> Image

Example Pair: Normal -> Image

Example Pair: Style Transfer

Image Query

Text

'A

high quality

image'

Generated (two examples)

'A

high quality

image'

'henry_john_ye

end_king_a351

9_the_old_mill

_small'

Example Pair: Misalignment

Image Query

Text

Generated (two examples)

'henry_john_ye

end_king_a351

9_the_old_mill

_small'

We show the qualitative results in Figure 5. Prompt Diffusion successfully learns the underlying correlation between the example pairs and re-maps the relationship onto query and output pairs, without the need of training or finetuning on the three new domains. The image generation is also well controlled by text guidance as shown in the results.

In Figure 6, we further evaluate Prompt Diffusion on two distinct tasks: style transfer and misaligned input-output example pairs. For the first task, as shown in the initial two rows, Prompt Diffusion is given a pair of image style transfer examples. It is then tasked with applying this style transfer to a new target image, accompanied by a general text description, such as 'A high quality image.' The findings indicate that Prompt Diffusion is not only capable of discerning the underlying task from the visual example pairs but can also adeptly carry it out on fresh queries. For the second task, detailed in

the final two rows, we investigate the impact of pixel alignment in the example pairs. Even when various regions in the example pair are concealed, resulting in pixel misalignment, Prompt Diffusion manages to execute the intended task proficiently without any significant disruption.

4.3 Image Editing

'A young lady with

yellow

roses.

'

Depth + Text -> Image

Input

Seg + Text -> Image

Hed + Text -> Image

One-Step

[Image -> Depth] + Text -> Image

Two-Step

[Image -> Segmentation] + Text -> Image

[Image -> Hed] + Text -> Image

-

Village Autumn - Digital

Prints

'

-

Lord

of the Rings Village

-

Digital Prints

'

Input

'A young lady with

white roses.

'

Input

'

photogasm_zena_hollo

way_beautifulbizarre

'

'

photogasm_zena_hollo

way_beautifulbizarre_

bl

ack_and_white

'

moonlight,

in winter,

with snow

'

Input

Depth Map Output

'tree, shore, leaves

in a

sunny day

'

Input

Depth Map Output

moonlight,

in winter,

with snow

'

Input

Seg Map Output

moonlight,

in winter,

with snow

'

Input

Hed Map Output

'tree, shore, leaves

in a

sunny day

'

Input

Seg Map Output

'tree, shore, leaves

in a

sunny day

'

Input

Hed Map Output

We show that Prompt Diffusion has strong image editing ability in two ways. First, if only the conditions of images are given, e.g. , depth/segmentation/hed maps, Prompt Diffusion is able to synthesize images based on the input image conditions and additional text descriptions. Second, if one base image is given, we could apply a two-step strategy to achieve controllable image editing: We first generate the image conditions and then generate edited images based on both the generated conditions and text descriptions, all within one model: Prompt Diffusion. As shown in Figure 7, with the help of image conditions, Prompt Diffusion is capable of performing consistent modifications specified by the text guidance input without worrying about undesired excessive changes.

4.4 Quantitative Results

We compare Prompt Diffusion with CN(FT) quantitatively. For inverse tasks, we measure the zeroshot FID on the test split of our base dataset [4]. We use the corresponding image condition, text input, and example pair as the input for either Prompt Diffusion or CN(FT), and generate 10k random images where each input produces one image. We then measure the FID using the 10k generated samples with the corresponding 10k test images used as reference. For forward tasks, we measure the zero-shot RMSE between the generated image conditions (e.g., depth/hed/segmentation maps) and the corresponding ones in the test set, where all image conditions are rescaled to [0 , 1] for computing the RMSE. Note for both Prompt Diffusion and CN(FT), we finetune them with the same number of 5000 steps. We show the quantitative comparison in Table 1. We observe that Prompt Diffusion has a comparable or even better performance compared to CN(FT).

5 Conclusion and Discussion

In this study, we introduce Prompt Diffusion, a novel in-context visual foundation model that leverages diffusion-based techniques to generate images from vision-language prompts. Our model is designed to be flexible and compatible with various vision-language tasks, allowing it to accommodate multiple use cases. We train Prompt Diffusion on six different vision-language tasks and conduct

text

example source    example target

Image query

output

text

Image query

output

scribble map

normal map

a house

example source    example target

a dog

extensive experiments to evaluate its performance. Our results demonstrate that Prompt Diffusion has a promising generalization ability, not only on trained tasks but also on unseen ones. Additionally, Prompt Diffusion enables controllable image editing, including style, artistic medium, and contextual changes, without introducing excessive undesired modifications.

While our model has demonstrated notable successes in in-context learning, it is important to acknowledge its limitations. Currently, the visual scope of our model is restricted by the limited number of real-life images in the training dataset used for finetuning. As our base dataset [4] is synthesized by Stable Diffusion with proper filtering, it is difficult for the model to generate highfidelity, real-life images. Moreover, our model is only trained jointly across six pre-defined tasks, and we believe that expanding the range of joint tasks would enhance Prompt Diffusion's in-context learning capability. As shown in the last row of Figure 9, it is difficult to predict the behavior of our model when the query image is misaligned in type with the source image in the example pair, and as shown in Figure 8, there are instances where our model struggles with new tasks. Prompt Diffusion is also limited when it comes to executing physical transformation tasks like rotation and flipping without additional fine-tuning specific to these tasks. To unlock more of Prompt Diffusion's in-context learning ability, it is possible to change the finetuning setting to training from scratch, but this would require significant computational resources, which presently limits its feasibility for us.

Prompt Diffusion represents a pioneering effort in unlocking the in-context learning capacity of diffusion-based models. We hope that this work will encourage more researchers to explore the potential of diffusion-based in-context learning and contribute to the advancement of this exciting field.

Acknowledgments

Z. Wang and M. Zhou acknowledge the support of NSF-IIS 2212418, NIH-R37 CA271186, and the NSF AI Institute for Foundations of Machine Learning (IFML).

References

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems , 35: 23716-23736, 2022.

Mohammadreza Armandpour, Huangjie Zheng, Ali Sadeghian, Amir Sadeghian, and Mingyuan Zhou. Re-imagine the negative prompt algorithm: Transform 2d diffusion into 3d, alleviate janus problem and beyond. arXiv preprint arXiv:2304.04968 , 2023.

Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image inpainting. Advances in Neural Information Processing Systems , 35: 25005-25017, 2022.

Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:2211.09800 , 2022.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

John Canny. A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelligence , (6):679-698, 1986.

Ting Chen, Saurabh Saxena, Lala Li, Tsung-Yi Lin, David J Fleet, and Geoffrey E Hinton. A unified sequence interface for vision tasks. Advances in Neural Information Processing Systems , 35:31333-31346, 2022.

Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In Computer VisionECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX , pages 104-120. Springer, 2020.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780-8794, 2021.

Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. Violet: End-to-end video-language transformers with masked visual-token modeling. arXiv preprint arXiv:2111.12681 , 2021.

Vidit Goel, Elia Peruzzo, Yifan Jiang, Dejia Xu, Nicu Sebe, Trevor Darrell, Zhangyang Wang, and Humphrey Shi. Pair-diffusion: Object-level image editing with structure-and-appearance paired diffusion models. arXiv preprint arXiv:2303.17546 , 2023.

Xizewen Han, Huangjie Zheng, and Mingyuan Zhou. CARD: Classification and regression diffusion models. In Advances in Neural Information Processing Systems , 2022.

Lisa Anne Hendricks, John Mellor, Rosalia Schneider, Jean-Baptiste Alayrac, and Aida Nematzadeh. Decoupling the role of data, attention, and losses in multimodal transformers. Transactions of the Association for Computational Linguistics , 9:570-585, 2021.

Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 , 2022.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Qingqing Huang, Daniel S Park, Tao Wang, Timo I Denk, Andy Ly, Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Frank, et al. Noise2music: Text-conditioned music generation with diffusion models. arXiv preprint arXiv:2302.03917 , 2023.

Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. arXiv preprint arXiv:2303.13439 , 2023.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM , 60(6):84-90, 2017.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 , 2019.

Kunchang Li, Yali Wang, Junhao Zhang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unifying convolution and self-attention for visual recognition. arXiv preprint arXiv:2201.09450 , 2022.

Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. Hero: Hierarchical encoder for video+ language omni-representation pre-training. arXiv preprint arXiv:2005.00200 , 2020.

Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B Hashimoto. Diffusion-lm improves controllable text generation. arXiv preprint arXiv:2205.14217 , 2022.

Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16 , pages 121-137. Springer, 2020.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.00927 , 2022.

Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems , 32, 2019.

Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. CoRR , abs/2108.01073, 2021. URL https://arxiv.org/abs/2108.01073 .

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837 , 2022.

Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv:2211.09794 , 2022.

Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 , 2021.

Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114 , 2021.

OpenAI. OpenAI GPT-4. https://openai.com/gpt-4/ , 2022.

Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988 , 2022.

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21(140):1-67, 2020. URL http://jmlr.org/papers/v21/20-074.html .

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern analysis and machine intelligence , 44(3):1623-1637, 2020.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. arXiv preprint arXiv:2112.08633 , 2021.

Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208.12242 , 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems , 35:36479-36494, 2022.

Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about a red circle? visual prompt engineering for vlms. arXiv preprint arXiv:2304.06712 , 2023.

Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15638-15650, 2022.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265. PMLR, 2015.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems , 32, 2019.

Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.08530 , 2019.

Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF international conference on computer vision , pages 7464-7473, 2019.

Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490 , 2019.

Korawat Tanwisuth, Shujian Zhang, Huangjie Zheng, Pengcheng He, and Mingyuan Zhou. POUF: Prompt-oriented unsupervised fine-tuning for large pre-trained models. arXiv preprint arXiv:2305.00350 , 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Jianfeng Wang, Xiaowei Hu, Zhe Gan, Zhengyuan Yang, Xiyang Dai, Zicheng Liu, Yumao Lu, and Lijuan Wang. Ufo: A unified transformer for vision-language representation learning. arXiv preprint arXiv:2111.10023 , 2021.

Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. arXiv preprint arXiv:2212.02499 , 2022.

Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting everything in context. arXiv preprint arXiv:2304.03284 , 2023.

Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193 , 2022.

Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Diffusion-GAN: Training GANs with diffusion. arXiv preprint arXiv:2206.02262 , 2022.

Zhendong Wang, Yifan Jiang, Huangjie Zheng, Peihao Wang, Pengcheng He, Zhangyang Wang, Weizhu Chen, and Mingyuan Zhou. Patch diffusion: Faster and more data-efficient training of diffusion models. arXiv preprint arXiv:2304.12526 , 2023.

Zhixin Wang, Ziying Zhang, Xiaoyun Zhang, Huangjie Zheng, Mingyuan Zhou, Ya Zhang, and Yanfeng Wang. DR2: Diffusion-based robust degradation remover for blind face restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1704-1713, 2023.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 , 2022.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 , 2022.

Saining Xie and Zhuowen Tu. Holistically-nested edge detection. In Proceedings of the IEEE international conference on computer vision , pages 1395-1403, 2015.

Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02080 , 2021.

Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Yi Wang, and Zhangyang Wang. Neurallift360: Lifting an in-the-wild 2d photo to a 3d object with 360° views. arXiv e-prints , pages arXiv-2211, 2022.

Xingqian Xu, Zhangyang Wang, Eric Zhang, Kai Wang, and Humphrey Shi. Versatile diffusion: Text, images and variations all in one diffusion model. arXiv preprint arXiv:2211.08332 , 2022.

Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Cpt: Colorful prompt tuning for pre-trained vision-language models. arXiv preprint arXiv:2109.11797 , 2021.

Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543 , 2023.

Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. What makes good examples for visual in-context learning? arXiv preprint arXiv:2301.13670 , 2023.

Huangjie Zheng, Zhendong Wang, Jianbo Yuan, Guanghan Ning, Pengcheng He, Quanzeng You, Hongxia Yang, and Mingyuan Zhou. Learning stackable and skippable lego bricks for efficient, reconfigurable, and variable-resolution diffusion modeling. arXiv preprint arXiv:2310.06389 , 2023.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision , 130(9):2337-2348, 2022.

A Ablation Study

We conduct an ablation study to investigate the effects of vision-language prompt, which consists of the example pair, text guidance, and image query. We show the generation results in Figure 9. In the first row, we keep the text guidance and image query fixed, while we use different random images to construct the example pairs. The results suggest that Prompt Diffusion is not sensitive to what images are used to construct the example pairs, while it cares about the underlying correlation between the images inside the example pairs. In the second row, compared to the first row, we only modify the image query part, and we could see the output images are dramatically changed but consistent with the new image queries. In the third row, apart from the first row, we modify the text guidance and observe the output images are modified accordingly. In the last row, we investigate the impact of using an image query that is intentionally set to be misaligned in type with the source image of the example image pair. Our results show that in such cases, the model's behavior often becomes difficult to predict. Overall, with our vision-language prompt where the image query is aligned with the source image in the example image pair, the example pair tells the model what the specific task is, the image query provides the image domain condition for the generation, and the text guidance allows diverse semantic modifications.

a blue

backpack

on the

beach

Example Pair

Text

Image Query

Output

a blue

backpack

on the

beach

Example Pair

Text

Image Query

Output

a blue

backpack

on the

beach

a blue

backpack

on the

beach

an old

backpack

on the

beach

a lego

backpack

on the

beach

a lego

backpack

on the

beach

a lego

backpack

on the

beach

B Comparison to ControlNet

We qualitatively compare Prompt Diffusion with ControlNet [69]. We follow the guidance of ControlNet, finetune a ControlNet, from the same Stable Diffusion checkpoint, on our dataset for each inverse task independently as our baseline, and we call it CN(FT). We show the comparison results in Figure 10. We observe the jointly trained Prompt Diffusion performs comparably well to the independently trained ControlNet, which implies no significant performance change with multi-task learning applied.

We further evaluate the generation ability of the task-specific ControlNet in Figure 11. We directly apply the independently finetuned ControlNet on one new task during inference. We could observe a large number of failures when directly applying ControlNet to new, unseen tasks.

Appendix

Ours

CN (FT)

Inv Depth Map

Text

Ours

CN (FT)

Inv Hed Map

Ours

CN (FT)

Inv Seg Map

'African elephant,  foggy morning, hot air

balloons landing on background …

'

- Village Autumn

-

Digital Prints

'

'

Bayek cavalca un cammello in Assassin's Creed

Origins

'

'

venice-balcomy-by-

'

'

photogasm_zena_holloway_

beautifulbizarre

'

C Model Architecture

We build Prompt Diffusion based on ControlNet [69]. We show the detailed model architecture in Figure 12. Note we lock the parameters of Stable Diffusion encoder blocks for finetuning to inherit the encoder capability that Stable Diffusion learns from large-scale pre-training. All the other parameters are open for finetuning. The Stable Diffusion branch takes images in the latent space as inputs, while the ControlNet branch takes the original images as inputs. The pretrained image encoder and decoder, which map images to latent space and re-map latent images to original images, are not shown in the illustration. The latent images are in four channels and eight times smaller in both height and width.

As mentioned in the main paper, our ControlNet branch takes three images as inputs. Two of them form the example pair, which is first concatenated in the RGB channel and then encoded by stacked convolution layers. The third image represents the image query and is encoded with independent stacked convolution layers. We show the architecture of the stacked convolution layers in Figure 13. We encode the example pair and the image query to the same dimensional latent embeddings and then sum them up as the inputs for the ControlNet branch.

D Potential Social Implications

The development of image generation models has the potential to have negative social impacts. These models can create realistic images that are difficult to distinguish from real photographs, which could lead to a proliferation of fake images used for malicious purposes. For example, these models could be used to create fake images of people committing crimes or engaging in immoral behavior, which could damage reputations and lead to false accusations. Additionally, the widespread use of image generation models could contribute to the erosion of trust in visual evidence, which could have implications for legal proceedings, journalism, and other fields that rely on visual evidence. There is also a risk that these models could perpetuate biases and stereotypes, as they may learn from biased datasets and generate images that reinforce those biases. It is therefore important to carefully consider the ethical implications of image generation models and take steps to mitigate their potential negative impacts.

Depth to Image

Trained

Text

Hed to Image

Seg to Image

Depth to Image

Seg to Image

Depth to Image

Inference

Trained

Inference

Trained

Inference

'African elephant,  foggy morning, hot air

balloons landing on background …

'

- Village Autumn

-

Digital Prints

'

'

Bayek cavalca un cammello in Assassin's Creed

Origins

'

'

venice-balcomy-by-

'

'

photogasm_zena_holloway_

beautifulbizarre

'

9

E More Examples

We show more exmaples of applying Prompt Diffusion here.

Text Encoder

Time Encoder

SD Encoder Block-1

64x64

SD Encoder Block-2

32x32

SD Encoder Block-3

16x16

SD Encoder

Block-4 8x8

SD Middle

Block 8x8

SD Decoder

Block-4 8x8

SD Decoder Block-3

16x16

SD Decoder Block-2

32x32

SD Decoder Block-1

64x64

𝐼 1

𝐼 2

𝐼 3

𝑵𝒐𝒊𝒔𝒆

Time

Text & Time

SD Encoder Block-1

64x64

SD Encoder Block-2

32x32

SD Encoder Block-3

16x16

SD Encoder

Block-4 8x8

SD Middle

Block 8x8

zero convolution

zero convolution

zero convolution

zero convolution

zero convolution

zero convolution

concat

Stacked Convolution

zero convolution

Stacked Convolution

+

Text Guidance

Prompt

𝐼 4

Stable Diffusion Branch

ControlNet Branch

Stacked Convolution Layers

Conv2d(in_channels=input_channels, out_channels=16,

Activation SiLU()

Conv2d(in_channels=16, out_channels=16,

Activation SiLU()

Conv2d(in_channels=16, out_channels=32,

Activation SiLU()

Conv2d(in_channels=32, out_channels=32,

Activation SiLU()

Conv2d(in_channels=32, out_channels=96,

Activation SiLU()

Conv2d(in_channels=96, out_channels=96,

Activation SiLU()

Conv2d(in_channels=96, out_channels=256,

Activation SiLU()

'

Suzy Parker in Lanvin-Castillo, photographed by

#fashion

'

'

for ELLE #fashion,

with a pink dress and red hair

'

'

ELLE #fashion,

with a pair of glasses

'

'

A Maine Coon Cat

'

'

A

red

Maine Coon Cat

'

'

A Maine Coon Cat

is Smiling

'

'

'

'

mountain

in the night

'

'

Famous Kirkjufellsfoss

mountain

'

'

a horse is

eating grass'

'

a

zebra

is eating grass

'

'

a horse is

drinking water

'

'

A gentle man wearing white clothes, wearing a

beard, posing in sea scenery

'

'

A gentle man

in a pirate costume

, posing in sea

scenery

'

'

A gentle man wearing white clothes posing in sea

scenery

with a mask

'

'

Girl Picking Poppies_OSG by One Simple Gallery

'

'

Purple Hemerocallis

by One Simple

Gallery

'

'

Hemerocallis

with

butterflies flying

'

Hed Input

Hed Input

'

Detail of Boar Lane, Leeds, 1881 by John Atkinson

'

'

in the night with one

'

'

with huge snow

'

Hed Input

'

henry_john_yeend_king_a3519_the_old_mill_small

.jpg

'

the_old_mill_small,

in the lord of the rings

'

henry_john_yeend_king_the_old_mill_small,

with

snow

'

Hed Input

'

Paintings

'

'

Paintings,

on a mountain

'

'

A pirate

'

'

A

business man

posing in a beautiful

village

.

'

'

A

solder

posing in the

desert

'

'An

old father

posing on

Mars

.'

'

Vestrahorn mountain is on the east coast of Iceland.

Photograph by landscape photographer Matt

McGee.

'

'

Vestrahorn

volcanic eruptions

on the east coast

of Iceland.

'

'

A Huge Iceberg on the Sea

.

'

'BUTTERMERE MORNING'

'

BUTTERMERE MORNING

with lightning

.'

'

BUTTERMERE MORNING

COVERED IN SNOW

'

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an image generation framework. Given the textual description of a task and an input-output example pair, Prompt Diffusion can inpaint the missing output in a consistent way like in the input-output example and the textual guidance instruction. Prompt Diffusion is trained (in a supervised manner) over 6 different tasks and shows some generalization to new tasks and demonstrates emerging image editing capabilities.\n\nSTRENGTHS:\n- Overall, I think the problem setting of visual prompting conditioned on both visual examples and text is novel and interesting.\n- The paper is well written and the presentation is clear\n- By starting from pretrained diffusion models, the authors bring the advantages of visual quality and text guidance to the prompt diffusion framework.\n\nWEAKNESSES:\nMy main concern is with the framing of the paper, claiming in-context learning and generalization to new tasks. I think this is a very broad claim which is not backed up by the results. Please see the questions box below for more elaborate feedback/questions.\n\n1. The authors claim the model is \"capable of in-context learning\" and \"generalization to new tasks\". Given the empirical results here I find this is hardly convincing. \n2. The model is trained on a discrete number of tasks, so it is unclear whether visual examples are actually needed or if text-only guidance is enough. \n3. In general, I think more results are necessary to establish the motivation for the new proposed framework.\n\n---\n\nPost rebuttal: given that the authors decided to acknowledge the limitation we discussed, I've decided to update my score.\n\nQUESTIONS:\n1. The model is \"capable of in-context learning\" and \"generalization to new tasks\". \n* Can the model extend to different input-output structures in the test time? For example, by utilizing more than one visual example? \n* Given that the input-output examples are concatenated in channels dimension, can the model extend to input-outputs that are *not* aligned pixel-wise?\n* Considering edge prediction as in-context learning after supervised training for depth/hed is a stretch. These tasks are super similar. Similarly, with normals given depth/segmentation supervision.\n* Given the previous points, is it possible that the model performs in-context learning of style (Pix2Pix prompting)? I find this claim more accurate, and I think this is still novel.\n\n2. The model is trained on a discrete number of tasks\n* Are visual examples actually needed? How does the model perform with standard image input image output with text guidance? (similar to InstructPix2Pix).\n\n3. In general, I think more results are necessary to establish the motivation for the new proposed framework. For example:\n* Which modality instructions are more important better? Is it text? image? \n* What are the computation tradeoffs? I assume the text is more efficient than images and easier to obtain. Textual prompts are faster, easier to obtain, and have less memory overhead.\n\nLIMITATIONS:\nI think the authors should consider emphasizing the following limitations which were not mentioned: \n* the current framework is not very flexible and it assumes a fixed structure of the input-output and query image. \n* is the in-context learning limited to style interpolation? (e.g normals based on depth/segmentation, edges instead of depth/hed)", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel prompt design and model called Prompt Diffusion, for in-context learning in vision-language tasks. The vision-language prompt is designed by replacing text examples with paired image examples and the text query with an image query. The paper conducts extensive experiments to demonstrate Prompt Diffusion as a strong versatile vision-language foundation model that is capable of in-context learning.\n\nSTRENGTHS:\nThis paper pioneered a new and timely problem setting: extending in-context learning beyond LLMs, to text2image diffusion models. The proposed model, Prompt Diffusion, integrates the learning of multiple tasks into one vision-language foundation model, and it acquires in-context learning ability by learning across multiple tasks and generalizes effectively across new, unseen tasks. \n\nThe framework consists of two main components: a vision-language prompt and a diffusion model. The vision-language prompt replaces text examples with paired image examples and the text query with an image query, allowing for a new input-output pair format that could generalize the input-output configuration of most vision-language tasks. The implementation borrowed ideas from ControlNet. The diffusion model then takes the vision-language prompt as input and is trained jointly on six different tasks using these prompts. The resulting Prompt Diffusion model becomes the first diffusion-based foundation model capable of in-context learning. \n\nThe model demonstrates high-quality in-context generation for the trained tasks and effectively generalizes to new, unseen vision tasks using their respective prompts. It also demonstrates strong image editing ability. The proposed framework aims to facilitate research into in-context learning for computer vision.\n\nWEAKNESSES:\nWhile this pilot study is highly interesting, it is hard to assess if the diffusion model has the potential to achieve truly generalizable “in-context learning” as LLMs do. Currently, the model is only trained jointly across six pre-defined tasks, and those tasks are similar dense prediction type. The limited diversity of tasks evaluated in this paper constitutes an important hurdle to assess the work’s true value. \n\nIt is also hard to predict whether expanding the range of joint tasks would scale up Prompt Diffusion’s in-context learning capability.  Additionally, the model has poor robustness in handling misaligned query images (Figure 7). I am curious how the authors think this can be fixed: shall we expect to encompass more diverse training tasks, training data, etc?\n\nThe current generation results of high-fidelity, real-life images look underwhelming. It is noted that the authors used SD v1.5 backbone. I’ll be curious to see if in-context learning can work better if starting from SD v2.1.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nThe authors have thoroughly discussed limitations in Section 5.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPrompt Diffusion is a novel framework that enables in-context learning in diffusion-based generative models. It consists of a vision-language prompt and a diffusion backbone, which is trained jointly on six different tasks using their respective prompts. The resulting Prompt Diffusion model becomes the first diffusion-based vision-language model capable of in-context learning, demonstrating high-quality in-context generation for the trained tasks while generalizing to unseen vision tasks.\n\nSTRENGTHS:\n1) The proposed vision-language prompt design is novel, which replaces texts with paired images and replaces the text query with an image query. This design allows for a new input-output pair format that is able to generalize the input-output configuration for most vision-language tasks. \n\n2) The Prompt Diffusion model is the first diffusion-based versatile vision-language foundation model capable of in-context learning. The model acquires in-context learning ability by multi-task learning pre-training, so as to generalize “zero-shot” to additional new tasks with no further adaptation\n\n3) The authors conducted extensive experiments to illustrate their capablity of in-context learning. An ablation study is conducted to investigate the effects of vision-language prompt, which further supports the effectiveness of the proposed design.\n\n4) The model also yields compelling text-guided image editing results, which demonstrate the controllability of the model in generating images based on text guidance.\n\nWEAKNESSES:\n1) I would be cautious to call the current model to have \" strong in-context learning ability\", since the unseen tasks tested in this paper are still similar to the known ones. For example, \"HED2Image\" task versus \"Canny2Image\" and \"Scribble2Image\" tasks. The paper would be strengthened greatly if more diverse unseen tasks could be learned in context.\n\n2) For text encoding, the authors used a pre-trained CLIP. I wonder that if replace CLIP with an LLM encoder module, some of which have its own in-context learning ability in language domain, will enhancethe in-context learning ability?  This is explored by a rencent work: \"LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models\". \n\n3) The current framework follows instructive pix2pix in most cases and the implementation leverages ControlNet straightforwardly. However, this is not my major concern due to the unconventionality and novelty of the proposed method.\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces Prompt Diffusion. It is a framework designed to facilitate in-context learning within diffusion-based generative models. \nBy providing a pair of task-specific example images, such as depth from/to image and scribble from/to image, along with textual guidance,  the designed solution can automatically comprehend the underlying task and replicate the same task on a new query image following the provided text instructions. \nSpecifically, the authors propose a vision-language prompt that can effectively model various vision-language tasks. The diffusion model, which takes the prompt as input, is trained in a joint manner on six distinct tasks utilizing these prompts.\n\nSTRENGTHS:\n--The paper is well-written and easy to follow. \n--The authors have done an excellent job of clearly articulating the motivation behind their work. \n--The problem statement is well-defined and the objectives are clearly outlined.\n\nWEAKNESSES:\n--  The work appears to be an extension of a previous work titled \"A Generalist Painter for In-Context Visual Learning\". The design principles and motivations are similar, while there are notable differences that distinguish this work from the previous one.\n-- Differences from Prior Work: The authors have introduced text commands into prompts, which adds a new dimension to the model and potentially increases its versatility and applicability. Additionally, the foundation model used in this work is different from the one used in the prior work. This change could potentially lead to different performance characteristics and should be explored in more detail.\n\nQUESTIONS:\nN.A.\n\nLIMITATIONS:\nOverall, this paper presents an interesting extension to previous work in the field. The clarity of the writing and the strong motivation make it a valuable contribution. However, a more detailed comparison with the prior work could strengthen the paper and provide more insights into the novelty and significance of the proposed approach.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}