SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We show the qualitative results in Figure 5. Prompt Diffusion successfully learns the underlying correlation between the example pairs and re-maps the relationship onto query and output pairs, without the need of training or finetuning on the three new domains. The image generation is also well controlled by text guidance as shown in the results.
In Figure 6, we further evaluate Prompt Diffusion on two distinct tasks: style transfer and misaligned input-output example pairs. For the first task, as shown in the initial two rows, Prompt Diffusion is given a pair of image style transfer examples. It is then tasked with applying this style transfer to a new target image, accompanied by a general text description, such as "A high quality image." The findings indicate that Prompt Diffusion is not only capable of discerning the underlying task from the visual example pairs but can also adeptly carry it out on fresh queries. For the second task, detailed in
the final two rows, we investigate the impact of pixel alignment in the example pairs. Even when various regions in the example pair are concealed, resulting in pixel misalignment, Prompt Diffusion manages to execute the intended task proficiently without any significant disruption.
4.3 Image Editing
Figure 7: Image Editing Results. We demonstrate that Prompt Diffusion excels in image editing tasks through two strategies: one-step and two-step. One-step: when only the image condition, e.g., depth/seg/hed maps, is given, our model would edit the condition for generation according to the text descriptions. Two-Step: when the original image is given, we first do forward sampling to sample image conditions and then conduct inverse tasks to edit the condition with text guidance. Two-Step provides more controllable editing.
We show that Prompt Diffusion has strong image editing ability in two ways. First, if only the conditions of images are given, e.g., depth/segmentation/hed maps, Prompt Diffusion is able to synthesize images based on the input image conditions and additional text descriptions. Second, if one base image is given, we could apply a two-step strategy to achieve controllable image editing: We first generate the image conditions and then generate edited images based on both the generated conditions and text descriptions, all within one model: Prompt Diffusion. As shown in Figure 7, with the help of image conditions, Prompt Diffusion is capable of performing consistent modifications specified by the text guidance input without worrying about undesired excessive changes.
We compare Prompt Diffusion with CN(FT) quantitatively. For inverse tasks, we measure the zero-shot FID on the test split of our base dataset [4]. We use the corresponding image condition, text input, and example pair as the input for either Prompt Diffusion or CN(FT), and generate 10k random images where each input produces one image. We then measure the FID using the 10k generated samples with the corresponding 10k test images used as reference. For forward tasks, we measure the zero-shot RMSE between the generated image conditions (e.g., depth/hed/segmentation maps) and the corresponding ones in the test set, where all image conditions are rescaled to  $[0,1]$  for computing the RMSE. Note for both Prompt Diffusion and CN(FT), we finetune them with the same number of 5000 steps. We show the quantitative comparison in Table 1. We observe that Prompt Diffusion has a comparable or even better performance compared to CN(FT).
In this study, we introduce Prompt Diffusion, a novel in-context visual foundation model that leverages diffusion-based techniques to generate images from vision-language prompts. Our model is designed to be flexible and compatible with various vision-language tasks, allowing it to accommodate multiple use cases. We train Prompt Diffusion on six different vision-language tasks and conduct
Table 1: Zero-Shot Quantitative Results. We provide the FID comparison for inverse tasks and RMSE comparison for forward tasks on our test dataset.
Methods	FID ↓ (Inverse Tasks)	RMSE ↓ (Forward Tasks)
Depth-to-Image	Hed-to-Image	Seg-to-Image	Image-to-Depth	Image-to-Hed	Image-to-Seg
CN(FT) [69]	19.81	13.07	20.71	0.20	0.18	0.36
Prompt Diffusion (ours)	18.60	13.35	19.46	0.21	0.14	0.31
Figure 8: We present some failure cases of Prompt Diffusion when generalizing to new, unseen tasks under ambiguous text guidance.
extensive experiments to evaluate its performance. Our results demonstrate that Prompt Diffusion has a promising generalization ability, not only on trained tasks but also on unseen ones. Additionally, Prompt Diffusion enables controllable image editing, including style, artistic medium, and contextual changes, without introducing excessive undesired modifications.
While our model has demonstrated notable successes in in-context learning, it is important to acknowledge its limitations. Currently, the visual scope of our model is restricted by the limited number of real-life images in the training dataset used for finetuning. As our base dataset [4] is synthesized by Stable Diffusion with proper filtering, it is difficult for the model to generate high-fidelity, real-life images. Moreover, our model is only trained jointly across six pre-defined tasks, and we believe that expanding the range of joint tasks would enhance Prompt Diffusion's in-context learning capability. As shown in the last row of Figure 9, it is difficult to predict the behavior of our model when the query image is misaligned in type with the source image in the example pair, and as shown in Figure 8, there are instances where our model struggles with new tasks. Prompt Diffusion is also limited when it comes to executing physical transformation tasks like rotation and flipping without additional fine-tuning specific to these tasks. To unlock more of Prompt Diffusion's in-context learning ability, it is possible to change the finetuning setting to training from scratch, but this would require significant computational resources, which presently limits its feasibility for us.
Prompt Diffusion represents a pioneering effort in unlocking the in-context learning capacity of diffusion-based models. We hope that this work will encourage more researchers to explore the potential of diffusion-based in-context learning and contribute to the advancement of this exciting field.
Z. Wang and M. Zhou acknowledge the support of NSF-IIS 2212418, NIH-R37 CA271186, and the NSF AI Institute for Foundations of Machine Learning (IFML).
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35: 23716-23736, 2022.
[2] Mohammadreza Armandpour, Huangjie Zheng, Ali Sadeghian, Amir Sadeghian, and Mingyuan Zhou. Re-imagine the negative prompt algorithm: Transform 2d diffusion into 3d, alleviate janus problem and beyond. arXiv preprint arXiv:2304.04968, 2023.
A Ablation Study
We conduct an ablation study to investigate the effects of vision-language prompt, which consists of the example pair, text guidance, and image query. We show the generation results in Figure 9. In the first row, we keep the text guidance and image query fixed, while we use different random images to construct the example pairs. The results suggest that Prompt Diffusion is not sensitive to what images are used to construct the example pairs, while it cares about the underlying correlation between the images inside the example pairs. In the second row, compared to the first row, we only modify the image query part, and we could see the output images are dramatically changed but consistent with the new image queries. In the third row, apart from the first row, we modify the text guidance and observe the output images are modified accordingly. In the last row, we investigate the impact of using an image query that is intentionally set to be misaligned in type with the source image of the example image pair. Our results show that in such cases, the model's behavior often becomes difficult to predict. Overall, with our vision-language prompt where the image query is aligned with the source image in the example image pair, the example pair tells the model what the specific task is, the image query provides the image domain condition for the generation, and the text guidance allows diverse semantic modifications.
Figure 9: Ablation study for different variations of the vision-language prompt in Prompt Diffusion. The last row shows when intentionally mismatching the type of the query image and that of the source image in the example pair, the behavior of Prompt Diffusion becomes difficult to predict.
B Comparison to ControlNet
We qualitatively compare Prompt Diffusion with ControlNet [69]. We follow the guidance of ControlNet, finetune a ControlNet, from the same Stable Diffusion checkpoint, on our dataset for each inverse task independently as our baseline, and we call it CN(FT). We show the comparison results in Figure 10. We observe the jointly trained Prompt Diffusion performs comparably well to the independently trained ControlNet, which implies no significant performance change with multi-task learning applied.
We further evaluate the generation ability of the task-specific ControlNet in Figure 11. We directly apply the independently finetuned ControlNet on one new task during inference. We could observe a large number of failures when directly applying ControlNet to new, unseen tasks.
Figure 10: We show the comparison of Prompt Diffusion with CN(FT), ControlNet that is finetuned specifically for each individual task, on the three inverse tasks.
C Model Architecture
We build Prompt Diffusion based on ControlNet [69]. We show the detailed model architecture in Figure 12. Note we lock the parameters of Stable Diffusion encoder blocks for finetuning to inherit the encoder capability that Stable Diffusion learns from large-scale pre-training. All the other parameters are open for finetuning. The Stable Diffusion branch takes images in the latent space as inputs, while the ControlNet branch takes the original images as inputs. The pretrained image encoder and decoder, which map images to latent space and re-map latent images to original images, are not shown in the illustration. The latent images are in four channels and eight times smaller in both height and width.
As mentioned in the main paper, our ControlNet branch takes three images as inputs. Two of them form the example pair, which is first concatenated in the RGB channel and then encoded by stacked convolution layers. The third image represents the image query and is encoded with independent stacked convolution layers. We show the architecture of the stacked convolution layers in Figure 13. We encode the example pair and the image query to the same dimensional latent embeddings and then sum them up as the inputs for the ControlNet branch.
D Potential Social Implications
The development of image generation models has the potential to have negative social impacts. These models can create realistic images that are difficult to distinguish from real photographs, which could lead to a proliferation of fake images used for malicious purposes. For example, these models could be used to create fake images of people committing crimes or engaging in immoral behavior, which could damage reputations and lead to false accusations. Additionally, the widespread use of image generation models could contribute to the erosion of trust in visual evidence, which could have implications for legal proceedings, journalism, and other fields that rely on visual evidence. There is also a risk that these models could perpetuate biases and stereotypes, as they may learn from biased datasets and generate images that reinforce those biases. It is therefore important to carefully consider the ethical implications of image generation models and take steps to mitigate their potential negative impacts.
Figure 11: We show that a ControlNet finetuned for an individual task on the dataset fails to generate high-quality images for another task, which validates the necessity of multi-task training.
E More Examples
We show more exemples of applying Prompt Diffusion here.
Figure 12: We show the detailed model architecture of Prompt Diffusion.
Figure 13: Stacked Convolution Layers.
"Suzy Parker in Larvin-Castillo, photographed by Georges Dambier for ELLE, 1952. #vantage #1950s #fashion"
Suzy Parker photographed by Georges Dombier for ELLE # fashion, with a pink dress and red hair .
"Suzy Parker photographed by Georges Dombier for ELLE # fashion, with a pair of glasses"
Figure 14: More Depth-to-Image Examples of Prompt Diffusion.
Figure 15: More Hed-to-Image Examples of Prompt Diffusion.
"Ritter hvanhoe Artwork by Otto Erdmann Classic Paintings"
"Ritter Ivanhoe Artwork by Otto Erdmann Classic Paintings, on a mountain"
Figure 16: More Seg-to-Image Examples of Prompt Diffusion.