File size: 5,154 Bytes
7c16991 9f6a4c3 7c16991 9f6a4c3 7c16991 9f6a4c3 7c16991 9f6a4c3 7c16991 3b67d79 7c16991 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 |
---
license: apache-2.0
---
# Virtual Artist (InstructG2I: Synthesizing Images from Multimodal Attributed Graphs - NeurIPs 2024)
<a href=''><img src='https://img.shields.io/badge/Project-Page-green'></a>
## Introduction
We propose a graph context-conditioned diffusion model called **InstructG2I** to generate images from multimodal attributed graphs (MMAGs).
InstructG2I first exploits the graph structure and multimodal information to conduct informative neighbor sampling by combining personalized page rank and re-ranking based on vision-language features.
Then, a Graph-QFormer encoder adaptively encodes the graph nodes into an auxiliary set of graph prompts to guide the denoising process of diffusion.
Finally, we propose graph classifier-free guidance, enabling controllable generation by varying the strength of graph guidance and multiple connected edges to a node.
## Installation
```bash
conda create --name instructg2i python==3.10
conda activate instructg2i
git clone https://github.com/PeterGriffinJin/InstructG2I.git
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia
pip install -e .
```
## Quick Start
Generate a picture called *a mountain in the blue sky* under Claude Monet's style.
```python
import os
from PIL import Image
from instructg2i import InstructG2IPipeline, get_neighbor_transforms
text_prompt = 'a mountain in the blue sky' # a man playing soccer, a man playing piano
neighbor_pic_dir = 'examples/monet_pictures'
neighbor_transforms = get_neighbor_transforms(resolution=256)
pipeline = InstructG2IPipeline.from_pretrained("PeterJinGo/VirtualArtist", neighbor_num=5, device='cuda:0')
neighbor_image = [neighbor_transforms(Image.open(f'{neighbor_pic_dir}/{n_file}').convert("RGB")) for n_file in os.listdir(neighbor_pic_dir)]
image_gen = pipeline(prompt=text_prompt, neighbor_image=neighbor_image, neighbor_mask=[1] * len(neighbor_image), num_inference_steps=100).images[0]
image_gen.show()
```
<!--  -->
<div align="center">
<img src="figs/example.png" alt="arch" width="300" height="300" />
</div>
Generate a picture called *a house in the snow* combining the style of Claude Monet and my little brother.
```python
import os
from PIL import Image
from instructg2i import image_grid, InstructG2IMultiGuidePipeline, get_neighbor_transforms
# load the model
pipeline = InstructG2IMultiGuidePipeline.from_pretrained("PeterJinGo/VirtualArtist", neighbor_num=5, device='cuda:0')
# configuration
text_prompt = 'a house in the snow' # a man playing soccer, a man playing piano
scale_as = [0, 3, 10]
scale_bs = [0, 5, 15]
# read the sampled neighbors
path1 = "examples/monet_pictures"
path2 = "examples/children_pictures"
neighbor_images = [[neighbor_transforms(Image.open(os.path.join(path1, n_file)).convert("RGB")) for n_file in os.listdir(path1)],
[neighbor_transforms(Image.open(os.path.join(path2, n_file)).convert("RGB")) for n_file in os.listdir(path2)]]
neighbor_masks = [[1,1,1,1,1],
[1,1,1,1,1]]
# generation
image_gens = []
neighbor_transforms = get_neighbor_transforms(resolution=256)
for scale_a in scale_as:
for scale_b in scale_bs:
graph_guidance_scales = [scale_a, scale_b]
image_gen = pipeline(prompt=text_prompt,
neighbor_images=neighbor_images,
neighbor_masks=neighbor_masks,
graph_guidance_scales=graph_guidance_scales,
num_inference_steps=100).images[0]
image_gens.append(image_gen)
res_grid = image_grid(image_gens, len(scale_as), len(scale_bs))
res_grid.show()
```
<!--  -->
<div align="center">
<img src="figs/example2.png" alt="arch" width="400" height="400" />
</div>
## Download Models
### Image Encoder
Create an image_encoder folder by ```mkdir image_encoder```, then place the files downloaded [here](https://drive.google.com/drive/folders/1AtbN401MDSVLZlH5webITfskkIIjUPLZ?usp=sharing) into the folder.
### InstructG2I checkpoints
The virtual artist InstructG2I checkpoint which is trained on Artwork graphs can be downloaded [here](https://drive.google.com/drive/folders/1ntmPgZXmb-M-k5M0Cnh34p0fxoeKnnYC?usp=sharing) or [here](https://huggingface.co/PeterJinGo/VirtualArtist).
```python
from huggingface_hub import snapshot_download
snapshot_download(repo_id="PeterJinGo/VirtualArtist", local_dir=your_local_path)
```
The InstructG2I checkpoints for the Amazon graph and Goodreads graph can be found [here](https://drive.google.com/drive/folders/1rPhc-LFoyqDrqn6gigTogFB7cEpRLx72?usp=sharing).
## Citations
If you find InstructG2I useful for your research and applications, please cite using this BibTeX:
```bibtex
@article{jin2024instructg2i,
title={InstructG2I: Synthesizing Images from Multimodal Attributed Graphs},
author={Jin, Bowen and Pang, Ziqi and Guo, Bingjun and Wang, Yu-Xiong and You, Jiaxuan and Han, Jiawei},
journal={arXiv preprint arXiv:2410.07157},
year={2024}
}
```
|