asi commited on
Commit
0b1e25d
•
1 Parent(s): 69f945b

:books: add readme

Browse files
Files changed (1) hide show
  1. README.md +140 -0
README.md CHANGED
@@ -1,3 +1,143 @@
1
  ---
 
 
 
 
 
 
 
 
 
2
  license: apache-2.0
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ language:
3
+ - fr
4
+
5
+ thumbnail: https://raw.githubusercontent.com/AntoineSimoulin/gpt-fr/main/imgs/logo.png
6
+ tags:
7
+ - tf
8
+ - pytorch
9
+ - gpt2
10
+ - text-to-image
11
  license: apache-2.0
12
  ---
13
+
14
+ <img src="https://raw.githubusercontent.com/AntoineSimoulin/gpt-fr/main/imgs/igpt-logo.png" width="200">
15
+
16
+ ## Model description
17
+
18
+ **iGPT-fr** 🇫🇷 is a GPT model for French pre-trained incremental language model developped by the [Laboratoire de Linguistique Formelle (LLF)](http://www.llf.cnrs.fr/en). We adapted [GPT-fr 🇫🇷](https://huggingface.co/asi/gpt-fr-cased-base) model to generate images conditionned by text inputs.
19
+
20
+ ## Intended uses & limitations
21
+
22
+ The model can be leveraged for image generation tasks. The model is currently under a developpment phase.
23
+
24
+ #### How to use
25
+
26
+ The model might be used through the 🤗 `Transformers` librairie. You will also need to install the `Taming Transformers` library for high-resolution image synthesis:
27
+
28
+ ```bash
29
+ pip install git+https://github.com/CompVis/taming-transformers.git
30
+ ```
31
+
32
+ ```python
33
+ from transformers import GPT2Tokenizer, GPT2LMHeadModel
34
+ from huggingface_hub import hf_hub_download
35
+ from omegaconf import OmegaConf
36
+ from taming.models import vqgan
37
+ import torch
38
+ from PIL import Image
39
+ import numpy as np
40
+
41
+ # Load VQGAN model
42
+ vqgan_ckpt = hf_hub_download(repo_id="boris/vqgan_f16_16384", filename="model.ckpt", force_download=False)
43
+ vqgan_config = hf_hub_download(repo_id="boris/vqgan_f16_16384", filename="config.yaml", force_download=False)
44
+
45
+ config = OmegaConf.load(vqgan_config)
46
+ vqgan_model = vqgan.VQModel(**config.model.params)
47
+ vqgan_model.eval().requires_grad_(False)
48
+ vqgan_model.init_from_ckpt(vqgan_ckpt)
49
+
50
+ # Load pretrained model
51
+ model = GPT2LMHeadModel.from_pretrained("asi/igpt-fr-cased-base")
52
+ model.eval()
53
+ tokenizer = GPT2Tokenizer.from_pretrained("asi/igpt-fr-cased-base")
54
+
55
+ # Generate a sample of text
56
+ input_sentence = "Une carte de l'europe"
57
+ input_ids = tokenizer.encode(input_sentence, return_tensors='pt')
58
+ input_ids = torch.cat((input_ids, torch.tensor([[50000]])), 1) # Add image generation token
59
+
60
+ greedy_output = model.generate(
61
+ input_ids.to(device),
62
+ max_length=256+input_ids.shape[1],
63
+ do_sample=True,
64
+ top_p=0.92,
65
+ top_k=0)
66
+
67
+ def custom_to_pil(x):
68
+ x = x.detach().cpu()
69
+ x = torch.clamp(x, -1., 1.)
70
+ x = (x + 1.)/2.
71
+ x = x.permute(1,2,0).numpy()
72
+ x = (255*x).astype(np.uint8)
73
+ x = Image.fromarray(x)
74
+ if not x.mode == "RGB":
75
+ x = x.convert("RGB")
76
+ return x
77
+
78
+ z_idx = greedy_output[0, input_ids.shape[1]:] - 50001
79
+ z_quant = vqgan_model.quantize.get_codebook_entry(z_idx, shape=(1, 16, 16, 256))
80
+ x_rec = vqgan_model.decode(z_quant).to('cpu')[0]
81
+ display(custom_to_pil(x_rec))
82
+ ```
83
+
84
+ You may also filter results based on CLIP:
85
+
86
+ ```python
87
+ from tqdm import tqdm
88
+
89
+ def hallucinate(prompt, num_images=64):
90
+ input_ids = tokenizer.encode(prompt, return_tensors='pt')
91
+ input_ids = torch.cat((input_ids, torch.tensor([[50000]])), 1).to(device) # Add image generation token
92
+
93
+ all_images = []
94
+ for i in tqdm(range(num_images)):
95
+ greedy_output = model.generate(
96
+ input_ids.to(device),
97
+ max_length=256+input_ids.shape[1],
98
+ do_sample=True,
99
+ top_p=0.92,
100
+ top_k=0)
101
+
102
+ z_idx = greedy_output[0, input_ids.shape[1]:] - 50001
103
+ z_quant = vqgan_model.quantize.get_codebook_entry(z_idx, shape=(1, 16, 16, 256))
104
+ x_rec = vqgan_model.decode(z_quant).to('cpu')[0]
105
+ all_images.append(custom_to_pil(x_rec))
106
+ return all_images
107
+
108
+ input_sentence = "Une carte de l'europe"
109
+ all_images = hallucinate(input_sentence)
110
+
111
+ from transformers import pipeline
112
+
113
+ opus_model = "Helsinki-NLP/opus-mt-fr-en"
114
+ opus_translator = pipeline("translation", model=opus_model)
115
+
116
+ opus_translator(input_sentence)
117
+
118
+ from transformers import CLIPProcessor, CLIPModel
119
+
120
+ clip_model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
121
+ clip_processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
122
+
123
+ def clip_top_k(prompt, images, k=8):
124
+ prompt_fr = opus_translator(input_sentence)[0]['translation_text']
125
+ inputs = clip_processor(text=prompt_fr, images=images, return_tensors="pt", padding=True)
126
+ outputs = clip_model(**inputs)
127
+ logits = outputs.logits_per_text # this is the image-text similarity score
128
+ scores = np.array(logits[0].detach()).argsort()[-k:][::-1]
129
+ return [images[score] for score in scores]
130
+
131
+ filtered_images = clip_top_k(input_sentence, all_images)
132
+
133
+ for fi in filtered_images:
134
+ display(fi)
135
+ ```
136
+
137
+ ## Training data
138
+
139
+ We created a dedicated corpus to train our generative model. The training corpus consists in text-image pairs. We aggregated portions from existing corpora: [Laion-5B](https://laion.ai/blog/laion-5b/) and [WIT](https://github.com/google-research-datasets/wit). The final dataset includes 10,807,534 samples.
140
+
141
+ ## Training procedure
142
+
143
+ We pre-trained the model on the new CNRS (French National Centre for Scientific Research) [Jean Zay](http://www.idris.fr/eng/jean-zay/) supercomputer. We perform the training within a total of 140 hours of computation on Tesla V-100 hardware (TDP of 300W). The training was distributed on 8 compute nodes of 8 GPUs. We used data parallelization in order to divide each micro-batch on the computing units. We estimated the total emissions at 1161.22 kgCO2eq, using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al., (2019)](lacoste-2019).