aayushgs commited on
Commit
2d892dd
1 Parent(s): 4d804f4

upload initial

Browse files
README.md ADDED
@@ -0,0 +1,148 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ pipeline_tag: image-to-text
3
+ tags:
4
+ - image-captioning
5
+ languages:
6
+ - en
7
+ license: bsd-3-clause
8
+ ---
9
+
10
+ # BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
11
+
12
+ Model card for image captioning pretrained on COCO dataset - base architecture (with ViT large backbone).
13
+
14
+ | ![BLIP.gif](https://cdn-uploads.huggingface.co/production/uploads/1670928184033-62441d1d9fdefb55a0b7d12c.gif) |
15
+ |:--:|
16
+ | <b> Pull figure from BLIP official repo | Image source: https://github.com/salesforce/BLIP </b>|
17
+
18
+ ## TL;DR
19
+
20
+ Authors from the [paper](https://arxiv.org/abs/2201.12086) write in the abstract:
21
+
22
+ *Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. However, most existing pre-trained models only excel in either understanding-based tasks or generation-based tasks. Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision. In this paper, we propose BLIP, a new VLP framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones. We achieve state-of-the-art results on a wide range of vision-language tasks, such as image-text retrieval (+2.7% in average recall@1), image captioning (+2.8% in CIDEr), and VQA (+1.6% in VQA score). BLIP also demonstrates strong generalization ability when directly transferred to videolanguage tasks in a zero-shot manner. Code, models, and datasets are released.*
23
+
24
+ ## Usage
25
+
26
+ You can use this model for conditional and un-conditional image captioning
27
+
28
+ ### Using the Pytorch model
29
+
30
+ #### Running the model on CPU
31
+
32
+ <details>
33
+ <summary> Click to expand </summary>
34
+
35
+ ```python
36
+ import requests
37
+ from PIL import Image
38
+ from transformers import BlipProcessor, BlipForConditionalGeneration
39
+
40
+ processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-large")
41
+ model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-large")
42
+
43
+ img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg'
44
+ raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')
45
+
46
+ # conditional image captioning
47
+ text = "a photography of"
48
+ inputs = processor(raw_image, text, return_tensors="pt")
49
+
50
+ out = model.generate(**inputs)
51
+ print(processor.decode(out[0], skip_special_tokens=True))
52
+
53
+ # unconditional image captioning
54
+ inputs = processor(raw_image, return_tensors="pt")
55
+
56
+ out = model.generate(**inputs)
57
+ print(processor.decode(out[0], skip_special_tokens=True))
58
+ ```
59
+ </details>
60
+
61
+ #### Running the model on GPU
62
+
63
+ ##### In full precision
64
+
65
+ <details>
66
+ <summary> Click to expand </summary>
67
+
68
+ ```python
69
+ import requests
70
+ from PIL import Image
71
+ from transformers import BlipProcessor, BlipForConditionalGeneration
72
+
73
+ processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-large")
74
+ model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-large").to("cuda")
75
+
76
+ img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg'
77
+ raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')
78
+
79
+ # conditional image captioning
80
+ text = "a photography of"
81
+ inputs = processor(raw_image, text, return_tensors="pt").to("cuda")
82
+
83
+ out = model.generate(**inputs)
84
+ print(processor.decode(out[0], skip_special_tokens=True))
85
+
86
+ # unconditional image captioning
87
+ inputs = processor(raw_image, return_tensors="pt").to("cuda")
88
+
89
+ out = model.generate(**inputs)
90
+ print(processor.decode(out[0], skip_special_tokens=True))
91
+ ```
92
+ </details>
93
+
94
+ ##### In half precision (`float16`)
95
+
96
+ <details>
97
+ <summary> Click to expand </summary>
98
+
99
+ ```python
100
+ import torch
101
+ import requests
102
+ from PIL import Image
103
+ from transformers import BlipProcessor, BlipForConditionalGeneration
104
+
105
+ processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-large")
106
+ model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-large", torch_dtype=torch.float16).to("cuda")
107
+
108
+ img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg'
109
+ raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')
110
+
111
+ # conditional image captioning
112
+ text = "a photography of"
113
+ inputs = processor(raw_image, text, return_tensors="pt").to("cuda", torch.float16)
114
+
115
+ out = model.generate(**inputs)
116
+ print(processor.decode(out[0], skip_special_tokens=True))
117
+ # >>> a photography of a woman and her dog
118
+
119
+ # unconditional image captioning
120
+ inputs = processor(raw_image, return_tensors="pt").to("cuda", torch.float16)
121
+
122
+ out = model.generate(**inputs)
123
+ print(processor.decode(out[0], skip_special_tokens=True))
124
+ >>> a woman sitting on the beach with her dog
125
+ ```
126
+ </details>
127
+
128
+ ## BibTex and citation info
129
+
130
+ ```
131
+ @misc{https://doi.org/10.48550/arxiv.2201.12086,
132
+ doi = {10.48550/ARXIV.2201.12086},
133
+
134
+ url = {https://arxiv.org/abs/2201.12086},
135
+
136
+ author = {Li, Junnan and Li, Dongxu and Xiong, Caiming and Hoi, Steven},
137
+
138
+ keywords = {Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences},
139
+
140
+ title = {BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation},
141
+
142
+ publisher = {arXiv},
143
+
144
+ year = {2022},
145
+
146
+ copyright = {Creative Commons Attribution 4.0 International}
147
+ }
148
+ ```
config.json ADDED
@@ -0,0 +1,170 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_commit_hash": null,
3
+ "architectures": [
4
+ "BlipForConditionalGeneration"
5
+ ],
6
+ "image_text_hidden_size": 256,
7
+ "initializer_factor": 1.0,
8
+ "logit_scale_init_value": 2.6592,
9
+ "model_type": "blip",
10
+ "projection_dim": 512,
11
+ "text_config": {
12
+ "_name_or_path": "",
13
+ "add_cross_attention": false,
14
+ "architectures": null,
15
+ "attention_probs_dropout_prob": 0.0,
16
+ "bad_words_ids": null,
17
+ "begin_suppress_tokens": null,
18
+ "bos_token_id": 30522,
19
+ "chunk_size_feed_forward": 0,
20
+ "cross_attention_hidden_size": null,
21
+ "decoder_start_token_id": null,
22
+ "diversity_penalty": 0.0,
23
+ "do_sample": false,
24
+ "early_stopping": false,
25
+ "encoder_hidden_size": 1024,
26
+ "encoder_no_repeat_ngram_size": 0,
27
+ "eos_token_id": 2,
28
+ "exponential_decay_length_penalty": null,
29
+ "finetuning_task": null,
30
+ "forced_bos_token_id": null,
31
+ "forced_eos_token_id": null,
32
+ "hidden_act": "gelu",
33
+ "hidden_dropout_prob": 0.0,
34
+ "hidden_size": 768,
35
+ "id2label": {
36
+ "0": "LABEL_0",
37
+ "1": "LABEL_1"
38
+ },
39
+ "initializer_factor": 1.0,
40
+ "initializer_range": 0.02,
41
+ "intermediate_size": 3072,
42
+ "is_decoder": true,
43
+ "is_encoder_decoder": false,
44
+ "label2id": {
45
+ "LABEL_0": 0,
46
+ "LABEL_1": 1
47
+ },
48
+ "layer_norm_eps": 1e-12,
49
+ "length_penalty": 1.0,
50
+ "max_length": 20,
51
+ "max_position_embeddings": 512,
52
+ "min_length": 0,
53
+ "model_type": "blip_text_model",
54
+ "no_repeat_ngram_size": 0,
55
+ "num_attention_heads": 12,
56
+ "num_beam_groups": 1,
57
+ "num_beams": 1,
58
+ "num_hidden_layers": 12,
59
+ "num_return_sequences": 1,
60
+ "output_attentions": false,
61
+ "output_hidden_states": false,
62
+ "output_scores": false,
63
+ "pad_token_id": 0,
64
+ "prefix": null,
65
+ "problem_type": null,
66
+ "projection_dim": 768,
67
+ "pruned_heads": {},
68
+ "remove_invalid_values": false,
69
+ "repetition_penalty": 1.0,
70
+ "return_dict": true,
71
+ "return_dict_in_generate": false,
72
+ "sep_token_id": 102,
73
+ "suppress_tokens": null,
74
+ "task_specific_params": null,
75
+ "temperature": 1.0,
76
+ "tf_legacy_loss": false,
77
+ "tie_encoder_decoder": false,
78
+ "tie_word_embeddings": true,
79
+ "tokenizer_class": null,
80
+ "top_k": 50,
81
+ "top_p": 1.0,
82
+ "torch_dtype": null,
83
+ "torchscript": false,
84
+ "transformers_version": "4.26.0.dev0",
85
+ "typical_p": 1.0,
86
+ "use_bfloat16": false,
87
+ "use_cache": true,
88
+ "vocab_size": 30524
89
+ },
90
+ "torch_dtype": "float32",
91
+ "transformers_version": null,
92
+ "vision_config": {
93
+ "_name_or_path": "",
94
+ "add_cross_attention": false,
95
+ "architectures": null,
96
+ "attention_dropout": 0.0,
97
+ "bad_words_ids": null,
98
+ "begin_suppress_tokens": null,
99
+ "bos_token_id": null,
100
+ "chunk_size_feed_forward": 0,
101
+ "cross_attention_hidden_size": null,
102
+ "decoder_start_token_id": null,
103
+ "diversity_penalty": 0.0,
104
+ "do_sample": false,
105
+ "dropout": 0.0,
106
+ "early_stopping": false,
107
+ "encoder_no_repeat_ngram_size": 0,
108
+ "eos_token_id": null,
109
+ "exponential_decay_length_penalty": null,
110
+ "finetuning_task": null,
111
+ "forced_bos_token_id": null,
112
+ "forced_eos_token_id": null,
113
+ "hidden_act": "gelu",
114
+ "hidden_size": 1024,
115
+ "id2label": {
116
+ "0": "LABEL_0",
117
+ "1": "LABEL_1"
118
+ },
119
+ "image_size": 384,
120
+ "initializer_factor": 1.0,
121
+ "initializer_range": 0.02,
122
+ "intermediate_size": 4096,
123
+ "is_decoder": false,
124
+ "is_encoder_decoder": false,
125
+ "label2id": {
126
+ "LABEL_0": 0,
127
+ "LABEL_1": 1
128
+ },
129
+ "layer_norm_eps": 1e-05,
130
+ "length_penalty": 1.0,
131
+ "max_length": 20,
132
+ "min_length": 0,
133
+ "model_type": "blip_vision_model",
134
+ "no_repeat_ngram_size": 0,
135
+ "num_attention_heads": 16,
136
+ "num_beam_groups": 1,
137
+ "num_beams": 1,
138
+ "num_channels": 3,
139
+ "num_hidden_layers": 24,
140
+ "num_return_sequences": 1,
141
+ "output_attentions": false,
142
+ "output_hidden_states": false,
143
+ "output_scores": false,
144
+ "pad_token_id": null,
145
+ "patch_size": 16,
146
+ "prefix": null,
147
+ "problem_type": null,
148
+ "projection_dim": 512,
149
+ "pruned_heads": {},
150
+ "remove_invalid_values": false,
151
+ "repetition_penalty": 1.0,
152
+ "return_dict": true,
153
+ "return_dict_in_generate": false,
154
+ "sep_token_id": null,
155
+ "suppress_tokens": null,
156
+ "task_specific_params": null,
157
+ "temperature": 1.0,
158
+ "tf_legacy_loss": false,
159
+ "tie_encoder_decoder": false,
160
+ "tie_word_embeddings": true,
161
+ "tokenizer_class": null,
162
+ "top_k": 50,
163
+ "top_p": 1.0,
164
+ "torch_dtype": null,
165
+ "torchscript": false,
166
+ "transformers_version": "4.26.0.dev0",
167
+ "typical_p": 1.0,
168
+ "use_bfloat16": false
169
+ }
170
+ }
handler.py ADDED
@@ -0,0 +1,59 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import requests
2
+ from typing import Dict, Any
3
+ from PIL import Image
4
+ import torch
5
+ import base64
6
+ from io import BytesIO
7
+ from transformers import BlipForConditionalGeneration, BlipProcessor
8
+
9
+ device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
10
+
11
+
12
+ class EndpointHandler:
13
+ def __init__(self, path=""):
14
+ self.processor = BlipProcessor.from_pretrained(
15
+ "Salesforce/blip-image-captioning-large"
16
+ )
17
+ self.model = BlipForConditionalGeneration.from_pretrained(
18
+ "Salesforce/blip-image-captioning-large"
19
+ ).to(device)
20
+ self.model.eval()
21
+
22
+ def __call__(self, data: Dict[str, Any]) -> Dict[str, Any]:
23
+ input_data = data.get("inputs", {})
24
+ encoded_images = input_data.get("images")
25
+
26
+ if not encoded_images:
27
+ return {"captions": [], "error": "No images provided"}
28
+
29
+ texts = input_data.get("texts", ["a photography of"] * len(encoded_images))
30
+
31
+ try:
32
+ raw_images = [
33
+ Image.open(BytesIO(base64.b64decode(img))).convert("RGB")
34
+ for img in encoded_images
35
+ ]
36
+ processed_inputs = [
37
+ self.processor(image, text, return_tensors="pt")
38
+ for image, text in zip(raw_images, texts)
39
+ ]
40
+ processed_inputs = {
41
+ "pixel_values": torch.cat(
42
+ [inp["pixel_values"] for inp in processed_inputs], dim=0
43
+ ).to(device),
44
+ "input_ids": torch.cat(
45
+ [inp["input_ids"] for inp in processed_inputs], dim=0
46
+ ).to(device),
47
+ "attention_mask": torch.cat(
48
+ [inp["attention_mask"] for inp in processed_inputs], dim=0
49
+ ).to(device),
50
+ }
51
+
52
+ with torch.no_grad():
53
+ out = self.model.generate(**processed_inputs)
54
+
55
+ captions = self.processor.batch_decode(out, skip_special_tokens=True)
56
+ return {"captions": captions}
57
+ except Exception as e:
58
+ print(f"Error during processing: {str(e)}")
59
+ return {"captions": [], "error": str(e)}
preprocessor_config.json ADDED
@@ -0,0 +1,25 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "do_normalize": true,
3
+ "do_pad": true,
4
+ "do_rescale": true,
5
+ "do_resize": true,
6
+ "image_mean": [
7
+ 0.48145466,
8
+ 0.4578275,
9
+ 0.40821073
10
+ ],
11
+ "image_processor_type": "BlipImageProcessor",
12
+ "image_std": [
13
+ 0.26862954,
14
+ 0.26130258,
15
+ 0.27577711
16
+ ],
17
+ "processor_class": "BlipProcessor",
18
+ "resample": 3,
19
+ "rescale_factor": 0.00392156862745098,
20
+ "size": {
21
+ "height": 384,
22
+ "width": 384
23
+ },
24
+ "size_divisor": 32
25
+ }
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:66c8aec8d91b5e74b86bde343ed95fb53e9ccf9ffc77c5093890df662d234e04
3
+ size 1879143921
special_tokens_map.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "cls_token": "[CLS]",
3
+ "mask_token": "[MASK]",
4
+ "pad_token": "[PAD]",
5
+ "sep_token": "[SEP]",
6
+ "unk_token": "[UNK]"
7
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cls_token": "[CLS]",
3
+ "do_basic_tokenize": true,
4
+ "do_lower_case": true,
5
+ "mask_token": "[MASK]",
6
+ "model_max_length": 512,
7
+ "name_or_path": "Salesforce/blip-image-captioning-large",
8
+ "never_split": null,
9
+ "pad_token": "[PAD]",
10
+ "processor_class": "BlipProcessor",
11
+ "sep_token": "[SEP]",
12
+ "special_tokens_map_file": null,
13
+ "strip_accents": null,
14
+ "tokenize_chinese_chars": true,
15
+ "tokenizer_class": "BertTokenizer",
16
+ "unk_token": "[UNK]",
17
+ "model_input_names": [
18
+ "input_ids",
19
+ "attention_mask"
20
+ ]
21
+ }
vocab.txt ADDED
The diff for this file is too large to render. See raw diff