PD0AUTOMATIONAL nielsr HF staff commited on
Commit
39d7e8c
0 Parent(s):

Duplicate from Salesforce/blip2-opt-6.7b-coco

Browse files

Co-authored-by: Niels Rogge <nielsr@users.noreply.huggingface.co>

.gitattributes ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tflite filter=lfs diff=lfs merge=lfs -text
29
+ *.tgz filter=lfs diff=lfs merge=lfs -text
30
+ *.wasm filter=lfs diff=lfs merge=lfs -text
31
+ *.xz filter=lfs diff=lfs merge=lfs -text
32
+ *.zip filter=lfs diff=lfs merge=lfs -text
33
+ *.zst filter=lfs diff=lfs merge=lfs -text
34
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,60 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: en
3
+ license: mit
4
+ tags:
5
+ - vision
6
+ - image-to-text
7
+ - image-captioning
8
+ - visual-question-answering
9
+ pipeline_tag: image-to-text
10
+ duplicated_from: Salesforce/blip2-opt-6.7b-coco
11
+ ---
12
+
13
+ # BLIP-2, OPT-6.7b, fine-tuned on COCO
14
+
15
+ BLIP-2 model, leveraging [OPT-6.7b](https://huggingface.co/facebook/opt-6.7b) (a large language model with 6.7 billion parameters).
16
+ It was introduced in the paper [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://arxiv.org/abs/2301.12597) by Li et al. and first released in [this repository](https://github.com/salesforce/LAVIS/tree/main/projects/blip2).
17
+
18
+ Disclaimer: The team releasing BLIP-2 did not write a model card for this model so this model card has been written by the Hugging Face team.
19
+
20
+ ## Model description
21
+
22
+ BLIP-2 consists of 3 models: a CLIP-like image encoder, a Querying Transformer (Q-Former) and a large language model.
23
+
24
+ The authors initialize the weights of the image encoder and large language model from pre-trained checkpoints and keep them frozen
25
+ while training the Querying Transformer, which is a BERT-like Transformer encoder that maps a set of "query tokens" to query embeddings,
26
+ which bridge the gap between the embedding space of the image encoder and the large language model.
27
+
28
+ The goal for the model is simply to predict the next text token, giving the query embeddings and the previous text.
29
+
30
+ <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/blip2_architecture.jpg"
31
+ alt="drawing" width="600"/>
32
+
33
+ This allows the model to be used for tasks like:
34
+
35
+ - image captioning
36
+ - visual question answering (VQA)
37
+ - chat-like conversations by feeding the image and the previous conversation as prompt to the model
38
+
39
+ ## Direct Use and Downstream Use
40
+
41
+ You can use the raw model for conditional text generation given an image and optional text. See the [model hub](https://huggingface.co/models?search=Salesforce/blip) to look for
42
+ fine-tuned versions on a task that interests you.
43
+
44
+ ## Bias, Risks, Limitations, and Ethical Considerations
45
+
46
+ BLIP2-OPT uses off-the-shelf OPT as the language model. It inherits the same risks and limitations as mentioned in Meta's model card.
47
+
48
+ > Like other large language models for which the diversity (or lack thereof) of training
49
+ > data induces downstream impact on the quality of our model, OPT-175B has limitations in terms
50
+ > of bias and safety. OPT-175B can also have quality issues in terms of generation diversity and
51
+ > hallucination. In general, OPT-175B is not immune from the plethora of issues that plague modern
52
+ > large language models.
53
+ >
54
+ BLIP2 is fine-tuned on image-text datasets (e.g. [LAION](https://laion.ai/blog/laion-400-open-dataset/) ) collected from the internet. As a result the model itself is potentially vulnerable to generating equivalently inappropriate content or replicating inherent biases in the underlying data.
55
+
56
+ BLIP2 has not been tested in real world applications. It should not be directly deployed in any applications. Researchers should first carefully assess the safety and fairness of the model in relation to the specific context they’re being deployed within.
57
+
58
+ ### How to use
59
+
60
+ For code examples, we refer to the [documentation](https://huggingface.co/docs/transformers/main/en/model_doc/blip-2#transformers.Blip2ForConditionalGeneration.forward.example).
config.json ADDED
@@ -0,0 +1,255 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_commit_hash": null,
3
+ "architectures": [
4
+ "Blip2ForConditionalGeneration"
5
+ ],
6
+ "initializer_factor": 1.0,
7
+ "initializer_range": 0.02,
8
+ "model_type": "blip-2",
9
+ "num_query_tokens": 32,
10
+ "qformer_config": {
11
+ "_name_or_path": "",
12
+ "add_cross_attention": false,
13
+ "architectures": null,
14
+ "attention_probs_dropout_prob": 0.1,
15
+ "bad_words_ids": null,
16
+ "begin_suppress_tokens": null,
17
+ "bos_token_id": null,
18
+ "chunk_size_feed_forward": 0,
19
+ "classifier_dropout": null,
20
+ "cross_attention_frequency": 2,
21
+ "cross_attention_hidden_size": null,
22
+ "decoder_start_token_id": null,
23
+ "diversity_penalty": 0.0,
24
+ "do_sample": false,
25
+ "early_stopping": false,
26
+ "encoder_hidden_size": 1408,
27
+ "encoder_no_repeat_ngram_size": 0,
28
+ "eos_token_id": null,
29
+ "exponential_decay_length_penalty": null,
30
+ "finetuning_task": null,
31
+ "forced_bos_token_id": null,
32
+ "forced_eos_token_id": null,
33
+ "hidden_act": "gelu",
34
+ "hidden_dropout_prob": 0.1,
35
+ "hidden_size": 768,
36
+ "id2label": {
37
+ "0": "LABEL_0",
38
+ "1": "LABEL_1"
39
+ },
40
+ "initializer_range": 0.02,
41
+ "intermediate_size": 3072,
42
+ "is_decoder": false,
43
+ "is_encoder_decoder": false,
44
+ "label2id": {
45
+ "LABEL_0": 0,
46
+ "LABEL_1": 1
47
+ },
48
+ "layer_norm_eps": 1e-12,
49
+ "length_penalty": 1.0,
50
+ "max_length": 20,
51
+ "max_position_embeddings": 512,
52
+ "min_length": 0,
53
+ "model_type": "blip_2_qformer",
54
+ "no_repeat_ngram_size": 0,
55
+ "num_attention_heads": 12,
56
+ "num_beam_groups": 1,
57
+ "num_beams": 1,
58
+ "num_hidden_layers": 12,
59
+ "num_return_sequences": 1,
60
+ "output_attentions": false,
61
+ "output_hidden_states": false,
62
+ "output_scores": false,
63
+ "pad_token_id": 0,
64
+ "position_embedding_type": "absolute",
65
+ "prefix": null,
66
+ "problem_type": null,
67
+ "pruned_heads": {},
68
+ "remove_invalid_values": false,
69
+ "repetition_penalty": 1.0,
70
+ "return_dict": true,
71
+ "return_dict_in_generate": false,
72
+ "sep_token_id": null,
73
+ "suppress_tokens": null,
74
+ "task_specific_params": null,
75
+ "temperature": 1.0,
76
+ "tf_legacy_loss": false,
77
+ "tie_encoder_decoder": false,
78
+ "tie_word_embeddings": true,
79
+ "tokenizer_class": null,
80
+ "top_k": 50,
81
+ "top_p": 1.0,
82
+ "torch_dtype": null,
83
+ "torchscript": false,
84
+ "transformers_version": "4.27.0.dev0",
85
+ "typical_p": 1.0,
86
+ "use_bfloat16": false,
87
+ "vocab_size": 30522
88
+ },
89
+ "text_config": {
90
+ "_name_or_path": "",
91
+ "_remove_final_layer_norm": false,
92
+ "activation_dropout": 0.0,
93
+ "activation_function": "relu",
94
+ "add_cross_attention": false,
95
+ "architectures": [
96
+ "OPTForCausalLM"
97
+ ],
98
+ "attention_dropout": 0.0,
99
+ "bad_words_ids": null,
100
+ "begin_suppress_tokens": null,
101
+ "bos_token_id": 2,
102
+ "chunk_size_feed_forward": 0,
103
+ "cross_attention_hidden_size": null,
104
+ "decoder_start_token_id": null,
105
+ "diversity_penalty": 0.0,
106
+ "do_layer_norm_before": true,
107
+ "do_sample": false,
108
+ "dropout": 0.1,
109
+ "early_stopping": false,
110
+ "enable_bias": true,
111
+ "encoder_no_repeat_ngram_size": 0,
112
+ "eos_token_id": 50118,
113
+ "exponential_decay_length_penalty": null,
114
+ "ffn_dim": 16384,
115
+ "finetuning_task": null,
116
+ "forced_bos_token_id": null,
117
+ "forced_eos_token_id": null,
118
+ "hidden_size": 4096,
119
+ "id2label": {
120
+ "0": "LABEL_0",
121
+ "1": "LABEL_1"
122
+ },
123
+ "init_std": 0.02,
124
+ "is_decoder": false,
125
+ "is_encoder_decoder": false,
126
+ "label2id": {
127
+ "LABEL_0": 0,
128
+ "LABEL_1": 1
129
+ },
130
+ "layer_norm_elementwise_affine": true,
131
+ "layerdrop": 0.0,
132
+ "length_penalty": 1.0,
133
+ "max_length": 20,
134
+ "max_position_embeddings": 2048,
135
+ "min_length": 0,
136
+ "model_type": "opt",
137
+ "no_repeat_ngram_size": 0,
138
+ "num_attention_heads": 32,
139
+ "num_beam_groups": 1,
140
+ "num_beams": 1,
141
+ "num_hidden_layers": 32,
142
+ "num_return_sequences": 1,
143
+ "output_attentions": false,
144
+ "output_hidden_states": false,
145
+ "output_scores": false,
146
+ "pad_token_id": 1,
147
+ "prefix": "</s>",
148
+ "problem_type": null,
149
+ "pruned_heads": {},
150
+ "remove_invalid_values": false,
151
+ "repetition_penalty": 1.0,
152
+ "return_dict": true,
153
+ "return_dict_in_generate": false,
154
+ "sep_token_id": null,
155
+ "suppress_tokens": null,
156
+ "task_specific_params": null,
157
+ "temperature": 1.0,
158
+ "tf_legacy_loss": false,
159
+ "tie_encoder_decoder": false,
160
+ "tie_word_embeddings": true,
161
+ "tokenizer_class": null,
162
+ "top_k": 50,
163
+ "top_p": 1.0,
164
+ "torch_dtype": "float16",
165
+ "torchscript": false,
166
+ "transformers_version": "4.27.0.dev0",
167
+ "typical_p": 1.0,
168
+ "use_bfloat16": false,
169
+ "use_cache": true,
170
+ "vocab_size": 50272,
171
+ "word_embed_proj_dim": 4096
172
+ },
173
+ "torch_dtype": "float32",
174
+ "transformers_version": null,
175
+ "use_decoder_only_language_model": true,
176
+ "vision_config": {
177
+ "_name_or_path": "",
178
+ "add_cross_attention": false,
179
+ "architectures": null,
180
+ "attention_dropout": 0.0,
181
+ "bad_words_ids": null,
182
+ "begin_suppress_tokens": null,
183
+ "bos_token_id": null,
184
+ "chunk_size_feed_forward": 0,
185
+ "cross_attention_hidden_size": null,
186
+ "decoder_start_token_id": null,
187
+ "diversity_penalty": 0.0,
188
+ "do_sample": false,
189
+ "dropout": 0.0,
190
+ "early_stopping": false,
191
+ "encoder_no_repeat_ngram_size": 0,
192
+ "eos_token_id": null,
193
+ "exponential_decay_length_penalty": null,
194
+ "finetuning_task": null,
195
+ "forced_bos_token_id": null,
196
+ "forced_eos_token_id": null,
197
+ "hidden_act": "gelu",
198
+ "hidden_size": 1408,
199
+ "id2label": {
200
+ "0": "LABEL_0",
201
+ "1": "LABEL_1"
202
+ },
203
+ "image_size": 364,
204
+ "initializer_factor": 1.0,
205
+ "initializer_range": 1e-10,
206
+ "intermediate_size": 6144,
207
+ "is_decoder": false,
208
+ "is_encoder_decoder": false,
209
+ "label2id": {
210
+ "LABEL_0": 0,
211
+ "LABEL_1": 1
212
+ },
213
+ "layer_norm_eps": 1e-05,
214
+ "length_penalty": 1.0,
215
+ "max_length": 20,
216
+ "min_length": 0,
217
+ "model_type": "blip_2_vision_model",
218
+ "no_repeat_ngram_size": 0,
219
+ "num_attention_heads": 16,
220
+ "num_beam_groups": 1,
221
+ "num_beams": 1,
222
+ "num_channels": 3,
223
+ "num_hidden_layers": 39,
224
+ "num_return_sequences": 1,
225
+ "output_attentions": false,
226
+ "output_hidden_states": false,
227
+ "output_scores": false,
228
+ "pad_token_id": null,
229
+ "patch_size": 14,
230
+ "prefix": null,
231
+ "problem_type": null,
232
+ "projection_dim": 512,
233
+ "pruned_heads": {},
234
+ "qkv_bias": true,
235
+ "remove_invalid_values": false,
236
+ "repetition_penalty": 1.0,
237
+ "return_dict": true,
238
+ "return_dict_in_generate": false,
239
+ "sep_token_id": null,
240
+ "suppress_tokens": null,
241
+ "task_specific_params": null,
242
+ "temperature": 1.0,
243
+ "tf_legacy_loss": false,
244
+ "tie_encoder_decoder": false,
245
+ "tie_word_embeddings": true,
246
+ "tokenizer_class": null,
247
+ "top_k": 50,
248
+ "top_p": 1.0,
249
+ "torch_dtype": null,
250
+ "torchscript": false,
251
+ "transformers_version": "4.27.0.dev0",
252
+ "typical_p": 1.0,
253
+ "use_bfloat16": false
254
+ }
255
+ }
merges.txt ADDED
The diff for this file is too large to render. See raw diff
preprocessor_config.json ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "do_convert_rgb": true,
3
+ "do_normalize": true,
4
+ "do_rescale": true,
5
+ "do_resize": true,
6
+ "image_mean": [
7
+ 0.48145466,
8
+ 0.4578275,
9
+ 0.40821073
10
+ ],
11
+ "image_processor_type": "BlipImageProcessor",
12
+ "image_std": [
13
+ 0.26862954,
14
+ 0.26130258,
15
+ 0.27577711
16
+ ],
17
+ "processor_class": "Blip2Processor",
18
+ "resample": 3,
19
+ "rescale_factor": 0.00392156862745098,
20
+ "size": {
21
+ "height": 364,
22
+ "width": 364
23
+ }
24
+ }
pytorch_model-00001-of-00004.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ba3e3ca5886b10af6acbb024062ac98d94804341ad785aa2caff556b94749d58
3
+ size 9801631972
pytorch_model-00002-of-00004.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2108f77ba152c5921e7f1a357c5f59ea71e084c250252e491d20f1b343b8c46c
3
+ size 9934790547
pytorch_model-00003-of-00004.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1e6a38430e6d4b0c3cd75a332b190dbcfd15c0c86f0ada4df144c58cc8e7e337
3
+ size 9934842089
pytorch_model-00004-of-00004.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b1d19ef7fbc256eadf222b928821245d29b2eafd708482266396c499dc8a52fe
3
+ size 2166169773
pytorch_model.bin.index.json ADDED
The diff for this file is too large to render. See raw diff
special_tokens_map.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "</s>",
4
+ "lstrip": false,
5
+ "normalized": true,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "eos_token": {
10
+ "content": "</s>",
11
+ "lstrip": false,
12
+ "normalized": true,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "pad_token": {
17
+ "content": "<pad>",
18
+ "lstrip": false,
19
+ "normalized": true,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ },
23
+ "unk_token": {
24
+ "content": "</s>",
25
+ "lstrip": false,
26
+ "normalized": true,
27
+ "rstrip": false,
28
+ "single_word": false
29
+ }
30
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
tokenizer_config.json ADDED
@@ -0,0 +1,41 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_bos_token": true,
3
+ "add_prefix_space": false,
4
+ "bos_token": {
5
+ "__type": "AddedToken",
6
+ "content": "</s>",
7
+ "lstrip": false,
8
+ "normalized": true,
9
+ "rstrip": false,
10
+ "single_word": false
11
+ },
12
+ "eos_token": {
13
+ "__type": "AddedToken",
14
+ "content": "</s>",
15
+ "lstrip": false,
16
+ "normalized": true,
17
+ "rstrip": false,
18
+ "single_word": false
19
+ },
20
+ "errors": "replace",
21
+ "model_max_length": 1000000000000000019884624838656,
22
+ "pad_token": {
23
+ "__type": "AddedToken",
24
+ "content": "<pad>",
25
+ "lstrip": false,
26
+ "normalized": true,
27
+ "rstrip": false,
28
+ "single_word": false
29
+ },
30
+ "processor_class": "Blip2Processor",
31
+ "special_tokens_map_file": null,
32
+ "tokenizer_class": "GPT2Tokenizer",
33
+ "unk_token": {
34
+ "__type": "AddedToken",
35
+ "content": "</s>",
36
+ "lstrip": false,
37
+ "normalized": true,
38
+ "rstrip": false,
39
+ "single_word": false
40
+ }
41
+ }
vocab.json ADDED
The diff for this file is too large to render. See raw diff