aliencaocao commited on
Commit
89119e7
1 Parent(s): 4917244

Upload folder using huggingface_hub

Browse files
README.md ADDED
@@ -0,0 +1,110 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ tags:
4
+ - vision
5
+ widget:
6
+ - src: https://huggingface.co/datasets/mishig/sample_images/resolve/main/cat-dog-music.png
7
+ candidate_labels: playing music, playing sports
8
+ example_title: Cat & Dog
9
+ ---
10
+
11
+ # SigLIP (large-sized model)
12
+
13
+ SigLIP model pre-trained on WebLi at resolution 384x384. It was introduced in the paper [Sigmoid Loss for Language Image Pre-Training](https://arxiv.org/abs/2303.15343) by Zhai et al. and first released in [this repository](https://github.com/google-research/big_vision).
14
+
15
+ Disclaimer: The team releasing SigLIP did not write a model card for this model so this model card has been written by the Hugging Face team.
16
+
17
+ ## Model description
18
+
19
+ SigLIP is [CLIP](https://huggingface.co/docs/transformers/model_doc/clip), a multimodal model, with a better loss function. The sigmoid loss operates solely on image-text pairs and does not require a global view of the pairwise similarities for normalization. This allows further scaling up the batch size, while also performing better at smaller batch sizes.
20
+
21
+ A TLDR of SigLIP by one of the authors can be found [here](https://twitter.com/giffmana/status/1692641733459267713).
22
+
23
+ ## Intended uses & limitations
24
+
25
+ You can use the raw model for tasks like zero-shot image classification and image-text retrieval. See the [model hub](https://huggingface.co/models?search=google/siglip) to look for
26
+ other versions on a task that interests you.
27
+
28
+ ### How to use
29
+
30
+ Here is how to use this model to perform zero-shot image classification:
31
+
32
+ ```python
33
+ from PIL import Image
34
+ import requests
35
+ from transformers import AutoProcessor, AutoModel
36
+ import torch
37
+
38
+ model = AutoModel.from_pretrained("google/siglip-large-patch16-384")
39
+ processor = AutoProcessor.from_pretrained("google/siglip-large-patch16-384")
40
+
41
+ url = "http://images.cocodataset.org/val2017/000000039769.jpg"
42
+ image = Image.open(requests.get(url, stream=True).raw)
43
+
44
+ texts = ["a photo of 2 cats", "a photo of 2 dogs"]
45
+ inputs = processor(text=texts, images=image, padding="max_length", return_tensors="pt")
46
+
47
+ with torch.no_grad():
48
+ outputs = model(**inputs)
49
+
50
+ logits_per_image = outputs.logits_per_image
51
+ probs = torch.sigmoid(logits_per_image) # these are the probabilities
52
+ print(f"{probs[0][0]:.1%} that image 0 is '{texts[0]}'")
53
+ ```
54
+
55
+ Alternatively, one can leverage the pipeline API which abstracts away the complexity for the user:
56
+
57
+ ```python
58
+ from transformers import pipeline
59
+ from PIL import Image
60
+ import requests
61
+
62
+ # load pipe
63
+ image_classifier = pipeline(task="zero-shot-image-classification", model="google/siglip-large-patch16-384")
64
+
65
+ # load image
66
+ url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
67
+ image = Image.open(requests.get(url, stream=True).raw)
68
+
69
+ # inference
70
+ outputs = image_classifier(image, candidate_labels=["2 cats", "a plane", "a remote"])
71
+ outputs = [{"score": round(output["score"], 4), "label": output["label"] } for output in outputs]
72
+ print(outputs)
73
+ ```
74
+ For more code examples, we refer to the [documentation](https://huggingface.co/transformers/main/model_doc/siglip.html#).
75
+
76
+ ## Training procedure
77
+
78
+ ### Training data
79
+
80
+ SigLIP is pre-trained on the English image-text pairs of the WebLI dataset [(Chen et al., 2023)](https://arxiv.org/abs/2209.06794).
81
+
82
+ ### Preprocessing
83
+
84
+ Images are resized/rescaled to the same resolution (384x384) and normalized across the RGB channels with mean (0.5, 0.5, 0.5) and standard deviation (0.5, 0.5, 0.5).
85
+
86
+ Texts are tokenized and padded to the same length (64 tokens).
87
+
88
+ ### Compute
89
+
90
+ The model was trained on 16 TPU-v4 chips for three days.
91
+
92
+ ## Evaluation results
93
+
94
+ Evaluation of SigLIP compared to CLIP is shown below (taken from the paper).
95
+
96
+ <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/siglip_table.jpeg"
97
+ alt="drawing" width="600"/>
98
+
99
+ ### BibTeX entry and citation info
100
+
101
+ ```bibtex
102
+ @misc{zhai2023sigmoid,
103
+ title={Sigmoid Loss for Language Image Pre-Training},
104
+ author={Xiaohua Zhai and Basil Mustafa and Alexander Kolesnikov and Lucas Beyer},
105
+ year={2023},
106
+ eprint={2303.15343},
107
+ archivePrefix={arXiv},
108
+ primaryClass={cs.CV}
109
+ }
110
+ ```
config.json ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "SiglipModel"
4
+ ],
5
+ "initializer_factor": 1.0,
6
+ "model_type": "siglip",
7
+ "text_config": {
8
+ "hidden_size": 1024,
9
+ "intermediate_size": 4096,
10
+ "model_type": "siglip_text_model",
11
+ "num_attention_heads": 16,
12
+ "num_hidden_layers": 24
13
+ },
14
+ "torch_dtype": "float32",
15
+ "transformers_version": "4.37.0.dev0",
16
+ "vision_config": {
17
+ "hidden_size": 1024,
18
+ "image_size": 384,
19
+ "intermediate_size": 4096,
20
+ "model_type": "siglip_vision_model",
21
+ "num_attention_heads": 16,
22
+ "num_hidden_layers": 24
23
+ }
24
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1d20cb4db4d01c674572146bb93e686a3bdf9a6c75334dad6653d02afd469d43
3
+ size 2610010520
preprocessor_config.json ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "do_normalize": true,
3
+ "do_rescale": true,
4
+ "do_resize": true,
5
+ "image_mean": [
6
+ 0.5,
7
+ 0.5,
8
+ 0.5
9
+ ],
10
+ "image_processor_type": "SiglipImageProcessor",
11
+ "image_std": [
12
+ 0.5,
13
+ 0.5,
14
+ 0.5
15
+ ],
16
+ "processor_class": "SiglipProcessor",
17
+ "resample": 3,
18
+ "rescale_factor": 0.00392156862745098,
19
+ "size": {
20
+ "height": 384,
21
+ "width": 384
22
+ }
23
+ }
special_tokens_map.json ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "eos_token": {
3
+ "content": "</s>",
4
+ "lstrip": true,
5
+ "normalized": false,
6
+ "rstrip": true,
7
+ "single_word": false
8
+ },
9
+ "pad_token": {
10
+ "content": "</s>",
11
+ "lstrip": true,
12
+ "normalized": false,
13
+ "rstrip": true,
14
+ "single_word": false
15
+ },
16
+ "unk_token": {
17
+ "content": "<unk>",
18
+ "lstrip": true,
19
+ "normalized": false,
20
+ "rstrip": true,
21
+ "single_word": false
22
+ }
23
+ }
spiece.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1e5036bed065526c3c212dfbe288752391797c4bb1a284aa18c9a0b23fcaf8ec
3
+ size 798330
tokenizer_config.json ADDED
@@ -0,0 +1,33 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "added_tokens_decoder": {
3
+ "1": {
4
+ "content": "</s>",
5
+ "lstrip": true,
6
+ "normalized": false,
7
+ "rstrip": true,
8
+ "single_word": false,
9
+ "special": true
10
+ },
11
+ "2": {
12
+ "content": "<unk>",
13
+ "lstrip": true,
14
+ "normalized": false,
15
+ "rstrip": true,
16
+ "single_word": false,
17
+ "special": true
18
+ }
19
+ },
20
+ "additional_special_tokens": [],
21
+ "clean_up_tokenization_spaces": true,
22
+ "do_lower_case": true,
23
+ "eos_token": "</s>",
24
+ "model_input_names": [
25
+ "input_ids"
26
+ ],
27
+ "model_max_length": 64,
28
+ "pad_token": "</s>",
29
+ "processor_class": "SiglipProcessor",
30
+ "sp_model_kwargs": {},
31
+ "tokenizer_class": "SiglipTokenizer",
32
+ "unk_token": "<unk>"
33
+ }