Hugging Face's logo

HuggingFaceM4
/
idefics-80b

Text Generation
Transformers
PyTorch
Safetensors
4 datasets
English
idefics
pretraining
multimodal
text
image
image-to-text
text-generation-inference
10 papers
Model card Files Community
23
Leyo commited on
Commit
0b721e1
•
1 Parent(s): 052d671

fix winoground numbers

Browse files
Files changed (1) hide show
  1. README.md +2 -2
README.md CHANGED
@@ -173,13 +173,13 @@ TODO: beautiful plots of shots scaling laws.
173
 
174
  | Model | Shots | VQAv2 (open-ended vqa acc) | OKVQA (open-ended vqa acc) | TextVQA (open-ended vqa acc) | VizWiz (open-ended vqa acc) | TextCaps (CIDEr) | Coco (CIDEr) | NoCaps (CIDEr) | Flickr (CIDEr) | ImageNet1k (accuracy) | VisDial (NDCG) | HatefulMemes (ROC AUC) | ScienceQA (accuracy) | RenderedSST2 (accuracy) | Winoground (group_score (image_score/text_score)) |
175
  |:-----------|--------:|-----------------------------:|-----------------------------:|-------------------------------:|------------------------------:|-------------------:|---------------:|-----------------:|-----------------:|------------------------:|-----------------:|-------------------------:|-----------------------:|--------------------------:|----------------------------------------------------:|
176
- | IDEFIX 80B | 0 | 60.1 | 45.2 | 31 | 36 | 56.9 | 91.9 | 65 | 53.7 | 74.3 | 48.9 | 60.7 | 69 | 60.6 | 8 (0.1875/0.225) |
177
  | | 4 | 63.5 | 52.4 | 34.7 | 45.9 | 77.9 | 109.3 | 101.1 | 69 | - | 48.7 | 58.7 | 66.3 | 64 | - |
178
  | | 8 | 64.6 | 55.3 | 35.5 | 49.3 | 82.6 | 114 | 104.8 | 74.4 | - | 48.2 | 57.9 | - | 64.3 | - |
179
  | | 16 | 65.4 | 56.9 | 36.4 | 51.6 | 85.2 | 116.6 | 105.7 | 76.9 | - | - | 56.1 | - | 66.9 | - |
180
  | | 32 | 66 | 58.1 | 37 | 52.6 | 86.1 | 116.5 | 106.4 | 79 | - | - | 54.4 | - | 68.1 | - |
181
  <br>
182
- | IDEFIX 9B | 0 | 50.9 | 38.5 | 25.9 | 35.6 | 25.4 | 46.1 | 36.9 | 27.3 | 70.7 | 48.8 | 51.8 | 44.3 | 61.9 | 5 (16.75/20.75)|
183
  | | 4 | 55.6 | 45.9 | 26.8 | 42 | 60.9 | 89 | 78.5 | 52.3 | - | 48.2 | 52.6 | 41.7 | 60.6 | - |
184
  | | 8 | 56.5 | 47.4 | 26.9 | 42.9 | 63.8 | 97 | 84.4 | 60.3 | - | 47.5 | 52.3 | - | 66.8 | - |
185
  | | 16 | 57.2 | 49.1 | 28.1 | 45 | 68.1 | 99.6 | 87.2 | 65 | - | - | 52.6 | - | 66 | - |
 
173
 
174
  | Model | Shots | VQAv2 (open-ended vqa acc) | OKVQA (open-ended vqa acc) | TextVQA (open-ended vqa acc) | VizWiz (open-ended vqa acc) | TextCaps (CIDEr) | Coco (CIDEr) | NoCaps (CIDEr) | Flickr (CIDEr) | ImageNet1k (accuracy) | VisDial (NDCG) | HatefulMemes (ROC AUC) | ScienceQA (accuracy) | RenderedSST2 (accuracy) | Winoground (group_score (image_score/text_score)) |
175
  |:-----------|--------:|-----------------------------:|-----------------------------:|-------------------------------:|------------------------------:|-------------------:|---------------:|-----------------:|-----------------:|------------------------:|-----------------:|-------------------------:|-----------------------:|--------------------------:|----------------------------------------------------:|
176
+ | IDEFIX 80B | 0 | 60.1 | 45.2 | 31 | 36 | 56.9 | 91.9 | 65 | 53.7 | 74.3 | 48.9 | 60.7 | 69 | 60.6 | 8 (18.8/22.5) |
177
  | | 4 | 63.5 | 52.4 | 34.7 | 45.9 | 77.9 | 109.3 | 101.1 | 69 | - | 48.7 | 58.7 | 66.3 | 64 | - |
178
  | | 8 | 64.6 | 55.3 | 35.5 | 49.3 | 82.6 | 114 | 104.8 | 74.4 | - | 48.2 | 57.9 | - | 64.3 | - |
179
  | | 16 | 65.4 | 56.9 | 36.4 | 51.6 | 85.2 | 116.6 | 105.7 | 76.9 | - | - | 56.1 | - | 66.9 | - |
180
  | | 32 | 66 | 58.1 | 37 | 52.6 | 86.1 | 116.5 | 106.4 | 79 | - | - | 54.4 | - | 68.1 | - |
181
  <br>
182
+ | IDEFIX 9B | 0 | 50.9 | 38.5 | 25.9 | 35.6 | 25.4 | 46.1 | 36.9 | 27.3 | 70.7 | 48.8 | 51.8 | 44.3 | 61.9 | 5 (16.8/20.8)|
183
  | | 4 | 55.6 | 45.9 | 26.8 | 42 | 60.9 | 89 | 78.5 | 52.3 | - | 48.2 | 52.6 | 41.7 | 60.6 | - |
184
  | | 8 | 56.5 | 47.4 | 26.9 | 42.9 | 63.8 | 97 | 84.4 | 60.3 | - | 47.5 | 52.3 | - | 66.8 | - |
185
  | | 16 | 57.2 | 49.1 | 28.1 | 45 | 68.1 | 99.6 | 87.2 | 65 | - | - | 52.6 | - | 66 | - |