Hugging Face's logo

HuggingFaceM4
/
idefics-80b

Text Generation
Transformers
PyTorch
Safetensors
English
idefics
pretraining
multimodal
text
image
image-to-text
text-generation-inference
10 papers
Model card Files Community
23
Leyo HF staff commited on
Commit
e23b75c
•
1 Parent(s): de16541

fix significant digits in table

Browse files
Files changed (1) hide show
  1. README.md +12 -11
README.md CHANGED
@@ -174,17 +174,18 @@ We perform checkpoint selection based on validation sets of TODO, and select the
174
  TODO: beautiful plots of shots scaling laws.
175
 
176
  | Model | Shots | VQAv2 (OE VQA acc) | OKVQA (OE VQA acc) | TextVQA (OE VQA acc) | VizWiz (OE VQA acc) | TextCaps (CIDEr) | Coco (CIDEr) | NoCaps (CIDEr) | Flickr (CIDEr) | ImageNet1k (accuracy) | VisDial (NDCG) | HatefulMemes (ROC AUC) | ScienceQA (accuracy) | RenderedSST2 (accuracy) | Winoground (group (text/image)) |
177
- |:-----------|--------:|---------------------:|---------------------:|-----------------------:|----------------------:|-------------------:|---------------:|-----------------:|-----------------:|------------------------:|-----------------:|-------------------------:|-----------------------:|--------------------------:|----------------------------------------------------:|
178
- | IDEFICS 80B | 0 | 60 | 45.2 | 30.9 | 36 | 56.8 | 91.8 | 65 | 53.7 | 74.3 | 48.8 | 60.6 | 68.9 | 60.5 | 8 (18.8/22.5)|
179
- | | 4 | 63.4 | 52.3 | 34.7 | 45.8 | 77.9 | 109.3 | 101.1 | 68.9 | - | 48.6 | 58.7 | 66.3 | 63.9 | - |
180
- | | 8 | 64.5 | 55.2 | 35.4 | 49.3 | 82.5 | 113.9 | 104.7 | 74.3 | - | 48.1 | 57.8 | - | 64.3 | - |
181
- | | 16 | 65.4 | 56.8 | 36.3 | 51.5 | 85.2 | 116.6 | 105.6 | 76.8 | - | - | 56 | - | 66.9 | - |
182
- | | 32 | 66 | 58 | 37 | 52.6 | 86.1 | 116.5 | 106.3 | 78.9 | - | - | 54.3 | - | 68 | - |
183
- | IDEFICS 9B | 0 | 50.9 | 38.4 | 25.9 | 35.5 | 25.4 | 46 | 36.8 | 27.3 | 70.7 | 48.7 | 51.7 | 44.2 | 61.8 | 5 (16.8/20.8)|
184
- | | 4 | 55.6 | 45.8 | 26.8 | 42 | 60.8 | 88.9 | 78.4 | 52.2 | - | 48.1 | 52.6 | 41.6 | 60.6 | - |
185
- | | 8 | 56.4 | 47.3 | 26.8 | 42.8 | 63.7 | 96.9 | 84.3 | 60.3 | - | 47.5 | 52.3 | - | 66.8 | - |
186
- | | 16 | 57.2 | 49 | 28.1 | 45 | 68 | 99.6 | 87.2 | 65 | - | - | 52.5 | - | 66 | - |
187
- | | 32 | 57.9 | 50.4 | 28.2 | 45.9 | 69.7 | 101.5 | 88.6 | 66 | - | - | 53.1 | - | 63.4 | - |
 
188
 
189
  # Technical Specifications
190
 
 
174
  TODO: beautiful plots of shots scaling laws.
175
 
176
  | Model | Shots | VQAv2 (OE VQA acc) | OKVQA (OE VQA acc) | TextVQA (OE VQA acc) | VizWiz (OE VQA acc) | TextCaps (CIDEr) | Coco (CIDEr) | NoCaps (CIDEr) | Flickr (CIDEr) | ImageNet1k (accuracy) | VisDial (NDCG) | HatefulMemes (ROC AUC) | ScienceQA (accuracy) | RenderedSST2 (accuracy) | Winoground (group (text/image)) |
177
+ |:-----------|--------:|---------------------:|---------------------:|-----------------------:|----------------------:|-------------------:|---------------:|-----------------:|-----------------:|------------------------:|-----------------:|-------------------------:|-----------------------:|--------------------------:|----------------------------------:|
178
+ | IDEFIX 80B | 0 | 60.0 | 45.2 | 30.9 | 36.0 | 56.8 | 91.8 | 65.0 | 53.7 | 74.3 | 48.8 | 60.6 | 68.9 | 60.5 | 8.0 (18.8/22.5)|
179
+ | | 4 | 63.4 | 52.3 | 34.7 | 45.8 | 77.9 | 109.3 | 101.1 | 68.9 | - | 48.6 | 58.7 | 66.3 | 63.9 | - |
180
+ | | 8 | 64.5 | 55.2 | 35.4 | 49.3 | 82.5 | 113.9 | 104.7 | 74.3 | - | 48.1 | 57.8 | - | 64.3 | - |
181
+ | | 16 | 65.4 | 56.8 | 36.3 | 51.5 | 85.2 | 116.6 | 105.6 | 76.8 | - | - | 56.0 | - | 66.9 | - |
182
+ | | 32 | 66.0 | 58.0 | 37.0 | 52.6 | 86.1 | 116.5 | 106.3 | 78.9 | - | - | 54.3 | - | 68.0 | - |
183
+ <br>
184
+ | IDEFIX 9B | 0 | 50.9 | 38.4 | 25.9 | 35.5 | 25.4 | 46.0 | 36.8 | 27.3 | 70.7 | 48.7 | 51.7 | 44.2 | 61.8 | 5.0 (16.8/20.8)|
185
+ | | 4 | 55.6 | 45.8 | 26.8 | 42.0 | 60.8 | 88.9 | 78.4 | 52.2 | - | 48.1 | 52.6 | 41.6 | 60.6 | - |
186
+ | | 8 | 56.4 | 47.3 | 26.8 | 42.8 | 63.7 | 96.9 | 84.3 | 60.3 | - | 47.5 | 52.3 | - | 66.8 | - |
187
+ | | 16 | 57.2 | 49.0 | 28.1 | 45.0 | 68.0 | 99.6 | 87.2 | 65.0 | - | - | 52.5 | - | 66.0 | - |
188
+ | | 32 | 57.9 | 50.4 | 28.2 | 45.9 | 69.7 | 101.5 | 88.6 | 66.0 | - | - | 53.1 | - | 63.4 | - |
189
 
190
  # Technical Specifications
191