HuggingFaceM4
/

idefics-80b

@@ -171,39 +171,19 @@ We perform checkpoint selection based on validation sets of TODO, and select the
 TODO: beautiful plots of shots scaling laws.
-| 80B IDEFIX             | 0-shots   | 4-shots   | 8-shots   | 16-shots   | 32-shots   |
-|:-----------------------|:----------|:----------|:----------|:-----------|:-----------|
-| VQAv2                  | 60.0%     | 63.4%     | 64.5%     | 65.4%      | 66.0%      |
-| OKVQA                  | 45.2%     | 52.3%     | 55.2%     | 56.8%      | 58.0%      |
-| TextVQA                | 30.9%     | 34.7%     | 35.4%     | 36.3%      | 37.0%      |
-| TextCaps               | 56.8%     | 77.9%     | 82.5%     | 85.2%      | 86.1%      |
-| Coco                   | 91.8%     | 109.3%    | 113.9%    | 116.6%     | 116.5%     |
-| NoCaps                 | 65.0%     | 101.1%    | 104.7%    | 105.6%     | 106.3%     |
-| Flickr                 | 53.7%     | 68.9%     | 74.3%     | 76.8%      | 78.9%      |
-| ImageNet1k             | 74.3%     |           |           |            |            |
-| VizWiz                 | 36.0%     | 45.8%     | 49.3%     | 51.5%      | 52.6%      |
-| VisDial (NDCG)         | 48.8%     | 48.6%     | 48.1%     |            |            |
-| HatefulMemes (ROC AUC) | 60.6%     | 58.7%     | 57.8%     | 56.0%      | 54.3%      |
-| ScienceQA (accuracy)   | 68.9%     | 66.3%     | -         | -          | -          |
-| RenderedSST2           | 60.5%     | 63.9%     | 64.3%     | 66.9%      | 68.0%      |
-| 9B IDEFIX              | 0-shots   | 4-shots   | 8-shots   | 16-shots   | 32-shots   |
-|:-----------------------|:----------|:----------|:----------|:-----------|:-----------|
-| VQAv2                  | 50.9%     | 55.6%     | 56.4%     | 57.2%      | 57.9%      |
-| OKVQA                  | 38.4%     | 45.8%     | 47.3%     | 49.0%      | 50.4%      |
-| TextVQA                | 25.9%     | 26.8%     | 26.8%     | 28.1%      | 28.2%      |
-| TextCaps               | 25.4%     | 60.9%     | 63.7%     | 68.0%      | 69.7%      |
-| Coco                   | 46.0%     | 88.9%     | 96.9%     | 99.6%      | 101.5%     |
-| NoCaps                 | 36.8%     | 78.5%     | 84.3%     | 87.2%      | 88.6%      |
-| Flickr                 | 27.3%     | 52.2%     | 60.3%     | 65.0%      | 66.0%      |
-| ImageNet1k             | 70.7%     |           |           |            |            |
-| VizWiz                 | 35.5%     | 42.0%     | 42.8%     | 45.0%      | 45.9%      |
-| VisDial (NDCG)         | 48.7%     | 48.1%     | 47.5%     |            |            |
-| HatefulMemes (ROC AUC) | 51.7%     | 52.6%     | 52.3%     | 52.5%      | 53.1%      |
-| ScienceQA (accuracy)   | 44.2%     | 41.6%     | -         | -          | -          |
-| RenderedSST2           | 61.8%     | 60.6%     | 66.8%     | 66.0%      | 63.4%      |
 # Technical Specifications

 TODO: beautiful plots of shots scaling laws.
+| Model      |   Shots |   VQAv2 (open-ended vqa acc) |   OKVQA (open-ended vqa acc) |   TextVQA (open-ended vqa acc) |   VizWiz (open-ended vqa acc) |   TextCaps (CIDEr) |   Coco (CIDEr) |   NoCaps (CIDEr) |   Flickr (CIDEr) |   ImageNet1k (accuracy) |   VisDial (NDCG) |   HatefulMemes (ROC AUC) |   ScienceQA (accuracy) |   RenderedSST2 (accuracy) |   Winoground (group_score (image_score/text_score)) |
+|:-----------|--------:|-----------------------------:|-----------------------------:|-------------------------------:|------------------------------:|-------------------:|---------------:|-----------------:|-----------------:|------------------------:|-----------------:|-------------------------:|-----------------------:|--------------------------:|----------------------------------------------------:|
+| IDEFIX 80B |       0 |                         60.1 |                         45.2 |                           31   |                          36   |               56.9 |           91.9 |             65   |             53.7 |                    74.3 |             48.9 |                     60.7 |                   69   |                      60.6 |                                                   8 (0.1875/0.225) |
+|            |       4 |                         63.5 |                         52.4 |                           34.7 |                          45.9 |               77.9 |          109.3 |            101.1 |             69   |                    -   |             48.7 |                     58.7 |                   66.3 |                      64   |                                                  - |
+|            |       8 |                         64.6 |                         55.3 |                           35.5 |                          49.3 |               82.6 |          114   |            104.8 |             74.4 |                    -   |             48.2 |                     57.9 |                   -   |                      64.3 |                                                  - |
+|            |      16 |                         65.4 |                         56.9 |                           36.4 |                          51.6 |               85.2 |          116.6 |            105.7 |             76.9 |                    -   |             -   |                     56.1 |                   -   |                      66.9 |                                                  - |
+|            |      32 |                         66   |                         58.1 |                           37   |                          52.6 |               86.1 |          116.5 |            106.4 |             79   |                    -   |             -   |                     54.4 |                   -   |                      68.1 |                                                  - |
+<br>
+| IDEFIX 9B  |       0 |                         50.9 |                         38.5 |                           25.9 |                          35.6 |               25.4 |           46.1 |             36.9 |             27.3 |                    70.7 |             48.8 |                     51.8 |                   44.3 |                      61.9 |                                                   5 (16.75/20.75)|
+|            |       4 |                         55.6 |                         45.9 |                           26.8 |                          42   |               60.9 |           89   |             78.5 |             52.3 |                    -   |             48.2 |                     52.6 |                   41.7 |                      60.6 |                                                  - |
+|            |       8 |                         56.5 |                         47.4 |                           26.9 |                          42.9 |               63.8 |           97   |             84.4 |             60.3 |                    -   |             47.5 |                     52.3 |                   -   |                      66.8 |                                                  - |
+|            |      16 |                         57.2 |                         49.1 |                           28.1 |                          45   |               68.1 |           99.6 |             87.2 |             65   |                    -   |             -   |                     52.6 |                   -   |                      66   |                                                  - |
+|            |      32 |                         58   |                         50.5 |                           28.2 |                          46   |               69.7 |          101.5 |             88.7 |             66   |                    -   |             -   |                     53.1 |                   -   |                      63.5 |                                                  - |
 # Technical Specifications