hop

README.md

@@ -9,6 +9,15 @@ datasets:
 - pixparse/pdfa-eng-wds
 - wendlerc/RenderedText
 - HuggingFaceM4/the_cauldron
+- teknium/OpenHermes-2.5
+- GAIR/lima
+- databricks/databricks-dolly-15k
+- meta-math/MetaMathQA
+- TIGER-Lab/MathInstruct
+- microsoft/orca-math-word-problems-200k
+- camel-ai/math
+- AtlasUnified/atlas-math-sets
+- tiedong/goat
 language:
 - en
 tags:
@@ -52,7 +61,7 @@ For optimal results, we recommend fine-tuning `idefics2-8b` on one's specific us
 
 As a starting point, we provide fine-tuning codes that can be adapted for one's particular scenario:
 - With the [TRL library](https://github.com/huggingface/trl): TODO
-- With the [Hugging Face Trainer](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#api-reference%20][%20transformers.Trainer): TODO
+- With the [Hugging Face Trainer](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#api-reference%20][%20transformers.Trainer): [Tutorial notebook](https://colab.research.google.com/drive/1rm3AGquGEYXfeeizE40bbDtcWh5S4Nlq?usp=sharing)
 
 
 # Technical summary
@@ -61,7 +70,21 @@ IDEFICS-2 exhibits strong performance for a model of its size (8B parameters) wh
 
 <details><summary>For more details, expand the result table.</summary>
 
-TODO: performance table
+| Model        | Open weights | Size | # tokens per image | MMMU   (val/test)   | MathVista (testmini) | TextVQA (val) | MMBench (test)| VQAv2 (test-dev) | DocVQA (test)
+|--------------|-------------|------|--------------------|-----------|-----------|---------|---------|---------|---------|
+| DeepSeek-VL  | ✅ |  7B   | 576                | 36.6/-   | 36.1      | -       | 73.2    |  -     |   -   |
+| LLaVa-NeXT-13B   | ✅ | 13B  | 2880               | 36.2/-   | 35.3      | 67.1    | 70.0  | 82.8 |   -   |
+| LLaVa-NeXT-34B | ✅ |  34B    | 2880                  | 51.1/44.7 | 46.5  | 69.5  | 79.3    | 83.7    |   -   |   -   |
+| MM1-Chat-7B  | ❌ | 7B   | 720                | 37.0/35.6 | 35.9      | 72.8    | 72.3    |   -   |    -   |
+| MM1-Chat-30B | ❌ | 30B    | 720                  | 44.7/40.3 | 39.4  | 73.5  | 75.1    |    83.7   |       |
+| Gemini 1.0 Pro | ❌ | ? |  ?  |  47.9/-  |   45.2   |    74.6    |   -    | 71.2 |  88.1  |
+| Gemini 1.5 Pro | ❌ | ? |  ?  |  58.5/-  |   52.1   |    73.5    |   -    | 73.2 |  86.5  |
+| Claude 3 Haiku |  ❌ |? |  ?  |  50.2/-  |   46.4   |    -    |   -    | - |  88.8  |
+|      |    |                  |  |       |    |     |
+| IDEFICS-1 instruct (32-shots) | ✅ |  80B |  -  |  -  |   -   |    39.3    |   -    | 68.8 |  -  |
+|      |    |                  |  |       |    |     |
+| IDEFICS-2 (w/o image splitting) | ✅ |  8B   | 64                 | 43.5/37.9 | 51.6      | 70.4    | 76.8    | 80.8 | 67.3 |
+| IDEFICS-2 (w/ image splitting) | ✅ |  8B   | 320                | 43.0/37.7 | 51.4      | 73.0    | 76.7    | 81.2 | 74.0 |
 
 </details>
 
@@ -71,6 +94,21 @@ TODO: performance table
 - We departed from the IDEFICS-1's architecture (gated cross-attentions) and **simplified the integration of visual features** into the language backbone. The images are fed to the vision encoder followed by a learned [Perceiver](https://arxiv.org/abs/2103.03206) pooling and a MLP modality projection. That pooled sequence is then concatenated with the text embeddings to obtain an (interleaved) sequence of image(s) and text(s).
 - All of these improvements along with better pre-trained backbones yield a significant jump in performance over IDEFICS-1 for a model that is **10x smaller**.
 
+IDEFICS-2 is trained in 2 stages for maximum efficiency. In a first stage, images are fed to the model at SigLIP's native resolution (squares of 384 x 384). In the second stage, images are fed to the model at their native resolution (with a maximum of 980 and a minimum of 378) and native aspect ratio. Since high resolution is necessary for OCR data, we add PDFA, Rendered-Text, and IDL to OBELICS, LAION Coco and PMD during that second stage. 
+
+Following this, we perform instruction fine-tuning on [The Cauldron](https://huggingface.co/datasets/HuggingFaceM4/the_cauldron), a collection of 50 manually curated vision-language datasets along with 9 text-only instruction fine-tuning datasets:
+- [OpenHermes-2.5](https://huggingface.co/datasets/teknium/OpenHermes-2.5)
+- [lima](https://huggingface.co/datasets/GAIR/lima)
+- [databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k)
+- [MetaMathQA](https://huggingface.co/datasets/meta-math/MetaMathQA)
+- [MathInstruct](https://huggingface.co/datasets/TIGER-Lab/MathInstruct)
+- [orca-math-word-problems-200k](https://huggingface.co/datasets/microsoft/orca-math-word-problems-200k)
+- [math](https://huggingface.co/datasets/camel-ai/math)
+- [atlas-math-sets](https://huggingface.co/datasets/AtlasUnified/atlas-math-sets)
+- [goat](https://huggingface.co/datasets/tiedong/goat)
+
+We use Lora to train the parameters initialized from pre-trained backbones and full fine-tuning for newly initialized parameters (modality connector), as we find this strategy to be more stable as long as more computationally efficient.
+
 More details (training procedure, data selection, hyper-parameters, etc.) along with lessons learned from our ablations will be available in an upcoming technical report.
 
 
@@ -199,7 +237,7 @@ Flash attention 2 support is available both for `idefics2-8b-base` and `idefics2
 
 </details>
 
-**4 bit quantization and module fusing**
+<!-- **4 bit quantization and module fusing**
 
 <details><summary>Click to expand.</summary>
 
@@ -228,7 +266,7 @@ model = AutoModelForVision2Seq.from_pretrained(
 ).to(DEVICE)
 ```
 
-</details>
+</details> -->
 
 # Bias, Risks, and Limitations