Introducing Idefics2: A Powerful 8B Vision-Language Model for the community
•
80
Victor Sanh PRO
VictorSanh
AI & ML interests
None yet
Articles
Organizations
Posts
5
Post
1939
Can't wait to see multimodal LLama 3!
We released a resource that might come in handy: The Cauldron 🍯
The Cauldron is a massive manually-curated collection of 50 vision-language sets for instruction fine-tuning. 3.6M images, 30.3M query/answer pairs.
It covers a large variety of downstream uses: visual question answering on natural images, OCR, document/charts/figures/tables understanding, textbooks/academic question, reasoning, captioning, spotting differences between 2 images, and screenshot-to-code.
HuggingFaceM4/the_cauldron
We released a resource that might come in handy: The Cauldron 🍯
The Cauldron is a massive manually-curated collection of 50 vision-language sets for instruction fine-tuning. 3.6M images, 30.3M query/answer pairs.
It covers a large variety of downstream uses: visual question answering on natural images, OCR, document/charts/figures/tables understanding, textbooks/academic question, reasoning, captioning, spotting differences between 2 images, and screenshot-to-code.
HuggingFaceM4/the_cauldron
Post
2672
New open multimodal model in town: Idefics2!
💪 Strong 8B-parameters model: often on par with open 30B counterparts.
🔓Open license: Apache 2.0.
🚀 Strong improvement over Idefics1: +12 points on VQAv2, +30 points on TextVQA while having 10x fewer parameters.
📚 Better data: boosting OCR capabilities with 6TB of documents to transcribe, and improving QA capabilities on charts/figures/diagrams.
🕵️♀️ Transparent training data: inspect and build upon all the data (10s of TB of data) we trained on.
🔲 More natural image processing: Incorporating strategies to treat images in their native resolution and native aspect ratio.
📸 High-resolution images: image resolutions up to 980 x 980 and integrating strategies that allow to trade computational efficiency for performance.
😎 2 checkpoints: Releasing both base checkpoint and instruction fine-tuned checkpoint. Chat version to come.
Ressources: HuggingFaceM4/idefics2-661d1971b7c50831dd3ce0fe
Blogpost: https://huggingface.co/blog/idefics2
💪 Strong 8B-parameters model: often on par with open 30B counterparts.
🔓Open license: Apache 2.0.
🚀 Strong improvement over Idefics1: +12 points on VQAv2, +30 points on TextVQA while having 10x fewer parameters.
📚 Better data: boosting OCR capabilities with 6TB of documents to transcribe, and improving QA capabilities on charts/figures/diagrams.
🕵️♀️ Transparent training data: inspect and build upon all the data (10s of TB of data) we trained on.
🔲 More natural image processing: Incorporating strategies to treat images in their native resolution and native aspect ratio.
📸 High-resolution images: image resolutions up to 980 x 980 and integrating strategies that allow to trade computational efficiency for performance.
😎 2 checkpoints: Releasing both base checkpoint and instruction fine-tuned checkpoint. Chat version to come.
Ressources: HuggingFaceM4/idefics2-661d1971b7c50831dd3ce0fe
Blogpost: https://huggingface.co/blog/idefics2
Collections
1
spaces
1
models
6
VictorSanh/test-victor
Updated
•
30
VictorSanh/idefics2-8b-docvqa-finetuned-tutorial
Updated
VictorSanh/vsft-idefics2-8b-hf
Updated
VictorSanh/idefics2-8b-docvqa-finetuned
Updated
VictorSanh/roberta-base-finetuned-yelp-polarity
Text Classification
•
Updated
•
84
•
2
VictorSanh/bart-base-finetuned-xsum
Text2Text Generation
•
Updated
•
12
datasets
8
VictorSanh/obelisc_20000_tr_199_w_xattn_opt_step-65000
Viewer
•
Updated
VictorSanh/LrvInstruction
Preview
•
Updated
•
4
•
6
VictorSanh/obelisc_850k_w_ldatopics_tr_199_w_xattn_opt_step-28000
Viewer
•
Updated
VictorSanh/obelisc_23k_w_ldatopics_tr_199_w_xattn_opt_step-28000
Viewer
•
Updated
VictorSanh/obelisc_23k_tr_190_01_64n_opt_step-34000
Viewer
•
Updated
VictorSanh/obelisc_23k_tr_199_w_xattn_opt_step-28000
Viewer
•
Updated
VictorSanh/obelisc_embedded_10k_v0
Updated
VictorSanh/dummy_test
Updated