Note Great overview on vision-language modelling approaches

Note - Among the first models to incorporate instruction fine-tuning in vision language models to improve multimodal chat capabilities - Generate 158k synthetically generated visual instruction samples using GPT-4 - Original LLaVA model incorporated a pretrained Vicuna LM and a pretrained CLIP vision encoder and fine-tuned end-to-end on generated vision-language instruction-following data

Note - Develop a multilingual LLM covering 10 languages using similar architecture as in LLaVA - use pretrained CLIP and Vicuna using a two-layer MLP with GELU as the projector between modalities - Multilingual dataset curated using a semi-automated translation pipeline - Translated LLaVA dataset

Note - Introduce Aya 23, a family of multilingual (text-only) language models supporting 23 languages based on Cohere’s “Command” model which are pre-trained using a data mixture that includes texts from 23 languages and fine-tuned on the Aya multilingual instruction data - Available in 8B and 35B sizes