OmniFusion Technical Report
Authors: name: Elizaveta Goncharova, name: Anton Razzhigaev, name: Matvey Mikhalchuk, name: Maxim Kurkin, name: Irina Abdullaeva, name: Matvey Skripkin, name: Ivan Oseledets, name: Denis Dimitrov, name: Andrey Kuznetsov

Last year, multimodal architectures served up a revolution in AI-based
approaches and solutions, extending the capabilities of large language models
(LLM). We propose an OmniFusion model based on a pretrained LLM and
adapters for visual modality. We evaluated and compared several architecture
design principles for better text and visual data coupling: MLP and transformer
adapters, various CLIP ViT-based encoders (SigLIP, InternVIT, etc.), and their
fusing approach, image encoding method (whole image or tiles encoding) and two
7B LLMs (the proprietary one and open-source Mistral). Experiments on 8
visual-language benchmarks show the top score for the best OmniFusion setup in
terms of different VQA tasks in comparison with open-source LLaVA-like
solutions: VizWiz, Pope, MM-Vet, ScienceQA, MMBench, TextVQA, VQAv2, MMMU. We
also propose a variety of situations, where OmniFusion provides highly-detailed
answers in different domains: housekeeping, sightseeing, culture, medicine,
handwritten and scanned equations recognition, etc. Mistral-based OmniFusion
model is an open-source solution with weights, training and inference scripts
available at https://github.com/AIRI-Institute/OmniFusion.