OmniFusion Technical Report
Authors: name: Elizaveta Goncharova; name: Anton Razzhigaev; name: Matvey Mikhalchuk; name: Maxim Kurkin; name: Irina Abdullaeva; name: Matvey Skripkin; name: Ivan Oseledets; name: Denis Dimitrov; name: Andrey Kuznetsov
Published: 2024-04-09 11:00:19

Abstract: Last year, multimodal architectures served up a revolution in AI-based
approaches and solutions, extending the capabilities of large language models
(LLM). We propose an OmniFusion model based on a pretrained LLM and
adapters for visual modality. We evaluated and compared several architecture
design principles for better text and visual data coupling: MLP and transformer
adapters, various CLIP ViT-based encoders (SigLIP, InternVIT, etc.), and their
fusing approach, image encoding method (whole image or tiles encoding) and two
7B LLMs (the proprietary one and open-source Mistral). Experiments on 8
visual-language benchmarks show the top score for the best OmniFusion setup in
terms of different VQA tasks in comparison with open-source LLaVA-like
solutions: VizWiz, Pope, MM-Vet, ScienceQA, MMBench, TextVQA, VQAv2, MMMU. We
also propose a variety of situations, where OmniFusion provides highly-detailed
answers in different domains: housekeeping, sightseeing, culture, medicine,
handwritten and scanned equations recognition, etc. Mistral-based OmniFusion
model is an open-source solution with weights, training and inference scripts
available at https://github.com/AIRI-Institute/OmniFusion.