MARVIS: Modality Adaptive Reasoning over VISualizations
Authors: name: Benjamin Feuer, name: Lennart Purucker, name: Oussama Elachqar, name: Chinmay Hegde

Scientific applications of machine learning often rely on small, specialized
models tuned to particular domains. Such models often achieve excellent
performance, but lack flexibility. Foundation models offer versatility, but
typically underperform specialized approaches, especially on non-traditional
modalities and long-tail domains. We propose MARVIS (Modality Adaptive
Reasoning over VISualizations), a training-free method that enables even small
vision-language models to predict any data modality with high accuracy. MARVIS
transforms latent embedding spaces into visual representations and then
leverages the spatial and fine-grained reasoning skills of VLMs to successfully
interpret and utilize them. MARVIS achieves competitive performance on vision,
audio, biological, and tabular domains using a single 3B parameter model,
achieving results that beat Gemini by 16\% on average and approach specialized
methods, without exposing personally identifiable information (P.I.I.) or
requiring any domain-specific training. We open source our code and datasets at
https://github.com/penfever/marvis