On Large Multimodal Models as Open-World Image Classifiers
Authors: name: Alessandro Conti, name: Massimiliano Mancini, name: Enrico Fini, name: Yiming Wang, name: Paolo Rota, name: Elisa Ricci
Organization: Multimedia and Human Understanding Group

Traditional image classification requires a predefined list of semantic
categories. In contrast, Large Multimodal Models (LMMs) can sidestep this
requirement by classifying images directly using natural language (e.g.,
answering the prompt "What is the main object in the image?"). Despite this
remarkable capability, most existing studies on LMM classification performance
are surprisingly limited in scope, often assuming a closed-world setting with a
predefined set of categories. In this work, we address this gap by thoroughly
evaluating LMM classification performance in a truly open-world setting. We
first formalize the task and introduce an evaluation protocol, defining various
metrics to assess the alignment between predicted and ground truth classes. We
then evaluate 13 models across 10 benchmarks, encompassing prototypical,
non-prototypical, fine-grained, and very fine-grained classes, demonstrating
the challenges LMMs face in this task. Further analyses based on the proposed
metrics reveal the types of errors LMMs make, highlighting challenges related
to granularity and fine-grained capabilities, showing how tailored prompting
and reasoning can alleviate them.