# Exploring the Limits of Large Language Models' Ability to Distinguish Between Objects
## 1 Introduction  
Large language models (LLMs) have demonstrated remarkable capabilities across a range of tasks, from text generation [1, 2, 3, 4, 5, 6, 7] to solving difficult Olympiad-level math questions with reasoning [8, 9]. Recent advancements have pushed LLMs toward applications in computer vision (vision-language models (VLMs)) and multimodal processing (MLLMs), allowing them to learn simultaneously from different data sources rather than solely text data. VLMs are trained on a diverse set of large data to exhibit state-of-the-art performance on different vision-text tasks [10, 11, 12, 13, 14, 15, 16], such as classification [17, 18, 19, 20] and segmentation [21, 22, 23]. The growing reliance on automated systems in fields such as quality control, surveillance, and automated inspection has increased the need for robust classification modelscapable of precise object recognition. VLMs offer an intriguing alternative to traditional computer vision models in these applications with their potential for zero-shot and few-shot learning, since they are flexible on newly seen situations. However, VLMs face unique challenges in visual recognition, particularly when distinguishing visually similar objects or processing complex scenes with multiple objects and classes (Figure 1). One critical yet under-explored aspect of these challenges is LLMs' ability to differentiate real objects from artificial ones and to classify human-like figures across different categories. In this study, we investigate these limitations by evaluating the performance of various VLMs in these specific classification tasks, highlighting areas that require improvement. We organized the evaluation of language models around two primary classification tasks: _real vs. fake object classification_ and _human vs. human-like object classification_ (Figure 2). The first evaluation challenges the models to distinguish real objects from their artificial counterparts across various categories, such as fruits, vegetables, and hands. The second evaluation requires models to differentiate human and human-like figures (mannequins, banners), which adds complexity in scenarios involving multi-class and multi-object settings. Through these tasks, we seek to assess state-of-the-art vision-language models' strengths and weaknesses in handling visually complex information and to identify the architectural and training modifications necessary to improve their classification performance. Figure 1: Illustrative examples of real-world challenges that VLMs struggle to comprehend. Figure 2: Scheme for evaluating the performance of vision-based LLMs in distinguishing realistic objects. By evaluating a range of VLMs with different architectures, sizes, and training configurations, we aim to provide a comprehensive analysis of their current performance to reveal both their capabilities and limitations in visual classification tasks. Our findings emphasize the necessity for advancements in multimodal integration, prompt engineering, and adaptive learning, which could enable LLMs to fulfill a broader range of practical applications that demand precise and reliable object classification. The primary contributions of this study are as follows:  
* We provide an in-depth empirical assessment of a broad range of vision-language models (VLMs) across two challenging tasks: differentiating real objects from artificial replicas and distinguishing real humans from human-like figures (e.g., mannequins, banners). Our evaluation is structured into step-by-step tasks that separately measure basic object recognition, authenticity verification, and spatial reasoning. This multi-task framework reveals how even high-performing models (e.g., GPT-4o) achieve near-perfect accuracy on isolated object identification (up to 98.6%) but suffer drastic performance drops (often near-zero) under more complex multi-object or nuanced authenticity conditions. * We introduce a novel evaluation approach that leverages tailored question prompts and a custom dataset.