# Exploring the Limits of Large Language Models' Ability to Distinguish Between Objects
## 4 Experiments
### Evaluation of Open-Vocabulary Vision Foundation Models
\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multicolumn{1}{c}{\multirow{2}{*}{**Model**}} & \multicolumn{2}{c}{**Q3\_1**} & \multicolumn{2}{c}{**Q3\_2**} & \multicolumn{2}{c}{**Q4\_1**} & \multicolumn{2}{c}{**Q4\_2**} \\ \cline{3-10}  & & **Step 1** & **Step 2** & **Step 1** & **Step 2** & **Step 1** & **Step 2** & **Step 1** & **Step 2** \\ \cline{3-10}  & & **Single** & **Single** & **Multi-Object** & **Multi-Object** & **Single** & **Single** & **Multi-Object** & **Multi-Object** \\ \hline Closed model & GPT-4o [49] & **95.77** & **56.34** & **100.00** & 25.00 & **100.00** & **71.83** & 87.50 & **12.50** \\ \hline \multirow{3}{*}{\begin{tabular}{c} Small-scale models \\ scale models \\ scale models \\ scale models \\ scale models \\ \end{tabular} } & \begin{tabular}{c} Owen2-VL-7B [16] \\ scale models \\ scale models \\ \end{tabular} & 76.06 & 46.48 & 87.50 & 12.50 & **100.00** & 59.15 & **100.00** & 0.00 \\  & Phi-3.5-3.8B [5] & 43.66 & 18.31 & 62.50 & 0.00 & 81.69 & 45.07 & **100.00** & 0.00 \\  & llama3-lava-next-8B [3] & 74.65 & 26.76 & 75.00 & 25.00 & 80.28 & 42.25 & 12.50 & 0.00 \\  & Ovisi1.6-German2-9B [50] & 85.92 & 50.70 & 87.50 & 12.50 & **100.00** & 61.97 & **100.00** & 0.00 \\  & MiniCPM-V-2.6-8B [51] & 83.10 & 43.66 & 87.50 & 0.00 & 98.59 & 52.11 & **100.00** & 0.00 \\ \hline \multirow{3}{*}{\begin{tabular}{c} Medium-scale models \\ scale models \\ \end{tabular} } & \begin{tabular}{c} Llama-3.2-1B [3] \\ scale models \\ scale models \\ \end{tabular} & 70.42 & 38.03 & 87.50 & **37.50** & 95.77 & 64.79 & **100.00** & 0.00 \\  & llama-v1.6-13B [5] & 92.96 & 32.39 & **100.00** & 0.00 & 97.18 & 49.30 & **100.00** & 0.00 \\  & l lartVL2.5-26B [51] & 84.51 & 52.11 & 87.50 & 0.00 & 91.55 & 64.79 & **100.00** & **12.50** \\  & lava-v1.6-34B [5] & 76.06 & 30.99 & 87.50 & **37.50** & 98.59 & 50.70 & 87.50 & 0.00 \\  & lternVL2.5-38B [51] & 88.73 & 50.70 & **100.00** & 0.00 & 91.55 & 61.97 & **100.00** & **12.50** \\ \hline \multirow{3}{*}{\
\begin{tabular}{c} Large-scale models \\ scale models \\ \end{tabular} } &
\begin{tabular}{c} Owen2-VL-7B [16] \\ scale models \\ scale models \\ \end{tabular} & 60.56 & 36.62 & **100.00** & 0.00 & **100.00** & 61.97 & **100.00** & 0.00 \\  & llama-3-2.90B [8] & 83.10 & 45.07 & **100.00** & 0.00 & 95.77 & 57.75 & **100.00** & 0.00 \\  & lternVL2.5-78B [51] & 81.69 & 47.89 & **100.00** & 12.50 & 94.37 & 59.15 & **100.00** & **12.50** \\  & lava-next-72B & 73.24 & 36.62 & 75.00 & 25.00 & 97.18 & 36.62 & **100.00** & 0.00 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Object classification results corresponding to Questions 3 and 4. Bold values indicate the best scores and underlined values indicate the second-best scores for each question. **Single-object Scenarios:** In tasks involving isolated objects, such as distinguishing between real and fake items like carrots, hands, oranges, and pimentos (peppers), the models failed to distinguish between the two categories entirely (Figure 7). Moreover, in some cases, the models were unable to identify the type of object. Despite the presence of distinguishable features, such as shape or color, the models often misclassified fake objects (Figure 8). **Multi-object Scenarios:** The models exhibited a complete inability to classify objects correctly in scenes containing multiple objects. For example, in a mixed arrangement of  
Figure 8: Examples of classification results in different categories. The top row shows results from the YOLO-World model. The bottom row presents results from the RAM model. (**a**) The top two oranges are fake oranges, and the bottom two oranges are real oranges. (**b**) Fake hand (**top**) and real hand (**bottom**). Different colored boxes are different labeled misclassifications. Figure 7: Examples of classification results in different categories using the YOLO-World model. The model misclassified real objects as fake. (**a**) Real orange. (**b**) Real hand. (**c**) Real pimento.