LVLM-Count: Enhancing the Counting Ability of Large Vision-Language Models
Authors: Muhammad Fetrat Qharabagh
Mohammadreza Ghofrani
Kimon Fountoulakis
Venue: Accepted by TMLR ()

Abstract: Counting is a fundamental operation for various real-world visual tasks, requiring both object recognition and robust counting capabilities. Despite their advanced visual perception, large vision-language models (LVLMs) are known to struggle with counting tasks. In this work, we evaluate the performance of several LVLMs on visual counting tasks across multiple counting and vision datasets. We observe that while their performance may be less prone to error for small numbers of objects, they exhibit significant weaknesses as the number of objects increases. To alleviate this issue, we propose a simple yet effective baseline method that enhances LVLMs’ counting ability for large numbers of objects using a divide-and-conquer approach. Our method decomposes counting problems into sub-tasks. Moreover, it incorporates a mechanism to prevent objects from being split during division, which could otherwise lead to repetitive counting—a common issue in a naive divide-and-conquer implementation. We demonstrate the effectiveness of this approach across various datasets and benchmarks, establishing it as a valuable reference for evaluating future solutions.

Published in Transactions on Machine Learning Research (02/2026)
LVLM-Count: Enhancing the Counting Ability of Large Vision-Language Models
Muhammad Fetrat Qharabagh Cheriton School of Computer Science University of Waterloo
mafetrat@nnaaerrloo.cr
Mohammadreza Ghofrani
Independent Researcher
Kimon Fountoulakis
Cheriton School of Computer Science University of Waterloo
Reviewed on OpenReview: https: // openreview. net/ forum? id= Cd19U0003
Abstract
Counting is a fundamental operation for various real-world visual tasks, requiring both object recognition and robust counting capabilities. Despite their advanced visual perception, large vision-language models (LVLMs) are known struggle with counting lasks. In this work, we evaluate the performance of several LVLMs on visual counting tasks across multiple counting and vision datascts. We observe that while their performance may be less prone to error for small numbers of objects, they exhibit significant weaknesses as the number of objects increases. To alleviate this issue, we propose a simple yet. effective baseline method that enhances LVLMs' counting ability for large numbers of objects using a divide-and- conquer approach. Our method decomposes counting problems into sub-tasks. Morcover, incorporates a mechanism to prevent objects from being split during division, which could otherwise lead repetitive counting-a common issuc in naive divide-and-conquer implementation. We demonstrate the effectiveness of this approach across various datasets and benchmarks, establishing it valuable reference for evaluating future solutions.
1 Introduction
Counting key cognitive task with broad applications in industry, healthcare, and environmental monitoring
al. 2015 Paul Cohen et al. 2017 Lempitsky & Zisserman 2010). It improves manufacturig, inventory, and quality control, ensures safety in medical settings, and helps manage resources in environmental efforts (Wang & Wang 2011 Zen et al.) 2012 Arteta et al. 2016). These applications often require distinguishing between objects of the same class with subtle variations, as well as recognizing complex concepts. Models trained solely on counting datasets struggle generalize to such scenarios due the limited availability of annotated data for finc-grained counting. Recent advancements large vision-language models (LVLMs), such GPT-4o (Achiam et al.) 22023), combined with the massive scale of web-scraped training cata, have enabled unprecedented zero-shot recognition capabilities, making them a promising candidate for handling complex and fine-grained counting tasks. However, evaluations of
(De Almeida et al. 2015 Cuerrero-Gómez-Ommeee
LVLMs have also revealed notable weaknesses in their numerical reasoning (Yin et al. 2023 Xu et al. 2024
Yang et al. 2023
In this work, we focus specifically on the visual counting, one of the most fundamental aspects of numerical reasoning. We observe that although LVLMs perform well in counting small numbers of typically fewer than 20 their accuracy deteriorates with larger quantities. Inspired by prior work the rapid and

---

Published in Transactions on Machine Learning Research (02/2020)
Inpul image
Target Segmentacion
Object-aware Division
Number Aggregation
Final Count
Area Detection
Grounding prompt E:: "brown eggs"
Segmentation prompt E: "brown egg"
Q: How many brown eggs the image?
Q: How many brown eggs in the image?
Object interest (E) extraction from Q using an LLM
Counting using LVLM
Figure 1: Illustration of our proposed pipeline. First, an expression (E') describing the area of interest is extracted from the prompted question (Q), such as "brown eggs". The expression is extracted using a large language model (LLM) which the same as LVLM our work. Then, E and the image are provided as input to a grounding model, such as the one by Liu et al. (2023) to detect the area of interest. Second, any objects corresponding to E are segmented. Third, in the object-aware division step, we use the segmentation masks to divide the detected area of interest without cutting through the objects of interest. Finally, the number of objects of interest in each sub-image is computed using an IVLM, and the results are aggregated.
accurate estimation of small quantities by Chattopadhyay et al. 2017 we propose a simple yet effective baseline method lo alleviate this issue. Leveraging a divide-and-conque approach, we engineer a simple pipeline that divides an image into carefully cut sub-images, and prompts the LVLM to count the objects of interest in each sub-image. The counts from the sub-images are then aggregated to make the final prediction. A key feature of our pipeline is mechanism that prevents objects of interest from being split by the dividing lines, which could otherwise lead to double-counting. To have this mechanism two existing pre-trained detection and segmentation models are employed in the pipeline. The workflow is illustrated in Figure 1
Initially, in our pipeline, the calegory name of the object of interest is extracted from the inpul question using an LLM (We use the IVLM as an L.T for this step). The area containing the objects of interest is detected in the image by grounding model, such as Liu et al. (2023), and then cropped. The cropping step removes irrelevant context from the image. Secondly, using an object detection model by [Lu et al. (2023), and segmentation model by Kirillov et al. (002)) the segmentation masks of the objects of interest. are created. Thirdly, we use mechanism that divides the image into multiple sub-images without cutting through the objects of interest. We call this mechanism object-aware division. The division positions are determined automatically using an unsupervised and non-parametric method based on object masks. Then we treat the object-aware division as a path-finding problem, avoiding the segmented objects as obstacles. A black-white image is built by converting all the masks into black and the rest of the image into white pixels. The binary image is converted into graph where only white pixels
connected nodes. Using the A algorithm [Russell & Norvig 2016), a path is found from one end to the other end of the image, ensuring objects remain intact. Finally, using an LVLM as counting tool, the objects of interest in the sub-images are counted and aggregated. Our contributions are summarized below:
1. We evaluate the counting performance of several recent. Large Vision-Language Models (LVLMs) multiple counting and vision datasets. We propose a simple yet effective baseline method, LVLM- Count, which enhances the counting performance of LVLMs without requiring additional training. Similar to standard counting with LVLMs, our method is a prompt-based approach that retains their zero-shot capabilities while addressing their difficulties in handling large numbers. Through experiments, we demonstrate that LVLM-COue Count improves LVLMs counting performance across the evaluated datascts.
We propose solution for object-aware division. Accurate division crucial, as parts of cut objects can lead to over-counting (see Figure 2aI. The proposed solution divides images without cutting through objects of interest. specified by arbitrary prompt.
2

---

Published in Transactions on Machine Learning Research (02/2026)
20.0
Absolute Counting Error
17.5
Original Image
15.0
Naive division
12.5
Obj.a division
10.0
7.5
5.0
2.5
0.0
10
20
30
40
50
60
Number of Blue Circles
(a) Naive division
(b) Object-aware division
(c) Counting error
Comparison of the naive and the object-aware division. The objects of interest the circles. In F Figure 2a we illustrate naive division of the input image, which is divided into equally sized sub-images with straight lines. In Figure 2b] we illustrate the object-aware division, which avoids cutting through circles. In Figure 2c] we illustrate the counting error of GPT-4o for images with randomly positioned circles. The absolute counting error the absolute difference between the ground truth and the number predicted by GPT-4o. The results are averaged over three trials.
Figure 2:
As a minor contribution, we create new benchmark to address some of the drawbacks of existing datasets. Prior datasets feature simple counting tasks, e.g. counting "strawberries" and lack intra-class complexity. To address these issues, we develop a challenging benchmark for counting emoji icons. The subtle variations within emoji classes make this benchmark uniquely difficult.
2 Related Work
Early counting models, referred to as class-specific, largeted counting problems for certain categories (Artela
et al. 2016 Babu Sam et al. 2022 Mundhenk et al. 2016 Xie et al. 2018), such as cars, people, or cells. Later, with the emergennce of stronger vision models and large-scale datasets, class-agnostic methods were proposed that could count objects from a wide variety of categories. However, most existing class-agnostic, or
open-world, models require visual exemplars of the target objects Duuiić et al., 2023
Gong et al. 2022
Lin
al.
2022
Liu al.
2022 Lu et al. 2019
Nguyen et al. 2022 Ranjan al. 2021 Shi al. 2022
et al.
2021
You et al.
2023
Text- counting models. With the advent of vision- language foundation models such as CLIP and GroundingDINO, text-based open-world models have been proposed that are trained specifically for counting. Leveraging the rich textual and visual feature extraction capabilities of foundation models, oblained through web-scale training, the text-based counting models by Amiini Naieni al. (2023 ; Dai et al. (2024): Kang
ct al. (0024; Amini-Naicni et al. (2024) have started to demonstrate comparable superior accuracy. In addition, Shi al. (2024) introduce TFOC, counting model that does not require any counting-specin training. Instead, they cast the counting problem as prompt-based segmentation task, using SAM (Kirillov et al. 2023) to obtain segmentation masks that determine the output number. Despite this progress, these models remain constrained by two principal limitations. First, their performance can degrade significantly under considerable distribution shift in the input samples. Second, the prompts they interpret are typically limited to simple object categorics or referring expressions; as task complexity increases, these models often fail. Employing LVLMs for counting offers potential pathway mitigate these issues.
Leveraging the concept of divide and conquer for counting. The concept of divide and conquer has been used in carly work (Chattopadhyay et al. 2017 Xiong et al. 2019 Stahl et al. 2018). Chattopadhyay el al. (2177) use an image-level divide and conquer approach and train a convolutional neural network (CNN) that can count objects from a predetermined and limited set of categories in sub-images. Xiong et al. 22019 propose applying the divide step on the convolutional feature map instead of the input image avoid repeatedly computing convolutional features for each sub-image, thereby improving efficiency. However, the
CNN in their work is only capable of counting single object category. Similar to Chattopadhyay al. 2017
3

---

Published in Transactions on Machine Learning Research (02/2026)
Stahl et al. (2018) also employ image-level division and train a CNN to count objects from a predetermined set of categories. Nonetheless, their method does not require local image annotations for training.
Assessment of LVLMs' counting performance. Several prior works have explored the visual counting capabilities of LVLMs as part of broader evaluations, underscoring the difficulties these models encounter in counting tasks Yin et al. (0023) ; Xu al. (2024); Yang et al. (2023). However, these studies have not focused on developing solutions to address these challenges. In this work, we conduct comprehensive quantitative asscssment of LVLMs' counting performance across diverse visual counting benchmarks. More importantly, propose a simple yet effective baseline method to improve their ability to count large numbers. Our bascline method is not tailored to any specific model and can be used in plug-and-play manner with different LVLMs. Regarding the categories outlined earlier, our method, LVLM-Count, is an open-workd, prompt-based counting approach that requires no additional training. To the best of our knowledge, we are the first to propose a dividd-aand-oonquer strategy that avoids splitting and potentially double-counting objects of interest. specified by an arbitrary text prompt.
3 LVLM-Count
Our proposed method aims to answer counting questions by dividing an image into sub-images while avoiding cuts through objects of interest. LVLM-Count consists of four key stages. First, in the "Area Detection" stage, we localize areas containing relevant objects. Second, in the "Target Segmentation" stage, we identify and segment the objects of interest. Third, in the "Object-aware Division" stage, we divide the localized areas into sub-images without cutting through the segmented objects. Finally, the LVLM counts the target objects in each sub-image and aggregates the results. Figure ! iillustrates the workflow of our method, which we detail in the following subscctions.
3.1 Area Detection
In this part of the pipeline, we assume that we are given a counting question Q along with an image. The question Q cont aintains&o an expression E that specifies a set of objects interest. The expression E distinguishes these objects from objects of other categories or the same category but with different attributes present in the image. By employing an LLM, the expression E' is extracted from Q. For example, let Q be "How many brown eggs are in the image?". Q is given to LLM, which is prompted to return the expression F, "brown eggs", referring to the objects we want, to count. After E is extracted, it is provided as input to GroundingDINO along with the image. The output of GroundiggDINO may be a single bounding box or a set of bounding boxes that have relevance to E beyond a certain threshold. These bounding boxcs often overlap and typically contain repeated objects. Thus, all the overlapping output bounding boxes are merged. After merging, a set of non-overlapping areas of interest may remain. We consider the non-overlapping areas as "detected areas", which are then cropped to be passed to the next stage. This process is illustrated in Figure 3
3.2 Target Segmentation
The cropped images from the first stage contain objects of interest, and the ultimate goal is to divide them without cutting through those objects. However, a prerequisite for implementing such mechanism to first detect and localize the objects of interest. Each cropped image fed into an open-world detection model along with E. The output of the open-world detection model produces bounding box for each object of interest. The bounding boxes then given input to segmentation model, which returns segmentation masks for the objects within each bounding box. We illustrate an example of this process in Figure 4
How to determine the bounding boxes. To determine the bounding boxes, we use GroundingDINO and set the bounding box probability threshold to a low value to avoid missing any object of interest. The bounding boxes alone cannot help with the object-aware division of the detected arcas due to their rigid structure, which includes redundant areas in the vicinity of the object and, in the worst case, overlaps with other bounding boxes. Our goal is to precisely locate the pixels of an object of interest.
4

---

Published in Transactions on Machine Learning Research (02/2026)
oggald
Figure 3: Illustration of the area detection stage of LVL.MCount. For this image, Q is set to "How many brown eggs are in the image". The LLM that used in this step returns an E' which is "brown eggs". E and the original image given input to GroundingDINO, which returns bounding box. If the grounding model returns multiple bounding boxes, they are merged to form the final detected area.
brown og 22)
brown egg (0
brown egg (0
brown egg (0.2)
brown egg (0
brown egg 17)
brown egg (0.
brown e0 15)
brown 60 own egg to vn egg
egg 0. 0.19)
(a) GroundingDINO output
(b) SAM output
Figure 4: Illustration of the target segmentation step of LVLM-Count. The goal is produce all the instance masks for F. set to "brown egg". The cropped detected area from Figure 3 together with F,, is given input.
to GroundingDINO, which produces the output shown in F Figure 4a Figure 4al is then given as input to SAM,
which produces the output shown in Figure 4b
How to determine the segmentation masks. We employ a pre-trained segmentation model, specifically SAM (Krillov et al., (002)). for the segmentation task. This model accepts bounding boxes as prompts and generates masks covering the most prominent objects within these boxes. However, do not use SAM's output masks directly, in crowded scenes cases with multiple occlusions, the masks often overlap, making it difficult to identify reliable division paths. To address this issue, apply several post-processing steps. First, we perform non-maximum suppression on SAM-generated masks to climinate those with significant overlaps corresponding to less certain bounding boxes. Additionally, we apply an erosion function to the segmentations, ensuring adjacent masks maintain minimum separation of two pixels. For detailed analysis and visual examples of how these post-processing operations enhance robustness in crowded scenes and occlusion cases, see Appendix C The final processed masks for each cropped area are then passed to the next stage of our pipeline.
5

---

Published in Transactions on Machine Learning Research (02/2026)
Robustness to the Accuracy of Area Detection and Target Segmentation Stages. Although we employ GroundingDINO state-of-the-art model for grounding and detection it lacks the flexibility and robustness of LVLMs when generalizing to unseen data and complex concepts. Consequently, we minimizes dependence on the accuracy of the detection model used for area detection and target segmentation. Rather than prioritixing accuracy, we emphasize: i) not missing any region containing objects of interest. during area detection, and ii) not missing segmentation masks for any objects of interest during target segmentation. These objectives are easily achieved by setting GroundingDNO's detection threshold to very low value in both stages.
A low threshold may lead to false positives. In area detection, this could result in regions that contain no objects of interest. However, since counting is performed by the LVLM in the final stage, these regions will be ignored. For target segmentation, false positive masks might occur, but the consequence is that some irrelevant objects will be protected from being cut by the division lines, just as the objects of interest are. Furthermore, as we will demonstrate in subsequent sections, all masks are removed from the sub-images after division, ensuring no noise propagates from this stage to the IVLM--ased counting stage.
We demonstrate the effectiveness and robustness of our strategy in the early pipeline stages by showing that LVLM-Count significantly improves LVLMs' performance on one of the most challenging counting bltasetssthi Penguin Dataset, Penguin Research (2016), as detailed in Section 4.2 This dataset is particularly difficult due to frequent occlusion, camouflage, and complex backgrounds Arteta et al. (2016). Furthermore, in our ablation studies, we test a variant of our pipeline that removes the detection model entirely. Instead, SAM is configured to generate segmentation masks for all objects in the scene. As shown in [Talle 5] our method still enhances counting performance in this configuration, underscoring its robustness and minimal dependence on the detection model during the initial pipeline stages.
3.3 Object-aware Division
In this stage, the cropped image is divided into appropriate sub-images so that no object of interest is cut by the dividing paths. The core idea is that the dividing paths should not intersect the pixels covered by the masks corresponding to the objects of interest. This step consists of two sub-steps. First, we decide the starting and ending points of the paths. Second, we draw the paths. Below, we describe how we approach these two sub-steps.
How to determine the starting and ending points of the paths. We utilize an unsupervised and conpparaecrric approach, to obtain the start and end points of the paths. A few pixels are sampled from each of the masks. To determine the location of the division paths on the ær-xxis, the samples taken from the masks are projected onto the z-axis. The projected points are automatically clustered using non non- parameeriii mean-shift algorithm[ (Comaniciu & Meer 2002). Once the clusters are identified, the point between the point with the highest x value in one cluster and the point with the lowest x value in the subsequent cluster is considered the r-coordinate of a division path. In effect, knowing the x-coordinate of a vertical path means that the coordinates of its endpoints are known. In particular, assuming height for an image crop, we consider P.- = (z. 0) and Pe- = (x. h) as the start and end points, respectively. Note that using this technique, we obtain the appropriate coordinates for the division paths, as well as the number of paths. For example, if there is only one cluster along the z-axis, no division is required, and if there are two clusters, one vertical path will divide the image into two parts. We illustrate this approach in Figure 5
How to draw the paths. Previous step obtains the endpoints of the division paths. Assume P. = (z, 0) and P. = (z, h) are the start and end points of a vertical division path, respectively. In an idcal case where there are no masks in the path of a straight line connecting the two points, this path will be drawn by connecting all the pixels on the straight line. However, there are potential masks that can be considered obstacles blocking the path. In other words, beginning from P., the line needs to go around these obstacles to reach Pe. Consequently, we treat this as 2-dimensional path-finding problem. To solve the problem, we build a 2D binary map, T, where the pixels covered by the masks are turned into black, indicating them as obstacles, and all the other pixels are turned into white, showing they are open for passage. This
In our implementation, we employ the MeanShift algorithm from the learrt-learn library Pedregosa et al. 2011 utilizing its default heuristic for automatic bandwidth estimation.
6

---

Published in Transactions on Machine Learning Research (02/2026)
P
P
Random sammpees from masks
Projection
Projection onto the 2n-xis
onto this
P
10c
25g
Cluster
Cluster 2
Cluster 3
Figure 5: Illustration of the unsupervised and non-parametric method to obtain the division points (P), Pe); and P2.). A few pixels are sampled (shown points inside the segmented objects) from the pixels composing target masks. The samples are projected onto the x-axis. The projected points are clustered using mean-shift clustering. The point in the middle of two consecutive clusters is considered vertical division point. Blue lines are solely for illustration
binary image IB is mapped into a graph G, where each white pixel is a node, and is connected to all its white neighboring pixels. We use the A (G, Ps. Pe. g) search algorithm to find a path that connects P. Pe. where the heuristic g is set to be Manhattan distance. The output of A is set of connected pixels that go around the obstacles and connect P. to Pe, creating an object-aware division path, shown in Figure 6 The path-finding algorithm is run for all division coordinates. Finally, we draw the image contours based on these division paths and take the area surrounded by each contour as a resulting sub-image. Note that although not part of the main LVLM-Count pipeline, the above two steps can be applicd to the y-axis in the same manner to obtain horizontal division paths. For further discussion, the reader may refer to Appendix S)
3.4 Target Counting
All the sub-images obtained from the cropped areas are gathered. Then, question Q and each sub-image are given as input to an LVLM. At the end of the loop, the recorded numbers for the sub-images are aggregated form the final answer. For images with very large number of objects, sometimes LVLMs refuse count, citing the large number. In those cases, a new prompt requires the model give the closest estimate of the number.
4 Experiments
In this section, we present the performance results of different LVLMs and their enhanced performance results using method counting-speeific dataset, counting benchmark taken from popular vision datasets, and a challenging counting benchmark that propose using emoji icons. Additionally, we show the success of our design in enhancing counting performance on one of the most challenging counting datasets that features heavy occlusions and complex backgrounds. The code to reproduce the experiments is available at our GitHub repository
20ur Hub repository is available httpps///gtttt...c..////aaaataas http:////sccaccccccaaaaaa/stt/tt

---

Published in Transactions on Machine Learning Research (02/2020)
P
P
P.
pl
P2
Figure 6: Illustration of object-aware division. The masks are turned into a black-and-while image. A dividing path is found by connecting Ps to Pe using the A search algorithm a graph that corresponds to the binary image. The only nodes the graph are white pixcls of the black-and-white image, which are connected to all other white pixels in their 3 x 3 neighborhood. The nodes and edges on the obtained paths have been colored red.
4.1 Datasets and Benchmarks
FSC-147 (Ranjan et al. 2021). FSC-147 is a counting dataset that contains 6135 images, spanning 147 different object categories such as kitchen utensils, offiee supplies, vehicles, and animals. The mumber of objects in each image ranges from 7 to 3731, with an average of 56 objects per image. The dataset is split. into training, validation, and test sets. A total 89 object categories are assigned to the training set, 29 to the validation set, and 29 to the test set, with different categories each split. The training set contains 3659 images, with the validation and test sets containing 1286 and 1190 images, respectively. For each image in the test set, single category name is given, and the expected output is the number of instances.
PASCAL VOC Benchmark We build counting benchmark from PASCAL VOC dataset (Everingham) et al. 2015). Similar to [hhattopadlyay et al. (2177,, we choose PASCAL VOC 2007 among other variants. This variant contains a training set of 2501 images, a validation set of 2510 images, and a test set of 4952 images, with 20 object categories that remain consistent across the splits. Each image includes annotations for instances of the object categories in the dataset. We first create 20 simple counting questions asking for the number of objects from each of the 20 categories for every image in the test set. Then, we randomly sample five questions from each ground truth count. This process resulted in 102 questions in total. Finally, we manually checked the ground truth counts and corrected them if required.
Emoji-Count. To our knowledge, no counting benchmark exists with large number of objects in a scene involving complex reasoning. To this end, we propose a challenging counting benchmark using emoji icons. From the 1816 standard emoji icons, we remove those that directly overlap with concepts demonstrated by other icons. We then group the remaining 1197 icons into classes. In cach class, there are icons from the same similar object categories, but with subtle differences that require complex reasoning to distinguish. For each of the 82 classes, empty 1024 x 1024 image is first created. This image is filled with up to six categorics chosen randomly from the class, with each category having a random count between 30 and 50 in the image. For each image, we create questions that ask the number of instances of the available categories in the image. This results in 415 image-question pairs. We illustrate two examples of this dataset in Figure 7
Penguin Benchmark. The challenging Penguin datasct Penguin Rescarch 2016) consistently exhibits heavy occlusion and complex background patterns that can easily be mistaken for penguins Arteta al. 2016)_ The test set of this dataset is quite large, containing thousands of images with penguin counts
8

---

Published in Transactions on Machine Learning Research (02/2026)
(a) Q: How many waning gibbous moons are there the image? Answer: 18.
(b) Q: How many clocks at time "two-thirly" are there the image? Answer: 15.
Figure 7: Illustaation of a smaller version of challenging cases from Emoji-Count. In Figure 7a the class name is "Moon Phase". In Figure 7b, the class name is "Clock Time".
ranging from 0 to 213. To build manageable benchmark, randomly sample 100 images while preserving balanced ground truth range. We refer the reader to Appendix D for more details about this benchmark.
4.2
Results
This section presents the numerical results of our experiments with the base LVLMs and their corresponding IVT.M-Count on each benchmark described in Section 4.1 In the experiments, we consider two baselines: (i) a baseline where the number of target segmentation masks taken as the final answer, and (ii) a baseline where an image of the generated segmentation masks is provided to GPT-4o along with a prompt to count them. For reference, we also report the performance of three srate-of-the-art (SOTA) text-based counting-specifn models: GroundingREC (Dai al. 2024), CountG (Amini-Naieni al. 2024), and DAVEpmm (Pelhan
et al. 2024).
It is important to note that the primary aim of this work is to propose a method that enhances the counting ability of an arbitrary large vision-language model (LVLM) across diverse datascts. The inclusion of these specialized models our comparison should not be interpreted as an allempt to replace or compete with them on specific counting tasks particurttillly those involving simple counting. These models are trained on dedicated counting datasets and perform efficiently on test samples from the same distribution or datasets with high category overlap. However, their performance may degrade significantly when applied to out-of- distribution samples. Furthermore, these models are constrained by the lype and complexity of the input prompts they can process; they typically only accept simple category names or referring expressions, and often fail when presented with complex phrases or sentences.
In contrast, our method designed to be comprehensive. Although it may not match the efficiency of task-spocific models on datascts they were trained on, it performs robustly across wide range of test distributions, despite not being explicitly trained on any of the tasks. Moreover, is capable of handling arbitrary prompt types and structures, regardless of complexity. We also report the performance of TFOC (Sh et al. 2024), SOTA training-free free model. In essence, TFOC is segmentation-baset counting approach, similar to our baselines, but employs more sophisticated methodology.
The above discussion is reflected in our experimental results. For the trained counting models, we report performance using weights obtained from training on the FSC-147 training set. These models perform well on the FSC-147 dataset and on the PASCAL VOC benchmark, which has high category overlap with FSC-147. However, on the Emnoji-Count dataset-whiee exhibits different distribution and requires complex
9

---

Published in Transactions on Machine Learning Research (02/2026)
understanding their performance drops considerably. In contrast, our method, LVLM-Count, maintains high performance even on challenging benchmarks such as Emoji-Count. Another experiment on the TallyQA- Complex benchmark, which contains real-world scenes with complex counting questions, further reveals the limited complex reasoning capability of conventional trained modcls. LVLMs and our proposed LVLM-Count, however, demonstrate superior ability to handle such challenging cases. The corresponding results on the TallyQA-Complex benchmark are provided in Appendix T
We also conduct ablation study on the effectiveness of different components of our pipeline in Appendix A In summary, the ablation experiments support three conclusions: (i) each component has positive effect over the bascline LVLM, and when combined, the full pipeline in Figure 1 achieves the best performance compared to different variants; (ii) naive division with straight lines cannot replace object-aware division; and (iii) LVLM-Count is robust to the accuracy of the target segmentation stage. For visual examples of LVTM-Connt's performance on the benchmarks introduced [Section 4.1 see [Appendix B. An inference time analysis is provided in Appendix JJ This analysis shows that the largest portion of the time spent querying the LVLM, which is step common to both the base LVLM and IVTM-Count. The other steps contribute much smaller share to the overall inference time.
FSC-147. We evaluate the performance of various LVLMs and their enhanced performance using our method, IVL.M-ouun, on the test set of the FSC-147 dataset. We consider two baselines: (i) the baseline where the number of target segmentation masks is taken as the final answer, and (ii) the baseline where an image of the generated segmentation masks is provided to GPT-4o with a prompt to count the number of masks. Our experiments involve GPT-40 (a leading proprietary model), as well as two open-source models: Qwen2 VL 72B AWQ (Yang et al. 2024] and Gemma 3 27B Team al. (2025). The expression E used in different stages of our method corresponds to the category name provided in the test set. A simple query Q in the form of "How many E are there?" is constructed and used as the text prompt for the LVLM during the counting stage. In all experiments, the detection thresholds are set to 0.1. The results are presented in [able I reporting the mean absolute orror (MAE) and root mean square error (RMSE). Our findings indicate that LVLM-Count improves the performance of all three LVLMs terms MAE. Interestingly, while the base Qwen2 and Gemma 3 models are much less powerful than their commercial counterpart, they outperform the base GPT-4o when integrated into our pipeline.
Table 1: Evaluation on the test set of the FSC-147 dataset. In all the tables of this section, the results for base IVLMs and LVLM-Count are reported over six trials. In the tables, columns marked with A show the performance difference between the LVLM-Count and the base LVLM uses. Green indicates improvement, while red represents degradation. Also, the 95% confidence intervals for the MAEs are reported in the column named MAE 95% Cl. To see the measured accuracy metrics for this dataset and the subsequent benchmarks, refer to Appendix Additionally, MAE analysis across different intervals of ground truth values for FSC-147 is provided in Appendix H
Method
Trained Model
MAE 4
A
MAE 95% CI
RMSE 4
A
TFOC (Shi et 2024
24.79
137.15
DAVE Pelhan et al.
2024
14.90
103.42
Count Amini-Natia ienii al. 2024
V 14.76
120.42
GroundingNeC (Da et al. 2024
10.12
107.19
Number of target segmentaton masks
X
44.14
154.39
GPT-4o counting target segmentaion masks
X
38.45
156.11
GPT-40
X
25.57
24.74, 26.39
137. .26
LVLM-Countt (GPT-44 IVLM)
X
17.86
4 7.71
16.96, 18.77
91.71
45.55
Gemma 27B
30.59
30.22, 30.97
132.61
LVLM-Count (Gemma 3 27B as LVLM)
X
20.25
10.34
19.662, 20.89
110.45
2/21.
Qwen2 VL 72B AWQ
X
34.18
[32.85, 35.51
149.49
LVLMM-ount (Qwen2 VL 72B AWQ as LVLM)
22.29
11.89
[21.63, 22.96
119.46
3.30.44
PASCAL VOC Benchmark. We evaluate the performance of three LVLMs GPT-4o, Qwen2, and Gemma 3-on this benchmark, both with and without CVLM-Count. As shown in Table 2 LVLM-Count consistently outperforms the base L.L.Ms.
10

---

Published in Transactions on Machine Learning Research (02/2026)
Table 2: Evaluation on the PASCAL VOC counting benchmark.
Method
MAE
A
MAE 95% C1
RMSE
A
TFOC (Shi al. 2024)
12.03
18.18
DAVEp Eprm Pelhan et al. 2024
12.39
22.81
Gount Amini-Naieni et al.
2024
2.81
7.01
GroundingRec (Dai ct al. 2024
4.05
7.80
Number of the largel segmentation masks
4.03
7.47
GPT-4o counting target segmentaion masks
6.60
15.09
GPT40
4.64
44.45, 4.82]
8.56
LVLM-Count (GPT4o as LVLM)
3.42
4 1.22
3.25, 3.59
7.17
1 1.39
Gemma 3 27B
3.40
3.24 .3555
7.54
LVLM-Count (Gemma 3 27B as LVLM)
2.97
4 0.43
[2822, 3.11]
6.16
4 1.38
Qwen2 VL 72B AWQ
4.80
[4.61, 4.99
8.71
LVLM-Count (Qwen2 VL 72B AWQ as LVLM)
4.12
4. 0.68
[3.87, 4.37]
7.76
J 0.95
Emoji-Count. We evaluate the performance of the LVLMs and their performance using IVT.M-Counl on the Emoji- benchmark. The results are shown in Table 3 understanding complex concepts. We observe that taking the number of masks as the final count performs particularly poorly, as for any object of interest in the image, the segmentation stage lends to segment all the objects and cannot distinguish between different icons. Although GPT-4o and Gemma 3 show reasonable performance, the other open-source model does not perform well. Nonetheless, the performance of all three base LVLMs is significantly enhanced by LVLM-Count, especially Qwen2, which performs almost on par with GPT-4o when IVT.M-Count used for it.
This is challenging benchmark, as it requires
Table 3: Evaluation on the Emojii-Count benchmark.
Method
MAE 4
A
MAE 95% CI RMSE +
A
TFOC Shi et al.) 2024
64.64
87.45
DAVE Polhan al. 2024
198.99
208.00
CountGD Naieni al.
2024
137.93
156.80
Gro (Dai et al.
2024
143.22
158.74
Number of the target segmentation masks
82.47
107.98
GPT-4o counting target segmentaion masks
107.72
162.12
GPT-4o
23.57
22.37, 24.76
36.97
IVT.MCCunt. (GPT-40 as IVLM)
16.57
47
16.25, 16.90
33.11
4 3.86
Gemma 3 27B
21.39
[21322, 21.45
24.04
LVLM-Count (Gemma3 27B as LVLM)
16.16
| 5.23
[15.76, 16.56
21.27
4 2.77
Qwen2 VL 72B AWQ
78.05
[72.17, 83.93
159.22
LVLM-Count (Qwen2 VL 72B AWQ as LVLM)
24.43
53.62
[23.65, 25.21]
43.38
115.84
Penguin Benchmark. We report the counting performance on the Penguin benchmark in Table 4 We evaluate two variants of LVLM-Count. The primary variant uses GroundingDINO in both the area detection and target segmentation stages. The alternative variant does not use detection model; instcad, SAM is configured to segment all entities in the image. Both variants improve MAE across all three LVLMs, highlighting LVLM-Count's robustness in scenarios with heavy occlusion and complex backgrounds conditions that pose significant challenges for arca detection and target segmentation. Additional details and a visual example showing the segmentation masks and division paths generated in both variants for a sample from the Penguin benchmark can be found in Appendix [)
11

---

Published in Transactions on Machine Learning Research (02/2026)
Table Evaluation on the Penguin benchmark.
Method
MAE 4
MAE 95% CI
RMSE 4
A
GP
35.18
[33.13, 37.23
45.76
LVLM-Count( Main variant, using GPT-4o
26.76
8.42
225.66 27.66
38.60
4 7.16
WYLM-Cout((SAMoon) variant, using GPT-40)
29.02
6.16
[28.21, 29.84
44.89
4 0.87
Gemma 3 27B
49.44
(48.94, 49.95
60.20
IVLM-Count( Main variant, using Gemma 3)
34.13
15.31
33..2, 36.53
44.11
16.09
JVLM-Counts SAM-only variant, using Gemma 3)
41.09
8.35
39.92, 42.25
53.01
4 7.19
Qwen2 VL 72B AWQ
44.02
[40.45, 47.60
65.59
IVLM-Count(( Main variant, using Qwen2)
28.21
15.81
22..48, 29.94
40.40
J. 25.19
WVL.M-Comt(SAMMonnyy using Qwen2) )
33.33
+ 10.69
[30.4,, 36.21]
47.55
4 18.04
5 Limitations
In this work, we quantitatively evaluated the visual counting performance of several IVLMs on multiple datasets. More importantly, we introduced a simple yet effective baseline method, LVLM-Count, that enhances the visual counting capabilities of LVLMs across the evaluated benchmarks. However, like any method, LVLM-Count has limitations. One limitation arises in cases where sub-images contain no objects of interest: the LVLM may occasionally predict non-zero value. This reflects broader weakness in LVLMs, and addressing it will require targeted improvements to ensure their accurate zero prediction in such scenarios. Another limitation occurs with open-source models during the target counting stage. Ideally, the LVLM's output for each sub-image should be numerical value. While top proprietary LVLMs, such as GPT-4o, offer functionalities like JSON schema to enforce structured responses (e.g., ensuring a numerical output), open-source models typically lack such features. For these models, the prompt must include explicit, instructions format the response correctly. For instance, an instruction like 'Place the final predicted number inside [[]" enables the use of regex searches to extract the number for aggregation.
Acknowledgments
K. Fountoulakis would like to acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC). Cette recherche a été financée par le Conseil de recherches en sciences naturelles et en génie du Canada (CRSNG), [RGPIN-2019904066, DGECR--019-001-17)
References
Manoj Acharya, Kushal Kafle, and Christopher Kanan. TallyQA: Answering complex counting questions. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 8076-8084, 2019.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint trXiv:2303.0877. 2023.
Niki Amini-Naieni, Kiana Amini-Naieni, Tengda Han, and Andrew Zisserman. Open-world text-specifed object counting. In 34th British Machine Vision Conference 2023, BMVC 2023, Aberdeen, UK, November 20-24, 2023. BMVA, 2023. URL https::/pappers http:/pppprrr..bbcoooo...222.2 wwwc2023. org/ 05100.000.
Niki Amini- Tengda Han, and Andrew Zisserman. CountGD: Multi-modal open-world counting. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Carlos Arteta, Victor Lempitsky, and Andrew Zisserman. Counting the wild. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIT 14, pp. 483 498. Springer, 2016.
12

---

Published in Transactions on Machine Learning Research (02/2026)
Deepak Babu Sam, Abhinav Agarwalla, Jimmy Joseph, Vishwanath A Sindagi, R. Venkatesh Babu, and Vishal M Patel. Completely self-supervised crowd counting via distribution matching. Tn Europeon Conference on Computer Vision, pp. 186 204, Springer, 2022.
Prithvijit Chattopadhyay, Ramakrishna Vedantam, Ramprasaath R Selvaraju, Dhruv Batra, and Devi Parikh. Counting everyday objects in everyday scenes. In Proceedings of the IEEE Conference on Computer Vision and Pallern Recognition, pp. 1135 1144, 2017.
Dorin Comaniciu and Peter Meer. Mean shift: A robust approach toward feature space analysis. IEEE Transactions pollern analysis and machine intelligence, 24(5):603 619, 2002.
Siyang Dai, Jun Liu, and Ngai-Man Cheung. Referring expression counting. In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition, pp. 16985-169988 2024.
Paulo RL De Almeida, Luiz S Oliveira, Alceu S Britto Jr, Eunclson J Silva Jr, and Alessandro Koerich. PKLot- A robust dataset for parking lot classification. Expert Systems with Applications, 42(11):4937-45) 49947-4499,, 2015.
Mark Everingham, SM Ali Eslami, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The PASCAL visual object classes challenge: A retrospective. International journal of computer vision, 111:98 136, 2015.
Shenjian Gong, Shanshan Zhang, Jian Yang, Dengxin Dai, and Bernt Schiele, Class-agnostic object counting robust to intraclass diversity. In European Conference on Computer Vision, pp. 388 403. Springer, 2022.
Ricardo Guerrero- Seneero-GGoee-OOmeeee Beatriz Torre-Jiménez, Roberto López-Sastre, Saturnino Maldonado- Bascón, and Daniel Rubro-Rubio. Extremely overlapping vehicle counting. In Pattern Recognition and Image Analysis: 7th Therian Conference, TbPRTA 2015, Sandiago de Compostela, Spain, June 17-19, 2015, Proceedings 7, pp. 423 431. Springer, 2015.
Philipp Kainz, Martin Urschler, Samuel Schulter, Paul Wohlhart, and Vincent Lepetit. You should use regression to detect cells. In Medical Image Computing and Computer-Assisse InterventionMIICCA 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 276-283. 283. Springer, 2015.
Seunggu Kang, WonJun Moon, Euiyeon Kim, and Jae-Pil Heo. VLCounter: Text-aware visual represent at ion for zero-shot object counting. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 2714-2722 2024.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Antti Lehmussola, Pekka Ruusuvuori, Jyrki Selinummi, Heikki Huttunen, and Olli Yli-Harja. Computational framework for simulating fluorescence microscope images with cell populations. IEEE transactions medical imaging, 26(7):1010-100) 2007.
Victor Lempitsky and Andrew Zisserman. Learning to count objects in images. Advances in Newral Information Processing Systems, 23, 2010.
Wei Lin, Kunlin Yang, Xinzhu Ma, Junyu Gao, Lingbo Liu, Shinan Liu, Jun Hou, Shuai Yi, and Antoni B Chan. Scale-prior deformable convolution for exemplar-guided class-agnostic counting. In The British Machine Vision Conference (BMVC), pp. 313, 2022.
Chang Liu, Yujie Zhong, Andrew Zisserman, and Weidi Xie. CounTR: Transformer-based generalised visual counting. In BMVA Press, 2022.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection. ar Xiu preprint arXio:2303.0544 0549999 2023.
13

---

Published in Transactions on Machine Learning Research (02/2026)
Erika Lu, Weidi Xie, and Andrew Zisserman. Class-agnostic counting. In Computer Vision-ACCV 2018: 14th Asion Conference Computer Vision, Perth, Australia, December 2 6, 2018, Revised Selected Papers, Port III 14, pp. 669 684. Springer, 2019.
T Nathan Mundhenk, Goran Konjevod, Wesam A Sakla, and Kofi Boakye. A large contextual dataset for classification, delection and counting of cars with deep learning. In Compuler Vision Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14, pp. 785 800. Springer, 2016.
Thanh Nguyen, Chau Pham, Khoi Nguyen, and Minh Hoai. Few-shot object counting and detection. European Conference on Computer Vision, pp. 348 365. Springer, 2022.
Joseph Paul Cohen, Genevieve Boucher, Craig A Glastonbury, Henry Z Lo, and Yoshua Bengio. Count-ception: Counting by fully convolutional redundant counting. In Proceedings of the IEEE International Conference Computer Vision Workshops, pp. 18-26, 2017.
Fabian Pedregosa, Gaäl Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825-2830, 2011.
Jer Pelhan, Vitjan Zarrlanik, Matej Kristan, et al. DAVE-A: Delect-and-verify paradigm for low-shot counting. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 223022 2024.
Penguin Research. Penguin research webpage, 2016. URL penguins/ Accessed: 2024-11-23
ww.ac.cc.... uk/ wwg-vgg///vvvvvv data/
Viresh Ranjan, Udbhav Sharma, Thu Nguyen, and Minh Hoai. Learning to count everything. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recoguition, pp. 3394 3403, 2021.
Stuart Russell and Peter Norvig. Artificial intelligence: a modern approach. Pearson, 2016.
Min Shi, Hao Lu, Chen Feng, Chengxin Liu, and Zhiguo Cao. Represent, compare, and learn: A similarity- aware framework for class-agnostic counting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pollern Recognition, pp. 9529 9538, 2022.
Zenglin Shi, Ying Sun, and Mengmi Zhang. Training free object counting with prompts. In Proceedings of the IEFE/CVF Winter Conference on Applications of Computer Vision, pp. 323 33-3,1, 2024.
Tobias Stahl, Silvia L Pintea, and Jan C Van Gemert. Divide and count: Generic object counting by image divisions. IEEE Transactions Image Processing, 1035) 2018.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint XiX::22003. 19786, 2025.
Nikola Dukić, Alan Lukežič, Vitjan Zavrtanik, and Matej Kristan. A low-shot object counting network with LEEE/CVF /CVF International Conference on Computer
iterative prototype adaptation. In Proceedings of the IEEE/ Vision, pp. 18872-18881, 2023.
Meng Wang and Xiaogang Wang. Automatic adaptation generic podestrian detector to a specific traffic scene. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2011, pp. 3401-3400 IEEE, 2011.
Jason Wei, Xuczhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thonght prompting clicits reasoning in large language models. Advances in neural information processing systems, 35:24824 248837 2022.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Weidi Xie, Alison Noble, and Andrew Zisserman. Microscopy cell counting and detection with fully convolutional regression networks. Compuler melhods in biomechanics and biomedical engineering: Imaging & Visualization, 6(3):283 292, 2018.
Haipeng Xiong, Hao Lu, Chengxin Liu, Liang Liu, Zhiguo Cao, and Chunhua Shen. From open set to closed set: Counting objects by spatial dividc-and-conquer. In Proceedings of the IEEE/CVF international conference computer vision, pp. 8362 8371, 2019.
Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Mong, Siyuan Huang, Yu Qiao, and Ping Luo. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, al. Qwen2 tochnical report. arXiv preprint arXiv:2407.10671 2024.
Shuo-Diao Yang, Hung-Ting Su, Winston H Hsu, and Wen-Chin Chen. Class-agnostic few-shot object counting. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 870-878, 2021.
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of Imms: Preliminary explorations with gpt-4v (ision). arXiv preprint 2ete23999.1444 9()::1, 2023.
Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Xiaoshui Huang, Zhiyong Wang, Lu Sheng, Lei Bai, et al. Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark. Advances in Neural Information Processing Systems, 36:26650-2666 2023.
Zhiyuan You, Kai Yang, Wenhan Luo, Xin Lu, Lei Cui, and Xinyi Le. Few-shot object counting with similarity-aware feature enhancement. In Proceedings of the IEFE/CVP Winter Conference Applications of Computer Vision, pp. 6315 6324, 2023.
Gloria Zen, Negar Rostamzadeh, Jacopo Staiano, Flisa Ricci, and Nicu Sebe. Enhanced semantic descriptors for functional scene catcgorization. In Proceedings of the 21st International Conference on Pattern Recognition, pp. 1985 1988. IEEE, 2012.
15

---

Published in Transactions on Machine Learning Research (02/2026)
Appendix Table of Contents
Ablation Study
17
B Visual Examples of LVLM-Count's Performance on the FSC-147 Dataset, PASCAL VOC, and Emoji-Count Benchmarks
17
C Robustness to Crowded Scenes and Heavy Occlusion
17
D Additional Information and Visual Examples from the Penguin Benchmark
19
E Real-world Application of LVLM-Count
21
F LVLM-Count's Power in Handling Multiple Object Categories in the Same Image
24
False Positive Masks at the Target Segmentation Stage
26
Performance Analysis of LVLM-Count for different ground truth ranges on FSC-147
dataset
27
Report of Various Accuracy Metrics for the Performance of LVLM-Count on the FSC-147
Dataset, PASCAL VOC, Emoji-Count, and Penguin Benchmarks
28
Inference Time Comparison
28
K Comparison of Base LVLMs and LVLM-Count's Counting Performance with SOTA Counting Models as a Reference Point
30
L LVLM-Comnt's Ability to Handle Visual Exemplars
31
M Chain of Thought Prompting for LVLM-Count
32
N Illustration of the Complete Workflow of the IVLM-Count. for Additional Image
33
Emoji-Count Benchmark Details
34
P Parsing Strategy for the Output of Open-source LVLMs
35
Q Non-zero Prediction of LVLMs for Subimages with No Objects of Interest
36
R Reproducibility Table for Implementation Parameter
38
S Dividing Images Along Both the Horizontal and Vertical Axes
38
Evaluation of the counting methods TallyQA-Complex
39
16

---

Published in Transactions on Machine Learning Research (02/2026)
A
Ablation Study
We examine the eflect of each stage in our method: (i) area detection and (ii) object-aware division using masks produced during target segmentation. We design different experiments to investigate each stage's individual contribution, well as their combined effect in our pipeline.
To demonstrate our method's minimal dependence on detection accuracy in the area detection and target segmentation stages, we conduct an experiment without area dctection and without using GroundinDDINO for segmentation masks. Instead, we configure SAM in "segment anything" mode, which generates masks for all entities in the scene. The results show that our method remains effective even without any detection model, confirming that IVLM-Count has minimal dependence the accuracy of these initial stages.
Additionally, conduct an experiment where both area delection and segmentation stages are excluded. In this case, object-aware division cannot be performed, so we divide images using equidistant straight lines into subimages (which we call "naive division"). We also run another experiment with area detection but without segmentation, applying naive division to the detected areas. The results demonstrate that unlike object-aware division, the naive approach unreliable due to repetitive counting of fragmented objects.
We perform ablation studies using GPT-4o, conducting experiments on the FSC-144 test set. Table 5 presents the results of these ablation experiments.
Table 5: An ablation study of LVLM-Coul on the FSC-147 test dataset. Columns marked with A show the performance difference between WVLM-Coun and the base LVLM. Green indicates improvement, while red represents degradation.
Method
MAE 4
A
RMSE 4
A
GPT-40
25.57
137.26
GPT-40 + Naive division
33.04
1 7.47
116.83
$ 20.43
GPT-4o + Area Detection + Naive division
32.69
1 7.12
102.41
$ 34.85
GPT-4o + Area detection
23.08
2.49
120.06
17.2
GPT-4o + Object-aware division (using SAM without GroundingDINO)
21.01
1 4.566
135.04
J. 2.22
GPT-4o Object-aware division
19.17
6.400
120.61
J. 16.65
GPT-4o + Area delection
Object-aware division, (equiv. to LVLM-Count)
17.86
4. 7.7.1
91.71
J. 45.55
B Visual Examples of LVLM-Count's Performance on the FSC-147 Dataset, PASCAL vOC, and Emoji-Count Benchmarks
This section presents several visual examples showcasing the performance of LVLM-Coumn on the FSC-147 dataset PASCAL VOC, and the Emoji-Count benchmarks. The LVT.M used in the pipeline to generate these visual examples is GPT-4o.
Figure 8 illustrates three examples of LVLM-Count's performance on FSC-147. Additionally, Figure 9 includes an example from the PASCAL VOC benchmark. Finally, we present three visual examples from the Emoji-Count benchmark in Figure these visual examples, LVLM-Count achieves more accurate results compared to the base GPT-4o.
C Robustness to Crowded Scenes and Heavy Occlusion
One of the key strengths of our approach is its robustness in crowded scenes and images with heavy occlusions. To achieve this robustness, the masks produced by SAM are not directly used for object-aware division. Instead, several important post-processing operations are applied beforehand. The most critical of these is non-maximum suppression, which removes lower-confidence masks that overlap with others. Additionally,
17

---

Published in Transactions on Machine Learning Research (02/2026)
18
16
Ground Truth: 41
Base GPT4o: 27
LVLM-Count: 41 strawberries
15
23
28
Ground Truth: 63
Base GPT4o: 58
LVLM-Count: 66 hot air balloons
19
7
12
Ground Truth: 38
Baes GPT4o: 44
LVLM-Count: sheep
Figure 8: Illustration of three examples of the performance of LVLM-Count on FSC-147. Top row: The object of interest is "strawberry". Middle row: The object of interest "hot air balloon". Bottom row: The object of interest is "sheep".
erosion function is applied to trim the outermost layer of each mask, ensuring at least two pixels of empty space between adjacent masks to allow for reliable placement of division lines. A polishing step further refines the masks by smoothing their surfaces and eliminating artifacts, preventing division lines from mistakenly passing through them.
Figure 11 demonstrates the strong performance our method on a randomly selected image from the web containing various types flowers. This scene features extreme occlusion; nonetheless, our method effectively handles the division task. It is important to note that the strong performance our method this challenging scenario not coincidental. Figure 12 illustrates the impact of our post-processing operations. The upper
18

---

Published in Transactions on Machine Learning Research (02/2026)
16
21
Ground Truth: 41
Base GPT4o: 27
LVLM-Count: 37 bottles
Figure 9: An example of the performance of LVLM-Count on the PASCAL VOC benchmark. The object of interest "bottle".
row displays the unprocessed SAM masks, while the bottom row shows the results after post-processing, which generates reliable paths for the division lines.
To further demonstrate the effectiveness of our method in crowded scenes, present Table 6 the performance of different LVLMs FSC-147 dataset samples where the ground truth count. is 100 or higher. The table contrasts their baseline performance with their performance when using our LVLM-Count method. The results show significant improvements when LVLM-Count is employed. Furthermore, Figure 13 presents three visual examples of LVLM-Count's performance on crowded images from FSC-147.
which compares
Table Evaluation on samples with ground truth counts equal to or higher than 100 from the test set of the FSC-147 dataset. Showing the effectiveness of our method for crowded scenes.
Method
MAE +
A
RMSE
GPT-4o
89.22
291.34
LVLM-Count (GPT-40 as LVLM)
57.87
$ 31.35
185.29
I 106.05
Gemma 3 27B
125.69
330.38
LVLM-Count (Gemma 27B as LVLM)
71.68
| 54.01
251.48
78.9
Qwen2 VL 72B AWQ
102.64
302.23
IVT.M-Cunnt (Qwen2 VL. 72B AWQ LVLM)
79.66
4 22.98
289.47
12.76
D Additional Information and Visual Examples from the Penguin Benchmark
To demonstrate the effectiveness of our strategy of setting low detection threshold to overcome challenges in area detection and target segmentation stages, we evaluate it benchmark taken from the Penguin dataset (Penguin Research 2016). The goal in this dataset is to count penguins in images. This is challenging since images the dataset consistently exhibit heavy occlusion and complex background patterns that can easily be mistaken for penguins Arteta et al.] (2016). This dataset consists of two splits: i) the mixed-site split, in which images from the same camera can appear in both the training and testing sets, and ii) the separate-site split, in which images in each set strictly belong to different cameras. Images in this dataset are annotated by multiple annotators, where each annotator might identify different number of penguins due to the challenges in locating them within the images. Since annotators usually undercount the penguins, similar to Arteta et al. (2016), we take the maximum number of penguins among the annotations as the ground truth and caleulate MAE and RMSE with respect to this value. For additional details regarding the dataset. and metric calculations, we direct readers to Arteta et al. (2016).
We construct our benchmark using the separate-site split. Given the dataset's substantial size, which comprises tens of thousands of images, we randomly select 100 samples from the chosen split. To ensure
19

---

Published in Transactions on Machine Learning Research (02/2026)
Be c de g a
de o
a gg God ae gggg
De ge o coo
ge a
do
agg
te
8
DU
DO
D
80
808
00 BOB
12
24
Ground Truth: 46
Base GPT4o: 20
LVLM-Count: 44 right arrow curving left(
15
14
Ground Truth: 36
Base GPT4o: 50
LVLM-Count: 43 woman with red hair (+%
12
8
10
Ground Truth: 30
Baes GPT4o: 50
LVLM-Count: 30 hibiscus
Figure 10: Three examples of the performance of LVLM-Count on the Emoji-Count benchmark. Top row: The object of interest is "right arrow curving left". Middle row: The object of interest is "woman with red hair". Bottom row: The object of interest is "hibiscus".
balance, the sampling probability for an image with specific ground-truth annotation count inversely proportional the frequency of that count in the entire split. We evaluate two variants of our pipeline. The main variant employs GroundingDINO with low threshold for area detection and target segmentation, while the alternative variant does not use GroundingDINO at all. In this variant, masks are obtained by configuring SAM to segment any entity in the image. The results presented in (Table 4 show that both variants significantly improve the counting performance of the LVLMss, demonstrating the robustness of the initial stages in our pipeline.
Figure 14 demonstrates the segmentation masks and division paths for the two variants on sample from the Penguin benchmark. In Figure 14 (a), GroundingDINO with low detection threshold is employed.
20

---

Published in Transactions on Machine Learning Research (02/2026)
20
17
Ground Truth: 47
Base GPT4o: 33
LVLM-Count: 44 flowers
Figure 11: An example of the performance of LVLM-Count on a random crowded image from the web, involving heavy occlusion. The object interest "flower".
Black-white
Paths
Original SAM masks
No- Possible Path
Processed SAM masks
Figure 12: The effect of post-processing the masks produced by SAM in making TVT.M-Count robust to crowded scenes with heavy occlusions. The object of interest is "flower".
It correctly identifies all penguin instances, though there some false positives. These detections subsequently segmented by SAM. Figure 14 (c) depicts the division paths found using these masks. In Figure 14 (b), GroundingDINO is not used, and SAM is configured to segment all entities in the scene. As a result, in addition to the penguins, large number of other objects are segmented. Nevertheless, Figure 14 (d) shows that the division paths derived from these masks successfully partition the image without cutting any penguins. Note that some excessively large masks generated by SAM have been removed due post-processing in our pipeline.
E Real-world Application of LVLM-Count
As stated in Seccionn
counting has numerous real-world applications, including but not limited to biology, health, industry, warchousing, and environmental monitoring. Below, we demonstrate the performance of LVLM-Coumt on examples from the following areas: i) biology tioloyyheealth, /health, ii) industry /warchousing, and iii) industtyywwaacchnseste
environmental monitoring. We also compare its results with those of the base LVLM (GPT-40 for the figures
21

---

Published in Transactions on Machine Learning Research (02/2026)
32
34
50
Ground Truth: 113
Base GPT4o: 70
LVLM-Count: 116 strawberries
60
52
100
Ground Truth: 210
Base GPT4o: 105
LVLM-Count: 212 green peas
18
108
45
Ground Truth: 161
Baes GPT4o: 57
LVLM-Count: 171 beans
Figure 13: Visual examples of the performance of WVLM-Count on three crowded scenes from the FSC-147 dataset.
in this section). Note that in all examples, the cluster-based approach automatically determines the start. and end points of the division paths.
In Figure 15) images of two laboratory samples analyzed using LVLM-Count. The first row shows image from dataset introduced by Lempitsky & Zisserman (2010), which contains simulated bacterial cells from fluorescence-light microscopy, created by Lchmussola et al. (2007). The second row shows an image from the BM dataset rodueed by Kainz el al. (2015). which contains bone marrow samples from eight healthy individuals. The standard staining procedure highlights the nuclei of various cell types in blue, while other cellular components appear shades of pink and red (Paul Cohen al. 2017). As observed, LVLM-Coml achieves much higher accuracy in counting bacterial cells and bone marrow muclei in the top and bottom rows of Figure 15] respectively, compared to the base IVLM, particularly for the bone marrow nuclei.
22

---

Published in Transactions on Machine Learning Research (02/2026)
18:500 HYPERT TRPE
(a) Target segmentation with GroundingDINO + SAM
(b) Segmentation using only SAM
100000
1C:000 HyPEEr Lie
(c) Division paths for GroundingDINO SAM masks
(d) Division paths for SAM-only masks
Figure 14: rausooation of robustness in detection during the initial pipeline stages: an example from the challenging Penguin benchmark. (a) GroundingDINO with a low threshold successfully detects all penguins, though a few false positives remain. These detections are then segmented by SAM. (b) Masks produced without a detection model: Here, SAM set to segment all entities in the image. Excessively large masks are filtered out during post-processing in our pipeline. (c) Division paths are derived using the masks generated by GroundingDLNO and SAM. (d) Division paths based on SAM-only segmentation effectively partition the image without cutting through any penguins. Note that the slight difference in the area of the images on the left is due to the existence of area detection stage.
In Figure 16, two images from industrial scenes analyzed using LVLM-Count. The top row shows sectional image of a stockpile of tree logs, and the bottom row shows image from an industrial area containing barrels of various colors. For the top image, the objects of interest are the tree logs, while for the bottom image, LVLM-Count is tasked with counting the blue barrels. In both cases, LVLM-Count's 's predictions are significantly closer to the ground truth values than those of the base LVLM, particularly for the tree logs, where the ground truth number is too large for the base IVT.M to estimate accurately.
Figure 17 shows an image sourced from a dataset (Penguin Research, 2016) created as part of an ongoing initiative to monitor the penguin population in Antarctica. This dataset comprises images captured hourly by network of fixed cameras installed at over 40 locations. Over several years, this effort has accumulated over 500,000 images. Zoologists use these images to identify trends in penguin population sizes at cach site, facilitating studies on polential correlations with factors such as climate change. Thus, determining the mumber of penguins in each image is crucial. Given the challenges of engaging human annotators to process such a vast datasot, automating the counting task highly dosirable (Arteta et al. 2016). XVLM-Count is prompted to count the number of penguins in the image, and as observed, its predictions are significantly closer to the ground truth than those of the base LVLMM
23

---

Published in Transactions on Machine Learning Research (02/2026)
47
49
55
Ground Truth: 169
Base GPT4o: 98
LVLM-Count: 151 bacterial cells
Ground Truth: 112
Base GPT4o: 78
LVLM-Count: 115 bone marrow nuclei
Figure 15: Performance of LVLM-Count real-world applications in biology/health The top row shows an image of simulated bacterial cells from fluorescence-ighh microscopy (Lempitsky & Zisserman 2010 with the objects of interest being "bacterial cells." The bottom row shows an image of bone marrow, with the nuclei various cell types highlighted blue (Kainz et al. 2015), and the objects of interest being "bone marrow nuclei."
F LVLM-Count's Power in Handling Multiple Object Categories in the Same Image
LVLM-Count is a highly effective method for handling counting tasks that involve multiple objects in the same image. Its strength in such scenarios stems from the capabilities of LVLMs to answer numerous visual questions about an image and its objects. Depending on the given text prompt, it can count instances of a single object category among others or instances of multiple object categories simultaneously. In this section, demonstrate how LVLM-Count performs in counting different objects of interest, determined simply by prompt, using an image with multiple object categories.
The image in Figure 18 contains three object categories: person, cow, and horse. In the top row, the object of interest is "cow." " We prompt WVLM-Count to count the cows. First, the masks are produced through the initial stages our pipeline, and then the cluster-based approach used to automatically determine the start and end points of the division paths. It can be observed that horses have also been masked cows. Nonetheless, this does not negatively impact the final answer; it merely causes the division lines to avoid cutting through the horses as well. The counting in LVLM-Count is performed by an LVLM (GPT-40 in this figure) and does not rely on the masks. We observe that GPT-4o successfully counts the number of cows in the resulting subimages, leading to the correct final answer.
In the middle row of Figure 18 the object of interest "person." LVLM-Coume again successfully counts the number of people accurately. A more interesting case is the bottom row of Figure 18] where both cows and persons are objects of interest. We prompt LVLM-Count to count the number of "cows and persons." " Similar to the first of the figure, there false positive masks here well. However, I.T.M-Count. successfully
24

---

Published in Transactions on Machine Learning Research (02/2026)
111
43
75
Ground Truth: 205
Base GPT4o: 101
LVLM-Count: 229 tree log
Ground Truth: 28
Base GPT4o: 41
LVLM-Count: 28 blue barrel
Figure 16: Performance of LVLM-Count on real-world applications in cettustry/warhhouste The top row shows an image of a stockpile of tree logs, with the objects of interest being "tree logs." The bottom row shows an aerial image of an industrial area containing barrels of various colors, with the objects of interest. being "blue barrels."
34
Ground Truth (Max): 174
Base GPT4o: 93
LVLM-Count: 168 penguin
Figure 17: Performance of IVLM-Count real-world applications in environmental monitoring. The image is sourced from (Penguin Research 2016), initiative to monitor the penguin population in Antarctica, with the objects of interest being "penguins."
counts the number of instances from both categorics combined since the counting ultimately performed by the LVLM. Note that the number of objects in this image is limited, and GPT-4o might answer these questions correctly without the need for the IVT.M-Count, pipeline. This image has been chosen to illustrats
25

---

Published in Transactions on Machine Learning Research (02/2026)
LVL Count's power in handling multiple objects in counting task rather than for comparison with the baseline I.T.M.
2
Number of Cows:
LVLM-Count: 3 Cows
K
1
2
Number Persons:
LVLM-Count: Persons
Number of Persons and Cows:
LVLM-Count: 7 Persons and Cows
Figure 18: Illustaaioon of the ability of LVLM-Count in counting an object of interest determined by a prompt when multiple object categories exist in single image. Top row: Object of interest is "cow". Middle Object of interest is "person". Bottom row: Object of interest is "person and cow"
G False Positive Masks at the Target Segmentation Stage
One of the reasons we task an LVLM to count the objects in the subimages instead of using the number of generated masks at the target segmentation stage as the final count of the objects of interest. is the existence of false positive masks. The GroundinDDINO model is responsible for detecting the objects of interest, determined by expression F, and passing the output bounding boxes to SAM for producing segmentation
26

---

Published in Transactions on Machine Learning Research (02/2026)
masks. Nonetheless, GroundigDLNOO is not strong LVLM in understanding expressions extracted from complex questions. Thus, it often returns bounding boxes for all instances of the object category mentioned in the expression, even if those instances do not satisfy other conditions in the expression.
For example, in the top row of Figure 19, E "brown egg". However, all the eggs have been segmented regardless their color. Thus, counting the masks results a significant error. Interestingly, as we can see, the false positive masks do not negatively affect LVLM-Count's final answer, as the counting is done by an LVLM at the final stage, which is much stronger than GroundingDINO at understanding referring expressions. The only effect is that the white eggs have not been cut through by the division lines either. In the bottom row, we have chosen an image from the challenging Emoji-Count benchmark. The image contains icons, all of which have an arrow but point in different directions. However, the objects of interest are only "right arrows curving left." Similar to the eggs example, taking the masks used for object- division results a significant error.
5
3
Ground Truth:
Number of Masks:
LVLM-Count: 8 brown eggs
D 0 be dr Goo of
geggg gg
Bg a
th a
Bogge
eea
a a o
o D da
aGu o 000
Pas o
a Bago DOu
0 o
B D Begc
000 80
D
0000000
12
Ground Truth:
Number of Masks: 245
LVLM-Count: 44 right arrow curving left (
Figure 19: Top row: The object of interest is "brown egg." However, all the eggs have been segmented because of the limitation of the GroundingDINO model understanding complex referring expressions. Regardless, LVLM-Count provides a significantly more accurate number compared to the number of masks. Bottom row: The object of interest is "right arrows curving left." Similar the image of the eggs, counting the number of masks results in very large error, while LVLM-Count provides much more accurate number.
H Performance Analysis of LVLM-Count for different ground truth ranges on FSC-147 dataset
To further investigate the performance our pipeline, we divide the ground truth values in the FSC-147 test set into intervals and plot the MAE for the base GPT-4o, Qwen2, and Gemma models alongside the results from WVLM-Count using each model, as shown in Figure 20 The first interval contains relatively small ground truth values, range in which LVLMs already perform well. As the ground truth values increase, the
27

---

Published in Transactions on Machine Learning Research (02/2026)
base models exhibit increasingly larger errors compared to LVLM- with the margin growing rapidly. This behavior is consistent with our observations of counting errors on the blue circles in F Figure? 2
302
302
(GPT4o)
(Gemma
728 AWQ)
Base (GPT4o)
Base (Gemma3 278)
Base (Owen2 72B AWQ)
402
MAE
MAE
MAE
203
100
102
10.201
(20, 100]
(100, 3001
1300 [3d0, end]
[0.20]
(20, 100]
(100, 300]
(300. endl
[0.20]
(20, 100]
(100, 3001
(300,end)
Ground Truth Range
Ground Truth Range
Ground Truth Range
(a) LVLM-Count using GPT-4o
(b) LVLM-Count using Gemma
(c) LVLM-Count using Qwen2
Figure 20: Performance analysis of our method, LVLM-Count, on the FSC-147 lest set using GPT-4o Figure 20a). Gemma 3 (Figure 20b and Qwen2 Figure 20ch. In the first interval, all base LVLMs exhibit lower MAE. However, in intervals with higher ground truth values, LVLMMCount achieves a lower MAE compared to the base LVLMs. Note that as the ground truth values grow higher, the margin of improvement increases rapidly.
Report of Various Accuracy Metrics for the Performance of LVLM-Count on the FSC-147 Dataset, PASCAL VOC, Emoji-Count, and Penguin Benchmarks
This section presents various accuracy measures for the experiments reported in Tables D 2 3 and The accuracy metrics are defined in Table 7 Following these definitions, Table shows the measured accuracies for the FSC-147 test set. Similarly, Tables g 10 and [ present the accuracy metrics for the PASCAL VOC, Emoji-Count, and Penguin benchmarks, respectively. The results in the tables demonstrate that IVLM-Count generally achieves a higher accuracy than the base LVLM it employs.
Table 7: Definitions of Various Accuracy Metrics. GT denotes the ground truth number.
Metric
Definition
Acc
Percentage of answers such that answer GT
AcosEl Percentage of answers such that lasswer GT
Acc
Percentage of answers such that answer GT I 3
Acci5
Percentage of answers such that lnswer GT 5
1cc+10
Pereentage of answers such that
answer GT 10
J Inference Time Comparison
LVLM- Countt is a pipeline consisting of multiple stages. However, this section, we demonstrate that the dominant. portion of the time during inference with the LVLMs tested in this paper, i.e. GPT4o, Gemma. 3,
28

---

Published in Transactions on Machine Learning Research (02/2026)
Table 8: FSC-147 Dataset. A (1) next to the measured accuracies for LVLM-Count indicates improvement over the base LVLM uses, while (4) indicates degradation compared to the corresponding base LVLM.
Method
Ace (%)
Acc#1 (%)
Acc:33 (%)
Accı5 (%)
Acc (%)
GPT40
12.24
26.22
42.10
52.41
66.58
LVLM-Come Count (GPT44 as LVLM)
14.45 (1)
28.99 (1)
47.25 (1)
57.08 (t)
70.45 (?)
Gemma 3 27B
8.71
18.94
32.55
41.57
55.15
LVLM-Count (Gemma 3 as LVLM)
11.37 (1)
25.46 (1)
41.51 (1)
52.07 (1)
67.09 (*)
Qwen2 VL 72B AWQ
20.81
36.83
46.95
62.07
LVLM-Count (Qwen2 VL 72B AWQ as LVLM) 9.61 (1)
23.95 (1)
43.05 (1)
53.78 (1)
69.08 (*)
Table 9: PASCAL VOC Benchmark. A (1) next to the measured accuracies for LVLM-Count indicates improvement over the base LVLM it uses, while a (4) indicates degradation compared to the corresponding base LVLM.
Method
Acc (%)
Acc#1 (%)
Acc+t3 (%)
Acc+55 (%)
Acc=t10 10 (%)
GPT4o
30.39
47.06
61.11
71.90
89.87
IVLM-Count (GPT4o IVLM)
32.35 (t)
51.96 (t)
73.20 (1)
82.35 (1)
93.46 (*)
Gemma 3 27B
29.41
54.58
75.49
85.62
92.16 12.16
IVTM-Count (Gemma 27B I.LM)
30.39 (t)
50.98 (4)
78.43 (1)
85.62 (-)
94.77 (*)
Qwen2 VL 72B AWQ
24.18
42.48
60.13
72.22
86.60
LVLM-Coum (Qwen2 VL 72B AWQ as LVLM)
28.76 (1)
46.73 (1)
67.32 (1)
78.43 (1)
89.87 (*)
Table 10: Emoji-Countt Benchmarkk. A (1) next to the measured accuracies for LVLM-Count indicates improvement over the base IVLM it uses, while a (4) indicates degradation compared to the corresponding base LVLM.
Method
Acc (%)
Acc+1 (%)
Acc+33 (%)
Acc+-5 (%)
Acc+t10 10 (%)
GPT4o
1.85
5.54
13.73
21.12
43.94
LVLM-Count (GPT1o as IVLM)
2.81 (t)
9.88 (?)
23.53 (f)
36.22 (1)
60.21 (t)
Gemma 27B
0.80
1.77
4.10
7.07
16.14
WYLM-Count (Gemma 3 27B as LVLM)
2.49 (1)
5.30 (1)
13.33 (1)
19.92 (1)
38.63 (1)
Qwen2 VL 72B AWQ
0.88
2.89
7.55
11.41
19.76
LVLM-Count (Qwen2 VL 72B AWQ as LVLM)
2.33 (1)
6.75 (1)
16.22 (1)
25.70 (1)
43.86 (t)
Table 11: Penguin Benchmark. A (1) next to the measured accuracies for LVLM-Count indicates improvement over the base LVLM it uses, while (1) indicates degradation compared to the corresponding base LVLM.
Method
Acc (%)
Acc:E1 (%)
Acc:33 (%)
Acc:t55 (%)
Acc:110 (%)
GPT40
2.00
3.67
8.33
12.00
22.00
LVLM-Count (o as LVLM)
2.67 (t)
6.33 (+)
14.33 (1)
18.67 (t)
33.00 (t)
Gemma 3 27B
1.33
1.67
7.00
8.67
16.00
WWLM-Count (Gemma 27B as LVLM)
1.00 (L)
3.67 (?)
9.33 (1)
12.67 (t)
23.67 (1)
Qwen2 VL 72B AWQ
1.67
2.00
6.33
10.33
16.67
LVLM-Count (Qwen2 VL 72B AWQ as LVLM)
1.67 (-)
5.33 (*)
11.67 (1)
18.00 (t)
31.00 (1)
and Qwen2, is spent querying the LVLM to count objects, and the additional steps in the pipcline comprise a small portion in comparison.
In the four stages the pipeline, the following main steps are taken: making a call to an LLM/LVLM to extract the object of interest from Q, calling GroundingDINO for area detection, calling GroundingDINO and SAM for target segmentation, performing some post-processing on the masks, running the mean-shift clustering algorithm, running the A search, and querying the LVLM for the sub-images. From the listed
29

---

Published in Transactions on Machine Learning Research (02/2026)
steps, the running time of the post-processing masks is negligible. The time required for the rest of the steps in optimized pipeline is demonstrated in [Table 12. To obtain the inference time for each step, we evaluated it on 1,000 images from the FSC-147 dataset and calculated the average.
Table 12: Inference time breakdown of the LVLM-Count pipeline for different LVLMs (Time in seconds)
Method
Extract E from Q
Area Detection
Target Segmentation
Clustering
4
Subimage Analysia
Total
IVLMMComt (GPT4))
0.11
1,63
2.56
LVLM-Coun (Gemms
LVLM Count (Qwea2 VL 72B AWQ)
0.10
1.17
2,07
As shown in Table 12, the dominant term in the inference time comes from calling the LVLM to count objects of interest in the sub-images. This occupies more than half of the total inference time. However, similar step occurs when querying the base LVLM to count objects interest in the main image. The additional steps present in I.Y..M-ount but absent the base L.T.M constitute smaller portion of the time. [Table 13) presents a comparison between the total inference time of LVLM-Count and the base LVLMs. This increase in inference time may be acceptable for most applications, given the substantial improvements in counting accuracy when using LVLM-Coumr
Table 13: Comparison of the inference time between the base LVLM and LVLM-Count
Method
Inference (s)
Base GPT4o
1.68
LVLM-Count (Using GPT-40)
2.56
Base Gemma 3 27B
1.52
LVLM-Count (Using Gemma 3 27B) Base Qwen2 VL 72B AWQ
2.01
1.46
LVLM-Count (Using Qwen2 VL 72B AWQ)
2.07
K Comparison of Base LVLMs and LVLM-Count's Counting Performance with SOTA Counting Models as a Reference Point
LVLM-Count designed to enhance the counting performance of LVLMs. Tables D 2 and 4 in the main paper present extensive experimental results demonstrating LVLM-Counts accuracy improvements over the base LVLMs. In this section, we include the results of state-of-the-art (SOTA) dedicated counting models for the experiments reported in the main paper. Specifically, we compare the performance of GROUNDINGRECC Dai et al. (2024), COUNTGD Amini-Naieni et al. (0024), and DAVEpm Pelhan et al. (2024), as well as TFOC Shi et al. (2024). The first three models have counting-spectig training, while the latter, though a dedicated counting model, does not require counting-specife training. Reporting these results provides additional insights into the progress of counting performance in LVLMs and their enhancod performance with LVLM-Count.
Table 14 presents the results of dedicated counting models alongside LVLMs and their corresponding LVLM- Count variants the FSC-147 test set. Note that GroundigRECC, CountDD, and DAVEpem the top three nighest--erforming text-based trained models on FSC-147 in the literature. Moreover, TFOC is the best-performing training-free free model on FSC-147. We observe that, in general, the dedicated counting models (henceforth referred to simply as counting models) outperform the base LVLMs. However, the use of LVLM-Count significantly narrows this performance gap.
For training- counting models, this is considered cross-dataset evaluation, meaning that they are trained on the training of FSC-147 and tested on the PASCAL VOC benchmark. We observe that some trained counting models still outperform
Results on the PASCAL VOC dataset are reported Table 15
3This is for where sub-image queries the LVT.M made parallel
30

---

Published in Transactions on Machine Learning Research (02/2026)
Table 14: Evaluation SOTA counting models on the FSC-1447 test set serves as reference point for assessing the performance of LVLMs and their corresponding LVLM-Count variants. The column "Trained Model" indicates whether a model has been trained on FSC--47. Columns marked with A show the performance difference between LVLM-Count and its base LVLM. Improvements shown in green, while degradations are shown in red.
Method
Trained Model
MAE
A
RMSE +
A
TFOC (Shi et al. 2024
X
24.79
137.15
DAVEpmm Pelhan al. 2024
14.90
103.42
CountGD Amini-Naieni
al.
2024
/
14.76
120.42
Grounding REC Dai et al.
2024
/
10.12
107.19
GPT-40
X
25.57
137.26
LVLM-Cour (GPT-40 as LVLM)
X
17.86
7.71
91.71
| 45.55
Gemma 27B
X
30.59
132.61
LVLM-Coume (Gemnma 3 27B as LVLM)
X
20.25
4 10.34
110.45
+ 22.16
Qwen2 VL 72B AWQ
X
34.48
149.49
LVLM-Count (Qwen2 VL 72B AWQ as LVLM)
X
22.29
4 11.89
119.46
4 30.44
LVLMs; however, this is likely due to the significant overlap between the categorics in PASCAL VOC and FSC-147. Nevertheless, LVLMMCCunt is very effective in narrowing the performance gap. Additionally, the results on the Emoji-Count benchmark are reported Table 16 This also a cross-dataset evaluation. In contrast PASCAL VOC, Emoji-Count consists of complex concepts and categories that do not overlap with FSC-147. As result, LVLMs, which have far stronger generalization power for unseen data and complex concepts, outperform counting models by large margin. Employing IVT.MCCoun widens this gap even further.
Table 15: Evaluation of SOTA counting models on the PASCAL VOC counting benchmark as reference point for the performance LVLMs and their corresponding LVLMM-ount, performance.
Method
MAE +
A
RMSE +
A
TrainingFree (Shi al. 2024)
12.03
18.18
DAVE ppm (Pelhan al. 2024
12.39
22.81
CountGD (Amini- Naieni al.
2024
2.81
7.01
GroundingRec (Dai et al.
2024
4.05
7.80
GPT4o
4.64
8.56
LVLM-Count (GPT4o as LVLM)
3.42
1.22
7.17
1.39
Gemma 3 27B
3.40
7.54
LVLM-Count (Gemma 3 27B as LVLM)
2.97
0.0.43
6.16
1.38
Qwen2 VL 72B AWQ
4.80
8.71
LVLM-Count (Qwen2 VL 72B AWQ as LVLM)
4.12
0.68
7.76
4 0.95
presents the results of prior counting models on the Penguin benchmark alongside the counting performance of LVLMs. We observe that prior training-baae based counting models significantly outperform LVLMs on this benchmark. However, similar to other datasets, LVLM-Count narrows this performance gap.
Finally, Table 17
L LVLM-Count's Ability to Handle Visual Exemplars
Exemplar-base basod counting methods face challenges due the difficulty of acquiring representative exemplars. Additionally, they often struggle with intra-class variability, as objects within the same category may exhibit diverse appearances, leading noisy matches and reduced accuracy. In contrast, text-based methods offer
31

---

Published in Transactions on Machine Learning Research (02/2020)
Table 16:
Evaluation of SOTA counting models on the Emoji-Count benchmark, serving as a reference point for the performance of LVLMs and their corresponding LVLM- MaM-Connt performance.
Method
MAE +
A
RMSE +
TFOC (Shi et al 2024
64.64
87.45
DAVE pm PPllhan et a. 2024
198.99
208.08
CountGD [Amini-Naiee et al.
2024
137.93
156.80
GroundingREEC Dai el al. 2024
143.22
158.74
GPT-40
23.57
36.97
LVLM-Count (GPTT40 LVLM) Gemma 3 27B
16.57
47
33.11
3.388
21.39
24.04
LVLM- Count (Gemma3 27B as LVLM)
16.16
5.23
21.27
2.77
Qwen2 VL 72B AWQ
78.05
159.22
LVLM-Count (Qwen2 VL 72B AWQ as LVLM)
24.43
4 53.62
43.38
115.84
Table 17:
Evaluation of SOTA counting models on the Penguin benchmark, serving as reference point for the performance of LVLMs and their corresponding LVLM-Count performance.
Method
MAE
A
RMSE 1
TFOC (Shi et al.
2024)
59.34
74.40
DAVE iPelhan et al.
2024
22.29
31.38
Count AAmini-Naa Naieni et al.
2024
17.07
25.08
GroundingREC Dai et al. 2024
22.04
26.75
GPT4o
35.18
45.76
IVT.M-Count, (GPT-40 as LVLM)
26.76
8.42
38.60
4 7.16
Gemma 3 27B
49.44
60.20
LVLM-Count (Gemma 3 27B as LVLM)
34.13
J 15.31
44.11
16.09
Qwen2 VL 72B AWQ
44.02
65.59
LVLM--Cunt (Qwen2 VL 72B AWQ as LVLM)
28.21
4 15.81
40.40
25.19
greater flexibility, as textual descriptions are easily provided, modifiable, and capable encoding abstract or fine-grained concepts. This adaptability makes based approaches particularly suitable for dynamic or open-set scenarios, where predefined exemplars impractical.
Our focus in this work, similar to many recent works Liu et al. (2023 Amini-Naieni al. (0023) is on bast- counting due to the above-mentioned reasons. Nevertheless, LVLM-Count is capable of working with visual exemplars alone as well as combinations of text prompts and visual exemplars, in addition to its text-only capabilitics. The adaptation is straightforward, as most LVLMs also accept visual prompts. In Table 18. we evaluate the base GPT-4o and IVLM-Count (which uses GPT-4o) on Emoji-Count with text-only prompts, visual exemplars, and combinations of text and visual exemplars. For the disual-exeppla version, single image of the target emoji icon is provided, and the LVLM is required to count similar instances. For the combined text-and-visual version, image of the target emoji icon alongside its name is provided to the model. The results show higher accuracy when combining text and visual exemplars, while the text-only and exemplar-only versions achieve approximately the same performance.
M Chain of Thought Prompting for LVLM-Count
In this work, our primary focus is on developing a pipeline to enhance the counting ability of LVLMs for counting prompts in general, rather than optimizing for specific prompting method. Nevertheless, we investigated the impact of Chaug-of-hhouhht (CoT) prompting Wei al. (2022) on visual counting task. For
experiments, we appended the phrase 'think step-by-step" to the counting prompt. As shown in Table 19
32

---

Published in Transactions on Machine Learning Research (02/2026)
Table 18: Evaluation of LVLM-Count's performance using text-only inputs, exemplar-only inputs, and a
combination of text and exemplars on the Amoji-Count datasct.
Method
MAE 4
RMSE +
GPT4o
22.51
35.94
IVTM-Count (GPT4o as LVLM,, text)
16.29
4 6.22
32.47
+ 3.47
LVLM-Count (GPT4o as LVLM, visual exemplars)
15.66
+ 6.85
31.69
+ 4.25
LVLM-Count (GPT4o as LVLM, text + visual exemplars)
13.46
4 9.05
24.35
11.59
the results the PASCAL VOC benchmark reveal that CoT negatively affects counting performance- not only for LVLM-Count but also for the base LVLM.
Table 19: Evaluation of CoT prompting for visual counting on PASCAL VOC.
Method
MAE
RMSE 4
GPT40
4.46
8.35
GPT40 with CoT prompting
5.69
10.10
LVLM-Count (GPT4o as LVLM)
3.31
7.11
LVLM-Count with CoT prompting (GPT4o as LVLM)
3.89
7.70
N Illustration of the Complete Workflow of the LVLM-Count for an Additional Image
In this section, we demonstrate the same steps illustrated for the example image of eggs in Figures 3 and 0 for an image of zebras drinking water.
The zebra image is passed to the pipeline along with the question Q =how many zebras are in the image?". First, E =zebraa" extracted using the LLM. Then the zebra image is passed to the area detection stage, where the prompt given to GroudingDLNO is "zebras". The output bounding boxes are merged, and the resulting area cropped, as illustrated in Figure 21] The cropped area is then passed to the target segmentation stage. At this stage, Gro detects the objects of interest defined by E as the input prompt. SAM then uses the output bounding boxes to produce segmentation masks for the zebras, as shown in Figure 22
Merge the overlapping bounding boxes
Output of the area detection stage
Figure 21: Illustaaioon of the area detection step of LVLM-Count for the zebra image. For this image, Q is set to "How many zebras are in the image?". The LIM used in this step returns an E, which is "zebra". The plural form of E, "zebras", and the original image are given as input to GroundingDINO, which returns some bounding boxes (left and upper right images) that are merged to form the final detected area.
33

---

Published in Transactions on Machine Learning Research (02/2026)
(a) Dro output
(b) SAM output
Figure 22: Illustration of the target segmentation step of LVLM-Count for the zebra image. The goal is to
produce all the instance masks for E set to "zebra". The cropped image from Figure 21 together with E, is given as input to GroundngDINO, which produces the output shown in Figure 22a input to SAM, which produces the outpul shown in Figure 22b
Figure 22a then given
After the larget segmentation stage, the masks are passed to the object-aware division slage. First, the masks are used in the cluster-based approach find the location of the start and end points of the division paths, i.c., (P,, P!) and (P?, P?)). Then these masks are turned into black-and-white image, which, in turn, is mapped to a graph. The division paths are then found by connecting each start point to its corresponding end point by running the A search algorithm on the graph. The found paths are mapped back into the image domain and drawn in red, as depicted in Figure 23 The image contours are determined based on the drawn red paths, and each contour's interior masked out independently to obtain the subimages. Finally, the subimages are given to the IVLM count the number of zebras in each.
PL
P
P
p?
P1
32
600
Projection of samples onto the x-axis
End points found by the cluster-based aproach
Figure 23: Left: Illustration of the unsupervised and non-parametric method to obtain the division points (P), P!) and (P2, P2). First, few pixels are sampled (shown as points inside the segmented objects) from the pixels composing each mask. The samples are projected onto the z-axis. The projected points are clustered using mean-shift clustering. The point in the middle of two consecutive clusters is considered vertical division point. The straight vertical lines are drawn just for better visualization of the division points. Right: Illustration of object-aware division. The masks from Figure 22b are turned into black-and-while image. A dividing path is found by connecting P. to Pe using the A' search algorithm in a graph that corresponds to the binary image, where the only nodes in the graph are white pixels, which are connected to all of their white neighboring pixels. The path mapped back to the pixel domain.
Emoji-Count Benchmark Details
In this section, we provide further details about the Emoji-Countt benchmark. The full benchmark, including all images and corresponding annotations, is publicly available in our GitHub repository. The emoji icons used in daily chats can be placed in classes closely related to a specific topie. While the icons within these groups often appear very similar, they possess subtle yet important differences. Identifying specific emoji among others based on these fine distinctions requires complex level of understanding from the counting model.
Our dataset was built from the 1,816 standard Apple emojis available at the time its creation. Each emoji has unique descriptive name We first grouped these emojis into 82 non-overlapping classes based on their name's topic. To prevent ambiguity for counting model, we then removed confusing icons, such as those with names that could also refer to more specific emoji in the class. This cleaning process left with 1,197
34

---

Published in Transactions on Machine Learning Research (02/2026)
icons. From each class, randomly selected six icons for inclusion; classes with fewer than six retained all their icons. F Figure 25b shows the six selected icons from two example classes.
To generate the benchmark images, we began with an empty 1024 x 1024 pixel white canvas for each class. We then populated each canvas with instances of the icons from that class. Note that each icon may also be referred to as an object category. The procedure was as follows: for each of the six icons a class, we generated a random number between 30 and 50. A number of instances equal this random value were then placed at random positions the canvas, ensuring that the icons did not overlap and remained completely within the canvas borders. Each image is annotated with the count for every icon type it contains. The benchmark comprises 415 questions in total. (Tblle 20 provides additional statistics for the benchmark, while Figure 25a shows the distributions of the ground truth counts.
Table 20: Statistics of the Emoji-Count benchmark dataset.
Metric
Value
Total Questions
415
Total Classes
82
Total Images
82
Total Categories
415
Average Questions per Image
5.06
Answer Range
30-50
Average Answer Value
39.87 + 6.33
P Parsing Strategy for the Output of Open-source LVLMs
Unlike GPT-4o, which provides the user with the ability to format the output answer using JSON schema, the open-source models used in this work, i.c., Qwen2 VI. 72B AWQ and Gemma 3 27B, do not have such option. When prompt these models to count the mumber of instances of an object of interest, the answer is often accompanied by descriptive text that does not follow specific format, making the extraction of the predicted object count difficult. To solve this issue, we include sentence in the counting prompt given to the IVLM that asks the model to put its final prediction inside double brackets. The template prompt is: "How many object-of- [Double Brackets]]".
are the image? Report your final answer with only one number inside
After the model generates answer, use Python's regular expression package to extract the predicted number from within the double brackets. Often, this number is word rather than numeral. Therefore, after extraction, we use Python's text2num package to convert the word into its numerical form. The following code snippet shows this number extraction process. Details unrelated to the regular expression have been omitted.
import
from text_to_aun import alpha2digit
text_prompp
("How many object-of-interee are in the image.
"Report your final ansver with only one number inside [[Double Brackets]
while True:
try:
output_texx = LVLM( text_proupt, image)
ansver re. Bea pntput_text[[]] group(()r)
count
"en")
11
count int (count)
13
35

---

Published in Transactions on Machine Learning Research (02/2026)
venicles
buildings
drok
food
frutts
Vegetables
office supplies
plants
tools
bugs
household
birdl
Fisthing & arreeccaiess
rocietts
body parts
Ceitles
animal face
musical instruments
camines & felines
eiectronics
people
space
people jab
time tools
people
weather
hands
japanese buttons
people sports
colored squares
clock time
zodiac signs
sport ball
arrOWs
moon phase
traffic signs
face emotion
signs
emotion
country flag
Caoleware
emation
road
umbbeelia
Seople stance
trinnless
people holding
book
Watler
Jockss
math
banknote
diamonds
medel
Smusscmentt park
purctuation Amphib ans reptiles
trains
moneyy emotion
dinosaurs
blced
mailbox
This pie chart illustrates the 82 classes in the Emoji-Count benchmark. Inside each class there are to type of emoji icons that are related to the topic of the class. For example, the emoji icons inside the plants and hands classes, have been demonstrated in Figure 25b
Figure 24:
13
except Exception:
14
pass
Despite the format instruction in the prompt, the models might occasionally fail to produce an answer in the required format. This results an error, which is handled by the code above. The model is then asked generate new output until a successful prediction is extracted.
Let us refer to unsuccessful attempt to extract the predicted number failure. We measure the average failure rate per image and report it for each of the datasels in the main paper for the two open-source models that we use, i.e., Qwen2 VL 72B AWQ and Gemma 3 27B. These results are reported in Table 21
Q Non-zero Prediction of LVLMs for Subimages with No Objects of Interest
mentioned, LVLMs occasionally predict non-zero values for images that contain no object of interest.
our pipeline, it very rare for subimages generated from images that contain a non-zcro number of objects of interest not contain any objects of interest. The area detection stage ensures that areas containing objects of interest cropped, and the clustering step in the object-aware division stage finds the division points
36

---

Published in Transactions on Machine Learning Research (02/2020)
Class
Object Categories
Emoji
pinching
25
2- right-faiing fist
3- backhand index pointing down
hands
writing hand
fist
Freqency
15
me hand
1- palm tree
2- seedling
3- maple leaf
plants
cherry blossom
hibiscus
6- rose
Ground Truth Count
(a) Emoji-Count ground truth count distribution
(b) Two class examples from Figure 24
Figure 25: The figure illustrates: (a) the distribution of ground truth counts the images of Emojj-Count. benchmark, and (b) the object categories of two example classes, plants and hands
Table 21: Average failure rate of extracting the predicted number of open-source models across different benchmarks
Failure Rate
Method
FSC-147
PASCAL VOC
Emoji-Count
Penguin
Base Gemma. 3 27B
0
0
IVT.MCount (Using Gemma 3 27B) Base Qwen2 VL 72B AWQ
R 0
0
R 0
(
0.07
0.06
0.09
0.13
LVLM-Count (Using Qwen2 VL 72B AWQ)
0.07
0.08
0.01
0.12
that the distribution of potential objects of interest in different subimages balanced. However, we provide quantitative information on this phenomenon for our PASCAL VOC benchmark.
In total, for only 9.80% of the images in the benchmark, at least of the subimages generated through our pipeline contains no instance of the queried objects. Half of these cases are images that have zero ground truth count to begin with. Excluding those cases, the percentage drops to 4.90%. We report the average non-zero prediction rate for the subimages with a zcro ground truth count and the average predicted values in Table 22] For each LVLM, we report. two cases. one case, the input lext is simply a counting prompt; the other case, the input text includes the following sentence in addition to the counting prompt: "Say if you do not any." For the sake of brevity in the table entries, let refer to this simple technique Zero-aware prompt. As observed in the table, this simple technique is quite effective in reducing the error. Note that the standard IVTMMCount. uses this technique.
Table 22:
Average non-zero prediction rate and average predicted values for the subimages with zero ground truth count in PASCAL VOC benchmark.
Method
Average Non-zero Prediction Rate
Average Predicted Value J
IVLM-Comn. (GPT4o w/o Zero-aware prompt)
0.39
0.78
LVLM-Count (GPT4o)
0.15
0.15
WLM--Cout (Gemua
w/o Zero-aware promptt)
0.45
WVLM-Cont (Gemin 27B]
0.18
Count (Qwen2 VT. 72B AWQ w/o Zero-aware prompt)
0.42
0.51
LVLM-Count (Qwen2 VL 72B AWQ)
0.24
0.27
37

---

Published in Transactions on Machine Learning Research (02/2026)
R
Reproducibility Table for Implementation Parameter
This section provides
Table 23 which lists the values of key parameters used in our implernentation.
Detection threshold for area detection: This threshold controls the acceptance of bounding boxes proposed by GroundingDINO, based on the probability of each box during the area delection stage.
Detection threshold for target segmentation: This parameter is defined similarly but applies to the target segmentation stage.
Mask NMS IoU threshold: This sets the maximum acceptable Intersection over Union (IoU) value between two masks in the NMS algorithm. If two masks exceed this loU, the one with the lower probability is removed.
Mask crosion thickness: This determines the size of square kernel (cv2.erode used to crode the outer layer of masks. This prevents obstructions for division paths when two adjacent masks overlap.
Mask refinement thickness: This defines the size of square kernel (cv2.morphologyss) used refine the surface of potentially pitted masks.
Table 23: Reproducibility table for implementation parameters
Parameter
Value
Detection threshold for area detection
0.1
Detection threshold for target segmentation
0.1
Mask NMS ToU threshold
0.4
Mask erosion thickness
pixels
Mask refinement thickness
pixels
S
Dividing Images Along Both the Horizontal and Vertical Axes
As described in Section 3.3 in the object-aware division stage of IVLM-Count, the divisions are vertical. This is to keep the pipeline simple. Nonetheless, it is possible to perform the same steps to obtain horizontal division paths. In Table 24| we report the performance of a variant of IVT.M-Cout that leverages horizontal divisions addition to vertical divisions on the FSC-147 dataset. Additionally, Figure 26 shows a visual example of this variant's performance on a sample from the FSC-147 dataset.
Table 24: Evaluation of version of LVLM-Coum that leverages divisions along both vertical and horizontal axes the FSC-147 dataset.
Method
MAE
A
RMSE
A
GPT-4o
24.48
125.38
IVTM-Count (GPT-4o as LVLM, vertical & horizontal divisions)
17.67
6.8.1
90.61
3/3447
Gemma 3 27B
30.59
132.61
IVLM-Count (Gemma 3 27B as LVLM, vertical & horizontal divisions)
18.44
12.15
103.91
28.7
Qwen2 VL 72B AWQ
34.18
149.49
IWLM-Count (Qwen2 VL. 72B AWQ as LVLM, vertical & horizontal divisions)
18.19
15.99
104.12
45.37
38

---

Published in Transactions on Machine Learning Research (02/2026)
12
12
15
3
4
4
7
10
Ground Truth: 69
Base GPT4o: 96
LVLM-Count: 74 strawberries
Figure 26: A visual example of the performance of LVLM-Count on a sample from FSC-147 when it leverages both horizontal and vertical divisions
T Evaluation of the counting methods on TallyQA-Complan
TallyQA (Acharya et al. 2019) an open-world spen-world counting dataset that includes complex counting questions involving relationships between objects, attribute identification, reasoning, and more. It is fairly large dataset, with the training set containing 249318 questions and the test set having 22991 simple and 22991 complex counting questions. The number of objects in each image ranges from 0 to 15. We randomly sampled 10 questions per ground truth count from the complex counting questions the test set. This resulted in 149 complex counting questions total, since some ground truth values have fewer than 10 samples available. We refer to this benchmark as the TallyQA-Compla benchmark.
We compare our method, using GPT-4o and Qwen2 VL 72B AWQ the LVLM, against their corresponding base models. For reference, also report the cross-domain performance of three of the best-performing trained counting models-Grounnn RroundingR..C CountGD, and DAVE
well as the only prior training-free model, TFOC. The results are shown in Table 25 Note that CountGD and DAVEpm use weights trained on FSC-147, while GroundingREC uses weights trained on the REC-SK dataset (Dai et al. 2024), which its authors introduced alongside the model. Our method improves the MAE over the base LVLMs. Moreover, note that SOTA counting models are outperformed by the base LVLM models, further confirming that LVLMs possess better generalization and complex reasoning capabilities for counting tasks.
Table 25: Evaluation of the performance LVLMs and their corresponding IVLMM-Cunt as well as SOTA counting models on the TallyQA- AallyQAACooppex Complex counting benchmark.
Method
MAE 4
RMSE 1
A
TFOC (Shi et al. 2024
12.41
22.80
DAVE Eprm Pelhan et al. 2024
28.36
57.56
Gount [Amini-Naien et al.
2024
9.78
17.21
GroundingRec Dai et al. 2024
5.83
10.13
GPT4o
2.60
4.74
IVLM-Count (GPT4o IVLM)
2.28
0.32
4.18
0.56
Qwen2 VL 72B AWQ
3.21
5.35
LVLM-Count (Qwen2 VL 72B AWQ as LVLM)
2.47
0.74
4.35
1.00
39


Reviews:
[{"review_id": "p2e20VzErh", "reviewer": "Reviewer_ZbBN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce a method to improve LVLMs' counting abilities. They main idea is to improve the division of images in a divide-and-conquer approach such that objects are not split apart. This improves the accuracy especially in images with many objects for several tested models (GPT-4o, Gemma 3, Qwen2). The idea of object-aware division is simple and straightforward, well explained, and experiments show that it is effective.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes, the claims are supported by experiments performed on several datasets (FSC-147, PASCAL VOC Benchmark, Emoji Count, Penguin Benchmark). The MAE and RMSE of the counted number of objects is consistently improved compared to the baseline approaches. Visualizations in the appendix show some of the divisions and look reasonable.", "audience_yn": "Yes", "audience_explanation": "Counting objects in images is an important real-world task that state-of-the-art methods cannot handle sufficiently well, yet. The paper improves results of this task successfully.", "requested_changes": "The paper lacks some details to make results reproducible:\n\nQ1) page 6: what is the \"very low\" value you set for GroundingDINO?\n\nQ2) When using MeanShift Clustering, which version of it do you use (maybe I missed the citation)? Which bandwidth do you use? This can change the results significantly. \n\nQ3) What is the runtime and complexity of your method?\n\nQ4) On page 7 you state \"for images with a very large number of objects, sometimes LVLMs refuse to count\" - When or how often does this happen? Why does this happen? For which LVLMs does this happen? What changes if just a part of the image is given?\n\nQ5) Can you elaborate more on the limitations? E.g., can the erosion cause very small objects to disappear? Experiments on images with very small objects that should be counted would be interesting.", "broader_impact_concerns": "Not applicable.", "additional_comments": "Given that this is not my primary area of expertise, I am not familiar with the state-of-the-art baselines the authors could have benchmarked against, so I cannot assess the completeness of their comparisons.", "extra_scores": {}, "extra_text": {}}, {"review_id": "xx0u1UKKW7", "reviewer": "Reviewer_ECMU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Contributions:**\nThe paper combines a pipeline of three types of VLMs (GroundingDINO, SAM, multimodal LLM such as Qwen2-VL) with a  more traditional computer vision algorithm for the task of counting.\nTo me the main contribution is enhancing existing “end-to-end” LLMs with more rule-based or structured steps that help with hard counting tasks. Just like in humans, once we go beyond a couple of items (as in this paper 20 or more items often), we need “external help” to keep track of things and categorize them: using our fingers to count, pointing individually at each item with our index finger to not double count, or going from top left to bottom right to not double count.\nSo it makes sense to not rely on an end-to-end pipeline.\n\nThe authors evaluate 3 off-the-shelf multimodal LLMs on 3 existing benchmarks as well as a new emoji-based counting tasks (an additional contribution).\n\nThey show that while VLMs are great on counting a few objects (e.g. below 20), they struggle with large numbers off-the-shelf. While their proposed LVM-Count method does not alleviate the issue completely, models are now closer to the true number.\n\n**Good:**\n\nThe paper adds a straightforward well-defined contribution to the literature, for a task where it does makes sense to combine end-to-end neural models (LLMs) with structured models or rules.\n\nThe paper is easy to follow. Especially the sequence of figures make it easy to understand the method in Section 3.\n\nFinally, from the appendix it is clear the authors have ran many additional experiments and explorations.\n\n**Weaknesses:**\n\nAt the beginning of section 4.2 the authors list four different details that are in different appendix sections. Instead of just listing them, it would be good to summarize such findings in the main paper nonetheless and then refer to more details in the appendix. Reading so many references to appendix without any actual insights can break the reading flow.\nEspecially something like Appendix K, where you show how established non-LLM baselines perform, should definitely go in the main paper and main tables.\n\nIn general there is several of ablations or baselines that come up during reading the paper and are not addressed (see below).", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claim of the paper is simple “Here is a method to improve on top of existing LVLM’s counting ability”, and the experiments show this accurately.", "audience_yn": "Yes", "audience_explanation": "multimodal LLMs are widely used and counting is one of their established failure modes. Thus, many researchers will care about how to address this.", "requested_changes": "I believe based on the definition of TMLR what counts as acceptance (“claims made in the submission supported by accurate, convincing and clear evidence”), this paper matches the TMLR criteria. But I do think some crucial things are still missing, open questions or ablations a reader might want answers for.\n\n**Needed changes:**\n\nMove more appendix ablations into the main paper and integrate them appropriately into the main flow. As is, there is very little analysis or ablations after showing the main results table.\nEspecially Appendix K contains very interesting baselines that often surpass the current method. This needs to be in the main paper, and contextualized: Why would one still prefer the given method? Why are those baselines good on some datasets and not others?\n\n**Optional:**\n\n1. Are there simpler simpler/different approaches?\nSo if the SAM model already segments the objects, why still ask LVLM to count? Can’t you just count the segmentation masks rule-based, making sure to not double-count?\nOr ask the LVLM/CLIP etc to classify each segmented area individually? \nE.g. the pipeline would be the same but in the step where you divide the image with the path finding algorithm, you instead pass each segmented area to a LVLM/CLIP?\nThis would result in a lot of LVLM calls per example, but would be worth to know as a reference?\nWould this not work because as you said SAM/GroundingDINO are not perfect so passing larger chunks of images to the LVLMs mitigates that?\nIt would be great to explore why asking the LLM to count in the last step of the pipeline is needed.\n\n2. Another ablation: The path finding algorithm vertically splits the image into 3-4 parts (based on examples I see in the appendix). Why not more splits, both vertical and horizontal, such that the LLM is asked to count on e.g. 5 or 10 sub-images?\n\n3. Section “3.3 Object-aware division” is confusing to follow → e.g. clarify that choosing x-axis is arbitrary and you might as well choose y-axis? Or could you choose both? Please try to make this section less about explaining the intricacies of the algorithm and more about the high-level design choices. Perhaps the actual algorithm could just be references to the appendix and just said it is a path finding algorithm to split the image into parts, such that we don’t cut through images.", "broader_impact_concerns": "The datasets used do not contain people, so I don't see direct ethical concerns. Fine-grained visual recognition can always be used in surveillance scenarios, but I do not think this specific paper requires a full discussion on this.", "additional_comments": "It would be interesting to address how counting (at least when framed as this more hybrid approach with several steps of segmentation etc) is not just a trivial extension of the segmentation task? What are unique challenges beyond segmentation and maybe referring expressions? Related work (2nd paragraph) seems to address this a little but does not say how it falls short exactly.\n\nI kept wondering as a reader: What exactly is the challenge here beyond established task formulations such as segmentation/referring expressions? And, is this pipeline the best way to address it?", "extra_scores": {}, "extra_text": {}}, {"review_id": "05xGW2sWUn", "reviewer": "Reviewer_bvKd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper identifies a real weakness in current LVLMs: they perform reasonably when only a few objects need to be counted, but their accuracy drops sharply as the numbers grow. To address this, the authors introduce LVLM-Count, a training-free, plug-and-play pipeline that combines expression extraction, area detection, segmentation, and—most notably—an object-aware division mechanism. The division avoids splitting through objects by combining clustering and A* path planning, which is both elegant and practical. The work also contributes a new benchmark, Emoji-Count, that stresses fine-grained distinctions and highlights where models still fail.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper evaluates multiple LVLMs across FSC-147, PASCAL VOC, Emoji-Count, and a Penguin benchmark with heavy occlusion. Across these, LVLM-Count consistently lowers MAE/RMSE vs. base models; e.g., on Penguin, GPT-4o MAE drops from 36.01 → 26.95 (main variant), with similar gains for Gemma 3 and Qwen2; a SAM-only variant also helps, underscoring robustness. The pipeline and object-aware division are described concretely (A* over mask graphs), and ablations/coarse analysis indicate why naïve splitting fails while object-aware paths help. The paper also reports the modest inference-time overhead (e.g., GPT-4o: 1.68s → 2.56s), and even notes that CoT prompting can hurt relying on VOC—usefully candid. Two caveats: statistics (variance/significance) are light, and some regressions exist on specific cases, but overall, the evidence is careful and convincing.", "audience_yn": "Yes", "audience_explanation": "Likely high. A training-free, drop-in baseline that helps counting is useful to both researchers and engineers. The division idea may transfer to other dense-reasoning tasks.", "requested_changes": "1. Add variability stats (95% CIs or bootstrapped SEs) for the headline numbers; call out regressions.\n2. Describe Emoji-Count in more detail: categories, sampling strategy, splits, and licensing. Provide class definitions/splits and usage terms so others can adopt the benchmark.\n3. Offer a clearer strategy for parsing outputs of open-source LVLMs (to handle non-numeric responses). Document the enforced schema (e.g., [[N]]) and failure rates; release the exact regex/parsers. \n4. Quantify the “false positive counts” on zero-object images and suggest remedies. Quantify the non-zero predictions and propose a guardrail.\n5. Gather implementation details (thresholds, NMS, erosion, mask refinements) into a single reproducibility table. Consider releasing a minimal config file. \n6. The resource link in §4 returned “The requested file is not found” when I checked (Oct 3, 2025, PST). Please fix and ensure code/data are accessible.", "broader_impact_concerns": "Stronger counting helps ecology/field studies but raises privacy risks in surveillance. I’d encourage deployment guidance (schema checks, zero-object guards) and reporting error bars to avoid over-trusting counts in the wild.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)