Improving CLIP Counting Accuracy via Parameter-Efficient Fine-Tuning
Authors: Ruisu Zhang
Yicong Chen
Kangwook Lee
Venue: Accepted by TMLR ()

Abstract: We focus on addressing the object counting limitations of vision-language models, with a particular emphasis on Contrastive Language-Image Pre-training (CLIP) models. Centered on our hypothesis that counting knowledge can be abstracted into linear vectors within the text embedding space, we develop a parameter-efficient fine-tuning method and several zero-shot methods to improve CLIP's counting accuracy. Through comprehensive experiments, we demonstrate that our learning-based method not only outperforms full-model fine-tuning in counting accuracy but also retains the broad capabilities of pre-trained CLIP models. Our zero-shot text embedding editing techniques are also effective in situations where training data is scarce, and can be extended to improve Stable Diffusion's ability to generate images with precise object counts. We also contribute two specialized datasets to train and evaluate CLIP’s counting capabilities. Our code is available at https://github.com/UW-Madison-Lee-Lab/CLIP_Counting.

Published in Transactions on Machine Learning Research (01/2025)
Improving CLIP Counting Accuracy via Parameter-Efficiet Fine- Tuning
Ruisu Zhang
rahong845@wussc.cdt
Deportment of Electrical and Computer Engineering University of Wisconsine Madiisn
Yicong Chen
ychen22299vvisc.com
Department of Electricol and Computer Engineering University of Wisconsin-Maann Madison
Kangwook Lee
kangwook lee @wiscc edu
Department of Electricol and Computer Engineering University of Wisconsin-Madison
Reviewed on OpenReview: http:://ppppneei...nn////omommoa
Abstract
We focus on addressing the object counting limitations of vision-language models, with a particular emphasis on Contrastive Languago-Image Pre-training (CLIP) models. Centered on our hypothesis that counting knowledge can be abstracted into linear vectors within the text embedding space, we develop parameter-efie efffieetr-efficient fine-tuming method and several zero- shot methods to improve CLIP's counting accuracy. Through comprehensive experiments, we demonstrate that our learning- based method not only outperforms full-model fine-tuning in counting accuracy but also retains the broad capabilities of pre-trained CLIP models. Our zero-shot text embedding editing techniques are also effective in situations where training data is scarce, and can be extended to improve Stable Diffusion's ability to generate images with precise object counts. We also contribute two specialized datasets to train and evaluate CL.PP's counting capabilities. Our code is available at /nttss://githbb.com com/ UV-Madison Lee- Leta-LLLLLLL//LLLLCCCCCiii
1 Introduction
Recent advancement of deep learning techniques has led to significant progress in vision-language mod-
els (Alayrac et al. 2022 Chen et al. 2023 Radford al. 2021 Singh et al. 2022 Liu et al.) 2023
et.
al.
2023a One such breakthrough is the development of Contrastive Language- Image Pre-training
(CLIP) [Radford et al.)
2021,, which is trained on a wide range of internet text-image pairs (Schuhmann 2021). CLIP has demonstrated strong performance in zero-shot learning tasks and serves as a backbone
et al.
for text-to-image generative models like Stable Diffusion. (Rombach et al. 2021
Despite its extensive deployment, CLIP exhibits limitations certain areas RRdffrd 2021 Thrush et al. 2022 Paiss et al. 2022), , such counting objects in images (Paiss et al. 2023). Counting is fundamental skill that requires the integration of visual and linguistic understanding, and it plays crucial role in numerous practical applications. Existing works have attempted to address the object counting
2021 Liu al,
limitations with CLIP models or CLIP-based models (Paiss et al. 2023 Jiang et al. 2023a Mestha et al. 2024 Binyamin et al. 2024). However, these methods often require extensive training on large datasels.
Our work socks a deeper understanding of CLIP's object counting capability and introduces a data-cfficient and computo-efficient approach enhancing it. We contribute two specialized datascts for training and evaluating CLIP's counting capabilities.

---

Published in Transactions on Machine Learning Research (01/2025)
Our key idea hypothesizes that counting knowledge can be abstracted vectors in the text embedding space, distinct from non-counting information. Our method seeks to find an object-agnostic counting vector that represents the concept of a count (e.g, "five"), which is not necessarily the text embedding of the count word. Once such a vector representation is identified, it is added to the text embedding of the original caption (e.g,, "an image of five dogs") to reinforce the counting signal. This idea of finding counting-specific vectors also provides a parameter-officent and scalable solution that avoids extensive model training, and can be universally applied in tasks that involve counting any given type of object.
In our paper, we develop a learning- based method and several zero-shot methods to obtain the counting representation. First, we employ counting-specific contrastive loss (Paiss et al.) 2023) to train these vectors. Our experiments reveal that this counting vector training not only outperforms traditional full-model fine-tuning methods in terms of counting accuracy but also avoids the issue of the pre-trained CLIP model losing its broad capabilities when fine-tuning the entire model on new data.
Furthermore, in scenarios where direct training data is scarce, we demonstrate that counting vector can be formulated following simple rules can be extracted from objects that CLIP counts more proficiently, To demonstrate the benefits of improved counting accuracy of CLIP, also test the fidelity on text-to-image models. In particular, we test our zero-shot method on Stable Diffusion models (Rombach et al. 2021 | and provide examples to show that it helps Stable Diffusion models generate images with precise object counts as specified in the caption.
Our contributions include: (i) a parameter-effiiient training method that enhances CLIP's counting accuracy while preserving its capabilities; (ii) zero-shot text. embedding editing techniques effective without training data; (iii) two novel datasets for finc-tuning and evaluating CLIP's counting ability; and (iv) demonstration of how our zero-shot approach guides lext-to-image models like Stable Diffusion to generate images with accurate object counts.
2 Related Work
Vision-language models Vision-languee models (VLMs) have achieved significant success in multimodal tasks by training on massive image-text datasets and excelling in zero-shot or fine-tuned downstream
tasks (Alayrac ct al.] 2022 Chen et al., 2023 Radford et al. 2021 Singh et al., 2022 Liu ot al., 2023 Li ct al,, 2023a). In this work, we will focus on the Contrastive Language-
Pre-training (CLIP) model trained by OpenAl (Radford al. 2021). CLIP is trained on 400 million image-caption pairs (Schuhmann et al.) 2021 using contrastive objective where matching text-image pairs are optimized to have low cosine distance, while mismatched pairs are pushed further apart. CLIP has demonstrated notable success across range of visual tasks due to its zero-shot capabilities. It also underpins text-to-image alignment in generative models like Stable Diffusion (Rombach et al. 2021
Limitations of vision-language models on counting While VLMs perform impressively across various tasks, they struggle with specific challenges, such as counting objects in images (Radford et al.) 2021 Liu
et al.
2021 Thrush et al.] 2022 Paiss et al. 2022 like counting objects within pictures (Paiss el al. 2023
In fact, the object counting problem has always been one of the important issues in the visual question
answering (VQA) field, and several studies have attempted to address it (Jiang et al. 2023b Xu et al. 2023
Zhang et al. 2018 Nguyen et al. 2021 Acharya et al. 2019
One reason for CLIP's limitations in object counting may stem from its mini-batch contrastive pre-training process. Typically, these models rarely process images of the same object in varying counts within a single training batch. As a result, CLIP models have minimal exposure to learning muanced differences in object counts during pre-training. While full-batch training could address this issue, it is often prohibitively costly and impractical.
One potential solution during pre-training is to strategically select mini- batches under certain conditions, allowing them to mimic full-batch optimization (Srcenivasan et al.) 2023). Using a tailored training set and loss function can better guide models in learning count. (Paiss et al.] 2023 Mestha al., 2024). For example, Paiss et al. (2023) fine-tunes pre-trained CLIP models using counting-specffic loss
2

---

Published in Transactions on Machine Learning Research (01/2025)
counting-relevant dataset filtered from the LAION-400M dataset (Schuhmann al. 2021 It also introduces new image-text counting benchmark, CountBench, used to evaluate a model's s understanding of object counting, which we also utilized in our experiments, Mestha et al. (0224) make further improvements by rodesigning the contrastive loss. Another work, CrowdCLIP (Liang ct al. 2023 focuses on the crowd counting problem, fine-tuning CLIP in an unsupervised manner to map crowd patches to count text. Concurrent. research has focused on enabling VLM-drriven image generation models to produce images with accurate item counts (Paiss et al., 2023 Li et al. 2023b Kang et al. 2023
Table 1: The counting evaluation of the fine-tuned CLIP-Count model varies across diverse objects. We use a CLIP-Count model checkpoint following its Visual Prompt Tuning and evaluate the Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) on our benchmark datasct Obj jcjecnttnn.. For comparison, we also compute MAE and RMSE for the pretrained CLIp-base- model. The results indicate that CLIP-Count struggles with small-count tasks, performing poorly and showing a significant gap compared to the pretrained CLIP model.
dogs
lions
chairs
gonts
cUws
cherries
boats
MAE
9.87
50.02
16.59
28.50
35.56
34.16
13.01
25.53
56.32
C.Int
RMSE
10.98
60.02
22.55
54.95
38.17
38.17
19.01
28.93
106.2
MAE
0.29
0.41
0.78
0.77
1.25
0.52
0.58
0.94
0.75
ELIP-base-1
RMSE
0.58
0.70
1.16
1.06
1.60
0.76
0.89
1.23
1.16
Density estimation framework vs. classification-basedd approaches in object counting Open- world counting tasks are often addressed using density estimation frameworks, which treat counting as a continuous regression problem, as demonstrated by CLIP-Count (Jiang et al. 2023b). In these methods, the model predicts a densily map over an image, with performance typically evaluated using mean square error (MSE). This framework works well for large-scale object counting in high-density scenarios, where continuous approximations are often sufficient. This approach optimally suits high-count datasets, where density maps and MSE are effective evaluators of performance.
In contrast, classification-basse based approaches like ours formulate counting as discrete classification task. Here, each potential object count is treated as a separate class, allowing the model to explicitly predict specific, integer- based counts. This task formulation is particularly well-suited for low-count scenarios, where the focus is on discrete accuracy rather than a smooth approximation.
While density estimation frameworks are powerful in high-density, large-count settings, they do not effectively capture the model's reasoning abilities in low-count scenarios, shown in Table [ Classification based approaches offer a more nuanced assessment, emphasixing object type recognition and specific count accuracy. Thus, our work extends prior research by highlighting the distinct advantages of classification frameworks for low-count precision and reasoning in vision-language models.
Linear word analogies and embedding editing Since word2vec (Mikolov et al., 2013a was developed, researchers have found that the differences between word embedding vectors could capture relationships between words (Mikolov et al.) 2013b Drozd et al. 2016 Ethayarajh et al. 2018 Allen & Hospedales) 2019). For instance, the vector direction from "queen" " to "king" captures a semantic transition from female to male. Building on this foundation, researchers have applied text embedding editing techniques to the field of image editing. Two works have explored the application of text embedding editing methods to image editing. One work (Parmar et al. 2023) discovers editing directions in the text embedding space and applies them to image edits, while leveraging cross-attention guidance to preserve the structure of image content. Another work (Nguyen el al. 2023 uses example pairs of 'before' and 'after' edits to infer lext-based editing directions, which are then applied to new images, similar to (Parmar et al. 2023 Our research makes distinct contributions: (i) Using orthogonal projections, we filter extrancous details to achieve procise text embedding edits; (ii) Instead of focusing solely on image editing, we transfer CLIP's counting ability across objects to improve counting-related tasks, such classification and generation.
3

---

Published in Transactions on Machine Learning Research (01/2025)
3 Methods
In this scction, outline our approach to investigating and validating the hypothesis that the knowledge related to counting in images can be represented in direction in the text embedding space such that it is independent of the object's embedding. Our approach enhances CLIP's counting abilities through learning-basse methods that leverage new datasets and zero-shot text embedding editing techniques. We introduce the representation of counting concepts as vectors in Section 3.3 our parameter-efiie
fine-tuning method in Section 3.4 and our zero-shot methods Section 3.5 In Section 3.6 we will introduce how we collect and apply new counting datasets. Prior the main method sections, we will first give brief overview of CLIP models in Section 3.1 and define the counting problem and how to evaluate CLIP's counting ability
in the following Section 3.2
3.1 Overview of CLIP
Contrastive Language-Image Pre-Traine Pre-Training (CLIP) Radford et al. 2021) is a pioncering model developed by OpenAI that effectively integrates visual and textual data processing. The fundamental concept of CLIP involves the simultaneous training of two distinct. encoders: an image encoder and a text encoder. These encoders are designed produce embeddings that are closely aligned for corresponding image-text pairs and distinct for non-matching pairs.
The primary objective of CL.IP is to minimize contrastive loss that encompasses both image-to-text and text-to-image directions. Specifically, for a given dataset with N samples, where μi E Rt denotes the normalized image embedding and vi denotes the normalized text embedding for the 7th sample, the CLIP loss function defined as:
LCLIP
Σ
log
exp(ps vt))
1 2NV
Σ
log
cxp(14 v4) k= explps vk) )
(1)
2N
N Ek=1
exp(u4 us)
where we use to denote the dot product between two vectors.
CLIP models are pre-trained on a large-scale dataset comprising 400 million image-text pairs sourced from the Internet. This extensive dataset enables CLIP to generalize well across different types of images and text found in real-world scenarios.
3.2 Evaluation of CLIP's counting accuracy
We start by defining the necessary notation. Let yobi E Rd be the CLIP text embedding vector for a caption like "an image of dogs", which identifies a specific object without indicating quantity, where d the embedding dimension of a CLIP model. o0 Rd represents the text embedding of caption that includes quantifier, where i is the quantity. For example, the embedding of the text "an image of three dogs" could be represented by U' .dog
Image Encoder
Projecton& Normalzation
To assess CLIP's counting performance, we set up an image classification task where the goal to find correct caption that describes the object count correctly in given image, illustrated in Figure
Rorattane matttttttnn
CLIP
Linaa Repeor
Text
Apar: Prediction Parrngg
Encoder
Following Paiss et al. 2023 we consider counts from two to ten as a nine-class classification task. Specifically, given an image with n specific objects and nine candidate captions (e.g., "an image of objects" for between two and ten), we first encode the image with CLIP's image encoder μ n and each caption with CL.P text encoder to nb The cosine similarity
Soda
Figure 1: An illustration of object counting as classification task with CLIP.
4

---

Published in Transactions on Machine Learning Research (01/2025)
between the y and each caption's text embedding U/ obi is computed and used to select the caption yielding the highest similarity score.
3.3 Representation of counting knowledge as vectors
We define the representation of counting knowledge for each number as a vector A, E Rd aligned with the dimensionality of CLIP's embedding space. Therefore, for the nine-class classification task that we consider, there is set of 9 vectors representing different. counts, each being denoted A, E Rd for i E 32, .... 10).
We then process these vectors, obtaining count vectors that are orthogonal to yobl, to climinate information associated with object representation yet not contributing to object count. Accordingly, we introduce A to denote the part of A that is orthogonal to yabj, demonstrated in the top figure in Figure 2
Then, we derive a Nounting-augmentin object representation D6 from original representation U; abj and the orthogonalized counting representation A;, such that
obj
A
D = ob) + A.
(2)
A
where A = A.
yobl, ob)
obj Vi
Note that our method only manipulates CLIP's text embedding and keeps its image embedding unchanged. The cosine similarity score between the original image embedding μoh and each manipulated text embedding bpb) is calculated to determine the object count in the caption with the highest similarity score.
A
oobi
The choice of forcing A; to be orthogonal to زoo instead of to obj is based on empirical results, elaborated in the ablation study Section 5.2 We hypothesize that o0t) already contains some level of counting information. Thus, if we lot A; to be orthogonal to vi there might be some loss of counting information. Similarly, the choice of bgth instead of yobi in Equation 2 is also based empirical studies. We also
Figure 2: Representing count- ing knowledge as vectors.
hypothesize that the existed counting information in وbو will reinforce the counting signal in 0ob)
3.4 Learning counting knowledge vectors via counting loss
In this section, we introduce learning-based method, also demonstrated in Figure 3 to obtain counting representation vectors A,, by minimizing a counting-specific loss defined in the CountBench paper (Paiss 2023). Specifically, given a pre-trained CLIP model, we freeze all its pre-trained weights and optimize only for i E (2, 10,, which has only 9d parameters.
et al.
To prepare for training, for each ground truth image-caption pair within the training dataset, we create eight counterfactual caption variants by manipulating only the object count in the original caption. For instance, if the correct caption is "two dogs," counterfactual variants include "three dogs," " "four dogs," "ten dogs." " Our objective is to enhance similarity scores between the text-image pairs with correct counts compared to the counterfactual pairs, thereby improving counting accuracy. The counting loss Lcount is defined by Paiss
ct al. 2023) follows:
Leount
1 N
Σ 2.j
log
exp(x* vit) exp(u ti + exp(ut (i,t))
(3)
where _² is the normalized image embedding of the 4th sample in one batch, G is the normalized text embedding of a caption containing ground truth count I of sample k; and uf represents the embeddings for a counting-specitn counterfactual text that contains the wrong count j.
In the original paper (Paiss et al. 2023), where the authors continue training the CLIP model on a large amount of counting data of 158K images, they also include CLIP's regular pre-training contrastive loss in
5

---

Published in Transactions on Machine Learning Research (01/2025)
counterfactual captions
Projection
pebi inaze
Enc lariar
CLIP Text
CLIP
Image Encoder
number
Encoder
mumbeeteea the
Captoon Captonegbeddigg
ground-truth
ground souths rebedddd
captions
image
min (L count)
(a) Our method fine-tumes only nine counting vectors.
Toxt Unear Projecion
Toat Linear Projettion
Imng Liner Projection
Tm Liment Poree
CLIP
CLIP
Text
min min count)
Image Encoder
Text Encoder
min(Lcowne)
Image
Encoder
(b) Fine-tuning the last. text. linear
(c) Fine-tuning the whole model.
text
projection layer.
Figure 3: Illustration of learning-based approach. As in Figure 3a follow Paiss al. 2023 to generate counterfactual captions by swapping only the number word in each ground-truth caplion, and fine-tune only nine counting vectors A; as defined in Section 3.3 to minimize counting loss Lcount that forces image embeddings to be far from counterfactual caption embedding yet close to ground-truth caption embeddings. We compare our special fine-tuiing methods to two other fine-tuning methods shown in Figure 3b and Figure 3e which involve training a larger number of model parameters.
the objective function L, such that L Mecount + LCLIP This is an explicit design to prevent. CLIP from forgetting its other non-counting related pre- crettrained knowledge. We will also investigate the necessity of LCLIp in our setting where only small amount of training data is available, in terms of its effectiveness in improving counting accuracy and preserving the CLIP model's pre-trained knowledge.
Recent work (Mestha et al. 2024) modifies Lcouut to be an N-way contrastive loss (Sohn 2016 and shows
that it improves model performance after training more than using Leount defined Equation 3 We will detail the mathematical formula of the contrastive counting loss in ablation study Section and compare results after training models with the contrastive counting loss.
3.5 Zero-shot methods to transfer counting knowledge from prior knowledge
Alongside fine-tuning, explore zero-shot methods to improve the CLIP model with counting capabilities without direct training counting tasks. Zero-shot methods are valuable in addressing the scarcity of labeled image-text pairs for counting tasks, enhancing the model's practical applicability in real-world scenarios. We propose several techniques to extract counting knowledge representation vector u, These methods can also be used as initialization when training counting vectors, which can effectively speed up the optimization process discussed in the earlier section.
Use text embedding of number words. We encode each number in its English word with CLIP text encoder and denote it Amu For example, the text embedding of word "two" is denoted as A mum Therefore, following Equation 2 the counting-augmentee object representation is calculated as 0. obj + -umm where A pum is A that's orthogonalized w.r.t bbb

---

Published in Transactions on Machine Learning Research (01/2025)
Extract counting knowledge from an easy-to-count object. This ap- proach is based on the observation that CLIP is more proficient at counting certain types of objects, such as dogs and cats, as elaborated in Section 4.4 Our key idea is that CLIP effectively counts certain object types, it already possesses some counting knowledge, at least for those objects. When we have the prior knowledge of which object CLIP can count effectively, we take it counting reference and extract A, from this object.
Arel
ree
vob
Specifically, as shown in Figure we define the counting representation extracted from any reference object as Arof, such that
4re
Ayef
A
(u. Jofe (uf=f
(uf v vo yiel
ref
(4)
y
The intuition behind this definition is that the counting information is encap- sulated in the direction moving from the non-quantitative representation (vtef) to the quantitative representation (ufe). Similarly, we obtain A by making ofo - ro orthogonal to uo to climinate information associated with the non- quantitative representation. We further ort orthogonalize it to the con-quantiaative representation of the target object before applying it. We denote the counting reprosentation after two steps of orthogonalization as Aref, such that
Figure 4: Extracting count- ing vector from a reference object.
A ref Arr zbj
(ob) yobj bbbj
(5)
Therefore, following Equation the counting-augmentn object representation is calculated vabi u. A
Extract knowledge from multiple objects. Instead of relying on specific objects known to be casier for CLIP to count, this method aggregates counting vectors from diverse set of objects, which aims to create robust counting mechanism. We prompt ChatGPT (OpenAl 2024), specifically to generate a list of 100 common daily objects and animals in their plural form. For cach object in the list, we calculate the counting-specifie vector Ar same as described in the above paragraph. We then average these 1 vectors across all objects in the list to create a generalized counting vector A Amulti = I00 E/ Aret This averaged vector Amulti, is used as the counting representation to calculate apbi = ob) + Amulte, hypothesizing that it encapsulates universal counting pattern applicable across different object types.
101-4-turbo-22 2-22-02224400 04-00----9
3.6 Development of new object counting benchmarks
To rigorously evaluate the counting capabilities of the CLIP model and the effectiveness of our methods, we have developed two datasets in addition using existing benchmark CountBench (Paiss et al. 2023
CountBench is an object counting dataset, collected from the LAION-400 dataset (Schuhmann et al. 2021
It comprises 540 images in total, with each numerical count represented by 60 respective images of diflerent. types of objects. Each image is accompanied by a caption describing the count of specific object.
A new diverse-source counting dataset. The first new benchmark DiverseCount consists of images automatically sourced from multiple sources, including the COCO Dataset (Lin et al., 2014h, Conceptual 12M Changpinyo et al. 2021,, YFCC100M (Thomee al. 2016), and SBU Captions Dataset (Ordonez) et al. 2011p. In addition, due to the fact that images with large counts (i.e., nine and ten) are scarce, to compose 150 images for each count, also manually collect some images from the Internet. Each text-image pair is manually reviewed, and captions are revised to climinate grammatical errors and irrelevant information. Images are also manually checked to avoid duplication ion. This process yielded a more comprehensive set of images with clear, concise captions, containing 1350 images in total.
An object-specific counting dataset. The second new benchmark ObjectCount focuses on object-specific counting to delvv deeper into how object types influence counting performance.
7

---

Published in Transactions on Machine Learning Research (01/2025)
It includes 360 images in total, with 10 images for each of nine different objects e ("dog", "cats", "lion", "chair", "goat", "cow", "cherry", "rose", "boat"), at each count level from two to five, all manually collected from the Internet. Counts are capped at five due to the scarcity of images with higher object counts. This dataset used to evaluate how counting ability varies across different object types and identify objects that are easier to count, which can serve as references zero-shot methods.
4 Experiments and Results
4.1 Experimental setup
Models. We evaluate our method three CLIP variants: clip-vit-bss--patchh3 wawiiiitargeppattoooe
patch16, and patch sizes, allowing each model to capture finer image details. Furthermore, clip-vit- -iip-vit-large-r pargh-- has a larger model size compared to the first two models. We test our zero-shot methods on the Stable Diffusion model, s -ifole-dff vv14, sourced from HuggingFace.
These models have progressively smaller
Learning-based methods implementation details. We compared novel approach, which involves optimizing counting vectors, with two conventional fine-tuning methods: 1) full fine-tuning of the pre-trained CLIP text encoder and 2) fine-tuning only pre-trained CLIP's last linear text projection layer, 3) prefix-tuning with prefix length 5 (Li & Liang 2021). Additionally, we test whether including the standard CLIP contrastive loss (LCLi)) enhances performance on small-scale datasets. For training the counting vectors, we set a higher learning rate of 10-3 We use lower learning rates of 10-1 for fine-tuning the text projection layer and 10-5 for the entire text model to minimize overfitting.
We divide our new dataset, DiverseCount, into training, validation, and Lest sets in 6:2:2 ratio. We three experiments with different random seeds and report the average test set scores to ensure robustness. We track performance across epochs by saving checkpoints and selecting the one with the lowest validation loss for final evaluations. We first evaluate each model on the DiverseCount lest set, followed by CountBench to measure generalization across data distributions.
To evaluate the impact of fino-tuning on unrelated tasks, we test the models onCIFAR10 Krizhevsky al, 2009 CIFAR100 (Krizhevsky al. 2009). Caltech101 (Fei-Fei et al.) 2004), EuroSAT (Helber et al. 2018
20191, and Food101 (Bossard et al. 2014). This assessment helps evaluate the role of Leur in preserving
CLIP's pre-trained capabilities.
Zero-shot methods implementation details. For zero-shot methods, we implemented different choices of counting vectors A; introduced in Section 3.5 including: 1) text embedding of number words Auum; 2) counting vectors Apulti extracted from common objects; and 3) counting vectors Aref extracted from an easy-to-count object. In the third set of experiments, we select 'cats' and 'dogs' as reference objects based on their strong performance in Table 4
We evaluate our method on our custom dataset ObjectCount, as introduced Section 3.6 as well as on the image counting benchmark, CountBench (Paiss al.) 2023). The counting task on ObjectCount has four classes (counts from two to five), while CountBench includes nine classes (counts from two to ten).
4.2 Effectiveness of learned counting knowledge vectors
We assess the offectivenoss of our methods in improving CLIP's counting accuracy using the DiverseCount test set and CountBench. The results in Table 2a indicate that training counting vectors and fine-tuning the last linear layer are more effective than fine-tuning the entire text model, even with far fewer paraneters. This supports our hypothesis that counting knowledge may be encapsulated in a specific dircction applicable across different objects. Training counting vectors, involving only 9d parameters, is computationally efficient. and maintains strong performance.
8

---

Published in Transactions on Machine Learning Research (01/2025)
Table 2: Accuracy (%) of 9-class classification on DiverseCount test splits and on CountBench, comparing different fine-tuning methods. All models are trained on the DiverseCount training split. We report the average aceuracy from three runs with different train/val/tess splits. Columns under L Loount represent methods that optimize only the counting loss, while columns under L = Leount LCLIP represent methods that optimize both the counting loss and CLIP's standard contrastive loss. "Cnt Vecs," "Proj," " "Text Model," and "Prefix Tuning" each indicate which parameter is exclusively trained: "counting vectors," "CLIP's text projection layer," " "CLIP's text model," " and "prefix embedding vectors," respectively. We boldface the highest score each row.
(a) Accuracy of 9-class classification on DiverseCount test splits, comparing different fine-tuning methods.
Learning- based methods
Model
Original
L
Lcoust
Lecuna + LELIP
Vocs (ours)
Proj
Text Model
Prefix Tuning
CntVecs (ours)
Proj
Text Model
Prefix Tuming
CLIP-base- base-32 32
28.17
37.78
38.16
33,92
14.55
38.53
37,66
34.17
15.67
16
28.66
38.28
38.53
35.41
13.81
38.66
38.03
36.01
17.91
CLIP-large-14
33.62
41.34
39.6
33.49
13.05
40.84
39.35
33.99
17.54
(b) Accuracy of 9-class classification CountBench, comparing different fine-tuning methods.
Learning- methods
Model
Original
L
Leount
Leousa
LeLi
Vecs (ours)
Proj
Text Model
Prelix Tuning
CntVecs (ours)
Proj
Text Model
Prefix Tunng
CLIP-base-32
30.69
33.12
27.47
32.83
10.21
33.4
27.75
33.62
15.32
CLIP- LLIP-base-16 28.73
30.66
27.39
28.51
15.32
30.89
27.25
29.47
18.51
(LIP-Large-1 31.97
39.2
31.4
32.33
10.45
39.41
31.83
32.47
13.65
When evaluated on CountBench (as shown Table 2b), all methods exhibit reduced effectiveness, likely due to a distribution gap between DiverseCount and CountBench. Training counting vectors continues to improve CLIP's counting accuracy on CountBench, demonstrating better generalization than other methods.
We also observe that prefix tuning, which fine-tunes similar number of parameters, does not improve performance and negatively impacts counting accuracy.
Moreover, the inclusion of LCLIP in the training objective does not significantly influence the outcomes across all training setups, as seen the three rightmost columns in Table 2
4.3 Impact on CLIP's performance in non-counting Tasks
Fine-tuming via directly updating CLIP's pre-trained weight has a significant impact on CLIP's performance in non-counting benchmarks, as detailed in Table 3 Both fine-tuming the entire text model and the last linear layer results in noticeable performance drops across all benchmarks, with finetuning the projection layer having a more pronounced effect.
Incorporating LOLP provides only marginal effectiveness in preventing the loss of pre-trained knowledge. This contrasts with the findings in the CountBench paper, where incorporating LCLiP was beneficial when fine-tuning CLIP with large dataset (i.c., 158K images), providing ample sources for the model to learn new things. In our experiments, where only a smaller dataset was available, LeLi did not demonstrate the same effectiveness, suggesting that its utility may be limited under conditions of restricted data availability.
However, training new counting vectors does not alter CLIP's model parameters. Thus, we can apply these vectors only in tasks specific to counting, while relying solely on the pre-trained CLIP model for all other non-counting tasks.
9

---

Published in Transactions on Machine Learning Research (01/2025)
Table 3: Performance on common benchmarks. We compare the performance of fine-tuned models against pre-trained models on common non-counting benchmarks. Fine-tuning either linear layer or the whole model will lead siginificant performance drop.
Learning-basse methods
Model
Benchmark
Original
L Lenunt
L Leount LCLIP
Proj
Text Model
Proj
Text Model
CIFAR10
88.95
83.43
89.01
82.72
88.95
CIFAR100
48.78
35.68
47.75
37.9
47 47.62
CLIP-base-33
Caltech101
80.18
71.50
80.33
71.15
80.3
EuroSAT
45.11
36.49
43.78
37.94
44.28
Food101
80.2
78.64
78.61
78.59
78.69
CIFAR10
88.35
79.92
88.31
81.59
88.38
CIFAR100
58.63
51.98
58.95
53.38
59.36
CLIP-base-16
Caltech101
76.78
70.70
75.13
71.18
75.67
Au
49.83
39.20
46.84
43.77
47.47
Food 85.57
82.97
85.26
85.69
83.49
CIFARI0
95.01
93.65
94.91
93.71
95.04
C10AR100
64.66
62.49
64.9
62.47
64.99
CLIP-lawd 1LPP-large-14 14
Caltech 101
81.02
75.50
80.20
75.71
80.56
EuroSAT
55.37
51.59
54.00
51.46
53.9
Food101
89.79
88.46
89.35
88.61
89.43
Table 4: The counting accuracy of CLIP varies across diverse objects. Pre-trained CLIP models
counting accuracy varies by object type. Still, all models consistently count dogs and cats more accurate than other objects.
average
dogs
cats
lions
chairs
goats
cows
cherries
roses
boats
CLIP-base-33 32
47.93
58.86
66.14
47.73
35.23
42.73
46.36
45.45
32.27
47.27
CLIP-base-. 16
50.33
74.77
74.77
54.32
47.05
32.73
55.00
35.00
34.09
45.23
CLIP- 14
60.86
75.23
79.09
65.45
52.95
44.77
65.00
53.86
56.82
54.55
4.4
CLIP's counting ability on different objects
Each column in Table represents the counting accuracy for a specific object in ObjectCount, with object names as column hcaders. The average accuracy across all objects is also displayed undor the "average" columm. We observe positive correlation between model size and average counting accuracy, with accuracies ranging from 47.93% 60.86%. However, there is significant variation in the models' counting abilities for different object types, suggesting that CLIP's counting capability is dependent the object.
Notably, all models consistently perform best when counting "dogs" and "cats", while their performance with other objects lacks consistency. We hypothesize that the superior performance on "dogs" and "cats" may result from their higher frequency in the pre-training dataset. In fact, when collecting our dataset. ObjectCount from the Internet, we do observe that images of dogs and cats are more accessible in larger volumes than images other objects.
4.5 Effectiveness of our zero-shot method
We evaluate our zero-shot methods on our custom dataset ObjectCount and CountBench, and report the results Table 5 Across all three CLIP models, there are noticeable improvements in counting accuracy when using zero-shot methods compared the default (baseline) settings. However, the effectiveness of each strategy varies by task and model size. For example, most methods more effective the -LIP-large- 14
10

---

Published in Transactions on Machine Learning Research (01/2025)
Table 5: CLIP's counting accuracy for image classification task on our custom dataset ObjectCount and CountBench (%), comparing results of zero-shot methods. We bold the higest score each row.
Model
DatasetMethod
original
obi + Amulti
1/3 + 2
obj
V/
ob)
+
A
dogs
obj
+ A cats
ObjectCount (4-class)
47.93
49.65
49.65
51.84
49.8
CLIP-base-37
CountBench (9-class)
30.69
31.91
31.7
28.3
29.36
ObjectCount (4-class)
50.33
53.16
52.55
55.45
54.77
base-base1l 16
CountBench (9-class)
28.73
27.02
31.06
29.15
28.51
ObjectCount (4-class)
60.86
65.58
64.77
64.12
64.5
CLIP-large-11 14
CountBench (9-class)
31.97
39.45
36.25
39.23
38.17
model, which has larger model size and higher resolution, with improvement close to or higher than 5%, The improvement is more significant on ObjectCount than on CountBench, likely because CountBench includes nine classes, making it more challenging.
4.6 Effectiveness of our method in improving text-to-image models' counting fidelity
Since our method directly enhances the counting capability of the CLIP model in zero-shot manner, we anticipate that using our approach will also aid models that utilize CLIP embeddings for image generation, such the Stable Diffusion model (Rombach et al., 2021], ,, in producing images with the correct number of objects. Therefore, we experiment with applying our method to Stable Diffusion. We select 10 image descriptions from CountBench (Paiss et al. 2023), as shown in Table 6 Each description features a commonly counted object and can serve as an appropriate image generation prompt for Stable Diffusion. We modify the counting number in each description to range from "two" to "ten," " forming 90 prompts total, each describing a unique combination of counting number and object. These prompts are provided to the Stable Diffusion model and we generate 20 images for each prompt. based on both our method and the original CLIP model, resulting in a total of 3,600 images. We use the YOLOv9 model (Wang & Liao 2024 for object detection to compare the object counts in images generated by our method and the original CLIP model. This allows us to determine how many instances of the specified object are present, using the detection results as ground truth.
Table 6: List of prompts used for image generation, with the counting numbers and objects highlighted in blue and red, respectively. For image generation, the counting numbers in each prompt are replaced with "two" to "ten".
No.
Prompt
An old building with ruined walls and four antique pink armchairs Vintage silver plate tablespoons, serving spoon set of two 1847 Rogers Ambassador pattern Forks with vegetables- four forks with different types of...
2
3
4
Set of four stemless red wine glasses
5
Row of five British Shorthair cats sitting on wooden tray, isolated on white background
6
Eight bottles of aguardiente on counter
7
Sct of four multicolored 'Penzance' small bowls
8
Photo of ten girafle portraits, isolated on a white background
9
Meet the MINI family, five cars in different styles, lined up in row
10 Photo of seven red plastic apples on white background
The experimental results are presented in Figure 5 which displays the counting accuracies images generated using the original CLIP model and our method, respectively, in the form of confusion matrices. We aggregate the image generation results for the ten types of objects. In the matrices, rows represent the specified counts

---

Published in Transactions on Machine Learning Research (01/2025)
in prompts, and columns represent the detected counts in generated images. The red boxes highlight the cells where the detected number of objects matches the counting number specified in the prompt, indicating the number of images correctly generated for each counting number. For prompts with counts from 2 to 5, our method sigmificantly improves Stable Diffusion's accuracy generating images with the correct object counts. However, for counting numbers from to 10, the ellectiveness of our method diminnihhss likely because the CLIP model's pretrained datasct lacks text labels corresponding to these higher counting numbers. Examples of images generated by Stable Diffusion are shown in Table and Appendix A Note that our method can be used in conjunction with existing methods for improving the fidelity of toct-to-image models, e.g., reinforcement learning- saaninggbasedd algorithms Faan et al. 2023
Fan & Lee 2023 Black et al. 2023
Stabie Difustoon with pasoline CUP
Stable Deftuston with Ours
26
16
25
12
20
Cruste
16
14 14
Coante
26
22
10
12
20
12 15
Ocroouted Frwwe
Ocrosuted
(a) Stable Diffusion with baseline CL.IP
(b) Stable Diffusion with our method
Figure 5: Confusion matrices comparing object counting accuracy in images generated by Stable Diffusion using (a) baseline CLIP and (b) our method. In each matrix, the rows represent the counts specified in the prompts for image generation, and the columns represent the counts detected in the generated images using the corresponding prompts. The red boxes highlight the cells where the detected number of objects matches the count specified in the prompt, indicating the ratio of generated images with correct counts for each row. We observe that our method is more effective in helping the Stable Diffusion model generate images with correct counts between 2 and 5, as shown by higher numbers in the red boxes, but less offective for counts from to 10.
5
Ablation Studies
5.1 Effectiveness of different contrastive loss designs
In our work, we adapt the count loss proposed by Paiss et al. it against a multi-class N-way loss, denoted by Leount, used in Mestha et al. 2204)) which is defined as:
(023), as shown in Equation We compare
exp(1* :
Lcount
log exp(u* Tit)) + Σ) exp(ut T/²))
(6)
Table shows that on the DiverseCount test split, fine-tuning with the contrastive counting loss Lcount performs slightly worse than Lcount.. However, applying Lggnnt further improves the accuracy of most fino-tuned models on the CountBench dataset, indicating that Leount might have a more robust and effective loss design when the models need to generalize to setting with larger distribution shift.
12

---

Published in Transactions on Machine Learning Research (01/2025)
Table 7: Selected results from Stable Diffusion (Rombach et al., 2021). - Images in the "Original" column are generated based on the input prompt in the same row, using different seeds, Images in the "Embedding edited" column are generated after applying our zero-shot method (using the same seeds), with the selection of "dog" the reference. After applying method, observe that Stable Diffusion is likely to generate images with the correct number of objects.
Input Prompt
Original
Embedding edited
"three lions"
"An old building with ruined walls and four antique pink armchairs"
"vintage silver plate tablespoons, serving spoon set of two"
"three dolphins jumping out of water"
"A picture of three cherries"
Table 8: Accuracy (%) of 9-class classification on DiverseCount test splits and CountBench of different fine-tuning methods, with contrastive count loss Leount- For each fine-tuning method, we bold scores if its higher when applying contrastive loss Leount vs. Leount We display changes compared to applying Lcount inside the parenthesizes right after each score.
DiverseCount
CountBench
Model
Cnt Vees (ours)
Proj
Text Model
Cnt (ours)
Proj
Text Model
CLIP-base-32
37.28 (-0.50)
37.78 (-0.38)
33.78 (-0.14)
33.55 (+0.433
28.25 (+0.788
34.62 (+1.79)
CLIP-base-B
38.03 (-0.25)
38.03 (-0.5)
35.53 (+0.12)
32.66 (+2.00)
28.66 (+1.27
29.4 (+1.89)
CLIP-large
39.48 (-1.86)
39.59 (-0.01)
34.37 (+0.88)
37.91 (-1.29) 32.69
(+1.299)
33.48 (+1.15)
5.2
Ablation studies of each component in the counting representation
5.2.1
Learning- based settings
To evaluate the impact of different design choices in learning-basee based settings, we analyze modifications A.-uohi representing and applying Â, In our main method, A, == A, goatl yooj which involves orthogonalization with respect to obb We study the effect of orthogonalization in the NoObjOrth experiment. We also study the impact the projection direction used during orthogonalization, by comparing the main results to the design such that. A; is forced be orthogonal to ei instead of ,o,, We name this group of experiments, where A, = A.- "A
" as in ChangeProjDir. In addition, when applying Á,, it is added to bb as shown in Equation 2 We study the effect of adding A onto ob such that ophi = bb + A, and named this group of experiments as ChangeAddDir.
Ablation study results are presented Table g under columns corresponding each ablation experiment. We find that in learning-baat based settings, there no significant difference between each modification and the main approach. This suggests that when the same modifications applied during training and inference,
13

---

Published in Transactions on Machine Learning Research (01/2025)
the orthogonalization method and direction might not significantly affect the learning of counting vectors. However, these factors have much more significant effects in zero-shot settings, we will demonstrate in the next section.
Table 9: Accuracy (%) of 9-class classification on DiverseCount test splits and CountBench of fine-tuning counting vectors, comparing different modifications with our method in the main paper. For each modification, we bold scores if its higher than score of the main method.
DiverseCount
CountBench
Model
Main method
NoabjOrth
ChangeProjDir
ChangeAddDir
Main method
NoübjOrth
ChangeProjDir
ChangeAddDir
SSIP-base-32
37.78
37.16
37.66
36.28
33.12
33.62
33.26
32.69
LLIP-base-11
38.28
38.65
38.78
38.03
30.06
31.7
31.48
32.22
CLIP- Largel 14
41.34
42.59
41.09
41.84
39.2
39.41
39.34
40.27
5.2.2 Zero-shot settings
In addition ablation study groups introduced in the previous section, including NoObjOrth, rojDr,,rojDir, ChangeAddDir, for zero-shot settings, add another ablation study group named NoRefOrth. This ablation group studies the effect of orthogonalizatton w.r.t. reference object's lext embeddings, by excluding projection on ref as defined in Equation 5 so that the orthogonalization only applies w.r.t. to target object's 's text - coby yob)
embeddings oo and that rel Apel
Figure and Table 10 demonstrate that the main method outperforms alternative modifications, particularly the CountBench dataset.
5.3 Applications other models
To evaluate the generalizability of our methods larger, more recent models, we apply our counting-specifi training techniques the SigLIP (Zhai et al.) 2023) and BLIP (Li al. 2022) models. SigLIP is an alternative to CLIP, pre-trained using simple pairwise sigmoid loss instead of CLIP's original loss function. The pairwise sigmoid loss improves SigLIP'ss wero-shot classification accuracy on ImageNet. BLIP, the other hand, leverages bootstrapped self-supervision techniques that iteratively refine its understanding of images and text without extensive labeled data. This approach enables BLIP to excel in scenarios where labeled data may be sparse or unavailable, making it well-suited for open-world applications and robust. zero-shot learning.
Our zero-shot method evaluated both SigLIP and BLIP. As shown in Table 11 our approach effectively enhances counting performance for these models under certain conditions. However, the improvements are not uniformly observed across all scenarios, indicating that our method's benefits may be context-dependent Future research should focus on refining and extending our approach to ensure broader applicability across diverse models and datasets.
5.4 Implications for broader applicability
Although our findings with SigLIP are preliminary, they suggest that the observed gains are not restricted to smaller CLIP models but may exlend to larger, stale-of-the-art architectures. Fine-tuning the text embedding space to incorporate numerical representations may be an efficient strategy for enhancing counting accuracy across diverse vision-language models. Future work could explore this approach in additional high-capacity models and evaluate on more diverse counting benchmarks, providing deeper insights into its adaptability and robustness.
14

---

Published in Transactions on Machine Learning Research (01/2025)
bip-wii-base-aalchhh
hipritt-arrge-palchhi
Accorsiy 1%
//
1/4
span
(a) Ablation studies of zero-shot methods on ObjectCount dataset.
clp-vit-bas--ssteh3
clpp-itt-bas-pptthh1
clp-sit-large-pathhh
70
Original
60
ChangehiddBlr
ChongsPropp
NolurOrth
NocltyDrch
50
Accur0 (80
40
2000
200
(b) Ablation studies of zero-shot methods CountBench dataset.
Figure 6: Ablation studies of zero-shot methods on ObjectCount dataset and CountBench dataset.
6 Conclusion and Discussion
6.1 Summary of contributions
In this study, explored the counting capabilities of CLIP models and introduced computationally elficient method for training counting vectors, along with several zero-shot text embedding editing techniques to enhance CLIP's counting accuracy. Our baaring-based approach demonstrates that targeted modifications to text embeddings can significantly improve object counting tasks without the need for extensive model retraining. This method not only proves to be effective but also preserves the broader capabilities of CL.IP, unlike other methods that might compromise the general performance of CLIP models.
Our zero-shot techniques particularly valuable in contexts where data is scarce full model retraining is impractical due to computational constraints. These methods have also shown promise when applied to text-to-image models like Stable Diffusion, indicating their potential applicability beyond the initial use case.
6.2 Performance variation across object categories
Our analysis reveals that CLIP's counting accuracy varies significantly across different object categories, with notable advantage for common objects such as cats and dogs. This outcome can be attributed to the pretraining data distribution of CLIP, where categories that are frequently represented in this dataset benefit from more comprehensive visual and linguistic representations, enabling more accurate counting results. Tn
15

---

Published in Transactions on Machine Learning Research (01/2025)
Table 10: CLIP's counting accuracy (%) on our custom ObjectCount and CountBench datasets for image classification tasks, comparing zero-shot methods. For each zero-shot method, we bold scores that are higher under the current modification. Changes relative the main method are shown in parentheses following each score.
(a) CLIP's counting accuracy for image classification task on custom dataset ObjectCount
Model
obj
Amulii
1/ + uqum
nbj
1i A
dogs
obl
+ Açate
CLIP-base-2 32
50.81
(+1.16)
44.87 (-4.78)
50.76 (-1.08)
48.79 (-1.01)
Change Add Dir
CLIP-base- 16
55.13
(+1.97)
48.79 (-3.76)
52.75 (-2.7)
51.39 (-3.38)
CLIP-1arge-14
63.79 (-1.79)
61.94 (-2.83)
63. (-0.99)
61.47 (-3.03)
CLIP-base
50.18
(+00.53)
43.46 (-6.19)
52.2 (+-0.36)
50.08 (+-0.28)
Change Proj Dir
CLIP-base- 16
53.76 (+0.6)
47.68 (-4.87)
55.25 (-0.2)
55.08 (+ (+-0.31)
CLIP-Large-14
65.68 (+0.1)
61.39 (-3.38)
65.05 (+0))33)
65.71 (+1122
CLIe-base- -2
49.34 (-0.31)
51.59 (-0.25)
50.15
(+ (+.3.3)
No Ref Orth
CLIP-base 16
53.81 (+0.65)
54.29 (-1.16)
53.21 (-1.56)
65.05 (-0.53)
63.99 (-0.13)
63.13 (-1.37)
CLIP-base 32
49.72 (+0.07)
49.47 (-0.18)
51.56 (-0.28)
49.93 (+0.133
No Target Orth
CLIP-base- 16
53.61 (+0.45)
53.05 (+0.5)
55.25 (-0.2)
54.42 (-0.35)
CLIP-large-14 65.71 (+0.13)
65.56 (+0.799
64.24 (+0.12)
64.5 (+0.0)
(b) CLIP's counting accuracy for image classification task on our custom dataset CountBench
Model
1 + A muulti
obj
+ m
2 + dogs
b A cats
CLIP-base-32
28.76
(-3.15)
23.82 (-7.88)
26.61 (-1.69)
27.04 (-3.32)
Change Add Dir
CLIP-base-10
28.29
(+1.27)
19.82 (-11.24)
26.73 (-2.42)
25.61 (-2.9)
CLIP-large-14 14
38.41 (-1.04)
20.6 (-15.65)
34.12 (-5.11)
36.48 (-1.69)
CLIP-base-33
30.04 (-1.87)
22.1 (-9.6)
26.61 (-1.69)
27.47 (-1.89)
Change Proj Dir
CLIP-base-16
26.28 (-0.74)
17.37 (-13.69)
27.84 (-1.31)
28.29 (-0.22)
CLIP- 14
36.48 (-2.97)
22.96 (-13.29)
36.48 (-2.75)
34.55 (-3.62)
CLIP-base-32
30.26 (-1.65)
27.9 (-0.4)
29.4 (+0.04)
No Ref Orth
CLIP-base-12
26.5 (-0.52)
28.73 (-0.42)
27.62 (-0.89)
CLIP-large-14
36.05 (-3.4)
36.91 (-2.32)
35.62 (-2.55)
CLIP-base-33
30.69 (-1.22)
28.54 (-3.16)
28.33 (+ 0.00)
29.18 (-0.18)
No Target Orth
CLIP-base- 16
26.73 (-0.29)
27.17 (-3.89)
28.95 (-0.2)
28.73 (+0.22)
CLIP- 14
37.12 (-2.33)
30.69 (-5.56)
36.91 (-2.32)
35.62 (-2.55)
contrast, less common niche object categories that appear less frequently in the pretraining data exhibit reduced accuracy in counting tasks due to less robust feature learning.
This performance variability highlights a key challenge in open-world scenarios, where models must generalize effectively across both familiar and unfamiliar categories. While CLIP's broad pretraining endows with inherently limited by the distributional biases present in the
impressive zero-shot learning capabilitics, dataset. As a result, when faced with objects outside the well-represented domain of its training corpus, CLIP may show diminished counting performance. This underscores the need for targeted fine-tuning
16

---

Published in Transactions on Machine Learning Research (01/2025)
Table 11: Counting accuracy (%) of SigLIP and BLIP models on our custom ObjectCount and CountBench datasets for image classification tasks, comparing zero-shot methods, The highest score in each row is shown in bold.
Model
Methodd
original
oihh Amuiti
ohj
Adaag
1/ + Açats
ObjectCount (4-class)
22.12
25.82
26.40
26.84
25.39
siglip- siglip-base-patchlee
CountBench (9-class)
14.08
9.01
9.45
9.01
8.57
ObjectCount (4-class)
23.01
23.12
26.69
21.84
22.73
Sggiiplarre-patchhee 224
CountBench (9-class)
12.31
12.53
14.51
12.31
10.33
ObjectCount (4-class)
24.32
24.34
27.6
29.55
26.02
blip hap-imaag--catiininng base
CountBench (9-class)
10.94
14.00
12.74
11.82
9.63
ObjectCount (4-class)
28.38
23.84
22.22
22.83
25.75
blip- halp-iaage-cattioning large
CountBench (9-class)
12.91
17.72
9.41
10.94
11.82
adaptive strategies to bolster the model's counting capabilities for underrepresented or novel object types, enhancing its generalization in truly open-world contexts.
6.3 Image encoder assumptions and considerations
In this work, our method exclusively focused on the text encoder while assuming that the image encoder's cxisting capabilitics were sufficient for counting. This choice was based on the hypothesis that CLIP's limitations in counting may be attributed to the representation numerical concepts within its text embedding space. By enhancing this aspect, we demonstrated that counting performance could be improved without modifying the image encoder.
However, acknowledge that the image encoder's limilations may also play role in cases where counting performance is suboptimal. Factors such as feature extraction quality, spatial representation, and object separation within images can impact counting effectiveness. Investigating the image encoder's capabilities in relation to counting would be a valuable extension for future work, particularly to understand how joint adaptations of the image and text encoders might. further improve counting tasks.
6.4 Count range and generalization limitations
Our study focused on count range of to 10, selected due to practical considerations such as dataset availability and the need for reliable image annotations. While methods shown to be effective within this range, we recognize that they may not generalize scamlessly to higher object counts without further adaptation. This is limitation of our current approach, and future research could investigate strategies for expanding the count range, such as using hierarchical representations or synthetic data augmentation to improve generalization to larger counts.
6.5 Limitations and future work
Despite the promising results, our approach has inherent limitations. One key issue is the lack a comprehensive theoretical understanding of why training counting-specifie vectors outperforms other methods. Further exploration of the underlying mechanisms CLIP's counting capabilitics could provide new insights and inspire more robust enhancement strategies for vision-language models.
Additionally, method is currently limited to scenarios involving single counted quantity and does not support complex visual scenes with multiple object compositions interactions between objects. For example, it cannot. handle queries that require counting multiple object types simultaneously, such as "five dogs and three cats." " Extending our method to handle multi-quantity counting would likely require more complex interactions within the text embedding space, and more advanced visual reasoning abilitics into the model. Techniques such as compositional embedding strategies or multi-object relational modeling may provide pathways to expand the applicability of our approach to more complex counting queries.
17

---

Published in Transactions on Machine Learning Research (01/2025)
Future work could focus integrating spatial and relational reasoning into vision-language models to enhance their ability to interpret contextual and spatial cues within images, improving both counting accuracy and general visual comprehension. Extending our approach to other visual tasks, such as object detection and complex counting scenarios, could broaden its applicability and reveal universal strategies for boosting performance across various vision-language architectures.
Another limitation of this work is that our study focuses on low-count, object-specific scenarios and evaluates performance using the custom DiverseCount and ObjectCount datasets. These benchmarks were chosen to align with our problem formulation, which frames counting classification task. However, recognize the importance of widely used benchmarks such as ShanghaiTech (Zhang et al. 2016) and UCF-QNRF et al. 2018), which provide comprehensive evaluations of counting methods. Although these data sets are typically used in density estimation tasks, they can also be adapted for classification based evaluation by aggregating object counts defined regions or across the entire image.
Our current work does not include experiments on these benchmarks, which limits the scope of our evaluation. Future research should explore the application of our methods to such datasets, enabling broader under- standing of their effectiveness across diverse counting paradigms and further validating their generalizability.
Acknowledgments
This work supported in part by NSF CAREER Award grant from AuriosaAI.
Amazon Research Award, and a
18

---

Published in Transactions on Machine Learning Research (01/2025)
References
Manoj Acharya, Kushal Kafle, and Christopher Kanan. Tallyqa: Answering complex counting questions. In The Thirty-Thun Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honobulu, Hawaii, USA, January 27 February 1, 2019, pp. 8076 8084. AAAI Press, 2019, doi: 10.1609/ 10.1609/aaai.va va3.a...33001.33111333 URL https://doi.orgg 1609// aaai. v33101. 33018076
Jean-Baptiste Alayrac Jeff Donahue, Pauline Lue, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Scrkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman,, and Karén Simonyan. Flamingo: a visual language model for few-shot learning. In NeurIPS, 2022. URL httpp:/ http:/pppprrs..ips... mips. 76011776c7b8817cccb111766666 Abstrract-Conferenncee
Carl Allen and Timothy Hospedales. Analogies explained: Towards understanding word embeddings. In International Conference on Machine Learning, pp. 223 231. PMLR, 2019.
Lital Binyamin, Yoad Tewel, Hilit Segev, Eran Hirsch, Royi Rassin, and Gal Chechik. Make it count:
Texl-lo-image generation with an accurate number of objects. arXiv preprinl 2xXiv:2406.10210 2024.
Kevin Black, Michael Janner, Yilun Du, Πya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement lcarning. CoRR, abs /t///2205..333 2023. doi: 00.48550/arxiv.2355.1
URL
https://doi.org//10 40w11//aa.50//.i........ 2305. 13301.
Lukas Bossard, Matthicu Guillaumin, and Lue Van Gool. Food-101 mining discriminative components with random forests. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Compuler Vision
ECCV 2014, pp. 461, Cham, 2014. Springer International Publishing. ISBN 11--3-319-1009990
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail tail visual concepts. In CVPR, 2021.
Xi Chen, Xiao Wang, Soravit Changpinyo, A. J. Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish V. Thapliyal, James Bradbury, and Weicheng Kuo. Pali: A jointly-scaled multilingual language-image model. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.nee 2023. URL http::/pppeeeeeeeeeeeeeedddddddd
Aleksandr Drozd, Anna Gladkova, and Satoshi Matsuoka. Word embeddings, analogies, and machine learning: Beyond king- man + woman queen. In Yuji Matsumoto and Rashmi Prasad (eds.), Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pp. 3519-3530, Osaka, Japan, December 2016. The COLING 2016 Organizing Committec. URL https: : 1332
Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. Towards understanding lincar word analogies. arXiv preprinl Xiv::11810.0 04882, 2018.
Ying Fan and Kangwook Lee. Optimizing DDPM sampling with shorteut fine-tuning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (cds.), International Conference on Machine Learning, ICML, 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 9623 9639, PMLR, 2023. URL https: 7/proceedings.oo mlr. mrt.:prres//2022faa223.b html
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadch, Kangwook Leo, and Kimin Loc. DPOK: reinforcement learning for fine-tuning text-to- image diffusion models. CoRR, abs//305. 1608. 2023. doi: 10.48550/arXitt 230.,..2305.6188 URL https: http://////1...55////r//.o. http:aawiiv.v......1 16381
19

---

Published in Transactions on Machine Learning Research (01/2025)
Li Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 Conference on Computer Vision and Pattern Recognition Workshop, pp. 178 178, 2004. doi:
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Introducing eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. In IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium, pp. 204-207. IEEE, 2018.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observalions and Remole Sensing, 2019.
Haroon Idrees, Muhmmad Tayyab, Kishan Athrey, Dong Zhang, Somaya Al-Maadeed, Nasir Rajpoot, and Mubarak Shah. Composition loss for counting, density map estimation and localization in dense crowds. In Proceedings of the European conference on compuler vision (ECCV), pp. 532 546, 2018.
Ruixiang Jiang, Lingbo Liu, and Changwen Chen. Clip-count: Towards text-guided zero-shot object counting. 2023a.
In Proceedings of the 31st ACM International Conference on Multimedia, pp.
Ruixiang Jiang, Lingbo Liu, and Changwen Chen. Clip-count: Towards text-guided zero-shot object counting. URL http:s///ool.oo91114885555///
CoRR, abs/2305.02277 2300400734, 2023b. doi: 2305.07304
Wonjun Kang, Kevin Galim, and Hyung Koo. Counting guidance for high fidelity text--o--ingeeeee synthesis. CoRR, hbs/2306.17567, 2023. doi: 10.48550/arXiv. 2306........6..7. URL http:://aol.org/1044555500 2306. 17567
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naunann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training large angusio--addvviionn assistant for biomedicine in one day. CoRR, abs/ //////2000000000 2023a. doi: 10.48550/arXiv 20..200000099 URL https://doi.orr org//1
48550//rXivv 2306.00890
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pro-training for unified vasion-language understanding and generation. In International conference on machine learning, pp- 12888 -19800 PMLR, 2022.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00. 10090,000000000 2021.
Yumeng Li, Margret Keuper, Dan Zhang, and Anna Khoreva. Divide bind your attention for improved generative semantic mursing. CoRR, abs/2330.11888 10864, 2023b. doi: (1.48550/arXiv.225 wtt....33...88.8. URL httpps://60i.org/10.. 48550//rKi5 33077.0866
Dingkang Liang, Jiahao Xie, Zhikang Zou, Xiaoqing Ye, Wei Xu, and Xiang Bai. Crowdelip: Unsupervised crowd counting via vision-language model. In IEEE/CVE Conference Compuler Vision and Pallern Recogmition, C'PR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 2893 2903. IEEE, 2023. doi: 000.:/222..2202022222 URL https:://oi....o https:///doo.or/1///// 1t..:0..1119v/// 1189/CVPR52729 20222227992222......
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision FCCV 2014, pp. 740-755, Cham, 2014. Springer International Publishing. ISBN 978-3-319-10602-1
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning. CoRR, abs/ 23044084488 2023. doi: 10.48550/ar www.ii.200..00..... 2304.0888.. URL http::://doi.org///// http://0....r///00.005/////////.
48550/ aar.vv.// 2304. 08485
20

---

Published in Transactions on Machine Learning Research (01/2025)
Nan Liu, Shuang Li, Yilun Du, Josh Tenenbaum, and Antonio Torralba. Learning to compose vi- sual relations. Tn Marc' Aurelio Ranzato, Alina. Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: An- nual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 23166 23178, 2021. URL http://pooeeddnggg.eerriiscccaao G308b2665550777455965555778333 Abstract.html
Harshvardhan Mcstha, Tejas Agrawal, Karan Bania, Yash Bhisikar, et al. Countclip-(re] tcaching clip to count to ten. arrie preprint ur urXiu:2406. 03586, 2024.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. XrXi preprint arXiv:1301.3781, 2013a.
Tomáš Mikolov, Wen-tau Yih, and GeofTrey Zweig. Linguistic regularities in continuous space word repre- sentations. In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pp. 746-751, 2013b.
Duy-Kien Nguyen, Vedamuj Goswami, and Xinlei Chen. Movie: Revisiting modulated convolutions for visual counting and beyond. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL httpps//opeerrvvveweeee//rrrrrrn
8e6BrwU6AjQ
Thao Nguyen, Yuheng Li, Utkarsh Ojha, and Yong Jae Lee. Visual instruction inversion: Image editing via visual prompting. CoRR, 2t///2307.1440 2023. doi: 10.45550/a7X1..2377.1.333 URL https:
7/doi. 48550/arXiv 2307 14331
OpenAL. Chatgpt: Language model. https://ww..openni.com/aaaaaaat 2024. Accessed: 2024-04-09.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using million captioned pho- tographs. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Percira, and K.Q. Weinberger (cds.), Advances in Neu- vul Information Processing Systems, volume 24. Curran Associates, Inc., 2011. URL https: couurpps.cc/paeee htt.://www....taatoooooo/t//taaa Patpheebbbbea-aaer...eeeee pdf
Roni Paiss, Hila Chefer, and Lior Wolf. No token left behind: Explainabiilty--iidee image classification and generation. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision ECCV 2022 17th European Conference, Tel Avio, Israel, October 23-27, 2022, Proceedings, Part XII, volume 13672 of Lecture Notes in Computer Science, pp. 334-350. Springer, 2022. doi:
wtt.:/0....////9...... 20. URL https://doi. org.10.....1/1.... 1007 3-188-3--33------- --1-----3--1-------- 8__0
Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Trani, and Tali Dekel. Teaching clip to count to ten. arXiv preprint arXio:2302. 12066, 2023.
Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu. Zero-shot laage-to-image translation. In Erik Brunvand, Alla Sheffer, and Michael Wimmer (eds.), ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH 2023, Los Angeles, CA, USA, August 6- 10, 2023, pp. ACM, 2023. doi: 10.1145 05155//5588432331155111
URL https::t////tt :/tp::://doi..o 110 1145111881431..23530
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Πlya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Reseorch, pp. 8748 8763. PMLR, 2021. URL html
mlr. http://iinnn........ress///////a
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution
image synthesis with latent diffusion models, 2021.
21

---

Published in Transactions on Machine Learning Research (01/2025)
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs, arXic preprint arXiv:2111.0211 2021.
Amanpreet Singh, Ronghang Hu, Vedamuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kicla. FLAVA: A foundational language and vision alignment model. In IEEE/CV /CVF Conference on Computer Vision and Pallern Recognilion, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 15617-15660 15629. IEEE, 2022. doi: 10.1109/CVPR52668.. 202..00.5.... URL https:://doi.org//1/ 1109/ CVPR52688
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL http:///ss/ooeeeeeoo....o//////s
Paper.
Kartik Sreenivasan, Keon Lee, Neong-Gwan Lee, Anna Lee, Jaewoong Cho, Jy-yong Sohn, Dimitris Papail- iopoulos, and Kangwook Lee. Mini-batch optimization of contrastive loss. In ICLR 2023 Workshop Mathematical and Empirical Understanding of Foundation Models, 2023.
Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfce100m: the new data in multimedia research. Comman. ACM, 59(2):64478 jan 2016. ISSN 0001-0782. doi: 001145/2812882 URL https://dbi.org// 20/.
1145/2812622
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for lisoo-innuuisic compositionality. In IEEE/CVE Conference Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 5228 5238. IEEE, 2022. doi: 10. 1119//CVRS2288.... 1022.0051
2022.00012222011111 URL https://doi.org/
Chien-Yar Wang and Hong- Yuan Mark Liao. YOLOv9: Learning what you want to learn using programmable gradient information. 2024.
Jingyi Xu, Hieu Le, Vu Nguyen, Viresh Ranjan, and Dimitris Samaras. Zero-shot object counting. In IEEE/CVE Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Conada, June 17-24, 2023, pp. 15548-15557. IEEE, 2023. doi: P0BPC/VV2222292...00000 URL https::/doi.orrg/ http:///ooo..oo///ooo0///////// 1wt.11111////.999.... 20W.1 01492
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- training. In Proceedings of the LEEE/CVF International Conference on Computer Vision, pp. 11975 11986, 2023.
Yan Zhang, Jonathon S. Hare, and Adam Prügel-Bennets Learning to count objects in natural images for visual question answering. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL
Yingying Zhang, Desen Zhou, Siqin Chen, Shenghua Gao, and Yi Ma. Single-image crowd counting via multi-columm convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 589-597, 2016.
22

---

Published in Transactions on Machine Learning Research (01/2025)
Appendix
A Effectiveness of our method in improving text-to-image models' counting fidelity
We provide more examples to show the effectiveness of applying method to Stable Diffusion (Rombach et al. 2021 to see if it can improve the counting fidelity of the text-to-image generation model. We show results from 3 prompts, where for each prompt, 30 images are generated with 30 unique random seeds. To compare our method with the unmodified Stable Diffusion baseline, images in the same row are generated using the same random seed. It is worth noting that our method is not always effective. However, does increase the likelihood of Stable Diffusion generating images with the correct object count.
"An old building with ruined walls and four antique pink armchairs"
"vintage silver plate tablespoons, serving spoon set of two"
"three lions"
Embedding edited
Embedding edited
Embedding edited
Original
Original
Original
23

---

Published in Transactions on Machine Learning Research (01/2025)
"An old building with ruined walls and four antique pink armchairs"
"vintage silver plate table Tablespoons, serving spoon set of two"
"three lions"
Embedding edited
Embedding edited
Embedding
Original
Original
Original
edited
24

---

Published in Transactions on Machine Learning Research (01/2025)
"An old building with ruined walls and four antique pink armchairs"
"vintage silver plate table tablespoons, serving spoon set of two"
"three lions"
Embedding edited
Embedding edited
Embedding
Original
Original
Original
edited
THIEE LIAAS
Timt: Lo Kotes
25


Reviews:
[{"review_id": "JZNZMODazQ", "reviewer": "Reviewer_PeLj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n - The proposed methods are simple and usable even in a small-scale data regime, or even without data (zero-shot text embedding editing).\n - The authors introduce two novel datasets for fine-tuning and evaluating counting accuracy\n - The paper also studies downstream use-cases of edited CLIP text encoders such as in Stable Diffusion\n - In general, the paper is easy to read. Provided figures 1-4 are helpful in terms of understanding the main contributions of the paper\n\nWeaknesses\n - Studied counts are limited to the range 2 to 10. In general, the methods do not generalize beyond the counts that have been explicitly trained.\n - The method is limited to a single counted quantity. A query like \"five dogs and three cats\" would not be directly feasible; at least it is not discussed how.\n - The paper focuses exclusively on the text encoder and implicitly assumes that the image encoder has the ability for counting (since it is not adapted). While the authors demonstrate that a pure focus on the text encoder can actually improve counting performance, a more detailed discussion and investigation of the image encoder's ability/limitations in terms of counting would be helpful.\n - The proposed datasets are not made publicly available. Also code for the methods is not provided\n - The studied CLIP models are relatively small and not state-of-the-art. To which extent would the gains observed by the authors transfer to stronger/more recent paper that are intrinsically stronger in counting? \n - Presentation of results in Section 4 and 5 could be improved; for instance: Table 2 does not highlight the best setting in bold, Figure 5 could use a range for the y-axis that focuses on the relevant part rather than showing 0-100% range,  Table 8 mixes non-comparable results on two different datasets into one table\n - Evaluation on Stable Diffusion is anecdotal, only evaluating the method on five (cherry-picked?) prompts.\n - As also discussed by the authors in Section 6, while the proposed methods are shown to be effective in the experiments, a better understanding of why and when this is expected to be the case remains open. \n - minor: Section 2 makes it sound as if OpenAI CLIP was trained on LAION-400M\n - minor: the studied diffusion model \"Stable Diffusion 1.4\" is somewhat outdated as of now. \n - minor: formatting of Section 3.2 is ugly (heading at the bottom of the page etc.). Also positioning of tables in the middle of the page in Section 4 is confusing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper focuses on improving the counting limitations of contrastively trained vision-language models like CLIP. The authors propose and study methods (both fine-tuning and zero-shot) for improving counting by learning linear direction in text embedding space that correspond to specific counts. The authors show that these methods outperform full model fine-tuning in the small data regime while maintaining CLIP's broad capabilities.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Where feasible, please address the non-minor points listed under weaknesses.", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Bwgvneq1aa", "reviewer": "Reviewer_RVAC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strength\n1. This paper improves CLIP counting accuracy via training a few parameters.\n2. Visualizations show that editing text embeddings is an effective way to enhance counting knowledge.\n\n# Weakness\n1. This method introduces nine counting vectors to manipulate the CLIP's text embedding. Each class corresponds to a certain number. There is a question of whether the method would still be practical with the number of objects increasing (e.g., 1000 objects in an image).\n\n2. The paper proposes prompts-related counting vectors. It is related to the prompt-tuning parameter-efficient fine-tuning methods. More discussions are needed between the proposed method and other prompt-tuning methods[1,2,3].\n[1] Visual prompt tuning.\n[2] The power of scale for parameter-efficient prompt tuning.\n[3] Prefix-Tuning: Optimizing Continuous Prompts for Generation.\n\n3. It has been widely demonstrated that parameter-efficient fine-tuning paradigms can improve model performance with a few tunable parameters. This method follows the widely used techniques, which may hurt the novelty.\n\n4. The novelty is limited. Previous methods [1,2] have provided explorations on improving CLIP's counting performance. For example, CrowdCLIP also transfers the counting problem into a classification problem by using CLIP. There needs to be more systemic comparisons with them.\n[1] CrowdCLIP: Unsupervised Crowd Counting via Vision-Language Model.\n[2] Teaching CLIP to Count to Ten.\n\n5. The experiments are not convincing. This paper claims that they aim to improve the counting accuracy. However, they do not conduct experiments on the counting datasets. In other words, to evaluate the counting performance, the authors could conduct comparison experiments on typical counting benchmarks, such as the ShanghaiTech and UCF-QNRF.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes learning-based and zero-shot methods to improve CLIP counting accuracy while retaining other non-counting related pre-trained knowledge and develops two object-counting benchmarks. Quantitative experiments on these two benchmarks and qualitative experiments employing the Stable Diffusion show that the proposed method improves CLIP's object counting accuracy across diverse model scales.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "see the weakness", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GTG9USjuzu", "reviewer": "Reviewer_q5MW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well-written and easy to follow. It only has a few grammar mistakes (which I will specify\nin the requested changes).\n- The proposed method is parameter-efficient compared to the full fine-tuning of CLIP.\n- The proposed method can be used to generate accurate counts of objects with Stable Diffusion. This\ncould be useful for generating synthetic data for object counting.\n- Focuses on CLIP, allowing the paper to provide a deep dive rather than a superficial investigation.\n- The experimental evaluation of counting 2-10 objects is thorough, covering 3 different CLIP backbones and reporting\nresults averaged over three random seeds.\n\nWeaknesses:\n- The paper does not consider counting more than 10 objects.\n- The related work does not provide a sufficient overview of prior open-world counting methods. These include CLIP-Count [1], CounTX [2], VLCounter [3] and other related works that use CLIP for object counting. Furthermore, unlike the proposed approach, these prior techniques can count any number of objects. For example, the test set these approaches are evaluated on includes images with over 1000 objects. Therefore, the paper needs to make it clearer why the proposed approach is useful over these prior methods.\n- The datasets that are proposed contain at most 10 objects and are limited in the categories of objects they contain. The paper does not mention FSC-147 [4] which is a standard counting benchmark that has images with thousands of objects and over 100 different categories of objects.\n- The paper does not evaluate the Stable Diffusion image generation method quantitatively.\n- The paper lacks a discussion on why certain categories are more difficult for CLIP to count. For example, perhaps the categories with the highest counting accuracy are in CLIP's large pretraining dataset. It would be good to connect this disucussion to the concept of 'open-world.'\n- A few grammar mistakes\n\n[1] R. Jiang, L. Liu, and C. Chen. Clip-count: Towards text-guided zero-shot object counting. In Proceedings of the ACM Multimedia Conference, 2023.\n\n[2] N. Amini-Naieni, K. Amini-Naieni, T. Han, and A. Zisserman. Open-world text-specified object counting. In Proceedings of the British Machine Vision Conference. British Machine Vision Association, 2023.\n\n[3] S. Kang, W. Moon, E. Kim, and J.-P. Heo. Vlcounter: Text-aware visual representation for zero-shot object counting. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.\n\n[4] V. Ranjan, U. Sharma, T. Nguyen, and M. Hoai. Learning to count everything. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a novel (and very parameter-efficient) method for improving pre-trained CLIP's counting accuracy, experimentally explores different text embedding editing techniques, introduces two new datasets for training and evaluating CLIP's counting accuracy, and shows how the presented approach can be combined with Stable Diffusion for generating images with accurate counts of objects. The proposed fine-tuning method only trains nine counting vectors. This is far fewer parameters than prior approaches to object counting. The paper focuses on counting at most 10 objects, and only evaluates the image generation approach qualitatively.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Add a discussion in the related work citing and discussing prior open-world counting approaches including the ones mentioned in the \"Strengths and Weaknesses\" section. Please emphasize how your approach is better than prior approaches to fine-tuning a low number\nof parameters for counting. For example, CLIP-Count uses visual prompt tuning to fine-tune only a subset of parameters for the counting task. Why is your approach better than this? CLIP-Count does not limit the number of objects it can count. Please also discuss why your datasets are necessary when much larger and more diverse counting datasets containing images with hundreds of objects (e.g., FSC-147) exist. **[CRITICAL]**\n2. Provide a quantitative evaluation of the Stable Diffusion approach for generating images with exact counts. For example, provide a set of text descriptions (maybe the ones from CountBench) and evaluate the number of objects generated by your approach versus the number of objects generated by the baseline and provide the resulting mean absolute errors for each approach. Since prior open-world counting methods already exist, I think the image generation is the most interesting and useful part of the paper, so it is important to evaluate it thoroughly. **[CRITICAL]**\n3. Please discuss why CLIP is more accurate at counting certain categories of objects than others. I see that you do mention that images of cats and dogs are very common. I would like you to take this further and discuss this topic in the context of 'open-world.' How well does CLIP perform on categories outside of its large pretraining dataset. **[WOULD STRENGTHEN THE WORK]**\n4. Please correct the following grammatical errors: **[CRITICAL]**\na) bottom of page 4, last paragraph: 'Similarity' should be 'similarly'\nb) Intro to section 3 includes grammatical error:\n\"In Section 3.6, we will introduce how we collect and apply new counting datasets. Prior to the main method sections, we will first give a brief **[missing word]** of CLIP models in Section 3.1 and define the counting problem and how to evaluate CLIP’s counting ability in following Section 3.2.\"", "broader_impact_concerns": "No ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Initially, reviewers appreciated the simplicity of the proposed approaches (PeLj, q5MW) and benchmarks (PeLj), the achieved results (RVAC), the downstream use cases (PeLj, q5MW), and the presentation (PeLj, q5MW). However, they also raised concerns, especially on the limited number of objects the models can handle (PeLj, RVAC, q5MW), missing experiments on standard counting datasets (RVAC, q5MW), the used CLIP models (PeLj), and missing discussions on the method choices (PeLj, RVAC) and challenges (q5MW).

The response and the revision addressed the main concerns by acknowledging the limitations and changing the focus on low-count settings, while also providing additional experiments and discussion of related works. All reviewers found the revision to be appropriate for this venue, and the AE agrees with their assessment. 

As a final, minor modification, it would be helpful to clearly elaborate (e.g., the limitations or Sec. 4.1) on the choice of datasets (and why others have not been adopted, e.g., due to the focus etc.).
Decision: Accept (poster)