Let's Roll a BiFTA: Bi-refinement for Fine-grained Text-visual Alignment in Vision-Language Models
Authors: Yuhao Sun
Chengyi Cai
Jiacheng Zhang
Zesheng Ye
Xingliang YUAN
Feng Liu
Venue: Accepted by TMLR ()

Abstract: Recent research has shown that aligning fine-grained text descriptions with localized image patches can significantly improve the zero-shot performance of pre-trained vision-language models (e.g., CLIP).
However, we find that both fine-grained text descriptions and localized image patches often contain redundant information, making text-visual alignment less effective.
In this paper, we tackle this issue from two perspectives: \emph{view refinement} and \emph{description refinement}, termed as \textit{\textbf{Bi}-refinement for \textbf{F}ine-grained \textbf{T}ext-visual \textbf{A}lignment} (BiFTA).
\emph{View refinement} removes redundant image patches with high \emph{Intersection over Union} (IoU) ratios, resulting in more distinctive visual samples.
\emph{Description refinement} removes redundant text descriptions with high pairwise cosine similarity, ensuring greater diversity in the remaining descriptions.
BiFTA achieves superior zero-shot performance on 6 benchmark datasets for both ViT-based and ResNet-based CLIP, justifying the necessity to remove redundant information in visual-text alignment.

Published in Transactions on Machine Learning Research (01/2026)
Let's Roll a BiFTA: Bi-refinement for Fine-grained Text-visual Alignment in Vision-Language Models
Yuhao Sun
yuhoo..unl (istudent.uimmed edu..ou
School of Computing and Information Systems The University of Melbourne
Chengyi Cai
chengyi. watregyi.ccii..o @eundmell. edu.aa
School of Computing and Information Systems The University of Melbourne
Jiacheng Zhang
haahhenng.hangg@@uummmncccl edu.
School of Computing and Information Systems The University of Melbourne.
Zesheng Ye
zesheng. www.:rgggge@umimmlbm.co
School of Computing and Information Systems The University of Melbourne
Xingliang Yuan
School of Computing and Information Systems The University of Melbourne
Feng Liu'
School of Computing and Information Systems The University of Melbourne.
Reviewed on OpenReview: https: openreview. net/ forum? id= 2mbkz22HH44
Abstract
Recent research has shown that aligning fine-grained text descriptions with localized image patches can significantly improve the zero-shot performance of pre-trained vision-language models (e-g., CLIP). However, find that both fine-grained lext descriptions and localized image patches often contain redundant information, making text-visual alignment less effective. In this paper, we tackle this issue from two perspectives: View Refinement and Description refinement, termed as Bi-refinement for Fine-grained Tenl-visual A Aiggmment (BiFTA). View refinement removes redundant image patches with high Intersection over Union (IoU) ratios, resulting in more distinctive visual samples. Description refinement removes redundant text descriptions with high pairwise cosine similarity, ensuring greater diversity in the remaining cleccriptions. BiFTA achieves superior zero-shot performance 6 benchmark datasets for both VaT-based and ResNet-based CLIP, justifying the necessity to remove redundant information in visual-text alignment.
1 Introduction
Drawing from the profound strides made in large-scale pre-training within natural language processing
Radford et al. 2018 2019 Devlin et al. 2019 Brown et al. 2020), the CLIP model (Radford et al. 2021
*Correspondnng author.
Code is available

---

Published in Transactions on Machine Learning Research (01/2026)
aligns vast collections of images with their corresponding natural language captions (c.g., "a photo of (label)") into unified embedding space using large datasets. The scaling of the pre-training data in CLIP empowers the model to deliver considerable performance in zero-shot classification (Radford et al. 2021 CLIP performs zero-shot classification by computing cosine similarity scores between image representations and textual label prompts so that the final prediction determined based on the distribution of these similarity scores (He& & Peng, 2017 Liang et al., 2020 Radford et al. 2021 Yan et al., 2025 Mahajan et al. )025). To push the limits of CLIP's zero-shot capabilities, several studies (Menon & Vondrick, 2023 Pratt et al. 2023) leverage the generative capabilities of Large Language Models (LLMs) to produce fine-grained. label-specific textual descriptions via. prompt templates (e.g., "describe what does a/an (label} look like"). Under this framework, CLIP model computes and integrates the cosine similarity scores between visual representation of the input image and each generaled textual descriptions. By enriching the textual modality, these refined prompts significantly boost CLIP's zero-shot classification accuracy. Recently, Li et al. (2024) propose weighted visual-tezt cross alignment score (WCA), which calculates and integrates a weighted cross-aligument score between LLM-generated label-specific descriptions and cropped image patches. By effectively refining the synergy between visual and textual representations, this cross-alignment scoring mechanism achieves the state-of-the thettrrrttrttt (SOTA) zero-shot performance on several downstream tasks.
However, as demonstrated in Figure ! we observe that both the LLM-generated textual descriptions and the randomly cropped image patches often exhibit redundant content. Such redundancy may introduce disproportionate contributions to the aross-alignment score computation, thereby compromising the accuracy of the resulting alignment. In specific, following WCA (Li et al.) 2024), we use random cropping to obtain localized image patches of an image sample (e.g., a border collie), as depicted in Figure | We find that these image patches often include certain redundant views exhibiting exceptionally high pairwise cosine similarities (i.e., approaching to 1), which is attributed to the randomness in the cropping method. Similarly, we observe that the diversity of LLM- -eMeraeee textual descriptions is often restricted by the invariant label-integrated prompt template, which directly leads to generate redundant text responses (see Figure D. In Figure 2 we further examine how the aforementioned redundancies affect the accuracy of cross-aligument score computation, using the state-of-the-art WCA scoring method as a representative example. We observe from the heatmap that the WCA score corresponding to the ground-truth label becomes more dominant after removing redundant image patches through simple Intersection over Union (IoU) filter. Since the WCA score is computed by aggregating similarity scores between image patches and the textual description, redundant patches introduce repetitive contributions that accumulate disproportionately in the final score. By contrast, after applying loU filter, the resulting WCA score distribution better aligns with the correct prediction. Furthermore, we find that as the IoU constraint becomes stricter (e.g., IoU = 0.5), the accuracy of the WCA score improves correspondingly. These findings suggest that randomly cropping image patches without enforcing overlap constraints may be suboptimal for accurately computing the cross-alignment score. Hence, these observations motivate us to remove redundant information within these image patches and textual descriptions.
To this end, we propose Bi-refinement for Fine-grained Tezt-visual A Alignment (BiFTA), new method to tackle the above-mentioned issue from two perspectives: view refinement. and descriplion refinement. Specifically, VR uses IoU as the filter metric to efficiently identify and eliminate redundant cropped image patches based on their overlaps of the bounding box. We aim to remove image patches with high IoU ratios, making the remaining visual samples more distinctive. In contrast, Descriplion Refmement (DR) first. computes the pairwise cosine similarity of the textual descriptions at the representation level, aiming to filter out redundant ones. Then, we select top-k textual descriptions that have the highest cosine similarities with the label caption ("a photo of a/an flabel}") ")) from the remaining ones.
Through extensive evaluations across six benchmark datasets, we show that BiFTA outperforms baseline methods by notably improving the zero-shot classification accuracy for both ViT-based and Res.Net-based CLIP, justifying the necessity to remove redundant information in visual-text alignment. We summarize the main contributions of our work as follows:
We observe that localized image patches and fine-grained textual descriptions often contain redundant information, making current. SOTA visual-text cross-alignment methods less effective.
2

---

Published in Transactions on Machine Learning Research (01/2026)
Weakness 1: Redundant Views
Weakness 2: Redundant Descriptions
A Border Collie is a
Cropped
B
medium-sized. athletic dog with black-and- white coat.
Patches
C
... type of herding dog that is bred in the Scottish Rorders region England.
medium sized, athletic dog with
slock, black and white coat.
Pairwise
c.mmmmo type of working dog.
Cosine
... type herding dog that bred
Similarity
the Scottish Borders region England
Repeated
Repeated
Figure 1: Weaknesses of weighted visual-text cross alignment (Li et al. 2024). Weakness 1: Pairwise similarity scores of highly overlapping crop bounding boxes. We demonstrate that image patches A, B, and C, exhibiting significant overlap and redundancy, which provide limited semantic information and consequently contribule minimally accurate classification. Weakness 2: Redundant textual descriptions generated by LLM. We gather textual descriptions from previous work and demonstrate that significant portion of these descriptions are redundant for given category, thereby diluting the contribution of meaningful and informative descriptions.
We propose an efficient data refinement method, namely BiFTTA, to mitigate such redundancy through VR and DR, enhancing the alignment between visual and textual modalitios.
We empirically show that BiFTA outperforms baseline methods by achieving significant improvements in zero-shot classification accuracy across 6 benchmark datasets with varions CLIP backbones.
2
Related Work
Zero-shot learning for Vision-Language Models. Vision-language models (VLMs) have shown their
emergent capabilities on image captioning, visual question answering and image classification, which are not
specifically pro-trained or explicitly finetuned for these downstream tasks (Radford et al. 2021 Cho et al.
2021:
Wang et al. 2021 Kim et al., 2021 Xue et al. 2021 el al. 2022a Alayrac el al., 2022 Chen el al,
2025
Cai al.) 2025c CLIP (Radford al.) (0221) demonstrates that integrating large-scale pre-training
imago-text pairs with contrastive loss function could enable zero-shot transfer to downstream tasks by
simply using natural language prompts. Similarly, ALIGN (Jia et al.) 2021] further demonstrates robust representation learning capability of VLMSs by pre-training noisy image-text pairs at large scale. By increasing the scale of the pre-training data and model size, Florence (Yuan et al. 2021 introduces a unified lasionelanguage foundation model capable zero-shot image classification and retrieval. On the other hand, CoCa (Yu al. 2022) combines contrastive and generative objectives improve zero-shot generalization across diverse tasks. The scaling of pre-training data and the contrastive learning paradigm contribute deeper visual-text alignment and visual understanding of the model.
Textual prompt engineering in VLMs. By scaling the training data, VLMs can learn and understand diverse visual concepts, which can then be transferred to downstrcam tasks through specific textual label
prompting (Radford et al. 2021 Jia el. al. 2021 Yuan et. al. 2021 Li & Liang 2021 Singh et al. 2022 Zhou
et al. 2022 Shu et al. 2022 Cai et al.) 2024b Cui et al. 2025 Ye et al. 2025 Yin et al. 2025 The
LLM-intograted textual description generation shows great generalization ability comparing with existing
2022b Wang al. 2022 Wu et al, 2023 Tanwisuth et al. 2023 Cai et al. 2025b). CLIP (Radford et al. 2021 achieves zero-shot classification
prompt-learnig methods, which often overfit training data (Li
3

---

Published in Transactions on Machine Learning Research (01/2020)
toU-0.55
loU-0.7
IoU-0.3
WCA
0.4124
Ground Truth: Goose
20nse
grose
goose
Suck
dowitcher
black swen
dowlicher
duck
Normaiized UCA Score
Top- Predictions
Craine bied
common redshank
Implis
red merganser
bustard
duniin
pelitar
bustard
camman gellinuul
ostrichh
limpkín
crane bird
common ga limute
laleehoree
bitterm
famiinge
duniin
10
dow toher
block owon
American coot
oystercatcher
Figure 2: We select an ImageNet image of a goose and display the Top-10 predictions ranked by WCA scores. The scores are normalized using softmax and its distribution is visualized using color intensity. The results show that applying an ToU-based filter to climinate duplicated image patches significantly enhances the precision of WCA scoring.
by generating classification weights through encoding textual descriptions that uses CLIP template and categories via its text encoder. then compares these text embeddings with image features extracted by the image encoder to determine the most likely class. Zhou et al. (2022) discover that manually prompt tuning is time-consuming task and propose CoOp, which models context words with continuous vectors. Subsequently, Menon & Vondrick [2023): Pratt et al. (2023) automatically generates textual category-specifi descriptions by leveraging LLMs with different prompt templates. These textual descriptions can accurately reflect visual features of images in each category. More recently, retrieval-augmeened generation (RAG) is proposed help to generate accurate descriptions categories, which is a training-Tres framework that
can be directly integrated during inference time (Yu et al.) 2024 Chan et al.) 2024 Guo al., 2024p. It retrieves semantically relevant documents by computing embedding vector similarity and provides the retrieved information supplementary context to LLMs, enabling more accurate and informed responses.
Fine-grained visual-text alignment. Weighted visual-text cross alignment (WCA) L et al. 2024 has done empirical observation on the embedding alignment between visual patches and textual descriptions. It uses random crops to augment image samples and utilizes cosine similarity to extract informative patches. Similarly, utilizes distinctive textual descriptions from LLM to cross-align with the image patches. There are uncertainties when cropping samples randomly, and the textual descriptions are not corresponding to fine-grained image patches. AttrVR (Cai et al., 2025a) uses descriptive and distinctive altributes of each categories from LLM outputs. In method, augment the textual descriptions by integrating various description generation methods to further select the high-quality textual descriptions.
3 Preliminary
CLIP Zero-shot Classification. CLIP (Radford et al. 2021 is a pro-trained VLM that consists of a text encoder fix : + and an image encoder /img X 7=> 22 where T is a discrete text space, X' is a continuous image space and Z C R" is a shared n-dimensional embedding space. These encoders take an image E and a text T' E as input pair (X,T), mapping them into the shared latent space Z. Then the similarity score between the image and text embedding is calculated
simCLIP (xncIIIPX777 cos( Zi, ZT)TT, with Z: Jimg(X), and Z. Sixt (T);
where cos(..) denotes the cosine similarity such that cos Z, Zi)
[2.11 2. and is temperature parameter.
2.
4

---

Published in Transactions on Machine Learning Research (01/2026)
For downstream classification tasks, the text encoder of CLIP model receives label prompt string F, (e.g.. "This is photo of a/an [y]"), where y E V/. Subsequently, CLIP model predicts the label which maximizes the probability of PCLIP (Y X),, given by:
arg max PCLIP = y IX) = exp(simerap (X, T.)) vEY
Eveyexp(sineg (X, r)
Here, the label y that maximizes the conditional probability PCLIP = XX) will be chosen, where the visual and textual representations exhibit the highest similarity within the embedding space Z. This enables zero-shot classification capability on CLIP model, as it can generalize to unseen categories without additional fine-tuning.
Weighted Cross- Alignment (WCA). WCA is a scoring method specifically designed to improve the visual-text cross-alignment capability of the CLIP model (Li et al. 2024). First, an original image X = x is randomly cropped with a window size ranging from [a, 3] E [0 for n times. It obtain n cropped image patches denote as I = rnd__ropp ____por))) i E [0. n), where rnd_ md__ropp-) is the random cropping function that obtains localized visual features (Li et al., 2024). On the other hand, WCA would prepare m textual descriptions T1, .,. Tm generated by LLMs, which encompass descriptive features of each category y e y. The textual descriptions are collected by leveraging a LLM with manually crafted prompt templates, such "Describe what a/an category looks like." (Pralt et al. can be presented as:
2023 Then the visual-text similarity score matrix
SiMCLIP (1)
siMCLIP (41, Tm)
simCLIP (In, simCLir (Ia, Tm
where (I,,,, represents the similarity scores between specific textual description T3 and an image patches 4. When applying WCA, the overall similarity score between image and label is as follows:
simwca = x, = = Σ M wvy, simeLIP(41,
(1)
i=1
where wy and fi are weights for image patch I and textual description Tj, respectively. They are obtained from the similarity between L; and the original image I, or T3 and the label prompt string Fy i.e. w; (aaxfcos([fxtl])
softmax cos(Simg((e)) (img((,))) and v, = pairs with aforementioned weights (Li ct al. 2024). ) However, redundant information may appear in image patches and textual descriptions, leading to the weaknesses shown in Figure I
fixe (T,)). WCA effectively aligns visual-text.
4 BiFTA: View Refinement and Description Refinement
The BiFTA framework seeks to eliminate redundant information from two primary perspectives: (1) VR
(Section 4.1 which removes overlapping image patches based on ToU scores; and (2) DR (Section 4.2). which integrates multiple methods for generating fine-grained textual descriptions and filters semantically similar embeddings using cosine similarity scores. An overview of BiFTA is illustrated in Figure 3 Our approach integrates efficient data filtering and refinement techniques to refine cross- alignnd image-text pairs, ultimately
enhancing the quality and accuracy of image classification. Furthermore, in Scction 4.4 we formalize the concept of Redundant Views/Descriplan and BiFTA- deduppiccated set.
4.1 View Refinement
As shown in Section 3 in WCA, for a single image x, a set of randomly cropped image patches V = {1,, l,.,,,,.,,,, } is created for subsequent cross-alignment. However, as illustrated in Figure the random cropping method frequently selects the same region or adjacent regions for cropping, which could affect the classification results. Therefore, without changing the size of the set [V] we make [V] into a queue to store all the image patches. Then we employ filtering function (iou(-) to ensure that each newly cropped image
5

---

Published in Transactions on Machine Learning Research (01/2026)
Description Refinement
View Refinement
IoU Filter
CoS Filter
Horder Collie
medium-siine sized, athlotic dog with
1.00
1.00
0.00
0.82
0.00
0.70
Image Empodler
Text Enooder
1.00 0.95
1.00
0.95
0.99
0.92
0.94
1.00
0.
type of harding dog that bred in Borders region England.
black- sack--nn--an white coot.
Soores
X
*
1.00
0.92
0:95
medium sixed, athiletic dog with
black and while coal.
1.00
0.00
1.00
common type of working dog.
Test Image
1.00
X
1.00
the Scotlish Borders region of England.
type herding dog
Image Flow
A Alignment
Text. Flow
Figure 3: An Overview of BiFTA. To reduce potential redundancy in views and descriptions, the randomly cropped views undergo filtering with the IoU filter (Section (.1).. while the randomly sampled description texts are processed using the CoS filter (Section 4.2 when computing the similarity between a single image and a single label. The similarity score is then calculated on the refined views and descriptions.
patch added to the current V does not have excessive overlap areas with the existing images in current queue. For newly cropped image I, = rnd_ _rdp_ro(()) the filtering function can be expressed as:
fio (T, = {
A TOU T.) < 1 E IoU Ii) 2 - 8
(2)
where ToU(.) is the Jaccard index (i.e., intersection over union) between two image patches, and 1 - is a hyperparameter representing the IoU threshold detailed in Section 5 When fiou(4, V) = 0, the view set V remains unchanged. However, if fio V) = 1, will be appended with the new view I, i.c., V V [U()). Thus, by maintaining the size of V consistent with WCA, method effectively introduces greater number of semantically independent views.
4.2 Description Refinement
In previous works, CuPL (Pratt et al. 2023 used label-integrated prompt templates such as "Describe what a/an [label] looks like" to obtain various appearance descriptions of different classes. Meanwhile, AttrVR (Cai et al. 2025a) employs prompts like "Describe the appearance of [task] [label]" to obtain DesAttr (which describes intra-class fcatures) and DistAttr (which distinguishes inter-class features). Both methods involve manual filtering remove noise from low-quality generations. As they are built. upon LLM-based generation with carefully designed templates, the resulting textual descriptions are semantically aligned and comparable in granularity instead of heterogeneous. Accordingly, initiate the DR process by taking the union of the two description sets produced by CuPL and AttrVR, treating it as a form of data augmentation. For a given label y, let us denote the three sets of descriptions as TCuPL (y)), DD)s DDist (y), respectively. We can then formulate the equation as:
(CPPL( ftLMsl/(oupl prompt)), DDDes = fLLM [des prompt)), Dist (v) fL.m(yl(dist prompt)),
where fLLM()) returns the LLM output given queries, and cupl prompt, des prompt, dist prompt are aforementioned prompts used by these methods.
Similar to the VR, in order to alleviate the redundancy when obtaining description set D {Ti, T2.... Tm we also employ filtering function (.,s(,,)) during random sampling. When sampling text T; E T(OPI U Des U Distt (y), the filter function fcos D) which determines whether new sampled T; should be included in current set D can be represented as fcos(T,, D) = (),(T.,D) (T,, D), which consists of a
hhtpss:/enn.eiipedla..ng//
index
6

---

Published in Transactions on Machine Learning Research (01/2026)
filter fcs(+;)) that removes similar descriptions:
(cs(Ti, {
AA E D. fix(()) (T), foxt (Ti))
<1-6
hE D. cos fta ( fixt (Ti))
21-c
and another filter STopk(ss) for eliminating noisy or irrelevant descriptions:
Topp (T;, D) { T, Top-k(eos(S<al (T Saxi (()) T
T, (op-k(cos( (Ty), fexi (T)) TED)
where is a hyperparameter representing the threshold, Top-k(-) returns the set of variable Ts corresponding to the top & function values, and T, is the label prompt string (e-g. "This is a photo of a/an [label]")
The use of fcs(5)) ensures that the selected description set contains as little repetitive or redundant textual content as possible. The use fTopk(-;) minimizes the presence of distracting descriptions in the candidate description set (e.g., noisy text generated by an LLM). When (T; D) = 0, the description set D remains unchanged. However, if fcos (T,, D) = D will be appended with the new description T,, i.c., D < DU (Ti).
In conclusion, DR first forms unified description pool by combining two high-quality description sets, then removes duplicate pairs and only keeps the top-k semantically matching pieces into our description pool. As result, the textual descriptions become more diverse under fixed set size [D], improving the effective utilization of informative descriptions.
4.3 Overall Pipeline
The overall pipeline of BiFTA is illustrated in Figure 3 From the Image Flow illustrated in the figure, BiFTA performs VR to the randomly cropped image patches, Each patch is stored in a fixed size queue V,, while each image patch is filtered using the IoU filter fioU, as defined in Eq. [ From the Text Flow illustrated on the other side of the figure, BiFTA conducts DR on a set of textual descriptions. The candidate texts are first filtered by the CoS filter fcas and then ranked according to their top-k similarity scores with the label prompt. Finally, the similarity between the refined views and descriptions is computed using Eq. D to yield the final prediction. The detailed algorithm is provided in Appendix A
4.4 Concept Formalization
In this section, we formally define the concept of Redundant /ieuus/Deccrptionss in Definition I and the Deduplicated Set in Definition 2
Definition 1. (Redundant Views Desuusspsrrittnnss.... Assuning 1; and I, are two views of image x, If 2 1,8, 8, where - 8 is the IoU threshold, then and 1ú are considered to have significant overlap and are regarded redundant views of each other. Similarly, for tertual descriptions Tp and Tg associated with label y, is
cosl fox(fxx(TT)) fext (Ta)) 21-c,
where - is the cosine similarity threshold, then the two descriptions are considered nearly identical and thus mutwally redundant.
Definition 2. TBiFTA-Deduplian
Set). Given a set of views V = (T1, I2...., In) of size n, V is a deduplicated view set if and only if
VV, ToU(,, 12) 1 -- 5.
Similarly, for set of textual descriptions D -Ti, (2,,,2,..... 3m, containing elements, D is deduplicated descriplion set if and only is
VT,,
cos(fexe (Tp), fix) < <1-6.
Through Definition 2 and Sections 4.1 4.3 we conclude that the view and description sets used by BiFTA satisfy the deduplicated set constraints. In contrast, the view and description sets employed by WCA do not meet these constraints, as no explicit doduplication is enforced during random image cropping or description sampling.

---

Published in Transactions on Machine Learning Research (01/2026)
5 Experiment
First, we outline the complete experimental settings in Section 5.1 In summary, we evaluate the refinement
performance of BiFTA through extensive experiments on 6 benchmark datasets and 5 different backbone architectures of the CLIP model. To further demonstrate the generalizability of our framework, we additionally conduct experiments on other VLM architectures in Appendix B including ALIGN (Jia et al.] 2021
AItLIP (Chen et al., 2022p, GroupViT (Xu et al., 2022), and SigLIP Zhhai et al. 2023 which suggest that
BiFTA consistently improves cross-alignment methods. In Section 5.2 we primarily present the zero-shot. classification performance of the CLIP model with ViT-B/32 backbone across 6 downstream tasks. The rest experimental results for other CLIP backbones are provided in Tables 11 in Appendix B In Section we conduct ablation studies to validate the design principles of VR and DR. For completeness, we further explore alternative refinement strategies in Appendix E. Finally, the limitations of our work are discussed in
5.3
Appendix G
5.1
Experimental Settings
Datasets. To evaluate BiFTA, we conduct experiments on 6 downstream classification tasks under a zero-shot
setting, including: (1) ImageNet (Deng et al.) 2009), a large-scale datasel comprising 1,000 diverse object classes; (2) CUB (Welinder et al. 2010), a fine-grained dataset of 200 bird species, focusing on subtle visual distinctions; (3) Oxford Pets (Parkhi et al. 2012), a dataset of 37 pet categories; (4) DTD (Cimpoi al.) 2014), texture dataset containing 47 categories of materials and surfaces; (5) Food101 (Bossard et al, 2014], a dataset of 101 food categories; and (6) Place365 (Zhou et al., 2017), scene recognition dataset with 365 categories of indoor and outdoor environments. These datasets span a wide range of categories, which encompass various visual domains such as scenes, lextures, food, animals and fine-grained objects. Thus, the evaluation ensures the robustness and generalizability of BiFTA across various real-world applications.
Basclincs. We evaluate the performance of BiFTA by comparing with competitive basclines on zero-shot classification: (1) CLIP (Radford el al.) 2021), a naive approach that incorporates manually crafted label prompt; (2) Ensemble CLIP (CLIP-E) Radford et al. 2021), an advance approach that incorporates series of label prompts; (3) CLIP-D (Menon & Vondrick 2023) an approach that utilizes category descriptions generated by a LLM instead of label prompting approach; (4) Wallle (Roth et al. 2023), a novel approach that replaces I.I.-generated category descriptions with random word descriptions; (5) CuPL. (Pratt et al., 2023 a method that leverages LLM and improves the quality and variety of textual descriptions compared with CLIP-D; (6) WCA (Li et al., 2024). a recently proposed method that computes cross- aoossalligmmnnt scores between localized image patches and fine-grained textual descriptions.
Implementation Details. We primarily use the pre-trained CLIP model for main experiments, including both Vision Transformer (ViT) and ResNet backbone architectures, specifically ViT-B/32, VIT-B/16, VIT- L/14, RN-50 and RN-101, These architectures are selected to enable thorough analysis of the proposed mothod across varying scales and complexitics. We keep the shared hyperparameters consistent with the settings in WCA (Li et al. (024)): we use n = 60 for the patch queue length [V] and m = 50 for the number of textual descriptions per category. We keep the cropping window size consistent with WCA, ranging from [otow; (high), where now 0.5 and Shigh 0.9 across all experiments. Additionally, we store the embeddings of localized image patches during the initial execution (Li et al.) 2024), which allows them to be reused when eva evaluating different sets of textual descriptions, thereby significantly reducing computational costs.
5.2 Zero-shot Classification Results
Tables to corresponding to a different CLIP backbone architecture.
2 present the zero-shot classification results across six downstream tasks, with each table
In Table 1 the classification performance of CLIP (B/32) underscores the consistent superiority of BiFTA over all baselines. In specific, BiFTA achieves a 3.33% improvement in accuracy over the WCA scoring method on the DTD dataset, which consists of diverse texture patterns. This improvement stems from the inherent challenge of the dataset: textual patterns in DTD are often homogencous, where random cropping strategy tends to produce semantically similar regions. In BIFTA, VR systematically eliminates redundant image
8

---

Published in Transactions on Machine Learning Research (01/2026)
Table 1: Zero-shot classífication accuracy (%) across various baseline methods with the pre-trained CLIP model (ViT-B/32). We report the averaged results and standard deviations a of 20 runs, with the improvement A (%) over the top-performing baseline WCA highlighted in green. The results of our method are highlighted and we use bold to represent the best-performing method.
Method
ImageNet
CUB
Oxford Pets
DTD
Food101
Place365
CLIP
62.05
51.21
85.01
42.93
82.60
38.51
CLIP-E
63.37
52.74
87.38
43.83
83.93
39.28
CLIP-D
63.01
52.69
84.46
44.20
84.12
39.90
Waffle
63.30
52.04
85.50
42.98
83.98
39.47
CuPL
64.37
49.76
87.03
47.50
84.20
39.08
WCA
66.49
56.74
89.05
49.89
86.11
40.55
BiFTA (ours)
66.83:10.04
58.24:10.17
89.74:10.14
53.22-10.26
86.43-10.06
41.55:10.06
A
+0.34
+1.50
+0.69
+3333
+0.32
+1.00
Table 2: Zero-shot classification accuracy (%) across various baseline methods with the pre-trained CLIP model (ViT-L/14). We report the averaged results and standard deviations a of 20 runs, with the improvement A%%) over the top-performing baseline WCA highlighted in green. The results of our method are highlighted and we use bold to represent the best-performing method.
Method
ImageNet
CUB
Oxford Pets
DTD
Food101
Place365
CLIP
73.48
62.12
93.24
52.61
92.55
39.63
CLIP-E
75.52
62.53
93.62
55.43
93.07
40.55
CLIP-D
75.03
63.26
93.30
55.05
93.03
40.55
Waffle
75.31
62.27
91.55
54.31
93.33
40.89
CuPL
76.62
62.15
94.33
60.59
93.37
40.77
WCA
77.32
65.12
94.67
61.74
93.93
42.19
BiFTA (ours)
77.82:00.04
65.6710.13
94.9610.10
62.45-00.26
93.9710000
42.98=0.05
A
+0.50
+0.55
+0.29
+0.71
+0.04
+0.79
Table 3: The average Zero-shot classification accuracy (%) over five different CLIP backbones across all downstream benchmarks. Improvements are shown in green.
Dataset
WCA
BiFTA
A
ImageNet
68.10
68.91
+0.81
CUB
55.31
56.05
+0.74
Oxford Pets
90.34
90.38
+0.04
DTD
52.77
54.52
+1.75
Food101
87.04
87.18
+0.14
Place365
40.22
41.15
+0.93
patches through the IoU filter function. The image patch queue V is thereby maintained as deduplicated set consisting of semantically independent candidates. In Eq. [ each selected image patch is assigned corresponding visual weight wy. This weight factor ensures that discriminative visual features dominate the computation of the cross-alignment score, thereby enhancing the model's robustness against repetitive patterns such as the lextures in DTD dataset. The text weight factor follows analogously, where it highlights semantically distinctive textual descriptions. In Table 2 we observe that BiFTA consistently outperforms all bascline methods when evaluated with a largo-scale backbone (ViT-I//14). Notably, achieves a (+0.50%) improvement on ImageNet, which significant gain for such a high-capacity model. Notably, BiFTA yields only marginal improvements large-scale datasets, such the +0.04% gain reported in Table 2 The results

---

Published in Transactions on Machine Learning Research (01/2026)
Table 4: Average classification accuracy (%) across various baseline methods with different CLIP models. The improvements (%% over the top-performing baseline (i.c., WCA) are highlighted in green. We use bold represent the best-performing method and underlined to represent the second-best method.
BiFTA (ours)
Model Architecture
CLIP
CLIP-E
CLIP-D
Waffle
CuPL
WCA
A
ViT-B/32
60.39
61.76
61.40
61.21
62.16
64.81
66.00
+1.19
-YT6
63.59
64.51
64.67
64.34
66.09
67.87
68.29
+0.42
ViT-L/14
68.94
70.12
69.87
69.61
71.31
72.50
72.98
+0.48
RN-50
56.97
58.64
38.39
57.92
60.01
62.00
62.54
+0.54
RN-101
59.14
60.50
59.22
58.89
59.04
61.14
62.03
+0.89
Table 5: Comparison of Top-1 accuracy (%) across alternative VLM architectures under zero-shot classification setting. All models are evaluated on the ImageNet- 1K benchmark.
VLMs
Vanilla
Vanilla_E
Vanilla_D
Waffle
CuPL
WCA
BiFTA
ALIGN 65.24
65.79
65.08
65.22
66.24
66.61
67.15
AltLIP
73.79
74.86
74.48
74.29
75.74
76.20
76.88
GioupViT
37.11
42.72
40.10
40.42
44.53
45.27
45.31
SigLIP
76.18
76.22
76.51
76.10
77.04
77.40
77.74
obtained with the other CLIP backbones and alternative VLM architectures are reported in Appendix B To provide a more holistic assessment of model performance, we further report results averaged over five different CLIP backbones across all downstream benchmarks in Table 3 While the average improvements over WCA on certain large-scale datasets remain modest (e.g., +0.04% on Oxford Pets and +0.14% on Food101), prior work has shown that large-scale, high-diversity benchmarks tend saturate more quickly than smaller or more homogeneous datasets (e.g., DTD CUB) that making incremental performance gains less visually prominent (Menon & Vondrick 2023 Roth et al. 2023). Overall, the averaged results indicate that method consistently purperfosms the baseline across diverse downstream benchmarks. In Table 4 report the performance of each CLIP backbone averaged over all evaluated benchmarks to provide a complementary view the results. specific, BiFTA achieves an average improvement 0.42% to 1.19% across different CLIP backbones compared to WCA. BiFTA exhibits similar trend observed original WCA that smaller backbone models (e.g., ViT-B/32) exhibit more significant improvements (+1.19%) comparing with their larger counterparts (e.g., ViT-L/14), (+0.48%). This trend highlights BiFTA's ability to greater enhance weaker backbone reprosentations, leading to relatively larger gains smaller models.
In Table 5 report supplementary experiment that evaluates the performance of BiFTA under alternative VLM architectures. This consistency underscores the adaptability and effectiveness of our refinement method when applied to a diverse range VLMs. For case of distinguishing VLM names from method names, we denote CLIP and two variants (CLIP-F and CLIP-D) as Vanilla, Vanilla E, and Vanilla_D in Table 5
For clearer visual comparison, Figure 4 illustrates the differences between BiFTA and WCA. On the left, given image of goose, BiFTA successfully filters out semantically independent regions (e.g., head, wings, neck, and fur), whereas WCA usually produces overlapping image patches (e.g., repeated neck regions background noise) so that ignores valuable features. This highlights how VR mitigates feature redundancy through loU-guided filtering, On the right, text samples describing a goose are presented. Descriptions refined by the CoS Filter are notably more diverse and distinctive; semantically meaningful prompts such as "plump, clongated body" and "a long, slender body" are preserved, while redundant phrases like "long nocks and webbed feet" are pruned. This demonstrates the effectiveness of DR in promoting semantic diversity among textual inputs.
10

---

Published in Transactions on Machine Learning Research (01/2026)
Sample Views of WCA
Sample Views of Ours (w/ loU Filter)
Sample Descriptions of W CA
Geese are walerfowl with long necks and webbed feet,
A goose is large bird with a long necks and webbed feet.
A goose is waterfowl with long necks and webbed feet.
Sample Descriptions of Ours (w/ CoS Filter)
The object goose is medium-sized bird with plump, elongated body.
Geese waterfowl with long necks and webbed feet.
The object goose is bird that typically has long, slender body covered feathers.
Figure 4: A visualization comparing the effectiveness of VR and DR in BiFTA against WCA. Left: with ToU filter, the cropped samples exhibit diverse and distinctive localized features. Right: with CoS filter, the texts can describe various local features of a category.
61.0
51.0
Accuracy (%)
50.5
Accuracy (%)
60.0
50.0
49.5
49.
0.6
0.8
20
100
IoU Threshold
Queue Length
Figure 5: Left: Accuracy of using alternative loU thresholds on DTD dataset with CLIP (B/32). Right: Accuracy of changing the size of [V] on DTD dataset with CLIP (B/32).
5.3 Hyperparameter Analysis and Ablation Studies
VR hyperparameter tuning. Figure 5 illustrates the impact of varying hyperparameter values on downstream performance: loU threshold 1 8 and patch queue length N -V. On the left, the classification accuracy varies as the changes, with accuracy gradually rising and then dropping r) increases, which suggest that moderate is needed to achieve optimal performance. In practice, lower 1) will result in critic cut-off to the cropped image patches, which often results in the deduplicated queue V containing an insufficient number of samples. To simply address this limitation, we re-sample from the existing queue until N samples are obtained. However, this procedure inevitably introduces redundant patches into V, thereby violating the principle of maintaining deduplicated set. shown the left panel of Figure 5 this trend further indicates that rodundant information can adversely affect downstream task performance. Notably, VR becomes ineffective as n approaches 1 due to the overly loose restriction on the cut-off. The performance exhibits decreasing trend once IoU exceeds 0.80, which is expected since the views longer effectively constrained by our VR. The right panel of Figure shows that increasing the queue length N has a consistently positive impact on classification accuracy, whereas the increasing trend reaches a plateau at 60. For consistency, we set 0.80 and N = 60 across all experiments.
DR hyperparameter tuning. We conduct an ablation study on the cosine similarity threshold c and the Top-k selection strategy used in DR. Using the ViT-B/32 CLIP model on ImageNet-1K, we evaluate zero-shot, performance under different combinations of (c, k) pairs. Rather than varying single parameter
11

---

Published in Transactions on Machine Learning Research (01/2026)
ImageNet
CUB
Oxford Pets
94
Accuracy (%)
12
Accuracy (%)
Accuracy (%)
00
70
86
884
82
80
RAG
attr/R
CuPL
Mix
RAG
attr/R
CuPL
Mix
RAG
attr/R
CuPL
Mix
DTD
Food101
Place365
68
Accuracy (%%
Accuracy (%)
Accuracy (%)
42
68
89
41
87
39
48
38
RAG
altrVR
CuPL
Mix
RAG
attrVR
CuPL
Mix
RAG
altrVR
CuPL
Mix
Figure 6: Comparing VR with different description sets and strategies on BiFTA. The results averaged
across 3 image encoder backbones of the CLIP model, where 'mix' is the strategy BiFTA finally utilized for VR.
Table 6: Ablation study on the DR hyperparametors: cosine threshold and Top-k selection using ViT-B/32 on ImageNet-1K.
(c, k)
w/ DR Acc. (%)
(0.90, 10)
65.02++0.07
(0.95, 20)
66.45-10.01
(0.99, 30)
66.70+0.04
(0.99, 40)
66.84+0.05
(0.99, 50)
66.83+0.04
independently, we consider (c, k) pairs, as a strict cosine threshold alone may leave an insufficient number of textual descriptions in the pool. For all main experiments, adopt (c = 0.99, k = 50), specified in Section [.1 In Table the ablation results show that retaining larger number of textual descriptions generally leads to improved performance, underscoring the importance of preserving sufficient semantic diversity. In contrast, overly restrictive cosine thresholds discard a substantial portion of candidate descriptions, resulting in reduced diversity and degraded performance. Overall, the setting (c = 0.99, k = 50) provides a favorable balance between diversity and redundancy, which yields stable and strong performance across benchmarks.
Ablation studies of alternative DR strategies. Figure t presents the ablation results of using different sets of textual descriptions. These descriptions generated by LLMs with various prompt template designs, as introduced in Section 4.2 Among all settings, only the mixed description set from our DR demonstrates consistently superior performance across most downstream tasks, with results averaged over three CLIP backbones: ViT-B/32, V16- and ViT-L/14. The dashed line indicates the average performance of WCA, which is the strongest baseline method. Relatively, BiFTA achieves varying degrees improvement depending on the description set. Notably, descriptions generated with RAG-prompt templates perform unsatisfactorily compared with other sets, and details of their implementation are provided in Appendix C We provide more ablation results of using different sets of textual descriptions in Appendix D In addition,
12

---

Published in Transactions on Machine Learning Research (01/2026)
Table 7: Ablation studies of comparing the performance of WCA and BiFTA between single modality refinement (w/o VR and w/o DR) and full refinements, using CLIP models (B/32, B/16 and L/14). VR: View Refinement; DR: Description Refinement. The best result for single dataset across each model is underlined, and the best averaged results (%) are highlighted in bold.
Oxford Pets
Methods
ImageNet
CUB
DTD
Food101
Place365
Avg.
WCA BiFTA (w/o VR)
66.49
56.74
89.05
49.89
86.11
40.55
64.81
66.51
58.11
88.56
51.27
86.41
41.80
65.44
B/32
BiFTA (w/o DR)
66.77
56.94
89.17
50.51
87.46
40.85
65.28
BiFTA (ours)
66.83
58.24
89.74
53.22
86.43
41.55
66.00
WCA BiFTA (w/o VR)
71.05
59.87
92.13
52.87
89.99
41.33
67.87
70.67
59.36
90.58
51.36
90.20
42.23
67. 67.40
B/16
BiFTA (w/o DR)
71.10
59.91
91.83
53.56
90.38
42.11
68.15
BiFTA (ours)
71.14
60.06
91.67
54.64
90.11
42.12
68.29
WCA BiFTA (w/o VR)
77.32
65.12
91.67
61.74
93.93
12.19
72.50
77.14
65.46
94.63
62.09
93.94
42.51
72.63
L/14
BIFTA (w/o DR)
77.89
65.56
94.78
62.17
94.04
42.58
72.84
BiFTA (ours)
77.82
65.67
94.96
62.45
93.97
42.98
72.98
CuPL equivalent to applying only the VR component of BiFTA, we also provide quantitative result Table 7 This suggests that even refining single modality (the visual side) is sufficient to improve zero-shot image classification performance.
Ablation studies of BiFTA w/o VR and DR. We also compare the experimental results of the WCA scoring method with a complete version of BiFTA and partial BiFTA in Table 7 We demonstrate that applying BiFTA refinement to a single modality is sufficient to improve cross auras-aligmment performance, For ImageNet and Food101 datasots, the models often exhibit better performance with BiFTA w/o DR, which indicates our merged description set might not be an optimal description set. Overall, BiFTA with dual refinements achieves the highest average improvement across all three CL.IP backbones, shown in Table 7
Table 8: Performance comparison among two alternative View Refinement (VR) strategies.
VR
ImageNett
CUB
Oxford Pets
DTD
Food101
Place365
Avg.
Soo feLIP
66.77
56.94 55.41
89.17 88.48
50.51
87.46
40.85
65.28 65.35
66.93
52.10
88.21
40.94
Table 9: Performance comparison among two alternative Description Refinement (DR) strategies.
DR
ImageNet
CUB
Oxford Pets
DTD
Food101
Place365
Avg.
fTe-ID fes
66.38 66.51
58.05
88. 74 88.56
51.25
86.19
42.14
65.46
58.11
51.27
86.41
41.80
65.44
VI E V, cos fing fing (4:)
<
fcup(4.
I E cos Simgl (I), (img([i)
>
VT E D, cosl Sembh Somb (Ti)
< -6 2 21- 1-ee
(TT.ID(TT,DD)
0 Ie E D, cos fembb(T) femb (T))
Exploring Alternative Refinement Strategies. Table g illustrate the result of the downstream tasks by utilizing alternative VR/DR strategies. For alternative VR strategies, first attempt to directly the
13

---

Published in Transactions on Machine Learning Research (01/2026)
Table 10: Runtime time of WCA and BiFTA,
Execution per image
Time
Generate and encode 100 crops
226.08 ms 10.07 ms
ToU filtering
20.61 ms + 7.77 ms
Total time
246.69 ms + 14.10 ms
image encoder of the CLIP model Jing to obtain the embeddings of each image patch. Then we incorporate the same cosine similarity function to remove redundant image patches, as shown in Eq. 3 As a result, SCLP offers slight improvements in zero-shot classification on rownst lasks. However, computing image patch embeddings is highly computationally intensive, since each sample must be divided into N patches. To make framework practical, data refinement procedure should be designed in lightweight manner that minimizes additional compulational overhead. In contrast, fioU achieves a favorable balance between efficiency and performance. We additionally explore an alternative VR strategy based on grid cropping instead of random cropping in Appendix which is more computational efficient approach.
For alternative DR strategies, we leverage TF-IDF to encode the text descriptions into text embeddings (Jones 2004), denoted as femb- Then, we incorporate femb into compositional function fIF-IDF to eliminate duplicate text descriptions, as shown Eq. 3 Similar to the proposed fcos in Section 4.2 we obtain the final filtering function fT, D) fIF-IDF (Ti, D) (Topk(T,, D). The results indicate that the two DR strategies yield nearly identical performance. Nevertheless, both consistently outperform the baselines. The core of DR lies in leveraging the fropk function to select the most semantically relevant descriptions to the label prompt after an initial filtering step with either fcs or JTF-IDF. Overall, lightweight yet principled refinement strategies are sufficient to yield competitive gains without incurring excessive computational overhead.
5.4 Runtime Comparison
BiFTA introduces only minor offline preprocessing costs and no additional inference-tine overhead. First, VR adds an IoU-based filtering step to the randon-crop procedure, incurring an average of 20.61 ms per image compared to 226.08 ms for crop generation and encoding, as shown in Table I0 The patch embeddings are cached and reused during subsequent inference (Li et al. [002]). Then, DR is performed entirely offline via cosino-similarity filtering and Top-k selection, requiring only an average of 42.36+8.65 ms per category. Overall, both VR and DR introduce only minor, one-time offline preprocessing costs, and BiFTA incurs additional inference-time overhead compared to WCA.
6 Conclusion
In this work, we identified a critical limitation in existing fine-grained visual-text alignment methods: the presence of redundant information in both localized image patches and LLM-generated textual descriptions. To address this, we propose BiFTA, a novel framework that introduces two key innovations: (1) VR via loU-based filtering to eliminate spatially overlapping image patches, and (2) DR through cosine similarity thresholding to remove semantically redundant textual descriptions. Our experiments across 6 benchmark datasets demonstrate that BiFTA consistently outperforms state-of-the-arr methods in zero-shot classification accuracy over the previous methods. The ablation studies validate the necessity of both components: IoU filtering ensures diverse visual features, while cosine-based text pruning enhances semantic specificity. Owing and iual-modality refinement, and can
to its flexible design, BiFTA naturally supports both be readily applied to broader prompt-learning frameworks beyond WCA. Importantly, the refinement process is decoupled from downstream methodologics, which enables broad applicability across diverse tasks and settings.
14

---

Published in Transactions on Machine Learning Research (01/2026)
7 Acknowledgements
YHS, CYC and JCZ are supported by the Melbourne Rescarch Scholarship. ZSY is supported by the Australian Research Council (ARC) with grant number DE240101089. FL is supported by the ARC with grant number DF240101089, I.P240100101, DP230101540 and the NSF&CSTRO Responsible AT program with grant number 2303037. This research was supported by The University of Melbourne's Research Computing Services and the Petascale Campus Initiative.
15

---

Published in Transactions on Machine Learning Research (01/2026)
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: a visual language model for few-shot learning. In NewrIPS, 2022.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 mining discriminative components with random forests. In ECCV, 2014.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Veelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In NewrIPS, 2020.
Voes,,
Chengyi Cai, Zesheng Ye, Lei Feng, Jianzhong Qi, and Feng Liu. Bayesian-guided label mapping for visual reprogramming. In NewrIPS, 2024a.
Chengyi Cai, Zesheng Ye, Lei Feng, Jianzhong Qi, and Feng Liu. Sample-specific masks for visual reprogramming- based prompting. In ICML, 2024b.
Chengyi Cai, Zesheng Ye, Lei Feng, Jianzhong Qi, and Feng Liu. Attribute-basee visual reprogramming for image classification with CLIP. arXiv preprint arXiv:2501.13982 2025a.
Chengyi Cai, Zesheng Ye, Lei Feng, Jianzhong Qi, and Feng Liu. Understanding model reprogramming for CLIP via decoupling visual prompts. In ICML, 2025b.
Lincan Cai, Jingxuan Kang, Shuang Li, Wenxuan Ma, Binhui Xie, Zhida Qin, and Jian Liang. From local details to global context: Advancing vision-language models with attention-based selection. In JCML, 2025c.
Chi-Min Chan, Chunpu Xu, Ruibin Yuan, Hongyin Luo, Wei Xue, Yike Guo, and Jie Fu, RQ-RAG: learning to refine queries for retrieval augmented gencration. arXiv preprint 0rXiv:2444.00611 2024.
Shiming Chen, Bowen Duan, Salman H. Khan, and Fahad Shahbaz Khan. Interpretable zero-shot learning with locally-aligned vision-language model. arriv 2506.23822, 2025.
Zhongzhi Chen, Guang Liu, Bo-Wen Zhang, Fulong Ye, Qinghong Yang, and Ledell Wu. Altelip: Altering the language encoder in clip for extended language capabilitics. In ACL (Findings), 2022.
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and-languagee tasks via text generation. In ICML, 2021.
Mircca Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vodaldi. Describing textures in the wild. In CVPR, 2014.
Fangming Cui, Yonggang Zhang, Xuan Wang, Xule Wang, and Liang Xiao. Generalizable prompt learning of clip: A brief overview. arXiv preprint arXiv:2508. 01263, 2025.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In CVPR, 2009.
Jacob Devlin, Meng-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
Zirui Guo, Lianghao Xia, Yanhua Yu, Tu Ao, and Chao Huang. Lightrag: Simple and fast retrieval-augmentod generation. arxiv preprint ar Xrx:224 10. 1000:24110777, 05779, 2024.
16

---

Published in Transactions on Machine Learning Research (01/2026)
Xiangteng He and Yuxin Peng. Fino-grained image classification via combining vision and language. In CVPR, 2017.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021.
Karen Spärck Jones. A statistical interpretation of term specificity and its application in retrieval. J. Documentation, 2004.
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-langee languugg transformer without convolution or region supervision. In ICML, 2021.
Jinhao Li, Haopeng Li, Sarah Monazam Erfani, Lei Feng, James Bailey, and Feng Liu. Visual-text cross alignment: Refining the similarity score in vision-language models. In ICML, 2024.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, 2022a.
Wanhua Li, Xiaoke Huang, Zheng Zhu, Yansong Tang, Xiu Li, Jie Zhou, and Jiwen Lu. Ordinalclip: Learning rank prompts for language-guided ordinal regression. In NewrIPS, 2022b.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACL, 2021.
Weixin Liang, James Zou, and Zhou Yu. ALICE: active learning with contrastive natural language explanations. In EMNLP, 2020.
Shweta Mahajan, Hoang Le, Hyojin Park, Farzad Farhadzadeh, Munawar Hayat, and Fatih Porikli. Attention guided alignment in efficient vision-language models. arXiv 2511. 2771, 2025.
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In ICLR, 2023.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, 2012.
Sarah M. Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification. In ICCV, 2023.
Alec Radford, Karthik Narasimhan, Tim Salimans, Nya Sutskever, et al. Improving language understanding by generative pre-training. OpenAl, 2018
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learnors. OpenAl blog, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Karsten Roth, Jac Myung Kim, A Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata. Waffling around for performance: Visual classification with random words and broad concepts. In ICCV, 2023.
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. In NeurIPS, 2022.
Amanpreet Singh, Ronghang Hu, Vedamuj Goswami, Guillamme Couairon, Wojeicch Galuba, Marcus Rohrbach, and Douwe Kiela. FLAVA: A foundational language and vision alignment model. In CVPR, 2022.
Korawat. Tanwisuth, Shujian Zhang, Huangjie Zheng, Pengcheng He, and Mingyuan Zhou. Pouf: Prompt- oriented unsupervised fine-tuning for large pre-trained models. In ICML, 2023.
Feng Wang, Manling Li, Xudong Lin, Hairong Lv, Alex Schwing, and Heng Ji. Learning to decompose visual features with latent textual prompts. In ICLR, 2022.
17

---

Published in Transactions on Machine Learning Research (01/2026)
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. In TCLR, 2021.
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, Belongie, and P. Perona. Caltech- Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-001, California Institute of Technology, 2010.
Chengyue Wu, Teng Wang, Yixiao Ge, Zeyu Lu, Ruisong Zhou, Ying Shan, and Ping Tuo. x-tuning: Transferring multimodal foundation models with optimal multi-task interpolation. In ICML, 2023.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In CVPR, 2022.
Hongwei Xue, Yupan Huang, Bei Liu, Houwen Peng, Jianlong Fu, Houqiang Li, and Jiebo Luo. Probing inter-modality: Visual parsing with self-attention for vision-and-language pre-training. In NeurIPS, 2021.
Jie Yan, Xiaoyuan Guan, Weishi Zheng, Hao Chen, and Ruixuan Wang. Global and local vision-language alignment for few-shot learning and few-shot OOD delection. In Medical Image Compuling and Compuler Assisted Intervention MICCAI 2025 28th International Conference, Daejeon, South Korea, September 23-27, 2025, Proceedings, Part V, 2025.
Zesheng Ye, Chengyi Cai, Ruijiang Dong, Jianzhong Qi, Lei Feng, Pin- Yu Chen, and Feng Liu. Neural network reprogrammability: A unified theme on model reprogramming, prompt tuning, and prompt instruction. arXiv preprint Xiv:2550 04650, 2025.
Xiaojie Yin, Qilong Wang, and Qinghua Hu. Constrained prompt enhancement for improving zero-shot generalization of vision-language models. arXiv 17417, 2025.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung. Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. TMLR, 2022.
Yue Yu, Wei Ping, Zihan Liu, Boxin Wang, Jiaxuan You, Chao Zhang, Mohammad Shoeybi, and Bryan Catanzaro. Rankrag: Unifying context ranking with retricval-augmented generation in 1lms. In NeurIPS, 2024.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Tuowei Zhou, and Pengchuan Zhang. Florence: A new foundation model for computer vision. arXie preprint arXiv:2111.11432 2021.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. ICCV, 2023.
Bolci Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE PA 2017.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. 1JCV, 2022.
18

---

Published in Transactions on Machine Learning Research (01/2026)
A Appendix 1: Additional Algorithms for the BiFTA Implementation
Algorithm Zero-Shot Classification Pipeline of BiFTA
1: Input Query image lo E IR IR#* x3; labels y E y; hyperparametens: patch queue Q, crop count N, textual description count M, a label prompt T, and pre-trained CLIP model with encoders [ing(:) (image) and f.x(() (text).
Initialize Q < [1] by Eq. 2
# Step 1: View Refinement.
for = 2 to N do
Generate 4 by Eq. 2
5: Check Is Redundant(I) by Algo. 2
6: Is Redundant (4) = false, (.push(I.)
7: Compute wi = softmax(code
(Lo), fing (1:)))
end for Step Description Refinement
for y y do
10: Obtain 7 = (T) where J> M
11:
Remove T, E T based on Algo. 3
12: Obtain 73 C TV from Line (11)
Initialize f,
13:
14: Select top-M T's by arg max cos(fix((T)) fixt (f,))
15: Compute Vy for all {T,}}- MM via Uy softmax(cos(ffrsso (Tyy f fexa (T₃))
16: Compute SCA via Eq.
17: end for
18: Output y" arg yy SCA
Algorithm Implementation of Redundant Image Patch Filtering
1: Input: A patch queue Q contains all the previous saved image patches, a new cropped image patch I, an ToU threshold 7 = 1 8.
2: Initialize Is Redundant = false
for 1= to [Q] do Compute тд loU(4,
if mm then
Is Redundant <- true, break 7: end if 8: end for 9. Output: Is Redundant
Algorithm Implementation of Redundant Textual Description Filtering
Input: A set of textual descriptions of label y; Tv (T)}/=1> where the number of descriptions the merged textual description set; text encoder.
is the tolerance for the duplicate texts, setting 1.0, fext is
2: for j = to J - do
3: for k j + to J do
4: Compute S = cos( (frx (T)), Fixe (Tx))
5: if S then
Remove Tş from T
end if
end for
end for
10: Obtain TV= T
11: Output: Tv
19

---

Published in Transactions on Machine Learning Research (01/2026)
B
Appendix 2: More Experimental Results
Table 11: Zero-shot classification accuracy (%) across various baseline methods with the pre-trained CLIP model (ViT-B/16). We report the averaged results and standard deviations a of 20 runs, with the improvement (%% over the top-performing baseline WCA highlighted in green and red. The results of our method are highlighted and bold to represent the best-performing method.
Method
ImageNet
CUB
Oxford Pets
DTD
Food101
Place365
CLIP
66.74
56.01
88.14
42.98
88.40
39.27
CLIP-E
68.37
56.16
89.10
45.27
88.83
40.30
CLIP-D
68.04
57.08
87.52
46.17
88.85
40.34
Waffle
68.12
56.89
86.51
44.68
89.06
40.76
CuPL
69.61
56.42
91.14
50.53
88.98
39.83
WCA
71.05
59.87
92.13
52.87
89.99
41.33
BiFTA (ours)
71.14:0.004
60.06:10.15
91.67+0.11
54.64+0..16
90.11::0.05
42.12:10.04
42.
A
+0009
+0.19
-0.46
+1.77
+0.12
+00.79
Table 12: Zero-shot classification accuracy (%) across various baseline methods with the ResNet- CL.IP model (RN-50). We report the averaged results and standard deviations of 20 runs, with the improvement A(%) over the top-performing baseline WCA highlighted in green. The results of our method are highlighted and we use bold to represent the best-performing method.
Method
ImageNet
CUB
Oxford Pets
DTD
Food101
Place365
CLIP
58.15
45.67
83.65
38.67
78.62
37.04
CLIP-E
59.82
46.58
85,66
41.22
80.82
37.73
CLIP-D
59.62
17.76
83.70
42.23
79.92
37.13
Waflle
59.82
46.76
83.54
38.88
80.74
37.77
CuPL
61.43
47.91
87.05
47.39
80.50
37.78
WCA
62.82
50.16
88,40
49.45
81.25
38.91
BiFTA (ours)
63.54400.06
50.43:1.117
88.74-0.12
51.41:10.22
81.39-10.09
39.70=10.06
A
+0.72
+0.27
+0.34
+1.96
+0.14
+0.79
Table 13: Zero-shot classification accuracy (%) across various baseline methods with the ResNet- CLIP model (RN-101). We report the averaged results and standard deviations a of 20 runs, with the improvement A (7%) over the top-performing baseline WCA highlighted green and red. The results of our method are highlighted and we use bold to represent the best-performing method.
Method
ImageNet
CUB
Oxford Pets
DTD
Food101
Place365
CLIP
61.26
49.34
84.96
40.05
82.44
36.77
CLIP-E
62.31
49.65
86.97
43.62
83.64
37.81
CLIP-D
60.65
50.29
82.53
42.82
83.25
35.75
Waffle
61.25
48.05
83.70
40.05
82.48
37.83
CuPL
61.43
42.85
87.63
43.83
82.74
35.77
WCA
62.82
44.64
87.43
49.91
83.92
38.11
BiFTA (ours)
65.24100.05
45.8610.12
86.8120.15
50.8710.23
84.02-0.05
39.40=10.06
A
+2.42
+1.22
-0.62
+0.96
+0.10
+1.29
Tables 13 provide the results of utilizing alternative architectures of the CLIP backbone. The results
show that BiFTA consistently outperforms baselines on downstream tasks with the ResNet-based backbone models, which evidently shows that our framework is able to generalize various CLIP style models.
20

---

Published in Transactions on Machine Learning Research (01/2026)
C Appendix 3: RAG-based Text Generation
We further explore alternative prompt design based on ReetrieallAuugmenteer Generation (RAG) to enrich textual descriptions with external knowledge. Specifically, we construct a knowledge database from a pre- processed Wikipedia corpu which contains approximately 1.8 million documents. All documents are truncated, tokenized, and encoded into embedding vectors using the eaxt-enbeddin---da subsequently stored in the ChromaDB vector database for semantic retrieval.
model, and
Due to practical constraints, we utilize subset of approximately 150k documents in our experiments, indexing the entire corpus would require prohibitively long preprocessing time. Given query prompt, documents are retrieved based on cosine similarity between the prompt embedding and document embeddings. The retrieved documents are then concatenated with the prompt template and fed into a GPT model to generate the final textual descriptions.
However, as shown in Figure G this RAG-based description generation strategy yields inferior performance when applied to CLIP-based zero-shot classification. We attribute this behavior to two key factors. First, Wikipedia articles typically contain long and diverse contextual information, while the retrieval queries are short prompt templates, making saatte--mmmlaritybaaaa based retrieval less effective at identifying visually relevant content. Second, and more importantly, Wikipedia articles often focus encyclopedic knowledge- such as historical background, taxonomy, or cultural context rather than fine-grained visual attributes that are critical for visual discrimination. For instance, the Wikipedia entry for a Persian cat primarily discusses breeding history and popularity, but provides limited information about localized visual characteristics.
such
These observations suggest that simply scaling the size of generic Wikipedia-based database is unlikely to substantially improve performance under the current retrieval and aligument framework. Instead, more promising direction lies in designing category-aware attribute-centric RAG strategies that retrieve documents explicitly aligned with patch-level visual semantics. We view such designs as an important avenue for future work, which would require a systematic investigation of suitable external resources and retrieval mechanisms tailored to visual recognition tasks.
D Appendix 4: Extra Ablation Studies
Table 14: The ablation study on a. This experiment is evaluated on the ImageNet dataset by leveraging CLIP (B/32). We set the upper bound B to 0.9 and report the Top-1 Accuracy (%).
8 = 0.9
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
WCA
66.4810.00
66.57-10.77
66.01
66.72-10.07
0600077007
66.8510.04
66.92-0.04 66.61=0.04
66.9311.05 66.55+0.07
Ours
66.9310.06
66.97=00.05
07.05=0.08
67.2440.04
66.83+10.04
66.88+0.04
Table 15: The ablation study on 8. This experiment is evaluated on the ImageNct datasct by leveraging CLIP (B/32). We set the lower bound a to 0.5 and report the Top-1 Accuracy (%).
3
a = 0.5
0.6
0.7
0.8
0.9
WCA
61.77+00.06
63.21+00.5
45.055
66.49+-0.07
66.06+t0.08
Ours
64.40
65.46.10.07
65.9110.05
66.83.10.04
66.73.10.08
Table 14
compare the Top-1 accuracy (%) of the WCA scoring method and BiFTA when varying the cropping size lower bound and upper bound B3. In Table 14 the results reveal significant, performance gap when the crop window range is large (c.g., 0.91, As a increases, image patches cropped with larger
15
21

---

Published in Transactions on Machine Learning Research (01/2026)
window sizes, which reduces the effectiveness of the view filtering under stricter loU thresholds. In Table 15 as B increases, there are more image patches with larger areas. We observe consistent upward trend in accuracy as / increases for both methods. The performance gap gets more pronounced when / is smaller. For example, at = 0.6, BiFTA achieves a 2.6% higher classification accuracy than WCA. This suggests that when the cropping windows are smaller, redundant small patches have a more significant negalive impact on the weighted scores, whereas this issue effectively addressed by integrating the VR introduced in BiFTA.
Based Figure Ö in Section 5.3 provide supplementary results for cach backbone. Figures to present the classification accuracy across all downstream tasks for 4 description sets. Each sub-figure represents the results obtained from CLIP ViT-B/32, ViT-B/16 and ViT-I//4, respectively. In conclusion, the mixed set of descriptions, combining the CuPL and AttrVR descriptions and filtering them based on CoS function and top-k similarities, demonstrates superior performance compared to other description sets.
ImageNet
CUB
Oxford Pets
68
60
90
Accuracy (%)
Accuracy (%)
Accuracy (%)
66
58
85
65
57
64
56
80
RAG
attrVR
CuPL
mix
RAG
att
CuPL
mix
RAG
att
CuPL
mix
DTD
Food 101
Place365
55
42
Accuracy (%)
Accuracy (%)
86
Accuracy (%)
40
85
50
84
38
RAG
attrV
CuPL
mix
RAG
aur
CuPL
mix
RAG
aur
CuPL
mix
Figure 7: Results of exploring textual description studies, using the results of CLIP (B/32). "RAG" computes similarity using RAG-generated descriptions. "attrVR" uses both descriptive and distinctive texts caleulate similarity. "CuPL" directly employs descriptions from the CuPL method. "mix" combines "attrVR" and "CuPL" descriptions.
22

---

Published in Transactions on Machine Learning Research (01/2026)
ImageNet
CUB
Oxford Pets
72
62
Acouracy (%)
71
Accuracy (%)
61
Accuracy (%)
42
60
69
88
48 w
68
8
66
56
RAG
attrVR
CuPL
mix
RAG
attr
CuPL
mix
RAG
attr//
CuPL
mix
DTD
Food101
Place365
68
81
43
66
Accuracy (%)
Accuracy (%)
90
Accuracy (%)
41
29
62
40
60
88
46
86
RAG
attrVR
CuPL
mix
RAG
attr/R
CuPL
mix
RAG
attrVR
CuPL
mix
Figure 8: Results of exploring textual description studies, using the results of CLIP (B/16).
ImageNet
CUB
Oxford Pets
79
68
78
67
Acouracy (%)
Accuracy (%)
Accuracy (%)
04
92 30
76
75
84
*
73
RAG
attrV/
CuPL
mix
RAG
attr/R
CuPL
mix
RAG
attr/R
CuPL
mix
DTD
Food101
Place365
96
44
Accuracy (%)
Accuracy (%))
Accuracy (%)
80
94
42
56
88
41
68
82
52
91
50
90
38
RAG
attrV/
CuPL
mix
RAG
attrVR
CuPL
mix
RAG
altr/R
CuPL
mix
Figure 9: Results of exploring textual description studies, using the results of CLIP (I/144).
23

---

Published in Transactions on Machine Learning Research (01/2026)
E
Appendix 5: Alternative VR strategy: Grid Crop
Table 16: Zero-shot classification performance of WCA with different grid-crop configurations using ViT-B/32 across multiple benchmarks.
WCA (ViT-B/32)
CUB
Food101
DTD
Oxford Pets
ImageNet
Place365
Avg. acc.
Grid crop (3x3)
36.77=0.15
68.09=0.08
46.03=0.20
76.43+0.18
50.67=0.05
33.88=0.05
51.98
Grid crop (4x4)
29.18-0.20
57.21-0.11
43.48-0.28
66. 0.36
43.89:10.05
31.8810.07
45.37
Grid crop (5x5)
21.83=0.23
45.19=0.09
40.19=0.38
56. 0.32
37. 37.65=0.11
29.34=:.09
38.41
Table 17: Zero-shot classification performance of VR with different grid-crop configurations using ViT-B/32 across multiple benchmarks.
VR (VIT-B/32)
CUB
Food101
DTD
Oxford Pets
ImageNet
Place365
Avg. acc.
Grid crop (3x3)
38.5910.22
68. 70.0088
48.06.1..25
77.1310.23
50.92:10.07
35.05:10.06
53.08
Grid crop (4x4)
31.65+0.277
57.88+0.08
44.86+0.25
67.73+0.25
44.17
33.01++0.06
46.55
Grid crop (5x5) 23.65:10.27
16.20:10.18
42.28:10.88
57.33
37.78:10.08
30.17:10.07
39.62
Compared to approaches that rely on CLIP embedding similarity for patch selection, grid cropping is more computationally efficient, and the resulting cropped patches are inherently non-overlapping. As shown in Tables 16 and 17 BiFTTA consistently outperforms WCA under grid-crop- basedd VR settings. Nevertheless, grid cropping itself leads to noticeable performance degradation compared to random cropping, with classification accuracy consistently decreasing as the grid resolution increases. This trend can be attributed to the reduced patch size induced by finer grid partitioning, where smaller patches contain substantially less semantic information and lead to worse cross alignment. similar pattern is also observed in Table [4] Overall, while grid cropping serves as viable alternative VR baseline, these results further demonstrate that BiFTA remains robust and consistently surpasses WCA across diverse VR configurations.
F Appendix 6: Utilizing advanced LLM
Table 18: Comparison using textual descriptions generated by more advanced language model.
Description set
ImageNet (%)
Original GPT-4o
66.83:10.04
66.64410.04
We further investigate the impact of using a stronger language model for description generation by constructing an alternative set of textual descriptions for ImageNet categories using gpt To ensure fair comparison, we adopt five representative prompt templates from CuPL and AttrVR and generate ten description samples per prompt for each class. The corresponding results reported in Table 18]
We observe slight performance degradation compared to the original description set, suggesting that although more advanced language models may produce richer or more detailed descriptions, such improvements do not directly translate into better performance under the current cross- alossaaiggment scoring framework. This result indicates that the effectiveness of description refinement is not solcly determined by the strength of the language model, but also by how well the generated descriptions align with patch-level visual representations. These findings point to an interesting future direction: designing prompt templates that are explicitly tailored to localized visual features, which may further enhance the effectiveness of description refinement.
24

---

Published in Transactions on Machine Learning Research (01/2026)
G Appendix 7: Limitation
One potential limitation we observed is that the textual descriptions generated by the LLM do not consistently focus on local features of the object. These descriptions often tend to be generic, making it difficult to associate specific parts of the text with corresponding local image patches from a human perspective. To address this, we aim to explore more advanced approaches to generate precise and localized descriptive texts that better align with the visual details of an image.
On the other hand, while BiFTA generally benefits from incorporating descriptions, its effectiveness relies the assumption that increased patch diversity improves fine- fine-graine -reaigai cross-modal alignment. In practice, we observe that this assumption may not always hold uniformly across all benchmarks. For instance, on Oxford Pets, BiF'TA sometimes exhibits slight performance drop compared to WCA, suggesting that certain patches filtered as redundant may still contain clas-ddisriminativee information. This observation highlights limitation of the current refinement strategy: it does not explicitly distinguish between truly redundant patches and visually similar yet semantically important ones. Identifying such cases and developing adaptíve refinement mechanisms that better preserve informative visual cues remain important directions for future work.
diverse visual views and textual
25


Reviews:
[{"review_id": "ktxNK5RSOo", "reviewer": "Reviewer_tbw9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary:\n\nThis paper proposes BiFTA (Bi-refinement for Fine-grained Text-visual Alignment), a post-hoc refinement framework built on top of vision–language models (like CLIP) and the recent WCA (Weighted visual-text Cross Alignment) scoring method. The main challenge addressed in the paper is the presence of substantial redundancy in randomly cropped image patches and LLM-generated descriptions, which degrades the quality of the cross-alignment score. BiFTA tackles this by View Refinement (VR) via IoU filtering and Description Refinement (DR) via cosine similarity filtering. The proposed approach is evaluated for zero-shot classification on 6 datasets and several backbones.\n\n\nStrength:\n\n-- The paper presents a well motivated problem formulation with Fig. 1 and Fig. 2 to empirically support this claim.\n\n-- Both VR and DR components are non-parametric and require no additional training: IoU-based filtering and cosine-similarity–based pruning are conceptually simple to understand. It is straightforward to drop into any WCA-like pipeline without retraining the VLM.\n\n-- The evaluations show consistent empirical gains across many datasets, architectures, and ablations.\n\n-- The overall paper is readable and well-organized with helpful figures.\n\nWeakness:\n\n-- The overall average improvements over WCA are very incremental, +0.09 for ImageNet in Table 1, +0.04 for Food101 in Table 2. Also no average accuracy of the datasets are provided in the tables.\n\n-- In Section 4.2, they take the union of CuPL, DesAttr, and DistAttr description sets before filtering. These mentioned sources have different statistical properties and noise levels. Taking the union creates a very heterogeneous pool. The paper neither justify why union is used nor provide an ablation on unions vs individual sources. Figure 6 shows that some sources (e.g., RAG) are harmful, which suggests union is not always beneficial.\n\n-- The hyperparameter sensitivity is insufficiently analyzed. IoU threshold and queue size are studied, but ϵ (cosine threshold) and Top-K in Description Refinement are not ablated. DR is critical to the method, and lack of analysis makes robustness unclear. Tables 6–7 also show that the choice of alternative refinement strategies produces similar results, suggesting that Top-K is the dominant operation, but this is not explored.\n\n-- Since the whole method manipulates the scoring matrix WCA, it is worthwhile to know the runtime overhead over WCA. It is important to check as  filtering requires repeated IoU checks and cosine similarity computation. Given the incremental improvements, it is critical to estimate the worth if the overhead is too high. Further, it is also unclear if similar gains appear for modern prompt-learning or patch-based methods beyond WCA (e.g., Tip-Adapter, RAG-augmented models, visual attribute prompting). This limits the generality of the proposed approach.\n\n-- Regarding VR also, the paper only compares IoU filtering with a CLIP-sim filtering alternative. However, simple deterministic strategies (grid crops, multi-scale crops, or attention-map driven crops) are plausible baselines and could outperform IoU filtering. These were not included.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The core claim of the paper, that removing redundant views and descriptions improves WCA-style zero-shot classification, is supported by reasonably clear and accurate evidence. The experiments show consistent, reproducible gains across multiple datasets and CLIP backbones. Also the ablations and qualitative examples align with the redundancy argument. However, some improvements are modest or occasionally negative. The theoretical analysis mainly justifies redundancy in principle rather than BiFTA’s specific design. Thus, while the evidence supports the main idea, it only moderately supports the stronger claims of broad impact and methodological novelty.", "audience_yn": "Yes", "audience_explanation": "The topic is directly relevant to the TMLR audience working on vision–language models and zero-shot classification. The paper provides a practical refinement technique that improves a widely used CLIP-based scoring method (WCA) and offers insights into redundancy in LLM-generated descriptions and patch-level alignment—issues. Even though the contribution is incremental, the findings would be of interest seeking lightweight, training-free improvements to multimodal alignment pipelines.", "requested_changes": "Alongside the points in the Weaknesses, here are the Queries/Suggestions:\n\n-- Right now BiFTA is tightly coupled to WCA’s patch–description matrix and weighting. Could the same refinement idea be applied to other CLIP enhancement methods (e.g., AttrVR, RAG-based prompt sets) that don’t use the WCA framework?\n\n-- The authors ablate IoU and queue length, but not the ϵ and K for DR. Some sensitivity analysis here would help understand robustness of DR and whether the same hyperparameters work across all datasets.\n\n-- Are there datasets or backbones where view/description refinement clearly degrades performance (more than ±0.5%)? If so, understanding limitations (e.g., classes where most “redundant” patches were actually important) could be insightful.", "broader_impact_concerns": "Nil", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lxUFtalpdD", "reviewer": "Reviewer_bh1B", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies a new method to improve the zero shot classification accuracy of pretrained vision language models. Prior work has suggested to use a diverse set of class description and image patches to estimate the probability that an image is from a certain class. In this work this approach is extended by filtering out duplicates from the image patches and text descriptions by relying on high overlap (Jaccard Index) and high embedding similarity. The paper validates the approach through extensive experiments. \n\nStrenghts:\n\n- Extensive evaluation\n- Method is clearly explained\n\nWeaknesses:\n\n- Improvement is not that significant (in most settings below 1%)\n- Theoretical explanation has problems\n- No runtimes reported", "claims_and_evidence_yn": "No", "claims_explanation": "I am no expert for the empirical setup but the results seem to provide some evidence for their method. However, I was not able to properly understand the theoretical statements in Proposition 3 and 4 and I am not sure whether they can be made rigorous. \n\nFirst, the statement containing $\\approx$ is non-rigorous. But on a more fundamental level it is not clear what the considered probability distribution is and for natural candidates the statement would not be true, e.g., if $I_i$ are i.i.d. samples from the random patches then the probability that $I_i$ and $I_j$ take certain (similar) values is not the same as the probability $I_i$ takes a certain value. I would guess that the actual statement should be that having both views $I_i$ and $I_j$ does not change the probability that the label of the instance is $y$ compared to having only $I_i$. While this seems intuitively reasonable a rigorous statement will still require additional assumptions (some type of continuity) on the CLIP model. For the statement of Proposition 4 similar issues arise. It is not clear to me, whether the Proposition suggests that WCA and BIFTA give rise to the same posterior probabilities but this is not quite clear, because duplication changes the prediction in WCA.\n\nI would suggest to remove the formal claims and only provide a hand-wavy intuition.", "audience_yn": "Yes", "audience_explanation": "People interested in zero-shot prediction might be interested in the findings of the paper.", "requested_changes": "As mentioned above, Proposition 3 and 4 need to be rewritten to be accurate mathematical statements or they should be removed from the paper. \n\nA better understanding of why the methods shows some improvements would strengthen the paper. E.g., at least for the IoU filters the distribution of the patches seems to be quite similar to the WCA approach, so the improvement might not result from a different distribution of image patches but higher diversity for the small sample size of the images. To test this one could compare to larger patch sample size for WCA. \n\nSpacing in formulas should be adjusted in many places.\n\nRuntimes should be reported (might have overlooked this).", "broader_impact_concerns": "N/A", "additional_comments": "Why don't you just use the Wikipedia article of a class name as context to generate text descriptions?", "extra_scores": {}, "extra_text": {}}, {"review_id": "E5PHf3XzeZ", "reviewer": "Reviewer_4Aac", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new method called BiFTA to improve the zero-shot performance of vision–language models (VLMs) such as CLIP. The method consists of two main components: (1) view refinement and (2) description refinement, which remove redundant image patches and redundant or irrelevant textual descriptions, respectively. Experiments on six benchmarks and across different architectures demonstrate the superiority and robustness of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The method is intuitively sound, and the improvement direction is well-motivated. Evaluations across various benchmarks and architectures, along with ablation studies, demonstrate the readiness and effectiveness of the proposed approach.", "audience_yn": "Yes", "audience_explanation": "Although the improvements on some benchmarks are minor, certain readers may be interested in further exploring the boundary of the WCA method—for example, investigating whether its performance could be enhanced by replacing the random cropping strategy with a more structured or deterministic one.", "requested_changes": "As this paper is a resubmission, I notice only a few adjustments:\n\n1. What is the computational efficiency of the proposed method compared with WCA?\n\n2. Is there any ablation study on the patch size and the length of the LLM-generated descriptions?\n\n3. Which LLM is used in the paper to produce the detailed descriptions? Any ablation on the LLM used, e.g. using more advanced LLM to provide better descriptions?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)