kNN-CLIP: Retrieval Enables Training-Free Segmentation on Continually Expanding Large Vocabularies
Authors: Zhongrui Gui
Shuyang Sun
Runjia Li
Jianhao Yuan
Zhaochong An
Karsten Roth
Ameya Prabhu
Philip Torr
Venue: Accepted by TMLR ()

Abstract: Continual segmentation has not yet tackled the challenge of improving open-vocabulary segmentation models with training data for accurate segmentation across large, continually expanding vocabularies. We discover that traditional continual training results in severe catastrophic forgetting, failing to outperform a zero-shot segmentation baseline. We introduce a novel training-free strategy, kNN-CLIP, which augments the model with a database of instance embeddings for semantic and panoptic segmentation that achieves zero forgetting. We demonstrate that kNN-CLIP can adapt to continually growing vocabularies without the need for retraining or large memory costs. kNN-CLIP enables open-vocabulary segmentation methods to expand their vocabularies on any domain with a single pass through the data, while only storing compact embeddings. This approach minimizes both compute and memory costs. kNN-CLIP achieves state-of-the-art performance across large-vocabulary semantic and panoptic segmentation datasets. We hope kNN-CLIP represents a significant step forward in enabling more efficient and adaptable continual segmentation, paving the way for advances in real-world large-vocabulary continual segmentation methods.

Published in Transactions on Machine Learning Research (07/2024)
KNN-CLIP: Retrieval Enables Training-Free Segmentation
on Continually Expanding Large Vocabularies
Zhongrui Gui'
Shuyang Sun' Runjia Li
Jianhao Yuan'
Zhaochong An?
Karsten Roth
Ameya Prabhu
Philip Torr
University of Oxford
*University of Copenhagen
*University of Tübingen
Reviewed on OpenReview:
Abstract
Continual segmentation has not yet tackled the challenge of improving open-vocabulary segmentation models with training data for accurale segmentation across large, continually expanding vocabularies. We discover that traditional continual training results in severe catastrophic forgetting, failing to outperform a zero-shot segmentation baseline. We intro- duce novel training-free strategy, KNN-CLIP, which augments the model with a database of instance embeddings for semantic and panoptic segmentation that achieves zero forgetting. We demonstrate that kNN-CLIP can adapt to contimually growing vocabularies without the need for retraining or large memory costs. kNN-CLIP enables open-vocabulary segmenta- tion methods to expand their vocabularies on any domain with a single pass through the data, while only storing compact embeddings. This approach minimizes both compute and memory costs. kNN-CLIP achieves state-of-the-art performance across large-vocabulary mantic and panoptic segmentation datasets. We hope kNN-CLIP represents a significant step forward in enabling more efficient and adaptable continual segmentation, paving the way for advances in real-world largo-vocabulary continual segmentation methods.
Mac 493 lcal
Caca-CAl
Clessic
232 kcal
Lamborghin Hurocán LP610
Farrar
Gwen Stacy
Miles Morales
Cap Ea Aruticaa
Iran Man
McDensic's 't's fries 444 kcal
Fict-o-Fan Fish 319 kcal
Big Mac 493 lica
Eanth
Repler1b499
Ash Teechum
De Sall
Micheel
Frandinn
Cirttnn
Fevvr Phillps
Rer
Enarge
INCE
EARTH
KEPLER-1649C
Figure 1: We propose kNN-CLIP to continually expand the vocabulary space of segmentation models. Our approach adapts to concept customization and identifying long-tails taiget concepts, a known challenge for CLIP models [Udandarao al. 2024). For concept customization, we build the supporting databasc for each long- tailed concept efficiently using C2C (Prabhu al.) 2023c). We use EntitySeg (Qi et al., 2022) generate class-agnostic masks for entities in the database and KNN-CLIP to label these masks. At inference time, filter the masks from EntitySeg based on confidence thresholds for both the mask and class predictions.
"Equal advising; correspondence Shuyang Sun ikevineundrobots.co ax..tttsss......... ukk and Ameya Prablu (ameya@prabbu.

---

Published in Transactions on Machine Learning Research (07/2024)
1 Introduction
The ultimate goal in image segmentation is to accurately segment any image according to text query across open-ended classes. This task was made more achievable through the rapid advances of vision- language models (VLMs) like CLIP (Radford et al.] 2021b), which when applied to semantic and panoptic segmentation, shows promise in handling a broad vocabulary of visual data (Zhou et al.) 2021 Rao et al.
2022 However, recent state-of-the-ae the-rrt--rrrtt-e open-vocabulary segmentation approaches (Yu et al. 2023 Liti
et al. 2023c) still rely on fine-tuning with datasets containing fine-grained annotations like masks and boxes. Although they excel common categorics on segmentation benchmarks, they fall short of achicving broad-
vocabulary segmentation (Li et al. 2023a Sun el al. 2023b).
We identify that the success of these models on standard datasets is due to fine-tuning specific datasets with detailed labels, such as COCO-Stuff (Caesar et al. 2018), which often overlap with the classes of many other benchmarks with fine-grained annotations. For example, COCO-Stuff and ADE-20K share 73 out of 150 classes. Crucially, removing this overlapping vocabulary results in poor segmentation performance (Sun et al. 2023a). Hence, the effective vocabulary of these open-vocabulary models is limited to COCO-Stuff, i.e. fine-tuning significantly narrows down the vocabulary size of these models, restricting their ability generalize to open-vocabulary of objects and concepts In this work, we answer the question: How do we cost-effectively enhance the performance of these models over continually growing vocabulory space?
First, we investigate continual training of these open-vocabulary models on data with new categories. We consistently find that continual training of these models causes catastrophic forgetting by overwriting past knowledge from updating a limited, poorly understood parameter space. Hence, we investigate how to expand the knowledge of the segmentation model given in-domain data without any Iraining.
Our novel method, KNN-CLIP, simply augments the CLIP segmentation models with a retrieval database that matches images with text descriptions. This database can be updated with new data in a single pass without storing any previous images or requiring training, similar to ACM (Prabhu el al.) 2023a]. Unlike conventional continual learning methods, kNN-CLIP ensures the model retains knowledge of previously encountered data (zcro forgetting), as the distance of a data point to itself in the retrieval database is zero. kNN-CLIP requires a single pass (ouline), is memory--fficient due to only storing features, and expands its vocabulary with minimal computational resources due to no additional training. The dynamic nature of kNN-CLIP is particularly advantageous for segmentation settings where large volumes of new data are collected daily (Gui et al. 2024), as retraining large foundation models frequently is too expensive.
We demonstrate the effectiveness of kNN-CLIP on state-of-the-art open-vocabulary semantic and panoptic segmentation benchmarks with the FC-CLIP (Yu et al. 2023) model. kNN-CLIP achieves notable perfor- mance increases (mIoU) across various challenging datasets: A-847, PC-459, and A-150. Specifically, we see improvements of +2.6, +1.7 and +7.2 points respectively, demonstrating effective segmentation across a continually-growvigg vocabulary space. This improvement comes with minimal computational and memory overheads and no training cost. As shown in Fig D we also qualitatively demonstrrte the effectiveness of our method on tasks beyond current benchmarks, such as concept customization and segmenting long-tailed tailed concepts like celebrities, demonstrating the effectiveness of retrieval for tasks considered challenging for CLIP models [Udandarao et al. 2024h. Overall, the key contributions of our study include:
Dymamically Updating Vocabulary Space: We observe that continual segmentation methods typically underperform compared to zcro-shot open vocabulary models. Therefore, we propose adapting the task of continually integrating new classes in continual learning to open vocabulary segmentation models, offering a more feasible solution for continual segmentation.
Trainéng-Free Continual Vocabulary Expansion: We introduce a novel technique, KNN-CLIP, that continually expands the vocabulary of image segnentatiin models beyond the vocabulary space of CLIP without additional training. This method utilizes a continually expanding support set, recasting the problem of concopt prediction as an image retricval task.
This phenomenon of degradation of open- vocabulary performance after fine-tuning CLIP models, known as concept forget- ting MMukhoti et al. 2023), was studied in classification tasks.
2

---

Published in Transactions on Machine Learning Research (07/2024)
Consistent Improvements Across Diverse Datasets: We present extensive experimental ev- idence of our approach's 's effectiveness, demonstrating substantial improvements in semantic and panoptic segmentation across datasets with long-tailed distributions (A-847, PC-459, A-150).
2 Related Work
aetrieval-Augmenntee Models. In the NLP domain, Retrioval-Angmented Generation (RAG) has been shown as a promising technique for enhancing Large Language Models (LLM) by leveraging external struc-
tured data (Guu et al. 2020; Wu et al., 2022: Borgeaud al.
2022 Liu et al. 2020 Peters et al.) 2019 Yu
ltt al. 2021 Izacard et al. 2022 Ram et al. 2023) Yuan et al.) 2024). The dynamic nature of RAG facilitates
continuous knowlodge updates, enabling models to integrate domain-speciffe information scamlessly. This is
particularly beneficial for knowledge-intensiie lasks (Lewis et al.
2020 Wang et al. 2022 Khandelwal et al.
2019 Shi et al.
2023 Petroni et al. 2023). Inspired by its success in NLP, researchers
are now exploring
the application of RAA in computer vision tasks (Chen et al.
2022a Marino et al. 2021
Roth et al.
2022:
Shen et al. 2022:
Sheynin et al. 2022 Yasunaga et al.) 2022
Blattmann et al., 2022
Chen et al.]
2022b
Liu et al. 2023b; Udandarao et al. 2023 Iscen et al.; 2023
Prabhu et al., 2023c Balazevic et al.
2023
For instance, REACT (Liu et al. 2023b]
presents a methodology aimed retrieving relevant knowledgee
and learning customized visual modules accordingly for specific domains. Meanwhile, SuS-X (Udandarao)
et al.
2023 and C2C (Prabhu et al. 2023c) introduce frameworks that utilize retrieved support sets for far more accurate CLIP classification to open-set domains. Although these two works show promisc in applying retrieval-based methods lo visual perceplion, they did not discuss how to apply such Lechniques successfully in visual segmentation. Hummingbird (Balazevic et al.) 2023) suggests employing a straightforward non- parametric nearest neighbour retrieval as the decoder for dense scene understanding tasks. Morcover, beyond integrating new visual features, RECO (Iscen et al.) (023) demonstrates the efficacy of incorporating text representations by cross-modal fusion alongside original and retrieved embeddings. However, both Hum- mingbird (Balazevic et al. 2023) and RECO (Iscen et al. 2023) require training to better integrate retrieval augmentation into their pipelines; in contrast, our method demands no training efforts while providing faster adaptability and strong performance.
Open Vocabulary Learning in Image Understanding. Benefiting from advancements in vision- language models (Radford et al. 2021a Jia et al., 2021), visual models have shown the potential to overcome the constraints of pre-defined closed-set concepts, leading to more flexible open vocabulary image understand-
ing [Li ct al. 2023b). This capability is particularly significant for dense prediction tasks YYu et al.. 2023
Rao et al.] 2022 Ding et al.] 2022 Chen et al. 2023 Karazija et al.] 2023 Liang et al.] 2023 Li et al.
2022 Xu et al.
2023a b), where approaches like Simeaseline (Xu et al. 2022b) and OVSeg [Lingg et al,
2023 leverage eross-modality supervision from CLIP to align class-agnostic mask proposals with language concopts. Building on these ideas, ODISE (Xu et al. 2023a) and FC-CLIP (Yu et al. 2023) further enhance performance in both semantic and panoptic segmentaa ation by improving visual encoders and mask proposal strategies. Despite these advancements, a critical limitation is the issue of catastrophic forgetting during fino-tuning on concept-restricted dense prediction datasets, which leads to a significant reduction in model vocabulary size [Xu et al.) 2022a Sun et al. 2023a). To mitigate this challenge, training-free free methods such as ReCO (Shin et al. 2022) and CaRt (Sun et al. 2023a) have been proposed to enable efficient adaptation of
CLIP models through recurrent pruning of the label space. While these methods avoid catastrophic forget-
ting, they still struggle to accommodate continually expanding vocabularies (Wang et al. 2023b Lüddeecke
& Fecke 2022: Xie al. 2023: Sun et al. 2002bb Seifi et al. 2024).. Therefore, in this work, we explore the use of retricval augmentation to enhance continual open vocabulary image undcrstanding.
Continual Learning for Semantic Segmentation. Unlike open vocabulary semantic segmentation, continual learning does not aim to initially encompass an extremely large vocabulary space but instead focuses on preserving the ability to expand this space continuuusly. However, continual learning is challenged by issues such as catastr forget and semantic drift. To address these problems, iCaRL (Rebulli et al. 2017) suggests replaying the most representative samples during the continual learning stage, and subsequent works have sought to optimize the associated memory burden (Cha et al. 2021 Zhu et al. [0233 Wang et al., 2023a). The use of web images for replay in a similar manner has also been explored (Liu al. 2023a) Additionally, ALIFF. (Oh et al. 2022) introduces feature-replay scheme to reduce memory requirements.
3

---

Published in Transactions on Machine Learning Research (07/2024)
Feature Extructor
Pooling
cat"
feature
"cat"
"cat"
embedding
Pooling
Feature Extractor
"dog"
ddog
feature
"dos"
embedding
Continually Expanding Database
Feature $
Pooling
Extructor
"Ilam" feature
"Ilama"
"Ilama"
embedding
Figure 2: Dynamic Database Construction. A key benefit of our methodology that it allows for the seamless integration of embeddings for new classes into our database, continuously expanding its vocabulary space.
Other research has delved into leveraging memorized features to preserve known knowledge (Yoon et al,
2022 Yu al. 2020 Wang al. 2021 Michieli & Zanuttigh, 2021 Douillard al.] 2021]. While the
of memorized features shows promise, significant limitation is that the representation capacity of these features might be insufficient, which can hinder the performance of continual learning (Yuan & Zhao 2023) Approaches such as employing hyper-class knowledge (Shi et al. 2022) or dynamically updating stored features [Liu et al.] 2022: Lin et al., 2022) attempt to mitigate this issue but do not fully circumvent it. Our method, which aims to expand the vocabulary space dynamically, resembles feature-replay methods as we also construct support of features. However, our approach differs by operating without the need for additional training, utilizing support set that stores powerful features learned through self-supervised learning techniques (Caron et al. 2021), with minimal memory requirements.
3 KNN-CLIP: Continually Expanding Retrieval-Augmente Dense Prediction
We introduce a novel, training-free framework for continual dense prediction with expanding vocabularios, applicable diverse domains and across dense prediction tasks. Inspired by the Retrieval -ugrieval-Augmentee Gen- eration (RAG) method used in Large Language Models, our framework leverages a customizable embedding database to incorporate domain-specific knowledge directly at inference, with additional training.
This approach is applicable to a range of dense prediction tasks including semantic and panoptic: image segmentation. We aim to augment existing open-vocabulary segmentation models. These models, including FC-CLIP Yu et al.) 2023) which we use as our baseline, usually have mask proposer trained to generate class-agnostic masks from extracted features. Then, the image CLIP features are pooled using these proposed masks and used to compute similarities with the text CLIP features class names for classification. Finally, mask predictions and their corresponding class predictions are combined give the result segmentation masks. We first evaluate the confidence level of CLIP classification results for each query mask. For queries falling below given confidence threshold, query embeddings. These embeddings are then matched against vectorized database using kNN scarch algorithm based on cosine similarities. For the most similar embeddings found, we create a set of confidence- aware pseudo-logits by stacking similarities across class vectors, guided by the labels of these embeddings. The results from this retrieval process are then combined with the initial CLIP results through given weighting parameter X. We now provide details of our training- free method. We first describe the embedding construction process and detail the query mechanisms.
extract DINO features and use mask pooling to generate
4

---

Published in Transactions on Machine Learning Research (07 (07/2024)
input image
Pooling
Feature Extractor
"dog":0.9 *cat":0.1
T-0.5 2-1
"dog":0.4 #cat":0.66
"dog":0.9 *ca6":0.6
Mask Proposer
Menging
base prediction
augmented prediction
Figure 3: Retrieval Augmentation from the Database. By retrieving similar features from the database, we integrate the retrieved information with our previous prodiction.
3.1 Designing Continually Expanding Embedding Databases
Database Construction. To obtaín distinctive vectorized representations for the database images, we employ mask-pooling features extracted by pretrained encoders from the images shown in Fig 2 Our input consists of an incoming image (æ; E R3xhx=) with its corresponding k class-agnostic masks, which could be semantic, instance dense masks, or bounding box masks (M; E Rkxhx=) and class annotations (C;) containing the class {cyy corresponding to mask (m. 1 = M,. We extract features of this image using a Vision Transformer (ViT, Dosovitskiy et al. (2021) )- b-asedd encoder (E), which pretrained DINOv2 Oquab al. 2023) this case, yielding h; = E(r)). Here, E Rdx xw" with d representing the embedding dimension of the feature, and h', w' represent the dimensions of the feature map. The embedding dimensions h', w/ are determined by image size divided by the patch size of the vision transformer, thus divided by 14 (the patch size of ViT-G) each, and we reduce the space requirements by storing small feature maps rather than input images. The ground-truth masks M; are resized to M E Rkxh xw" to align with the feature map dimensions. By applying mask average pooling along H and w', the final embedding sets {eig) ( on image 2, is obtained by:
En Eno (hy m/y) End mly Jj=1s where where Eg E Rkd
E, = (ei
(1)
Then, the vectorized embedding database is constructed by concatenating the embedding sets and class sets from all images, so that Dvec = {E, ) fer, cr - if there are n' embeddings in total. New embeddings can also be continually added to the database repeating the above process. Note that we do not store any images, enabling low storage costs and allowing deletion of previously seen data.
One appealing characteristic of our framework is the dynamic extendability of the database and the cus- tomization of the database construction based personalized interests. Depending on the purpose specified by the user-defined applications (fine-grained settings such as smart retail, species classification, etc.), the database could be sourced from various domains through manual curation, online web-based collection, synthetic data sources.
3.2 Inference Using the Continually Expanding Embedding Databases
The embedding database established serves as the retrieval source during inference to enhance the capabilities for various downstream visual models, as shown in Fig Given a test image along with a set of query mask proposals M, x and corresponding feature embeddings acquired using DINOv2 (Oquab et al. 2023) through the same method outlined in See S.13.1 we conduct a k-Nearest neighbour (kNN) search for each query eg within the embedding database Decc as follows:
kNN*(e,)
e Drecs Ssortedl: k]);== 33 ee)
(2)
||
5

---

Published in Transactions on Machine Learning Research (07 (224)
prediction
image
mask decoder
pixel decoder
mask prediction
based CLIP
mask pooling
class embeddings
In-voceb class probabilites
text embeddings
late fusion
prediction
DINOV2
mask pooling
query embeddings
retrieval augmented cless probsbilttes
retrieval dalabase
in-vocab branch
out-vocab branch (retrieval augmented)
Figure 4: Augmenting FC-CLIP. We integrate the retrieval augmentation module to the state-of-the-art segmentation model, FC-CLIP. FC-CLIP includes an in-vocabulary branch and out-of-vocabulary branch. We don't shown the original the out-of-vocabulary branch using DINOv2 features and retrieved information.
branch here for simplicity. We use kNN-CLIP to augment
We then utilize the retrieved sample class labels {ey} and cosine similarity scores {sy}* to construct confidence-aware pseudo-logits Prot RC for retrieval-based prediction as follows:
Σ (8: I[es = -)) +C) D erp 1 (s; I[c, 1)) +
eap
Pret
Vj E
(3)
where we accumulate the class-wise similarity over the class vector and normalize it into a probability distribution through soft operation. We consider the higher similarity between instance embeddings with more frequent retrieval reflects higher confidence in the kNN-based prediction. We then leverage the prediction logits to augment the raw prodiction logits Prase E R from CLIP through class-wise logit modification follows:
Pfinal XPret Pret" (1 T(maar ((maar(Poase)) Paase) > T)) + Poase T[maar( (Posse) > T)
where we set a confidence threshold on raw prediction with the logit lower than replaced by the corresponding value in retrieval-based prediction.
Our construction of pseudo-logits using a weighting based cosine similarity only enhances the downstream performance tasks classes whose labels are stored in the embedding database. If the true label of query is not in the database, or if the cosine similarity of a retrieved result is too low, our method will naturally prefer the original predictions. This preserves the broad zero-shot capabilitics and the knowledge gained during pretraining, which are adept al making accurate predictions for concepts they were trained on (Udandarao et al. 2024). This flexible design enables us to dynamically expand to larger vocabularies and effectively addresses the recently highlighted issuc of concept forgetting (Mukhoti et al. 2023).
3.3 Augmenting FC-CLIP
We integrate methodology into the current state-of-the-ar open-vocabulary segmentation model, FC- CLIP (Yu et al. 2023). which contains three main components. The model includes mask generator, in-vocabulary classifier, and an out-of-vocabulant classifier. All these components are built on top of a shared CLIP backbone. The pixel decoder and mask decoder generate class-agnostic masks, following Fask2Former (Cheng et al. 2022). Then, by mask-pooling the pixel features from the pixel decoder using the class-agnostic masks proposed, the in-vocabulary classifier gives the in-domain class embeddings; by mask pookino the CLIP features, the out-vocabulary classifier yields its class embeddings. The results from both classifiers then fused to give the final prediction.

---

Published in Transactions on Machine Learning Research (07/2024)
We use retrieval augmentation module to enhance the performance of the out-of-vocabulan classifier of FC-CLIP, which is the main reason for the open-vocabulary capability of FC-CLIP. The original out-of- vocabulary classifier leverages the cross-modal CLIP model for open-vocabulary classification. As in Fig 4 to enable retricval augmentation, we additionally pass the imput image into a frozen pretrained DINOv2. We then conduct mask-pooling on the DINO feat and obtain our query embeddings as See 3.1
We then
compute the retrieval augmented class probabilities as described in See 3.2 We fuse the retrieval augmented class probabilities with the original oul-of-vocabulary class proba derived from CLIP linearly in classifier combined with the results of the in-vocabulary classifier to give final class prediction, following FC-CLIP. We then derive the segmentation masks from mask predictions and class predictions following Mask2Former and FC-CLIP.
Eq. Z Then, the retrieval augmented results of the out-of-
4 Experiments
We present the results of our training-free approach, aimed at improving open-vocabulary dense predic- tion for large-scale datascts, including semantic and panoptic segmentation, testing continually expanding vocabularies customized contexts.
4.1
Implementation Details
Database Construction. Our database created by extracting features from each dataset's training sets using DINOv2 (Oquab et al., 2023) with a ViT-Giant architecture, featuring register tokens (Darcet al., 2023 We select "keys" our feature representation. We resize the images to 518 518 and acquire an image feature of dimensions 1536 x 37 x 37, as the patch size of chosen ViT is 14. We then conduct mask average pooling to shrink the feature dimension to 1536. Then, we store the shrunken feature with a dimension of 1536 and its corresponding label e into the database using FAISS. On average, a stored feature occupies 6kB of space. Note that we do not store past images alongside.
k-Nearest neighbours Search. We employ FAISS for feature retrieval using the cosine similarity met- ric. Our method uses brute force approach for identifying the closest embeddings, but we also explore approximate search methods like Hierarchical Navigable Small Worlds (HNSW) to improve efficiency at in- ference time (Prabhu et al. 2023a). We shall compare HNSW approximation with exact-search and detail the tradeofls in our ablations. The number of nearest neighbours retrieved is 16 across dense visual lasks, with further details in the ablation section.
4.2 Catastrophic Forgetting Restricts Open- Vocabulayy Capabilities of Models
Motivation. We study how the open-vocabulary performance of dense predictors varies as these models are trained to recognize new classes. Specifically, we compare their segmentation performance before and after training and report performance deterioration.
Setup. We adopt incremencre continual learning setting adopted from CoMFormer (Cermelli et al, 2023). We benchmark this on the semantic segmentation benchmark of COCO Panoptic as base dataset (having 330K images and 80 classes) and ADE20K (Zhou et al.) 2019) as a continual dataset (having 27K images and 150 classes). We start from the model weights of FC-CLIP which was trained on COCO Panoptic, and then incrementally update the base model on ADE20K, with 5, and, 30 classes per timestep and a computational budget of one epoch (Prabhu et al., 2023b a). We follow Fask22ormer (Chong et al.] (022) to use their loss function on an FC-CLIP (Yu et al. 2023) model and evaluate mloU across all classes in COCO Panoptic and ADE20K respectively after continual training.
Results. As shown in Table I when using naive continual lcarning bascline, regardless of the number of incremental classes, we observe consistent performance degradation of around 6 mIoU and 24 mloU on the continual learning dataset ADE20K and the base dataset COCO-Panoptic respectively. The degradation on ADE20K can be attributable to catastrophic forgetting when learning across timesteps. Moreover, this causes concept forgetting, i.c., degradation of the spen-vocabulay performance by a large margin, as measured by performance on COCO-Panoptic This indicates that there is pressing need for techniques to allow
7

---

Published in Transactions on Machine Learning Research (07/2024)
Table 1: Vocabulary Narrowing FC-CLIP. We observe that further training FC-CLIP within a compu- tational budget leads to significantly poorer performance on the downstream ADE20K dataset, while our retrioval augmentation method circumvents the problem in compute efficient manner. We also observe significant. vocabulary narrowing in COCO Panoplic dataset, which serves as the original training set of
FC-CLIP.
Method
Base Dataset
Continual Dataset
Incremental Class
COCO Panoptic
ADE20K
FC-CLIP Yu et al.
2023
COCO Panoptic
63.7
34.1
COCO Panoptic
ADE20K
43.0 (-20.7)
28.9 (-5.2)
FC-CLIP Yu et al.
2023
COCO Panoptic
ADE20K
10
38.9 (-24.8)
27.7 (-6.4)
COCO Panoptic
ADE20K
30
39.6 (-24.1)
26.3 (-7.8)
KNN-FC-CLIP (Ours)
COCO Panoptic
ADE20K
65.5
(+1.8)
41.3 (+7.2)
Table 2: Comparisons with Continual Learning. Retrieval augmentation circunvents catastrophic
forgetting occuring in continual learning.
Method
Base Dataset
Continual Dataset
Incremental Class
ADE20K (mloU)
FC-CLIP Yu et al.
2023
COCO Panoptic
34.1
COCO Panoptic
ADE20K (150 classes)
28.9 (-5.2)
FC-CLIP
Yu et al.
2023
COCO Panoptic
ADE20K (150 classes)
10
27.7 (-6.4)
COCO Panoptic
ADE20K (150 classes)
30
26.3 (-7.8)
ADE20K (100 classes)
ADE20K (50 classes)
5
26.3
2armer
Cheng et al.
2022
ADE20K (100 classes)
ADE20K (50 classes)
10
31.0
ADE20K (100 classes)
ADE20K (50 classes)
50
38.1
ADE20K (100 classes)
ADE20K (50 classes)
5
30.9
Former Cermelli al.
2023
ADR20K (100 classes)
ADE20K (50 classes)
32.3
ADE20K (100 classes)
ADE20K (50 classes)
50
38.4
KNN-FC-CLIP (Ours)
COCO Panoptic
ADE20K
41.3 (47.22)
segmentation models to continually expand their vocabulary across novel concepts without losing their open-vocabulary segmentation capabilities. We show that our proposed training-free method kNN-CLIP allows effective alleviation of concept forgetting leading to significant performance improvement of +7.2 mIoU in ADE20K and +1.8 mIoU in COCO Panoptic.
4.3 Comparison with Continual Segmentation Approaches
Sctup. We compare our method under the setting of continual learning against the popular supervised method, Mask2Former (Cheng et al. 2022) and the current state-of-the-arl, continual learning method, CoMFormer (Cermelli et al. 2023). We use the same experimental protocol as previously described for our method. These supervised models were initially trained on the first 100 classes and subsequently updated incrementally with the next 50 classes. We then evaluated the performance of these models using the mean Intersection Union (mIoU) metric across all 150 classes from the ADE20K dataset.
Results. As shown in Table 2 Mask2Former (Cheng et al. 2022 and CoMFormer (Cermelli et al. 2023 exhibit large performance degradation when we reduce the number of incremental classes within 50 classes. In contrast, kNN-CLIP demonstrates a substantial advantage in preserving previous knowledge and mitigating knowledge loss, and it is inherently unaffocted by the number of incremental classos. This resilience is attributed lo our dynamic and continuously expanding embedding database, which significantly enhances our method's effectiveness in continual learning scenarios.
4.4 Retrieval Enhances Panoptic Segmentation
Setup. Our exploration extends to panoptic segmentation, further validating the discernability of instance- level open-vocabulary recognition. We evaluate our method ADF20K and COCO Panoptic datasets. To
8

---

Published in Transactions on Machine Learning Research (07/2024)
Table 3: Open-vocabulary panoptic segmentation performance. Retrieval augmentation showcases
improvements in panoptic segmentation.
ADE20K
COCO Panoptic
Method
PQ
AP
mloU
PQ
AP
mloU
ODISE [uu et al. 2023a
22.6
14.4
29.9
55.4
46.0
65.2
ODISE (
23.4
13.9
28.7
45.6
38.4
52.4
FC-CLIP (baseline)
26.8
16.8
34.1
54.4
44.6
63.7
KNN-FC-CLIP(Our)
29.6
17.5
41.3
54.8
44.8
65.5
A w.r.t baseline
+2.8
+0.7
+7.2
+0.4
+0.2
+1.8
Table 4: Open-vocabulary semantic segmentation performance. Retrieval augmentation demon-
strates state-of-the-ard performances on open-vocabulary semantic segmentation
Method
Training Dataset
A-847
PC-459
A-150
PC-59
PC-21
SimBaseline Hsia et al.
2022
COCO Stuff
15.3
74.5
ZegFormer (Ding al.
2122
COCO Stuff
16.4
73.3
L Li et al. 2022
COCO Stuff
3.8
7.8
18.0
46.5
OVSeg
Liang
2023
COCO Stuff
9.0
12.4
29.6
55.7
SAN Xu et al.
20235
COCO Stuff
13.7
17.1
33.3
60.2
OpenSeg Ghiasi et al.
2022
COCO Panoptic + COCO Caption
6.3
9.0
21.1
42.1
ODISE (caption) (uu et al.
2023a
COCO Panoptic + COCO Caption
11.0
13.8
28.7
55.3
82.7
MaskCLIP Zhou et al.
2021
COCO Panoptic
8.2
10.0
23.7
45.9
ODISE (u et
2023a
COCO Panoptic
11.1
14.5
57.3
84.6
FC-CLIP(baseling ) Yu et al.
2023
COCO Panoptic
14.8
18.2
34.1
58.4
81.8
(NN-FC-CCIPPours) A w.r.l baseline
COCO Panoplic
17.4
19.9
41.3
62.8
85.3
+22.6
+11.7
+7..2
+-4.4
+.3.5
assess the performance comprehensively, we employ three critical metrics: Panoptic Quality (PQ), Average Precision (AP), and mean Intersection over Union (mloU).
Results. As detailed in Table retrieving information from additional support sets dramatically increases performance on ADE20K, with PQ, AP, and mIoU rising by +2.8, +0.77, +7.2 respectively without any training, Overall, the performance of open-vocabulary panoptic segmentation can be significantly boosted without any training.
Improving Base Dataset Performance. We highlight that we observed improvements of +0.4 in PQ, +0.2 in AP, and +1.8 mIoU for COCO Panoptic dataset itself. This highlights that using retrieval even from the baseline model's training dataset, in this case, COCO Panoptic, significantly enhances segmentation accuracy. Our method complements advances in open-vocabulary panoptic segmentation tasks, with these results demonstrating the consistent improvements achieved.
4.5 Retrieval Enhances Semantic Segmentation
Setup. Our study extends studying the impact of training-free continual vocabulary expansion of kNN-CLIP to semantic segmentation, testing its efficacy across dense prediction tasks. We analyze the performance of KNN-CLIP over five diverse datasets: ADE20K and A847 (Zhou et al. 2201),, containing 27K images, Pascal Context(PC)-59/ -994/99 (Mottaghi et al. 2014 and Pascal VOC-21 (Everingham et al. 2010) containing 10K images. These datasets include pixel-level annotation on various granularity covering a wide-range of semantic concepts from 150, 847, 21, 59, and 459 classes, respectively. We use the same FC-CLIP backbone in Sec 4.1 For all benchmarks, leverage the mloU metrie evaluate segmentation performance.
SOur method aims continually enhance the performance of open-vocabulary segmentation models. For evaluation, the training each dataset, which is unseen other methods, construct embedding database. Our approach significantly improves performance of zero-shot baselines continual learning setting.

---

Published in Transactions on Machine Learning Research (07/2024)
Table 5: Comparisons with Hummingbird. Our method adopts similar nearest neighbors retrieval
module as Hummingbird but shows significant improvements.
Dataset
Hummingbird Hummingbird++ kNN-CLIP (Ours)
PASCAL VOC
76.9
77.3
85.3 (+8.0)
ADE20K
35.0
35.8
41.3 (+5.5)
Table 6: (Left) Confidence Threshold and Confidence Weighting Ablations. Comparisons of dif- ferent choices of iypee-parameters on A-847. (Right) Number of Retrieved Nearest Neighbors Ab- lations. Comparisons of different choices of number of retrieved nearest neighbors on A-847 with T 0.7 and À 1.2.
No. Neighbors (k)
Confidence Weighting
Confidence Threshold
ADE-847 (mIoU)
14.79
Number of Neighbors
mloU
16
1.0
0.65
17.28
14.79 (baseline result)
16
1.2
0.65
17.27
13.77
16
1.4
0.65
17.33
4
14.84
16
1.0
0.7
17.43
16.60
16
1.2
0.7
17.45
16
17.45
16
1.4
0.7
17.42
17.24
16
1.0
0.75
17.35
64
16.64
16
1.2
0.75
17.42
1.4
0.75
17.41
16
1.0
0.85
17.25
Results. The effectiveness of our method clearly demonstrated in Table showcasing significant im- provements across various benchmarks. When compared the bascline FC-CLIP model, we boost mIoU by +2.6, +1.7, +7.2, +4.4, +3.5, across A-847, PC-459, A-150, PC-59, and PC-21, respectively. With retrieval augmentation, the performance long-tailed datasets, such as A-847 and PC-459, is generally improved. Our method aims to complement advances in open-vocabulary semantic segmentation and these results underscore the robustness and adaptability of our approach handling complex segmentation tasks.
4.6 Comparisons with Retrieval-based Approaches
Comparison. We also compare our results with Hummingbird (Balazevic et al. 2023 which proposes novel pretraining framework that is suitable for fast downstream adaptions. Hummingbird employs nearest neighbour retrieval for predicting the classes of patches in target images, while we use kNN to augment the prediction of each proposed class agnostic mask. We compare our semantic segmentation results with Hummingbird's best-performing model on Pascal VOC and ADE20K. The results of both methods are derived by performing nearest. neighbour retrieval from the evaluated downstream datasets without any further training.
Results. As shown in Table
5 our method outperforms the best Hummingbird model with an increase of +8.0 and +5.5 mloU in PASCAL VOC and ADE20K respectively. The result demonstrates that predictions based on class agnostic masks are more robust, and the information in image patches might not be sufficient for nearest neighbour retrieval. The result. further showcases the strong performance and fast adaptability of our straightforward method over other retrieval- bassed approaches.
4.7 Ablations
Effect Confidence Threshold and Confidence Weightings. We examine the impact of varying hyper-parameter settings on performance, focusing on two key parameters: the confidence threshold and the retrieved confidence weightings. The selection of the confidence threshold is primarily inffuenced by the accuracy of the kNN retrieval module across different datasets. Our experiments carried out the A-847
10

---

Published in Transactions on Machine Learning Research (07/2024)
Table 7: Ablations on Approximate Nearest Neighbors Search. Comparisons of Inference FPS using Brute Force KNN and HNSW built with efConstruction 100 and efSearch 64 for COCO and ADE20K datasets. All results were obtained using single A40 GPU with CUDA 11.8 and PyTorchh 2.0.0. The average
runtime was calculated across the entire validation set and includes post-processing time.
ADE20K
COCO
method
FPS
mloU
FPS
mloU
ODISE
0.41
29.9
0.39
65.2
FC-CLIP
2.86
34.1
2.84
63.7
RA-FC-CLIP (Brute Force)
1.31
41.3
(+7.2)
0.83
65.5 (+1.8)
RA-FC-CLIP(HNSW)
1.52 (x1.16)
34.2 (+0.1)
1.51 (x1.82)
64.5 (+0.8)
dataset. As shown in Table G (Left), the optimal performance, measured in terms of the highest mIoU, is achieved when the confidence threshold, denoted as T in Sec 3.2 is set to 0.7, and the confidence weighting, represented by in the same section, is set at 1.2.
Effect of Numbers of neighbours Retrieved. In dense visual tasks, such as semantic segmentation, our approach is enhance the pseudo-logits for classification of each query mask by incorporating information from the 16 closest neighbours. Additionally, we conduct experiments to examine the impact of varying the number of nearest neighbours retrieved. The results are shown in Table 6 (Right). These experiments are carried out using the previously determined optimal hyper-parameters, where T is set to 0.7 and X is set to 1.2.
Approximate Nearest Neighbour Search. To optimize our inference time, we explore approximate nearest neighbour search algorithms reduce runtime. We apply Hicrarchical Navigable Small World to our feature retrieval module. As expected, we observe trade-off between inference time and model performance. As in Table 7 it can be seen that adopting HNSW leads to faster runtime but makes compromises with mloU. With the medium size of the support set, such as ADE20K including approximately 220k features, brute force search is preferred; if the size of the support set is large, as for COCO, which has 860k features, approximate nearest neighbour search offers a better balance between inference time and accuracy.
Inference Time. We assess the inference time based on our implementation of FC-CLIP, taking into account only the time required infer an image. The retrieval database is preconstructed, ensuring that our evaluation focuses solely on the inference cost. By integrating an additional feature extractor, DINOv2, into the existing architecture, we observe compromise in inference speed shown in Table Furthermore, the kNN search module contributes to the slower inference time due to our utilization of brute force search. We observe that inference speed can be enhanced by implementing approximate neighbour search techniques as alternatives to brute force scarch, but approximate neighbour scarch leads to a slightly worse performance.
5 Conclusion
In our study, we examine the constraints related to the vocabulary used in earlier open-vocabulary meth- ods that rely on CLIP, and we also highlight how performance diminishes when Vision- Languug Models (VLMs) are fine-tuned using downstream annotations. Instead of adopting continual learning approaches, we introduce a novel method called KNN-CLIP, which does not require training and utilizes nearest neighbour retrieval to predict dense visual tasks. To accommodate ever-growing vocabularies, we employ a dynamic retrieval database, circumventing the need for retraining. Our approach demonstrates strong performance across several datasets and showcases significant improvements over other methods.
Broader Impact. We aim to enable dense prediction models across a large and expanding vocabulary of classes However, siguificant advancements are required before deployment in critical applications like self- driving cars or medical imaging, pending thorough testing and validation. Additionally, to prevent misuse, we explicitly prohibit our approach's 's use in military or surveillance contexts.
11

---

Published in Transactions on Machine Learning Research (07 (224))
References
Ivana Balazevic, David Steiner, Nikhil Parthasarathy, Relja Arandjelovié, and Olivier Henaff. Towards in-context scene understanding. Advances in Neural Information Processing Systems, 2023.
Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Björn Ommer. Rotrieval-angmented diffusion models. Advances in Neural Information Processing Systems, 14115309-153222 2022.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pp. 2206 2240. PMLR, 2022.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 1209-1218, 2018.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-superiseed vision transformers. In Proceedings of the IEEE/CVP international conference on computer vision, pp. 9650 9660, 2021.
Fabio Cermelli, Matthieu Cord, and Arthur Douillard. Comformer: Continual learning in semantic and panoptic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3010-3020, 2023.
Sungmin Cha, YoungJoon Yoo, Taesup Moon, et al. Ssul: Semantic segmentation with unknown label for excmplar-bass basod class-inermmental learning. Advances neural information processing systems, 34: 10919-10930) 2021.
Jun Chen, Deyao Zhu, Guocheng Qian, Bernard Ghanem, Zhicheng Yan, Chenchen Zhu, Fanyi Xiao, Mo- hamed Elhoseiny, and Sean Chang Culatana. Exploring voca-vocayylary semantic segmentation without human labels. arXiv preprint arXiv:2306.0044 00450, 2023.
Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, and William W Cohen. Murag: Multimodal retrieval- augmented generator for open question answering over images and text. aie preprint 2rXi::2110.02928. 2022a.
Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W Cohen. Re-imagen: Retrieval-angmeened text-to-image generator. arXiv preprint arXiv:2209.1444 2022b.
Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexander Kirillov, Rohit Girdhar, and Alexander G Schwing. Mask2former for video instance segnentation. In CVPR, 2022.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint arXiw:2309.16588 2023.
Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11583-151111 -15922, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa, Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021.
Arthur Douillard, Yifu Chen, Arnaud Dapogny, and Matthieu Cord. Plop: Learning without forgetting for continual semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition (CVPR), pp. 4040 40500, June 2021.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voe) challenge. International journal of computer vision, 88:303 338, 2010.
12

---

Published in Transactions on Machine Learning Research (07/2024)
Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Y Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. In European Conference on Computer Vision, pp. 540-557. Springer, 2022.
Shengxi Gui, Shuang Song, Rongjun Qin, and Yang Tang. Remote sensing object detection in the deep lcarning a review. Remote Sensing. 16(2), 2024. ISSN 2072-4292 doi: 10.33900/1166220327. URL
https://wae http:///wwwwpp.....// 2omm/2072-492211//222
Kelvin Guu, Kenton Lee, Zora Tung, Pamppong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929 3938. PMLR, 2020.
Hsuan-An Hsia, Che-Hsien Lin, Bo-Han Kung, Jhao-Ting Chen, Danicl Stanley Tan, Jun-Cheng Chen, and Kai-Lung Hua. Clipcam: A simple baseline for zero-shot text-guided object and action localization. ICASSP 2022-2022 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4453 4457. IEEE, 2022.
Ahmet Iscen, Mathilde Caron, Alireza Fathi, and Cordelia Schmid. Retrieval-enhanced contrastive vision- text models. arXiv preprint Xiv:22300 07196, 2023.
Gautier Izacard, Patrick Lewis, Maria. Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning with retrieval augmented language models. orXiv preprinl ar Xrxiv:2008. f0::220000399, 2022.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp. 4904 4916. PMLR, 2021.
Laurynas Karazija, Tro Laina, Andrea Vedaldi, and Christian Rupprecht. Diffusion models for zero-shot open-vocabulary segmentation. arXiv preprint wxXiv22306009114 2023.
Urvashi Khandolwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Goneralization through memorization: Nearest neighbor language models. arXiv preprint Xrxiv:1911.00 2019.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrievallaugmmened generation for knowledge- intensive ulp tasks. Advances in Neural Information Processing Systems, 33:9459 9474, 2020.
Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language-driven semantic segmentation. In International Conference on Learning Representations, 2022.
Feng Li, Hao Zhang, Huaizhe Xu, Shilong Liu, Lei Zhang, Lionel M Ni, and Heung- Yeung Shum. Mask dino: Towards unified transformer-basee framework for object detection and segmentation. In Proceedings of the TEEF/CVE Conference on Computer Vision and Pallern Recognilion, pp. 3041-3050, 2023a.
Runjia Li, Shuyang Sun, Mohamed Elhoseiny, and Philip Torr. Oxfordtvg-hic. Can machine make humorous captions from images? In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 20293 20303, 2023b.
Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculeson. Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogrition, pp. 7061 7070, 2023.
Zihan Lin, Zilei Wang, and Yixin Zhang. Contimual semantic segmentation via structure preserving and projected feature alignment. Furopenn Conference on Computer Vision, pp. 361- Springer, 2022.
Chang Liu, Giulia Rizzoli, Francesco Barbato, Umberto Michieli, Yi Niu, and Pictro Zanuttigh. Re- call+: Adversarial web-based replay for continual learning in semantic segmentation. arXio preprint 2Xii:2309.10477 2023a.
13

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Haotian Liu, Kilho Son, Jianwei Yang, Ce Liu, Jianfeng Gao, Yong Jac Lee, and Chunyuan Li. Learning cus- tomized visual models with retrieval-angmented knowledge. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 15148 15158, 2023b.
Jie Liu, Yanqi Bao, Guo-Sen Xie, Huan Xiong, Jan-Jakob Sonke, and Eistratios Gavves. Dynamic prototype convolution network for few-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11553 11562, 2022.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-sel object detection. arXiv preprint arXit:2303.05499 2023c.
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. K-bert: Enabling language representation with knowledge graph. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 34, pp. 2901 2908, 2020.
Timo Lüddecke and Alexander Ecker. Image segmentation using text and image prompts. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognation, pp. 7086-7096, 2022.
Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Gupta, and Marcus Rohrbach. Krisp: Integrating implicit and symbolic knowledge for open-domain knowledge-based vqa. In Proceedings of the IEEE/CVP Conference Compuler Vision and Pallern Recognalion, pp. 14111 14111-14121, 2021.
Umberto Michieli and Pietro Zanuttigh. Continual semantic segmentation via repulsion-autraction of sparse and disentangled latent representations. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pp. 1114-1124, 2021.
Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong- Whong Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 891-898, 2014.
Jishnu Mukhoti, Yarin Gal, Philip HS Torr, and Puncet K Dokania. Fino-tuning can cripple your foundation model; preserving features may be the solution. arXi preprint arXiv:2308.1330 2023.
Youngmin Oh, Donghyeon Baek, and Bumsub Ham. Alife: Adaptive logit regularizer and feature replay for incremental semantic segmentation. Advances in Neural Information Processing Systems, 35:14516 14528, 2022.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin FllNouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint ar Xix::334.00198, 2023.
Matthew E Peters, Mark Neumann, Robert L Logan IV, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A Smith. Knowledge enhanced contextual word reprosentations. arXiv preprint arXiv: (..00004100. 064, 2019.
Fabio Petroni, Samuel Broscheit, Aleksandra Piktus, Patrick Lewis, Gautier Izacard, Lucas Hosscini, Jane Dwivedi-Yu. Maria Lomeli, Timo Schick, Michele Bevilacqua, et. al. Improving wikipedia verifiability with ai. Nature Machine Intelligence, 5(10):1142 1148, 2023.
Ameya Prabhu, Zhipeng Cai, Puneet Dokania, Philip Torr, Vladlen Koltun, and Ozan Sener. Online con- tinual learning without the storage constraint. arXio preprint arXi::2355.092 2023a.
Ameya Prabhu, Hasan Abed Al Kader Hammoud, Puneet Dokania, Philip HS Torr, Ser-Nam Lim, Bernard Ghanem, and Adel Bibi. Computationally budgeted continual learning: What does matter? In CVPR, 2023b.
Amcya Prabhu, Hasan Abed Al Kader Hammoud, Ser-Nam Lim, Bernard Ghanem, Philip HS Torr, and Adel Bibi. From categories classifier: Namc-only continual learning by exploring the web. arXiv preprint arXiv:2311.11200 2023c.
14

---

Published in Transactions on Machine Learning Research (07/2024)
Lu Qi, Jason Kuen, Yi Wang, Jiuxiang Gu, Hengshuang Zhao, Philip Torr, Zhe Lin, and Jiaya Jia. Open world entity segmentation. MI, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkinn Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021a.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021b.
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Browe and Yoav Shoham. In-context anggentd- language models. arXiv preprint arXiv:2302. 00083, 2023.
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Landuage-guided dense prediction with context-aware prompling. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 18082-18066 2022.
Sylvestre-Alviee Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Karsten Roth, Latha Pemula, Joaquin Zepeda, Bernhard Schölkopf, Thomas Brox, and Peter Gehler. To- wards total recall in industrial anomaly detection. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition (CVPR), pp. 14318-14328, June 2022.
Soroush Soifi, Danicl Olmoda Reino, Fabicn Despinoy, and Rahaf Aljundi. Annotation free semantic sog- mentation with vision loundation models. arXiv preprind orXiv:2403. 093007 2024.
Sheng Shen, Chunyuan Li, Xiaowei Hu, Yujia Xie, Jianwei Yang, Pengchuan Zhang, Zhe Gan, Lijuan Wang, Tu Yuan, Ce Liu, et al. K-lite: Learning transferable visual models with external knowledge. Advances in Newral Information Processing Systems, 35:1555: 15573, 2022.
Shelly Sheynin, Oron Ashual, Adam Polyak, Uriel Singer, Oran Gafni, Eliya Nachmani, and Yaniv Taigman. Knn-diffusion: Image generation via. large-scale retrieval. or preprinl arXiv:2204.02: 022444 2022.
Guangchen Shi, Yirui Wu, Jun Liu, Shaohua Wan, Wenhai Wang, and Tong Lu. Incremental few-shot semantic segmentation via embedding adaptive-update and hyper-class representation. In Proceedings of the 30th ACM international conference on multimedio, pp. 5547 5556, 2022.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. Replug: Retrieval-augmentee black-box language models. arXiv preprint arXiv:2301.12652 2023.
Gyungin Shin, Weidi Xie, and Samuel Albanie. Reco: Retrieve and co-segment for zero-shot transfer. Advances in Neural Information Processing Systems, 35:33754-33767, 2022.
Shuyang Sun, Runjia Li, Philip Torr, Xiuye Gu, and Siyang Li. Clip mn: Segment countless visual concepts without training endeavor. arXiv preprint arXim:2312.076611 2023a.
Shuyang Sun, Weijun Wang, Qihang Yu, Andrew Howard, Philip Torr, and Liang- Chieh Chen. Remax: Relaxing for better training on efficient panoptic segmentation. arXiv preprint arXiw:2306. 17319, 2023b.
Vishaal Udandarao, Ankush Gupta, and Samuel Albanic. Sus-x: Training-free name-only transfer of vision- language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2725-22 2736, 2023.
15

---

Published in Transactions on Machine Learning Research (07/2024)
Vishaal Udandarao, Ameya Prabhu, Philipp Torr, Matthias Bethge, Adel Bibi, and Sammel Albanie. No "zero- shot" without exponential data: Pretraining concept frequency determines multimodal model performance. arXiv preprint, 2024.
Guankun Wang, Long Bai, Yanan Wu, Tong Chen, and Hongliang Ren. Rethinking exemplars for contin- ual semantic segmentation in endoscopy scenes: Entropy-bar -basdd mini-batch pscudo-replay. Computers in Biology and Medicine, 165:107412, 2023a.
Haoxiang Wang, Pavan Kumar Anasosalu Vasu, Fartash Faghri, Raviteja Vemulapalli, Mehrdad Farajtabar, Sachin Mehta, Mohammad Rastegari, Oncel Tuzel, and Hadi Pouransari. Sam-clip: Merging vision foun- dation models towards semantic and spatial understanding. arXiv preprint vrXic:2310.15308 2023b.
Shuohang Wang, Yichong Xu, Yuwei Fang, Yang Liu, Siqi Sun, Ruochen Xu, Chenguang Zhu, and Michael Zeng. Training data is more valuable than you think: A simple and effective method by retrieving from training data. arXiv preprint arXiv:2203.0 08773, 2022.
Yulin Wang, Gao Huang, Shiji Song, Xuran Pan, Yitong Xia, and Cheng Wu. Regularizing deep networks with semantic data augmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44 (7):3733 3748, 2021.
Yuhuai Wu, Markus N Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers. arXiv preprinl trXii:2203.088 089113 2022.
Bin Xie, Jiale Cao, Jin Xie, Fahad Shahbaz Khan, and Yanwei Pang, Sed: A simple encoder-decoder for open-vocabulary semantic segmentation. arXi preprint ar 2rX1:2331.155577, 2023.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Bycon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In CVPR, 2022a.
Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2955 2966, 2023a.
Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for open-vocabulary semantic segmentation with pre-trained vision- lasoua-anguuage model. In European Conference on Computer Vision, pp. 736-753. Springer, 2022b.
Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open- -pca-vocayylary semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2945 2954, 2023b.
Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zel and Wen-lau Yih. Retrieval-augmented multimodal language modeling, arXiv preprinl arXiv:2211. 12561, 2022.
Jeongbeen Yoon, Dahyun Kang, and Minsu Cho. Semi-supervised domain adaptation via sample-to tomple- sclf-distillation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1978 1987, 2022.
Lu Yu, Bartlomicj Twardowski, Xialci Liu, Luis Herranz, Kai Wang, Yongmei Cheng, Shangling Jui, and Joost van de Weijer. Semantic drift compensation for class-incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6982 6991, 2020.
Qihang Yu, Ju He, Xueqing Deng, Xiaolui Shen, and Liang- Chich Chen. Convolutions die hard: Open- vocabulary segmentation with single frozen convolutional clip. arXiv preprint arXim:2308.0244 2023.
Wenhao Yu, Chenguang Zhu, Yuwei Fang, Donghan Yu, Shuohang Wang, Yichong Xu, Michael Zeng, and Meng Jiang. Dict-bert: Enhancing language model pre-training with dictionary. arXio preprint arXiv:21 100.04990, 2021.
16

---

Published in Transactions on Machine Learning Research (07/2024)
Bo Yuan and Danpei Zhao. A survey continual semantic segmentation: Theory challenge, method and application. arXie preprint 0r 2023.
Jianhao Yuan, Shuyang Sun, Daniel Omeiza, Bo Zhao, Paul Newman, Lars Kunze, and Matthew Gadd. Rag- driver: Generalisable driving explanations with retrieval-augmented in-context learning in multi-modal large language model. arXiv preprint arXiv:2402.1088 10828, 2024.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Autonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302 321, 2019.
Chong Zhou, Chen Change Loy, and Bo Dai. Denseclip: Extract free dense labels from clip. arXiv preprinl arXiv:2112.01071, 2021.
Lanyun Zhu, Tianrun Chen, Jianxiong Yin, Simon See, and Jun Liu. Continual semantic segmentation with automatic memory sample selection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3082 3092, 2023.
17


Reviews:
[{"review_id": "Co6lGmvbz0", "reviewer": "Reviewer_Rana", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper opens with the claim: \"The holy grail in image segmentation is to accurately segment any concept images based on a text query.\" I don't agree with this, or at least I don't understand it. What is a \"concept image\" exactly? \u0013I don't think related work on image segmentation makes much fuss about concept images.  \n\n\nThe key question raised in this work is \"Can we enhance these models to a continually-growing vocabulary with data without catastrophic forgetting?\" There must be some typo here, or words missing, because I am getting lost in the sentence at the \"with data\" part. It doesn't seem like the question should be emphasizing this, since the related works all use data as well. The paper makes other surprising claims about data too, like training \"to data\" (in the next sentence). I cannot make sense of this, except to say that these sentences are broken, and clear meaning is missing.\n\nThe authors claim that their approach is \"effortless\". This does not seem true. The implementation details make it very clear that some effort is required. \n\nThe paper claims that its main contribution is to \"uncover the paradox where fine-tuning VLM models for segmentation using downstream annotations dramatically reduces their ability to recognize the broad VLM vocabulary\". This seems covered by work that was actually described as motivational in this paper, like \"CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor\" by Sun et al 2023. \n\nThe paper claims that its second main contribution is \"Training-Free Continual Vocabulary Expansion\", but this also is covered by \"CLIP as RNN\" (which is training-free), and other works like MaskCLIP and ReCO and CaR. \n\n\nIn the method, before describing any database construction, the paper boasts that \"One appealing characteristic of our framework is the dynamic extendability of the database and the customization of the database construction based on personalized interest.\" For a comment like this to make sense, it needs to appear after the introduction of the component being described (\"the database\"). \n\n\nThe paper says that the image dimensions are \"usually shrunken by a magnitude of 10 each\" when referring to the stride of the ViT. I have actually never seen a ViT with stride 10. What weights are these? Was this ViT trained from scratch?\n\nThe paper claims that \"efficiently representing images with small feature maps guarantees the memory efficiency of our method\". This seems obviously false, because anything can be done afterward to increase memory consumption. \n\nThe paper does not make it very clear what exactly the input setup is.  The introduction makes it sound like CLIP is getting turned into an open-vocabulary segmentation model via some labelled dataset, but then  it is revealed that the images already have \"query masks\", and then the task seems to be merely labelling. Also it's not clear what the model/pre-training setup is: the intro makes it sound like CLIP is the main thing, but then later DINO appears, and some unrelated ViT appears as well, and also a \"Mask Proposer\" shows up in Figure 3  but never appears in the method. \n\nwith it’s corresponding -> with its corresponding\n\n\nThe paper places great emphasis on \"we do not store any images for replay\", but this is true for all training-free methods. Replay does not make any sense when there's no training.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a method for open-vocabulary panoptic segmentation, using a combination of CLIP, DINO, and other models. The key idea is to pool the features of labelled objects, to create a database of labelled features, and then use these features to help vote for classification of new query masks. The main benefit here is that new \"concepts\" can be added by simply appending them to the database, and it is not necessary to fine-tune any model weights.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I pointed out many weaknesses, but my hunch is that these are not critical weaknesses/errors in method or motivation, but are simply the result of unclear writing. So, these can be addressed by careful rewriting. \n\nHarder to address might be: issues with the stated key contributions, which seem to actually be contributions of prior work.\n\nFinally, it would greatly strengthen the paper if the method were made more clear. What exactly is the input? How many pre-trained models are used, and where do they come from? Once these details are more clear, then it would also be interesting to find out how much specific details matter, like what if different models were used instead? In this sense, it would be great to convey a method that appears more general-purpose.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "clZfX1c9cI", "reviewer": "Reviewer_2MDe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- The experiments cover several types of segmentation datasets. A comparison against supervised method method (Mask2Former) and class incremental method (CoMFormer) helps contextualize the methods performance. A comparison is made to Hummingbird is section 4.6 (Table 5).\n\n- The ablations evaluate the most obvious hyper-parameters ($k$, $\\lambda$, and $T$).\n\n- The main strength is the demonstration that a simple nearest neighbour approach is able to boost performance on a sophisticated task like open vocabulary panoptic segmentation.\n\n## Weaknesses\n\n- Figure 1 teases concept customization, but this is not discussed nor evaluated in the manuscript. As it has been advertised centrally in page 1 figure 1 it deserves an evaluation.\n\n- The numbers reported for Mask2Former in Table 2 are far below the numbers reported in the Mask2Former paper. Which table in the Mask2Former paper (https://arxiv.org/pdf/2112.01527) should I look at to compare? Which type of segmentation task is being evaluated in section 4.2 / Table 2?\n\n- The details of how Hummingbirg and the proposed method differ have not been discussed making the comparison in Table 5 less useful to the reader. Please elaborate in the manuscript (or appendix).\n\n## Clarification questions\n\n- In section 4.1, does the \"linear combination of original predicted class probabilities, with the retrieved prediction class probabilities\" refer to Eq. 4?\n\n- In the same paragraph, which equation does the \"geometric ensemble\" refer to?\n\n- In Table 4, which dataset is used to compute the database of embeddings ($D_{vec}$)? Are all methods in the table seeing the same number and types of annotations?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript proposes to augment open vocabulary segmentation models with a database of embeddings ($D_{vec}$) for the task of expanding their segmentation capabilities to new categories. This is achieved by combining the segmentation predictions of the base model with a prediction computed from the KNN embeddings retrieved from $D_{vec}$.\n\nThe manuscript reports that without this augmentation, the base model, which is normally fine-tuned on a segmentation dataset, performs poorly on categories not labeled in that segmentation dataset. The KNN approach is presented within the framework of class incremental learning.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Concerns have already been raised in the section above.", "broader_impact_concerns": "Segmentation can be used in sensitive applications such as medical, military, and surveillance. A broader impacts statement would be useful.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7631SYQrey", "reviewer": "Reviewer_XLqz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n - The work studies an interesting and relevant problem setting (Open-Vocabulary Segmentation with a Growing Vocabulary)\n - The approach makes intuitive sense, is simple, and a logical continuation of prior work\n - Overview over related works is solid\n - The authors evaluate the method extensively and kNN-CLIP performs favourably to baselines.\n\nWeaknesses:\n - The exposition of the method (Section 3) lacks clarity (e.g. on the usage of CLIP and DINO)\n - Mathematical notation lacks rigour (see comments on requested changes)\n\nOverall, I think this is a promising work but it needs more care in writing and exposition. See requested changes below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces kNN-CLIP, a method for Open-Vocabulary Segmentation in a continual training setting (continually-growing vocabulary). Its main contribution is proposing a method to prevent catastrophic concept forgetting.  This method (kNN-CLIP) adds a retrieval mechanism to CLIP such that novel concepts (extended vocabulary) are not stored in CLIP's weights but in the concept bank over which retrieval is performed. This approach is related to retrieval-augmented generation in large-language models. The author study the proposed kNN-CLIP on a number of segmentation benchmarks and compare it to baselines, to which it performs favourably.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Page 2 third paragraph: \"we can guarantee that the model never forgets previously seen data\": please specify what specifically your work guarantees (in a more formal way)\n- Figure 2: using the term \"support vector\" here is suboptimal, since it is never introduced in the main document and also is too reminiscent on SVMs\n - The introduction of Section 3 mentions the usage of CLIP+DINO, but the rest of the section never clarifies where and how they are used.\n - \"The embedding dimensions h′, w′ are determined by image size divided by the patch size of the vision transformer, thus usually shrunken by a magnitude of 10 each,\": isn't this usually 8, 16 or 32?\n - \"The ground-truth masks Mi are resized to M′ i ∈ Rk×h′×w′ to align with the feature map dimensions.\": how are they resized? NN-interpolation? wouldn't a soft interpolation (bilinear or similar) make more sense here?\n - The notation in Equation 1 is  not sound: (i) there should not be an equal sign in the definition of the set. (ii) \"R^{k×d} = {...}\" this is wrong, the defined set is not equal to R^{k×d} but to E_i. I understand what the authors want to say, but I would ask them to fix the notation here.\n - Equation 2: there is nothing in the notation which makes the set \"kNN_k(eq)\" actually contain the k-nearest neighbours. There is no sorting of the index j according to the respective cosine similarities.\n - Equation 3: please clarify why no temperature parameter is used. Usually, CLIP uses a temperature when mapping from cosine similarities to confidences\n - Equation 4: please clarify why using a hard-threshold-based combination is preferable over a soft-thresholding\n - In the ablations  (Table 6)  both the number of neighbours and the confidence threshold are varied. However, they are only varied independently and not jointly. E.g could it be that 64 neighbours and a higher threshold would yield a better result than the currently best combination of k=16 and T=0.7? \n- \"ViT-Giant architecture, featuring 4 register tokens.\" please cite the work introducing register tokens.\n- Please clarify what a \"geometric ensemble\" is.\n- \"We first train the models like FC-CLIP [...]\": please make the paper self-contained by explaining briefly how FC-CLIP was trained.\n- Table 2 (or one of the others): please report the memory overhead of kNN-CLIP for the retrieval database. Even if it is small, it is relevant to know \"how small\". \n - Number of neighbours in retrieval (Table 7): does it make sense to choose this constantly regardless of the size of the retrieval database? Specifically initially, when the database is very small, retrieving 16 neighbours sounds like a lot.\n- since inference time drops substantially with kNN-CLIP: Is using DINO ViT-G really required or would smaller ViTs be an alternative?", "broader_impact_concerns": "No concerns on broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After the authors' revision, most of the reviewers' concerns have been addressed. The reviewers unanimously recommend acceptance of the paper.

I find the work valuable and recommend its acceptance and publication. 

Additionally, I have the following suggestions for the authors as they prepare the final version:

* Figures 2-4 are not in vector graphics format, causing them to become blurry when the PDF is zoomed in. It is recommended to use vector graphics for these figures.
* A limitations section could be added.
* As suggested by Reviewer Rana, the method could be clarified further.
* Proofread the paper again to avoid any typographical errors.
Decision: Accept (poster)