SPTNet: An Efficient Alternative Framework for Generalized Category Discovery with Spatial Prompt Tuning
Authors: Hongjun Wang
Sagar Vaze
Kai Han
Venue: ICLR 2024 poster (2024)
Keywords: Generalized Category Discovery
Novel Category Discovery
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We introduce SPTNet, a two-stage adaptation approach for Generalized Category Discovery (GCD) that optimizes both model and data parameters, achieving superior performance to existing methods by considering the spatial properties of image data.

Abstract: Generalized Category Discovery (GCD) aims to classify unlabelled images from both ‘seen’ and ‘unseen’ classes by transferring knowledge from a set of labelled ‘seen’ class images. A key theme in existing GCD approaches is adapting large-scale pre-trained models for the GCD task. An alternate perspective, however, is to adapt the data representation itself for better alignment with the pre-trained model. As such, in this paper, we introduce a two-stage adaptation approach termed SPTNet, which iteratively optimizes model parameters (i.e., model-finetuning) and data parameters (i.e., prompt learning). Furthermore, we propose a novel spatial prompt tuning method (SPT) which considers the spatial property of image data, enabling the method to better focus on object parts, which can transfer between seen and unseen classes. We thoroughly evaluate our SPTNet on standard benchmarks and demonstrate that our method outperforms existing GCD methods. Notably, we find our method achieves an average accuracy of 61.4% on the SSB, surpassing prior state-of-the-art methods by approximately 10%. The improvement is particularly remarkable as our method yields extra parameters amounting to only 0.117% of those in the backbone architecture. Project page: https://visual-ai.github.io/sptnet.

Published as a conference paper at ICLR 2024
SPTNET: AN EFFICIENT ALTERNATIVE FRAMEWORK FOR GENERALIZED CATEGORY DISCOVERY WITH SPA- TIAL PROMPT TUNING
Hongjun Wang 1
Sagar Vaze
Kai Han't
Visual Al Lab, The University of Hong Kong 2 Visual Geometry Group, University of Oxford h njwaan@cceneet.. ..k...k..kkkkkkk.
sagar@robots. ac.uk
kaihanx @hau
ABSTRACT
Generalized Category Discovery (GCD) aims to classify unlabelled images from both 'seen' and 'unseen' classes by transferring knowledge from a set of labelled 'seen' class images. A key theme in existing GCD approaches is adapting large- scale pre-trained models for the GCD task. An alternate perspective, however, is adapt the data representation itself for better alignment with the pre-trained model. As such, in this paper, we introduce a two-stage adaptation approach termed SPTNet, which iteratively optimizes model purameters (i.e., model-finetuning) and data parameters (i.e., prompt learning). Furthermore, we propose a novel spatial prompt tuning method (SPT) which considers the spatial property of image data, enabling the method to better focus on object parts, which can transfer between seen and unseen classes. We thoroughly evaluate our SPTNet on standard benchmarks and demonstrate that our method outperforms existing GCD methods. Notably, we find our method achieves an average accuracy of 61.4% on the SSB, surpassing prior oftte-of-the-rtt methods by approximately 10%. The improvement is particularly remarkable as our method yields extra parameters amounting to only 0.117% of those in the backbone architecture. Project page: https: /// visual-af ..w.aii.giti.. github. / sptnet.
INTRODUCTION
Deep learning models have been extensively studied in image recognition He et al. (2016); Krizhevsky et al. (2017), typically relying on large-scale annotated data, as well as a 'closed-wordd' assumption: that the data to be classified shares the same classes the labelled training data. However, this assumption limits application to real-world scenarios where the target data contains 'unseen' classes images alongside 'seen' ones Han et al. (2019; 2020; 2021); Fini et al. (2021); Wen et al. (2023); Jia al. (2021); Zhao & Han (2021). Recently, Category Discovery (CD) has emerged as practical open- open-world learning problem, where a model trained using partially labelled data is tasked to categorize unlabelled data that may originate from unseen categories. Initially, it studied as Novel Category Discovery (NCD) Han et al. (2019) focusing on unlabelled data exclusively from unseen categories. Subsequently, it extended to Generalized Category Discovery (GCD) Vaze et al. (2022) encompassing unlabelled data from both seen and unseen categories.
-ftte-of-the-art GCD methods Vaze et al. (2022); Cao et al. (2022); Wen et al. (2023) employ pre-trained self-supervised models, such as DINO Caron et al. (2021), and partially fine-tune their parameters the target task, taking advantage of the strong generalization properties of these representations. In this paradigm, data remains fixed while iterating over the model. However, fully fine-tuning a large pre-trained model can lead to overfitting to the labelled data, and is computationally expensive. Instead of focusing solely on the model, we find that alternatives which manipulate the data to cater to the model, are both more efficient and can also achieve better GCD performance. Specifically, visual prompting methods (e.g., Jia al. (2022); Bahng al. (2022)), have recently been explored to improve model capability by modifying the input or intermediate features through the addition of extra learnable tokens. Although these methods are effective in fully supervised learning, they do not improve representations for generalization and struggle to achieve satisfactory performance in the open- world GCD task. A natural approach to integrating both ad advantages
1 Corresponding author.

---

Published as a conference paper at ICLR 2024
is to simultaneously optimize the model and data parameters. However, this non-convex bilevel optimization often leads to sub-optimal solutions for both sets of parameters.
Inspired by the popectation-maximzaaieee techniques for bilevel optimization Engelmann et al. (2020); Byeon & Van Hentenryck (2022), we introduce a two-stage iterative learning framework called SPTNet for GCD, optimizing both model parameters (i.e. finettiinn and data parameters (i.e. prompt learning). More specifically, the framework includes two phases: (1) In the first phase, the backbone model is frozen, and only the prompts are adjusted. (2) In the second phase, we fix the prompt parameters and update the backbone model with a contrastive loss, using an augmented data pair constructed by the raw image together with its prompted version. The prompts and model are alternately trained until convergence. In this way, our learned prompt can be considered as a learned augmentation, targeted for the downstream recognition task (see Fig. 1).
(EM) algorithm Dempster al. (1977) and decomposition
Following arguments in the GCD literature Vaze et al. (2022), that object parts are an effective vehicle to transfer knowledge between 'seen' and 'unseen' categories, we propose Spatial Prompt Tuning (SPT), which learns pixel-level prompts around local image regions. Unlike previous methods (e.g-, Jia et al. (2022); Bahng et al. (2022)) that introduce learnable tokens to the hidden model space, or wrap prompts around the entire image border, SPT divides the original image into patches and attaches prompts to each patch in pixel space. The objective of SPT is to achieve improved alignment between the large pre-trained model and discriminative image regions the target task. We conduct experiments on seven datasets using the standard evaluation protocol in the GCD setting. Our method achieves an average accuracy of 61.4%, which is higher than the previous otate-of-the-art methods by around 10%, in proportional terms, on the SSB benchmark Vaze et al. (2021). Remarkably, this improvement is achieved by introducing only 0.117% extra parameters compared to all ViT- Base parameters, demonstrating the efficiency and effectiveness of our approach.
Our contributions can be summarized follows: (1) We introduce a two-stage iterative learning framework called SPTNet, integrating advantages of both model parameters (i.e., model-finctuning and data parameters (i.e., prompt learning) learning for GCD. (2) We propose a new spatial prompt method (called SPT) to adapt the data representation for better alignment with the pre-trained model. The method learns independent prompts for different spatial regions and introduces only 0.039% additional parameters compared to all ViT-Base parameters. (3) We conduct comprehensive evaluations of our method on seven datasets, including three generic (i.e., CIFAR-10, C00AR-100, and ImageNet- 100) and four fine-grained benchmarks (CUB, Stanford Cars, FGVC-Aireraf, and Herbarium 9). Our method outperforms state-of-the-an methods in most cases.
2 RELATED WORK
Semi-supervised learning (SSL) alleviates the issue of inadequacy of labelled data for training. which learns from both labelled and unlabelled data from predefined classes to get strong classifi- cation model. Consistency-basee Mixmatch Berthelot et al. (2019) and Fixmatch Sohn et al. (2020), operate by enforcing model pre- diction consistency under various perturbations of the unlabelled data or over the course of training. Recent methods, such as Chen et al. (2020b;e; 2021), have shown improved SSL performance by introducing contrastive learning (e.g., Chen et al. (2020a), He al. (2020)). Several works Wang et al. (2022b); Rizve et al. (2022); Wang et al. (2024); Sun et al. (2024) extend the standard SSL to an open-world setting.
approaches, including Mean-teacher Tarvainen & Valpola (2017),
Novel category discovery (NCD) aims at categorizing unlabelled images from unseen classes by transferring knowledge from labelled data of seen classes Han et al. (2019). Various approaches have been proposed to address NCD, for example, Han et al. (2019) introduces two-stage training cluster the unlabelled data. Han et al. (2020; 2021); Zhao & Han (2021) utilize ranking statistics to generate pseudo positives among unlabelled novel classes. Zhong et al. (2021b) transfers semantic knowledge through Mix augmentation between seen and novel classes, as well as reliable novel anchors with other examples. Zhong et al. (2021a) proposes neighborhood contrastive loss and hard-negative generation process by mixing novel and seen classes. Fini et al. (2021) reformulates the NCD problem into classification based on dynamic class assignments using Baakhomn-Knoph algorithm. Jia et al. (2021) addresses multi-modal NCD by inter- and modal contrastive learning with permutation-ensemlie statistics. Gu et al. (2023) proposes a novel knowledge distillation framework, which utilizes our class-relation representation to regularize the learning of novel classes.
method, which first utilizes metric learning, followed by learning
ranking
2

---

Published as a conference paper at ICLR 2024
Stage ones Flx 730k update P.t
Stage twee Fix P. updato FANK
Every
Block
Tranfooeer
Trann former Block
Block m
Transformes
Projector
MIP
Block
Transformer
Transformeer Blcck
Transformer Block m
Projector
C
P1
+
Every step
Forward Backward
Frozon
Tunod C. Objection Lanston
X.X' Diferont views of Input P. Spatiai promots
Figure 1: The overall framework of SPTNet. SPTNet alternates between data parameter tuning (stage one) and model parameter tuning (stage two). The data parameters are learnable prompts, for which we introduce spatial prompts Ps. The model parameters include the parameters of the top layer of the Transformer backbone F and a projection head H.
Generalized Category Discovery (GCD) extends NCD by categorizing unlabelled images from both seen and unseen categories (Vaze et al. (2022)). Vaze et al. (2022) tackles this issue by tuning the representation of the pre-trained ViT model with DINO (Caron et al. (2021); Oquab et al. (2024)) with contrastive learning, followed by semi-supervised k-means clustering. ORCA Cao et al. (2022) considers the problem from semi-supervised learning perspective and introduces an adaptive margin loss for better intra-class separability for both seen and unseen classes. CiPR Hao et al. (2024) introduces method for more effective contrastive learning and hierarchical clustering method for GCD without requiring the category number in the unlabelled data be known priori. SimGCD Wen et al. (2023) proposes a parametric method with entropy regularization to improve performance. DCCL Pu et al. (2023) improves clustering accuracy by alternating between estimating underlying visual conceptions and learning conceptional representations. They also introduce a dynamic conception generation and update mechanism to ensure consistent conception learning. PrompiCAL Zhang et al. (2023) introduces two-stage framework that iteratively generates and refines affinity graphs based on the model's current understanding of the data to enhance the semantic discriminativennes of pre-trained vision transformers. GPC Zhao al. (2023) proposes a GMM-based method that can jointly learn robust representation for GCD and estimate the unknown category number. We also note the concurrent work Vaze et al. (2023) which improves GCD performance with a student-teacher mechanism.
Prompt learning, as the representative of data parameters learning methods, targets at simply prepending a few extra tokens to the input and provides an effective and efficient solution that matches the performance of fully fine-tuning, commonly used in Natural Language Processing (NLP). Recently, prompting learning has been used in vision tasks. Particularly, Visual Prompt Learning (VPT) Jia et al. (2022) has been introduced to optimize extra visual prompts on top of pre-trained ViT backbone to achieve strong object recognition performance, Bahng et al. (2022) learns an additional "border" of input images as prompts to adapt large-scale pre-trained models, which improves the models' classification accuracy. There are also some works which utilize prompts to deal with different tasks, such as classification with imbalanced data Dong et al. (2022) or domain shift Wang et al. (2022a). Shtedritski et al. (2023); Khattak al. (2023) offer the possibility of manipulating both textual and visual modalities through prompting.
3 METHODS
3.1 PRELIMINARIES
Problem statement. Assume that we have the open-world dataset D. comprising two subsets:
labelled set D, {(x,, N C X; 3/j and an unlabelled set D, (x,) 1 X, E x3 H and W are the height and width of the image. 31- C1 and ) = C= UC2 are the label space of labelled and the unlabelled samples. C. C,. and C2 denote the label set for 'All'. 'Old", and 'New' categories, respectively. The objective of GCD is to categorize all the unlabelled images in D,,, having access to labels only in D;. For simplicity, hereafter we omit the subscript for each image X₁.
C X,, where
Architecture. We consider a parametric model consisting of feature extractor F and a projection head H. For an image X from D, we can obtain its class prediction by ğ = H(F(X)). Specifically, employ Vision Transformer (VIT) Dosovitskiy al. (2020) the architecture. We consider
3

---

Published as a conference paper at ICLR 2024
the Transformer encoder as F and simple multilayer perceptron (MLP) as H. In VIT. an image X first divided into n patches (x²,... ;..,,,"),,, where x' R2xhx= and n (H x / w). The patches are then mapped into d-dimensional latent space by a shared linear projection layer e.
Together with an extra learnalle classification token CLS. the full model is formulated as:
(x x²) = =0(X)
Fg = "a.r));-.;e(x"")
CLSS; E.j = L:([CSi-11; Ei-1))
ÿ H((LSas),
where o--) denotes the "patheify" operator to divide the input image X into patches (x* : xx), L, denotes the i-th layer of the ViT and denotes concatenation.
Model/Data parameters. We consider optimizing both model parameters and data parameters for GCD. Optimizing the model parameters is the most common way to train or fine-tune model by minimizing the loss function on a given dataset. Previous GCD methods, such as Vaze et al. (2022); Wen et al. (2023); Pu et al. (2023); Zhang et al. (2023), fine-tune the final transformer block and the linear projection layer of a pre-trained ViT model. In our method, the model parameters are in FF and H. Recently, visual prompt learning techniques have been introduced to effectively adapt pre-trained large-scale models to different downstream tasks, without the need of tuning the model parameters, We refer to such techniques as optimizing data parameters. Particularly, VPT Jia et al. (2022) inserts sequence of learnable embeddings in the input for each Transformer encoder layer L; of the ViT model. Specifically, VPT freezes the feature extractor F but learns a set of prompt tokens P. [r]:j = 1, .... b} with pi E R as part of the input for layer L, The input can be denoted as [CLS,; P;; E.J. Instead of inserting several tunable parameters to each layer, Bahng et al. (2022) attaches learnable parameters Pg to the border of the raw input image as (x") (x'..... xx")) = ++X++ P.).
We propose Spatial Prompt Tuning (SPT) for GCD, as will be described in Section 3.3, which attaches
a learnable prompt for each image patch. Namely, we learn a set of prompts P3 = {p':1 j =1....
and attach the prompts to the input image patches by (a* + pp. x" + +p") = ((X) + P..
3.2 SPTNET: AN ALTERNATE PROMPT LEARNING FRAMEWORK FOR GCD
Large-scale pretraining (e.g., DINO Caron et al. (2021) seff-supervision) is the key ingredient in existing GCD methods Vaze et al. (2022); Wen et al. (2023); Pu et al. (2023); Zhang et al. (2023). As GCD requires learning from unlabelled data, contrastive self self-supervised learning is the natural choice, which uses data augmentations to create different views the same input image. These augmentations provide an inductive bias as to what is (not) semantically meaningful in an image. clear but unexplored option that enables efficient adaptation of
In this context, prompt tuning pre-trained models. Our insight is that the learned prompt can also be used to generate a novel view, making suitable choice for the contrastive framework. Simultancously optimizing the model and prompts seems appealing, but it results in instability and sub-optimal solutions for both data and model parameters.
To mitigate the issue, inspired by EM algorithm Dempster et al. (1977), we propose SPTNet, two-stage alternative learning framework for GCD. The overall framework is illustrated in Fig. 1. The learning objective includes both representation learning and parametric classification, while our framework alternates between data parameter and model parameter optimization using the same learning objective.
Specifically, in vanilla contrastive learning, two different views, X and X', of the same input image are constructed as a positive pair. A set of other images are drawn from the dataset as negative samples N (XX) {X,;4 :4 1, I.... .Q). Then, the parameters of F can be updated by the InfoNCE loss Oord et al. (2018) using the data triplet (X,X', (',(NN(()),,
exp(cos( F (F(X),
Cun (X. X', F..)
log E (F(() ))/T))
(2)
where cos(s,) ) is the cosine similarity between embedding feature vectors and Tu is a temper- ature hyperparameter. Analogous to Eq. (2), supervised contrastive loss Khosla et al. (2020)
Camp (X, (X))),,( label y in the mini-batch.
utilizes a set of positive samples P(X) having the same class
Next, to assign labels to input instances, we use parametric methods to classify them into seen or new classes, as commonly done in image recognition. In supervised contrastive learning, this is
4

---

Published as a conference paper at ICLR 2024
achieved through the simultaneous optimization of F and H using the cosine-softmat enoss-yntropy loss Gidaris & Komodakis (2018):
exppcos(HH(F(X)),,))))))
cls (x, Uj H, F, Ts) = Σ Yvc log IOI exp(ccos((((((())) Wx)//
(3)
where H(F(X)) and Wx are the fz-noraalieed feature and the prototype vector of class respectively, X' is also used the above loss, as an additional augmented version of X. For the unsupervised counterpart. X' is removed from the input while the prediction label for self-distillation. The loss can be denoted as Cel (X, ; loss L can be written as:
H(F(('))) () is used as pseudo
HH F, Tz). Therefore, the overall
Cds) + X(Cuue + Cels =A,
where X, are the balance factors, and A represents the the mean-entropy antropy-maxiimsstie maximisation regu- lariser Assran et al. (2022), computed by taking the entropy of the mean prediction of all samples in mini-batch.
Our SPTNet alternates between optimizing data parameters and model parameters as follows:
Stage one: Fix F&H and update P... In the first stage, we attach the same set of spatial prompts P. to the input images, X, X', and (XX). The framework is trained with the loss in Eq. (4), while the image patches are replaced by their "prompted'' version. For example, ((X) in Eq. (1) is replaced by o(X) + Ps. and the same applies to and ((X). ). During training, we freeze the parameters of F&H and only update the prompt parameters of P.. It is worth noting that, to facilitate the generalization, the weight decay for optimizing P, set zero to prevent prompts from being sparse. Meanwhile, during the learning in Stage two, our spatial prompting acts as a strong data augmentation. Increasing the variation in the parameters of P. leads to more diverse 'prompted' image pairs to benefit the representation learning. It was also noted in the literature that more diverse data augmentation is helpful for representation learning based on contrastive learning (e.g., HaoChen et al. (2021)).
Stage two: Fix P. and update F&H. In the second stage, we freeze prompt parameters P. and learn the parameters of H and the top layer in F, again, with the loss in Eq. (4). With our spatial prompt learning strong augmentation, we aim to obtain representation that can better distinguish samples from different classes, as the core mechanism of contrastive learning involves implicitly clustering samples from the same class together.
Different from prior works that apply only hand-crafted augmentations, we propose to consider prompting the input with learnable prompts, i.e., (xX) + Pe, as new type of augmentation. The 'prompted' version of the input can be adopted by all loss terms. In this way, our framework can enjoy learned augmentation that varies throughout the training process, enabling F to learn discriminative representations. Each stage optimizes the parameters for & iterations.
3.3 SPATIAL PROMPT TUNING
Naively applying existing prompt tuning methods does not lead to satisfying performance (see 2nd and 3rd rows in Table 4). We speculate that prompts in the hidden model space rather than input space make it harder to align inputs within the contrastive framework, denced by our empirical results in Fig. 5. Besides, a key insight in GCD is that object parts effective in transferring knowledge between old and new categories Vaze et al. (2022). Therefore, we propose Spatial Prompt Tuning (SPT) to serve as a learned data augmentation that enables the model to focus on local image object regions, while adapting the data represen- tation from the pre- pre-trained ViT model and maintaining the align- ment with it. In SPT, we inject small number learnable param- eters into the input image patches and keep the backbone JF and the projection head H. frozen during training. Unlike existing methods that introduce learnable tokens hidden model space Jia et al.
evi-
W
SPT
(b) SPT Global
Figure 2: (a) An example of applying Spatial Prompt Tuning (SPT) to an image with a height H and width W. For each image patch x with height h and width w, attach spatial prompts P. of size m to it. (b) Joint spatial and global prompts for SPTNet.
5

---

Published as a conference paper at ICLR 2024
(2022) or wrap prompts around the entire image border Bahng et al. (2022), SPT divides the image
into patches and attaches learnable prompts to each partch. Specifically, let (x ,... x = (X) be the set of image patches divided from image X. For each patch x² E R SPT wraps instance-agnostic prompts P. around it in a rectangular shape with a width of m. as illustrated in Fig. (a). Thus, there are 6m(h- + w 2m) learnable parameters for the prompts of each patch.
Our SPTNet proceeds alternatively between the two stages and gradually learns the spatial prompts shared across all images. As revealed in Zhao & Han (2021). both global and local spatial information benefits novel category discovery. Therefore, apart from SPT tokens, SPTNet also wraps an additional global prompt around the entire image like Bahng et al. (2022), as illustrated in Fig. 2 (b).
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. We evaluate the effectiveness of SPT on three generic image recognition datasets (i.e., CIFAR-10/100 Krizhevsky et al. (2009) and ImageNet-100 Tian et al. (2020)), three fine-grained datasets (i.e., CUB Welinder et al. (2010), Stanford Cars Krause et al. (2013), and FGVC-Aireraft Maji et al. (2013)) contained in Semantic Shift Benchmark (SSB) Vaze et al. (2021), and the challenging large-scale fin-grained dataset Herbarium- 19 Tan et al. (2019). For each dataset, first subsample [C1] seen (labelled) classes from all classes. Following Vaze et al. (2022), we subsample 80% samples in CIFAR-100 and 50% samples in all other datasets from the seen classes to construct Dr, while the remaining images are treated as Du. The statistics of the datasets can be found Table 1.
Table 1: Dataset statistics and training configurations.
Labelled
Unlabelled
Configs
Dataset
#Num
#Class
ANum
AClass
wes
lxp
Md,
CIFAR Krizhevsky et (2009)
12.5K
37.5K
10
30-3
5e-4
20
CIFARIDO Krizbevaky et al. (2009)
20.0K
80
30.0K
100
1c-3
Sc-4
1.0
0
20
]
Image 100 Tian et (2020)
31.9K
50
95.3K
100
3--3
Se-4
10.0
20
1
Herbariom
(2019)
8.9K
25.4K
683
5e-4
20
CUB Welinder ot al. (2010)
1.5K
100
4.5K
200
0.05
5c-4
25.0
0
20
Stunford Cars Krause
(2013)
2.0K
98
6.1K
196
0.05
5e-4
25.0
20
RGVC-Airenaf Maji al. (2013)
1.7K
5.0K
50
5e-4
20
Evaluation protocol. We use clustering accuracy (ACC) evaluate the model performance, as per standard practice. During the evaluation, we compare the ground-truth labels y with the predicted
labels ùi and measure the ACC by ACC = |o. Σ = PPul I (v+ ((x)), where G represents the optimal permutation that gives the matching between the predicted labels with the ground truth.
Implementation details. We develop our SPTNet upon the SimGCD Wen et al. (2023) baseline and apply the spatial prompt tuning on the pre-trained ViT-B/I6 backbone Caron et al. (2021). Specifically, we take the final feature corresponding to the CLS token from the backbone as the image feature, which has dimension of 768. For the feature extractor F. we only fine-tun tune the last block. We set the spatial prompt size m to 1, while the globe prompt size to 30 which is the default value in Bahng et al. (2022). It is worth noting that our method yields extra parameters amounting only 0.117% of those in the backbone architecture (see Appendix A for details). The two stages alternate every k = 20 iterations. All prompts are trained for 1,000 epochs with batch size of 128. We utilize SGD as the optimizer for training, employing different learning rates (1 1r₁) and weight decay parameters ppper-parameters. determined on the
(wdp. wdb) to update prompts and the model. The training validation data splits, are shown in Table 1. We set the balancing factor À to 0.35 and the temperature values Tu and T. to 0.07 and 1.0, respectively, following Wen et al. (2023). For the temperature values To and T. in the classification losses, we also set them to 0.07 and 0.1. All experiments are conducted using an NVIDIA GeForce RTX 3090 GPU.
4.2 MAIN RESULTS
Evaluation on generic datasets. We evaluate SPTNet three generic datasets, CIFAR- 10, CIFAR- 100 and ImageNet- 100. We compare SPTNet with previous state-of-ther the--artt methods and two concur- rent methods (DCCL Pu et al. (2023) and PromptCAL Zhang et al. (2023)). The results are shown in Table 2. We can see that our method consistently outperforms previous sfate-of-the-arr methods. Specifically, SPTNet surpasses the baseline SimGCD by 0.4% on CIFAR- 1.9% on CIFAR- 100, and 2.5% on ImageNet-100 for 'All' classes; it also outperforms concurrent methods on both CIFAR- 100 and ImageNet- 100. SPTNet performs on par with PromptCAL. on CIFAR-10 10 but with much fewer learnable parameters and shorter training time (see Table 13). Note that for CIFAR-10 and
6

---

Published as a conference paper at ICLR 2024
CIFAR-100, the images are of extremely low-resolution (32 x 32). As such, limited information is provided in each patch, leading to limited gains from our proposed (local) spatial prompting. On ImageNet-1000 100, performance boosts are difficult to yield, as the original DINO backbone is already highly tuned for this dataset. This is evidenced by the gains (usually) being substantially less between the previous ffte-of-thhe-r and the simple k-means on raw DINO features Vaze et al. (2022).
Table 2: Evaluation three generic image recognition datasets, Bold values represent the best results, while underlined values represent the second best results.
CIFAR-10 10
CIFAR- 100
ImageNet-100
Method
All
Old
New
All
Old
New
All
Old
New
X-means Arthur Vassilvitskii (2006)
83.6
85.7
82.5
52.0
52.2
50.8
72.7
75.5
71.3
RankStatss Han al. (2021)
46.8
19.2
60.5
58.2
77.6
19.3
37.1
61.6
24.8
UNO+ Fini et al. (2021)
68.6
98.3
53.8
69.5
80.6
47.2
70.3
95.0
57.9
GCD Vaze al. (2022)
91.5
97.9
88.2
73.0
76.2
66.5
74.1
89.8
66.3
ORCA Cao et al. (2022)
96.9
95.1
97.8
74.2
82.1
67.2
79.2
93.2
72.1
SimGCD Wen el al. (2023)
97.1
95.1
98.1
80.1
81.2
77.8
83.0
93.1
77.9
DCCL Pu et al. (2023)
96.3
96.5
96.9
75.3
76.8
70.2
80.5
90.5
76.2
PromptCAL Zhang al. (2023)
97.9
96.6
98.5
81.2
84.2
75.3
83.1
92.7
78.3
SPTNet (Ours)
97.3
95.0
98.6
81.3
84.3
75.6
85.4
93.2
81.4
Evaluation on fine-grained datasets. Table 3 presents the results on fine-grained datasets including the SSB benchmark and Herbarium 19 dataset. The unsatisfactory performance of k-means and ORCA highlights the difficulty in discovering fine- aine-grined categories due to large intra-clast and small inter-class variations. In contrast, SPTNet demonstrates superior performance SimGCD, DCCL, and PrompiCAL. achieving an average absolute improvement of ~5% and an average proportional improvement of 10% across all evaluated datasets in SSB, specifically on 'All' classes. As there a clear semantic axis in SSB benchmark, and data augmentations implicitly define this 'semantic axis' or taxonomy in contrastive learning, SPT as a learned data augmentation ultimately enhances the GCD performance. This indicates that global and local prompts assist the model in focusing on details that dominate correctness in fine-graine one-grained recognition in GCD.
Table 3: Evaluation on the Semantic Shift Benchmark (SSB) and Herbarium 19. Bold values represent the best results, while underlined values represent the second best results.
CUB
Stanford Cars
FGVC-Aireratt
Herbarium19
Method
AlI
Cldd
New
All
Clid
New
All
Oid
New
All
Old
New
k-means Arthar& Vassilvieskii (2006)
34.3
38.9
32.1
12.
10.6
13.8
12.9
12.9
12.8
13.0
12.2
13.4
RankStatse Tam et (202))
33.3
51.6
24.2
28.1
61.8
12.1
27.9
55.8
12.8
27.9
55.8
12.8
UNO+ Fini et al. (2023)
35.1
25.1
35.5
70.3
18.6
28.3
53.7
14.7
28.3
53.7
14.7
GCD Vase al. (2022)
51.3
48.7
39.0
57.6
29.9
45.0
41.1
46.9
35.4
51.0
27.0
ORCA
(2022)
51.9
42.2
52.0
31.4
SimGCD Wen (2023)
Pu (2023)
PrompeCAL Zhang et al. (2023)
62.1
50.
70.1
52.2
52.3
37.0
28.9
SPTNe (Ours)
65.8
68.3
65.1
59.0
79.2
40.1
61.8
58.1
43.4
35.2
4.3 ABLATION STUDY
In this part, primarily focus the challenging SSB assess the effectiveness of different components and report the averaged results among CUB, Stanford Cars, and FGVC-Aircraft. As our method employs the same pre-trained model and objective function as SimGCD, we consider SimGCD as the baseline method for comparison.
Effect of prompt-related techniques. We first experiment with the strong SimGCD baseline using the recommended configuration in Wen et al. (2023). Table presents the results of the ablation study on the components of our SPT. The 2nd row shows the performance of adopting the VPT method on the pre-trained SimGCD. Comparing with the raw SimGCD base in the 1st row, the performance after adopting VPT is dropped. After adopting the global prompt (3rd row) on the pre-trained SimGCD, the performance is increased by 0.6% on 'All' classes. This indicates that naively applying existing prompt tuning methods does not yield satisfactory performance on GCD; the improvement by the global prompt, though marginal, is still encouraging, as it suggests that the pixel-level prompt method is suitable for the GCD setting when compared with VPT. Adopting our SPT (4 row) on pre-trained SimGCD gives a relatively larger improvement of 1.8% on 'All' classes. The effectiveness of our proposed method may be attributed to the spatial design for exploring semantic discrimination in local regions. Our alternate training strategy (5th & 7th rows) can effectively improve the performance, demonstrating its effectiveness. We also explore a variant of SPT by using shared prompts across all patches t6th row), which also demonstrates promising performance. After further introducing the global prompts t7th & 8th row), the performance further improved. The 8th row corresponds to our default SPTNet, which achieves the best performance. We refer to the variant in the 5th row as SPTNet-S (Shared), and the variant in the 6th row as SPTNett- (Patch). Both of these variants
7

---

Published as a conference paper at ICLR 2024
Table 4: Comparison on effectiveness of different prompting methods on SSB. We report the average test accuracy score over all component datasets of SSB (i.e., CUB, Stanford Cars and FGVC-Aircraft). 'Shared' and 'Alter' refer represents SPTNet and rows (5) and (6) represent its two variants SPTNet-S and SPTNet-P.
single shared prompt for all patches and alternative learning. Row (9)
No Method config
Procupt config
All
Old
New
(1)
None (haseline)
51.5
(2) SimGCD et (2023)
VPI Jia al. (2022)
54.4
64.7
49.1
(3)
+Gobal Bahng et al. (2022) +SPT
56.7
61.699
53,5+20
(4)
57.-1.8
67.2+17
53,3+1.8
(4)
Glohal Bahng ot al. (2022)
57.8
66
53.8133
(5) -Aller
Shared
60.5
68.6131
56,515.0
(6)
+SPT
59.1-30
68.5+30
51,5+10
(7) -Alber
Shared & Global Bahng al. (2023)
60
69.0
57.35.8
(8)
SPT & Global Bohig et al. (2022)
61,4
60.9*44
57,5+6.0
relatively more parameter efficient (see Appendix A for details), spatially SPTNet-S, while obtaining superior performance (more results of these two variants can be found in Appendix C).
Effect of different training strategies. To investigate the impact of different training strategies, we conduct additional experiments on both generic and fine-grained datasets. We consider two different training strategies, namely, (i) end-to-end (3rd row): both the data parameters and the (
model parameters are jointly trained in an end-to-end fashion; (ii) data fasafirst (4th row): the prompt parameters are optimized first, followed by the model parameters; (iii) model first (5th row): the model parameters are optimized first, followed by the prompt parameters; and (iv) alternative (6th row): our alternative training strategy, which optimizes model and data parameters alternatively, every other k iterations. The results are presented in Table Comparing rows (3) with the SimGCD baseline in row (I), we can see that SPTNet consistently outperforms SimGCD and our alternative training strategy leads the best performance. Since the SPTNet is built upon the pre-trained SimGCD, one might wonder about the performance of further fine-tuning SimGCD. In the 2nd row, we show the results after further fine-tuming the pre- pre-trained SimGCD. An improvement can be achieved, while the margin is significantly smaller compared with the improvement achieved by SPTNet. This suggests that both our SPT and alternate training strategy are beneficial for GCD.
Table 5: Evaluation on ImageNet- 100 and SSB using different training strategies.
ImageNet-100
SSB
No
Methods
All
Old
New
All
Old
New
(1) SimOCD Wen et al. (2023)
83.0
93.1
77.9
56.1
65.5
51.5
(2) SimGCD (further fine-tune)
84.3
93.1
79.7
57.0
66.0
52.3
(3) SPT (end-to-end)
84.1
92.8
80.0
58.6
67.4
53.2
(4) SPTNet (data first)
83.5
92.9
77.7
58.0
66.4
51.9
(5) SPTNet (model first)
84.8
93.3
80.6
59.2
67.8
54.9
(6) Set (alternative)
85.4
93.2
81.4
61.4
69.9
57.5
Effects of alternating frequency and prompt size. In our alternative training strategy, we alternate between the data and model parameter optimization every other A: iterations. Meanwhile, we also need to determine the spatial prompt size m. In Fig. 3, we present the average ACC on SSB with varying k and mn respectively. For k, we do not observe significant differences among different choices, and thus use moderate value of 20 leads to better performance. When m is too large, the image content might be over-oceluded, causing difficulty for the model to properly recognize the object. We also show the effects of global prompt size Appendix D.
default choice. For m, we find that a smaller value generally
[a) Attematinn frequency
(b) Prompt
Figure 3: Effects of different choices of alternating frequency (a) and prompt size (b) on SSB (i.e., CUB, Stanford Cars and FGVC-Aireraft. We report the averaged results and show the influence on 'All', 'Old' and 'New' classes.
8

---

Published as a conference paper at ICLR 2024
SimGCD
SIM
SPTNet-P
SPTNett
bird
automobll
Unseen
frog
horse
Figure 4: t-SNE visualization of representations on CIFAR-10. SPTNet produces the most discrimi- native representations among all compared methods.
4.4 QUALITATIVE COMPARISON
How do prompts affect the representations? To investigate the influence of different visual prompts, we visualize representations on CIFAR- 10 through t-SNE Van der Maaten Hinton (2008) in Fig. We compare representations of the SimGCD baseline, SimGCD+VPT, SPTNet, as well as SPTNet-P (which contains only the spatial prompts). They correspond the models in row (1), row (2), row (8), and row (6) in Table 4. Comparing the representations of SimGCD and SimGCD+ VPT, VPT appears to have negative impact on the representation, leading to clutter between seen and unseen classes (e.g., bird and dog) in the GCD setting. This is also aligned with the deteriorated performance of SimGCD+ VPT in Table 4. Both SPTNet-P and SPTNet produce more discriminative features and more compact clusters than SimGCD. Thanks to the global prompt, SPTNet further improves the representation over STPNet-P.
How do prompts affect the model's attention? The attention map provides very helpful clues to understanding the Transformer-based models' focus on the input. We extract the attention maps for the CLS token from different attention heads in the last layer of the ViT backbone and show the top 10% most attended patches in Fig. 5. We observe that for SimGCD and SimGCD+Global (i.e., row (4) in Table 4), different heads may focus on the same region (e.g.. s. in the 'seen' example, hy/hs/f10 of SimGCD and ha/hs//g of SimGCDD+Global) and some heads may attend to the background regions (e.g., in the 'unseen' example, h2/ha/h? for SimGCD) and h /y/ for SimGCD+Globall In contrast, SPT and SPT&Global attend to more diverse regions of the object and focus more on the foreground object regions, with the latter performing better.
SimGCD
SimGCD+Global
SPTNet-P
SPTNet
Seen class
Unseen class
Figure 5: Attention visualization of different heads (numbered as hı to h12). The top 10% attended patches are shown in red.
More results and analysis can be found in the Appendix.
CONCLUSION
In conclusion, we have introduced SPTNet, an efficient framework for Generalized Category Discov- ery (GCD). We propose a two-stage alternative optimization scheme, optimizing both model and data parameters, to enhance alignment between the pre-trained model and the target task. Additionally, we introduce spatial prompt tuning (SPT) method focus on object parts and facilitate knowledge transfer between seen and unseen classes, Experimental evaluations demonstrate the superiority of SPTNet over existing methods.

---

Published as a conference paper at ICLR 2024
Acknowledgements This work is supported by Hong Kong Research Grant Council Early Career Scheme (Grant No. 27208022), National Natural Science Foundation China (Grant No. 62306251), and HKU Seed Fund for Basic Research.
REFERENCES
David Arthur and Sergei Vassilvitskii. k-means++: The advantages of careful seeding. Technical report, Stanford, 2006.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. In ECCV, 2022.
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv: 2203. 17274, 2022.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. NeurIPS, 2019.
Geunyeong Bycon and Pascal Van Hentenryck. Benders subproblem decomposition for bilevel problems with convex follower. INFORMS Journal on Computing, 2022.
Kaidi Cao, Maria Brbic, and Jure Leskovec. Open-world semi-supervised learning. In ICI.R, 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Ting Chen, Simon Korbllihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. In NeurIPS, 2020b.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint wrXiv:2003.0429) 2020c.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, 2021.
Arthur P Dempster, Nan Laird, and Donald Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society: series B (methodological), 1977.
Bowen Dong, Pan Zhou, Shuicheng Yan, and Wangmeng Zuo. Lpt: Long-tailed prompt tuning for image classification. ICLR, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020.
Alexander Engelmann, Yuning Jiang, Boris Houska, and Timm Faulwasser. Decomposition of nonconvex optimization via bi-level distributed aladin. IEEE Transactions Control of Network Systems, 2020.
Enrico Fini, Enver Sangineto, Stéphane Lathuilière, Zhun Zhong, Moin Nabi, and Elisa Ricci. A unified objective for novel class discovery. In ICCV, 2021.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In CVPR, 2018
Peiyan Gu, Chuyu Zhang, Ruijie Xu, and Xuming He. Class-relation knowledge distillation for novel class discovery. In ICCV, 2023.
Kai Han, Andrea Vedaldi, and Andrew Zisserman. Learning to discover novel visual categories via deep transfer clustering. In ICCV, 2019.
10

---

Published as a conference paper at ICLR 2024
Kai Han, Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Automatically discovering and learning new visual categories with ranking statistics. ICLR, 2020.
Kai Han, Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Autonovel: Automatically discovering and learning novel visual categories. IEEE TPAMI, 2021.
Shaozhe Hao, Kai Han, and Yee K Wong. Cipr: An efficient framework with cross-instance positive relations for generalized category discovery. TMLR, 2024.
Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. In NeurlPS, 2021.
Kaiming Hc. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie. Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022.
Xuhui Jia, Kai Han, Yukun Zhu, and Bradley Green. Joint representation learning and novel category discovery on single- and multi-modal data. In ICCV, 2021.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In CVPR, 2023.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NeurIPS, 2020.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV workshop, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Communications of the ACM, 2017.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv: 2013.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint
2018
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby. Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po- Yao Huang. Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Ar- mand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. TMLR, 2024.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. ICCV, 2019.
Nan Pu, Zhun Zhong, and Nicu Sebe. Dynamic conceptional contrastive learning for generalized category discovery. In CVPR, 2023.
Mamshad Nayeem Rizve, Navid Kardan, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Openidn: Learning to discover novel classes for open-world semi-supervised learning. In ECCV, 2022.
Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about a red circle? visual prompt engineering for vims. In ICCV, 2023.
11

---

Published as a conference paper at ICLR 2024
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Do- gus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In NewrIPS, 2020.
Yiyou Sun, Zhenmei Shi, and Yixuan Li. A graph-theoretic framework for understanding open- opendworld semi-supervised learning. In NeurlPS, 2024.
Kiat Chuan Tan, Yulong Liu. Barbara Ambrose, Melissa Tulig, and Serge Belongie. The herbarium challenge 2019 dataset. arXiv preprint arXiv: 1906. 0936,0537,, 2019.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight- consistency targets improve semi-supervised deep learning results. In NeurIPS, 2017.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV, 2020.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 2008
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: good
closed-set classifier is all you need. In ICLR, 2021.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. In CVPR, 2022.
Sagar Vaze, Andrea Vedaldi, and Andrew Zisserman. No representation rules them all in category discovery. In NeurLPS, 2023.
Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained transformers:
An occam's 's razor for domain incremental learning. In NeurIPS, 2022a.
Yidong Wang, Hao Chen, Yue Fan, Wang Sun, Ran Tao, Wenxin Hou, Renjie Wang, Linyi Yang, Zhi Zhou, Lan-Zhe Guo, et al. Usb: A unified semi-supervised learning benchmark for classification. In NeurlPS, 2022b.
Yu Wang, Zhun Zhong, Pengchong Qiao, Xuxin Cheng, Xiawu Zheng, Chang Liu, Nicu Sebe, Rongrong Ji, and Jie Chen. Discover and align taxonomic context priors for open-world semi- supervised learning. In NeurlPS, 2024.
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-uesd birds 200. 2010.
Xin Wen, Bingchen Zhao, and Xiaojuan Qi. Parametric classification for generalized category discovery: A baseline study. ICCV, 2023.
Sheng Zhang, Salman Khan, Zhiqiang Shen, Muzammal Naseer, Guangyi Chen, and Fahad Khan. Promptcal: Contrastive affinity learning via auxiliary prompts for generalized novel category discovery. In CVPR, 2023.
Bingchen Zhao and Kai Han. Novel visual category discovery with dual ranking statistics and mutual knowledge distillation. In NeurIPS, 2021.
Bingchen Zhao, Xin Wen, and Kai Han. Learning semi-supervised gaussian mixture models for generalized category discovery. ICCV, 2023.
Zhun Zhong, Enrico Fini, Subhankar Roy, Zhiming Luo, Elisa Ricci, and Nicu Sebe. Neighborhood contrastive learning for novel class discovery. In CVPR, 2021a.
Zhun Zhong, Linchao Zhu, Zhiming Luo, Shaozi Li, Yi Yang, and Nicu Sebe. Openmix: Reviving known knowledge for discovering novel visual categories an open world. In CVPR, 2021b.
12

---

Published as a conference paper at ICLR 2024
APPENDIX
Discussion on different variants of SPTNet
14
Training configurations for SPTNet-P and SPTNet-S
15
C Benchmarking results of SPTNet-P and SPTNet-S
16
D Effects of global prompt size m
17
Visualization of learned prompts
18
F Results based DINOv2
19
G Robustness of SPTNet for GCD with domain shifts
20
H Unknown category number
21
I Performance and time efficiency
22
J Theoretical analysis on our alternate training
23
K More visualization and analysis of attention maps
25
Broader impacts
36
13

---

Published as a conference paper at ICLR 2024
A DISCUSSION ON DIFFERENT VARIANTS OF SPTNET
As discussed in Section 3.3, default SPTNet has both spatial and global prompts. In Section 4.3, we also introduce two variants of SPTNet with reduced prompt parameters, SPTNet-P (Patch) and SPTNet-S (Shared). SPTNet-P attaches only the spatial prompts without the global prompt to the input (row Table 4). The spatial prompts vary different patches. SPTNet-S attaches a single shared spatial prompt without the global prompt to the input (row in Table 4). In Figure 6, we compare the prompts of SPTNet, SPTNet-P and SPTNet-S.
As SPT wraps small number of parameters around the raw input image in rectangular shape with a width of m. As also discussed in Section 3.3, the number parameters for the spatial prompt of 1, and the number of patches = 196. (16 16 2) = 5, 880. 196 such prompts give 196 x 5, 880 = 35, 280 = 0.034M parameters. As for the global padding, the number of parameters is 6m H W 2m+)) Let H/ W 224. The number of parameters for the global prompt is x 30 x (224 + 224 - 60) = 69, 840. Therefore, the total number of parameters for SPT & Global is 35, 280 + 69, 840 105, 120. As the backbone model, ViT-B, has 86M parameters, SPTNet, SPTNet-P, and SPTNet-S only introduce 0.117%, 0.039%, and 0.0002% extra parameters compared to ViT-B, respectively.
each patch is (h + 2m). Let /
16,
The number of parameters for single spatial prompt 6
=m
H
H
W
W
W
(a) SPTNet
(b) SPTNet-P
(c) SPTNet-S
Figure 6: Prompts of SPTNet, SPTNet-P and SPTNet-S. SPTNet has both distinct spatial prompts and global prompt: SPTNet-P has multiple distinct spatial prompts; SPTNet-S has single shared spatial prompt.
14

---

Published as a conference paper at ICLR 2024
B TRAINING CONFIGURATIONS FOR SPTNET-P AND SPTNET-S
We show the training configurations for SPTNet-P and SPTNet-S in Table 6 and Table respectively. The prompt size of SPT m is set to by default for both SPTNet-P and SPTNet-S. We set the global prompt size mr in SPTNet to 30 (see Appendix D). For the ViT model, specifically, we resize all the input images into 224 224, so we have w x 196 patches with a resolution of 16 16 pixels.
Table 6: Training configurations for SPTNet-P SPTNet-S.
Configs
Dataset
1r6
wdb
1rp
wdp
k
CIFARIO Krizhevsky et al. (2009)
3e-3
5e-4
20.0
20
CIFARIO0 Krizhevsky al. (2009)
le-3
5e-4
1.0
20
ImageNet-100 Tian et al. (2020)
3e-3
5e-4
10.0
20
Herbarium 19 Tan al. (2019)
5e-3
5e-4
1.0
20
CUB Welinder et al. (2010)
0.05
5e-4
25.0
20
Stanford Cars Krause et al. (2013)
0.05
5e-4
10.0
20
FGVC-Aireraft Maji al. (2013)
0.05
5e-4
1.0
20
Table Training configurations for SPTNet.
Configs
Dataset
1rb
wcb
1 p
wdp
m
m
CIFARIO Krizhevsky et (2009)
3e-3
5e-4
0
30
CIFAR1O0 Krizhevsky al. (2009)
3e-3
5e-4
0
30
ImageNet-100 Tian et al. (2020)
3e-3
5e-4
5.0
0
20
30
Herbarium 19 Tan ct al. (2019)
5c-3
5c-4
1.0
0
20
30
CUB Welinder et al. (2010)
0.05
5e-4
25.0
0
20
30
Stanford Cars Krause et al. (2013)
0.05
5c-4
10.0
0
20
30
FGVC-Aircraft Maji et al. (2013)
0.05
5e-4
1.0
0
20
30
15

---

Published as a conference paper at ICLR 2024
C BENCHMARKING RESULTS OF SPTNET-P AND SPTNET-S
We further evaluate the performance of the more parameter-efficien SPTNet variants, SPTNet- and SPTNet-S, Table and Table 9. As can be seen, SPTNet, SPTNet-P and SPTNet-S consistently outperform the baseline in all cases.
Table 8: Evaluation on three generic image recognition datasets. Bold values represent the best results, while underlined values represent the second best results.
CIFAR-I 10
CIFAR- 100
ImageNet- 100
Method
All
Old
New
Old
New
New
SimGCD Wen et al. (2023)
97.1
95.1
98.1
80.1
81.2
77.8
83.0
93.1
77.9
SPTNet-P (Ours)
97.5
95.2
98.5
82.0
85.5
75.0
85,5
93.9
81.2
SPTNet-S (Ours)
97.5
95.9
98.3
81.0
83.8
75.4
85.5
94.1
81.2
SPTNet (Ours)
97.3
95.0
98.6
81.3
84.3
75.6
85.4
93.2
81.4
Table 9: Evaluation on the Semantic Shift Benchmark (SSB) and Herbarium 19. Bold values represent the best results, while underlined values represent the second best results.
CUB
Stanford Cars
FGVC-Airerat
Herbarium19
Method
All
Old
New
All
Old
New
All
Old
New
All
Old
New
SimGCD Wen al. (2023)
60.3
65.6
57.7
53.8
71.9
45.0
54.2
59.1
51.8
43.0
58.0
35.1
SPTNet-P (Ours)
64.6
70.5
61.6
55.6
74.4
46.5
57.2
60.6
55.5
43.3
58.0
35.5
SPTNet-S (Ours)
65.0
69.1
62.9
60.1
75.3
52.8
56.3
61.4
53.8
43.4
58.6
35.2
SPTNet (Ours)
65.8
68.8
65.1
59.0
79.2
49.3
59.3
61.8
58.1
43.4
58.7
35.2
16

---

Published as a conference paper at ICLR 2024
D EFFECTS OF GLOBAL PROMPT SIZE m
As the global prompt size m² + may affect the performance of SPTNet, we experiment with different global prompt sizes, namely m = 1, 10, 20, 30, 40, 50. We measure accuracy the CUB dataset using the same architecture and configurations in the main paper. Fig. 7 demonstrates that m 30 yields the best overall performance, which is the default setting in our main paper.
CUB
75.0
70.5
71.0
69.3
68.7
68.8
68.0
ACC (%)
65.0 6
65.8
65.2
65.
85.4
64.6
64.3
62.8
63.0
6..
62 1
56.6
+O'
O New
59.0
m+
Figure 7: Performance of SPTNet with different global prompt sizes m on CUB. We show the influence 'All', 'Old' and 'New' classes. When m is set 30, SPTNet achieves the best performance on 'All' categories.
17

---

Published as a conference paper at ICLR 2024
E VISUALIZATION OF LEARNED PROMPTS
We visualize different prompts after convergence in Fig. 8. Except for the end-to-end strategy, SPT and its variants commonly exhibit active parameters in prompts. This is attributed to their instance-agnostic nature, which enables them to handle variations in object locations across the dataset. Consequently, parameter values do not degrade to zero a region patch that background in one input but foreground in another. It is also worth noting that most prompts (particularly located the borders) are deactivated when employing the end-to-end strategy. We hypothesize that this is due to the network unintentionally adopting a "shortcut" approach, where it only updates model parameters to achieve invariance to learned augmentation while optimizing both model and data (prompt) parameters simultaneously. This also validates the need for alternate training when learned augmentation is applied.
SPTNet (end-to-end)
SPTNet-S
SPTNet-P
SPTNet
Figure 8: Visualization different learned prompts. Parameters of SPT and variants are mostly active, while most prompts are much less active when employing the end-to-end strategy.
18

---

Published as a conference paper at ICLR 2024
F RESULTS BASED ON DINOV2
Here, replace the pre-trained DINO model with the recently improved DINOv2 model Oquab et al. (2024) which is empowered with stronger representation capacity from unlabelled data. Results are shown in Table 10. We can observe that the stronger DINOv2 representation indeed enhances model performance as expected, especially in the 'New' categories, and SPTNet and the two variants still consistently outperform SimGCD.
Table 10: Evaluation CUB and ImageNet-100 using the pre-trained DINOv2 model. Bold values represent the best results, while underlined values represent the second best results.
CUB
ImageNet 100
Method
All
Old
New
All
Old
New
DINO+SimGCD Wen cl. (2023)
60.3
65.6
57.7
83.0
93.1
77.9
DINO+SPTNNe-P (Ours)
64.6
70.5
61.6
85.5
93.9
81.2
DINO+SPTNet-S (Ours)
65.0
69.1
62.9
85,5
94.1
81.2
DINO+SPTNeI (Ours)
65.8
68.8
65.1
85.4
93.2
81.4
DINOv2+SimGCDD Wen ct al. (2023)
67.4
69.5
66.3
88.5
96.2
84.6
DINOv2+SPTNet-P (Ours)
69.0
69.7
68.7
90.5
96.3
87.5
DINOv2+SPTNat-5 (Ours)
69.0
70.3
68.3
90.6
96.4
87.6
DINOv2+SPINet (Ours)
69.2
69.0
69.3
90.1
96.1
87.1
19

---

Published as a conference paper at ICLR 2024
G ROBUSTNESS OF SPTNET FOR GCD WITH DOMAIN SHIFTS
To validate the robustness of SPTNet, we test method in more challenging setting, GCD with domain shifts. We conduct experiments on the largest UDA dataset DomainNet Peng et al. (2019), containing about 0.6 million images with 345 categories distributed among six domains. We apply the data construction process Vaze al. (2022) to construct the 'Old', 'New' and 'All' splits based on DomainNet and evaluate different methods on our constructed data. To account for domain shifts that were not considered in Vaze al. (2022), construct the partially labelled data by using images from both the 'real' domain and the 'painting' domain to train the model. Specifically, we utilise subset of labelled images from select classes in the 'real" domain, along with unlabelled images from all classes the 'painting' domain. We assess the model's performance on both the 'real' and 'painting' domains. Additionally, we evaluate the model on images from other previously unseen domains, including "quickdraw', 'sketch', 'infograph', and 'clipart'. Results are shown in Table 11. Compared with other baseline methods in the vanilla GCD setting, we find that SPTNet can perform well on (i) labelled and seen domain (i.e., 'real"), (ii) unlabelled but seen domains (i.e., 'painting") and (iii) unseen domains (i.e., others, including 'quickdraw', 'sketch', 'infograph', and 'clipart').
Table 11: Evaluation the DomainNet benchmark. The model is trained the 'real' and 'painting' domains and we report the respective results on real, painting and the remaining four domains (i.e., others). Bold values represent the best results, while underlined values represent the second best results.
Real
Painting
Others
Methods
All
Old
New
All
Old
New
All
Old
New
RankStats+
34.1
61.9
19.7
29.7
49.7
9.6
14.3
25.5
5.5
UNO+
44.2
72.2
29.7
30.1
45.1
17.2
14.0
23.4
7.4
ORCA
49.8
23.5
38.5
7.1
10.4
19.5
8.1
GCD
47.3
53.6
44.1
32.9
41.8
23.0
15.2
22.0
11.1
SimGCD
61.3
77.8
52.9
34.5
35.6
33.5
16.7
22.5
12.2
SPTNet (Ours)
63.1
75.9
56.4
39.2
43.1
35.2
17.4
22.2
13.6
20

---

Published as a conference paper at ICLR 2024
H UNKNOWN CATEGORY NUMBER
As the total number of categories (GT) cannot be accessed in the real-world setting, we evaluate our SPTNet-P with an estimated number of categories using an off-the-shelf method Vaze et al. (2022) (see Table 12) We also evaluate our method with varying category numbers (see Fig. 9). Our evaluation includes two representative datasets: CUB for fine-grained and ImageNet- 100 for generic classification tasks. We find that our method consistently outperforms SimGCD on both datasets when the exact number of categories is unknown.
Table 12: Performance of SPTNet-P and the baseline method SimGCD with an estimated number of categories on CUB and ImageNet- 100. Bold values represent the best results.
CUB
ImageNet- 100
Method
C
All
Old
New
All
Old
New
SimGCD Wen et al. (2023)
GT (200/100)
60.3
65.6
57.7
83.0
93.1
77.9
SPTNett-P (Ours)
GT (200/100)
64.6
70.5
61.6
85,5
93.9
81.2
SimGCD Wen et al. (2023)
Est. (231/109)
61.0
66.0
58.6
81.1
90.9
76.1
SPTNett-P (Ours)
Est. (231/109)
65.2
71.0
62.3
83.4
91.8
74.6
CUB
Imageteet-100
103
08.1
93.9
93.0
70.1
70.8
872
23
00.3
04.0
865
65.1
83.7
ACC (80
812
70.3
MCD
521
80
73
Old New
60
0.5
Figure 9: Performance with varying category numbers. We experiment with category numbers obtained by multiplying the GT number with different factors C" = {0.1, 0.5, 1.0, 2.0, 10.0).
We also evaluate our SP on CIFAR-100 dataset with fewer known categories. The results shown in Fig. 10, where 50% of the samples from known classes are labelled. The results indicate that SPTNet is robust in few-class scenarios and outperforms the concurrent method, PromptCal Zhang et al. (2023). It more challenging for models to infer novel semantic clustering when fewer classes are known due to semantic shifts, resulting in decreased performance for all methods. This further demonstrates the effecttiveness of our proposed method.
CIFAR-100
CIFAR-100
#00
85.5
79.5
74
78.
75.0
70.9
ACG
ACC
68.1
70.0
652
66.3
66.6
05.2
Old
Oid
New
80
50
25
10
50
50
25
|G|
|G|
(b) PromptCal Figure 10: Performance with a varying number of known classes [C,
(a) SPTNet-P
21

---

Published as a conference paper at ICLR 2024
I PERFORMANCE AND TIME EFFICIENCY
To assess the practicality of different methods, conducted further comparisons in terms of accuracy, training time per epoch, and inference time. The results are presented Table 13. Our proposed SPTNet demonstrates superior accuracy while obtaining mostly the best time efficiency.
Table 13: Time efficiency of different methods on ImageNet- 100 and SSB. Bold values represent the best results.
ImsrgeNet-
Accunacy (All))
Training (Sco)
Inference (See)
Accuracy (AII)
Training time (Sod)
(Soe)
GCD (2022)
SimGCD Wen et (2023)
847
391
56.1
17
PrompCAL Zhang (2023)
X3.1
1817
393
55.1
492
103
SPTNNr (Oir))
85.4
4K3
601
61.4
32
17
22

---

Published as a conference paper at ICLR 2024
J THEORETICAL ANALYSIS ON OUR ALTERNATE TRAINING
To estimate the model parameters 0, it is common to introduce the log-likelihood function L(0) In P(X 0). This function quantifies the likelihood of the parameter given the data X. As the natural logarithm function, In X, is monotonically increasing, maximizing P(X () is equivalent to objective.
maximizing (().. In other words, maximizing the log- oog-likelihood function L(() achieves the same
The EM algorithm is an iterative procedure designed to maximize L(0). Let 0, denote the current estimate for after the i-th iteration. Our goal is to calculate an updated estimate 9 that maximizes L(0):
L( - L (0:) I 0) I 0z)
In
Σ P(X ,0)(""))("")) 10)
- PI 0c)
plin
P Pppt 0x
ln
Σ P(X lim 1pm,,0)((p"" 10)
-In 0,)
P Pppin x,
In
P (x,0))
P(X p OpP 10)
II (X 01)
(5)
P Ppllin
X, 0))
piin
[P(X pt lm
O)P(pIm 10)
P (p 1m0
X,0+)
In (X (0,)
P (p (x,e)
pio
[P(X 20 (p 10)
Σ P (p
0,) In
(p X, 0,) P (X 0,)
pd
A H (t)
Let 1 (0) 0.) = L (0.) + H1 (0, 0z) which is bounded above by the likelihood function L (0). Let
= 0z. we observe that:
(0, 0²) = L (0.) + (0c 0z)
P(X p1 ,0.) P (plim (..)
= (0.) + Σ P (p lon (,,0,) In
P (plm X,0,) (X 0.)
p
P (X, 0i)
# (0,) + Σ P (p lmn 0, In
(6)
p P(X, p lmn (b))
plo
= (0)) + Σ (p x,0
p
L (0))
Note that in general, P(X 12: 0,) Py | 0.) cannot be equal to P(X, 12, | 0.) since p is conditioned on both X and 8. Consequently, the factorized form does not equal the joint distribution. As result, for 0 = 0,, the functions (0 0c) and L(0) are not equal.
Our objective is to find the 0 that maximizes the function L(0). While 1(0)0.) and L(0) may not be equal for the current estimate 0 O.. it still holds that 1(0)0.) is bounded by L(0). Therefore, increasing 1(0)8, will also increase L(0). To achieve the greatest increase in L(0). the EM algorithm selects updated 0,41 that maximizes 100)0)):
Oz+1 = arg max (1(0)) (0.))
P(X Imm (p 1:m 8)
(7)
arg max
L ((,)) +
P( 1m X,0) In
P P(X 0.) P 1pm X, 0z)
pmn
23

---

Published as a conference paper at ICLR 2024
Ignoring terms which are constant w.r.t. A. the equation can be further deduced:
0,11 = arg max
P (p
(, In P(X "0p(",))(t"")) (0)
pin
Σ P (p (,) In P(X. 10m.p,00 Pppl:m000
arg max
P(ptin 0) P(O)
(8)
Σ P (p 1mn
(2 In P(x, p 1mm 10)
arg max
0
(x P(X, 1m [0)}
arg max
The alternate training algorithm thus consists of iterating (1) E-step: Determine the conditional expectation Ex,00 [In P(X.pl 0)} (2) M-step: Maximize this expression with respect to evident that end-to-end training for maximizing L(0) is not equivalent to two-stage learning I(010)) in the converged state, as verified in Eq. (6). Another advantage of two-stage learning is that provides framework for better estimation for both model and data parameters. This is further supported by the evidence presented in Fig. 8, where end-to-end training leads to sub-optimal solutions for pl:..
24

---

Published as a conference paper at ICLR 2024
K MORE VISUALIZATION AND ANALYSIS OF ATTENTION MAPS
We visualize attention maps from different heads in the last layer of the ViT backbone for multiple
datasets. The query position is selected either as the CLS token (in Fig. 11, Fig. 12, Fig. 13) or the local region on the edge of the foreground object (in Fig. 14, Fig. 15, Fig. 16, Fig. 17). We show the top 60% most attended patches in red for different attention heads. We observe that SPTNet-P and SPTNet attend more salient object regions, likely due to their ability to learn local invariance. Besides, SPTNet-P and SPTNet cover more diverse regions of the salient object regardless of query positions, illustrating more diverse attention patterns across heads. A similar phenomenon can be found in Stanford Cars and FGVC-Airerale Fig. 16 for the edge position, as well as in the generic dataset (e.g., ImageNet) in Fig. 13 and Fig. 17.
Fig. 11 and Fig. 12 for the CLS token and Fig. and
We also investigate the impact our proposed SPT by separating the prompt and backbone from SPTNet-P. We remove the prompt component from a well-trained SPTNet-P and visualize the attention maps by feeding raw images to the backhone only, referred as SPTNet-P (w/o prompt)'. Upon comparing the attention maps with and without patch-wise prompts, we observe that SPTNet-P with prompts (i.e., the 3rd column) exhibits clearer attention foreground objects compared to SPTNet-P without prompts (i.e., the 2nd column). This indicates that the learned prompts help elicit critical features for recognition. Additionally, when considering generic dataset like ImageNet- 100, we notice that there is no significant difference between the attention maps of models with and without prompts, resulting in an inferior performance boost compared to the fine-grained datasets.
SPTNet-P (w/o prompt)
SimGCD
SPTNet-P
SPTNet
Seen class
Unseen class
Figure 11: Attention visualization on Stanford Cars, for different attention heads in the last layer of the VIT backbone, by querying the CLS token.
To explore the difference of performance boost between generic and fine-grained datasets, transfer a pre-trained model from fine-grained dataset to generic one in Fig. 22. Since the overlap between ImageNet-100 100 and fie-grained datasets only contains various types of birds,
select some bird
samples from ImageNet- ImageNet- 100 and (2) trained on CUB. The model trained CUB appears to focus more on local regions, while the one trained on ImageNet- 100 pays more attention to the entire object. Furthermore, based on the quantitative comparison presented in Section 4.3 and Appendix C. we observe that SPTNet-P outperforms SPTNet on the ImageNet- CUB. Additionally, as depicted in Figure 22, we can observe that when more diverse attention is focused different regions of the object, it corresponds to improved performance. This indicates that
as inputs and visualize the attention map of two models: (1) trained on
dataset but performs worse than SPTNet on
25

---

Published as a conference paper at ICLR 2024
SPTNet-P (w/o prompt)
SimGCD
SPTNet-P
SPTNet
Seen class
Unseen class
Figure 12: Attention visualization FGVC-Aireraft for 12 different attention heads in the last layer of the VIT backbone, by querying the CLS token.
the ability of the model concentrate attention various object regions is beneficial for achieving better results. For instance, when comparing 'SPTNet-P' and 'SPTNet' trained on the ImageNet- 100 dataset, we observe that 'SPTNet-P" exhibits higher concentration on the objects compared 'SPTNet.' This observation aligns with the qualitative comparison, indicating that 'SPTNet-P" performs better than 'SPTNet' on ImageNet-100 100. Similarly, when considering 'SPTNet-P (from CUB)' and 'SPTNet (from CUB)' trained the CUB dataset, notice that 'SPINett' demonstrates a stronger focus on the objects compared to 'SPTNet-P'. This observation is consistent with the qualitative comparison, suggesting that 'SPTNet' outperforms 'SPTNet-P' on CUB.
26

---

Published as a conference paper at ICLR 2024
SPTNet-P (w/o prompt)
SimGCD
SPTNet-P
SPTNet
Seen class
Unseen class
Figure 13: Attention visualization ImageNet- 100, for 12 different attention heads in the last layer of the VIT backbone, by querying the CLS token.
27

---

Published as a conference paper at ICLR 2024
SPTNet-P prompt)
SimGCD
SPTNet-P
SPTNet
Sooo class
Unsoon claxs
Figure 14: Attention visualization on CUB, for 12 different attention heads in the last layer of the VIT backbone, by querying the point marked as green.
28

---

Published as a conference paper at ICLR 2024
SimGCD
SPTNet-P
SPTNet
Seen class
Unseen class
Figure 15: Attention visualization Stanford Cars, for different attention heads in the last layer of the ViT backbone, by querying the point marked as green.
29

---

Published as a conference paper at ICLR 2024
SimGCD
SPTNet-P
SPTNet
Seen class
Unseen cleess
Figure 16: Attention visualization FGVC-Aireraft, for 12 different attention heads in the last layer of the VIT backbone, by querying the point marked as green.
30

---

Published as a conference paper at ICLR 2024
SPTNet-P (wio prompt)
SimGCD
SPTNet.P
SPTNet
Soen class
Figure 17: Attention visualization on ImageNet- 100, for 12 different attention heads in the last layer of the VIT backbone, by querying the point marked green.
31

---

Published as a conference paper at ICLR 2024
SPTNet-P (w'o prompt)
SimGCD
SPTNet-P
SPTNet
Unseen
Figure 18: Attention visualization on CUB, for 12 different attention heads in the last layer of the ViT backbone, by querying the CLS token. SPTNet-P and SPTNet can automatically identify salient objects, likely due to their ability to learn local invariance. Upon comparing the attention maps with and without patch-wise prompts, more concentrated attention on the salient object compared to SPTNet without prompts (i.e., the 2nd column). This indicates that our learned prompts help elicit critical features for recognition.
observe that SPTNet with prompts (iee., the 3rd column) exhibits
32

---

Published as a conference paper at ICLR 2024
SPTNet-P
SimGCD
(w/o prompt)
SPTNet-P
SPTNet
Seen class
Unseen class
Figure 19: Attention visualization Stanford Cars, for 12 different attention heads in the last layer of the ViT backbone, by querying the CLS token. SPTNet-P and SPTNet can automatically identify salient objects, likely due to their ability to learn local invariance. Upon comparing the attention maps with and without patch-wiss prompts, we observe that SPTNet with prompts (i.e., the 3rd column) exhibits more concentrated attention on the salient object compared to SPTNet without prompts (i.e., the 2nd column). This indicates that our learned prompts help elicit critical features for recognition.
33

---

Published as a conference paper at ICLR 2024
SPTNet-P
SimGCD
(wlo prompt)
SPTNet-P
SPTNet
Seen class
Unseen class
Figure 20: Attention visualization on FGVC-Aireraft, for 12 different attention heads in the last layer of the VIT backbone, by querying the CLS token. SPTNet-P and SPTNet can automatically identify salient objects, likely due to their ability to learn local invariance. Upon comparing the attention maps with and without patch-wise prompts, we observe that SPTNet with prompts (i.e., the 3rd column) exhibits more concentrated attention on the salient object compared SPTNet without prompts (i.e., the 2nd column). This indicates that our learned prompts help elicit critical features for recognition.
34

---

Published as a conference paper at ICLR 2024
SPTNet-P
SimGCD
(w/o prompl)
SPTNet-P
SPTNet
Seeen class
Unseen class
Figure 21: Attention visualization on ImageNet- 100. for 12 different attention heads in the last layer of the ViT backbone, by querying the CLS token. SPTNet-P and SPTNet can automatically identify salient objects, likely due their ability learn local invariance. Upon comparing the attention maps with and without patch-wise prompts, we observe that SPTNet with prompts (i.e., the 3rd column) exhibits concentrated attention the salient object compared SPTNet without prompts (i.e., the 2nd column). This indicates that our learned prompts help elicit critical features for recognition.
SPTNet-P
SPTNet-P from CUR
SP
SPTNet frrm CUR)
Figure 22: Attention visualization for models trained on ImageNet- 100 and CUB by applying them
to bird images from ImageNet-106 100. The models trained on CUB are marked as '(from CUB)'.
35

---

Published as a conference paper at ICLR 2024
L BROADER IMPACTS
Our study is among the efforts extend the capability of AI systems from the closed world to the open world. Particularly, it will play a positive role in fostering next-generation Al systems with the capability of categorizing and organizing open-world data automatically. However, our method still has several limitations. First, though we have achieved encouraging results on the public datasets, the interpretability still needs improvement, as the underlying principles of how the decisions are made by the systems remain not crystal clear. Second, the cross-domain robustness is not satisfactory. as can be seen from the results on the setting of GCD with domain shifts, though our method has achieved the best overall results and new class discovery results, the performance still has significant room to improve. Additionally, in the vanilla GCD setting, methods typically rely on pre-trained model (e.g., DINO) feature extractor, which may inherit its drawbacks (e.g., discrimination and privacy issues).
36


Reviews:
[{"review_id": "9WnW4wG8s4", "reviewer": "Reviewer_bfKG", "summary": "This paper approaches Generalized Category Discovery (GCD) from an alternate perspective, which optimizes data and model parameters by prompt learning and finetuning, respectively. To this end, they propose a visual prompt learning method that learns data representation to better focus on object parts for generalizability. They achieve further performance improvement compared with the previous arts and investigate their approach with sufficient experimental analyses.", "questions": "The questions are naturally raised in the weaknesses section.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-organized to present their approach. They first tackle the previous GCD methods and redefine the problem with their own perspective. Then, they show their proposed methods based on their objective. It is easy to follow their objective that they propose spatial prompt tuning for better generalization on both seen and unseen classes.\n- The authors demonstrate the effectiveness of SPNet in their framework with sufficient experimental analysis. Their in-depth analysis shows that their proposed method clearly contributes to performance improvement.", "weaknesses": "- Although the authors explain the necessity of an alternative training strategy by referring to the EM algorithm, this reviewer did not reach the reasoning behind this explanation. This reviewer can agree that the authors demonstrate this training strategy empirically. It does not seem to be a specialized method for GCD. This reviewer recommends explaining in more detailed reasoning to choose this strategy if the authors have more reasons than the only empirical observation.\n- Even though the authors investigate the alternative training strategy by ablation study, this reviewer suggests presenting the visualization of representation during training at each switch to show the representation is enhanced as the objective of their approach.\n- As far as this reviewer’s understanding, their framework can be utilized for zero-shot learning tasks such as open-set recognition and open-vocabulary semantic segmentation, which evaluate the model on both seen and unseen classes, not only GCD. This reviewer agrees that their results show efficiency and efficacy. This reviewer believes that the experiment results on the closely related task strengthen their study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YKfPHA76rX", "reviewer": "Reviewer_Pson", "summary": "The paper addresses the Generalized Category Discovery (GCD) problem, which involves training on labeled images from seen classes to classify both seen and unseen class images. The authors propose an alternative training method for prompts and model parameters, specifically introducing a spatial prompt tuning method that adds image prompts on a patch-wise basis. Their approach achieves a significant performance improvement of about 10% over existing methods using fewer parameters.", "questions": "The paper could strengthen its reasoning and analysis linking the simple strategy proposed to the task of GCD. While it demonstrates a substantial performance increase with a straightforward approach, there is a lack of analysis or reasoning provided in the paper, making it difficult to directly correlate the performance improvements with their causes.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "- The method proposed is both simple and highly effective, appearing easy to implement.\n\n- Overall, except for the lack of method reasoning, the text is readable and has high-quality writing.\n\n- Comprehensive evaluations across various datasets and against state-of-the-art methods are conducted, with thorough ablation studies and analyses supporting the proposed method.", "weaknesses": "- The paper's main weakness lies in the lack of a detailed explanation of why the proposed method significantly improves GCD performance. While the alternative training of model and prompt, which enables more fine-grained augmentation, is acknowledged, the paper does not thoroughly describe how this relates to the GCD problem and why it leads to better performance. \n\n- The reasoning behind why SPTNet outperforms Global Prompt in GCD is supported only by experimental results and not by direct consideration of object parts, with insufficient evidence contrary to Vaze et al. (2022)'s findings. \n\n- The paper needs analysis of how alternative training induces changes and what benefits it has over end-to-end or completely separate two-stage learning strategies in terms of Expectation-Maximization (EM) learning aspects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tKqjIZFDiL", "reviewer": "Reviewer_E8qj", "summary": "- Generalized Category Discovery (GCD) is the problem of leveraging information from known classes in labeled data to automatically identify known and unknown classes in unlabeled data. Authors propose a two stage aporoach, named SPTNet for GCD. \n- SPTNet iteratively optimizes \"model\" parameters of large self-supervised networks and \"data\" parameters (i.e. prompt tuning methods). The former adapts the model to the data while the latter adapts the data to improve the model's capability of identifying categories. \n- Authors propose a novel Spatial Prompt Tuning (SPT) method that enables the model to focus on object parts an show remarkable improvements across several GCD benchmarks with very few additional trainable parameters.", "questions": "**Suggestions**:\n- The writing of the paper can be improved. Few sections (ablation, Section 3.2) need improvement. \n- Font of text in Fig.1 font is too small and I recommend increasing that. \n- Difference between SPTNet and SPTNet-P is not clear in the main paper. Readers have to look to supplementary to get clarity. Since this is an important part of the contribution, its better authors move Fig. 5 to the main paper. Also provide an example of computing total number of parameters of SPT in the supplementary. \n\n**Questions**:\n- What happens when you train Bahng et. al's Global prompts by increasing the number of parameters to match SPT? I understand its not a whole lot of parameters but how much would that change the performance of using Global prompts?\n- In the paragraph below **Stage 2: Fix p_{1:n}...**, authors present that they use the spatial prompts as augmentations. But by using learned prompts as augmentation, you are asking the network to be invariant to it, how does this help? The intuition behind using this and why would it improve the performance? How much does it improve the performance of contrastive loss? If that is a significant improvement, then that would be a good result for self-supervised literature.\n\n**Please address all the concerns and questions raised above for me to improve my ratings**", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The idea of optimizing the model and data parameters to improve GCD has some merit especially for discovery in fine-grained datasets. \n- The experiments section supports most of the claims (except a few discussed in weaknesses) made in the paper and show the merit of the proposed approach.", "weaknesses": "**Interpretability of SPT**:\n- Authors claim that SPT enables the model to focus on parts of objects, but the way it is designed, there are learnable parameters around each patch. Are the learned parameters, after convergence, sparse in nature? Are there more non-zero values around discriminative regions of objects and zero around patches belonging to background? The SPT setup in its current form is not very interpretable and the experiments certainly do not validate the claim that SPT is better than Bahng et al. because it enables the model to focus on object parts. \nCan this claim be validated/negated if, instead of around a patch, SPT is applied as learnable horizontal or vertical stripes in the image (maintaining the same number of parameters as the original SPTNet). If these experiments achieve the same performance, then the claim made by the authors is not true. I would like to hear the authors' thoughts on this.\n**Ablation experiments**:\n- Table-4 is not presented efficiently in my opinion and needs more attention. From rows 6 and 7, without Global prompting, SPT-S is better than SPT-P. But there is no experiment which uses Global + SPT-S with alternate training in the table. \n- I recommend adding one component at a time to the baseline, makes the table more readable than its current form. I believe most of the experiments are in there and all one would need is to rearrange the rows accordingly. \n- Also the accompanying text to Table-4 has some mistakes which make it harder to read the table. For example. Rows 5,6 compare the effect of alternate training. But in the text, authors explain that these two rows show the benefit of global prompting. Kindly make the table and text consistent and readable to improve reading experience.\n**Alternate training**: \n- In Fig. 3(a), for each k, are epochs adjusted accordingly? I believe this is important because authors report that with smaller k, the model underfits. But what about the experiment of training the model parameters to convergence, followed by training the SPT to convergence (k=1). This experiment is crucial to understand why the alternate training is required. Please provide the details of this experiment. \n**Qualitative results**:\nAuthors show attention maps in Fig. 4b and claim that with SPT, the heads cover the object. I do not see a difference between visualizations of SimGCD and SPTNet to be honest. Instead of showing 4 examples with all the heads, I recommend authors to be more specific and show exactly (by highlighting the region) what they want the readers to focus on. In its current form Fig. 4 does not add anything new to the discussion and can be removed entirely to make space for more important experiments (suggested above).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)