Federated Generalized Novel Category Discovery with Prompts Tuning
Authors: Lei Shen
Nan Pu
Zhun Zhong
Mingming Gong
Dianhai Yu
Chengqi Zhang
Bo Han
Venue: Accepted by TMLR ()

Abstract: Generalized category discovery (GCD) is proposed to handle categories from unseen labels during the inference stage by clustering them. Most works in GCD provide solutions for unseen classes in data-centralized settings. However, unlabeled categories possessed by clients, which are common in real-world federated learning (FL), have been largely ignored and degraded the performance of classic FL algorithms. To demonstrate and mitigate the harmful effect of unseen classes, we dive into a GCD problem setting applicable for FL named FedGCD, analyze overfitting problem in FedGCD in detail, establish a strong baseline constructed with state-of-the-art GCD algorithm simGCD, and design a learning framework with prompt tuning to tackle both the overfitting and communication burden problems in FedGCD. In our methods, clients first separately carry out prompt learning on local data. Then, we aggregate the prompts from all clients as the global prompt to help capture global knowledge and then send the global prompts to local clients to allow access to broader knowledge from other clients. By this method, we significantly reduce the parameters needed to upload in FedGCD, which is a common obstacle in the real application of most FL algorithms. We conduct experiments on both generic and fine-grained datasets like CIFAR-100 and CUB-200, and show that our method is comparable to the FL version of simGCD and surpasses other baselines with significantly fewer parameters to transmit.

Published in Transactions on Machine Learning Research (07 (07/2025)
Federated Generalized Novel Category Prompts Tuning
Discovery
with
Lei Shen
calshen@comp.hhbu.eeuu
TMLR Group, Department of Computer Science Hong Kong Baptist University
Nan Pu
wan.puu@umii..t
Department of Information Engineering and Computer Science University of Trento
Zhun Zhong
stunchonngll70@mmiil.co
School of Computer and Information
Hefei University of Technology
Mingming Gong
mingming- www.gggggngggnnnnnbneeeeooooe edu.ou
School of Mathematics and Statistics University of Melbourne
Dianhai Yu* Baidu Inc.
yudianhai@baidaa.om
Chengqi Zhang
Deportment of Data Science and Artificial Intelligence Hong Kong Polytechmic University
Bo Han"
bhanonl@comp..o hkbbu, edu. Isk
TMLR Group, Department of Computer Science Hong Kong Baptist University
Reviewed on OpenReview: https: openreveew. net/ forum? id- dVMESumMlo
Abstract
Generalized category discovery (GCD) is proposed to handle categories from unseen labels during the inference stage by clustering them. Most works in GCD provide solutions for unseen classes lata-centralized settings. However, unlabeled categorics possessed by clients, which common in real-world federated learning (FL), have been largely ignored and degraded the performance of classic FL algorithms. To demonstrate and mitigate the harmful effect of unseen classes, dive into GCD problem setting applicable for FL named FedGCD, analyze overfitting problem in FedGCD in detail, establish strong baseline constructed with state-of-the-art GCD algorithm simGCD, and design a learning framework with prompt tuning to tackle both the overfitting and communication burden problems in FedGCD. In our methods, clients first separately carry out prompt learning on local data. Then, we aggregate the prompts from all clients as the global prompt to help capture global knowledge and then send the global prompts to local clients to allow
Correspondence Dianhai Yu SyndianhaiQhaidh.ccmm and Han <bhanml@comp. hww.:mmppp.bb........k...

---

Published in Transactions on Machine Learning Research (07 /225)
0.66
M Mmn
0.8
Cliest 0
Client
Accuracy
0.62
Accuracy
0.6
Clieal 2
Chint
Cliest
0.58
0.4
Client
Client
0.54
Global mode!
0.2
Clinnt
50
100
150
200
50
100
150
200
50
100
150
200
(a) Global Model
(b) ACC on Local Clients
(c) Loss on Local Clients
Figure 1: Overfitting of fine-tuning the last block with simGCD on Cifar-100. Horizontal axis represents global training epoch in Federated Learning. Subfigure (a) shows the accuracy of global model during training process. Subfigure (b) and (c) report accuracy and loss on local clients, respectively.
access to broader knowledge from other clients. By this method, we significantly reduce the parameters needed to upload in FedGCD, which is common obstacle in the real application of most FL algorithms. We conduct experiments on both generic and fine-grained datasets like CIFAR-100 100 and CUB-200, and show that our method is comparable to the FL version of simGCD and surpasses other baselines with significantly fewer parameters to transmit.
1 Introduction
With advanced supervised learning techniques which have been greatly developed in recent years, machines can approach human performance in some types of tasks such as classification and segmentation. However, this enormous ability of machines relies heavily the annotations given by people and requires large amount of labeled data, which may be deficient in many real scenarios due to the varying data distributions/tase /tasks and the substantial labor required. Several fields like domain adaplation (DA) and domain generalization (DG) are proposed to tackle the situation where distribution changes, while all these methods fail to solve the problem where the tasks met by the models are varied. On the other hand, cross-task transfer learning Transfer (TL) strategies have gained great success and are widely applied with the labels of new tasks (Vaze et al. 2022p. However, transfer learning cannot help new tasks without the labels from new tasks. To deal with the problem of the lack of labels in new tasks, novel category discovery (NCD) and generalized category discovery (GCD) first learn a feature extractor with semi-supervised representation learning for making full use of both the labeled and unlabeled data. Then, a nonpaaramttric clustering or parametric classification method is applied to cluster the unlaboled data from new tasks (Wen et al. 2023a
Most previous works in FL assume that the data from clients are fully labeled and all classes in test daía appear in training data (McMahan al. (017).. However, this assumption is becoming harder to maintain for FL systems than in the data-centralized setting. Thercfore, considering GCD problems in the FL sctting is of great importance for real-world applications. Most previous work in GCD only provides solutions for unseen classes in the data-centralized setting, while similar situations are more common in FL as described in Section 2.4 In the meantime, FL setting brings more challenges to the GCD problem since both representatioe learning and classification are more demanding to learn with distributed or even heterogeneous data in each client, which is often encountered complicated in the data distributed scenario, where each client owns significantly less data compared with the centralized setting. In Federated Generalized Category Discovery (FedGCD), overfitting for labeled (old) class leads to performance descent in novel class. Moreover, we witness test performance gap between local training and test data due to data deficiency to train a complete large-scale model or even layer of it, as demonstrated in Figure [ We empirically show our method can effectively alleviate the overfitting problem in FedGCD by local prompt tuning with less but more effective trainable parameters in Section 5.6
FI.. We found the overfitting problem of GCD even worse and more
Morcover, the real application of FL always meets strict restrictions for communication costs. Our prompt tuning paradigm perfectly responds the requirement of reducing communication costs in the real employment of Fl. algorithms. Previous FedGCD requires training the whole model from scratch (Zhang et al. 2023a
2

---

Published in Transactions on Machine Learning Research (07/2025)
fine-tuning the last blocks in local clients (Zhang et al. 2023a Pu et al 2023a Previous FedGCD methods like FedoSST, and AGCT, require the transmission of the whole local models, which typically consist of millions of trainable parameters, together with auxiliary components such as local centroids and Gaussian mixture models. The training paradigm causes a great communication burden for both local clients and the contral server. Our method FedGCD-P exploits recent Parameter-Ffficien Fine-Tuning (PEFT) methods to tackle this challenging but important problem: how can we effectively learn both local and global presentation without modifying a large number of parameters, which costs unbearable communication bandwidth and demands the local client to great computation ability? Specifically, adopt prompt tuning (Jia et al, 2022) to achieve this goal: by only modifying a small number of prompt parameters, we extract useful presentation for local datasets, which significantly reduces communication and computation costs, making our method far more practical and resource-friendly.
We conduct experiments on commonly-used classification datasets such as CTFAR-100 and CUB-200 with a self-supervised pre-trained Vision Transformer HINO-ViT-B/IG demonstrating that our method obtains stale-of-the-arl performance with only 0.7% parameter transmission.
We summarize contributions as follows:
We first discover the overfitting problem of FedGCD, which significantly degrades the performance of previous FedGCD methods and causes heavy system overhead in federated systems, thus is of great. significance for improving FedGCD methods especially in the time of large models.
We propose FedGCD-P, a FedGCD method based on prompt tuning to handle the overfitting problem of FedGCD, and mitigate the communication burdens in generic FL, settings.
We empirically demonstrate the effectiveness of our method to be comparable or even outperform baselines with the transmission of significantly fewer parameters by solving the overfitting problem.
2
Related Work
2.1
Federated Learning
Federated learning (FL) was first proposed by (McMahan et al., 2017) as decentralized training paradigm without sharing training data. However, naive FedAvg faced many problems, two severe problems are (1) Data Heterogeneity: In the real application, different. distributions always occur in different clients, which violates the basic ii.d. (independent and identically distributed) assumptions of most machine learning algorithms. Especially when using FedAvg, heterogeneons data will cause severe client drift. To solve this problem, plenty of works have been proposed to restrict local updates either with a regularization term (Acar et al. 2020 Li et al. 2021a) or by correcting the local update (Karimireddy et al. 2020 to prevent them from diverge too much from global model; (2) Communication cost: FedAvg ask the full transmission of the whole model or gradients of it, which of thousands of parameter and non-trivial to communicate, A large number of parameter decoupling methods (Arivazhagan al.) 2019 Collins al.) 2021 Chen & Chao 2021 Jiang & Lin 2022) have been proposed to update only part of the model to reduce the communication cost of transmitting the model or gradients, while keeping similar even better results compared to update the whole model. However, due to communication and computation resource limitations, large-scale vision Transformers (ViT) are explored in FL with a relatively low frequency. few previous works (Che et al. 2023 Sun et al. 2023 Wei et al. 2024) focused on fully supervised and close-set settings, which is a typical setting in idealistic FL. However, when applying FL in the real world, the fully labeled data and close-set assumption are not applicable, therefore making those works impractical in such a situation.
2.2
Federated semi-supervised Learning
To tackle the practical issue of full-supervisod assumption above, there are alrcady some works attempting combine unsupervised or semi-supervised together with FL, to enable real-world FL even when label missing exists in local clients. Some of them (Albaseer al. 2020 Li et. al. 2023 utilize model trained labeled
3

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Sup Loss
Distillation Loss
Self-sup CL
Supervised CL
Sharpen
Gound Truth
Student Logits
Teacher Logits
Local Training
Local Fine-tuning
Local Prompts
Average
Boardcast
Prompt Aggregation
Global Prompt
CLS
MLP
CLS
MLP
CLS
MLP
ViT Encoder
ViT Encoder
ViT Encoder
Client 1
Client N
Figure 2: Overall Framework of FedGCD-P. The images with solid line frame labeled data, while the images with dotted line frame are unlabeled data. We fix the encoder of the Vision Transformer. At the beginning of one federated round, Global Prompt serves as the initial value of Local Prompts, while the Local Prompts are obtained from local fine-tuning, While at the end of the federated round, Local Prompts aggregate by average to get Global Prompts.
data to produce pscudo labels to carry out semi-supervised learning in FL. While some previous works (Zang et al. 2020 Long et al. 2021) use consistency regularization, a semi-supervised technique commonly used, aiming at minimizing the consistency loss between two representations of the same sample. There are also works like FedMatch (Jeong et al. 2020) combining pseudo label and consistency regularization to boost the performance of semi-supervised learning, However, all those works assume close-set setting, which is not completely practical in FL, e.g. some clients may have not observed data from some classes previously and thus are not possible to correctly label them.
2.3 Generalized Category Discovery
Due to the frequent happening of unlabeled class in training data and test data. Novel category discovery (NCD) (Han et al. 2019b) is first proposed for leveraging training data and cluster test data from unlabeled
2021 Zhong et al. 2021a Fini et al.) 2021 for NCD also have shown the ability to cluster unknown class data with knowledge learned from unlabeled classes. However, these works are all based on the assumption that all the test data comes from unknown classes, which is impractical most application scenarios. For example, (Han et al.] 2019b) involves a procedure to predict the number of new categories the unlabeled data. In this process, the estimated number of classes is considered non-overlapped, and overlapped classes in labeled and unlabeled data will bias the prediction of the number of unknown classes. Therefore, generalized category discovery (GCD) has been proposed by (Vaze et al. 2022 which extends the classes of test data to both unknown classes in unlabeled training data and classes already appeared in labeled training data. At the same time, Open-world Semi-supervised Learning (Open- (Open-world world SSL) (Cao et al.) 2022) proposed a very similar problem setting with GCD. Later, DCCL (Pu et al. 2023b) introduce Conceptional Contrastive Learning (CCL) to enhance representation learning by exploiting hicrarchical classification information, and
classes more reasonably. Several previous works (Han et al. 2019a Jia et al. 2021 Zhao & Han
their cluster methods follow the cemi-supervised cluster paradigm GCD (Vaze
2022 While simGCD
and GPC (Zhao al. 2023) propose to exploit parametric classification
Wen et al. 2023b with prototypes
4

---

Published in Transactions on Machine Learning Research (07/2025)
learning, respectively through direct optimization and EM-like optimizing Gaussian Mixture Models (GMM). PromptCAT. (Zhang et al. 2023b) and SPTNet separately introduce GCD methods based on soft prompt and spatial prompt tuning to solve GCD problems. However, all the above works discuss the GCD problem in a centralized setting, leaving the GCD problem in the Federated setting an aider-dicuusee field. To the best of our knowledge, only two previous works FedoSSI, (Zhang et al. 2023a) and AGCL (Pu et al. 2023a have discussed the GCD problem in the Federated setting, and our work first discusses the overfitting problem of
existing GCD methods and explore prompt learning in the GCD problem in Federated setting.
2.4 Federated GCD
In federated learning, data are decentrally scattered among all clients, and it is hard to require all participants to label local data; therefore, a great number of unlabeled data exist in FL systems. Ignoring these unlabeled data will greatly damage the performance of federated models when faced with the same class data as the unlabeled data and misclassify the unlabeled new class data. into the labeled classes. To leverage these unlabeled data properly and correctly cluster the new task data, formulating federate generalized category discovery (FedGCD) and designing corresponding generalized category discovery methods to solve the FedGCD problem is necessary and beneficial. However, to the best of our knowledge, there are only two previous works FedoSSL (Zhang et al. 2023a) and FedGCD (Pu et al., 2023al) attempted to solve the GCD problem under the federated setting. Below we discuss them in detail and compare their differences with our work.
FedoSSL. In FedoSSL (Zhang et al. 2023ap, local feature extractors trained with unbiased semi- supervuper learning loss in local clients and aggregate all local feature extractors to obtain a global one. However, a traditional CNN architecture is adopted by FedoSSL, which may face the problem of not having enough representation ability due to the lack of training data in the federated setting. Different from FedoSST, to enhance the representation ability of the global federated learning model, we utilize prompt learning with joint training of large-scale pre-trained visual transformer (ViT), aiming at better discovering semantic clusters in the unlabeled data. As for the clustering of unlabeled data, a protolype learning method is exploited to carry out clustering in the server in FedoSSL, The way of clustering with uploading local centroids may be exposed to the risk of privacy leakage, while replacing Sinkhorn-Kinn Knop based clustering with general k-means method may witness a performance drop as stated in FedoSSL (Zhang et al., 2023a).. Therefore, with a more representative feature extractor to obtain better semantic features, we choose to take advantage of k-means to cluster the unseen categories without auxiliary information.
AGCL. AGCL (Pu et al. 2023a] attempts to share information through the Gaussian mixture model (GMM), however, such methods are faced with certain privacy risks, since a generative model trained with local data may reveal private information about the clients. What's 's more, AGCL (Pu et al.) 2023ab fine-tunes and transfers the last block of the ViT, which costs large comnunication overhead as Table 2 shown and makes the method unpractical for closed-source pre-trained model following the trend in current Large Language Models, c.g. GPT-4, Clande 3.5.
3 Problem Definition
Before introducing the FedGCD setting, we introduce the formulation of the GCD problem (Vaze et al., 2022), FL setting (McMahan et al.) 2017 Smith et al. 2017) and present some preliminaries. Finally, we introduce the FedGCD setting defined in Pu et al. 2023a).
3.1 Generalized Category Discovery
Our GCD setting follows \aaze et al. 2022). Specifically, we assume that the training datasct D Dt U Da comprises two subsets: a labeled set D, {a, J C X, x 34, with its label space Yi Cseens and an unlabeled set D,, = {a, N2 C X,, with its underlying label space Va = = Cseen UCUnscen- Here, C, C Cseen: and CÜnseen denote the label set for All, Seen, and Unseen classes, respectively. Following (Vaze al. 2022 assume the number of the All classes |C| is known.
5

---

Published in Transactions on Machine Learning Research (07 (025)
3.2
Federated Learning
FL algorithms can be split into Generic FL (GFL) and Personalized FL (PFL) according to the place where the inference happens, and they focus on different optimization goals with varied application scenarios. Below we briefly introduce both kinds of FL. In this work, we mainly focus on the Generic FL setting.
3.2.1 Generic FL
The GFT, aims to make Nr clients collaboratively learn a global model parameterized as 0 used conduct prediction on the server. Fach client has its local training dataset, we denote the training dataset of client n as D Thus, the local objective function (,) on client n is also different from client to client. The global optimization object of GFL is defined (Karimireddy et al. 2020
N1.
N1.
min Ca(0)= = Σ (0) = Σ Pm E, EDE (((,
(1)
mmm
where (In, Un) E D is the sample from D (n) is the prediction of the trained model, d is the number of model parameters, > 0 and Σ Pn = 1. Usually, Pm n's samples in local training dataset and N - No. The global model refers the model obtained from optimizing the GFL objective.
Na N, where No denotes the number of client
3.2.2 Personalized FL
While the object of GFL is to learn a model suitable for all training distribution, the PFL requires less from single model and chooses to learn multiple personalized models fitting on local datasets separately: (Chen
& Chao 2021
et al. 2021b):
N2 Σ Pr E.. ED (()( ( R(2, (o.,,.,...,
min
(2)
0,91 ONz
where Q is the collaboration scheme of clients, R is the regularizer (Chen & Chao 2021 both of which vary from algorithm to algorithm, The obtained personalized models are named as Personalized Models (PMs).
3.3 Federated Generalized Category Discovery
From the introduction of problem formulation of NCD and FL, We now introduce the formal formulation of FedGCD (Pu et al. 2023ah. We assume there are Nr. clients in the FL system, and client maintains local model 6". A global model is initialized in two ways according to the parameter part, (1) randomly initialized, (2) initialized with pre-trained model (Caron et al. 2021) like in our experiments. After initialization, the global model is distributed to all clients, Le. Vn E (0, I,..,
,II..,,,,,, Nr,, 0g of. Given the local labeled training No. E X x 5% with the corresponding data set XL,, and label set Doh and the local unlabeled training dataset D = {I, b))
dataset on n-th client D (( w
No.x E Nn client n is supposed to train its local model 0", In order to keep the same setting with FedGCD Pu et al. 2023a), we also assume the label space of each client cannot be the same as or include that another client, i.e. for client and j. + j. y and yj, y V # y. or D/A Following the common FL setting (McMahan et al. 2017), we simulate and control the data and label heterogeneity that often exists in both real-world FL and GCD applications, with the Latent Dirichlet Sampling (Dir) partition method (Hsu et al., 2020). In our setting, mainly consider the Generic FL setting where a global model is used for inference by all clients in the FL system.
4 Methodology
4.1 Local Prompt Tuning
From the observation above, we think about what may cause the performance gap between training and test data and the reason why similar phenomenon hasn't been observed in centralized setting. The assumption

---

Published in Transactions on Machine Learning Research (07 (07/2025)
made by us is that the local clients own significantly fewer data samples in their number compared with centralized settings. Moreover, less diversity due to the notorious problem called data heterogeneity in FL significantly increases the risk of overfitting the training on local clients. To alleviate the effects brought by those problems, we propose to apply prompt tuning for local training. To effectively tune the local prompt, we adopt the loss style from state of art GCD algorithmi (Wen al. 2023b). To further boost prompt luning on local clients, we also propose a loss applying to the prompt itself. As shown in Figure 2 our overall learning objective composed of four parts. We first introduce supervised classification loss and self-distillation loss, together with prompt-tuning to FedGCD setting.
Loss on [CLS] token. Fine-tuning of Vision Transformers on downstream classification tasks always relies on [CLS] token to utilize supervision information. Therefore, for prompt luning, we follow previous works to output class logit with a multilayer perceptron (MLP) as classification prediction and pseudo label, and apply cross-entropy loss for comparing model prediction and label together with the pseudo label. Our loss function on MLP consists of two parts: supervised classification loss and self-distillation loss with entropy regularization. The denotation follows those in the Section 3.3 For the supervised part, given p(a;) is the output logit of MLP for local sample 2; D4, supervised classification loss is defined as
Let No
Σ CE(sottmxx(((r))).uk) xED!
where is the class label of the sample V;. CE(s)) denotes cross-entropy loss. As for the self-distillation part, it involves cross-entropy of put logit of x; sharpened with a relatively low temperature тs and logit of different views (random argumentations) x with a sharper temperature Te as the pseudo label.
LSD(D!)
Σ
CE(sotmnax(((z))/t... (ton:///)))///)toooommmmmt/c))))
(4)
N
where A.g(D!) denote the augmented local training data.
After introducing the loss function associated with the output of MLP, we also apply supervised and self- supervised contrastive loss for better representation ability with prompts. For supervised contrastive loss, class labels are used to identify if two samples are from the same class instead of providing direct classification guidance, and this loss aims to reduce the distance of samples from the same class in feature space and increase that of samples from different classes, the specific form as follows:
Lsup- Coup-con (D%)
NE Σ Σ nd a;ED #yEN(a.)
exp(21; 2,)
(5)
Ex.ED. exp(zi, 2n)
where N (x;)) the set that all samples own the same label y with Tr, ie. N(x;)) = fo E D4 y = yi 2s is the feature of 24, As for the self-supervised contrastive loss, we define it the following form:
exp(zr, 2/)
Loel (D.) = NI
Σ
Σ EAngtoo) x
(6)
Ex.EDS exp(zi,
D.
where of is the feature of augmented image 2/ With the four parts of loss, we have the total loss function below:
L(Dİ)) = X (D)) + Lssup-com com (D+)) + - A) (LsD(D!)) + Luup con (D+))
(7)
where À is a hyper- parameea controlling the importance of supervised and unsupervised learning objectives.
On each client participating in current. training, the overall learning objective is in the form of equation 7
7

---

Published in Transactions on Machine Learning Research (07 (025))
Loss on [PMT] tokens. After implementing the loss [CLS] token, we attempted to carry out prompt tuning local clients, surprisingly finding prompt tuning instead of fine-tuning the whole block effectively relieves the overfitting problem. We assumed that fine-tuning the prompt with only the learning objective of the output of [CLS] token was not sufficient for full utilization knowledge from local data, especially the labeled data. Motivated by this assumption, we thought the lack of supervision and self-supervision on the output of prompt tokens was the root of the problem. Therefore, we added auxiliary MLP for [PMTT tokens to assist the supervision from labeled samples and the self-supervision from all samples, our training objective for PMT] tokens follows the above overall learning objective in equation 7 After applying Loss [PMT) tokens, the total loss turns into:
Liatal (D)) = L(D()) LPMT (D
where (PM)(DD)) is the loss applied to the outpl of MLP for [PMT] tokens. However, our empirical results show the learning objective prompt tokens doesn't significantly improve the representation ability of prompts, which will elaborate more in Section 5.3
4.2 Global Aggregation
We only aggregate the prompt part of the Vision prompt transformer. In practice, this aggregation scheme greatly reduces the communication burden of both clients and the server, logether with the storage burden of servers in FL systems with plenty of clients. Our aggregation scheme for local prompts from participating clients follows the classic Federated algorithm FedAvg (McMahan et al. 2017). which averages the prompt from clients who lake part in the current training round by its number of training samples.
NL
0
where 0441 is the global model parameters at t++1 round, 0f is the local model parametes of n-th client at round. After aggregating the prompt parameters, the server sends the global prompt back to clients, clients update their local prompt with the global prompt. Through this aggregation process, clients benefit from the knowledge of other clients.
5 Experiments
Table 1: Comparison of different methods on CIFAR-10, CTFAR-100, CUB-200, and Oxford-Pet datasets. f: As ACGL (Pu et al.
2023a) official implementation hasn't 't been release, collect the origin result of ACGL
from Pu et al. 2023a * FedoSSL failed to converge on CIFAR-10 and CIFAR-100 in our setting following
their typer-parameters parameters, and we rerun them with modified parameters but still failed.
Method
CIFAR-10 (%)
CIFAR- 100 (%)
Imagenct-108 (%)
CUB-200 (%)
Oxford-Pel (%)
All
Seen Unseen
All
Seen
Unseen
All
Seen
Unseen
All
Seen
Unseen
All
Seen Unseen
clicuts (full participation)
Fod-GCD
93.3
96.3
91.3
63.6
63.4
63.8
77.3
84.7
70.0
49.1
52.4
45.9
84.6
$2.8
$6.4
Fod-SimGCD
86.4
92.3
80.4
66.0
62.1
69.8
76.3
88.7
63.8
63.0
63.4
62.5
82.6
$1.9
83.3
#
65.6
65.5
65.7
29.6
33.2
26.0
16.6
59.1
34.0
52.0
56.5
47.7
76.0
81.5
67.0
AGCL
82.5
83.4
82.2
51.2
54.0
T3.1
78.1
67.0
53.1
52.9
51.2
81.4
$2.0
50.7
FedGCD-P
94.0
94.1
69.3
71.3
67.2
87.3
70.8
61.3
63.5
39.2
83.1
85.4
50.7
Table 2: # parameters needed to transmit versus methods on CIFAR-100.
Method
Fed-GCD
Fed-SimGCD
FedoSSL
AGCL
FedGCD-P
# Parameters
7.1M
7.1M
7.9M
8.7M
55.3k
8

---

Published in Transactions on Machine Learning Research (07/ (225))
5.1 Experimental Setup
Datasct. To fairly evaluate the performance of FedGCD methods, we conduct comparison for all methods
on two commonly-used generic image classification datasets (i.e., CIFAR-10 [Krzhhvskyy et al. 2009). CIFAR-
100 Krizhevsky et al., 2009)) and two fine-grained image classification datasets (i.e., CUB-200 (Welinder
let al. 2010), Oxford- Pet (Parkhi et al, 2012). We leverage the Latent Dirichlet Sampling (Dir) partition method (Hsu et al. 2020 to split the training set into N² subsets, cach of which is stored in local clients as
its local training data. After partitioning the whole training dataset into heterogeneous subsets, we sample subset of half the classes as "Seen" categories in the original training set, and only local data from these classes are treated potentially labeled data. In these potentially labeled data, 50% of instances of each labeled class are randomly sampled to form the labeled set. The remaining local data are treated as unlabeled data when training. We set N = 5 and partition the whole dataset into local data with Dir(0.05) in our main experiments and explore more settings in Section 5.5 and Section 5.4
Evaluation Protocols. Different from GCD who use the sami-supervised k-means evaluate the final performance of models, we evaluate the global model performance with clustering accuracy (ACC) of prediction directly with k-means for all methods except for FedoSSL. We choose k-means because (1) it is more efficient in terms of running time; (2) differences bet ween semi-supervised k-means and k-means have little effect on the comparison of different GCD methods. We follow the original evaluation method of FedoSSL due to its contribtton related to its novel classifier calibration module. The cluster accuracy is defined as ACC= = M M 13): p" (d)),, where y denotes the ground truth label, and &h denotes the prediction given by k-means on extracted features, p-()) denotes the optimal permutation maximize the ACC of the overall predictions. Besides overall clustering accuracy testing on 'All' classes, we also measure the clustering accuracy for "Seen" and "Unseen" categories individually. Following common practice in GCD, we apply k-means on the feature outputted by (CT.S) token of VIT to get the final clustering results.
Baselines and implementation details. We compare FedGCD-P with two strong baselines adapted from centralized GCD. GCD (Vaze et al., 2022) is the most classic algorithm designed for GCD problems, while simGCD (Wen et al., 2023b) is the seate-of-the-att method for GCD in the centralized setting. Specifically, we utilize the original GCD methods in local client training to fine-tune the last block of ViT in both methods, and aggregate the trained local model with FedAvg MMcMahan et. al. 2017/. We also compare FedGCD-P with existing FedGCD works FedoSSL (Zhang et al. 2023a) and AGCL (Pu et al.) 2023a).. As ACGL (Pu 2023a official implementation hasn't 't been released, we collect the bascline result of ACGL from (Pu ct al.
ct al.
2023a We set the batch size to 128 for all methods and datasets. All methods are optimized by SGD with a momentum of 0.9 and weight decay of 1 x 10-4 for 200 epochs with a cosine annealing schedule starting from a learning rate of 0.1. All methods use dino-vitb16 (Caron et al. 2021) as the backbone following the previous setting in GCD works (Vaze et al.) 2022 Pu al.) 2023a). The hyper-parameters X controlling weight. of supervised and self-supervised learning are set to 0.35 for Fed-GGD, Fed-SimGCD, and FedGCD-P. During training process, we define the initial random seed as 2023, then the random seed is increased by every global round, while when federatedly partitioning the datasets, the random seed is set as O. Our local training epoch is set as 1. Other parameters and implementation details follow the official implementation of baselines.
Hardware and Software Configuration. We conduct experiments using NVIDIA V100 32GB GPU, 20 Intel(R) Xeon(R) Silver 4114 CPUs @ 2.20GHz. The operating system is Oracle Linux Release 6. The pytorch version is 1.12.1. The numpy version is 2.0.1. The cuda version is 11.8. All our experiments are carried out in single GPU to avoid unexpected influence contrastive learning of splitting batch data to multiple GPUs
UEK
5.2 Experiment Results
From Table [ we observe that FedGCD-P significantly outperforms other baselines on the CIFAR-10 and CIFAR-100, while FedGCD-P ranked second in both CUB-200 and Oxford-Pet with slight with the best results. What's notable for the advantage of FedGCD-P is its significantly lower communication cost compared with other baselines. As shown in Table 2 FedGCD-P transmits only 0.70 0.78% of parameter compared
9

---

Published in Transactions on Machine Learning Research (07 (07/2025)
with basclines. This advantage is in the requirement of real-world FL applications where many clients like edge devices may be short of communication bandwidth.
5.3 Ablation Study of FedGCD-P
We attempt to explain the effects of the components in our methods, with analysis towards different loss terms of FedGCD-P: (1) self-supervised contrastive loss(self- CL)); (2) supervised contrastive loss (sup-CL); (3) self-distilaation loss with entropy rogularization (SD); (4) supervised classification loss (CLS); ; (5) prompt- tuning (PMT): If the column "PMT" disabled, we only fine-tune the last block of ViT; while enabling "PMT" is to fine-tune [PMT] tokens (not include [CLS] token) instead of last block; (6) loss on prompt tokens (L-PMT). We have the following observation and conclusion: (1) imbalance between supervised and unsupervised learning has catastrophic effect on FedGCD methods, which may lead to failure of convergence, e.g. (1) and (3) columns of Table 3 we believe this unique issue with FedGCD problems is associate with its decentralized setting and great heterogeneity make naive FedAvg aggregation unstable for the convergence of the global model, so we claim a more robust aggregation scheme required to help more robust convergence of FedGCD methods, and believe to be promising future direction for FedGCD problems; (2) we show components of our method FedGCD-P except for I-PMT vital for achieving the best performance FedGCD problems, especially the improvement brought by introducing prompt tuning, which we will
experimentally verify later in Section 5.6
Table 3: Ablation Study of FedGCD-P.
CIFAR-100 (%)
Oxford-Pet (%)
self-CL
sup-CL
SD
CLS
PMT
L-PTT
All
Seen
Unseen
All
Seen
Unseen
X
X
55.0
56.3
53.7
75.7
66.5
85.6
/
X
X
X
X
63.6
63.4
63.8
84.6
82.8
86.4
J
/
X
X
X
56.5
55.6
57.4
78.9
68.2
90.2
V
X
66.0
62.1
69.8
82.6
81.9
83.3
(5)
/
V
X
69.9
68.4
71.5
85.9
86.9
84.9
V
V
69.3
71.3
67.2
83.1
85.4
80.7
/
x
V
46.7
46.9
46.5
79.4
78.8
80.1
5.4 FedGCD-P under different degrees of heterogeneity
In this section, we discuss a significant problem in FL named data heterogeneity, and empirically show the robustness towards different degrees of heterogeneity of baselines and our method. From Table we found besides the abnormal convergence of FodoSSL, all basclines sufferod with the increase of heterogeneity from Dir(0.2) to Dir(0.05), especially Fed-SimGCD. The performance of FedGCD-P is at medium level compared with other baselines.
5.5 FedGCD-P under different numbers of clients
In this section, test the scalability of different methods in FedGCD setting. We compare the performance of all methods under the FL system with 5 clients and 10 clients. In both settings, all clients participate in the local training and upload their local models for aggregation. From Table 5 we see the growth of the number of clients only slightly influences the performance of FedGCD-P and witness a relatively balanced decrease instead of in Fed-SimGCD, whose 'Seen' is increased while 'Unseen' greatly decreases. The phenomena show the supervisod and unsupervisod learning FodGCD-P is relatively balanced. We attributo the good scalability the alleviation of the overfitting problem in local data, which deteriorates with the increase of the number of clients and decrease of data samples each client.
10

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table 4: Comparison of different degrees of heterogeneity Dir(0.2) and Dir(0.05), Nr = 5, on dataset CIFAR-100.* we found FedoSSL cannot converge on CIFAR-10 and CIFAR-100 in our setting following their and we attempt rerun them with modified parameters but still cannot converge normally.
Hcterogencity
Dir(O.2)
Dir(0.05)
All Seen
Unseen
All
Seen
Unseen
5 clients (full participation)
Fed-GCD
66.0
64.6
67.3
63.6
63.4
63.8
Fed-SimGCD
71.6
69.9
73.3
66.0
62.1
69.8
FedoSSL* *
30.1
26.3
33.9
52.0
56.5
47.7
AGCL
56.1
56.8
55.3
54.2
54.6
54.0
FedGCD-P
71.7
72.6
70.8
69.3
71.3
67.2
Table Comparison of different methods on Nr and N₂ = 10, on dataset CTFAR-100, with Dir(0.05). *:
found FedoSSL cannot converge on CIFAR-100 100 in our setting following their hyper-parameters and we allempt to rerun them with modified parameters but still cannol converge normally.
#clients
10
All
Seen
Unseen
All
Seen
Unseen
Fed-GCD
63.6
67.1
63.8
62.7
64.7
60.6
Fed-SimGCD
66.0
62.1
69.8
63.6
67.7
59.4
FedoSSL*
52.0
56.5
47.7
45.8
44.0
47.6
AGCL
54.2
54.6
54.0
52.2
53.6
52.4
FedGCD-P
69.3
71.3
67.2
69.0
71.1
66.9
5.6 Prompt can effectively alleviate overfitting
We empirically testify our motivation that prompt tuning can be utilized to alleviate the overfitting problem of local training. From Figure 3 other baselines' performances constantly decrease with the communication round after short period of increase. It clearly observed that FedGCD-P prevents the overfitting of the global model.
0.8
0.6
0.6
Accuracy
Accuracy
0.4
FedOCD-P P
FedGCD-P
Fed Red-SimGCD
Fod-SimGCD
0.2
Ped-GCD
Fed-GCD
FedoSSL
0.2
FodoSSL
0.0
0
50
100
150
200
50
100
150
200
(a) CIFAR-10 100
(b) CUB-200
Figure 3: Performance methods on CIFAR-100 and CUB-200 with Dir(0.05), 5 clients.
6 Conclusion
In this paper, we introduce and formulate a foderated generalizod category discovery setting, We observe and dive into the common phenomena of overfitting problems in this FedGCD setting. We propose a global FedGCD framework based a prompt tuning framework to solve the problem. We also show some
11

---

Published in Transactions on Machine Learning Research (07 (0/5))
intermediate experiment results on both generalized and fino-grained datasets, and empárically testify claim that can significantly alleviate overfitting by fine-tuning prompt instead of ViT itself. For future work, we believe more advanced PEFT methods like lora can be incorperated into FedGCD-P to more effectively alleviate the overfitting problem due to deficient samples in local training.
Acknowledgments
LS and BH were supported by the NSFC General Program No. 62376235, CCF-Baidu Open Fund, HKBU Faculty Niche Research Areas No. RCCENNAA-G//2223///S// and HKBU CSD Departmental Incentive Scheme.
References
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Foderated learning based on dynamic regularization. In International Conference on Learning Representations, 2020.
Abdullatif Albaseer, Bekir Sait Ciftler, Mohamed Abdallah, and Ala Al-Fuqaha. Exploiting unlabeled data in smart cities using federated edge learning. In 2020 International Wireless Communications and Mobile Computing (IWCMC), pp. 1666-1671, 2020. doi:
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers. arXiv preprint arXiv:1912.000 001180 2019.
Kaidi Cao, Maria Brbic, and Jure Leskovee. Open-workd semi-supervised learning. International Conference on Learning Representations (ICLR), 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Tianshi Che, Ji Liu, Yang Zhou, Jiaxiang Ren, Jiwen Zhou, Victor Sheng, Huaiyu Dai, and Dejing Dou. Fedorated learning of large language models with parameter-effcient prompt tuning and adaptive optimiza- tion. Proceedings of the 2023 Conference on Empirical Melhods Natural Language Processing, pp- 7871-7888, 2023.
Hong- You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning for image classification. In International Conference Learning Representations, 2021.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representations for personalized federated learning. In International conference on machine learning, pp. 2021.
PMLR,
Enrico Fini, Enver Sangineto, Stéphane Lathuilière, Zhun Zhong, Moin Nabi, and Elisa Ricci. A unified objective for novel class discovery. In Proceedings of the IEEE/CV International Conference on Compuler Vision, pp. 9284 9292, 2021.
Kai Han, Sylvestre-Alvis Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Automatically discovering and learning new visual categories with ranking statistics. In International Conference Learning Representations, 2019a.
Kai Han, Andrea Vedaldi, and Andrew Zisserman. Learning to discover novel visual categories via deep transfer clustering. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 8401 8409, 2019b.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Federated visual classification with real-world data distribution, 2020.
12

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Wonyong Joong, Jachong Yoon, Eunho Yang, and Sung Ju Hwang. Federated semi-supervised learning with inter-client consistency & disjoint learning. In International Conference on Learning Representations, 2020.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference Computer Vision, pp.
Springer, 2022.
Xuhui Jia, Kai Han, Yukun Zhu, and Bradley Green. Joint representation learning and novel category discovery on single-and multi-modal data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 310-619, 2021.
Liangze Jiang and Tao Lin. Test-time robust personalization for federated learning. In The Eleventh International Conference on Learning Reyresentations, 2022.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In Inter- national conference on machine learning, pp. 5132 5143. PMLR, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ming Li, Qingli Li, and Yan Wang. Class balanced adaptive pscudo labeling for federated semi-supervised learning- In 2023 IEEE/CVE Conference on Computer Vision and Pallern Recognition (CVPR), pp. 16201. TEFF, 2023.
Qinbin Li, Bingsheng He, and Dawn Song. Modeelcontrrativee federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10713-10722, 2021a.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In ICML, 2021b.
Zewei Long, Jiaqi Wang, Yaqing Wang, Houping Xiao, and Fenglong Ma. Fedcon: A contrastive framework for federated semi-supervised learning. arXiv preprint orXim:2109.04593 2021.
Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication- efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273-1282. PMLR, 2017.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3498-34880 3505, 2012, doi: 10.1109/CVPR.20 6248092.
Nan Pu, Zhun Zhong, Xinyuan Ji, and Nicu Sebe. Federated generalized category discovery. arXio preprinl arXiv:2305.14107 2023a.
Nan Pu, Zhun Zhong, and Nicu Sebe. Dynamic conceptional contrastive learning for generalized category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7579-7588, 2023b.
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Amect Talwalkar. Federated multi-task learning. Advances in neural information processing systems, 30, 2017.
Jingwei Sun, Ziyue Xu, Hongxu Yin, Dong Yang, Daguang Xu, Yiran Chen, and Holger R Roth. Fedbpt: Efficient federated black-box prompt luning for large language models. 2023.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7492-7501, 2022.
Guoyizhe Wei, Feng Wang, Anshul Shah, and Rama Chellappa. Dual prompt tuning for domain-aware federated learning, 2024. URL http:////ppppeeeeeeeeeee////tttp
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-uesd birds 200. 09 2010.
13

---

Published in Transactions on Machine Learning Research (07/2025)
Xin Wen, Bingchen Zhao, and Xiaojuan Qi. Parametric classification for generalized category discovery: A baseline study. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp- 16590 16600, 2023a.
Xin Wen, Bingchen Zhao, and Xiaojuan Qi. Parametric classification for generalized category discovery: A baseline study. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 16590 16600, 2023b.
Jie Zhang, Xiaosong Ma, Song Guo, and Wenchao Xu. Towards unbiasod training in federated open-world semi-supervised learning. arXiv preprinl. arXiv:2305.00771, 2023a.
Sheng Zhang, Salman Khan, Zhiqiang Shen, Muzammal Naseer, Guangyi Chen, and Fahad Khan. Prompical: Contrastive affinity learning via auxiliary prompts for generalized novel category discovery, 2023b.
Zhengming Zhang, Zhewei Yao, Yaoqing Yang, Yujun Yan, Joseph E Gonzalez, and Michael W Mahoney. Benchmarking semi-supervised federated learning, arXiv preprint arXie:2008.11964 17(1):2, 2020.
Bingchen Zhao and Kai Han. Novel visual category discovery with dual ranking statistics and mutual knowledge distillation. Advances in Neura Information Processing Systems, 14:22982-22999) 2021.
Bingchen Zhao, Xin Wen, and Kai Han. Learning semi-supervised gaussian mixture models for generalized category discovery. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp.
2023.
Zhun Zhong, Enrico Fini, Subhankar Roy, Zhiming Luo, Elisa Ricci, and Nicu Sebe. Neighborhood contrastive learning for novel class discovery. In Proceedings of the LEEE/CVE conference on computer vision and pattern recognition, pp. 10867 10875, 2021a.
Zhun Zhong, Linchao Zhu, Zhiming Luo, Shaozi Li, Yi Yang, and Nicu Sebe. Openmix: Reviving known knowledge for discovering novel visual categories in an open world. In Proceedings of the IEEE/CV CVF conference on computer vision and pattern recognition, pp. 9462-9470, 2021b.
14


Reviews:
[{"review_id": "wjyFqgKRd6", "reviewer": "Reviewer_hQ5H", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n* The problem is very interesting.\n\n* Prior work section is comprehensive.\n\n* Prompt tuning (in general, PEFT methods) reduces the computation cost in FL.\n\n* The authors conduct thorough ablation experiments to highlight the contribution of each component,\n\n\n**Weaknesses**\n\n* The novelty is very limited. The paper uses prompt tuning and self-distillation to improve the performance and avoid overfitting.\n\n* Prompt tuning is only one of the PEFT methods. The paper does not compare with other PEFT methods, such as LoRA.\n\n* Is the unseen data really unseen? All the ViT models have already been trained on common datasets, so it is difficult to judge whether the improvements come from learning the unseen data or reducing the overfitting.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses generalized category discovery (GCD) within the federated learning (FL) setting, where clients possess unlabeled data from unseen categories. The authors propose FedGCD-P, a framework utilizing prompt tuning. This method involves clients performing prompt learning locally and then aggregating these prompts globally, allowing for efficient knowledge sharing and improved performance on both seen and unseen categories, as demonstrated through experiments on various datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Page 1 refers to an appendix. However, the paper does not have any appendices.\n\n* The federated setting does not seem very realistic. All experiments are with only 5 - 10 clients (with a full participation ratio), which is very small for FL.\n\n* The overfitting problem is something already known. The model can easily overfit wherever we have small datasets compared to the number of trainable parameters. \n\n* Some of the training details, such as seeds, are missing.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RskUeRSIgp", "reviewer": "Reviewer_MGkK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## **Strengths**\n\n- The paper identifies and articulates the overfitting issue in FedGCD settings, which is a under-discussed problem.\n- The proposed prompt-based method is experimentally validated and shows performance improvements.\n\n---\n\n## **Weaknesses**\n\n1. **Typographical and Citation Issues**  \n   - There are several minor typographical errors that need attention. For instance, the phrase “as described in Appendix ??” lacks a proper reference and breaks the reading flow.\n\n2. **Writing and Clarity**  \n   - Some sentences are overly complex and difficult to follow. For example, the sentence starting with *“Similar to GCD...”* on page 2 is long and nested—splitting it into shorter sentences would help clarity.  \n   - There are inconsistencies in notation, particularly in Sections 3.1 and 3.3. Label sets are referred to alternately as **$\\mathcal{L}_n^L$** and **$Y_l$**, which can be confusing.  \n   - Font styles such as italic and calligraphic are used inconsistently (e.g., **$\\mathcal{L}_I^L$** vs. **$L_n^L$** in Section 3.3). Please ensure consistent formatting throughout.\n\n3. **Missing or Ambiguous Information**  \n   - Figure 1 lacks context. It’s unclear whether subfigure (a) illustrates standard GCD or the FedGCD setting. Providing this clarification, along with a brief description of the dataset used, would improve readability.  \n   - Figure 2 lists four loss types, but the ablation study in the text discusses six. The additional two are not clearly introduced in the method section, which could confuse readers.\n\n4. **Potential Contradiction in Claims**  \n   - Section 4.1 states that overfitting is not observed in centralized settings, yet Figure 1 appears to suggest otherwise. The sentence *“Similar to GCD [29], we found the overfitting problem of GCD even worse and more complicated in the data-distributed scenario”* seems to imply overfitting is present in both. Please clarify the distinction more explicitly to avoid confusion.\n\n5. **Unclear Purpose of Equations (5) and (6)**  \n   - Equations (5) and (6) are nearly identical, differing only by the dataset to which $x_i$ belongs. However, this distinction isn’t well explained. It would help to explicitly clarify why both are needed and what unique purpose each serves.\n\n6. **Fairness of Baseline Comparisons**  \n   - The comparison with AGCL may not be fair. The reported performance of Fed-GCD in the AGCL paper (e.g., 80.7 on CIFAR-10) is significantly lower than what is reported here (90.3), suggesting possible differences in setup.  \n   - Either re-run AGCL under the same experimental configuration or adjust this paper’s settings to match AGCL’s to ensure a fair comparison.\n\n7. **Ablation Study Concerns**  \n   - In the third ablation experiment, removing SD (a proposed module) actually improves performance. This casts doubt on whether SD is truly beneficial. Please discuss this result in more detail and justify the component’s inclusion.\n\n8. **Questioning the Necessity of the Solution**  \n   - The proposed method reduces overfitting by essentially applying a lighter fine-tuning strategy. Could simpler techniques like early stopping, reducing local epochs, or lowering the learning rate achieve similar results? Please clarify why the prompt-based approach is preferable or uniquely effective.\n\n9. **Generalization to FedGCD**  \n   - It’s unclear whether the proposed solution is specifically designed for FedGCD. Overfitting in ViTs is a common issue in FL settings in general. The paper would benefit from a clearer explanation of why this approach is particularly suited to FedGCD rather than FL more broadly.\n\n10. **Motivation Could Be Stronger**  \n    - Assuming the core focus of the paper is to tackle overfitting in FedGCD, the motivation could be more explicitly and convincingly presented. Positioning overfitting as a **central challenge** and comparing the proposed solution to **existing mitigation strategies** would better highlight the novelty and necessity of this work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper identifies the overfitting issue in Federated Generalized Category Discovery (FedGCD) tasks and proposes a prompt-based method to mitigate it. The proposed approach achieves performance improvements while reducing communication costs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the issues in the **Weaknesses** section.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SjCUbEF5r7", "reviewer": "Reviewer_MvSh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- The existence of novel categories in the FL setting is a practical observation. I think this is a practical problem but has received relatively few observations. \n- The observation that prompt tuning addresses the overfitting issue is interesting, and, if better justified, will bring insights to the broader community. \n\n# Weaknesses. \n- The paper claims that 'discovering the overfitting problem of federated GCD' is one of its main contributions. However, in my opinion, the study into this phenomenon is weak. Figure 1 is the only figure that explains the observation, but the descriptions are very vague, and does not transmit sufficient insights to readers. Although I think that this observation will be of interest to a wider audience, I think it requires more efforts from the audience, e.g. how is it different from common overfitting phenomena in FL, what are the concrete observations, and what factors may be causing this issue. \n- The paper applies prompt tuning technique for federated GCD, which may limit its practical applications. From my understanding, prompt tuning can only be applied on vision transformer models, but not on CNN models, which still plays a non-negligible role in vision recognition. Therefore, the authors may need to justify the wide applicability of the proposed FedGCD-P. \n- The methodology is vaguely described. From my understanding, the methodology significantly relies on [CLS] and [PMT] tokens, but these relevant tokens are not described sufficiently in the paper, and it may be hard to readers to understand their meanings and their importance. In addition, the section 'loss on PMT token' is also very vaguely described. Finally, the self-supervised contrastive loss in Eqn. 4 is not easy to understand. Since it also involves the concept of $N(x_i)$, it seems not applicable to unlabeled data, which limits its practicality as a self-supervised loss. \n- Experiments can also be improved. In my opinion, since 'overfitting' is a key concept and motivation of this paper, more efforts should be put into analyzing it. and the current Fig. 3 is insufficient. At least, the authors should show the improvement of training accuracy alongside the decrease of test accuracy. Some more possible experiments include, e.g. analyzing the gap between training & test, analyzing how the heterogeneity plays an important role in the train-test gap, and how the prompt tuning parameters (e.g. # tokens) impact the overfitting phenomenon. \n- Minor points\n    - In large language models, it is more or less known that full fine-tuning will lead to overfitting, while PEFT can mitigate it (e.g. Xu et al. 2023). The authors may need to justify how different is the task of GCD in terms of 'full fine-tuning leads to overfitting, while PEFT does not'.  \n    - In the Introduction, there is a broken reference (Appendix ??). The appendix is also nowhere to be found. \n\n\nXu et al. Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment. Arxiv 2023", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a method for federated generalized novel category discovery (GCD). The paper identifies overfitting as one key observation when we apply ordinary GCD algorithms in the federated scenario. The authors propose a method that combines prompt tuning with existing GCD algorithms and apply it in FL, called FedGCD-P. Experiments show that the proposed FedGCD-P achieves higher accuracy and transmits considerably less data compared to existing baselines.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Explain the phenomenon of overfitting in federated GCD in more details, e.g. what may be its contributing factors, what is the difference between GCD tasks and ordinary FL tasks, etc. \n2. Explain the methodology more clearly (See weaknesses above). \n3. Justify the wide applicability of the proposed FedGCD-P. \n4. Provide more analysis on how PEFT addresses the overfitting issue of FedGCD.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)