Flipped Classroom: Aligning Teacher Attention with Student in Generalized Category Discovery
Authors: Haonan Lin
Wenbin An
Jiahao Wang
Yan Chen
Feng Tian
Mengmeng Wang
QianYing Wang
Guang Dai
Jingdong Wang
Venue: NeurIPS 2024 oral (2024)
Keywords: generalized category discovery
semi-supervised learning
open world learning
Primary area: machine_vision
TL;DR: FlipClass introduces a dynamic teacher-student framework for GCD, aligning the teacher's attention with the student's feedback to ensure consistent pattern learning and synchronized classification across old and new classes.

Abstract: Recent advancements have shown promise in applying traditional Semi-Supervised Learning strategies to the task of Generalized Category Discovery (GCD). Typically, this involves a teacher-student framework in which the teacher imparts knowledge to the student to classify categories, even in the absence of explicit labels. Nevertheless, GCD presents unique challenges, particularly the absence of priors for new classes, which can lead to the teacher's misguidance and unsynchronized learning with the student, culminating in suboptimal outcomes. In our work, we delve into why traditional teacher-student designs falter in generalized category discovery as compared to their success in closed-world semi-supervised learning. We identify inconsistent pattern learning as the crux of this issue and introduce FlipClass—a method that dynamically updates the teacher to align with the student's attention, instead of maintaining a static teacher reference. Our teacher-attention-update strategy refines the teacher's focus based on student feedback, promoting consistent pattern recognition and synchronized learning across old and new classes. Extensive experiments on a spectrum of benchmarks affirm that FlipClass significantly surpasses contemporary GCD methods, establishing new standards for the field.

Flipped Classroom:
Aligning Teacher Attention with Student in Generalized Category Discovery
Haonan Lin
Wenbin An
Jiahao Wang
Yan Chen
Feng Tian
Mengmeng Wang*
Guang Dai*
Qianying Wang
Jingdong Wang
School of Comp. Science & Technology, MOEKLINNS Lab, Xi'an Jiaotong University
School of Auto. Science & Engineering, MOEKLINNS Lab, Xi' an Jiaotong University
3 College of Comp. Science Technology, Zhejiang University of Technology SGIT AI Lab, State Grid Corporation of China Lenovo Research 6 Baidu Inc
Abstract
Recent advancements have shown promise in applying traditional Semi-Superving Learning strategies to the task of Generalized Category Discovery (GCD). Typically, this involves a teacher-student framework in which the teacher imparts knowledge to the student to classify categories, even in the absence of explicit labels. Nev- ertheless, GCD presents unique challenges, particularly the absence of priors for new classes, which can lead to the teacher's misguidance and unsynchronized learning with the student, culminating in suboptimal outcomes. In our work, we delve into why traditional teacher-student designs falter in open-world generalized category discovery as compared to their success in closed-wort -wolld semi-supervised learning. We identify inconsistent pattern learning across attention layers the crux of this issue and introduce FlipClass-a a method that dynamically updates the teacher align with the student's 's attention, instead of maintaining static teacher reference. Our teacher-student attention alignment strategy refines the teacher's s focus based on student feedback from an energy perspective, promoting consistent pattern recognition and synchronized learning across old and new classes, Exten- sive experiments on spectrum of benchmarks affirm that FlipClass significantly surpasses contemporary GCD methods, establishing new standards for the field.
Introduction
Teacher-Student architecture has proved its effectiveness in Semi-Supervised Learning (SSL. 66
32 153,
which aims to take advantage of large collection of unlabeled data, reducing the expensive costs of annotation [57 90 89 [. Previous approaches tend to model p (sstudenttreacher), where the teacher typically acts as a fixed point of reference, providing a form of "supervision prior" to guide 60] This supervision comes from labeled data and is asymmetrical: while the teacher has robust prior knowledge and provides a stable learning signal, the student's s knowledge is incomplete and evolving. The student learns from both the teacher's 's supervision and the data it is exposed to, trying to emulate the teacher by aligning its predictions with those of the teacher.
the student [25] 42
"Corresponding Authors. Equal contribution. This work was partly completed during internship SGIT AI Lab, State Grid Corporation of China.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Learning Gap
2
Model Comparison
Inner Feedback
Learning Unegnclcunied (TSCM)
Teacher- Student
Consistency Model (TSCM)
Erm.....
Over
Confident
New
Ba
Scb
Model
Parmenin
Augmentation distintion
leamning gap
inconsistent features
Flipped Classroom
Altention Inconsitency,
Learning Synchronized (FlipCless)
Consistency Model
(FlipClass)
D
Teecher
Attention
Adeptive to
Student
tch
Attention uodate
Attention alignment
(TVi)
(Tr))
D xbxx
Studient
consistent
features
0LB (9ra)
Now (0ma)
Ckay, which Incusing on?
100
190
200
Teacher
Student
Teacher
Data
Data
Figure 1: Left: Learning effects of traditional Teacher- Stuuher-Student Consistency Model (TSCM, e.g.,
SimGCD (75) and our Flipped Classroom Consistency Model (FlipClass) on Stanford Cars B3). Middle: Model comparison between TSCM and our FlipClass, where Deww refers to data belonging to new classes. Right: Illustration of the inner feedback mechanism in FlipClass, where teacher attention is adapted to the student, leading to the alignment of attention.
Teacher-Student designs traditionally rely on a closed-workd assumption, where is expected that the teachers have supervision prior to all classes they will face while instructing students [6] B)
However, real-world applications often involve dynamic and open environments, where instances belonging to new classes may appear [84] 30
31 41. In such cases, discovering novelties could enable models to adapt new information and evolve continually as biological systems do [14, 49). Recently, Generalized Category Discovery (GCD) 66) stands out by challenging models categorize unlabeled data containing both old and new classes using only partial labels for training. Although recent GCD methods have adapted closed world Teacher-Student strategies with notable success (75 50), the transition is not seamless and presents several challenges.
Challenge I: Learning gap. Fig. top left illustrates the learning evolution of student and teacher. The previous seacher-student models result in unsynchronized learning and a significant learning gap in new classes. The ideal learning dynamic between the teacher and student should be cohesive, which requires teaching students accordance with their aptitude.
Challenge II: Discrepancies in features. The learning gap arises from the teacher's fast pace, leading to large discrepancies representations between aeeelly-uummetee data (teacher) and strongly- augnened data (student), especially for new classes (Fig. [ middle). This causes significant prediction differences, making consistency loss optimization difficult and hindering effective student learning. Over time, the iterative learning process exacerbates this misalignment.
Challenge III: Attention inconsistency. Inadequate supervision for new classes leads to inaccurate instructions from the teacher, causing the student to focus on different parts than the teacher (Fig. right). Imagine a classroom transitioning to new subject. Without proper guidance, students attention diverges from the teacher's. ss resulting in confusion and ineffective learning.
To up, the challenges of previous teacher-student models arise from inadequate supervision on new classes and the gap between weakly and strongly augmented data. This results in attention inconsistency (Chall. III), which leads discrepancies predictions and representations (Chall. II), ultimately causing significant performance gap (Chall. 1). Addressing these challenges requires developing teacher-student dynamics that align the evolving knowledge of both teacher and student (Fig. I bottom left). These findings lay the foundation for our approach, FlipClass, which models the teacher's 's posterior p sreeache//student from the energy perspective of attention, building an adaptive teacher to bridge the learning gap. FlipClass offers a plug-and aad--ndd-pal -pay solution to foster an interactive learning environment where the student can influence the teacher's guidance in real-time, allowing teachers to tailor their instructions based on students' current attention 7 I. By aligning attention,
FlipClass ensures that the learning pace of the teacher and student is in syne, leading to improvement on both old and new classes. Our contributions are summarized as:

---

Sorted Pseudo Label Quality (CUB)
Consistency Loss (CUB)
Categorize Errors SimGCD (CUB)
6-
:lld classes
New classes
Consistency Loss
Old clusstre
0.8-
Accuracy
0.6-
GT Class
19.29%
15.09%
04-
8.
New
15.76%
26.54%
New
Sorted Pseudo Labe) Quality (SCars)
6
Consistency Loss (SCars)
Predicted Class
1.0
:Old classes
New classes
Consletancy Losa
Old classes
SimGCD (SCars)
New classes
Accuracy
08-
GT Class
10.55%
17.61%
R.
New
11.52%
43.48%
02-
200
Old Predicted
New Class
Classes
Epoch
Figure 2: Exploring prior gaps between SSL and GCD on SCars and CUB datasets. Left: Accuracy of sorted pseudo labels for old and new classes. Middle: Consistency loss trends over epochs, illustrating challenges in optimization and slower convergence for new classes. Right: Categorize errors [75], where "True Old" refers to predicting an 'Old' class sample to another 'Old' class, while 'False Old" indicates predicting an 'Old' class sample as some 'New' class.
(1) Empirical analysis: We highlight the challenge of applying the closed-word Teacher-Student paradigm to the spen-world scenario of GCD. Our in-depth analysis identifies attention misalignment between teacher and student as the key issue hindering synchronized learning between them.
(2) Methodology: Based on these analyses, we propose flexible and effective method, FlipClass, which enables teachers to adapt and respond to student feedback to synchronize their learning progress, thereby leading to an overall improvement teaching outcomes.
(3) Superiority: FlipClass consistently outperforms state-of-the-art generalized category discovery methods on both coarse-grainee and fine-grained datasets.
2 Background
We first provide some background on semi-supervised learning and generalized category discovery to better contextualize our analysis. Let us consider a data set consisting of labeled data Dz, :=1
[(x{. y()) Nz from |Cxl old classes and unlabeled data Dy = (x") Nu that may contain instances from
j=1
both old classes Cx and new classes Cv, with C = Ck U UC For a data instance x, let pmny fe(x)) denote the predicted class distribution produced by the model f.
2.1 Integrating SSL Techniques into Consistency Loss Framework
In semi- sumissupervised learning (SSL), the goal is to enhance model performance by leveraging unlabeled data, traditionally drawn from the same class spectrum as the labeled data [90]. The goal of SSL can be formalized by integrating three fundamental techniques: (/) Consistency regularization ensures that the model outputs consistent predictions for augmented versions of the same instance. This technique utilizes different transformations to test the robustness of the model's predictions, promoting stability across variations in the input data [ generate artificial labels for unlabeled data by adopting "hard" labels (that is, the argmax of the model output) and keeping only the labels where the highest class probability exceeds a predefined threshold (62] 47 57
59 (36. (2) Pseudo-labeling utilizes the model to
37 23). (3) Teacher-Student model incorporates a structured learning relationship where the teacher model, typically trained on reakky-auggentee instances, generates high-quality pseudo labels. These pseudo labels are then used to guide the training of the student model, which processes strongly-augmented instances. This approach helps improve the generalization capabilities of the student model by learning from the refined knowledge and stable supervision signals provided by
the teacher [66 43 77 12 53 Several methods integrate some of these techniques and achieve
advanced performance in SSL 63 83
78
87..
We unify these SSL techniques into one consistency
loss:
Lcons
Σ
H( fo(a(x))) fx))(
(1)
Dull
xEDU
3

---

where cross-entropy H(-.-) measures consistency for regularization, while the prediction Pm fo(x)) serves as a pseudo label. This setup captures a teacher-student dynamic, where a (xx) and Ax)) represent the teacher (weakly-augmenteed) and student (strongly-augmentee)) instances, respectively.
2.2 Class Prior Gap between SSL and GCD
The GCD task pushes the boundaries of SSL by questioning the closed- assumption that all classes in the unlabeled dataset Dy are previously known [69]. Instead, GCD incorporates new classes CN into the unlabeled dataset, demanding that the model learn to recognize and then correctly classify them 4 21 80 (1. In this open- open-world setting, SSL methods face obstacles with new classes due to the lack of supervision [24 [6], resulting in significantly lower quality of pseudo-labels for these new classes than for the old ones (Fig. 2
left). This gap exacerbates the complexity of optimizing the consistency loss Eq. [ for new classes, leading to learning instability and slow convergence (Fig. 2 middle). Such optimization issues lead severe prediction bias, resulting in new classes' performance lagging behind that of old classes (Fig. [ right), underlining the limitations of existing SSL techniques GCD scenarios. More empirical analysis can be found in Appendix B.2
How Consistency Loss Goes Awry: Unraveling the Pitfalls
Acknowledging challenges presented by the absence of prior knowledge for new classes in traditional semi-supervised learning (SSL. methods is the first step toward addressing the complexities of open- world tasks. We further identify that the 'prior gap' manifests as issues in learning synchronization and representation discrepancy (Sec. (.1). Our analysis targets the minimization of the energy function between teacher and student representations to bridge the 'prior gap'. We find that aligning their attention on similar patterns reduces energy, indicating effective alignment and learning /Sec.(.2) This key understanding paves the way for the development of our proposed methods, aiming to synchronize teacher- student attentions for improved model learning dynamics (Sec. 4
3.1
What to Bridge the Class Prior
The challenge of optimizing consistency loss leads to a learning gap between the student and the teacher, particularly evident when dealing with new classes (Fig. top left). This gap causes the student to plateau, as it cannot keep pace with the teacher's s more advanced understanding, which in turn restricts the teacher's progress in classes Moreover, this learning gap also manifests itself in the divergent representations between teacher and student (Fig. [ middle), specifically for new classes. Based these observations, we revisit consistency loss (Eq. [ in the
setting. Insight 3.1. The large discrepancy between fo (x)) and fo(A(x)) complicates maintaining consis- tency across the model predictions. To narrow this divide, an intuitive idea is to align fe(a(()) more closely with fo (xx).. simplifying the optimization of Lcons:
Lcons
Σ d(pm(y fo(a(x)) AR), XEDU
Pmly AR). l Fo(
Dul
where AR aims to pull (x)) closer to fx(A(x)). Ideally, AR would be adaptive, scaling with the discrepancy between fo (xx)) and fa(a(xx), while avoiding make them too similar, which enables model to find a shortcut of Lcons-
To design it, we delve into the vision transformer, representation encoder that has significantly advanced the performance of the GCD task. We found that the self-attention mechanism excels at capturing critical image patterns: depicted in Fig. leftt, deeper features (after the 8th layer) reveal semantic, high-level commonalities (e.g., car shell) across all images; and the shallow features are more attuned high-frequency, low-level details (e.g., color and texture).
3.2 Inconsistent Patterns Spoil the Whole Barrel
Inconsistent patterns can disrupt learning, making it crucial to align stored and queried patterns effectively. To address this, we draw inspiration from the Hopfield Network (2 - an associative memory model known for its energy-based mechanism that naturally pulls similar patterns together

---

Teacher
AILS
Alto
Alte
AlCs 8
Alta
Inc
Energy EEO, K3
10-
Student
60
Alt
Alnn
Layers -11
epoch epoch 40 - epoch 60
Figure 3: Left: Attention heatmaps for teacher and student across attention layers. Right: Energy trend over epochs, with lower energy indicating less discrepancy pattern recognition between teacher and student.
(see Appendix A.1 for details). We follow Ramsauer et al. to define the energy function for a state pattern (query) Rd, parameterized by stored (key) patterns X [x] .XX] RdxN
((5:)) 5 - Ise(X (oe(X'E.)). +c, with Ise(v, B= log exp(v:)
(3)
Minimizing E(E; X) resembles retrieving stored pattern x; that most similar to the query É. and log-sum sum-exp (lse) function is parameterized by 0 and is preset constant. Particularly, the first term ensures the finiteness of the query, while the second term measures the alignment of the query with each stored pattern. The update rule for state pattern is equivalent gradient descent update of minimizing the energy with step size the most similar stored pattern:
[54] ensures that the query moves closer to
5ー = - smmBE"X)x
Moreover, the energy function is closely related to the Transformer's 's self-attention mechanism |67 (Appendix A.112). By extending the energy model from self-attention to cross-attention, we model the dynamics between student and teacher learning patterns. Taking the student representations f, = (x)) as examples, we have Q, = f Wo and K, f, W.. By applying Eq. to key and query matrices, we energy functions track the teacher-student relationship:
E(Q.:K, = diag K,K,")
> 1=1
Ise Q,k,,
(5a)
E(K,) Ise (idiag( K, 1,,,
log
1=1 2 _kr..kk.1 tc,
exp
(5b)
where E( Q.; K2) indicates the alignment in learning patterns of the student and teacher; k, denotes the i-th row vector of K, and a 2 Intuitively, Ise Bee(Qkk", B) captures the smooth maximum aligmment between student queries qs./ and teacher keys k,,- Specifically, it nudges each teacher key k,j towards semantic alignment with its most corresponding student query qs.j- The regularization term diag K,K,?) acts as constraint on the energy levels of teacher's 's representation k, guarding against any disproportionate increase during the maximization of Ise( Q.k This ensures that no individual teacher representation becomes too closely mirrored in the student's representation, maintaining diverse learning trajectory.
When applying dosed-world consistency regularization to the GCD task, it becomes difficult to gradually reduce the energy (QQ,; high energy demonstrated flaw in the previous methods: teachers and focuseed on identifying putterns that were inconsistent, leading to divergent learning puths. Specifically, when teachers and students focus similar patterns (e.g., taillights), energy is reduced, indicating beiter prediction consistency and effective learning. In contrast, when their attention is distracted, the energy rises, leading to severe inconsistencies in predictions and making the optimization of Lcons more difficult.
Insight 3.2.
00:KK, K2) as training progresses (Fig. right). The sustained
5

---

FlipClass: Teacher-Student Attention Alignment
4.1 Teacher Attention Update Rule
Based on Insights 3.1 and 3.2, our objective is to minimize the energy function E(Qs; K,) teacher and student representations, thereby easing the optimization of Leons.
between
Theorem 4.1. The minimization can be formulated as obtaining maximum posteriori probability (MAP) estimate of teacher keys K, given a set of observed student queries Q,:
p(K.QQ.) (K))))
(6)
p(Q.)
where Ki) and p(K,) are modeled by energy functions Eq. 5a and 5b respectively. We approximate the posterior inference by the gradient of the log posterior, estimated as:
VK, log (K, =VVV, EQQs: K,) +VK,E(K,))
sm (5Q, Q aT D
sm (ddias(K K,
where sm(v) := exp (v Ise(v, 1)) and (-) is vector-to do-iooooo---g matrix operator. Incorporating Eq. 4 the update rule of teacher keys K, derived as follows:
Kinddate
K,
Yapdote
(sm Qw%)
Yreg al + D sm
diag (xx*)) kw%)
(8)
where a, Yupdate and Yeeg are hyper-parameters.
For a proof, refer to Appendix A.2 The teacher-attention update rule in Theorem 4.1 minimizes an implicit energy function determined by student queries and teacher keys. It serves as using the student queries to search for the most similar teacher patterns in the stored set. As illustrated in Fig. 4 the update rule adjusts the teacher's attention in the direction of student attention, facili- tating the retrieval of related patterns and improving semantic alignment. This de- sign establishes a bidirectional information flow: the teacher not only imparts advanced knowledge to the student, but also adjusts guidance based on the student's learning ef- fects, achieving a more cohesive learning dynamic.
Transformer Encoder (ith block)
Ki
Kcrodsted
Projector
Candiaan Curt
you this ar?
Attnn
Rulee SiR
Key
Lp
coas
Sorny cannot
Attn Query
Instruction...
Projector
Studont Out
Attention Aligmment
Le
Figure 4: Framework of FlipClass demonstrating teacher-student interaction, where teacher's and stu- dent's 's attention is aligned by teacher's 's updating (Eq. [). Then Leep and Lcons are combined for optimization.
4.2 Representation Learning and Parametric Classification
Contrastive learning plus consistency regularization under the parametric paradigm has been demon- strated effective in GCD task [75]. Formally, given two views (random augmentations and x²) of the same image mini-batch B, the supervised and self- supersuueeeieee contrastive loss is written as:
exp(z) /T/
Lxp
log
EPSi exp(z T.)TT)
IEB
qEN,
exp(z 2,/ Tw)
CK
log
EPS exp(z 2 Tn)
FEB
where the feature 2, fx;;) and is fznorrmalised, and are temperature values. For Li N; indexes other images in the same batch that hold the same label as x,. The representation learning loss is balanced with 2: Leep (1 2)p Aepp where B' corresponds to the labeled subset of B.

---

Table 1: Evaluation the Semantic Shift Benchmark (SSB). Bold values represent the best results,
while underlined values represent the second- results.
CUB
Stanford Cars
Aircraft
Methods
Backbone
All
Old
New
All
Old
New
All
Old
New
Avg.
GCD
2022
DINO
51.3
56.6
48.7
39.0
57.6
29.9
45.0
41.1
46.9
45.1
XCon
2022
DINO
52.1
54.3
51.0
40.5
58.8
31.7
47.7
44.4
49.4
46.8
CiPR 2023
DINO
55.6
47.0
PCAL
2023
DINO
62.9
64.4
62.1
50.2
70.1
40.6
52.2
52.2
52.3
55.1
SimGCD
2023
DINO
60.3
65.6
57.7
53.8
71.9
45.0
54.2
59.1
51.8
56.1
AdaptGCD
2024
DINO
66.6
66.5
66.7
48.4
57.7
39.3
53.7
51.1
56.0
56.2
AMEND
2024
DINO
64.9
75.6
59.6
56.4
73.3
48.2
52.8
61.8
48.3
58.0
GCA 2024
DINO
68.8
73.4
66.6
54.4
72.1
45.8
52.0
57.1
49.5
TIDA 2024
DINO
54.7
72.3
46.2
54.6
61.3
52.1
μGCD 2024
DINO
65.7
68.0
64.6
56.5
68.1
50.9
53.8
55.4
53.0
58.7
CMS 2024
DINO
68.2
76.5
64.0
56.9
76.1
47.6
56.0
63.4
52.3
60.4
InfoSieve
2024
DINO
69.4
77.9
65.1
55.7
74.8
46.4
56.3
63.7
SPTNet
2024
DINO
65.8
68.8
65.1
59.0
79.2
49.3
59.3
61.8
58.1
61.4
FlipClass (Ours)
DINO
71.3
71.3
71.3
63.1
81.7
53.8
59.3
66.9
55.4
64.6
Improvement
DINO
+5.5
+2.5
+6.2
+4.1
+2.5
+4.5
+0.0
+5.1
-2.7
+3.2
GCD 2022
DINOV2
71.9
71.2
72.3
65.7
67.8
64.7
55.4
47.9
59.2
64.3
SimGCD 2023
DINOv2
71.5
78.1
68.3
71.5
81.9
64.6
49.9
60.9
60.0
μGCD [2024]
DINOv2
74.0
75.9
73.1
76.1
91.0
68.9
66.3
68.7
65.1
72.1
FlipClass (Ours)
DINOv2
79.3
80.7
78.5
78.0
88.0
73.2
71.1
75.1
69.1
76.1
Improvement
DINOv2
+5.3
+4.8
+5.4
+1.9
-3.0
+4..3
+4.8
+6.4
+4.0
+4.0
The consistency regularization objectives (Eq. D are then simply cross-entropy loss E(q', Σ (k) log p(k) between the predictions and pscudo-labels or ground-truth labels:
Lies (i). s
for unlabeled, for labeled.
Lcons
18
10 Lier ((vi,))) Pi)
The one-hot labels correspond to and the soft pseudo-label q is produced by the teacher instance a (x); Moreover, a mean-entropy regularizer [6) H(5) = = 5 k) log p(k), is included to encourage diverse predictions. The combined classification loss, Lcons, balances unsupervised and supervised terms with a parameter And the overall training objective is Leep + Lcons-
Experiments
5.1 Experimental Settings
Datasets. We evaluate the effectiveness of FlipClass on three generic image recognition datasets
(i.e., CIFAR- 10/100 34 and ImageNet-100 100 (20). three fine-grained datasets (68] (i.e., CUB [71].
Stanford Cars 33 and FGVC-Aireraft [46) contained in Semantic Shift Benchmark (SSB) 68 and
the challenging datasets Herbarium-19 19
65 ImageNet-1k [20]. For each dataset, we first subsample
|Cyl seen (labeled) classes from all classes. Following GCD [69], we subsample 80% samples in
CIFAR- and 50% samples in all other datasets from the seen classes construct D,. while the remaining images are treated as D, (refer to Table g.
Evaluation Protocols. The performance was evaluated by measuring accuracy between the model's cluster assignments and ground-true -ttuh labels on the test set, with three aspects: all instances (All), instances from old categories (Old), and instances from new categories (New). The number of
unlabeled dataset (IC,,) often unknown. Following previous studies 64 85), we set K (cluster number) equal to |Cal, approximate cluster estimation is usually feasible in the real world. The estimation of the number of categories unlabeled datasets can be found in the Appendix C44 Further implementation details can be found in Appendix D.I
categories

---

CUB
SCars
AirCraft
Cifar10
Cifar100
80%
70%
100%
70%
60%
60%
60%
98%
80%
50%
95%
50%
40%
40%
70%
Old
New
All
New
Old New
Old
New
(I) Ours
(2) w/o Strong Aug
(3) w/o Attn Align
(4) Attn Align w/o Reg
Figure 5: Ablation study results for FlipClass, indicate the critical role of strong augmentations, attention alignment, and regularization in model performance across multiple datasets.
Table 2: Evaluation on three generic image recognition datasets.
CIFARTO
CIFARTO0
ImageNet-100
Methods
Backbone
Old
New
New
All
Old
New
Avg.
GCD 2022
DINO
91.5
97.9
88.2
73.0
76.2
65.5
74.1
89.8
66.3
79.5
AdaptGCDD
2024
DINO
93.2
94.6
92.8
71.3
75.7
66.8
83.3
90.2
76.5
82.6
InfoSicve
2024
DINO
94.8
97.2
93.7
76.9
78.4
73.9
80.5
92.8
74.4
84.1
CiPR 22023
DINO
97.7
97.5
97.7
81.5
82.4
79.7
80.5
84.9
78.3
86.6
SimGCD 2023
DINO
97.1
95.1
98.1
80.1
81.2
77.8
83.0
93.1
77.9
86.7
GCA 2024
DINO
95.5
95.9
95.2
82.4
85.6
75.9
82.8
94.1
77.1
86.9
TIDA 2024
DINO
98.2
97.9
98.5
82.3
83.8
80.7
CMS 2024
DINO
82.3
85.7
75.5
84.7
95.6
79.2
AMEND 2024
DINO
96.8
94.6
97.8
81.0
79.9
83.8
83.2
92,9
78.3
87.0
SPTNet
[024]
DINO
97.3
95.0
98.6
81.3
84.3
75.6
85.4
93.2
81.4
88.0
FlipClass (Ours)
DINO
98.5
97.6
99.0
85.2
84.9
85.8
86.7
94.3
82.9
90.1
Improvement
DINO
+1.2
+2.6
+0.4
+3.9
+0.6
+10.2
+1.3
+1.1
+1.5
GCD 2022
DINOv2
95.2
93.9
77.3
82.8
66.1
81.3
94.3
74.8
AMEND [024)
DINOv2
97.7
96.6
98.3
83.5
83.0
84.5
87.3
95.1
83.4
89.5
FlipClass (Ours)
DINOv2
99.0
98.2
99.4
91.7
90.4
94.2
91.0
96.3
88.3
93.9
Improvement
DINOv2
+1.3
+1.1
+8.2
47.4
+9.7
+3.7
+1.2
+4,9
+4.3
5.2
Experimental Results
We compare SOTA methods with ours in GCD using features from both DINO [5] and DINOv2 [51. Our approach shows significant performance improvement, particularly in the recognition of 'New' classes across both the SSB fine-grained benchmark (Tab. [T] and generic image recognition datasets (Tab. consistently surpassing existing SOTA methods. Moreover, in fine-gree fine-grained image classification Tab. []. recognizing subtle differences between closely related categories is crucial, which is in contrast coarse-gras gaarse- datasets where the visual differences between classes are more obvious. In fine- mne-grained settings, the risk of the model generating incorrect pseudo labels is higher, which makes consistency regularization counterproductive (Sec. 2.2). However, the results across these datasets demonstrate our model's capability to effectively adapt consistency regularization strategies from dlosed-world settings to more complex open-we world scenarios. Moreover, the balanced accuracy between new and old classes on the CUB dataset (Tab. [ also observed with methods like PCAL, CiPR, and AdaptGCD, can be attributed to the dataset's small size (6,000 images) and large class split (200). This limited data reduces the likelihood of overfitting to old classes, promoting more uniform performance across both categories. Additional results on Herbarium 19 and ImageNet 1k are detailed in the Appendix C.2
5.3 Analysis and Discussion
Ablation Studies. Our ablation study, shown in Fig. underscores the significance of our design choices. First, we replace the student's augmentations with the teacher's, s, i.e., using only weak augmentations, the performance on 'New" classes significantly declines (2nd of bars). This underscores the importance of strong augmentations for the student, which are essential to bolster generalizability. Then we validate the importance of attention alignment Eq. 3 set of bars), we see performance drop across both 'Old' and 'New" classes, affirming that our attention alignment strategy is crucial for maintaining consistent learning pace between teacher and student, leading to sustained performance gains. Finally, the 4th set of bars verifies the role of regularization during

---

(1)
1,2666
Before Training
Logits Alignment
Consistency Loss
Ours
Old
Stroeg Ave Aug
Stroiz Aus
Stecep And
Sirse Aue
(2)
Aug
1.4211
(3)
(4)
ILS372
(LT821
Accuracy
Figure 6: Accuracy and representation alignment with different strategies: (1) initial state, (2) distribution alignment, (3) FixMatch, and (4) our teacher-attention update. Performance on 'New' and 'Old' classes are shown, alongside alignment of teacher (red) and student (blue) representation.
10.. ACC Bave (in 30
SCare
CUB
Tomn Ere 962
CIRAR10O
CUBI
20
15
89
-IX
CORAL
COBAL
Oo Rixe Noe Filoo OM Tix New
Tee-Ol Filx New Bike O1l Toa New
New
Trp 7
(a) Comparison of attention alignment methods, show-
(b) Categorize errors for FlipClass CIFAR100 and CUB, showing the reduction in prediction bias for 'False Old' and 'False New' classes with different up- date rates (Yupdate), highlighting model robustness.
casing the effectiveness our teacher-attention update
strategy over alternatives in improving classification accuracy for 'Old' (solid) and 'New" (dotted) classes.
Figure 7: Attention alignment methods comparison and categorize errors with different update rates.
the attention update, which integrates the prior energy of the teacher, preventing any single student pattern from overly influencing the teacher's attention.
Enhanced Consistency through Attention Alignment. To validate our assumption modeling R in Eq. 2 we showcase how distribution-base strategies (distribution alignment [T, FixMatch method, attention alignment, achieve consistency on new classes from CUB dataset (Fig. 6). Our method stands out by significantly reducing the discrepancy between the representations of teacher (weakly-augmented) and student
(63) and our representation-baxe
data. This representation based alignment leads to more consistent learning process and is evidenced by the superior accuracies we achieve for both 'Old' and 'New" classes.
Design Choice of Attention Alignment. We experimented with various techniques to model R in Eq. 2 including scheduled data augmentation (SDA), increasing similarity between Q₃ and K, via l2 norm, Kullback-Leibler divergence (KLD) or CORrelation ALignment (CORAL) loss (see details in Appendix D.I). As shown in Fig. 7a our teacher-attention update strategy outperforms these alternatives on both CUB and SCars datasets.
FlipClass mitigates prediction bias. We verify the effectiveness and robustness of FlipClass, by diagnosing the model's classification errors under four different (Yupdate) as defined in Eq. 8 As
depicted in Fig. 7b both "False New" and "False Old" errors are consistently mitigated where Old' class samples are mistakenly labeled as 'New' and vers. Moreover, as illustrated in Fig (bb bottom, FlipClass outperforms leading methods [55] by moving closer the true class distribution, yielding higher and less biased accuracies across all classes.
Does the improved energy dynamic make for performance gains? Fig. 8a top shows that aligning attention in these deeper (9-10) layers yields the highest performance on SCars dataset. And Fig. Sal bottom displays greater reduction of energy (QQ; K,) in deeper layers. Moreover, this trend highlights that attention alignment constantly maintains lower energy levels than without, indicating improved alignment of student patterns with teacher updating.
How does FlipClass change the representations? Fig. 8b showcases representation enhance- ments with FlipClass against the leading method, InfoSieve [35], using the same t-SNE and PCA components to ensure consistent projection space and scale. FlipClass forms clusters with higher compactness and purity, demonstrating enhanced feature discrimination and less inter-class confusion. The zoom-in comparison on CUB dataset is provided in Fig. [ which showcases that FlipClass achieves more distinct and well-separated clusters. Further, we assess prediction bias and class-

---

Effects of Attention-Update Layers
InfoSieve
FlipClass (Ours)
Larm lpen
Energy Trend Across Attention Layers and Epochs
Energy -E[Q:; K.)
1.0
Accuracy
Frequecc Acturacy
Prequency
3.00
Epoch
Epoch
Epoch
Epoch
8.5
w/o attention update - attention update
pePce prokiina
0.0
Clargance perdecioo
(a) Top: performance regarding attention attention-update layers. Bottom: Energy tracking across attention layers and epochs. (Both on SCars dataset)
Class Accuracy
Aligned Predictions
GT Labels
(b) Comparison of representations and classwise accuracy
between InfoSieve (SS) and FlipClass on CUB.
Figure 8: Attention alignment improves energy dynamic and brings performance gains.
specific accuracies. Unlike InfoSieve's skewed predictions, FlipClass aligns better with class distributions, and significantly improves over the tail classes in CUB dataset (More experiments in Appendix OK
InfoSieve
FlghCles
InfoSieve
FlipClass
Class
Class
Class
Class
Closs
Cliss 30
Class All
Class
Class
Class R3
Figure 9: Zoom-in comparison of InfoSieve and FlipClass the CUB dataset using t-SNE and PCA. FlipClass shows improved cluster separation and compactness.
Conclusion
This paper introduces FlipClass, dynamic teacher-student attention alignment strategy for improving learning synchronization, providing new view on applying closed-world models to open-world task of GCD. By aligning the attention of teacher and student, FlipClass bridges the learning gap between them, resulting in performance improvement on both old and new classes. Extensive experiments and analysis demonstrate that FlipClass outperforms existing state-of-the-arr methods across diverse datasets by a large margin.
7 Acknowledgement
This work was supported by National Science and Technology Major Project 20022D01171022 National Natural Science Foundation of China (62293551, of China Knowledge Centre for Engineering Science and Technology, Project of Chinese academy of engineering "The Online and Offline Mixed Educational Service System for 'The Belt and Road" Training in MOOC China". "LENOVO-XJTU" Intelligent Industry Joint Laboratory Project.
Project
References
[1] Gökçe Akçayır and Murat Akçayır. The flipped classroom: review of its advantages and challenges. Computers & Education,
2018.
[2] S-I Amari. Learning patterns and pattern sequences by self-organizing nets of threshold elements.
IEEE Transactions computers, 100(1)):1197-1200 1972.
10

---

[3] Wenbin An, Wenkai Shi, Feng Tian, Haonan Lin, DianYing Wang, Yaqiang Wu, Mingxiang Cai. Luyan Wang. Yan Chen, Haiping Zhu, al. Generalized category discovery with large language models in the loop. arXiv preprint arXiv:23/2. 10897, 2023.
[4] Wenbin An, Feng Tian, Qinghua Zheng, Wei Ding, Ying Wang, and Ping Chen. Generalized category discovery with decoupled prototypical network. In Proceedings of the AAAI Conference on Artificial Intelligence, number 11, pages 12527-12535, 2023.
[5] Wenbin An, Feng Tian, Wenkai Shi, Yan Chen, Yaqiang Wu, Qianying Wang. and Ping Chen. Transfer and alignment network for generalized category discovery. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 10856-10864, 2024.
[6] Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. In European Conference on Computer Vision, pages 456-473. Springer, 2022.
[7] Isaiah T Awidi and Mark Paynter. The impact of a flipped classroom approach on student learning experience. Computers & education, 128:269-288 2019.
[8] Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. Advances in neural information processing systems, 27, 2014.
(9] Jianhong Bai, Zuozhu Liu, Hualiang Wang, Ruizhe Chen, Lianrui Mu, Xiaomeng Li, Joey Tianyi Zhou, Yang Feng, Jian Wu, and Haoji Hu. Towards ietributioonagnnssie generalized category discovery. Advances in Neural Information Processing Systems, 36, 2024.
[10] Anwesha Banerjee, Liyana Sahir Kallooriyakath, and Soma Biswas. Amend: Adaptive margin and expanded neighborhood for efficient generalized category discovery. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2101-2110, 2024.
[11] David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. In International conference on learning representations, 2020.
[12] Zhaowei Cai, Avinash Ravichandran, Subhransu Maji. Charless Fowlkes, Zhuowen Tu, and Stefano Soatto. Exponential moving average normalization for self-supervised and semi- supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 194-203, 2021.
[13] Kaidi Cao, Maria Brbic, and Jure Leskovec. Open-world semi-supervised learning. arXiv preprint arXiv:2102.033526 2021.
[14] Susan Carey. The origin of concepts. Journal of Cognition and Development, 1(1):37-411 2000.
[15] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[16] Florent Chiaroni, Jose Dolz, Ziko Imtiaz Masud, Amar Mitiche, and Ismail Ben Ayed. Mutual information based generalized category discovery. arXiv preprint arXiv:2212.00334, 2, 2022.
[17] Sua Choi, Dahyun Kang, and Minsu Cho. Contrastive mean-shiff learning for generalized category discovery. arXiv preprint 04X44.2404.09451, 2024.
[18] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition workshops, pages 702-703, 2020
[19] Mete Demircigil, Judith Heusel, Matthias Löwe, Sven Upgang, and Franck Vermet. On a model of associative memory with huge storage capacity. Journal of Statistical Physics, 168:288 299, 2017, URL httphhttts/aaat semanticscholar. prg/CorpusIpp 119331112

---

[20] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[21] Yixin Fei, Zhongkai Zhao, Siwei Yang, and Bingchen Zhao. Xcon: Learning with experts for fine-grained category discovery. arXiv preprint wxxv:2208.00898, 2022.
[22] Enrico Fini, Enver Sangineto, Stéphane Lathuilière, Zhun Zhong, Moin Nabi, and Elisa Ricci. A unified objective for novel class discovery. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 9284-9292, 2021.
[23] Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. Advances in neural information processing systems, 17, 2004.
[24] Lan-Zhe Guo, Yi-Ge Zhang. Zhi-Fan Wu, Jie-Jing Shao, and Yu-Feng Li. Robust semi- supervised learning when not all classes have labels. Advances in Neural Information Processing Systems,35:33055 35:3305-3317, 2022.
[25] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31, 2018.
[26] Kai Han, Andrea Vedaldi, and Andrew Zisserman. Learning to discover novel visual categories via deep transfer clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8401-8409, 2019.
[27] Kai Han, Sylvestre-Alvise Rebuffi, Sehastien Ehrhardt, Andrea Vedaldi, and Andrew Zisser- man. Automatically discovering and learning new visual categories with ranking statistics. In International conference on learning representations, 2020.
[28] Shaozhe Hao, Kai Han, and Kwan- Yee K Wong. Cipr:
An efficient framework with cross- instance positive relations for generalized category discovery. arXiv preprint arXiv:2304.06928, 2023.
[29] Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method improve robustness and uncertainty. 2019.
arXiv preprini
[30] Yen-Chang Hsu, Zhaoyang Lv, and Zsolt Kira. Learning cluster in order transfer across domains and tasks. In International conference on learning representations, 2018.
[31] Yen-Chang Hsu, Zhaoyang Lv, Joel Schlosser, Phillip Odom, and Zsolt Kira. Multi-class classification without multi-class labels. In International conference on learning representations, 2019.
[32] Zhanghan Ke, Daoye Wang, Qiong Yan, Jimmy Ren, and Rynson WH Lau. Dual student: Breaking the limits of the teacher in semi sumirvipe learning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6728-672 6636. 2019.
[33] Jonathan Krause, Michael Stark, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
[34] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009. URL. httpttttt//s/wt... toronto. edu/ waddu//kizz/ wawii/iialel http://riii/ereiaie wtwtr/arrnnnnn-etuuee---eeeeeeee features- 10.9-TR.pd..
[35] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[36] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In Interna- tional conference on learning representations, 2017.
12

---

[37] Dong- Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. Workshop on challenges in representation learning. ICML, 3(2):896, 2013.
[38] Wenbin Li, Zhichen Fan, Jing Huo, and Yang Gao. Modeling inter-class and intra-class constraints in novel class discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3449-3458 2023.
[39] Haonan Lin, Wenbin An, Yan Chen, Feng Tian, Wei Ding, QianYing, Wang, Ping Chen, Yaqiang Wu, mingxiang cai, and Guang Dai. Semantic-enhanced prototypical network for universal novel category discovery, 2024. RRLLittps://ppereeiiie
net/Eorum?id-Uoolivb
[40] Haonan Lin, Wenbin An, Yan Chen, Feng Tian, Yuzhe Yao, Wei Ding, Qianying Wang. and Ping Chen. A tri-branch network with prototype-aware matching for universal category discovery. In 2024 IEEE International Conference on Multimedia and Expo (ICME), pages 1-6. IEEE, 2024.
[41] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribttion detection. Advances neural information processing systems, 33:21464-21475, 2020.
[42] Bruno Loureiro, Cedric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Mare Mezard, and Lenka Zdeborová. Learning curves of generic features maps for realistic datasets with a teacher-student model. Advances in Neural Information Processing Systems, 34:18137-18151, 2021.
[43] Yucen Luo, Jun Zhu, Mengxi Li, Yong Ren, and Bo Zhang. Smooth neighbors on teacher graphs for semi-supervised learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1896-8905, 2018
[44] Shijie Ma, Fei Zhu, Zhun Zhong, Xu-Yao Zhang, and Cheng-Lin Liu. Active generalized category discovery. arXiv preprint MXXiv:2403.04277 2024.
[45] James MacQueen et al. Some methods for classification and analysis of multivariate observations. Proceedings of the fifth Berkeley symposium on mathematical statistics and probubility, 1(14): 281-297, 1967.
[46] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification aircraft. arXiv preprint 1rxXiv:330.515555 2013.
[47] Geoffrey J McLachlan. Iterative reclassification procedure for constructing an asymptotically op- timal rule of allocation in discriminant analysis. Journal of the American Statistical Association, 70(350):365336) 1975.
[48] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii Virtual adversarial training: a regularization method for supervised and supervvied learning. IEEE transactions on pattern analysis and machine intelligence, 41(8):1979-1992 2018
[49] Gregory Murphy, The big book of concepts. MIT press, 2004.
[50] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European conference on computer vision, pages 69-84. Springer, 2016.
[51] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07/99 07/3, 2023.
[52] Jona Otholt, Christoph Meinel, and Haojin Yang. Guided cluster aggregation: A hierarchical approach to generalized category discovery. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2618 2627,, 2024.
[53] Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V Le. Meta pseudo labels. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 11557-11568, 2021.
[54] Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlovié, Geir Kjetil Sandve, et al. Hopfield networks is all you need. In International conference on learning representations, 2021.
13

---

[55] Sarah Rastegar, Hazel Doughty, and Cees Snoek. Learn to categorize or categorize to learn? self-coding for generalized category discovery. Advances in Neural Information Processing Systems, 36, 2024.
[56] Mamshad Nayeem Rizve, Navid Kardan, and Mubarak Shah. Towards realistic semi-supervised learning. In European Conference on Computer Vision, pages 437-455. Springer, 2022.
[57] Chuck Rosenberg, Martial Hebert, and Henry Schneiderman. Semi-supervised self tiffnirainng of object detection models. Carnegie Mellon University, 2005.
[58] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115())211-2 2015. doi: 10. 10.7/00/s111 00777s11263--00 1263-015-0086-y
[59] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi-supervised learning. Advances in neural information processing systems, 29, 2016.
[60] Felix Sarnthein, Gregor Bachmann, Sotiris Anagnostidis, and Thomas Hofmann. Random teachers are good teachers. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 30022-30041. PMLR, 23-29 Jul 2023. URL https: pptppe///prcceeingg.. mmr..mmr......... press/ w202/sarnthein23a. html
[61] Walter J Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E Boult. Toward open set recognition. IEEE transactions on pattern analysis and machine intelligence, 35(7): 1757-1772, 2012.
[62] Henry Scudder. Probability of error of some adaptive pattern-recognitionn machines. IEEE Transuctions Information Theory, 11(3)363-377, 1965.
[63] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raf- fel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-608. 608, 2020.
[64] Yiyou Sun and Yixuan Li. Opencon: Open-world contrastive learning. Transactions on Machine Learning Research, 2022.
[65] Kiat Chuan Tan, Yulong Liu, Barbara Ambrose, Melissa Tulig, and Serge Belongie. The herbarium challenge 2019 dataset. arXiv preprint arXiv: arXiv:1906.00 05372, 2019.
[66] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged con- sistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017.
[67] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neurul information processing systems, 30, 2017.
[68] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier is all you need? In International conference on learning representations, 2022.
[69] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 7492-7501, 2022.
[70] Sagar Vaze, Andrea Vedaldi, and Andrew Zisserman. No representation rules them all in category discovery. Advances in Neural Information Processing Systems, 36, 2024.
14

---

[71] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. Technical report, California Institute of Technology, 2011.
[72] Hongjun Wang, Sagar Vaze, and Kai Han. SPTNet: An efficient alternative framework for generalized category discovery with spatial prompt tuning. In The Twelfth International Conference on Learning Representatioss 2024. URL
wawpaarriiieeee/forrrnnn net/forum?
id=3QLkwU40EE
[73] Ye Wang. Yaxiong Wang, Yujiao Wu, Bingchen Zhao, and Xueming Qian. Beyond known clusters: Probe new prototypes for efficient generalized class discovery. arXiv preprini arXiv:2404.0899) 08995, 2024.
[74] Yu Wang, Zhun Zhong, Pengchong Qiao, Xuxin Cheng, Xiawu Zheng, Chang Liu, Nicu Sebe, Rongrong Ji, and Jie Chen. Discover and align taxonomic context priors for open-world semi-supervised learning. Advances in Neural Information Processing Systems, 36, 2024.
[75] Xin Wen, Bingchen Zhao, and Xiaojuan Qi. Parametric classification for generalized category discovery: baseline study. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16590-16600, 2023.
[76] Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data aug- mentation for consistency training. Advances in neural information processing systems, 33: 6256-6268, 2020.
[77] Qizhe Xie, Minh- ThanTThag Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10687-10698, 2020.
[78] Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi-supervised learning with dynamic thresholding. In International Conference on Machine Learning, pages 1 11525-1 1536. PMLR, 2021.
[79] Muli Yang, Yuehua Zhu, Jiaping Yu, Aming Wu, and Cheng Deng. Divide and conquer: Compositional experts for generalized novel class discovery. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14268 14277, 2022.
[80] Qing Yu, Daiki Ikami, Go Irie, and Kiyoharu Aizawa. Self-labeling framework for novel cate- gory discovery over domains. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3161-3169, 2022.
[81] Alan L Yuille and Anand Rangarajan. The concave-convex procedure (ccop). Advances in neural information processing systems, 14, 2001.
[82] Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023 6032, 2019.
[83] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang. Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling, Advances in Neural Information Processing Systems, 8408-184141 2021.
[84] He Zhang and Vishal M Patel. Sparse representation--asee open set recognition. IEEE transactions on pattern analysis and machine intelligence, 39(8): 1690-166,, 2016.
[85] Sheng Zhang, Salman Khan, Zhiqiang Shen, Muzammal Naseer, Guangyi Chen, and Fa- had Shahbaz Khan. Promptcal: Contrastive affinity learning via auxiliary prompts for gener- alized novel category discovery. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3479-3488, 2023.
[86] Bingchen Zhao and Kai Han. Novel visual category discovery with dual ranking statistics and mutual knowledge distillation. Advances in Neural Information Processing Systems, 34: 22982-22994 2021.
15

---

[87] Mingkai Zheng, Shan You. Lang Huang, Fei Wang, Chen Qian, and Chang Xu. Simmatch: Semi-supervised learning with similarity matching. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pages 14471- 14471-14481, 2022.
[88] Zhun Zhong, Enrico Fini, Subhankar Roy, Zhiming Luo, Elisa Ricci, and Nicu Sebe. Neighbor- hood contrastive learning for novel class discovery. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 10867-10875, 2021.
[89] Zhi-Hua Zhou. A brief introduction weakly supervised learning. National science review, (1):44-53, 2018.
[90] Xiaojin Jerry Zhu. Sepi-supervised learning literature survey. University of Wisconsin-Madison Department of Computer Sciences, 2005.
16

---

Appendix
Table of Contents
A Theory Assumptions and Proofs
18
A.1 Preliminaries: Hopficld Network Energy Function
18
A.2
Derivation of The Teacher Atlention Update Rule
19
Extended Experimental Analysis of Attention Alignment
21
B.1
Representation Discrepancy of Old and New Classes
21
B.2
Enhanced Consistency Loss Optimization
21
B.3
Attention Alignment Bridges Prior Gap and Benefits Synchronized Learning
22
B.4
Attention Specialization in Deep Network Layers
23
B.5
Attention Alignment in Layer Selection
23
B.6
Time Efficiency Attention Alignment
24
More Experimental Results
25
Main Results with Error Bars
25
Results Complex Datasets
25
Clustering and Per-class Prediction Distribution
25
4
Robustness to Number of Classes
26
C.5
Results with Varying Proportion of Old Classes
27
Experimental Settings
28
Implementation Details
................................
28
D.2
Design of Data Augmentation
................................
28
D33
Datasets
29
D.4
Other Alignment Strategies
30
E
Related Works
30
Consistency Regularization
30
Novel Category Discovery
31
E.3
Generalized Category
Caterrr Discovery
31
F
imitations and Future Work
31
Broader Impacts
31
17

---

A Theory Assumptions and Proofs
A.1 Preliminaries: Hopfield Network Energy Function
A.1.1 Global Convergence of Hopfield Network Energy Function
We first provide the formulation of Hopfield network energy function, and present its convergence, which build the fundamental of our one-step Teacher-Attentinn Update Strategy. Ramsauer al. proposed a new energy function that is modification of the energy of modern Hopfield networks 19]. and new update rule which can be proven to converge to stationary points of the energy.
Given N stored (key) patterns X, E Rd represented by the matrix X = (X)....,NN) with the state (query) pattern & Rd, the energy function E of the modern Hopfield networks can be expressed:
E = exp(1se(1, 1, E)),
where Ise(B.x) = B log اي exp(3x;) is the log-sum-exp function (Ise) for 0 < B. Ramsauer jet vaal.,then proposed to take the logarithm of the negative energy of modern Hopfield networks and add a quadratic term of the current state to ensure that the norm of the state vector & remains finite and the energy is bounded, reads:
1
E = BBe x 55555 B log N 3 M².
(9)
Using p sofumax(BXX E), the update rule is defined as:
nnew f(5) Xp Xsoftmax(8X'
(10)
which is the Cancave-Conver Procedure (CCCP) for minimizing the energy and can be proven as converging globally.
Theorem A.1 (Global Convergence (Zangwill): Energy). The update rule Eq. 10 converges globally: For g++l = f()"), the energy E(g') E(E") for - oo and a fixed point g.
Proof. The Concave-Convex Procedure (CCCP) minimizes a function that is the sum of concave function and convex function. And since Ise is convex, -Ise concave function. Therefore, the energy function E(g) the sum of the convex function Ei (g) = 1 + C1 and the concave function Ez(5) = -lse:
E(E) = Ei + E2(5).
(11)
(E) = 5 5" 2 M2 5 g+ C₁.
1
E2(g) -Ise XT)
where Ci does not depend on g.
The Concave-Convex Procedure (CCCP) applied to E is
(gE1() -ve
(12)
which results in the update rule:
gl+l Xp' Xsoftmax(8X
(13)
where p' softmax(BXT This is the update rule in Eq. 10
A.1.2 Hopfield Update Rule is Attention of The Transformer
The Hopfield network update rule is the attention mechanism used in transformer. Assume N stored (key) patterns x, and S state (query) patterns r; with dimension dx, we can have X = (X),.... xy) and R (ri...., rs)" combine the and as row vectors. Define the key as k; = W qi W and multiply the result of the update rule (Eq. 10 with Wv. By defining the matrices
18

---

= (XWR)", Q (RW))', and V XW W W',,, where Wx E Rdexds We e Rdxdd,
Wv E Rdixdo If B 1/vdj and softmax e RN is changed to row vector, there is:
softmax
QKT
V softmax e XW
(14)
vdx
where the left part is the transformer attention, while the right part is the update rule Eq. multiplied by Wy.
A.2 Derivation of The Teacher Attention Update Rule
We first provide several lemmas for the derivation of the teacher atteettn update rule (Eq. Lemma A.2. For a given column vector E RN,, we have:
i log Σ exp( = softmax(Bx)
(15)
dx
Proof. Consider = Ein exp(8x,) and f(x) = log S.
afix) 8x
can be computed as:
0f(x) 0
1 as
log S
0xj
dxj
5
0xj
The partial derivative ds is: oxj
as à
N (xp = Σ exp(Bx,) =
N Σ [m
B exp((xx))0 Bexp(xx,)
0xj 0xj
0x,
7=1
where öij is the Kronecker delta.
Substitute as back into the expression for @f(x) 0xy
xxy
0s(x) 1
0x; Bexp((xx) = =B- exp(Ax,)
S
Recognize that (xp()x)) the j-th component of the softmax function applied Bx:
exp(Bx,) exp(Xx,)
softmax(Bx) (Bx),
ΣN exp(Bxi)
Therefore, we have:
i log EN exp(Bx:)
= softmax(Bx))
dx
Putting it back into vector notation:
log N exp(Bx;)
=B- softmax(6x)
0x
log Σ exp((xx
i log NN exp()x;
Since
B-1
we can have:
dx
i log Σ exx(
softmax(Bx)
dx
This confirms the lemma.
Lemma A.3. Let k; denote the i-th row vector of K E Nxd, Then, we have:
ok,k
(e.¹ 2e("(eNN)
2e
(16)
öK
where eÑ represents an N N-dimensional column vector where only the i-th entry is 1, with all other entries set zero.
19

---

Proof. First, note that the expression k,k? can be equivalently rewritten as e, k (e,))
To find the derivative of k/k/ with respect to K, we need to consider the individual elements of K. Let K = [k k j. Therefore, k/ is the i-th row of K, and we denote this as k K,,-
The differential of k, is given by:
d(k,k)) d(k) K..) d(k).) K,, (K.))
Since K ((,,, = ex (e, /KK,, = (e, (ef²) (()) and similarly for the transpose term, we get.
d(k,k)) =e (e dKK, + K. (e, (e())
Thus, we can summarize the derivative as:
ō T 2e; (e, (K,
0K
which matches Eq. 16
Lemma A.4.
Vxlse (Kk softmax(8Qkk) Q
(17)
Proof. Define z Qk?, we can rewrite Ise B) as:
Ise B) = 8 log Σ exp(Bz))
Using Lemma A.3 we have:
Velse(z, B) softmax (Bz)
(18)
Substitute z Qk? back to the expression, we have:
θz Vxlse(QK,, V,lse(z,B))
0K
With dz, Sh.
Qj: and Eq. 18 we have:
V Vklse Vklse(Qk/ softmax((Qkk)
Thus, the lemma is proved. Lemma A.5.
@diag KKT)
2K
(19)
ÖK
where diag(A) (A) denotes the trace of A.
Proof. Let us construct a column vector x whose i-th element is given by x; = k;kk /2. Then, using
Lemmas A.2 and A.3 and the chain rule, we have:
i log ΣN exp [kik?
ddiag
KK?
aK dx; Ølse(x, 1)
oK dx:
Σ eef (e,²) (/sftmaax()));
Because of Lemma A.2 and A.3
Σ k: (x)]; 2K
This proves the lemma.
20

---

Derivation of Eq. 7 Now we proof that we can approximate the posterior inference of p(K, | (s) by the gradient of the log posterior. estimated as:
VK, log p(,))Q,)
(Vx, E(Qui VVK E(K)))
sm BBQKK Q alI D sm ( d K, ddiag(K,k,)) K,
where sm(v) softmax(v) exp (v- lse(v, v. 1)) and D))) is dettor-toddaggonalmmaiix operator. Moreover, recall Eq. 5a and 5b
the energy EQQ ; K,) and E(K,) are denoted as:
E(Q.:K,) diag(K,K/
Σ i=1
Ise Q Q,KK, +c.
E(K,) = lse diagg diia K, K,K,),,
log
exp
- kki tc,
2
i=1
Proof.
Using Lemmas A.4 and A.5
we have:
VK, ((Qui ak; smQ,,,m
Since K, can be expressed as Ise( kkik?) incorporating Lemma A.5
we have:
Vx, EK,) D
diag diag(K,k,))) K,
2
Put them together, we have:
VK, log p (K) = sm sm(Bk.K)))
1 (ccl+
[ didag(KiK))) K.,
D sm
This matches Eq. 7
B Extended Experimental Analysis of Attention Alignment
In this section, begin by examining the representation discrepancy between old and classes, highlighting alignment issues (B.I) Enhanced consistency loss optimization is then detailed, showing
improvements in learning stability (B.2). We discuss how attention alignment bridges the prior gap and benefits synchronized learning, enhancing overall performance (B.3). The focus then shifts to attention specialization in deep network layers (B.4), and the performance impact of layer selection for attention alignment on different dataset (B.5). Finally, we represent the negligible impact on computational cost of the Attention Alignment strategy (B.6), thereby proving its practical viability.
B.1 Representation Discrepancy of Old and New Classes
In GCD, directly applying consistency regularization leads to challenges, especially for new classes. Due to the lack of prior knowledge, the teacher struggles guide the student, resulting in repre- sentation discrepancies between the teacher (weakly-augmented) and student (strongly-auge arrnnnly-augmenneed) This is evident in Fig. [0 (left), where new classes show poor alignment between teacher and student representations compared to known classes.
This discrepancy causes unsynchronized learning, shown in Fig. 10 (right). The two main phenomena observed are the learning gap and learning regression. The learning gap indicates that the student struggles to reach the teacher's 's level of understanding, particularly for classes, leading to stagnation. Learning regression affects the teacher, hampering improvement for new classes and causing regression in known classes due the alignment efforts with the student.
B.2 Enhanced Consistency Loss Optimization
Building on the previous discussion on the representation discrepancy of old and new classes, we address the challenges in optimizing consistency loss (Sec. 2.2 that contribute to the learning gap. To
21

---

Before Training
After Training
Learning Unsynchronization
Learning Effect (Acc.)
0.8
learning regression
Old Classes
0.6
0.4
learning gap
0.2
New Classes
0
50
100 Epoch
150
200
Known Strong Aug (Student)
New Strong Aug (Student)
Known Weak Aug (Teacher)
Student
Teacher
New Weak Aug (Teacher)
Figure 10: Left: Comparison of representation discrepancy with respect old and classes before
and after training, showing the misalignment of student (blue) and teacher (red). Right: Learning unsynchronization between teacher and student with trends of learning regression and learning gap for old and new classes
SCars
CUB
FlipClass (SCars)
FlipClass (CUB)
Coonsteary Loes
GT Class New Old
2.53%
16.77 16.77%
GT Class New Old
10.50%
18.20%
6.10%
40.10%
5.10%
23. 60%
100
Epuch
Epoch
New Predicted Class
Old
New Predicted Class
Old
StsOCD
Tickao +000
(a) Consistency loss optimization SCars and CUB, (b) Categorize errors of FlipClass. Compared to comparing FlipClass with various update rates (Yupdate) those SimGCD (Fig. right), the reduced errors to the SimGCD baseline, demonstrating more rapid and on 'False New' represent that FlipClass mitigates the stable convergence.
overfitting of old classes brought hy the prior gap-
Figure 11: Attention alignment bridges the prior gap with better-converged consistency loss and leads to less biased predictions.
verify how FlipClass improves this process, track the consistency loss Lcons classes. As shown in Fig. [Ia FlipClass with various update rates (Yupdate) demonstrates faster and more stable convergence compared to the SimGCD baseline. Specifically, the experiments on SCars and CUB datasets reveal that FlipClass streamlines the optimization process of consistency loss, leading to more rapid and stable convergence.
B.3 Attention Alignment Bridges Prior Gap and Benefits Synchronized Learning
The enhanced consistency loss optimization further aids in achieving consistency between the student and the teacher, reflecting in two main aspects. Firstly, mitigates the effects of prior gap, as shown in Fig. [1b] Compared to the categorize errors of SimGCD (Fig. 2). FlipClass reduces the 'False New' error (where the model incorrectly predicts new classes as old classes). This indicates that FlipClass can mitigate overfitting on old classes due to the lack of prior knowledge about new classes. Secondly, as shown in Fig. 12 compared to the traditional teacher-student model used in generalized category discovery (e.g.. SimGCD), which suffers from a learning gap, FlipClass successfully bridges the learning gap. It achieves better teacher-student learning effects, ensuring more synchronized and stable learning outcomes.
22

---

SCars
CUB
Learning Unsynchronized
Learning Synchronized
Learning Unsynchronized
Learning Synchronized
Learnnn EBsc [Ave.
01.8
0.8
Loaringg Elfier [Aoc.]
00.8
11.6
3.6
0.5
0.6
0.6
0.1
0.1
0.1
0.2
0.2
0.2
100 Epoch
200
100
200
100 Epoch
200
100
200
Epoch
Epoch
O3 (Teh)
Old (Sta)
New (Tch)
New (Stu)
Old (Tdh)
Old (Stx)
New (Teh)
New (Sha)
Figure 12: Learning curves for SCars and CUB datasets. FlipClass achieves better synchronized and stable learning effects compared to the traditional teacher-student model.
All Acc
Old Ace
New Acc
0.7
0.4
0.6
0.4
0.3
0.5
2 Layers
2 Layers
Layers
0.3
Layers
Layers
Layers
0.4
0.2
5
11
Layers
Layers Attn 4
Layers
Attn 0
Attm 2
Attn
Attn 8
Attn 10
AEETTY
Atin
Atin 2
Attn 4
Alin
Atin 8
Alin 10
Attention heatmap accuracy per layer on SCars dataset, with deeper layers focused local semantic features and carlier layers on general features, indicating better transfer learning for old and new classes with attention alignment deeper network layers.
Figure 13:
B.4 Attention Specialization in Deep Network Layers
Moreover, the SCars dataset, the model shows greater energy reduction in deeper layers, suggesting a focus on local semantics over general ones. This illustrated in Fig. [3 where deeper layers emphasize local semantic parts, and earlier texture, color). This local semantic concentration enhances transferability across 'Old' and 'New' classes, with attention alignment in deeper layers (9-10) yielding the highest performance. We observe that different heads attend to disjoint regions of the image, focusing on important parts. After training with our method, attention heads become more specialized semantic parts, displaying more concentrated and local attention. Our model learns to specialize attention heads (shown as columns) different semantically meaningful parts, improving transferability between labeled and unlabeled categories.
associated with general semantics (e.g.,
B.5 Attention Alignment in Layer Selection
Furthermore, we provide the performance of layer selection for attention alignment on CUB and Cifar- 10. this context, "2 layers" means performing attention alignment in the two layers depicted the Fig. 14 As shown, Cifar-10 (coarse-grained) tends to achieve higher performance when alignment is performed in the middle layers (4-5), while CUB (fine-grained) achieves higher performance with alignment in deeper layers, which aligns with the trend of SCars (Fig. 13) The difference stems from
23

---

Acc
Acc
New Acc
Cifar10
0.975
0.98
0.965
0.970
Layers
0.960
Layers
Layers
Layers
0.965
1
2
3
4
5
7
8
10 11
2
3
5
7
8
10 11
1
2
3
4
5
6 Layers
7
8
10
11
Layers
Layers
All Acc
Old Acc
New Acc
0.8
0.8
CUB
0.7
0.7
0.7
Layers
Layers
Layers
0.6
Layers
0.6
Layers
Layers
0.6
11
7
Layers
Layers
Layers
Figure 14: Layer selection performance for attention alignment on Cifar- and CUB datasets. Higher performance is observed in middle layers for Cifar- 10 and deeper layers for CUB, indicating the need to tailor attention alignment to the nature of the data.
the dataset nature. CIFAR-10 benefits from middle-layer alignment, capturing general features like shapes and textures, which suffice for its simpler categories. Conversely, CUB requires deeper layer alignment for detailed features needed distinguish similar bird species. Deeper layers provide the refined features critical for complex tasks.
B.6 Time Efficiency of Attention Alignment
potential concern is whether attention alignment increases the time cost. Since perform the alignment in a one-loop manner, the additional time overhead is negligible. This is demonstrated in Table showing that the training and inference times for FlipClass comparable to those of SimGCD.
Table 3: Comparison of training and inference times for SImGCD ImageNet-100 and AirCraft datasets, with 200 epochs
ImageNet-100
AirCraft
Method
Training time (Min) Infer. time (Sec) Training time (Min) Infer. time (Sec)
SimGCD 75
1639
591
224
17
FlipClass (Ours)
1648
591
229
17
We further analyze the potential extra overhead of our attention update strategy compared to conventional self-attention block:
Overall Training Cost: For Vision Transformer (ViT) with L layers, the training cost includes (1) self-attention cost, OLL N df); (2) feed-forward network (FFN) cost, (LL N d2 dm). Here, N is the sequence length, dg is the feature dimension, and dm is the hidden dimension in the FFN.
Attention Update Cost: Our update strategy applies only 2-3 layers (see Appendix B.5 with each update having the same cost one FFN operation per layer.
In sum, we add 2-3 extra FFN-like operations during the forward pass. And there is no impact on the backward pass, as only the last block is updated without introducing extra parameters for optimization. Therefore, the additional overhead minimal, with only a slight increase in training time, as shown in Table

---

Table Time cost (seconds) per forward pass on CUB and Stanford Cars.
Setting
CUB
SCars
w/o Attention Updating
65s
93s
w/ Attention Updating (2 layers)
71s
101s
w/ Attention Updating layers)
76s
109s
Table Complete results of FlipClass in have five independent runs with random seeds.
All
Old
New
Dataset
Mean
Std
Mean
Std
Mean
Std
CIFAR10
98.5
+ 0.1
97.6
0.2
99.0
+ 0.4
CIFAR100
85.2
0.3
84.9
0.5
85.8
+ 1.2
ImageNet-100
86.7
= 0.5
94.3
1.2
82.9
+ 0.9
CUB
71.3
= 1.4
71.3
#3.2
71.3
+ 2.0
Stanford Cars
63.1
+ 0.7
81.7
1.8
53.8
+ 1.7
FGVC-Aircraft
59.3
= 1..8
66.9
3.66
55.4
4.20
In summary, the attention update operation introduces small extra overhead equivalent 2- 3 additional FFN operations, which remains insignificant relative to the overall training cost, especially given the performance improvements achieved by FlipClass.
C More Experimental Results
We detail our experimental results, including main outcomes with error bars for statistical significance We evaluate performance on complex datasets (C.2). analyze clustering and per-class prediction examine robustness to varying numbers of old and new classes
(C.I).
distributions (C3) Additionally,
(C.4) and investigate how different proportions of old classes affect performance (C.5) demonstrating
method's stability and adaptability.
C.1 Main Results with Error Bars
5 error bars to provide a clear understanding of the statistical significance and variability of the main results in experiments. Specifically, include both the mean and standard deviation (std) values for the performance of our FlipClass across different datasets and class types (All, Old, New). The standard deviations are calculated from five independent runs with random seeds, offering insight into the consistency of our method's performance.
C.2 Results on Complex Datasets
Here, we discuss the performance of various methods on challenging datasets, specifically Herbar- ium19 [65] and ImageNet- IK [35]. Herbarium 19, a long-tailed dataset, poses significant challenges due the varying frequencies of different categories, leading unbalanced cluster sizes. Table 6 demonstrates the robustness of our proposed FlipClass, in handling such frequency imbalances and its ability accurately distinguish categories even with few examples. Table fablee]]showcase the performance on ImageNet- MrageNet-1K.. large-scale generic classification dataset, in evaluating the model's s capability in real-world applications of generalized category discovery. The results demonstrate the robustness of FlipClass for tasks involving both familiar and unfamiliar data in complex, real-world scenarios.
C.3 Clustering and Per-class Prediction Distribution
Clustering Analysis. Fig. presents visual comparison of the clustering results obtained with FlipClass against the existing state-of-the-art method, InfoSieve (55), on Cifar- and Cifar- 100 datasets. On Cifar-10, 10, FlipClass forms clusters that exhibit higher compactness and purity, indicating
25

---

Table Performance comparison of different methods on the Herbarium 19 dataset.
Herbarium19
Methods
Pretraining
All
Known
Novel
k-means 1967
DINO
13.0
12.2
13,4
ORCA 2021
DINO
20.9
30.9
15.5
RS+ [2020
DINO
27.9
55.8
12.8
UNO+ 2021
DINO
28.3
53.7
12.8
GCD 2022
DINO
35.4
51.0
27.0
CMS 2024
DINO
36.4
54.9
26.4
OpenCon on 2022
DINO
39.3
58.9
28.6
InfoSieve
2024
DINO
41.0
55.4
33.2
MIB 2022
DINO
42.3
56.1
34.8
PCAL 2023
DINO
37.0
52.0
28.9
SimGCD 2023
DINO
44.0
58.0
36.4
AMEND 2024
DINO
44.2
60.5
35.4
μGCD [2024]
DINO
45.8
61.9
37.2
FlipClass (Ours)
DINO
46.3
60.2
40.7
Table 7: Performance comparison of different methods on the ImageNet-IK dataset.
ImageNet-1K
Methods
Pretraining
All
Known
Novel
GCD 69
DINO
52.5
72.5
42.2
SimGCD 2023
DINO
57.1
77.3
46.9
FlipClass (Ours)
DINO
59.2
78.9
49.5
enhanced feature discrimination and reduced interclass confusion. In contrast, on Cifar-100, although InfoSieve forms visually more compact clusters, these clusters show less purity, with higher incidence of false class predictions.
Prediction Distribution and Class-specific Accuracies. Fig. evaluates the prediction distribution and class-specific accuracies of FlipClass compared InfoSteve [55]. FlipClass demonstrates better fit to the true distribution, whereas InfoSieve shows skewed predictions. Moreover, FlipClass significantly outperforms InfoSieve in recognizing tail classes on the CUB and Stanford Cars datasets, improving accuracy and reducing prediction bias.
C.4 Robustness to Number of Classes
Varying Number of Classes during Clustering. In the main experiments (Section 5) the class number, K, is assumed as known prior following prior works 669 64
75 70), however, this Fig. 17 we conduct experiments when
setting has been questioned as impractical 9 this assumption is removed, evaluating results with different numbers of classes, where the ratio changes from 80% 200% compared to the ground truth number of classes. During clustering (e.g.. KMeans (5)). a predefined class number lower than the ground truth significantly limits the ability to discover new classes, causing the model focus more on old classes. Conversely, increasing the class number results in less harm to the generic image recognition datasets (e.g., Cifar-100) and can even be beneficial for some fine-grate gra-nraid,,, long-tailed datasets (c.g., CUB). This phenomenon occurs because overestimating the number of classes allows the model to maintain higher flexibility and adaptability in recognizing new classes in these fine- ime-ggrinedd challenging datasets. For fine-grained, class-distribution biased datasets like CUB, overestimating the class number helps capture subtle differences between closely related categories, thereby improving class separation and reducing prediction bias. However, for generic datasets like C0far-100, the visual differences between classes are more pronounced, and overestimating the class number can introduce unnecessary complexity, leading to overfitting and decreased performance.
Estimation Number of Classes. Additionally, to further validate the robustness of our model, we trained FlipClass using estimated number of classes in the dataset, where the number of classes
26

---

GCD
InfoSieve
FlipClass
Cifar10
Cifar
Figure 15: Comparison of clustering results on Cifar-10 and Cifar-100 datasets using GCD, InfoSieve, and our FlipClass.
Cifar10
CUB
SCars
InfoSieve
Accuracy
10000
Frequency
Accuracy
1.0
Frequency
Accuracy
1.0
Preeuencyy
S0
50
0.5
0123456783
Classes
0.0
Cluss prediction
0.0
FlipClass
Accuracy
1.00
5000
Frequene
Accuracy
1.0
Frequency
Accuracy
1.0
Frequency
0.75
0,5
Classes
0.0
by count)
0.0
(verted by predictios count)
Figure 16: Prediction distribution and class-specific accuracies of InfoSieve and FlipClass
Cifar-10, CUB, and Stanford Cars datasets.
predicted using the over-clustering method from GCD 669 We obtained similar predicted number of classes as SimGCD. As expected, our method performs worse on Cifar-100 when using an estimated number of classes due the mismatch between the estimated and actual class distribution. Interestingly, the performance of SimGCD (traditional teacher-student model) improves on CUB with both new and old classes, while our FlipClass makes improvements on old classes but decrease in new classes.
C.5 Results with Varying Proportion of Old Classes
In the primary experiments, we fix the number of the old classes |Cel (details in Appendix D.3) Here, we experiment with our method by changing the class split setting. Specifically, on Cifar- 100 ([Crl = 80) and CUB ((Crl 100) datasets, we test with fewer old classes, as shown in Fig. 18 For Cifar- 10f and CUB, as the number of old classes decreases, the accuracy for both old and new classes slightly declines but remains stable. This demonstrates FlipClass's effectiveness in leveraging additional old class information and robustness in handling varying numbers of known classes.
27

---

CIFARIO [K- (K0)
InageeNel- 1001 (K-100)
CUB [K 200)
Stanford Cats [K- 196)
80
New ACC (6)
8/1
70
80
501
62
75
6C
Ratio to GT Class Number
Racio to GT Class Number
Ratio to GT Class Number
Ratio to GT Class Nuraerr
85
Oli Are (86
85
95
90
75
90
05
70
Ratio
Class
Hailo
Closss
Cluss Nunden
Ratio
Notaler
Figure 17: Results with varying numbers of classes during clustering, where the ratio changes from 80% 200% of the ground truth number of classes.
Table 8: Performance of FlipClass and the baseline method SimGCD with estimated number of categories on CUB and Cifar 100. Bold values represent the best results.
CUB
Cifar100 Old
Method
|Canl/Crrl
All
Old
New
All
New
SimGCD 75
GT (200/100)
60.3
65.6
57.7
80.1
81.2
77.8
FlipClass (Ours)
GT (200/100)
71.3
71.3
71.3
85.2
84.9
85.8
SimGCD 75
Est. (231/109)
61.0
66.0
58.6
81.1
90.9
76.1
FlipClass (Ours)
Est. (299/108)
70.5
72.7
69.4
84.2
84.3
84.1
D Experimental Settings
D.1 Implementation Details
We develop our FlipClass upon the SimGCD [75] baseline on the pre-trained ViT-B/16 DINC [15] Specifically, we take the final feature corresponding the CLS token from the backbone as the image feature, which has a dimension 768. For the feature extractor F, we only fine-tune the last block. We set the balancing factor A to 0.35 and the temperature values Tn and T. to 0.07 and 1.0, respectively, following SimGCD. For the temperature values T, and TN in the classification losses, we also set them to 0.07 and 0.1. For update rule (Eq. [8.
O, B 1, Yupdate 0. and Yrgg = 0.5. All experiments are conducted using a single NVIDIA A100 GPU with 200 epochs, which find sufficient for the losses plateau.
D.2 Design of Data Augmentation
In this experimental setup, we design both weak and strong augmentations for the teacher and student networks. For weak augmentation, we use common techniques such as RandomCrop for all datasets, aiming to pass less perturbed versions of the input images to the teacher network. For the strong augmentation that is applied to the images fed to the student, we incorporate more aggressive transformations to expose the student to wider range of variations. Specifically, we add RandomResizedCroo with a scale range of 0.3 to 1.0, which allows for more aggressive cropping and resizing. Additionally, we include Gaussian blurring to simulate different levels of image blurriness. For datasets that are used for generic recognition tasks, we further enhance the strong augmentation by including ColorJitten with probability 0.8 and RandomGrayscall with probability 0.2. Solarization inverts pixel values above a threshold, simulating the effect of solarizing image, while Grayscale converts the image to black and white, reducing color information. These additional augmentations help expose the student network to even more diverse image variations, improving its robustness and generalization capabilities.
and
28

---

Cifar100
CUB
Accuracy (%)
Accuracy (%
80
70
70
50
60
20
100
Number of Old Classes [cp
Number of Old Classes |cr]
Old
New
Figure 18: Results with varying the number of old classes |Crl.
D.3 Datasets
Table 9: Statistics of the datasets used in our experiments.
CIFARIO
CIFAR100
ImageNet100
CUB200
Stanford Cars
Aircraft
Herharium
y
5
80
50
100
98
50
341
L
10
100
100
200
196
100
683
D
12.5k
20k
31.9k
1.5k
2.0k
1.7k
8.9k
D
37.5k
30k
95.3k
4.5k
6.1k
5.0k
25.4k
We follow the dataset settings from earlier works [75] 70 to subsample the training dataset. Specif- ically, 50% known categorics and all samples of unknown categories are used for training. For considered known during training, whereas for Cifar- 100, 80% the categories are known during training. Detailed statistics are displayed in Table 9 Below is summary of the datasets and how their combination supports experiments in generalized category discovery:
all datasets except Cifar- 50% of the categories
Cifar-10/10d 34 are coarse-grained datasets consisting of general categories with low-resolution images and even class distribution.
ImageNet-100//Kf) is subset of 100/IK categories from the coarse gaarse-grained ImageNe 35 58 dataset. It includes a large scale of high-resolution real-world images with evenly distributed classes.
is widely used for fine-grained image recognition, containing different bird species distinguished by subtle details.
CUB Caltech-UCCD Birds-200-201
Stanford Cars 33
fine fine-grained dataset of various car brands, providing multi-vies objects for class detection and scene understanding, challenging real-world applications in distinguishing subtle appearance differences.
FGVC-Aircrafi (Fine-Grained Visual Classification of Aircraft nized in a three-level hierarchy. At the finer level, differences between models subtle but visually measurable. Unlike animals, aircraft are rigid and less deformable, presenting variations in purpose, size, designation, structure, historical style, and branding.
46 is fine-grained dataset orga-
deacription
https://ww.rooots. uw. http://wwww.oo.oooooooooooooaaaa

---

Herbarium 19 (FGVC 2019 Herbarium Challenge 65 provides a curated dataset of over 46,000 herbarium specimens across 680 species, presenting long-tailed distribution and challenges for species recognition.
D.4 Other Alignment Strategies
In the ablation studies (Section 5.3), we apply different strategies such distribution alignment M. FixMatch [63] and CORAL. These strategies are employed to encourage the consistency between the teacher and student, therefore modeling R in Eq. We briefly provide the main idea of these alignment strategies below.
Distribution alignment is designed by maintaining rumning average of the model's predictions
on unlabeled data p(y). Given the model's prediction q = (m x,) on an unlabeled example Xu, q is scaled by the ratio p(y)/p(y) and then renormalize the result to form a valid probability distribution: q Normalize q X p(y) (()).
FixMatch is semi-supervised learning method that combines consistency regularization and pseudo- labeling. works by first generating pseudo-labels for unlabeled images using the model's 's predictions on weakly enhanced versions of those images, retaining only high-confidence predictions as following:
uB
1
Σ 1 (max (gb) r) H (Âb. Pm (y A (u)))),
Cu
BB
where r is a scalar hyperparameter denoting the threshold above which a pscudo-label should be retained. Then, the model is trained to predict these pseudo-labels using strongly enhanced versions of the same images. The loss function consists of two cross-entropy terms: a supervised loss for labeled data and unsupervised loss for unlabeled data, where the unsupervised loss utilizes the pseudo-label, calculated from weakly enhanced images and the model's predictions on strongly enhanced images.
CORAL (CORelation ALignment) aligns the second-order statistics (covariances) of two spaces. Specifically, CORAL minimizes the difference in covariance matrices between the student and teacher representations (S and T). The goal of CORAL is to find a transformation for that minimizes the Frobenius norm of the difference between the covariance matrices of S and T, denoting Cs and Cr, respectively. CORAL minimizes the following objective:
min
||Cs Crllf. S'
where Cs' is the covariance matrix of the transformed student representations S', and ||-I| denotes the Frobenius norm.
E
Related Works
E.1
Consistency Regularization
In semi-supervised learning (SSL), the goal is to enhance model performance by leveraging unlabeled data, traditionally drawn from the same class spectrum as the labeled data [90]. A key strategy in SSL, consistency regularization (36). in recent years, centers on promoting model stability by ensuring instance) and the student instance (strongly-augmented 87 83). Building the II-Model's 's teacher-student
that the teacher instance (weakly-augment instance) yield coherent predictions [I 63 78
framework, several approaches have advanced its capabilities [66, 43 77
12 5
MeanTeacher
(66) deploys an exponential moving average of the model parameters stablize the teacher's output. NoisyStudent [77] employs a self-training strategy that incorporates noise into the student model's training, cycling the improved student back into the teacher role. Previous methods in SSL. have largely concentrated on promoting the teacher's performance, often overlooking whether the student can keep pace, and neglecting the harmony of interaction. Our approach pivots synchronizing the teacher's 's and student's 's attention, a shift that's 's especially pivotal in GCD, where consistency challenged by the introduction of new classes. This strategy ensures balanced teacher-student dynamic, crucial for effective consistency regularization in the open- world setting.
ihtpps: htwp:://wwww... htw.://wwgg...o/////oorrrr.aaaaa
30

---

E.2 Novel Category Discovery
Novel category discovery (NCD) first formalized as cross-task transfer in [30, which aims to discover unseen categories from unlabeled data that have nonoverlapped classes with the labeled ones. Earlier works [31 26
79 38 mostly maintain two networks for learning from labeled and unlabeled data respectively. AutoNovel (27] introduces three-stage framework. Specifically, the model firstly trained with the whole dataset in self-supervised manner and then fine-tuned only with the fully-supervised labeled set to capture the semantic knowledge for the final joint-learning stage. UNO [22] addresses the problem by jointly modeling the labeled and unlabeled sets to prevent the model from overfitting to labeled categories. Similarly, NCL [88] generates pairwise pseudo labels for unlabeled data and mixes samples in the feature space to construct hard negative pairs.
86
E.3 Generalized Category Discovery
Generalized Category Discovery (GCD) extends NCD by categorizing unlabeled images from both seen and unseen categories [69]. which tackles this issue by tuning the representation of the pre-
trained VIT model with DINO (115), [51] with contrastive learning, followed by semi-supervised k-means clustering. ORCA [3] considers the problem from a semi-supervised learning perspective
and introduces an adaptive margin loss for better intra-class separability for both seen and unseen classes. CiPR (28) introduces a method for more effective contrastive learning and a hierarchical clustering method for GCD without requiring the category number in the unlabeled data to be known priori. SimGCD 75 proposes a parametric method with entropy regularization to improve performance. TIDA (74] discovers multi- guatiigranullrity semantic concepts and then leverages them to enhance representation learning and improve the quality of pscudo labels. Moreover, ,GCD [70] take leap forward by extending the MeanTeacher paradigm the GCD task. Instead of managing dual models as in μGCD, our approach achieves teacher-student consistency more effectively within a single-model structure, streamlining computational demands. Crucially, we found that the learning discrepancy between teachers and students in the open-world context is the reason why consistency is difficult achieve, and solved this problem by synchronizing the attention of teachers and students.
F
Limitations and Future Work
Catastrophic Forgetting. While our method achieves significant improvement on new classes, the performance on old classes, particularly on CUB, lacks compared to the state-of-the-art methods. We attribute this to phenomenon akin to catastrophic forgetting, where the model forgets previously learned concepts. Addressing these issues is essential for enhancing the robustness and effectiveness of the proposed methods.
Sub-optimal K Estimation. As shown in Appendix C.4 for fine- grained, class-distribution biased datasets like CUB, overestimating the class number helps capture subtle differences between closely related categories, thereby improving class separation and reducing prediction bias. However, for coarse- coarse-grained datasets like CIFAR- -00, overestimating the class number can introduce unnecessary complexity, leading to overfitting and decreased performance. Some works have delved into this path and show promising performance [73 74 9 highlighting the potential of tailored K estimation strategies to balance complexity and performance across different types of datasets.
Data Augmentation to Enhance Teacher-Student Consistency. Effective data augmentation techniques has been investigated a lot in semi-supervised learning [18 82 48 76 29], the techniques for generalized category discovery are still lacking, which affects the consistency between the teacher and student models. The strength of data augmentation for new classes needs careful control to avoid ineffective learning due excessive noise or insufficient variability. Additionally, preventing data leakage during augmentation is critical, as pretrained diffusion models can compromise evaluation integrity by leaking training data. Addressing these issues is essential for enhancing the robustness and effectiveness of the proposed methods.
G Broader Impacts
Our study extends the capability of Al systems from the closed world the open world, fostering Al systems capable of categorizing and organizing open-world data automatically. While Generalized
31

---

Category Discovery (GCD) has many real-world applications, it can be unreliable and must be applied with caution. Currently, supervised learning with extensive fine annotations is the mainstream solution for many computer vision tasks, but the cost and difficulty of obtaining these annotations can be prohibitive. Our work addresses this issue by advancing an open-set eemi-supervised learning paradigm, significantly reducing the need for precise annotations and promoting the application of Al models in areas where annotations are difficult to obtain.
This work provides a new idea for open-sel semi-supervised learning. Specifically, while conventional approaches apply closed-wond world semi- cemi-suprvvised learning techniques generalized category discovery, they rarely consider the attention alignment gap between teacher and student models. We point out that bridging this gap can significantly improve learning efficiency and accuracy. We hope that this methodology can be generalized to more relevant label-efficient tasks, promoting broader applications of Al in scenarios with limited labeled data.
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction (Section accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work discussed in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
33

---

Justification:
Concerning the proposition 4.1 Appendix A.1 and A provide its corresponding
proofs, with a short proof sketch to provide intuition.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Refer to Appendix D.I for implementation details, the provided code in the supplementary materials ensure the reproducibility well.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
34

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Refer to provided code in the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wttp:ccppppcccceeeeeeeeee..c.ett http:///gggeeee/ooeSSuuiiiiisss Coooeooomisssssseeeeeeee www.iissssiiiiiciico for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Table specifies all the training and test detail. Appendix D.I provides the employed hyperparameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Appendix provides the statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
35

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix D.I and provide sufficient information on the computer resources and the related costs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttt:://eeete ctcpppps.ccppuiiie http:///cc.////ceccccccceeeeeeee
Answer: [Yes]
Justification: The authors have reviewed and understand the NeurlPS Code of Ethics, and confirm that their research conforms to it in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Potential societal impacts of the work are discussed in Appendix G
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
36

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The original papers of assets, and the corresponding versions are provided in Section 5) References and Appendix D.I
Justification:
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
37

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We will well document the assets when we officially release our code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not describe potential risks incurred by study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
38

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
39


Reviews:
[{"review_id": "HWLdFtyvW3", "reviewer": "Reviewer_MqmB", "summary": "The paper introduces FlipClass, a dynamic teacher-student attention alignment strategy designed to address the challenges of Generalized Category Discovery (GCD) in open-world scenarios. Unlike traditional teacher-student frameworks, FlipClass updates the teacher’s attention to align with the student’s evolving focus, thereby promoting consistent pattern recognition and synchronized learning across both old and new classes. Extensive experiments validate FlipClass’s superiority over existing methods, establishing new benchmarks in GCD performance.", "questions": "1.\tWhile the paper claims that FlipClass significantly improves attention alignment (a design of $\\R$ in Insight 3.1), could you provide more detailed quantitative metrics and comparative analyses with other attention alignment strategies to strengthen this claim?\n2.\tIn Table 4, why does the accuracy of new classes increase significantly at the expense of the accuracy of old classes on the CUB dataset? Can you explain this phenomenon?", "limitations": "The necessary experiment of ablation study misses explanations for the strong augmentation and regularization.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tInnovative Approach: \n–\tThe dynamic teacher-student attention alignment strategy is novel in continuously updating the teacher's focus based on the students, ensuring synchronized learning and significantly advancing over static models.\n2.\tInterpretability:\n–\tClear visualizations, including attention heatmaps and energy dynamics, effectively demonstrate how attention alignment between teacher and student improves learning outcomes.\n3.\tTraining Details:\n–\tThe detailed description of the teacher-attention update rule and the theoretical foundation provided in the appendices contribute to a thorough understanding of the training process.\n4.\tExperimental Validation:\n–\tThe extensive experiments conducted on various benchmarks, strongly support the claims made by the authors; the experimental analysis clearly demonstrates the importance of attention alignment, further validating the approach.", "weaknesses": "1.\tExperiments:\n–\tCertain details in the ablation study, like the impact of strong augmentations and regularization, could be more clearly explained.\n–\tThe font size in Fig. 6 is too small and should be increased for better readability.\n2.\tWriting and Presentation:\n–\tConsistency in terminology and notation throughout the paper needs improvement to avoid confusion. For example, clearly distinguishing between different types of augmentations and regularization techniques used in the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive feedback and constructive comments.  Before addressing your inquiries, we wish to clarify certain weaknesses highlighted in the review that we believe require further elucidation.\n\n\n**Response to W1**: \n>  ***Certain details in the ablation study, like the impact of strong augmentations and regularization, could be more clearly explained.***\n\nThank you for your suggestion. \n\nThe impact of strong augmentations and regularization are as follows:\n- **Strong augmentations** are introduced to expose the student network to a **wider range of image variations**, thereby **enhancing its robustness and generalization capabilities**. \n- **Regularization** during the attention update **integrates the prior energy of the teacher**, preventing any single student pattern from overly influencing the teacher’s attention. \n\nThese clarifications have been added to Section 4 in the revised manuscript.\n\n\n&nbsp;&nbsp;\n\n**Response to W2**: \n>  ***Writing and Presentation: Consistency in terminology and notation throughout the paper....***\n\nThanks for your detailed review. We've revised these notations and fonts to improve the readability of our paper.\n\n---\n\n&nbsp;&nbsp;\n\n**Response to Q1**: \n>  ***While the paper claims that FlipClass significantly improves attention alignment (a design of $\\color{red}\\Re$ in Insight 3.1), could you provide more detailed quantitative metrics and comparative analyses with other attention alignment strategies to strengthen this claim?***\n\nCertainly! Thank you for the suggestion.\n\nThank you for your advice. Below, we provide detailed **quantitative comparison of different alignment strategies between the student and teacher attention maps**. \n\nThe strategies we experimented with include *$l_2$ loss, Kullback–Leibler divergence (KLD) loss, and Correlation Alignment (CORAL) loss*, as well as the *Semantic-Guided Multi-Attention Alignment (MAL)* method [1].\n\n| Attention Alignment Methods | CUB  |      |      | SCars |      |      |\n| --------------------------- | ------ | ------ | ------ | ------- | ------ | ------ |\n|                             | All  | Old  | New  | All   | Old  | New  |\n| **Ours (*FlipClass*)**        | **71.3** | **71.3** | **71.3** | **63.1**  | **81.7** | **53.8** |\n| $l_2$ Loss                  | 62.1 | 63.6 | 61.4 | 48.2  | 64.0 | 40.3 |\n| KLD Loss                    | 64.5 | 70.3 | 61.7 | 52.7  | 72.8 | 42.6 |\n| CORAL                       | 61.1 | 67.7 | 57.8 | 48.3  | 67.9 | 38.5 |\n| MLA                         | 68.3 | 70.4 | 67.2 | 56.9  | 73.4 | 48.7 |\n\n\nOur method achieves the highest accuracy across both datasets, demonstrating the effectiveness of our energy-based alignment strategy. \nThis approach allows for dynamic and nuanced adjustment of the teacher's attention, leading to better alignment with the student's evolving synchronized learning. \n\nWhile MAL shows closer performance to our method compared to other strategies, it still does not match our accuracy, underscoring the unique advantages of our energy-based strategy.\n\n\n&nbsp;&nbsp;\n\n**Response to Q2**: \n>  ***In Table 1, why does the accuracy of new classes increase significantly at the expense of the accuracy of old classes on the CUB dataset?***\n\nThank you for your thorough review.\n\nAs shown in Table 1, methods such as *PCAL, $\\mu$GCD*, and *AdaptGCD* also achieve comparatively similar accuracy of new and old classes on the CUB dataset. \n\nWe attribute this to the small scale of the CUB dataset, which contains only 6,000 images with a large class split (200). \nThis smaller dataset size might reduce the tendency to overfit the old classes, leading to a more balanced accuracy across new and old classes.\n\n&nbsp;&nbsp;\n\n*[1] Semantic-guided multi-attention localization for zero-shot learning. NeurIPS. 2019.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1SPL5DIdhZ", "reviewer": "Reviewer_MRfK", "summary": "This work proposes an attention alignment technique based on the Hopfield network energy function. Specifically, this work proposes to update the teacher model to increase the posterior teacher likelihood given the current student, which is modeled with the Hopfield network energy-based model. The teacher update process derived from the conditional score function is proven to be globally convergent. Experimental results showcase its considerable improvements over previous state-of-the-art baselines. The attention alignment technique is well-motivated with in-depth analysis. The contributions of this work include (1) investigating and discovering the attention alignment inconsistency between student and teacher models for the generalized category discovery problem, (2) proposing a theoretically inspired attention alignment method to address this issue, which is guaranteed to converge globally, and (3) achieving considerable performance gains compared with previous sota models.", "questions": "questions:\n\n- Baselines: Some baselines on generic and fine-grained datasets are not consistent in Tables 1 & 2, could you please tell us the reasons? It is better to make them consistent unless there is any special reason.\n- References: Is there any related work on utilizing the attention alignment technique?\n- Figures: Are the two methods visualized in figure 8b within the same projection space and with the same scale? Better provide zoom-in comparisons since there are two many classes.", "limitations": "Social impact: current methods are not applicable to real-world safety-demanding applications.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "strengths:\n\n- The idea of updating the teacher by aligning attention layers of the student is innovative.\n- The experimental results are comprehensive.\n- Different alignment strategies have been compared and the superior efficacy of attention alignment has been validated.\n- The global convergence of the update rule of the teacher model is proved.", "weaknesses": "weaknesses:\n\n- Writing: It is highly recommended to put the related work in the main content rather than in the appendix.\n- Ablations: There lacks the analysis on the hyperparameter \\alpha, and the rationale behind setting \\alpha=0 is to be explained.\n- line 141: incorrect symbol\n- Ablations: Better to provide comparisons in the ablations on varying class numbers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable feedback. Before addressing your inquiries, we wish to clarify certain weaknesses highlighted in the review that we believe require further elucidation.\n\n**Response to W1**\n> ***Put the related work in the main content rather than in the appendix.***\n\nThanks for your suggestions! \n\nWe've incorporated a concise version of related work in appendix into the revised main content, and make the experimental analysis more concise while keep it easy to follow.\n\n&nbsp;&nbsp;\n\n**Response to W2**\n> ***Analysis on the hyperparameter \\alpha, and the rationale behind setting \\alpha=0.***\n\nThank you for pointing this out. \n\n**Recap of Eq. 8**: the regularization term introduced by the prior energy prevents a single teacher key from dominating the attention, while the attention term updates the teacher keys in the direction of student queries, improving the consistency of focused patterns.\n\nIn practice, we found that a **nonzero** $\\alpha$ **(in Eq. 8) often leads to over-penalization, causing some teacher keys to vanish, and we found that setting $\\alpha=0$ gives the most consistent results**.\n\nDuring experiments, we conducted an **analysis of $\\alpha$ on the Stanford Cars dataset** to determine the optimal value and applied it to all datasets. \n\n**The results are provided in Table 1 in the attached PDF**.\n\nThese results highlight that $\\alpha=0$ yields the best performance across all categories. \n\nWe hope this clarifies our rationale.\n\n&nbsp;&nbsp;\n\n**Response to W3**\n> ***line 141: incorrect symbol.***\n\nWe sincerely appreciate your meticulous review, and we have corrected the mentioned typo in our revised manuscript. \n\n\n&nbsp;&nbsp;\n\n**Response to W4**\n> ***Provide comparisons in the ablations on varying class numbers.***\n\nYes, we agree. \n\nWe have included ablations on varying class numbers in **Appendix C.4 \"Robustness to Number of Classes\"**. Please kindly refer to it.\n\nWe will also consider incorporating these results into the main content.\n\nIn summary, our model demonstrates robust performance across different numbers of classes, maintaining consistent accuracy and stability. \n\n---\n\\\n**Response to Q1**\n> ***Some baselines on generic and fine-grained datasets are not consistent in Tables 1 & 2, could you please tell us the reasons?***\n\nSure, I'd like to, and thanks for your meticulous review. \n\nThe reasons for the inconsistency in baselines between Tables 1 and 2 are as follows:\n1. Inconsistency between Tables 1 and 2: **In detail, *XCon*, *PCAL*, *$\\mu$GCD* [1] are missed in Table 2, which are present in Table 1**.\n2. **Page Limits**: Due to space constraints, we removed some methods (*e.g., XCon*, *PCAL*) from Table 2 on generic datasets, since this dataset is less challenging compared to fine-grained datasets. \\\n**The results for XCon and PCAL on both generic and fine-grained datasets are provided in Table 2 in the attached PDF**.\n3. **Unavailable Code**: $\\mu$GCD [1] did not release their code, and their official paper does not report results on generic datasets. We attempted to reproduce their results but were unsuccessful. \nTherefore, we only reported their results on fine-grained datasets.\n4. **Method Ordering**: We **sorted the methods based on their All Acc**, which may result in different orders between Tables 1 and 2.\n\n&nbsp;&nbsp;\n\n**Response to Q2**\n> ***Is there any related work on utilizing the attention alignment technique?***\n\nYes, and thanks for the suggestions. Here is a concise summary of the attention alignment strategies in related works:\n- *RDAN* [2]: Utilizes a dual attention network to infer visual-semantic alignments by aligning attention across image regions and textual descriptions.\n- *MAL* [3]: Employs multi-attention localization to discover discriminative parts of objects for zero-shot learning, aligning attention based on semantic guidance.\n- *Alignment Attention* [4]: Focuses on aligning key and query distributions to improve the accuracy and effectiveness of attention mechanisms.\n- *Multi-level Representation Learning* [5]: Uses semantic alignment to enhance multi-level representation learning, aligning attention across different levels of representation.\n\nDifferences and Contributions of *FlipClass*:\n- **Generalized Category Discovery**: FlipClass is tailored for generalized category discovery in open-world settings, addressing challenges in semi-supervised learning scenarios and maintaining robust performance across different datasets and distributions.\n- **Teacher-Student Framework**: Unlike other methods that focus on visual-semantic or multi-level semantic alignments, FlipClass integrates attention alignment within a Teacher-Student framework. \nThis allows the teacher's guidance to adapt based on the student's learning, promoting better synchronization and performance.\n\nWe also provided experimental comparison with MAL [3] in **Response to Q1** with ***Reviewer MqmB***, please kindly refer to it.\n\n&nbsp;&nbsp;\n\n**Response to Q3**\n> ***Are the two methods visualized in figure 8b within the same projection space and with the same scale?***\n\nYes, during visualization. We utilize the same components for t-SNE and PCA to ensure the results of compared methods are projected in the same space with the same scale.\n\n> ***Better provide zoom-in comparisons.***\n\nWe appreciate your advice, and please kindly refer to **Fig.1 in the attached PDF**, which present the zoom-in comparison for better analysis of the representation space.\n\n&nbsp;&nbsp;\n\n*[1] μGCD: No representation rules them all in category discovery. NeurIPS, 2024.*\n\n*[2] Multi-level visual-semantic alignments with relation-wise dual attention network for image and text matching. IJCAI. 2019.*\n\n*[3] Semantic-guided multi-attention localization for zero-shot learning. NeurIPS. 2019.*\n\n*[4] Alignment attention by matching key and query distributions. NeurIPS. 2021.*\n\n*[5] Multi-level representation learning with semantic alignment for referring video object segmentation. CVPR. 2022.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ND3rYxOSMz", "reviewer": "Reviewer_P1mR", "summary": "The paper introduces FlipClass, a novel method addressing the challenges of Generalized Category Discovery (GCD) in open-world scenarios. It identifies the misalignment of attention between teacher and student models as a key issue hindering effective learning, especially when new classes are introduced. To tackle this, FlipClass dynamically updates the teacher's attention based on feedback from the student, promoting synchronized learning and consistent pattern recognition. Extensive experiments demonstrate that FlipClass outperforms contemporary GCD methods.", "questions": "1. Does the alignment of attention in the paper require the introduction of some prior knowledge?\n\n2. Do distributional alignment[1, 2] and this type of representation alignment conflict with each other? Have you conducted further experiments on combining them?\n\n3. From the description in the paper, the representation alignment of FilpClass seems to be distribution-agnostic. Can its performance still be maintained under long-tail distribution conditions[3]?\n\n[1] Open-World Semi-Supervised Learning. ICLR 2022.\n[2] Robust Semi-Supervised Learning when Not All Classes have Labels. NeurIPS 2022.\n[3] Towards Distribution-Agnostic Generalized Category Discovery. NeurIPS 2023.", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "1. This paper proposes an algorithm for Generalized Category Discovery. The authors underscore the significance of representation alignment between teacher (weak augmented) and student (strong augmented).\n\n2. The idea about aligning representation is interesting.\n\n3. The authors have provided the source code for reproduction.", "weaknesses": "1. The content regarding \"the Hopfield Network\" and the underlying motivation could be enhanced，and the current presentation is not clear and optimal.\n\n2. The SOTA method SPTNet [1] should also be compared in the paper. \n\n3. The differences between distribution alignment [2,3] and representation alignment in the paper need further discussion to make it easier for readers to understand your contributions.\n\n[1] SPTNet: An efficient alternative framework for generalized category discovery with spatial prompt tuning. ICLR 2024.\n[2] Open-World Semi-Supervised Learning. ICLR 2022.\n[3] Robust Semi-Supervised Learning when Not All Classes have Labels. NeurIPS 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your meticulous review and valuable feedback.\n\n**Response to W1**\n> ***The content regarding \"the Hopfield Network\" and the underlying motivation could be enhanced.***\n\nThank you for the suggestions! We have moved the **state-query capacity of the Hopfield Network** from Appendix A.1 to the main body and added more related work as suggested by ***Reviewer MRfK***.\n\nWe clarified the motivation by showing how the Hopfield Network's pattern storage and retrieval align with our goal of attention alignment. Its dynamic updates and energy minimization inspire our method to adjust the teacher's attention based on the student's focus, ensuring better synchronization and learning outcomes.\n\n&nbsp;&nbsp;\n\n**Response to W2**\n> ***SPTNet [1] should also be compared in the paper.***\n\nThanks for the suggestion. Below, we compare *FlipClass* with *SPTNet* [1] and report the accuracy of Old, New, and All classes. \n\n1. Superiority: Our method achieves the best results across all datasets, as shown in **Table 3 in the attached PDF**.\n2. Comparison: While both *SPTNet* and *FlipClass* target Open-World semi-supervised learning, *SPTNet* focuses on **information extraction at the pixel level**. \nIn contrast, *FlipClass* emphasizes **matching supervision signals at the representation level**.\n\n&nbsp;&nbsp;\n\n**Response to W3**\n> ***Differences between distribution alignment [2,3] and representation alignment.***\n\nThank you for this insightful comment.\\\nDistribution alignment in [2,3] utilizes KL divergence to regularize the predicted probabilities to be **close to a prior probability distribution $P$ to better learn new classes**.\n\nIn comparison, our representation alignment **does not require this prior $P$ of classes**. \nAdditionally, our approach focuses on aligning the learned representations between teacher and student, **improving consistency without relying on a predefined class distribution**. \nThis makes our method more adaptable and effective in diverse scenarios.\n\n---\n\\\n**Response to Q1**\n> ***Does the representation alignment require prior knowledge?***\n\nNo, it does not. **Unlike [2,3], our method inherits from the Teacher-Student framework, where the teacher provides higher quality supervision signals, \nforming a weak prior to guide the student's learning (i.e., $p(student∣teacher)$)**. \n\nSince this weak prior can have flaws and may not reflect the true distribution accurately, *FlipClass* further formulates \n$p(teacher∣student)$ to enable iterative mutual learning between the teacher and student. \nThis approach helps avoid incorrect guidance stemming from the teacher's overconfidence in its weak prior.\n\n&nbsp;&nbsp;\n\n**Response to Q2**\n> ***Can distributional alignment [2,3] be combined with FlipClass's representation alignment?***\n\nInteresting question! Yes, our representation alignment can be incorporated with other distribution alignment strategies.\n\nThe distribution alignment can be applied by adjusting the parametric classification loss defined in Sec 4.2 L198-203, seamlessly collaborating with our representation alignment in Sec 4.1.\\\nWe conducted experiments on *Stanford Cars*, by combining these distribution alignment strategies [2,3,4].\n**The results are as provided in Table 4 in the attached PDF**.\n\nThe reasons why directly combining these distribution alignment strategies does not bring improvement are:\n1. ***ReMixMatch*** [4]: \\\nReMixMatch maintains a running average of the model's predictions on unlabeled data, $\\tilde{p}(y)$, which scales the model prediction $q$ by the ratio $p(y)/ \\tilde{p}(y)$, forming a valid probability distribution: \n​$\\tilde{q} =\\text{Normalize}(q×p(y)/ \\tilde{p}(y))$. \\\nHowever, during the early training stage **in Open-World scenarios, the model's predictions are inaccurate on new classes, prone to making incorrect predictions of prior distributions \n($\\tilde{p}(y)$) on unlabeled data**. \n2. ***Prior Distribution Alignment*** [2,3]: \\\nThis approach regularizes the model with maximum entropy regularization using an even-distributed prior distribution. \nThis works on *Cifar* and *ImageNet*, which have balanced class distributions. \\\nHowever, **the *SSB Bench*** (*e.x.*, *Stanford Cars*) **has unbalanced class distributions (shown in Fig. 15 in Appendix C.3)**. \nWithout prior knowledge of the class distribution (especially on new classes), applying this naive distribution alignment degrades performance on both new and old classes.\n\nFurther exploration of distribution alignment on unbalanced datasets is an interesting direction, and we'll work on this to combine it with representation alignment.\n\n&nbsp;&nbsp;\n\n**Response to Q3**\n> ***Performance under long-tail distribution conditions [5]?***\n\nThank you for the suggestion to validate our method under long-tail distribution conditions. \\\nYes, *FlipClass*'s representation alignment is indeed **distribution-agnostic**.\\\n**We conducted experiments following the setting of [5]. The results are shown in Table 5 in the attached PDF.**\n\nWhile *FlipClass* does not surpass BaCon [5], it still performs well.\\\nUnlike non-parametric methods (*GCD, OpenCon, BaCon*), *FlipClass* and *SimGCD* reduce clustering time costs. \nAlthough non-parametric methods are often more robust, *FlipClass*'s attention alignment strategy demonstrates effective, distribution-agnostic performance.\n\n&nbsp;&nbsp;\n\n**We've included above explanation and extended experiments in the revised manuscript. Once again, thanks for your constructive suggestions.**\n\n&nbsp;&nbsp;\n\n*[1] SPTNet: An efficient alternative framework for generalized category discovery with spatial prompt tuning. ICLR 2024.*\n\n*[2] Open-World Semi-Supervised Learning. ICLR 2022.*\n\n*[3] Robust Semi-Supervised Learning when Not All Classes have Labels. NeurIPS 2022.*\n\n*[4] Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. ICLR. 2020.*\n\n*[5] Towards Distribution-Agnostic Generalized Category Discovery. NeurIPS 2023.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eUSZ46lBzn", "reviewer": "Reviewer_Btrz", "summary": "This paper targets the task of generalized class discovery (GCD), and argues that the existing teacher-student learning framework suffers from three challenges: 1) learning gap between old and new classes, 2) feature discrepancies between augmented images, and 3) attention inconsistency between teacher and student. These challenges, originating from inadequate supervision on new classes and the gap between weakly and strongly augmented data, largely hinder the performance of existing methods. In light of this, this paper proposes both empirical and theoretical analysis of the aforementioned issues, and introduces a novel method that can synchronize the learning progress of teacher and student models, which largely improves the GCD performance.", "questions": "Please see above.", "limitations": "The limitations are adequately discussed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well organized and clearly written. The motivation is also strong, which is focused on largely-ignored perspectives in GCD literature.\n1. The paper delivers useful empirical and theoretical insights, associated with an effective plug-and-play solution.\n1. The experimental results showcase the superiority of the proposed method, which largely surpasses the existing SOTA methods.", "weaknesses": "1. Given the strong performance shown in Tables 1 and 2, I'm curious about what cost it takes to achieve that, since the modification is simply a new attention operation that offers the ability to sync the learning paces of teacher and student models. In this regard, I'd like to know the potential extra computation overhead (e.g., theoretical/empirical analysis) compared with the conventional self-attention block.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thorough review. We appreciate your attention to detail regarding the potential extra computation overhead compared with conventional self-attention block.\n\n**Response to Q1**\n> ***Potential extra computation overhead (e.g., theoretical/empirical analysis) compared with the conventional self-attention block.***\n \nHere is a detailed comparison of the computation overhead:\n\n**1.Overall Training Computation Cost**:\nConsidering $L$ layers, the computation costs for a Vision Transformer (ViT) model are as follows:\n\n1. **Self-Attention Cost:** $O(B \\cdot N^2 \\cdot d)$\n2. **Feed-Forward Network (FFN) Cost:** $O(B \\cdot N \\cdot d^2)$\n\n**2.Attention Update Computation Cost**:\n\nOur attention update strategy is applied to only 2-3 layers (the choice of layers are explained in L261-265 and Appendix B.5), and each updating cost is the same as the FFN cost per layer:\n$$ O(B \\cdot N \\cdot d^2) $$\n\n**3.Comparison of Overheads:**\n\nDuring the forward pass, we **add only 2-3 extra FFN-like operations**. \\\nThis updating does not affect the backward pass, as we only update the last block and no extra parameters are introduced to optimize.\n\n**4.Empirical Analysis:**\n\nThe additional overhead is minimal, with only a slight increase in training time (table below), demonstrating that the computational cost is negligible compared to the performance improvements achieved by *FlipClass*.\n\n| Setting                         | CUB  | SCars |\n| ------------------------------- | ---- | ----- |\n| w/o Attention Updating          | 65s  | 93s   |\n| w Attention Updating (2 layers) | 71s  | 101s  |\n| w Attention Updating (3 layers) | 76s  | 109s  |\n*Time Cost (s) of One Forward Pass on CUB and Stanford Cars.*\n\nIn summary, **the attention update operation introduces a small extra overhead equivalent to 2-3 extra FFN operations, which is insignificant compared to the overall training cost**.\n\n---\n\\\n**We have promptly integrated this analysis into our manuscript to enhance the clarity. Once again, we extend our sincere appreciation for your insightful feedback.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## **General Response to All Reviewers**

We sincerely thank all reviewers for the time they spent reviewing our manuscript and for their thoughtful feedback. We appreciate that the reviewers found our paper theoretically and methodologically novel, with strengths such as:
- the **idea** of dynamic teacher-student attention alignment strategy to be **innovative and interesting** (***Reviewers MqmB, MRfK, P1mR***); 
- our proposed method to be **well-organized, clearly written**, and providing **considerable contributions** to the field (***Reviewers Btrz, P1mR***); 
- and overall our **analysis to be comprehensive**, with **detailed experimental validation and theoretical insights** (***Reviewers MqmB, MRfK, Btrz***).

The attached PDF includes (1) **the analysis of hyperparameter $\alpha$**, (2) **zoom-in cluster visualization** (***Reviewer MRfK***), (3) **extended comparison with SPTNet**,  (4) **combinations with distribution alignment strategies**, and (5) **results on long-tailed distribution conditions** (***Reviewer MRfK***). 

We have provided detailed responses to individual reviewers below, and have included additional experiments suggested by the reviewers in the Author Response PDF.

We are also pleased to publicly release all code.

Please let us know if you have any additional questions or concerns. We are happy to provide clarification.

Meta-review: All reviewers agree that the presented work is innovative, of broad interest to the NeurIPS community, and well-evaluated experimentally (although some questions had to be cleared up during the discussion phase). Given the clear consensus on the quality of this paper, acceptance is warranted. Authors are advised to include the clarifications that emerged during the discussion phase in their camera-ready version.
Decision: Accept (oral)