Flipped Classroom: Aligning Teacher Attention with Student in Generalized Category Discovery

Authors: haonan  lin
wenbin  an
jiahao  wang
yan  chen
feng  tian
mengmeng  wang
guang  dai
qianying  wang
jingdong  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-24 05:00:28

Abstract: Recent advancements have shown promise in applying traditional Semi-Supervised Learning strategies to the task of Generalized Category Discovery (GCD). Typically, this involves a teacher-student framework in which the teacher imparts knowledge to the student to classify categories, even in the absence of explicit labels. Nevertheless, GCD presents unique challenges, particularly the absence of priors for new classes, which can lead to the teacher's misguidance and unsynchronized learning with the student, culminating in suboptimal outcomes. In our work, we delve into why traditional teacher-student designs falter in generalized category discovery as compared to their success in closed-world semi-supervised learning. We identify inconsistent pattern learning as the crux of this issue and introduce FlipClass—a method that dynamically updates the teacher to align with the student's attention, instead of maintaining a static teacher reference. Our teacher-attention-update strategy refines the teacher's focus based on student feedback, promoting consistent pattern recognition and synchronized learning across old and new classes. Extensive experiments on a spectrum of benchmarks affirm that FlipClass significantly surpasses contemporary GCD methods, establishing new standards for the field.

Main paper: 
 introduction: 
 Teacher-Student architecture has proved its effectiveness in Semi-Supervised Learning (SSL) [66, 32, 53] , which aims to take advantage of a large collection of unlabeled data, reducing the expensive costs of annotation [57, 90, 89] . Previous approaches tend to model 𝑝(student|teacher), where the teacher typically acts as a fixed point of reference, providing a form of "supervision prior" to guide the student [25, 42, 60] . This supervision comes from labeled data and is asymmetrical: while the teacher has robust prior knowledge and provides a stable learning signal, the student's knowledge is incomplete and evolving. The student learns from both the teacher's supervision and the data it is exposed to, trying to emulate the teacher by aligning its predictions with those of the teacher.Figure 1 : Left: Learning effects of traditional Teacher-Student Consistency Model (TSCM, e.g., SimGCD [75] ) and our Flipped Classroom Consistency Model (FlipClass) on Stanford Cars [33] . Middle: Model comparison between TSCM and our FlipClass, where D new refers to data belonging to new classes. Right: Illustration of the inner feedback mechanism in FlipClass, where teacher attention is adapted to the student, leading to the alignment of attention.Teacher-Student designs traditionally rely on a closed-world assumption, where it is expected that the teachers have supervision prior to all classes they will face while instructing students [61, 3] . However, real-world applications often involve dynamic and open environments, where instances belonging to new classes may appear [84, 30, 31, 41] . In such cases, discovering novelties could enable models to adapt new information and evolve continually as biological systems do [14, 49] . Recently, Generalized Category Discovery (GCD) [69] stands out by challenging models to categorize unlabeled data containing both old and new classes using only partial labels for training. Although recent GCD methods have adapted closed-world Teacher-Student strategies with notable success [75, 50] , the transition is not seamless and presents several challenges.Challenge I: Learning gap. Fig. 1 top left illustrates the learning evolution of student and teacher. The previous teacher-student models result in unsynchronized learning and a significant learning gap in new classes. The ideal learning dynamic between the teacher and student should be cohesive, which requires teaching students in accordance with their aptitude.Challenge II: Discrepancies in features. The learning gap arises from the teacher's fast pace, leading to large discrepancies in representations between weakly-augmented data (teacher) and strongly-augmented data (student), especially for new classes (Fig. 1 middle ). This causes significant prediction differences, making consistency loss optimization difficult and hindering effective student learning. Over time, the iterative learning process exacerbates this misalignment.Challenge III: Attention inconsistency. Inadequate supervision for new classes leads to inaccurate instructions from the teacher, causing the student to focus on different parts than the teacher (Fig. 1 right). Imagine a classroom transitioning to a new subject. Without proper guidance, students' attention diverges from the teacher's, resulting in confusion and ineffective learning.To sum up, the challenges of previous teacher-student models arise from inadequate supervision on new classes and the gap between weakly and strongly augmented data. This results in attention inconsistency (Chall. III), which leads to discrepancies in predictions and representations (Chall. II), ultimately causing a significant performance gap (Chall. I). Addressing these challenges requires developing teacher-student dynamics that align the evolving knowledge of both teacher and student (Fig. 1 , bottom left). These findings lay the foundation for our approach, FlipClass, which models the teacher's posterior 𝑝(teacher|student) from the energy perspective of attention, building an adaptive teacher to bridge the learning gap. FlipClass offers a plug-and-play solution to foster an interactive learning environment where the student can influence the teacher's guidance in real-time, allowing teachers to tailor their instructions based on students' current attention [7, 1] . By aligning attention, FlipClass ensures that the learning pace of the teacher and student is in sync, leading to improvement on both old and new classes. Our contributions are summarized as: where "True Old" refers to predicting an 'Old' class sample to another 'Old' class, while 'False Old" indicates predicting an 'Old' class sample as some 'New' class.(1) Empirical analysis: We highlight the challenge of applying the closed-world Teacher-Student paradigm to the open-world scenario of GCD. Our in-depth analysis identifies attention misalignment between teacher and student as the key issue hindering synchronized learning between them.(2) Methodology: Based on these analyses, we propose a flexible and effective method, FlipClass, which enables teachers to adapt and respond to student feedback to synchronize their learning progress, thereby leading to an overall improvement in teaching outcomes.(3) Superiority: FlipClass consistently outperforms state-of-the-art generalized category discovery methods on both coarse-grained and fine-grained datasets. 
background: 
 We first provide some background on semi-supervised learning and generalized category discovery to better contextualize our analysis. Let us consider a data set D consisting of labeled data D 𝐿 = {(x ℓ 𝑖 , 𝑦 𝑖 )} 𝑁 𝐿 𝑖=1 from |C 𝐾 | old classes and unlabeled data D 𝑈 = {x 𝑢 𝑗 } 𝑁 𝑈 𝑗=1 that may contain instances from both old classes C 𝐾 and new classes C 𝑁 , with C = C 𝐾 ∪ C 𝑁 . For a data instance x, let 𝑝 m (𝑦 | 𝑓 𝜽 (x)) denote the predicted class distribution produced by the model 𝑓 .In semi-supervised learning (SSL), the goal is to enhance model performance by leveraging unlabeled data, traditionally drawn from the same class spectrum as the labeled data [90] . The goal of SSL can be formalized by integrating three fundamental techniques: (1) Consistency regularization ensures that the model outputs consistent predictions for augmented versions of the same instance. This technique utilizes different transformations to test the robustness of the model's predictions, promoting stability across variations in the input data [8, 59, 36] . (2) Pseudo-labeling utilizes the model to generate artificial labels for unlabeled data by adopting "hard" labels (that is, the argmax of the model output) and keeping only the labels where the highest class probability exceeds a predefined threshold [62, 47, 57, 37, 23] . (3) Teacher-Student model incorporates a structured learning relationship where the teacher model, typically trained on weakly-augmented instances, generates high-quality pseudo labels. These pseudo labels are then used to guide the training of the student model, which processes strongly-augmented instances. This approach helps improve the generalization capabilities of the student model by learning from the refined knowledge and stable supervision signals provided by the teacher [66, 43, 77, 12, 53] . Several methods integrate some of these techniques and achieve advanced performance in SSL [63, 83, 78, 87] . We unify these SSL techniques into one consistency loss:EQUATIONwhere cross-entropy 𝐻 (•, •) measures consistency for regularization, while the prediction 𝑝 m (𝑦 | 𝑓 𝜽 (x)) serves as a pseudo label. This setup captures a teacher-student dynamic, where 𝛼(x) and A (x) represent the teacher (weakly-augmented) and student (strongly-augmented) instances, respectively.The GCD task pushes the boundaries of SSL by questioning the closed-world assumption that all classes in the unlabeled dataset D 𝑈 are previously known [69] . Instead, GCD incorporates new classes C 𝑁 into the unlabeled dataset, demanding that the model learn to recognize and then correctly classify them [4, 21, 80, 5] . In this open-world setting, SSL methods face obstacles with new classes due to the lack of supervision [24, 56] , resulting in significantly lower quality of pseudo-labels for these new classes than for the old ones (Fig. 2 left ). This gap exacerbates the complexity of optimizing the consistency loss Eq. 1 for new classes, leading to learning instability and slow convergence (Fig. 2 , middle). Such optimization issues lead to severe prediction bias, resulting in new classes' performance lagging behind that of old classes (Fig. 2 , right), underlining the limitations of existing SSL techniques in GCD scenarios. More empirical analysis can be found in Appendix B.2. 
how consistency loss goes awry: unraveling the pitfalls: 
 Acknowledging challenges presented by the absence of prior knowledge for new classes in traditional semi-supervised learning (SSL) methods is the first step toward addressing the complexities of openworld tasks. We further identify that the 'prior gap' manifests as issues in learning synchronization and representation discrepancy (Sec. 3.1). Our analysis targets the minimization of the energy function between teacher and student representations to bridge the 'prior gap'. We find that aligning their attention on similar patterns reduces energy, indicating effective alignment and learning (Sec. 3.2). This key understanding paves the way for the development of our proposed methods, aiming to synchronize teacher-student attentions for improved model learning dynamics (Sec. 4).The challenge of optimizing consistency loss leads to a learning gap between the student and the teacher, particularly evident when dealing with new classes (Fig. 1 top left). This gap causes the student to plateau, as it cannot keep pace with the teacher's more advanced understanding, which in turn restricts the teacher's progress in new classes Moreover, this learning gap also manifests itself in the divergent representations between teacher and student (Fig. 1 middle), specifically for new classes. Based on these observations, we revisit consistency loss (Eq. 1) in the closed-world setting. Insight 3.1. The large discrepancy between 𝑓 𝜽 (𝛼(𝑥)) and 𝑓 𝜽 (A (𝑥)) complicates maintaining consistency across the model predictions. To narrow this divide, an intuitive idea is to align 𝑓 𝜽 (𝛼(x)) more closely with 𝑓 𝜽 (A (x)), simplifying the optimization of L cons :EQUATIONwhere Δℜ aims to pull 𝑓 𝜽 (𝛼(x)) closer to 𝑓 𝜽 (A (x)). Ideally, Δℜ would be adaptive, scaling with the discrepancy between 𝑓 𝜽 (A (𝑥)) and 𝑓 𝜽 (𝛼(𝑥)), while avoiding make them too similar, which enables model to find a shortcut of L cons .To design it, we delve into the vision transformer, a representation encoder that has significantly advanced the performance of the GCD task. We found that the self-attention mechanism excels at capturing critical image patterns: as depicted in Fig. 3 left, deeper features (after the 8th layer) reveal semantic, high-level commonalities (e.g., car shell) across all images; and the shallow features are more attuned to high-frequency, low-level details (e.g., color and texture).Inconsistent patterns can disrupt learning, making it crucial to align stored and queried patterns effectively. To address this, we draw inspiration from the Hopfield Network [2] -an associative memory model known for its energy-based mechanism that naturally pulls similar patterns together EQUATIONMinimizing 𝐸 (𝝃; X) resembles retrieving stored pattern x 𝑖 that is most similar to the query 𝝃, and log-sum-exp (lse) function is parameterized by 𝛽 > 0 and 𝑐 is a preset constant. Particularly, the first term ensures the finiteness of the query, while the second term measures the alignment of the query with each stored pattern. The update rule for a state pattern 𝝃 is equivalent to a gradient descent update of minimizing the energy 𝐸 with step size 𝜂 = 1 [54] , ensures that the query moves closer to the most similar stored pattern:EQUATIONMoreover, the energy function is closely related to the Transformer's self-attention mechanism [67] (Appendix A.1.2). By extending the energy model from self-attention to cross-attention, we model the dynamics between student and teacher learning patterns. Taking the student representations f 𝑠 = 𝑓 (A (x)) as examples, we have Q 𝑠 = f 𝑠 W 𝑄 and K 𝑠 = f 𝑠 W 𝐾 . By applying Eq. 3 to key and query matrices, we set energy functions to track the teacher-student relationship:EQUATIONEQUATIONwhere 𝐸 (Q 𝑠 ; K 𝑡 ) indicates the alignment in learning patterns of the student and teacher; k 𝑡 ,𝑖 denotes the 𝑖-th row vector of K 𝑡 and 𝛼 ≥ 0. Intuitively, lse(Q 𝑠 k 𝑇 𝑡 ,𝑖 , 𝛽) captures the smooth maximum alignment between student queries q 𝑠,𝑖 and teacher keys k 𝑡 ,𝑖 . Specifically, it nudges each teacher key k 𝑡 , 𝑗 towards a semantic alignment with its most corresponding student query q 𝑠, 𝑗 . The regularization term diag(K t K t 𝑇 ) acts as a constraint on the energy levels of teacher's representation k 𝑡 ,𝑖 , guarding against any disproportionate increase during the maximization of lse(Q s k 𝑇 𝑡 ,𝑖 , 𝛽). This ensures that no individual teacher representation becomes too closely mirrored in the student's representation, maintaining a diverse learning trajectory. Insight 3.2. When applying closed-world consistency regularization to the GCD task, it becomes difficult to gradually reduce the energy 𝐸 (Q 𝑠 ; K 𝑡 ) as training progresses (Fig. 3 right ). The sustained high energy demonstrated a flaw in the previous methods: teachers and students focused on identifying patterns that were inconsistent, leading to divergent learning paths. Specifically, when teachers and students focus on similar patterns (e.g., taillights), energy is reduced, indicating better prediction consistency and effective learning. In contrast, when their attention is distracted, the energy rises, leading to severe inconsistencies in predictions and making the optimization of L cons more difficult.4 FlipClass: Teacher-Student Attention Alignment 
teacher attention update rule: 
 Based on Insights 3.1 and 3.2, our objective is to minimize the energy function 𝐸 (Q 𝑠 ; K 𝑡 ) between teacher and student representations, thereby easing the optimization of L cons . Theorem 4.1. The minimization can be formulated as obtaining a maximum a posteriori probability (MAP) estimate of teacher keys K 𝑡 given a set of observed student queries Q 𝑠 :EQUATIONwhere 𝑝(Q 𝑠 |K 𝑡 ) and 𝑝(K 𝑡 ) are modeled by energy functions Eq. 5a and 5b, respectively. We approximate the posterior inference by the gradient of the log posterior, estimated as:EQUATIONwhere sm(v) := exp (vlse(v, 1)) and D (•) is a vector-to-diagonal-matrix operator. Incorporating Eq. 4, the update rule of teacher keys K 𝑡 is derived as follows:EQUATION)where 𝛼, 𝛾 update and 𝛾 reg are hyper-parameters. For a proof, refer to Appendix A.2. The teacher-attention update rule in Theorem 4.1 minimizes an implicit energy function determined by student queries and teacher keys. It serves as using the student queries to search for the most similar teacher patterns in the stored set. As illustrated in Fig. 4 , the update rule adjusts the teacher's attention in the direction of student attention, facilitating the retrieval of related patterns and improving semantic alignment. This design establishes a bidirectional information flow: the teacher not only imparts advanced knowledge to the student, but also adjusts guidance based on the student's learning effects, achieving a more cohesive learning dynamic.Contrastive learning plus consistency regularization under the parametric paradigm has been demonstrated effective in GCD task [75] . Formally, given two views (random augmentations x 𝑖 and x ′ 𝑖 ) of the same image in a mini-batch B, the supervised and self-supervised contrastive loss is written as:L 𝑠 rep = 1 |B ′ | ∑︁ 𝑖 ∈B ′ 1 |N 𝑖 | ∑︁ 𝑞∈N 𝑖 -log exp(z 𝑇 𝑖 𝑧 ′ 𝑞 /𝜏 𝑐 ) 𝑖 ′ ≠𝑖 exp(z 𝑇 𝑖 𝑧 ′ 𝑖 ′ /𝜏 𝑐 ) , L 𝑢 rep = 1 |B| ∑︁ 𝑖 ∈B -log exp(z 𝑇 𝑖 z ′ 𝑖 /𝜏 𝑢 ) 𝑖 ′ ≠𝑖 exp(z 𝑇 𝑖 𝑧 ′ 𝑖 ′ /𝜏 𝑢 ), The consistency regularization objectives (Eq. 1) are then simply cross-entropy loss ℓ(𝑞 ′ , 𝑝) = -𝑘 𝑞 ′ (𝑘) log 𝑝(𝑘) between the predictions and pseudo-labels or ground-truth labels:whereL cons = 1 |B| 𝑖 ∈B ℓ(𝑞 ′ 𝑖 , 𝑝 𝑖 ) -𝜀𝐻 ( p) for unlabeled, 1 |B ′ | 𝑖 ∈B ′ ℓ(𝑦 𝑖 , 𝑝 𝑖 )for labeled.The one-hot labels 𝑦 𝑖 correspond to x 𝑖 , and the soft pseudo-label 𝑞 ′ 𝑖 is produced by the teacher instance 𝛼(x) 𝑖 . Moreover, a mean-entropy regularizer [6] , 𝐻 ( p) = -𝑘 p(𝑘) log p(𝑘), is included to encourage diverse predictions. The combined classification loss, L cons , balances unsupervised and supervised terms with a parameter 𝜆. And the overall training objective is L rep + L cons . 
experimental settings: 
 Datasets. We evaluate the effectiveness of FlipClass on three generic image recognition datasets (i.e., CIFAR-10/100 [34] and ImageNet-100 [20] ), three fine-grained datasets [68] (i.e., CUB [71] , Stanford Cars [33] , and FGVC-Aircraft [46] ) contained in Semantic Shift Benchmark (SSB) [68] , and the challenging datasets Herbarium-19 [65] , ImageNet-1k [20] . For each dataset, we first subsample |C 𝑙 | seen (labeled) classes from all classes. Following GCD [69] , we subsample 80% samples in CIFAR-100 and 50% samples in all other datasets from the seen classes to construct D 𝑙 , while the remaining images are treated as D 𝑢 (refer to Table 9 ).Evaluation Protocols. The performance was evaluated by measuring accuracy between the model's cluster assignments and ground-truth labels on the test set, with three aspects: all instances (All), instances from old categories (Old), and instances from new categories (New). The number of categories in the unlabeled dataset (|C 𝑢 |) is often unknown. Following previous studies [64, 85] , we set 𝐾 (cluster number) equal to |C 𝑢 |, as approximate cluster estimation is usually feasible in the real world. The estimation of the number of categories in unlabeled datasets can be found in the Appendix C.4. Further implementation details can be found in Appendix D.1. We compare SOTA methods with ours in GCD using features from both DINO [15] and DINOv2 [51] . Our approach shows significant performance improvement, particularly in the recognition of 'New' classes across both the SSB fine-grained benchmark (Tab. 1) and generic image recognition datasets (Tab. 2), consistently surpassing existing SOTA methods. Moreover, in fine-grained image classification (Tab. 1), recognizing subtle differences between closely related categories is crucial, which is in contrast to coarse-grained datasets where the visual differences between classes are more obvious. In fine-grained settings, the risk of the model generating incorrect pseudo labels is higher, which makes consistency regularization counterproductive (Sec. 2.2). However, the results across these datasets demonstrate our model's capability to effectively adapt consistency regularization strategies from closed-world settings to more complex open-world scenarios. Moreover, the balanced accuracy between new and old classes on the CUB dataset (Tab. 1), also observed with methods like PCAL, CiPR, and AdaptGCD, can be attributed to the dataset's small size (6,000 images) and large class split (200). This limited data reduces the likelihood of overfitting to old classes, promoting more uniform performance across both categories. Additional results on Herbarium 19 and ImageNet-1k are detailed in the Appendix C.2.Ablation Studies. Our ablation study, shown in Fig. 5 , underscores the significance of our design choices. First, we replace the student's augmentations with the teacher's, i.e., using only weak augmentations, the performance on 'New' classes significantly declines (2nd set of bars). This underscores the importance of strong augmentations for the student, which are essential to bolster generalizability. Then we validate the importance of attention alignment in Eq. 8 (3rd set of bars), we see performance drop across both 'Old' and 'New' classes, affirming that our attention alignment strategy is crucial for maintaining a consistent learning pace between the teacher and student, leading to sustained performance gains. Finally, the 4th set of bars verifies the role of regularization during the attention update, which integrates the prior energy of the teacher, preventing any single student pattern from overly influencing the teacher's attention.Enhanced Consistency through Attention Alignment. To validate our assumption on modeling ℜ in Eq. 2, we showcase how distribution-based strategies (distribution alignment [11] , FixMatch [63] ) and our representation-based method, attention alignment, achieve consistency on new classes from CUB dataset (Fig. 6 ). Our method stands out by significantly reducing the discrepancy between the representations of teacher (weakly-augmented) and student (strongly-augmented) data. This representation-based alignment leads to a more consistent learning process and is evidenced by the superior accuracies we achieve for both 'Old' and 'New' classes.Design Choice of Attention Alignment. We experimented with various techniques to model ℜ in Eq. 2, including scheduled data augmentation (SDA), increasing similarity between Q 𝑠 and K 𝑡 via ℓ 2 norm, Kullback-Leibler divergence (KLD) or CORrelation ALignment (CORAL) loss (see details in Appendix D.1). As shown in Fig. 7a , our teacher-attention update strategy outperforms these alternatives on both CUB and SCars datasets.FlipClass mitigates prediction bias. We verify the effectiveness and robustness of FlipClass, by diagnosing the model's classification errors under four different (𝛾 update ) as defined in Eq. 8. As depicted in Fig. 7b , both "False New" and "False Old" errors are consistently mitigated-where 'Old' class samples are mistakenly labeled as 'New' and vice-versa. Moreover, as illustrated in Fig. 8b bottom, FlipClass outperforms leading methods [55] by moving closer to the true class distribution, yielding higher and less biased accuracies across all classes.Does the improved energy dynamic make for performance gains? Fig. 8a top shows that aligning attention in these deeper (9-10) layers yields the highest performance on SCars dataset. And Fig. 8a bottom displays a greater reduction of energy 𝐸 (Q 𝑠 ; K 𝑡 ) in deeper layers. Moreover, this trend highlights that attention alignment constantly maintains lower energy levels than without, indicating improved alignment of student patterns with teacher updating.How does FlipClass change the representations? Fig. 8b showcases representation enhancements with FlipClass against the leading method, InfoSieve [55] , using the same t-SNE and PCA components to ensure consistent projection space and scale. FlipClass forms clusters with higher compactness and purity, demonstrating enhanced feature discrimination and less inter-class confusion.The zoom-in comparison on CUB dataset is provided in Fig. 9 , which showcases that FlipClass achieves more distinct and well-separated clusters. Further, we assess prediction bias and class- A.1 Preliminaries: Hopfield Network Energy FunctionWe first provide the formulation of Hopfield network energy function, and present its convergence, which build the fundamental of our one-step Teacher-Attention Update Strategy. Ramsauer et al.proposed a new energy function that is a modification of the energy of modern Hopfield networks [19] , and a new update rule which can be proven to converge to stationary points of the energy.Given 𝑁 stored (key) patterns x 𝑖 ∈ R 𝑑 represented by the matrix X = (x 1 , . . . , x 𝑁 ) with the state (query) pattern 𝝃 ∈ R 𝑑 , the energy function 𝐸 of the modern Hopfield networks can be expressed:𝐸 = exp(lse(1, X 𝑇 𝝃)),where lse(𝛽, 𝑥) = 𝛽 -1 log 𝑁 𝑖=1 exp(𝛽x 𝑖 ) is the log-sum-exp function (lse) for 0 < 𝛽. Ramsauer et al. then proposed to take the logarithm of the negative energy of modern Hopfield networks and add a quadratic term of the current state to ensure that the norm of the state vector 𝝃 remains finite and the energy is bounded, reads:EQUATION)Using 𝑝 = softmax(𝛽X 𝑇 𝝃), the update rule is defined as:EQUATIONwhich is the Concave-Convex Procedure (CCCP) for minimizing the energy 𝐸 and can be proven as converging globally.Theorem A.1 (Global Convergence (Zangwill): Energy). The update rule Eq. 10 converges globally: For 𝝃 𝑡+1 = 𝑓 (𝝃 𝑡 ), the energy 𝐸 (𝝃 𝑡 ) → 𝐸 (𝝃 * ) for 𝑡 → ∞ and a fixed point 𝝃 * .Proof. The Concave-Convex Procedure (CCCP) [81] minimizes a function that is the sum of a concave function and a convex function. And since 𝑙𝑠𝑒 is a convex, -lse a concave function. Therefore, the energy function 𝐸 (𝝃) is the sum of the convex function 𝐸 1 (𝝃) = 1 2 𝝃 𝑇 𝝃 + 𝐶 1 and the concave function 𝐸 2 (𝝃) = -lse:EQUATION𝐸 1 (𝝃) = 1 2 𝝃 𝑇 𝝃 + 𝛽 -1 ln 𝑁 + 1 2 𝑀 2 = 1 2 𝝃 𝑇 𝝃 + 𝐶 1 ,𝐸 2 (𝝃) = -lse(𝛽, X 𝑇 𝝃), where 𝐶 1 does not depend on 𝝃.The Concave-Convex Procedure (CCCP) applied to 𝐸 isEQUATIONwhich results in the update rule:EQUATIONwhere 𝑝 𝑡 = softmax(𝛽X 𝑇 𝝃 𝑡 ). This is the update rule in Eq. 10. □The Hopfield network update rule is the attention mechanism used in transformer. Assume 𝑁 stored (key) patterns x 𝑖 and 𝑆 state (query) patterns r 𝑖 with dimension 𝑑 𝑘 , we can have X = (x 1 , . . . , x 𝑁 ) 𝑇 and R = (r 1 , . . . , r 𝑆 ) 𝑇 combine the x 𝑖 and r 𝑖 as row vectors. Define the key as k 𝑖 = W 𝑇 𝐾 x 𝑖 , q 𝑖 = W 𝑇 𝑄 r 𝑖 , and multiply the result of the update rule (Eq. 10) with W 𝑉 . By defining the matricesK = (XW 𝐾 ) 𝑇 , Q = (RW 𝑄 ) 𝑇 , and V = XW 𝐾 W 𝑉 = K 𝑇 W 𝑉 , where W 𝐾 ∈ R 𝑑 𝑥 ×𝑑 𝑘 , W 𝑄 ∈ R 𝑑 𝑟 ×𝑑 𝑘 , W 𝑉 ∈ R 𝑑 𝑘 ×𝑑 𝑜 . If 𝛽 = 1/ √𝑑 𝑘 and softmax ∈ R 𝑁 is changed to a row vector, there is:EQUATIONwhere the left part is the transformer attention, while the right part is the update rule Eq. 4 multiplied by W 𝑉 .We first provide several lemmas for the derivation of the teacher-attention update rule (Eq. 8). Lemma A.2. For a given column vector x ∈ R 𝑁 , we have:EQUATIONProof. Consider 𝑆 = 𝑁 𝑖=1 exp(𝛽x 𝑖 ) and 𝑓 (x) = log 𝑆, 𝜕 𝑓 (x) 𝜕x can be computed as:𝜕 𝑓 (x) 𝜕x 𝑗 = 𝜕 𝜕x 𝑗 log 𝑆 = 1 𝑆 • 𝜕𝑆 𝜕x 𝑗The partial derivative 𝜕𝑆 𝜕x 𝑗 is:𝜕𝑆 𝜕x 𝑗 = 𝜕 𝜕x 𝑗 𝑁 ∑︁ 𝑖=1 exp(𝛽x 𝑖 ) = 𝑁 ∑︁ 𝑖=1 𝜕 𝜕x 𝑗 exp(𝛽x 𝑖 ) = 𝑁 ∑︁ 𝑖=1 𝛽 exp(𝛽x 𝑖 )𝛿 𝑖 𝑗 = 𝛽 exp(𝛽x 𝑗 )where 𝛿 𝑖 𝑗 is the Kronecker delta.𝜕x 𝑗 back into the expression for 𝜕 𝑓 (x) 𝜕x 𝑗 :𝜕 𝑓 (x) 𝜕x 𝑗 = 1 𝑆 • 𝛽 exp(𝛽x 𝑗 ) = 𝛽 • exp(𝛽x 𝑗 ) 𝑆 Recognize that exp(𝛽x 𝑗 ) 𝑆is the 𝑗-th component of the softmax function applied to 𝛽x:softmax(𝛽x) 𝑗 = exp(𝛽x 𝑗 ) 𝑁 𝑖=1 exp(𝛽x 𝑖 ) = exp(𝛽x 𝑗 )𝑆 Therefore, we have:𝜕 log 𝑁 𝑖=1 exp(𝛽x 𝑖 ) 𝜕x 𝑗 = 𝛽 • softmax(𝛽x) 𝑗Putting it back into vector notation:𝜕 log 𝑁 𝑖=1 exp(𝛽x 𝑖 ) 𝜕x = 𝛽 • softmax(𝛽x) Since 𝜕 log exp(𝛽x) 𝜕x = 𝛽 -1 𝜕 log 𝑁 𝑖=1 exp(𝛽x 𝑖 ) 𝜕x, we can have:𝜕 log exp(𝛽x 𝑖 ) 𝜕x = softmax(𝛽x)This confirms the lemma. □ Lemma A.3. Let k 𝑖 denote the 𝑖-th row vector of K ∈ R 𝑁 ×𝑑 . Then, we have:EQUATIONwhere e 𝑁 𝑖 represents an 𝑁-dimensional column vector where only the 𝑖-th entry is 1, with all other entries set to zero.Proof. First, note that the expression k 𝑖 k 𝑇 𝑖 can be equivalently rewritten as e 𝑁 𝑖 k 𝑖 k 𝑇 𝑖 (e 𝑁 𝑖 ) 𝑇 . To find the derivative of k 𝑖 k 𝑇 𝑖 with respect to K, we need to consider the individual elements of K.Let K = [k 𝑇 1 ; k 𝑇 2 ; • • • ; k 𝑇 𝑁 ]. Therefore, k 𝑇 𝑖 is the 𝑖-th row of K, and we denote this as k 𝑇 𝑖 = K 𝑖,: . The differential of k 𝑖 k 𝑇 𝑖 is given by: 𝑑 (k 𝑖 k 𝑇 𝑖 ) = 𝑑 (K 𝑇 𝑖,: K 𝑖,: ) = 𝑑 (K 𝑇EQUATIONSubstitute z = Qk 𝑇 𝑖 back to the expression, we have:∇ K lse(Qk 𝑇 𝑖 , 𝛽) = ∇ z lse(z, 𝛽) • 𝜕z 𝜕K With 𝜕𝑧 𝑗𝜕k 𝑖 = Q 𝑗,: and Eq. 18, we have:EQUATIONwhere diag(A) denotes the trace of A.Proof. Let us construct a column vector x whose 𝑖-th element is given by x 𝑖 := k 𝑖 k 𝑇 𝑖 /2. Then, using Lemmas A.2 and A.3 and the chain rule, we have:𝜕diag(KK 𝑇 ) 𝜕K = 𝜕 log 𝑁 𝑖=1 exp 1 2 k 𝑖 k 𝑇 𝑖 𝜕K = ∑︁ 𝑖 𝜕x 𝑖 𝜕K 𝜕lse(x, 1) 𝜕x 𝑖 = ∑︁ 𝑖 e 𝑁 𝑖 (e 𝑁 𝑖 ) 𝑇 K[softmax(𝑥)] 𝑖 Because of Lemma A.2 and A.3 = ∑︁ 𝑖 e 𝑁 𝑖 k 𝑖 [softmax(x)] 𝑖 = 2KThis proves the lemma. □ Derivation of Eq. 7. Now we proof that we can approximate the posterior inference of 𝑝(K 𝑡 | Q 𝑠 ) by the gradient of the log posterior, estimated as:∇ K 𝑡 log 𝑝(K 𝑡 |Q 𝑠 ) = -∇ K 𝑡 𝐸 (Q 𝑠 ; K 𝑡 ) + ∇ K 𝑡 𝐸 (K 𝑡 ) = sm 𝛽Q s K t 𝑇 Q 𝑠 -𝛼I + D sm 1 2 diag(K t K t 𝑇 ) K 𝑡 ,where sm(v) = softmax(v) := exp (vlse(v, 1)) and D (•) is a vector-to-diagonal-matrix operator. Moreover, recall Eq. 5a and 5b, the energy 𝐸 (Q 𝑠 ; K 𝑡 ) and 𝐸 (K 𝑡 ) are denoted as:𝐸 (Q 𝑠 ; K 𝑡 ) = 𝛼 2 diag(K t K t 𝑇 ) - 𝑁 ∑︁ 𝑖=1 lse(Q 𝑠 k 𝑇 𝑡 ,𝑖 , 𝛽) + 𝑐, 𝐸 (K 𝑡 ) = lse 1 2 diag(K t K t 𝑇 ), 1 = log 𝑁 ∑︁ 𝑖=1 exp 1 2 k 𝑡 ,𝑖 k 𝑇 𝑡 ,𝑖 + 𝑐,Proof. Using Lemmas A.4 and A.5, we have:∇ K 𝑡 𝐸 (Q 𝑠 ; K 𝑡 ) = 𝛼K 𝑡 -sm(𝛽Q 𝑠 K 𝑇 𝑡 ) • Q Since K 𝑡 can be expressed as lse( 1 2 k 𝑖 k 𝑇 𝑖 ), incorporating Lemma A.5, we have:∇ K 𝑡 𝐸 (K 𝑡 ) = D sm 1 2 diag(K t K t 𝑇 ) K 𝑡Put them together, we have:∇ K 𝑡 log 𝑝(K 𝑡 |Q 𝑠 ) = sm 𝛽Q s K t 𝑇 Q 𝑠 -𝛼I + D sm 1 2 diag(K t K t 𝑇 ) K 𝑡 ,This matches Eq. 7. □In this section, we begin by examining the representation discrepancy between old and new classes, highlighting alignment issues (B.1). Enhanced consistency loss optimization is then detailed, showing improvements in learning stability (B.2). We discuss how attention alignment bridges the prior gap and benefits synchronized learning, enhancing overall performance (B.3). The focus then shifts to attention specialization in deep network layers (B.4), and the performance impact of layer selection for attention alignment on different dataset (B.5). Finally, we represent the negligible impact on computational cost of the Attention Alignment strategy (B.6), thereby proving its practical viability.In GCD, directly applying consistency regularization leads to challenges, especially for new classes. Due to the lack of prior knowledge, the teacher struggles to guide the student, resulting in representation discrepancies between the teacher (weakly-augmented) and student (strongly-augmented). This is evident in Fig. 10 (left), where new classes show poor alignment between teacher and student representations compared to known classes.This discrepancy causes unsynchronized learning, as shown in Fig. 10 (right). The two main phenomena observed are the learning gap and learning regression. The learning gap indicates that the student struggles to reach the teacher's level of understanding, particularly for new classes, leading to stagnation. Learning regression affects the teacher, hampering improvement for new classes and causing regression in known classes due to the alignment efforts with the student.Building on the previous discussion on the representation discrepancy of old and new classes, we address the challenges in optimizing consistency loss (Sec. 2.2) that contribute to the learning gap. To verify how FlipClass improves this process, we track the consistency loss L cons on new classes. As shown in Fig. 11a , FlipClass with various update rates (𝛾 update ) demonstrates faster and more stable convergence compared to the SimGCD baseline. Specifically, the experiments on SCars and CUB datasets reveal that FlipClass streamlines the optimization process of consistency loss, leading to more rapid and stable convergence.The enhanced consistency loss optimization further aids in achieving consistency between the student and the teacher, reflecting in two main aspects. Firstly, it mitigates the effects of prior gap, as shown in Fig. 11b . Compared to the categorize errors of SimGCD (Fig. 2 ), FlipClass reduces the 'False New' error (where the model incorrectly predicts new classes as old classes). This indicates that FlipClass can mitigate overfitting on old classes due to the lack of prior knowledge about new classes. Secondly, as shown in Fig. 12 , compared to the traditional teacher-student model used in generalized category discovery (e.g., SimGCD), which suffers from a learning gap, FlipClass successfully bridges the learning gap. It achieves better teacher-student learning effects, ensuring more synchronized and stable learning outcomes. Moreover, on the SCars dataset, the model shows a greater energy reduction in deeper layers, suggesting a focus on local semantics over general ones. This is illustrated in Fig. 13 , where deeper layers emphasize local semantic parts, and earlier ones are associated with general semantics (e.g., texture, color). This local semantic concentration enhances transferability across 'Old' and 'New' classes, with attention alignment in deeper layers (9-10) yielding the highest performance. We observe that different heads attend to disjoint regions of the image, focusing on important parts. After training with our method, attention heads become more specialized to semantic parts, displaying more concentrated and local attention. Our model learns to specialize attention heads (shown as columns) to different semantically meaningful parts, improving transferability between labeled and unlabeled categories.Furthermore, we provide the performance of layer selection for attention alignment on CUB and Cifar-10. In this context, "2 layers" means performing attention alignment in the two layers depicted in the Fig. 14 . As shown, Cifar-10 (coarse-grained) tends to achieve higher performance when alignment is performed in the middle layers (4-5), while CUB (fine-grained) achieves higher performance with alignment in deeper layers, which aligns with the trend of SCars (Fig. 13 ). The difference stems from the dataset nature. CIFAR-10 benefits from middle-layer alignment, capturing general features like shapes and textures, which suffice for its simpler categories. Conversely, CUB requires deeper layer alignment for detailed features needed to distinguish similar bird species. Deeper layers provide the refined features critical for complex tasks.A potential concern is whether attention alignment increases the time cost. Since we perform the alignment in a one-loop manner, the additional time overhead is negligible. This is demonstrated in Table 3 , showing that the training and inference times for FlipClass are comparable to those of SimGCD. In sum, we add 2-3 extra FFN-like operations during the forward pass. And there is no impact on the backward pass, as only the last block is updated without introducing extra parameters for optimization. Therefore, the additional overhead is minimal, with only a slight increase in training time, as shown in Table 4 . We detail our experimental results, including main outcomes with error bars for statistical significance (C.1). We evaluate performance on complex datasets (C.2), analyze clustering and per-class prediction distributions (C.3). Additionally, we examine robustness to varying numbers of old and new classes (C.4), and investigate how different proportions of old classes affect performance (C.5), demonstrating our method's stability and adaptability.Table 5 reports error bars to provide a clear understanding of the statistical significance and variability of the main results in our experiments. Specifically, we include both the mean and standard deviation (std) values for the performance of our FlipClass across different datasets and class types (All, Old, New). The standard deviations are calculated from five independent runs with random seeds, offering insight into the consistency of our method's performance.Here, we discuss the performance of various methods on challenging datasets, specifically Herbar-ium19 [65] and ImageNet-1K [35] . Herbarium19, a long-tailed dataset, poses significant challenges due to the varying frequencies of different categories, leading to unbalanced cluster sizes. Table 6 demonstrates the robustness of our proposed FlipClass, in handling such frequency imbalances and its ability to accurately distinguish categories even with few examples. Table 7 showcases the performance on ImageNet-1K, a large-scale generic classification dataset, in evaluating the model's capability in real-world applications of generalized category discovery. The results demonstrate the robustness of FlipClass for tasks involving both familiar and unfamiliar data in complex, real-world scenarios.Clustering Analysis. Fig. 15 presents a visual comparison of the clustering results obtained with FlipClass against the existing state-of-the-art method, InfoSieve [55] , on Cifar-10 and Cifar-100 datasets. On Cifar-10, FlipClass forms clusters that exhibit higher compactness and purity, indicating enhanced feature discrimination and reduced interclass confusion. In contrast, on Cifar-100, although InfoSieve forms visually more compact clusters, these clusters show less purity, with a higher incidence of false class predictions.Prediction Distribution and Class-specific Accuracies. Fig. 16 evaluates the prediction distribution and class-specific accuracies of FlipClass compared to InfoSieve [55] . FlipClass demonstrates a better fit to the true distribution, whereas InfoSieve shows skewed predictions. Moreover, FlipClass significantly outperforms InfoSieve in recognizing tail classes on the CUB and Stanford Cars datasets, improving accuracy and reducing prediction bias.Varying Number of Classes during Clustering. In the main experiments (Section 5), the class number, 𝐾, is assumed as a known prior following prior works [69, 64, 75, 70] , however, this setting has been questioned as impractical [9, 74, 73] . In Fig. 17 , we conduct experiments when this assumption is removed, evaluating results with different numbers of classes, where the ratio changes from 80% to 200% compared to the ground truth number of classes. During clustering (e.g., KMeans [45] ), a predefined class number lower than the ground truth significantly limits the ability to discover new classes, causing the model to focus more on old classes. Conversely, increasing the class number results in less harm to the generic image recognition datasets (e.g., Cifar-100) and can even be beneficial for some fine-grained, long-tailed datasets (e.g., CUB). This phenomenon occurs because overestimating the number of classes allows the model to maintain higher flexibility and adaptability in recognizing new classes in these fine-grained, challenging datasets. For fine-grained, class-distribution biased datasets like CUB, overestimating the class number helps capture subtle differences between closely related categories, thereby improving class separation and reducing prediction bias. However, for generic datasets like Cifar-100, the visual differences between classes are more pronounced, and overestimating the class number can introduce unnecessary complexity, leading to overfitting and decreased performance.Estimation of Number of Classes. Additionally, to further validate the robustness of our model, we trained FlipClass using an estimated number of classes in the dataset, where the number of classes is predicted using the over-clustering method from GCD [69] . We obtained a similar predicted number of classes as SimGCD. As expected, our method performs worse on Cifar-100 when using an estimated number of classes due to the mismatch between the estimated and actual class distribution. Interestingly, the performance of SimGCD (traditional teacher-student model) improves on CUB with both new and old classes, while our FlipClass makes improvements on old classes but sees a decrease in new classes.In the primary experiments, we fix the number of the old classes |C ℓ | (details in Appendix D.3).Here, we experiment with our method by changing the class split setting. Specifically, on Cifar-100 (|C ℓ | = 80) and CUB (|C ℓ | = 100) datasets, we test with fewer old classes, as shown in Fig. 18 . For Cifar-100 and CUB, as the number of old classes decreases, the accuracy for both old and new classes slightly declines but remains stable. This demonstrates FlipClass's effectiveness in leveraging additional old class information and robustness in handling varying numbers of known classes. We develop our FlipClass upon the SimGCD [75] baseline on the pre-trained ViT-B/16 DINO2 [15] . Specifically, we take the final feature corresponding to the CLS token from the backbone as the image feature, which has a dimension of 768. For the feature extractor F, we only fine-tune the last block. We set the balancing factor 𝜆 to 0.35 and the temperature values 𝜏 𝑢 and 𝜏 𝑐 to 0.07 and 1.0, respectively, following SimGCD. For the temperature values 𝜏 𝑡 and 𝜏 𝑠 in the classification losses, we also set them to 0.07 and 0.1. For update rule (Eq. 8), we set 𝛼 = 0, 𝛽 = 1, 𝛾 update = 0.1 and 𝛾 reg = 0.5. All experiments are conducted using a single NVIDIA A100 GPU with 200 epochs, which we find sufficient for the losses to plateau.In this experimental setup, we design both weak and strong augmentations for the teacher and student networks. For weak augmentation, we use common techniques such as RandomHorizontalFlip and RandomCrop for all datasets, aiming to pass less perturbed versions of the input images to the teacher network. For the strong augmentation that is applied to the images fed to the student, we incorporate more aggressive transformations to expose the student to a wider range of variations. Specifically, we add RandomResizedCrop with a scale range of 0.3 to 1.0, which allows for more aggressive cropping and resizing. Additionally, we include Gaussian blurring to simulate different levels of image blurriness. For datasets that are used for generic recognition tasks, we further enhance the strong augmentation by including ColorJitter with probability 0.8 and RandomGrayscale with probability 0.2. Solarization inverts pixel values above a threshold, simulating the effect of solarizing an image, while Grayscale converts the image to black and white, reducing color information. These additional augmentations help expose the student network to even more diverse image variations, improving its robustness and generalization capabilities. We follow the dataset settings from earlier works [75, 70] to subsample the training dataset. Specifically, 50% of known categories and all samples of unknown categories are used for training. For all datasets except Cifar-100, 50% of the categories are considered known during training, whereas for Cifar-100, 80% of the categories are known during training. Detailed statistics are displayed in Table 9 . Below is a summary of the datasets and how their combination supports experiments in generalized category discovery:Cifar-10/1003 [34] are coarse-grained datasets consisting of general categories with low-resolution images and even class distribution.ImageNet-100/1K4 is a subset of 100/1K categories from the coarse-grained ImageNet5 [35, 58] dataset. It includes a large scale of high-resolution real-world images with evenly distributed classes.CUB (Caltech-UCSD Birds-200-2011) 6 [71] is widely used for fine-grained image recognition, containing different bird species distinguished by subtle details.Stanford Cars7 [33] is a fine-grained dataset of various car brands, providing multi-view objects for class detection and scene understanding, challenging real-world applications in distinguishing subtle appearance differences.FGVC-Aircraft (Fine-Grained Visual Classification of Aircraft)8 [46] is a fine-grained dataset organized in a three-level hierarchy. At the finer level, differences between models are subtle but visually measurable. Unlike animals, aircraft are rigid and less deformable, presenting variations in purpose, size, designation, structure, historical style, and branding.Herbarium 19 (FGVC 2019 Herbarium Challenge) 9 [65] provides a curated dataset of over 46,000 herbarium specimens across 680 species, presenting a long-tailed distribution and challenges for species recognition.In the ablation studies (Section 5.3), we apply different strategies such as distribution alignment [11] , FixMatch [63] and CORAL. These strategies are employed to encourage the consistency between the teacher and student, therefore modeling ℜ in Eq. 2. We briefly provide the main idea of these alignment strategies below.Distribution alignment is designed by maintaining a running average of the model's predictions on unlabeled data p(𝑦). Given the model's prediction 𝑞 = 𝑝 model (𝑦 | x 𝑢 ) on an unlabeled example x 𝑢 , 𝑞 is scaled by the ratio 𝑝(𝑦)/ p(𝑦) and then renormalize the result to form a valid probability distribution: q = Normalize(𝑞 × 𝑝(𝑦)/ p(𝑦)).FixMatch is a semi-supervised learning method that combines consistency regularization and pseudolabeling. It works by first generating pseudo-labels for unlabeled images using the model's predictions on weakly enhanced versions of those images, retaining only high-confidence predictions as following:ℓ 𝑢 = 1 𝜇𝐵 𝜇𝐵 ∑︁ 𝑏=1 1 (max (𝑞 𝑏 ) ≥ 𝜏) H ( q𝑏 , 𝑝 m (𝑦 | A (𝑢 𝑏 ))) ,where 𝜏 is a scalar hyperparameter denoting the threshold above which a pseudo-label should be retained. Then, the model is trained to predict these pseudo-labels using strongly enhanced versions of the same images. The loss function consists of two cross-entropy terms: a supervised loss for labeled data and an unsupervised loss for unlabeled data, where the unsupervised loss utilizes the pseudo-labels calculated from weakly enhanced images and the model's predictions on strongly enhanced images.CORAL (CORelation ALignment) aligns the second-order statistics (covariances) of two spaces. Specifically, CORAL minimizes the difference in covariance matrices between the student and teacher representations (S and T). The goal of CORAL is to find a transformation for S that minimizes the Frobenius norm of the difference between the covariance matrices of S and T, denoting C 𝑆 and C 𝑇 , respectively. CORAL minimizes the following objective:min 𝑆 ′ ∥C 𝑆 ′ -C 𝑇 ∥ 2 𝐹 ,where C 𝑆 ′ is the covariance matrix of the transformed student representations S ′ , and ∥.∥ 𝐹 denotes the Frobenius norm.In semi-supervised learning (SSL), the goal is to enhance model performance by leveraging unlabeled data, traditionally drawn from the same class spectrum as the labeled data [90] . A key strategy in SSL, consistency regularization [36] , in recent years, centers on promoting model stability by ensuring that the teacher instance (weakly-augmented instance) and the student instance (strongly-augmented instance) yield coherent predictions [11, 63, 78, 87, 83] . Building on the Π-Model's teacher-student framework, several approaches have advanced its capabilities [66, 43, 77, 12, 53] . MeanTeacher [66] deploys an exponential moving average of the model parameters to stablize the teacher's output. NoisyStudent [77] employs a self-training strategy that incorporates noise into the student model's training, cycling the improved student back into the teacher role. Previous methods in SSL have largely concentrated on promoting the teacher's performance, often overlooking whether the student can keep pace, and neglecting the harmony of interaction. Our approach pivots to synchronizing the teacher's and student's attention, a shift that's especially pivotal in GCD, where consistency is challenged by the introduction of new classes. This strategy ensures a balanced teacher-student dynamic, crucial for effective consistency regularization in the open-world setting.Novel category discovery (NCD) is first formalized as cross-task transfer in [30] , which aims to discover unseen categories from unlabeled data that have nonoverlapped classes with the labeled ones. Earlier works [31, 26, 86, 79, 38] mostly maintain two networks for learning from labeled and unlabeled data respectively. AutoNovel [27] introduces a three-stage framework. Specifically, the model is firstly trained with the whole dataset in a self-supervised manner and then fine-tuned only with the fully-supervised labeled set to capture the semantic knowledge for the final joint-learning stage. UNO [22] addresses the problem by jointly modeling the labeled and unlabeled sets to prevent the model from overfitting to labeled categories. Similarly, NCL [88] generates pairwise pseudo labels for unlabeled data and mixes samples in the feature space to construct hard negative pairs.Generalized Category Discovery (GCD) extends NCD by categorizing unlabeled images from both seen and unseen categories [69] , which tackles this issue by tuning the representation of the pretrained ViT model with DINO ( [15] , [51] ) with contrastive learning, followed by semi-supervised k-means clustering. ORCA [13] considers the problem from a semi-supervised learning perspective and introduces an adaptive margin loss for better intra-class separability for both seen and unseen classes. CiPR [28] introduces a method for more effective contrastive learning and a hierarchical clustering method for GCD without requiring the category number in the unlabeled data to be known a priori. SimGCD [75] proposes a parametric method with entropy regularization to improve performance. TIDA [74] discovers multi-granularity semantic concepts and then leverages them to enhance representation learning and improve the quality of pseudo labels. Moreover, 𝜇GCD [70] take a leap forward by extending the MeanTeacher paradigm to the GCD task. Instead of managing dual models as in 𝜇GCD, our approach achieves teacher-student consistency more effectively within a single-model structure, streamlining computational demands. Crucially, we found that the learning discrepancy between teachers and students in the open-world context is the reason why consistency is difficult to achieve, and solved this problem by synchronizing the attention of teachers and students.Catastrophic Forgetting. While our method achieves significant improvement on new classes, the performance on old classes, particularly on CUB, lacks compared to the state-of-the-art methods. We attribute this to a phenomenon akin to catastrophic forgetting, where the model forgets previously learned concepts. Addressing these issues is essential for enhancing the robustness and effectiveness of the proposed methods.Sub-optimal 𝐾 Estimation. As shown in Appendix C.4, for fine-grained, class-distribution biased datasets like CUB, overestimating the class number helps capture subtle differences between closely related categories, thereby improving class separation and reducing prediction bias. However, for coarse-grained datasets like CIFAR-100, overestimating the class number can introduce unnecessary complexity, leading to overfitting and decreased performance. Some works have delved into this path and show promising performance [73, 74, 9] , highlighting the potential of tailored 𝐾 estimation strategies to balance complexity and performance across different types of datasets.Data Augmentation to Enhance Teacher-Student Consistency. Effective data augmentation techniques has been investigated a lot in semi-supervised learning [18, 82, 48, 76, 29] , the techniques for generalized category discovery are still lacking, which affects the consistency between the teacher and student models. The strength of data augmentation for new classes needs careful control to avoid ineffective learning due to excessive noise or insufficient variability. Additionally, preventing data leakage during augmentation is critical, as pretrained diffusion models can compromise evaluation integrity by leaking training data. Addressing these issues is essential for enhancing the robustness and effectiveness of the proposed methods.Our study extends the capability of AI systems from the closed world to the open world, fostering AI systems capable of categorizing and organizing open-world data automatically. While Generalized Category Discovery (GCD) has many real-world applications, it can be unreliable and must be applied with caution. Currently, supervised learning with extensive fine annotations is the mainstream solution for many computer vision tasks, but the cost and difficulty of obtaining these annotations can be prohibitive. Our work addresses this issue by advancing an open-set semi-supervised learning paradigm, significantly reducing the need for precise annotations and promoting the application of AI models in areas where annotations are difficult to obtain.This work provides a new idea for open-set semi-supervised learning. Specifically, while conventional approaches apply closed-world semi-supervised learning techniques to generalized category discovery, they rarely consider the attention alignment gap between teacher and student models. We point out that bridging this gap can significantly improve learning efficiency and accuracy. We hope that this methodology can be generalized to more relevant label-efficient tasks, promoting broader applications of AI in scenarios with limited labeled data.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: Refer to provided code in the supplementary materials.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: Appendix C.1 provides the statistical significance of the experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The original papers of assets, and the corresponding versions are provided in Section 5, References and Appendix D.1.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: We will well document the assets when we officially release our code.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not describe potential risks incurred by study participants.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.https://huggingface.co/facebook/dino-vitb16https://www.cs.toronto.edu/~kriz/cifar.htmlhttps://www.kaggle.com/c/imagenet-object-localization-challenge/overview/ descriptionhttps://www.image-net.org/download.phphttps://www.vision.caltech.edu/datasets/cub_200_2011/https://www.kaggle.com/datasets/jessicali9530/stanford-cars-datasethttps://www.robots.ox.ac.uk/~vgg/data/fgvc-aircraft/https://www.kaggle.com/c/herbarium-2019-fgvc6 
 
 Appendix:  


Decision: All reviewers agree that the presented work is innovative, of broad interest to the NeurIPS community, and well-evaluated experimentally (although some questions had to be cleared up during the discussion phase). Given the clear consensus on the quality of this paper, acceptance is warranted. Authors are advised to include the clarifications that emerged during the discussion phase in their camera-ready version.

Reviews: review: paper_summary: paper_summary: The paper introduces FlipClass, a dynamic teacher-student attention alignment strategy designed to address the challenges of Generalized Category Discovery (GCD) in open-world scenarios. Unlike traditional teacher-student frameworks, FlipClass updates the teacher’s attention to align with the student’s evolving focus, thereby promoting consistent pattern recognition and synchronized learning across both old and new classes. Extensive experiments validate FlipClass’s superiority over existing methods, establishing new benchmarks in GCD performance.
strength_weakness: strength_weakness: 1.	Experiments:
–	Certain details in the ablation study, like the impact of strong augmentations and regularization, could be more clearly explained.
–	The font size in Fig. 6 is too small and should be increased for better readability.
2.	Writing and Presentation:
–	Consistency in terminology and notation throughout the paper needs improvement to avoid confusion. For example, clearly distinguishing between different types of augmentations and regularization techniques used in the experiments.
questions: questions: 1.	While the paper claims that FlipClass significantly improves attention alignment (a design of $\R$ in Insight 3.1), could you provide more detailed quantitative metrics and comparative analyses with other attention alignment strategies to strengthen this claim?
2.	In Table 4, why does the accuracy of new classes increase significantly at the expense of the accuracy of old classes on the CUB dataset? Can you explain this phenomenon?
limitations: limitations: The necessary experiment of ablation study misses explanations for the strong augmentation and regularization.
paper_summary: paper_summary: This work proposes an attention alignment technique based on the Hopfield network energy function. Specifically, this work proposes to update the teacher model to increase the posterior teacher likelihood given the current student, which is modeled with the Hopfield network energy-based model. The teacher update process derived from the conditional score function is proven to be globally convergent. Experimental results showcase its considerable improvements over previous state-of-the-art baselines. The attention alignment technique is well-motivated with in-depth analysis. The contributions of this work include (1) investigating and discovering the attention alignment inconsistency between student and teacher models for the generalized category discovery problem, (2) proposing a theoretically inspired attention alignment method to address this issue, which is guaranteed to converge globally, and (3) achieving considerable performance gains compared with previous sota models.
strength_weakness: strength_weakness: weaknesses:

- Writing: It is highly recommended to put the related work in the main content rather than in the appendix.
- Ablations: There lacks the analysis on the hyperparameter \alpha, and the rationale behind setting \alpha=0 is to be explained.
- line 141: incorrect symbol
- Ablations: Better to provide comparisons in the ablations on varying class numbers.
questions: questions: questions:

- Baselines: Some baselines on generic and fine-grained datasets are not consistent in Tables 1 & 2, could you please tell us the reasons? It is better to make them consistent unless there is any special reason.
- References: Is there any related work on utilizing the attention alignment technique?
- Figures: Are the two methods visualized in figure 8b within the same projection space and with the same scale? Better provide zoom-in comparisons since there are two many classes.
limitations: limitations: Social impact: current methods are not applicable to real-world safety-demanding applications.
paper_summary: paper_summary: The paper introduces FlipClass, a novel method addressing the challenges of Generalized Category Discovery (GCD) in open-world scenarios. It identifies the misalignment of attention between teacher and student models as a key issue hindering effective learning, especially when new classes are introduced. To tackle this, FlipClass dynamically updates the teacher's attention based on feedback from the student, promoting synchronized learning and consistent pattern recognition. Extensive experiments demonstrate that FlipClass outperforms contemporary GCD methods.
strength_weakness: strength_weakness: 1. The content regarding "the Hopfield Network" and the underlying motivation could be enhanced，and the current presentation is not clear and optimal.

2. The SOTA method SPTNet [1] should also be compared in the paper. 

3. The differences between distribution alignment [2,3] and representation alignment in the paper need further discussion to make it easier for readers to understand your contributions.

[1] SPTNet: An efficient alternative framework for generalized category discovery with spatial prompt tuning. ICLR 2024.
[2] Open-World Semi-Supervised Learning. ICLR 2022.
[3] Robust Semi-Supervised Learning when Not All Classes have Labels. NeurIPS 2022.
questions: questions: 1. Does the alignment of attention in the paper require the introduction of some prior knowledge?

2. Do distributional alignment[1, 2] and this type of representation alignment conflict with each other? Have you conducted further experiments on combining them?

3. From the description in the paper, the representation alignment of FilpClass seems to be distribution-agnostic. Can its performance still be maintained under long-tail distribution conditions[3]?

[1] Open-World Semi-Supervised Learning. ICLR 2022.
[2] Robust Semi-Supervised Learning when Not All Classes have Labels. NeurIPS 2022.
[3] Towards Distribution-Agnostic Generalized Category Discovery. NeurIPS 2023.
limitations: limitations: NA
paper_summary: paper_summary: This paper targets the task of generalized class discovery (GCD), and argues that the existing teacher-student learning framework suffers from three challenges: 1) learning gap between old and new classes, 2) feature discrepancies between augmented images, and 3) attention inconsistency between teacher and student. These challenges, originating from inadequate supervision on new classes and the gap between weakly and strongly augmented data, largely hinder the performance of existing methods. In light of this, this paper proposes both empirical and theoretical analysis of the aforementioned issues, and introduces a novel method that can synchronize the learning progress of teacher and student models, which largely improves the GCD performance.
strength_weakness: strength_weakness: 1. Given the strong performance shown in Tables 1 and 2, I'm curious about what cost it takes to achieve that, since the modification is simply a new attention operation that offers the ability to sync the learning paces of teacher and student models. In this regard, I'd like to know the potential extra computation overhead (e.g., theoretical/empirical analysis) compared with the conventional self-attention block.
questions: questions: Please see above.
limitations: limitations: The limitations are adequately discussed.

Comments: title: Thanks for Your Further Response
Further Reply to Reviewer P1mR
comment: Dear Reviewer MRfK,

We greatly appreciate your helpful comments and your satisfaction with our responses! We will add these related works on attention alignment to our revised manuscript, and make comprehensive revisions based on the above important discussions and highlight them.

Thanks again for your valuable suggestions and comments. We really enjoy communicating with you and appreciate your efforts.
I appreciate the authors' great effort in providing further details. My primary concerns on related work and $\alpha$ has been addressed. I hope the author could add these related works on attention alignment to the final script. I would like to increase the score by one.
Dear Reviewer Btrz,

Thank you again for reviewing our manuscript. We have tried our best to address your questions (see our rebuttal above), and revised our paper by following suggestions from all reviewers.

Please kindly let us know if you have any follow-up questions or areas needing further clarification. Your insights are valuable to us, and we stand ready to provide any additional information that could be helpful.
Dear Reviewer MRfK,

Thank you again for your time and insightful comments! We have comprehensively revised our work according to your comments (please kindly refer to the rebuttal above). We hope we have addressed your concerns regarding the **analysis and explanation of the role of $\alpha$ in Eq. (8)**, **compared methods in Tables 1&2** and **related work on attention alignment**, *etc*. 

**Since the discussion is about to close, we would be grateful if you would kindly let us know of any other concerns and if we could further assist in clarifying any other issues.**

Thanks a lot again, and with sincerest best wishes.

Authors
Dear Reviewer MqmB,

Thank you again for reviewing our manuscript. We have tried our best to address your questions (see our rebuttal in the top-level comment and above), and revised our paper by following suggestions from all reviewers.

Please kindly let us know if you have any follow-up questions or areas needing further clarification. Your insights are valuable to us, and we stand ready to provide any additional information that could be helpful.
Thank you very much for your response!
Dear Reviewer P1mR,

We greatly appreciate your satisfaction with our responses, and very glad you increase the rating! We will make comprehensive revisions to our work based on your comments in order to further improve the quality of our work.

Thanks again for your valuable suggestions and comments. We enjoy communicating with you and appreciate your efforts!
The authors have addressed my concerns. Thanks AC and authors.
Thanks for your good responses! And you have addressed my concerns. I will increase my score to 7, this is a comprehensive paper.
Thank you to the authors for this response. Dear reviewer P1mR: Could you check whether the authors addressed your points? It would be particularly helpful to receive your update.
Thank you to the authors for this response. Dear reviewer MRfK: Could you check whether the authors addressed your points? It would be particularly helpful to receive your update.