Title: DAA: Amplifying Unknown Discrepancy for Test-Time Discovery
Venue: NeurIPS 2025
Accepted: yes
Keywords: Test-Time Discovery

Novel Category Discovery

Online Learning

Test-Time Training
Abstract: Test-Time Discovery (TTD) addresses the critical challenge of identifying and adapting to novel classes during inference while maintaining performance on known classes, which is a capability essential for dynamic real-world environments such as healthcare and autonomous driving. Recent TTD methods adopt training-free, memory-based strategies but rely on frozen models and static representations, resulting in poor generalization. In this paper, we propose a Discrepancy-Amplifying Adapter (DAA), a trainable module that enables real-time adaptation by amplifying feature-level discrepancies between known and unknown classes. During training, DAA is optimized using simulated unknowns and a novel warm-up strategy to enhance its discriminative capacity. To ensure continual adaptation at test time, we introduce a Short-Term Memory Renewal (STMR) mechanism, which maintains a queue-based memory for unknown classes and selectively refreshes prototypes using recent, reliable samples. DAA is further updated through self-supervised learning, promoting knowledge retention for known classes while improving discrimination of emerging categories. Extensive experiments show that our method maintains high adaptability and stability, and significantly improves novel class discovery performance. Our code will be available.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a Discrepancy-Amplifying Adapter (DAA) method to enhance the Test-Time Discovery task. Specifically, the DAA module amplifies feature discrepancies between known and unknown classes for test-time adaptation. Besides, a Short-Term Memory Renewal (STMR) mechanism is employed to ensures effective prototype management through adaptive memory updates. Extensive experiments demonstrate its superiority in adaptability and stability, improving novel class discovery performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.This work introduces a trainable DAA module in the training phase, which simulates unknown categories and enhance the model’s discriminative ability.\n\n2.This paper proposes a STMR mechanism in the testing phase, dynamically updating stored representations through a queue mechanism to support prototype updates, sample replay, and test-time fine-tuning of DAA. \n\n3.Comprehensive evaluation across multiple datasets show significant improvements over existing methods in both real-time and post evaluations.\n\nWeaknesses:\n\n1.The heuristic-based memory update strategy may not perform optimally in highly noisy environments, potentially limiting its effectiveness.\n\n2.The evaluation benchmarks are relatively small in scale, which may be insufficient to fully demonstrate the method's effectiveness.\n\nQUESTIONS:\n1.When simulating unknown classes, this method employs an interpolation of two different known classes as described Eq.1. My concern is that the simulated class still within the range of known distributions. Will the method remain effective when dealing with unknown classes that lie outside these distributions?\n\n2. Can the proposed method be extended to address continual learning scenarios?\n\n3.It is suggested to provide an overall algorithm for clearer illustration.\n\nLIMITATIONS:\nSee weakness.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the task of Test-Time Discovery (TTD) the problem of identifying and adapting to novel classes during inference while maintaining performance on known classes, a setting specifically relevant in dynamic environments like autonomous driving and healthcare. While prior approaches to TTD are typically training-free and rely on static model representations (e.g., via frozen backbones and hash-based memory updates), the authors propose a more adaptive and learnable alternative.\n\nTheir core contribution is the Discrepancy-Amplifying Adapter (DAA), a trainable module designed to amplify the feature discrepancies between known and unknown classes during both training and test time. DAA is preconditioned using a warm-up phase that simulates unknown class features via mixup and gaussian noise, and is trained with a dual loss—preserving known-class features via MSE while enforcing contrastiveness for unknowns.\n\nTo handle continual adaptation during test time, the authors introduce a Short-Term Memory Renewal (STMR) mechanism a queue-based dynamic memory system that manages prototypes of both known and discovered unknown classes. STMR periodically refreshes the memory with reliable, updated feature representations and supports DAA's self-supervised test-time adaptation. This setup avoids representational drift and enables the model to remain plastic to novel classes while stable on the known set.\n\nExtensive experiments on CIFAR100, CUB, and Tiny-ImageNet benchmarks demonstrate the proposed method outperforms both training-free and training-based baselines in real-time and post-hoc evaluation across clustering, classification, and forgetting metrics. Ablations further support the complementary roles of DAA and STMR. The method also maintains competitive knowledge retention while improving discriminative capability for new class discovery.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n### Quality\n\n- Well-Motivated and Technically Sound Solution: The paper addresses a genuinely hard and under-explored problem test-time discovery of novel classes in an online setting by integrating a trainable module (DAA) into a previously training-free setup.\n- Strong Experimental Validation: The authors perform thorough empirical evaluation across three benchmark datasets (CIFAR100D, CUB-200D, Tiny-ImageNetD), using appropriate metrics for both clustering quality and classification robustness (e.g., HCA, ARI, NMI, KA, KF). Both real-time and post-evaluation are considered, and the paper demonstrates consistent superiority over baselines including HM [16] and prompt-based continual learning methods (e.g., GMP, L2P).\n- Solid Ablation and Analysis: The ablation studies are clear and informative, isolating the effects of each component (DAA, STMR, test-time training). \n\n### Clarity\n- Well-Structured Presentation: The paper is clearly written and well-organized. The motivation for each component (DAA, STMR) is carefully laid out with visual illustrations (e.g., Fig. 1b, Fig. 2, Fig. 3). \n- Clear Definitions and Objectives: The formulation of the TTD task is precise, with explicit delineation from closely related paradigms like TTA and NCD. \n\n### Significance\n- Addressing a Critical Gap in TTD: The paper goes beyond the limitations of training-free test-time methods (which rely on frozen backbones and static prototypes) and introduces a practical, adaptive framework. This directly advances the state of the art in TTD, a task that is gaining traction for real-world applications in robotics and healthcare.\n- Potential for Broader Impact: The architectural idea of plug-in adapters for open-world adaptation could generalize to other settings such as continual learning.\n\n### Originality\n- Novel Adapter Design for TTD: While adapters and test-time training are known concepts, their use in the open-world discovery setting is novel.\n- STMR as a Lightweight Memory Mechanism: The Short-Term Memory Renewal (STMR) module is a notable contribution that balances model plasticity and stability, improving prototype quality without incurring the long-term errors seen in training-free memory-based approaches.\n\n## Weaknesses\n\n- Hyperparameter Sensitivity Unclear: The performance of DAA and STMR depends on hyperparameters (e.g., warm-up length, memory size, threshold γ, weighting factors λ₁ and λ₂). Although some sensitivity is analyzed (Fig. 6), a more comprehensive exploration would strengthen claims of robustness.\n-  Assumption of Class-Purity in Early Unknowns: The method assumes that initial unknown class predictions can seed meaningful prototypes. This assumption might not hold in highly noisy or fine-grained scenarios, leading to potential error accumulation in STMR. For example if the original dataset is contaminated with noisy data will this method be still competitive with methods based on hash lists and static representations.\n- Key implementation details are missing (e.g., architecture of DAA, training dynamics, runtime performance). Some are essential for reproducibility and could have been briefly summarized in the main paper.\n\nQUESTIONS:\n#### 1. **Clarify the DAA Adapter Architecture**\n- **What is the architecture used for the DAA adapter?**  \n  Please include a detailed architectural description (e.g., number of layers, type of transformations, parameter count) in the **main paper or appendix**, as this is critical for **reproducibility** and assessing overhead.\n\n#### 2. **Streamline the Contribution Statement**\n- In the **introduction**, consider consolidating the contributions into **two key components**:\n  - The **Discrepancy-Amplifying Adapter (DAA)**  \n  - The **Short-Term Memory Renewal (STMR)**  \n  These are the core innovations driving the improved performance. The SOTA results, while strong, are a consequence of these design choices rather than a standalone contribution.\n\n#### 3. **Clarify Robustness to Noise and Ambiguity**\nPlease explain how your method handles the following scenarios:\n- **Noisy or mislabeled known-class data** that may distort the feature space and lead to false discoveries or prototype pollution.\n- **Early unknown-class samples** that are **unrepresentative or ambiguous**, e.g., outliers, atypical images, or class boundary cases. How resilient is your method to such issues?\n\n#### 4. **Hyperparameter Sensitivity and Tuning Strategy**\nThe method introduces several important hyperparameters:\n- Similarity threshold **γ**\n- Memory size\n- Warm-up duration\n- Loss weights **λ₁** and **λ₂**\n- Mixup beta parameters to generate early unknowns\n\nHowever, there is minimal discussion of how these were selected or how sensitive the model is to their variation.  \nPlease include:\n- A brief **hyperparameter sensitivity analysis**, and\n- Whether the values are transferable across datasets or require tuning per setup.\n\n#### 5. **Justify Memory Design Choices**\nYour memory module is a core component—please expand on the rationale and trade-offs:\n- **Why was FIFO** selected over alternatives such as **exemplar-based sampling** or **confidence-based filtering**?  \n- **How frequently is memory renewal triggered**, and how was this interval chosen? Is it a fixed schedule or adaptive?\n\nLIMITATIONS:\nYes, the authors have acknowledged some limitations of their approach in the conclusion specifically the computational cost introduced by the warm-up and adaptation stages, and the heuristic nature of the memory update strategy (STMR), especially under noisy conditions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel approach to test-time discovery - the task of detecting and adapting to previously unseen categories during inference. This problem lies at the intersection of test-time training and novel category discovery. The core contribution is the introduction of the Discrepancy-Amplifying Adapter (DAA), which combines a trainable feature transformation with a short-term memory renewal strategy (STMR). The first component employs a warm-up strategy with self-supervised learning to regularize the feature embedding space, encouraging the clustering of samples within the same category. The second component incorporates a memory bank that stores representative samples for replay, effectively preserving knowledge of previously seen categories and mitigating catastrophic forgetting when performing test-time training.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:** The derivation from training-free in the TTD task appears reasonable, and the extensive evaluations on benchmark datasets provide some evidence supporting the effectiveness of the proposed approach.\n\n**Weaknesses:**  \n1. Overall, the paper requires significant revisions, as the current writing makes it difficult to follow and understand. As a result, it is hard to assess the technical contributions of this paper effectively.\nThe sim( ) operator on line 181 is undefined.\nIn Equation 2, what is $x_k$? \nThe captions for figures should be self-contained; for example, what are the units and meaning of the x and y axes?\nEquation 7 is confusing. Why is there a $L_{kn}$ term inside the definition of $L_{kn}$; similar for $L_{un}$? \nWhat does “real-time” eval and “post-eval” rigorously mean? There are some descriptions in the caption of Table 1, but they seem to be unclear.\n\n2. There are several concerns about the clarity of the proposed method that may affect the reproducibility of this work:\nThe choice of $\\gamma$ is non-trivial. How did the authors select this value? A sensitivity analysis for this parameter should be provided.\nIn Line 141, the loss function is described as a 'weighted combination'; however, Equation 3 does not include any coefficients, implying that the two loss terms are equally weighted.\nWhat is the architecture of DAA?\nWhat is the typical size of the memory queue?\n\n3. The objective of the warm-up stage requires further justification. In Equation 1, do $x$ and $x’$ come from two different categories? If they are randomly selected from one category, they should not create any unknown category. For Eq 2, why is MSE used, while the contrastive loss can still be used if we consider $DAA(r(x))$ and $r(x)$ is a “positive-pair” in the regular contrastive loss?\n\n4. For the  STMR, how to decide when the renewal step is performed. In line 161, “it will be triggered every several batches,” which seems to be unclear.\n\nQUESTIONS:\n1. I suggest that the authors address the questions in the strengths and weaknesses sections.\n2. Regarding checklist item 3, the authors state that 'We provided the assumptions and proof.' However, the submitted paper and  supplemental materials do not include any assumptions or theorems.\n\nLIMITATIONS:\nThe limitations of this work are briefly described at the end of the conclusion section. A more detailed discussion on potential strategies to address these challenges would strengthen the paper. The authors should also discuss other limitations, such as using the memory queue at test-time limits the real-world adoption, and the extra training might impose computational overhead compared to training-free approaches. It would be valuable to assess whether the observed accuracy gains justify the increased computational cost.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes DAA, a trainable module for Test-Time Discovery that amplifies feature discrepancies to detect novel classes while preserving known-class performance. Experiments show improved novel class discovery and stable performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The problem addressed in this work is well-motivated.\n2. The paper is solidly organized, and the use of illustrative figures effectively helps readers grasp the core challenges and the proposed solution.\n3. Although the proposed DAA strategy is relatively simple, it effectively addresses limitations of existing methods and achieves notable performance improvements.\n\nWeaknesses:\n1. According to the strict definitions of test-time adaptation/training, accessing source datasets during inference is typically disallowed. If the test-time discovery (TTD) setting is an extension built on these assumptions for novel class discovery, then the reliance on source-known samples in DAA’s Short-Term Memory raises concerns. The authors appear to directly store known samples from the source dataset during test time. I suggest conducting an ablation study where the memory pool is constructed solely from test-time samples, to evaluate whether such a change significantly degrades performance.\n2. The implementation details of DAA are insufficiently described. Does it explicitly incorporate any mechanism designed to preserve or enhance feature-level discrepancy for unknown samples in the feature space?\n3. Eq.(2) is unclear:\n (a) How are the class labels for synthesized unknown samples defined?\n (b) Are the two loss terms in Eq.(2) weighted equally, or is there a balancing factor?\n4. The memory update mechanism in Short-Term Memory is not clearly explained. Is the sample filtering policy purely based on FIFO, or are there additional criteria for selecting reliable samples?\n\nQUESTIONS:\nsee weaknesses and try to solve all of my concerns.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-11"}