Title: Partition-Then-Adapt: Combating Prediction Bias for Reliable Multi-Modal Test-Time Adaptation
Venue: NeurIPS 2025
Accepted: yes
Keywords: test-time adaptation

multi-modal domain shifts
Abstract: Existing test-time adaptation (TTA) methods primarily focus on scenarios involving domain shifts in a single modality. However, they often prove ineffective when multiple modalities simultaneously undergo domain shifts, as they struggle to identify and utilize reliable samples within testing batches amid severe prediction bias. To address this problem, we propose Partition-Then-Adapt (PTA), a novel approach combating prediction bias for TTA with multi-modal domain shifts. PTA comprises two key components: Partition and Debiased Reweighting (PDR) and multi-modal Attention-Guided Alignment (AGA). Specifically, PDR evaluates each sample’s predicted label frequency relative to the batch average, partitioning the batch into potential reliable and unreliable subsets. It then reweights each sample by jointly assessing its bias and confidence levels through a quantile-based approach. By applying weighted entropy loss, PTA simultaneously promotes learning from reliable subsets and discourages reliance on unreliable ones. Moreover, AGA regularizes PDR to focus on semantically meaningful multi-modal cues. Extensive experiments validate the effectiveness of PTA, surpassing state-of-the-art method by 6.1\% on Kinetics50-MC and 5.8\% on VGGSound-MC, respectively. Code of this paper is available at https://github.com/MPI-Lab/PTA.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the under-explored problem of multi-modal test-time adaptation (MM-TTA) when all modalities simultaneously undergo domain shift. To mitigate prediction bias under this setting, the authors propose Partition-Then-Adapt (PTA),  which contains two key components: Partition and  Debiased Re-weighting (PDR) and Attention-Guided Alignment (AGA) . Extensive experiment show that PTA achieves good performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This paper raises a meaningful question: how to design a test-time adaptation (TTA) scheme that mitigates domain shift when multiple modalities are involved. The authors observe that existing MM-TTA approaches focus only on single-modal domain shifts within multi-modal tasks; when all modalities are simultaneously shifted, performance degrades markedly. Hence, investigating how to develop an MM-TTA method for this more challenging setting is both timely and significant.\n\n2. The paper is well organised,  clear writing, and figures/tables support the narrative effectively.\n\n3. The authors benchmark PTA against MM-TTA methods, report clear SOTA gains, and conduct ablations on PDR, AGA, weighting strategies, and hyper-parameters, which help the reader understand each design choice.\n\n\nWeaknesses:\n1. Is “high class frequency ⇒ high bias” always valid? In PDR, samples whose predicted class appears more frequently than the batch mea  are deemed more biased. Yet class frequency can be high simply due to small batch size or genuine class imbalance in the target domain.  Please  provide results for PTA’s sensitivity to different batch sizes.\n\n2. Possible misconception in the MMD explanation (§3.3) \n    Minimising the squared MMD in Eq. (4) actually reducesthe distance between $A^+$ and $A^-$, i.e., makes the two distributions more similar.\n   However, the text claims the goal is to “push them apart”. Could the authors reconcile this apparent contradiction or correct the description?\n\n3. Definition of the confidence term $\\mathcal K$\n  Are they the maximum Softmax probabilities of the current classifier? Please add a precise definition.\n\nQUESTIONS:\nPlease refer to Weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors address the problem of test-time adaptation (TTA) under multi-modal domain shifts, specifically in the context of video and audio co-classification using pre-trained video and audio encoders. While prior TTA methods for multi-modal settings primarily focused on single-modality domain shifts, the authors propose a novel method, called Partition-Then-Adapt (PTA), to tackle domain shifts in both modalities simultaneously.\n\nPTA consists of two key components:\n- Partition and Debiased Reweighting (PDR), which identifies reliable and unreliable samples in a test batch by quantifying prediction bias—measured as the frequency of each predicted label relative to the batch average. Samples with lower bias are considered reliable, while those with higher bias are considered unreliable. A quantile-based reweighting strategy is applied to focused on reliable samples and remove effect of unreliable ones.\n- Attention-Guided Alignment (AGA), which aligns the attention maps of unreliable samples with those of reliable ones by minimizing the maximum mean discrepancy (MMD) between their attention distributions.\n\nExperiments on Kinetics50-C and VGGSound-C demonstrate the effectiveness of the proposed method, showing consistent improvements over existing TTA approaches.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The paper is well-structured and easy to follow.\n- The authors address the problem of multi-modal domain shifts, which has been underexplored in the literature.\n- The proposed method is well-motivated, and the Attention-Guided Alignment (AGA) component effectively mitigates the issue of uniform attention maps that can arise when PDR is used for adaptation.\n\n### Weaknesses\n- I believe that the method may require a large batch size to accurately estimate prediction bias and perform effective partitioning, which could limit its applicability in low-latency or memory-constrained environments.\n- The key assumption—that multi-modal domain shifts lead to performance drops even for high-confidence predictions (as shown in Figure 1(c))—is only validated using CAV-MAE. The generality of this observation across other model architectures is not validated.\n\nQUESTIONS:\n- Why does the SuMi method fail to adapt the pre-trained model stably across all benchmarks?\nAccording to Tables 6 and 7, SuMi not only performs inconsistently under multi-modal domain shifts but also fails to improve performance under single-modal shifts. An in-depth analysis of SuMi's failure modes would benefit readers.\n\n- Why do multi-modal domain shifts degrade model calibration more severely than single-modal shifts, as shown in Figure 1(c)?\nThis observation is central to the motivation of PTA but is not sufficiently explained. Additional insights into why the interaction between modalities causes such a pronounced miscalibration—possibly due to conflicting noise patterns or misaligned feature distributions—would strengthen the justification for the proposed method.\n\n- Why is the proposed method still effective under single-domain shifts?\nWhile PTA is designed for multi-modal shifts, the results suggest that it remains effective in single-modality scenarios. A deeper explanation is warranted: for instance, does prediction bias still emerge due to label imbalance or temporal inconsistencies even in single-domain cases? Clarifying this would demonstrate the generality of the framework.\n\n- Can the proposed method outperform existing single-modality TTA methods on standard benchmarks such as CIFAR-C or ImageNet-C?\nIf so, a comparison table on these datasets would help illustrate the broader applicability of PDR and AGA, even beyond multi-modal settings. However, due to time constraints, this is a minor suggestion.\n\n- Can the proposed method also be applied to other multi-modal classification models?\n\nLIMITATIONS:\nAuthors discuss the limitations in Appendix. However, I encouraged authors to fix the following points to improve the quality of the paper.\n\n1. I suggest the authors include the efficiency of the Source (i.e., the original pre-trained model without adaptation) in Figure 6. Since TENT involves only a few trainable parameters, I assume the Source would be slightly faster. However, readers may want a direct comparison of the efficiency (e.g., latency or parameter count) between the Source model and various TTA methods. This is a minor suggestion for improving clarity.\n\n2. Additionally, I recommend that the authors explicitly specify the unit for the number of trainable parameters shown in Figure 6. I assume it is in millions, but making this clear in the figure caption or axis label would avoid confusion. Moreover, in Tables 6 and 7, the authors only highlight (bold) their own method's results. To provide a fair comparison and improve readability, it would be better to bold the best values across all methods, not just the proposed one.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper highlights the limitations of using confidence or entropy-based measures for selective test-time adaptation, particularly in multi-modal domains. To address this, the authors propose a novel Partition then Adapt (PTA) strategy, which first clusters data before applying adaptation, aiming to improve robustness and reliability.

STRENGTHS AND WEAKNESSES:
**Strengths**
- The paper addresses an important and practical issue that can arise in multi-modal domains, where traditional confidence-based measures may fail.
- The proposed method leverages attention mechanisms effectively to improve adaptation.
- The ablation study is thorough and helps clarify the contribution of each component.
- Extensive experiments across multiple datasets support the efficacy of the proposed approach.

**Weaknesses and Questions**

- The proposed method, PTA, quantifies prediction bias based on prediction frequency. Specifically, according to Eq. (1), if a sample's predicted label has a higher frequency than the average frequency of predicted labels within the batch, the sample is considered a biased (i.e., unreliable) one. While the empirical results based on this partitioning are convincing, it would be helpful to include a higher-level explanation or intuition for why partitioning reliable and unreliable samples based on prediction frequency should work. Could the authors provide supporting evidence or cite prior work that inspired this approach?

QUESTIONS:
Please see weaknesses and questions section.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces PTA for test-time adaptation under multi-modal domain shifts. The method contains two components, Partition and Debiased Reweighting (PDR) and multi-modal Attention-Guided Alignment (AGA). PDR splits the test batch into reliable and unreliable subsets using a bias indicator, and then employs quantile-based strategy to jointly consider prediction bias and confidence to reweight samples. AGA adds an MMD-based regularization loss on attention maps.\n\nSTRENGTHS AND WEAKNESSES:\n## Strength\n\n- This paper is well written and follows a good structure.\n- Promising results are obtained in the studied benchmarks.\n- The effectiveness of the proposed two components is also shown.\n\n## Weakness\n\n- Some claims might not be convincing. For example, the authors claim “Fig. 1 illustrate the effectiveness of existing reweighting strategies diminishes under multi-modal domain shifts” in Line 116. However, Fig. 1 only evaluates the pretrained model, and dose not provide direct evidence or analysis for current methods (e.g., SuMi, READ) in multi-modal domain shifts. The logical connection about the motivation part is weak and requires more elaboration to be convincing.\n- The authors experimentally demonstrate the effectiveness of proposed sample reweighting strategy. However, the core idea of splitting samples into reliable/unreliable subsets based on confidence/bias indicators and give unreliable samples little or negative weights has been introduced by earlier methods (e.g. DeYO[1]). The authors are encouraged to conduct a deeper comparison and discuss more about how PDR differs from, or improves upon those prior reweighting strategies.\n- Some necessary citations might be missing and related work might not be sufficient. For instance, using weighted entropy loss in TTA (Line 152) has already been proposed in TTL[2].\n- Key experimental details are missing. (1) What does the “source model” denote? (2) There lacks essential details about CAV-MAE architecture, such as the number of attention layers used for fusion. (3) It is not clear how many attention layers are updated during adaption and what fraction of total parameters they represent. (4) In Table1, labeling the video+audio modality corruptions as Kinetics50-C may cause confusion with the single-modality Kinetics50-C benchmark. (5) It is not entirely clear what each number in Table 1 and 2 means? (e.g., average over six audio corruptions in Table 1?)\n- Baseline reproduction discrepancies. It's not clear why results of SOTA methods (e.g., SuMi, READ) differ from their original papers in Table 6 and Table 7.\n- Concerns about hyper-parameter fairness for baseline comparison. Figure 5 shows that learning rate and updated modules greatly influence TTA performance, yet PTA and baselines use different learning rates. It is encouraged to supplement comparisons with different learning rates and updating modules, which would make the comparisons more comprehensive.\n- Although the experiments demonstrate the method’s effectiveness, it remains unclear whether PTA can generalize to other single-modal and multimodal datasets (e.g, image-text). It is encouraged to conduct experiments to verify the method’s applicability to vision-language models (e.g., CLIP in TDA[5]) and image shift contexts (e.g., ImageNet-C in FOA[3],  CIFAR10-C in SoTTA[4]). Such experiments would validate PTA’s broader applicability.\n\n[1] Entropy is not enough for test-time adaptation: From the perspective of disentangled factors. In ICLR, 2024\n\n[2] Test-Time Low Rank Adaptation via Confidence Maximization for Zero-Shot Generalization of Vision-Language Models. In WACV, 2025\n\n[3] Test-Time Model Adaptation with Only Forward Passes. In ICML, 2024\n\n[4] Robust Test-Time Adaptation on Noisy Data Streams In NeurIPS, 2023\n\n[5] Efficient Test-Time Adaptation of Vision-Language Models. In CVPR, 2024\n\nQUESTIONS:\nSee Weakness\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the test-time adaptation problem in multi-modal scenarios where both single-modality and multi-modality domain shifts may exist. The authors propose an approach dubbed Partition-Then-Adapt (PTA) to address this new setting. PTA consists of two main components, i.e., Partition and Debiased Reweighting (PDR) and multi-modal Attention-Guided Alignment (AGA). PDR leverages the relative prediction frequency to split testing samples into reliable and unreliable subsets and adopts contrary adaptive weighting for the entropy minimization of the two sets. AGA can further regularize the adaptation of unreliable sets to focus on semantically meaningful multi-modal cues. Extensive empirical results demonstrate PTA consistently outperforms state-of-the-art TTA methods on multi-modal tasks.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The problem of test-time adaptation (TTA) is practical and significant. Moreover, according to the authors, the submission is the first to study multi-modal TTA with multiple domain shifts instead of single-modality shifts.\n\n- The proposed PTA method is novel, especially the AGA component, which specifically addresses the challenge of multi-modal adaptation with unreliable samples.\n\n- Experiments and ablation studies are very comprehensive. Extensive experiments demonstrate the superior performance of PTA over other TTA methods.\n\n## Weaknesses\n- The PDR component lacks solid support, such as a theoretical analysis of its reasonability.\n- The presentation lacks clarity. The introduction of PDR is confusing due to imprecise notation. The specific meanings of $Z$ and $K$ are unclear—for example, it’s not specified whether they refer to scalars or vectors, or to individual samples or mini-batches.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}