Title: Graph–Smoothed Bayesian Black-Box Shift Estimator and Its Information Geometry
Venue: NeurIPS 2025
Accepted: yes
Keywords: label shift

target shift

domain adaptation
Abstract: Label shift adaptation aims to recover target class priors when the labelled source distribution $P$ and the unlabelled target distribution $Q$ share $P(X \mid Y) = Q(X \mid Y)$ but $P(Y) \neq Q(Y)$. Classical black‑box shift estimators invert an empirical confusion matrix of a frozen classifier, producing a brittle point estimate that ignores sampling noise and similarity among classes. We present Graph‑Smoothed Bayesian BBSE (GS‑B$^3$SE), a fully probabilistic alternative that places Laplacian–Gaussian priors on both target log‑priors and confusion‑matrix columns, tying them together on a label‑similarity graph. The resulting posterior is tractable with HMC or a fast block Newton–CG scheme. We prove identifiability, $N^{-1/2}$ contraction, variance bounds that shrink with the graph’s algebraic connectivity, and robustness to Laplacian misspecification. We also reinterpret GS‑B$^3$SE through information geometry, showing that it generalizes existing shift estimators.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the ​​Graph-Smoothed Bayesian Black-Box Shift Estimator (GS-B$^3$SE)​​ for label shift adaptation. Traditional Black-Box Shift Estimation (BBSE) estimates target class priors by inverting a classifier’s confusion matrix but suffers from two limitations: 1) sensitivity to sampling noise​​: Error amplification with limited validation data; 2) ignored semantic relationships​​: Independent per-class estimation without statistical sharing.\n\nThe proposed GS-B$^3$SE is a fully probabilistic alternative that places Laplacian–Gaussian priors on both target log-priors and confusion-matrix columns, tying them together on a label-similarity graph. The key idea is to couple both the target log-prior vector and the columns of the confusion matrix through a Gaussian Markov random field defined on a label-similarity graph.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths​​:\n\n- Novel formulation​​: First unified Bayesian framework integrating graph regularization with confusion matrix uncertainty quantification.\n\n- ​​Theoretical rigor​​: Complete proofs for identifiability, optimal $N^{-1/2}$ convergence, and variance reduction scaling with algebraic connectivity $\\lambda_2(L)$.\n\n- Multidisciplinary insight​​: Information geometry analysis (Section 6) reveals how graph priors act as $e-$convex barriers, linking optimization to Riemannian dynamics.\n\nWeaknesses​​:\n\n- ​​Computational overhead​​: HMC requires 4 chains × 1.5k samples (NVIDIA T4) for K=100; wall-clock time for Newton-CG not benchmarked.\n\n- ​​Underutilized geometric insights​​: Natural gradient flow (Section 6.1) lacks empirical optimization comparisons versus EM/variational methods.\n\nQUESTIONS:\nI'm curious about the practical scenarios where this 'Label Shift' methodology would be most applicable. At its core, the paper addresses robust estimation of target class priors under distribution shift, formalized as a shift in P(Y) while assuming P(X∣Y) remains constant. This raises three key questions:\n\n- ​​Applicability Conditions​​: When is the label shift framework the appropriate modeling choice compared to other distribution shift paradigms? What diagnostic tests exist to validate its core assumption in real-world applications?\n\n- ​​Causal Implications​​: Does the formulation P(X∣Y)=constant implicitly assume Y→X causation? If so, how does this align with domains where features X may causally influence labels Y?\n\n- Practicality Concerns​​: The proposed solution appears computationally intensive relative to the problem's conceptual framework. I wonder whether the complexity is justifiable for deployment scenarios where predictions derive from fixed embeddings. (As a non-expert, I acknowledge this perspective may benefit from domain-specific context.)\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the label shift problem, where the source and target domain share the same conditional distribution, $P_s (X│Y)=P_t (X|Y)$ but differ in class priors, $P_s (Y)≠P_t (Y)$. A common approach to this problem is the Black-Box Shift Estimator (BBSE) method, which estimate a confusion matrix C and an empirical target prior $P ̂_t (Y)$ from finite samples to recover the true target prior $P_t (Y)$ , as $P_t (Y)=C^(-1) P ̂_t (Y)$. However, BBSE ignores the finite sample noise and semantic structure information during estimation. Therefore, this paper proposes a hierarchical Bayesian model (In Line 133-134) to improve the BBSE method. \n\nIn this model, the authors introduce a similarity graph on labels and utilize the graph Laplacian as covariance matrix in the distributions of latent variables, like θ and $ϕ_i.$\nWith these latent variables θ and $ϕ_i$, the model can estimate both the empirical target prior $P ̂_t (Y)$ and the confusion matrix $C$, then the BBSE method can be applied to recover the true target prior.\n\nSTRENGTHS AND WEAKNESSES:\n- Strengths\n\t1. This paper proposes a novel method to improve BBSE, which couples the target prior and confusion matrix through a hierarchical Bayesian model.\n\t2. The underlying idea that replacing the point estimate of BBSE with Bayesian estimate to calculate the confusion matrix and empirical target prior is reasonable.\n\t3. The propose method is theoretically solid. In Proposition 2, the authors give an upper bound of estimation error for θ.\n\n- Weaknesses\n\t1. The main weakness lies in the experimental section. (1) Lack of ablation study. In Table 3, the proposed GS-B3SE outperforms other methods. However, it relies on CLIP embeddings to construct the similarity graph, and there is no ablation comparing alternative graph construction methods. Since CLIP [1] demonstrates strong zero-shot performance on datasets like CIFAR-10 (zs acc: 91.3) and CIFAR-100 (zs acc: 65.1), the authors should provide additional experiments to clarify whether the observed improvements stem from the proposed method itself rather than the use of CLIP features. (2) Inconsistent graph construction. In Lines 274–278, the method for constructing the similarity graph differs across datasets. For MNIST, image embeddings are used, whereas for CIFAR-10 and CIFAR-100, text embeddings are employed. This inconsistency raises important questions: How do these different types of features impact the final performance? For a given dataset, how should one decide which type of embedding to use for graph construction?\n\t2. In Line 110-112, the authors assume the number of predicted samples per classes, $n_i^S$, follows a multinomial distribution. However, since $n_i^S$ is actually non-negative, this assumption may not hold.\n\t3. In Proposition 2, the notation $θ_0$ is not defined.\n\t4. Proposition 2 illustrates how the minimum eigenvalue of the graph Laplacian influences the estimation error. However, it seems that label similarity (or edge weights) does not directly impact the estimation of variables. Can we directly optimize the graph Laplacian with specific constrains?\n\nQUESTIONS:\nPlease refer to the weaknesses part.\n\nLIMITATIONS:\nThe proposed method is novel and theoretically solid. However, the empirical evaluation is not convincing. And the presentations have some flaws.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new method called GS-B³SE to address a practical problem in machine learning: label shift. This problem refers to a scenario where the class proportions (P(y)) change between the training and test sets, while the class-conditional feature distributions (P(x|y)) remain the same.\n\nExisting black-box methods (like BBSE) typically provide only a single point estimate, which makes them sensitive to sampling noise and completely ignores the potential semantic similarity between different classes (e.g., 'cat' and 'dog' are more similar than 'cat' and 'car').\n\nThe authors' core contribution is a fully Bayesian framework to address these issues. They don't just estimate a single target distribution q; instead, they introduce probabilistic priors for both the target distribution q and the classifier's confusion matrix C. The most crucial step is their use of a pre-constructed \"class similarity graph\" to impose a smoothing constraint on these priors. This graph structure allows semantically similar classes to \"borrow\" information from each other during parameter estimation. The benefit of this approach is that the model can better handle classes with sparse data, and the resulting estimates are more robust due to the consideration of uncertainty.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- The idea of introducing a graph structure into label shift estimation and innovatively using it to simultaneously smooth the priors for both the target distribution q and the confusion matrix C is very clever.\n\n- The theoretical analysis is robust. Furthermore, the paper interprets the method from an information geometry perspective, which undoubtedly adds depth to the work.\n\n- The paper is written clearly, making it easy for readers to follow the authors' line of thought.\n\n## Weaknesses\n\n- The entire model's core advantage is built upon the pre-given class similarity graph, L. This graph is constructed using embeddings from a pre-trained model (like CLIP) and remains fixed. This raises a significant question: To what extent does the quality of this graph determine the model's performance ceiling? What happens if the semantic similarity captured by the embeddings does not align with the classifier's confusion patterns? Although the paper analyzes the model's robustness to an inaccurate graph in Proposition 2, the bias term ||(L-L0)θ0|| remains, and it's unclear to the reader how large this term might be in practice. Positing graph learning as future work is reasonable, but this is indeed a core limitation of the current method.\n\n- Model inference relies on HMC sampling or a block Newton-CG optimizer. HMC is notoriously slow, especially for high-dimensional posteriors. While the authors mention a \"fast\" Newton-CG scheme, they provide no comparison of model runtimes in the experimental section. For a module positioned as a \"pure post-processing\" step, its practicality diminishes if its computational overhead is orders of magnitude higher than baselines. Its scalability, especially as the number of classes K becomes large (e.g., on CIFAR-100), is a question that needs to be answered.\n\n- The model assumes that the columns of the confusion matrix C are similar because \"columns corresponding to neighbouring predicted classes exhibit similar shape.\" Column i of C, C_:,i, represents the probability distribution over predicted labels for a sample whose true class is i. Why should one assume that the predicted distribution for a true 'truck' is similar in shape to the predicted distribution for a true 'bus'? While plausible, this is not self-evident. An alternative approach, such as smoothing the rows of C, seems equally plausible (i.e., for a given predicted label like 'truck', the distribution of its true source labels is similar to that for 'bus'). A more detailed discussion and justification for this specific design choice would be beneficial.\n\nQUESTIONS:\n- Regarding Graph Quality and Sensitivity: The model's performance appears to be highly dependent on the pre-constructed graph. To better understand this dependency, could you conduct some \"destructive\" experiments? For instance, how much does performance degrade when using a lower-quality graph (e.g., a k-NN graph generated from random embeddings)? This would help us more intuitively understand the practical significance of the robustness bound in Proposition 2.\n\n- Regarding Computational Efficiency: Thank you for the detailed experiments. Could you supplement them with the actual wall-clock runtimes for GS-B³SE and the main baselines (e.g., MLLS, BBSE)? This comparison would be particularly valuable for the CIFAR-100 dataset (with K=100) and would allow potential users to better weigh the trade-off between improved accuracy and computational cost.\n\n- Regarding Model Design Choices: As mentioned earlier, you chose to smooth the column vectors of the confusion matrix. Could you elaborate on the intuition behind this design? Did you consider or experiment with other smoothing approaches, such as smoothing the row vectors? Is there theoretical or empirical evidence to suggest that smoothing the columns is the superior choice?\n\nLIMITATIONS:\nThe limitations are discussed in the conclusion section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
By treating the confusion matrix as latent variables, this manuscript considers a fully Bayesian alternative to classical black-box label-shift estimators in the standard label-shift problem. The estimator can computed via HMC or Newton–conjugate-gradient optimizer. Numerical simulation and real-data experiments are run to test the proposed algortihm.

STRENGTHS AND WEAKNESSES:
I'm not an expert in label shift or ML with graphs. So, probably I am not able to comment on the strengths and the weakness. Nevertheless, I do have a few questions regarding the submission. Please see next section.

QUESTIONS:
1. In practice, how is the similarity graph constructed? I suppose such graph is usually noisy in real-world data. How will the noise affect the performance of the proposed algorithm?

2. How is the computational complexity of the algorithm compared with the standard BBSE? I suppose sampling from and inferring with the posterior distribution can be extremely time consuming in such Bayesian setting?

3. One of the motivation of the algorithm is to deal with the situation with rare classes. How does the algorithm perform in such case? I suppose in this case, the choice of prior distribution should affect the performance severely? Will it make the proposed algorithm less useful in such extreme condition?

LIMITATIONS:
Please see the questions above, especially on the question for the rare classes case.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03