Title: Kernel conditional tests from learning-theoretic bounds
Venue: NeurIPS 2025
Accepted: yes
Keywords: kernel methods

hypothesis testing

statistical learning
Abstract: We propose a framework for hypothesis testing on conditional probability distributions, which we then use to construct *statistical tests of functionals of conditional distributions*.
These tests identify the inputs where the functionals differ with high probability, and include tests of conditional moments or two-sample tests.
Our key idea is to transform confidence bounds of a learning method into a test of conditional expectations.
We instantiate this principle for kernel ridge regression (KRR) with subgaussian noise.
An intermediate data embedding then enables more general tests — including *conditional two-sample tests* — via kernel mean embeddings of distributions.
To have guarantees in this setting, we generalize existing pointwise-in-time or time-uniform confidence bounds for KRR to previously-inaccessible yet essential cases such as infinite-dimensional outputs with non-trace-class kernels.
These bounds also circumvent the need for independent data, allowing for instance online sampling.
To make our tests readily applicable in practice, we introduce bootstrapping schemes leveraging the parametric form of testing thresholds identified in theory to avoid tuning inaccessible parameters.
We illustrate the tests on examples, including one in process monitoring and comparison of dynamical systems.
Overall, our results establish a comprehensive foundation for conditional testing on functionals, from theoretical guarantees to an algorithmic implementation, and advance the state of the art on confidence bounds for vector-valued least squares estimation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel framework for hypothesis testing on conditional probability distributions, specifically developing a conditional two-sample test. The primary goal is to determine if the relationship between inputs (termed covariates) and outputs is the same across two different phenomena, with applications in fields like process monitoring, robotics, and medical studies.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. A central idea is the transformation of confidence bounds from learning methods into conditional two-sample tests. This principle is applied and demonstrated using Kernel Ridge Regression (KRR) and conditional kernel mean embeddings.\n\n2. To enhance practical usability, the paper introduces bootstrapping schemes. \n\n\n**Weaknesses**\n\n1. The central null hypothesis, as stated in Equation (5), is E(p1)(x) = E(p2)(x). This is a test for the equality of conditional expectations, not a conditional two-sample test in the general sense. \n\n2. The authors dedicate a substantial portion of the paper to laying out a highly abstract mathematical foundation involving Markov kernels, subgaussian processes in Hilbert spaces, and reproducing kernel Hilbert spaces (RKHS) for vector-valued functions. While this formalism is necessary for the paper's specific theoretical proofs, it is disproportionate to the narrowness of the final result. \n\n3. The entire framework is built on transforming the confidence bounds of Kernel Ridge Regression (KRR) into a statistical test. Consequently, the method inherits all the limitations of KRR. The most critical of these is the \"well-specified\" assumption, which requires that the true conditional expectation function lies within the chosen RKHS. This is a strong assumption that is difficult, if not impossible, to verify in practice, casting doubt on the reliability of the test's guarantees in real-world applications where model misspecification is the norm.\n\n4.  The paper claims an advantage in its ability to provide \"local\" guarantees, identifying at which covariates two functions differ. However, the value of this localization is severely diminished when the test can only detect differences in the mean. A practitioner could use this test and find no significant difference across the entire covariate space, incorrectly concluding the conditional distributions are identical, while in reality, one might have collapsing variance or other critical behavioral changes that the test simply cannot see.\n\nQUESTIONS:\n1. While the paper introduces bootstrapping schemes to avoid tuning inaccessible parameters, are there specific scenarios or types of datasets where these schemes might face practical limitations or lead to less accurate results? For instance, how robust are they to outliers or highly noisy data?\n\n2. The framework is built upon kernel methods. What guidance can be provided on the selection of appropriate kernel functions for different types of data and applications to ensure optimal performance of the conditional two-sample test? Are there specific types of kernels that are more suited for certain conditional distribution characteristics?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper's major contributions are:\n\n- A framing of the relationship between conditional two-sample testing – particularly when it is desired to find for _which_ conditioning values those distributions differ – and learning-theoretic bounds on estimation error of regression.\n- A new learning-theoretic bound for the estimation error of kernel ridge regression.\n- A bootstrapping scheme for KRR that avoids using the overly-conservative and hard-to-estimate precise form of the bounds derived for the test.\n\nSTRENGTHS AND WEAKNESSES:\n- The connection between estimation and testing is clear, and while the formal result of Theorem 4.1 is straightforward, setting it out in this way is illuminating. Connecting to the no free lunch theorem and the hardness result of Shah and Peters for conditional dependence is particularly helpful.\n  - It might be _more_ helpful to expand further, perhaps in an appendix, on the relationship between conditional two-sample testing to conditional independence testing. It is probably not obvious to all readers that the problems can be directly reduced to one another, nor that doing so actually breaks the jointly-continuous assumption that was vital to the construction of Shah and Peters but does not invalidate hardness in this setting.\n  - That this construction allows directly using time-uniform bounds is exciting! The paper could probably emphasize the practical implications of this a little bit more.\n\n- Theorem 4.3, and the generalization D.23, appear novel and interesting.\n  - It's a little disappointing that they don't yield practical tests, but this is also fairly expected; for instance, the 2012 paper after which this paper styles its title constructed bounds based on concentration inequalities (much simpler in the non-conditional setting).\n  - Appendix C's discussion of the relationship to the results of [40] in particular seems quite sparse. These authors also had a more recent followup generalizing the setting: [https://www.jmlr.org/papers/v25/23-1663.html](Li et al. 2024). Especially, they remove the trace-class assumption in a slightly different way, but their construction (as discussed eg in Remark 2) seems closely connected to your UBD definition.\n\n- The bootstrap scheme is straightforward and seems fine as a heuristic.\n  - It would be good to relate this to other bootstraping schemes for kernel ridge regression, especially the scheme of [Singh and Vijaykumar (2023)](https://arxiv.org/abs/2302.06578) who provide finite-sample guarantees. Can their scheme be used here?\n\n- Generally: I imagine some of the other reviewers will complain about the readability of this paper. It is certainly not a particularly approachable paper to non-experts in the field, but that seems somewhat inevitable for a paper taking on this set of results and framed in this way. This highlights a bit that Theorem 4.3/D.23, while relevant to the test procedure, is (a) a completely standalone result and (b) not actually used in the test procedure. Another strategy could be to write one more-accessible paper about conditional testing and its relationship to estimation error along with the heuristic for estimating it, and another paper about the theorem; I'm not sure this is necessarily a better strategy, but some reviewers might disagree.\n\nQUESTIONS:\n- How does Theorem 4.3/D.23, particularly in case 2, compare to the results of https://www.jmlr.org/papers/volume25/23-1663/23-1663.pdf that follow up [40]?\n- Can the Singh and Vijaykumar KRR bootstrap help here?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors establish conditions under which confidence bounds for KRR are equivalent to credible intervals given by Gaussian Process regression. They then show how they can be converted into conditional hypothesis tests. The main idea seems to be to base the hypothesis test on the pointwise overlap between two GP posteriors: the more overlap, the more likely one fails to reject the null hypothesis (at that particular input).\n\nSTRENGTHS AND WEAKNESSES:\nI preface this review by stating that the research area and proof techniques are quite unfamiliar to me. I can't fairly evaluate the impact or originality of the work.\n\n**Strengths.**\n* The writing is very clear, and the main argument is not too hard to follow.\n* There are no clear errors in the mathematical framing.\n* The idea is pretty neat -- in general it seems useful to find Rosetta stones between learning theory and statistical testing, and this result seems to be a big one. To be honest, I'm a bit surprised that this has not been done before!\n\n**Weaknesses.**\n* The technical formality obscures the main idea, in my opinion. I understand that rigor is important to some researchers, but I found it quite difficult to parse the theorem statements. Perhaps informal versions of the theorems can be given early in the paper.\n\nQUESTIONS:\n1) Why frame the learning algorithm as KRR with confidence bounds? The equations appear much more similar to GP regression with credible intervals given by the posterior covariance. A similar point is mentioned in passing in line 308, but to me this seems like the core insight behind the whole theorem! If that's true, then I think this should be emphatically mentioned earlier.\n\n2) If it is true that the proposed test is equivalent to examining the (pointwise) posterior distribution of the difference between the two estimators and testing whether the credible interval contains 0, then can you clarify how Theorem 4.3 is different from the standard theory of GP regression?\n\n3) It is known that functions that are less smooth w.r.t. to chosen kernel (i.e., have larger RKHS norm) are harder to learn (in KRR and GP regression). Intuitively, it seems that one should be less likely to reject the null in the case that the data generating processes is not smooth (and thus hard to learn). However, my understanding is that the posterior variance is blind to the alignment between the kernel and E[p](x), so the hypothesis test cannot depend on the smoothness of E[p](x). Can you provide some intuition as to why the hypothesis test does not seem to depend on this? Is it simply that the estimation error is contained in the posterior mean rather than the variance? (It could be that I'm simply mistaken about the relationship between estimation error and hypothesis testing.)\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a conditional two-sample testing framework using kernel methods with finite-sample guarantees. The approach builds confidence bounds via kernel ridge regression (KRR), leveraging existing tools like Abbasi-Yadkori's bounds. While the core idea of using KRR confidence intervals for testing is intuitive, the claimed ability to perform pointwise inference in regions with vanishing probability mass appears to conflict with known impossibility results (the paper even acknowledges this in Section 2: \"Conditional testing is notoriously hard—there exists no nontrivial test with prescribed level against all alternatives.\" ). The authors propose an ad-hoc bootstrapping scheme to address conservative non-asymptotic confidence regions, but this lacks theoretical justification. More fundamentally, the well-specified assumption (that true conditional means lie in the RKHS) is both unrealistic and fails to address how the method circumvents the acknowledged impossibility results for conditional testing.\n\nThe paper proposes a framework for conditional two-sample testing that focuses on identifying covariate values x where the conditional expectations $\\mathbb{E}\\_{p_1}[Y\\mid X=x]$ and $\\mathbb{E}\\_{p_2}[Y\\mid X=x]$ differ. Concretely, the authors reduce the testing problem to comparing two Kernel Ridge Regression (KRR) estimators. The core idea (Theorem 4.2) is that if each KRR estimator $f_{D_i}$ comes with a confidence bound $B_i(D_i,x)$ (i.e. with probability $1-\\delta_i$ we have $|f_{D_i}(x) - \\mathbb{E}[Y\\mid X=x]|\\le B_i(D_i,x)$ for all x in some set S and all sample sizes in N), then one can reject the null $H_0:\\mathbb{E}\\_{p_1}[Y\\mid x]=\\mathbb{E}\\_{p_2}[Y\\mid x]$ whenever $\\| f_{D_1}(x) - f_{D_2}(x)\\| > B_1(D_1,x) + B_2(D_2,x)$. Under these conditions the authors claim that the test has (pointwise) type‐I error bounded by $\\delta_1+\\delta_2$. So, the paper then aims to supply the required confidence bounds by deriving novel KRR concentration results. In Theorem 4.3 the authors claim a time-uniform (anytime) bound for vector-valued KRR under fairly general “uniform-block-diagonal” (UBD) operator-valued kernels $K(x,x')=\\iota(K_0(x,x')\\otimes {\\rm Id}_V)\\iota^{-1}$ and subgaussian noise. This result extends classic scalar KRR bounds to infinite-dimensional outputs and non-trace-class kernels (covering, e.g., conditional mean embeddings). Combining these bounds Theorem 4.4 yields a concrete test: both data sets produce KRR predictors and associated posterior variances $\\sigma(x)$, and one rejects $H_0(x)$ if $|f_1(x)-f_2(x)| > \\beta_1,\\sigma_1(x)+\\beta_2,\\sigma_2(x)$, where $\\beta_i$ depend on confidence levels. Theorem 4.4 shows this test has level $\\alpha_1+\\alpha_2$ over the covariate region X (or a chosen subset S). The authors also propose a practical bootstrap calibration (Appendix B) to estimate the thresholds $\\beta_i$ from data, avoiding manual tuning of RKHS-norm bounds. Empirically, the paper demonstrates in synthetic experiments that the proposed test controls type I error and can achieve higher power than a global conditional-distribution test (Hu & Lei [5]) when the difference in conditional means is localized in rarely-sampled regions of X. They also present toy illustrations of process change-detection using a sliding-window test on linear dynamical systems (Appendix A.3). Overall, the paper claims to provide a “comprehensive foundation” for conditional two-sample testing – bridging theoretical guarantees and practical implementation\n\nSTRENGTHS AND WEAKNESSES:\n### __Strengths__\n\n- __Related works:__ The authors did a good job reviewing related works and properly citing why conditional testing is a hard problem. \n\n- __Theoretical framework:__ The reduction from conditional testing to learning-theoretic confidence bounds in Theorem 4.2 (any two regression learners with high-probability error bounds yield a valid conditional test) seams elegant. \n\n- __Handling non-i.i.d. data:__ By allowing online sampling (a process of transition pairs) the method covers dependent or sequentially collected data. This is a strength over classical tests that require i.i.d. samples; the authors correctly emphasize that “outputs may not be independent”.\n\n\n### __Weaknesses__\n\n-  __Theoretical Issues:__ The method's claim to detect differences in vanishing-probability regions ($\\theta\\rightarrow 0$) lacks proper theoretical support and seem to contradict well-known impossibility results. From my point of view, the  __authors fail to__: \n(a) Specify necessary assumptions (e.g., density bounds, smoothness) to circumvent impossibility results.\n(b) Explain how their well-specified assumption and UBD kernels with trace-class constraints overcome fundamental limitations \nIn that regard, I suspect that the well-specified assumption and UBD kernel structure may implicitly restrict the complexity of joint distribution of Y|X to simple scenarios, significantly limiting practical applicability.\n\n- __Scope:__  The core method explicitly tests equality of conditional expectations. If two conditional distributions differ in higher moments (e.g. variance) but have the same mean, the basic test will not flag a difference. The authors mention that one can embed outputs or transform data (Algorithm 3 for CMMD) to compare other moments or full distributions, but these extensions are not the main focus. It should be emphasized that the presented test is primarily for conditional mean differences, and that extending to full distributional tests requires additional work.\n\n- The proposed __bootstrapping method lacks theoretical guarantees__ and may overfit\nThe well-specified assumption (that the true conditional means lie in the RKHS) is overly restrictive in practice. There is no guarantee that the chosen UBD kernel can adequately model arbitrary conditional distributions, especially without strong prior knowledge.\n\n- __Empirical Shortcomings:__ \n     - No validation on non-IID or sequential data despite claims \n     - Experiments limited to toy examples: \n     - Low-dimensional X (in dimension 1 or 2)\n     - No testing on high-dimensional complex objects or mis-specfied settings\n\n- __Baseline Comparisons:__\n     - Only compared to one baseline ([5]), omitting modern alternatives \n     - No ablation study to demonstrate UBD's contribution \n\n\nI summary, I cannot recommend this submission for acceptance due to: \n1. Insufficient theoretical justification for key claims (particularly rare-region testing) \n2. Lack of theoretical guarantees for the bootstrapping approach \n3. Narrow experimental validation that doesn't support practical utility\n\nQUESTIONS:\nNA\n\nLIMITATIONS:\nNot discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA general framework is proposed for kernel-based conditional two-sample tests that use sequences of input-output measurements, which are not necessarily independent and identically distributed (i.i.d.). The noises are assumed to be sub-Gaussian. The null hypothesis tested is that of equal conditional expectations given a covariate (input). A key advantage is that the output can be abstract, originating from a separable Hilbert space. This feature, combined with the theory of kernel mean embedding, allows the framework to be applied to test other hypotheses, such as those involving higher moments or even full conditional distributions. The paper also introduces the concept of uniform-block-diagonal kernels. An abstract test is first defined by building on finite-sample confidence bounds for learning algorithms. The paper's main results come in the form of confidence bounds for vector-valued kernel ridge regression (KRR) methods, summarized by Theorem 4.3 and later generalized by Theorem D.23 in the supplementary material. Based on these findings, kernel-based two-sample tests are defined and experimentally demonstrated using toy examples.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths of the paper include:\n- The paper addresses the crucial statistical problem of two-sample homogeneity tests (initially, for conditional expectations, which can applied for conditional distributions, as shown in the Appendices), proposing and rigorously studying a novel and promising framework for these problems.\n- The approach is highly abstract, allowing for a wide range of design choices. Its flexibility is demonstrated in the appendix, for example, through an example of process monitoring.\n- The concepts are clearly defined, and the theorems are precisely stated and proven.\n- The results are placed in context, with their connections to existing bounds further discussed in Appendix C.\n- The proposed methods apply to non-i.i.d. data and allow online data generation.\n- The specific tests and toy examples provided effectively illustrate the approach.\n- The code for the numerical experiments is made available online.\n\nWeaknesses of the paper:\n- The current title is overly general. A more specific title that highlights that the tests are based on guarantees of vector-valued KRR would be beneficial.\n- Subgaussianity is a core assumption of the framework, and therefore, it should be explicitly mentioned in the abstract.\n- In Section 4, both the null hypothesis (H_0) and the alternative hypothesis (H_1) should be explicitly defined.\n- Curiously, the paper's core contributions appear to be the confidence bounds for vector-valued KRR methods (Theorem 4.3), with the two-sample tests emerging as a straightforward corollary. Yet, the paper identifies the tests themselves as its main contributions.\n- The abstract also refers to material which can only be found in the supplementary material (e.g., process monitoring).\n- The numerous technical assumptions required may limit the practical applicability of the results beyond toy examples.\n- Since the tests are based on kernel methods, their scalability might be an issue.\n- Even the applications to the toy examples need a heuristic bootstrap step.\n- The conclusion's claim of presenting a complete pipeline from conceptual foundations to practical implementations seems an exaggeration. Practical issues are only briefly touched upon, with, for example, the bootstrap scheme detailed solely in the appendix.\n- A minor, yet annoying issue is that many formulas in the supplementary material extend beyond the margins.\n- I had the impression that the authors tried to squeeze too much material into the paper, and valuable contributions are therefore placed in the supplementary material, due to lack of space (e.g., generalized confidence bounds for vector-valued least squares).\n\nQUESTIONS:\n- Are these tests mainly useful for small sample settings or do you expect them to be scaleable?\n- What is the primary obstacle to achieving a distribution-free test? Is it possible to eliminate the bootstrap step?\n\nLIMITATIONS:\nThe paper adequately addresses its limitations; however, a deeper discussion on its practical applicability is noticeably absent.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-13"}