Zipper: Addressing Degeneracy in Algorithm-Agnostic Inference
Authors: Geng Chen
Yinxu Jia
Guanghui Wang
Changliang Zou
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Asymptotic normality
Cross-fitting
Goodness-of-fit testing
Model-free
Variable importance.
Primary area: interpretability_and_explainability
TL;DR: Addresses degeneracy issues in algorithm/model-agnostic goodness-of-fit inference such as assessing variable importance.

Abstract: The widespread use of black box prediction methods has sparked an increasing interest in algorithm/model-agnostic approaches for quantifying goodness-of-fit, with direct ties to specification testing, model selection and variable importance assessment. A commonly used framework involves defining a predictiveness criterion, applying a cross-fitting procedure to estimate the predictiveness, and utilizing the difference in estimated predictiveness between two models as the test statistic. However, even after standardization, the test statistic typically fails to converge to a non-degenerate distribution under the null hypothesis of equal goodness, leading to what is known as the degeneracy issue. To addresses this degeneracy issue, we present a simple yet effective device, Zipper. It draws inspiration from the strategy of additional splitting of testing data, but encourages an overlap between two testing data splits in predictiveness evaluation. Zipper binds together the two overlapping splits using a slider parameter that controls the proportion of overlap. Our proposed test statistic follows an asymptotically normal distribution under the null hypothesis for any fixed slider value, guaranteeing valid size control while enhancing power by effective data reuse. Finite-sample experiments demonstrate that our procedure, with a simple choice of the slider, works well across a wide range of settings.

Zipper: Addressing Degeneracy in Algorithm-Agnostic Inference
Changliang Zou NITFID, School of Statistics and Data Science, LPMC, KLMDASR, and LEBPS, Nankai University gengchen... stat@gmail com, yxjia@mail nankai edu. cn, ghwang. nk@gmail. com, zoucl macc@nankai.. wwwnkaii.du..c.. cn
Geng Chen"
Yinxu Jia" Guanghui Wang*t
Abstract
The widespread use of black box prediction methods has sparked an increasing interest algoritmm/model-annosses agnostic approaches for quantifying goodness-of fit, with direct ties to specification testing, model selection and variable importance assessment. A commonly used framework involves defining a predictiveness criterion, applying cross-fitting procedure to estimate the predictiveness, and utilizing the difference in estimated predictiveness between two models as the test statistic. However, even after standardization, the test statistic typically fails converge non-degenerate distribution under the null hypothesis of equal goodness, leading to what is known as the degeneracy issue. To addresses this degeneracy issue, we present simple yet effective device, Zipper. draws inspiration from the strategy of additional splitting of testing data, but encourages an overlap between two testing data splits in predietiveness evaluation. Zipper binds together the two overlapping splits using slider parameter that controls the proportion overlap. Our proposed test statistic follows an asymptotically normal distribution under the null hypothesis for any fixed slider value, guaranteeing valid size control while enhancing power by effective data reuse. Finite-sample experiments demonstrate that our procedure, with simple choice of the slider, works well across wide range of settings.
Introduction
Consider predicting response Y TR from covariates X E RP, Due to the popularity of black box prediction methods like random forests and deep neural networks, there has been a growing interest in the so-called "algorithm (or model)-agnostic" inference the goodness-of-fi (GoF) in regression 2 5. This framework aims to assess the appropriateness of a given model for prediction compared to potentially more complex (often nigher-dimensional) model. A common approach involves defining a predictiveness criterion, employing either the sample-splitiigg or cross- fitting strategy to estimate the predictiveness of the models, and examining the difference in predictiveness. The main focus of this work is to address the issue of degeneracy encountered in predictiveness-compariss based test statistics.
1.1
Goodness-of-Fit Testing via Predictiveness Comparison
Let P represent the joint distribution of (Y,X) and consider a class F of prediction functions that effectively map the covariates to the response. Define a criterion C(f, P). which quantifies the predictive capability of a prediction function f E F. Larger values of C(J, P) indicate stronger
All authors contributed equally to this work and are listed in alphabetical order. Guanghui Wang and Changliang Zou corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

predictive capability. The optimal prediction function within the class F is determined as f E arg fer C(f, P). Therefore, C(f, P) represents the highest achievable level of predictiveness within the class F. Examples of C include the (negative) squared loss C(J. P) = =- [( for continuous responses and the (negative) cross-entropy loss C(f, P) = E|Y log (XX + (1
Y) log - [X))] for binary responses, where F. denotes the expectation under P.
In GoF testing problems, there are typically two classes of prediction functions F and Fs. where Fs is a subset of F. Define a dissimilarity measure Vis = C(f. P) C(fs. P), which quantifies the deterioration in predictive capability resulting from constraining the model class to Fs. Here, Is E arg jaFF C(ʃ, P) denotes the optimal prediction function within the restricted class. Since Fs S F, it follows that vss 2 0. The GoF testing is then formulated as
Ho Vs 0 versus H1 .bs > 0.
(I)
The formulation of assessing a scalar predictiveness quantity is inspired by the work of Williamson al. [5], which focuses on evaluating variable importance. Other predictiveness or risk quantities have also been explored by 77 [9] Here, we highlight a few examples where the aforementioned framework can be directly applied.
(Specification testing) Model specification testing aims to evaluate the adequacy of class of postulated models, such as parametric models, say, examining whether the conditional expectation E(Y | X)= = go (X) holds almost surely [00] where ge is a known function up to an unknown parameter 0. Under the framework
we can consider F as generally unrestricted class, while Fs represents class of parametric models.
(Model selection) GoF testing can also be employed identify the superior predictive model from a set of candidates [1] 12). This situation often arises when choosing between two prediction strategies, such as unregularized model and regularized one. Testing Ho is to assess whether the inclusion of a regularizer provides benefits for predictions.
(Variable importance mcasure) A recently popular aspect of GoF testing involves evaluating the significance of a specific group of covariates U in predicting the response, where X = (UT, VT)T. This assessment can be incorporated into (1) by defining a subset of prediction functions Fs F that disregard the covariate group t when making predictions. When utilizing the squared loss, the simplifies to the LOCO (Leave Out COvariates) variable
importance measure [3] 14 5 15] 16), which quantifies the increase in prediction error resulting from the removal of U. Specifically, taking v/s ElLY - E(Y V E((Y- E(Y X))² corresponds testing for conditional mean independence
18.
The measure vs possesses the advantages of being both model-free and algorithm-agnosse It is not tied to a specific model and remains independent of any particular model fitting algorithm. This flexibility makes ths versatile quantity for evaluating goodness-of-f enabling us to leverage diverse machine learning prediction algorithms in its estimation.
1.2 The Degeneracy Issue
Let Z = (Y.X) ~ P. and suppose we have collected a set of independent realizations of Z as Z, = (Y, for = I...., n. To effectively estimate ths while ensuring algorithm-agnostic inference, the sample-splitting or cross-fitting has recently gained significant popularity. This approach relaxes the requirements imposed on estimation algorithms, allowing for the utilization of flexible machine learning techniques [ 19
21 Taking sample-splittin as example, the data is divided into a training set and a testing set. Based on the training data, estimators f and fi are obtained for the optimal prediction functions f and fs. respectively. The dissimilarity measure v can then be evaluated using the testing data, i.e., thn.s = C(fa, pkt) - C(fts Pte), where pte represents the empirical distribution function of the testing data. Notably, in the context of measuring variable importance, it has been established that when the > O, the estimator yh.s exhibits asymptotic linearity under certain assumptions [5]. Similar results are also applicable to the problem of comparing multiple algorithms models [2].
However, the situation becomes distinet when considering the null hypothesis Hg v/s = 0. Recent
studies have drawn significant attention to a phenomenon known as degeneracy 22 23 15 18 5)

---

Consider the simplest scenario where there are no covariates and the objective is to test whether
μ := =Y) = 0: see also Example in Lei .. In this case, we set F R and Fs {0). Using the squared loss, we have ihs F(Y2) - E{(Y [] = nr² The estimator for the based on sample-spliting is s = 24. ( where Yur and y denote the sample means of the training and testing data, respectively. When μr # O, vn((,,,s (4) follows asymptotic normal distribution. However, when μl 0, Vniins = Op(n=1/3) indicating the presence of the degeneracy phenomenon. While inference at a n-rate remains feasible under degeneracy in this specific example, it is crucial to recognize that this would not hold true for intricate models and black box fitting algorithms. Williamson et al. [5] provide evidence for the occurrence of degeneracy in a general variable importance measure v/s, where the influence function becomes exactly under the null hypothesis. It is therefore required to address this degeneracy issue a generic manner to perform algorithm-agnoste statistical inference.
1.3 Existing Solutions
By using the fact that the influence functions of the individual components C(f, P) and C(fs. P) in thy remain nondegenerate even under Ho. Williamson al. [5] proposed an additional data split of the testing data, where the two predictiveness functions are evaluated on two separate testing data splits. This approach ensures nondegenerate influence function under Ho, therefore restoring asymptotic normality. A similar approach has been independently explored by Dai et al.
23 However, performing additional data splits may reduce the actual sample size used in the lesting, leading to substantial loss of power. Alternatively, Rinaldo et al. [13 and Dai et al. [23] introduced data perturbation methods, where independent zero-mean noises are added to the empirical influence functions. These methods also restore asymptotic normality. However, determining the appropriate amount of perturbation achieve desirable Type I error control remains heuristic process. Furthermore, Verdinelli and Wasserman [22) proposed expanding the standard error of the estimator mitigate the impact of degeneracy.
1.4 Our Contributions
In this paper, we introduce the Zipper device for algorithm-agnostic inference under the null hypothe- sis Hu of equal goodness. Our approach is inspired by the method of additional splitting of testing data, as demonstrated in the works of Williamson et al. (5) and Dai et al. [23] for assessing variables with zero-importance. Instead of creating two distinct testing data splits to evaluate the discrepancy of predictiveness criteria between the expansive and restricted models, we encourage an overlap between them. The Zipper device serves to bind together the two overlapping splits, with a slider parameter 7 E [0. 1) controlling the proportion of overlap. To ensure stable inference and accommo- date versatile machine learning prediction algorithms, we incorporate a K-fold cross-fitting scheme. We will demonstrate that the proposed test statistic follows an asymptotically normal distribution Ho for any fixed value 7, ensuring valid size control while providing satisfactory power enhancement.
1.5 Related Works
For variable importance assessments, in addition to LOCO methods within our framework, Shapley
value-based measures are commonly used 224 25 26 These measures, which estimate the incremental predictive accuracy contributed by a specific variable across all possible covariate subsets, reveal complex inter- varialle relationships but at considerable computational expense. Furthermore, conditional randomization tests [28] [29 offer robust alternative when covariate distributions are known or can accurately estimated. These methods are especially beneficial in semi-supervised settings with extensive unlabeled data. Additionally, LIME (30) focuses on estimating variable importance locally for a specific instance, while LOCO methods are designed to assess global variable importance.
1.6 Organization
The remainder of our paper is structured as follows. In Section 2 we introduce the Zipper device for addressing the degenerate issue, and present the asymptotic behaviors of the method. Finite-sampi sample experiments are presented Section 3 Section 4 concludes the paper. Proofs of theorems and additional numerical results are provided in Appendix.
3

---

2 Our Remedy
2.1 The Zipper Device
To initiate the process, we randomly partition the data into K folds, denoted as Di.... Dx. ensuring that each fold is approximately of equal size. For a given fold index E (1...., , Kj. we construct estimators f and Sk for the oracle prediction functions f and fs correspondingly, using the data excluding the fold Dj. To activate the Zipper device, we further randomly divide D; into two splits, labeled DR,A and DR. with approximately equal sizes, allowing for an overlap denoted as D.. Specifically, D U Dx.B= = Dx and D O Da.B = Dr.o- Let DR.a = D D and DE.6 DE.B' DD.o represent the noveoverlg parts. For simplicity, and 'D.AAl= [Dx.B], where [A| represents the cardinality of set A. Let T = DD. [Dx.A represent the proportion of the overlap. Visualize the two splits DR,A and Pa. two pieces of fabric or other materials. The term "Zipper" is derived from the analogy of using a zipper mechanism to either separate join them by moving the slider т; v/s = C(f, P)- C(fs, , P) based on the kth fold of testing data DR. we denote Pk.n.J as the empirical distribution of the data split DR.J, where I E to, a, bj. We can estimate ihs by Ck. Ck.n,S, where
assume that [D&l n/K
Figure []
To evaluate the discrepancy measure
Ck =C(fk,ns Pk.no) + (1- (1-7)C(fknn) P Pk, and Ck,nss = = Ps....) + (1 (ffknns
represent weighted aggregations of empirical predictiveness criteria corresponding to overlapping and non-overlapping parts, used for estimating C(f, P) and C(fs, P), respectively. By employing the cross-fitting process, we obtain the final estimator v/s, denoted as
K
S K
(Can - Ckn.s).
(2)
1=1
which is the average over all testing folds.
Notably, if fully open the Zip- per, setting - = and leaving DE.o empty, our approach aligns with the vanilla data splitting strategy utilized in Williamson et al. [5] and Dai et al. [23] for assessing variables with zero- importance. Conversely, when we completely close Zipper with = 1, our method essentially in- volves evaluating the predictive- ness discrepancy using the iden- tical data split DR. = De.A
22.A
Cifenr Plrn)
Den' Ui Pann) De.t Clfen Pann))
Di C(imm.s Pawn)
D
Figure Illustration of the mechanism of the Zipper device based on the kth fold of testing data.
Dr.B Dr. which is known to result in the phenomenon of degeneracy [5] Therefore, to avoid such
degeneracy, we restrict the slider parameter т E [0, 1); see also Remark 2.5 below.
2.2 Asymptotic Linearity
We start by investigating the asymptotic linearity of the proposed test statistic uh..s in ( which serves as a foundation for establishing the asymptotic distribution under the null hypothesis, as well as for analyzing the test's power.
Theorem 2.1 (Asymptotic linearity). If Conditions (CI)-(C4) in Section A hold for both tuples (P,F, fw.n and (P, , fs, fk.,,)), then
bn.s v/s n2
1
Σ
(lzi)
Σ
k=1
1:Z,EDE.a
:2,.DD.
(3)
+
Σ
(0(2,) os(2i)) + 10p(n-1/2
i:Z.ED..o

---

where = C(f. P) and os Pråz P). Here, Clf.P:h) represents the
Gâteaux derivative of P + C(f, P) P in the direction h, and óN denotes the Dirac measure Consequently, for any T E [0, 1),
(2/ - [) - vs) 4 NO.,
(Vn.s
where V (1 (o² +b}) + TTS o² = E(0 os (()) and vi ()(((Z) (z{]]
Remark 2.2. The conditions in Theorem 2.1 derived from Williamson et al. 15), which outline specific requirements concerning the convergence rate of estimators obtained from flexible black box prediction algorithms and the smoothness of the predictiveness measures. The validity of Theorem 2. relies on the asymptotic linear expansions of ((fk,a Pk.n.1) and C(fk,n n.) for E {o, a, bj: Lemma [.T or Theorem 2 in Williamson et al. [5]. Given the asymptotic linearity of Sn.S. readily obtain its asymptotic distribution by observing the independence of data in U DR,a- UK Da.b and D well the facts that U A== DR.al
I Da.bl (1 r) 7)
and U k= DR.ol= (2/(2- T).
The asymptotic linearity of vn.s in (3) exhibits distinct behaviors depending on the validity of Ho- Under Ho 4s = 0. we have Øs almost surely, due Is almost surely. Consequently, the overlapping terms Di.zEDe.. { 7x) (ss{ for k = 1 K vanish. As result, 3 simplifies
to
k
K Σ k=
Vn,5 ths
Σ k= ED
Σ
(ZZ)
Σ
os(Zi) + tor(n
(4)
n/ r)
k=1&2,
In this case, the dominant term compares the means two independent samples, each with a size of (1 (2 r). Additionally, it can be deduced that v/,,- - T) 2 + ozs) under Ho- Conversely, under H1 vss > 0, we can reformulate 33 as
K
Vn,s Vs
1 n/(2
(()))
Σ
Σ
dss 6s( up(nnnnn// 1/2).
k=
k=
Here, the leading term compares the means of two samples that have an overlap, with each sample having size of n/(2- - T).
2.3 Null Behaviors
To conduct the test, is crucial to have consistent estimator for the variance v3,r (1 (o2 + +oz) under Ho. Following the plug-in principle, we derive an estimator S,S.r == (1
>>-K + 02 ss,,,, where for each (1,...,K).
Σ
[C(fk,n Pa.fkin,P...bbe 82, - pk,n))²
1D&
i
Σ
(C(Junnss Pk 8z. p,,
i:Z:ED2
and Pk.n represents the empirical distribution of Dr. The consistency of this estimator is demonstrated Proposition 2.3
Proposition 2.3. If Conditions (C4)-(C5) in Section hold for both tuples ff and (P,Fs, Js, Jkn,s) then, v².s.r 24 v2,-- as n + do under for any E [0, 1).
Remark 2.4. Computing Gâteaux derivatives C for certain predictiveness measures can be challenging.
In many cases, the predictiveness measure takes the form C(J, P) [YY, ((x))) where is
known function. In such situations, it has been found that the Gâteaux derivative be expressed
as (>(2) == g(y, f(x)) E'g(Y, f(X))) see, for example, Appendix in Williamson al. (5). Consequently, when f f. the empirical Gâteaux derivative becomes Č(fkn Pk ò₂ Pkn)

---

g(v, fx.())) - " 9Y, (X.)))) This formulation allows for the identification of the variance estimator k. as the sample variance of (g(Y,, (x,))) simplifying the computation. Moreover, Condition (C5) is immediately satisfied (see Section B.2. Examples of such function include the squared loss and cross-entropy loss.
Based on Theorem 2. and Proposition utilizing Slutsky's lemma, we can conclude that the normalized test statistic
T. = (n/ /n. //V.S.. 4 ())
under Ho for any [0, 1). For prespecified significance level (0, we reject the null hypothesis T 21-a, where Za denotes the a quantile of N(0,).. A summary of the entire testing procedure can be found in Section[[]
Remark 2.5. In our asymptotic analysis the null behavior, we explicitly exclude the case of r = due to the resulting degeneracy phenomenon. Specifically, under Ho. when 7 = 1, the linear leading term of 33) becomes exactly zero. Therefore, including T = can introduce a distortion the Type I error.
2.4 Power Analysis
Next, we turn attention the power analysis of the proposed test under the alternative hypothesis
H1 : Us > 0.
Theorem 2.6 (Power approximation), Suppose the conditions stated in Theorem and Proposition
2.3 hold. Then for any [0, 1), the power function Pr(T, 21-ax H1) (T) o(1), where
[/s.
(0)
21 + fn/ - 1/)
Gs, =
VS.r
VSSr
(0) (1-- 23) and & denotes the distribution funcctiooooonnnniionn of (0, 1). Furthermore, if S.T
Cov (d(Z), (Z)) 0, then (s,,,((T)) increases with т.
Remark 2.7. The form of the power function can be directly derived from Theorem 2.1 and the fact that the estimator of standard deviation Vn,S.r & V/srr as n + oc under H1. Recall that v/.rr = (1 (o- + Trg. which is provably a decreasing function of 7 when Cov(o(Z), (2s(2) 0. Consequently, the approximate power function (..,,.(TT) increase with r. For more details, please refer to Section
(0)
Remark 2.8. The requirement Cov db d) (2)) > is relatively benign. For instance, when considering v = F](Y F(Y V) F[((Y -F(Y x))² within the framework of evaluating
variable importance, this condition is readily satisfied; refer to B
Consider the "unzipped" version of the proposed test with = O, which has been explored in the works of Williamson et al. [5] and Dai et al. 223 for assessing variable importance. According to Theorem 2.6] the approximate power function reduces to
(n/ (n22) (02 + 03)1/2
Gs (0) = +p -211-a +
aligning with the findings in Dai et al. [23] In contrast, for the Zipper with 7 [0, 1), we have
Gs (T) _ (-21-a+ {n/(2 7/
2 Gs,n.
(02 + 00
due to the facts that VS.r < MS.r, vs.r S (0"+0 (00 our method surpasses the vanilla sample-splitting or frttinnitting based inferential procedures that correspond to = 0. The improved power can be attributed to two sources: the introduction of the overlap mechanism 7 (corresponding to Inequality (ii)), and the utilization of the variance estimator v2 in,s, (Inequality (i))
if Cou(o(Z), os 2 0. Consequently,

---

Remark 2.9. As discussed, V Srr is inconsistent for the limiting variance V/ under H1 when 0 T If the objective construct valid confidence interval for the dissimilarity measure Vs. it is crucial to use a consistent variance estimator regardless of whether Ho holds or not. This can be achieved by incorporating an additional plug-in estimator of 1/s, which is component of the asymptotic variance VSS.. For detailed construction of valid confidence interval, please refer to Section in Appendix.
2.5
Tradeoff
Our asymptotic analysis demonstrates that the Zipper device ensures a valid testing size for any fixed slider parameter [0, 1). Moreover, the slider moves away from 0, the power improves. In practical scenarios with finite sample sizes, selecting an appropriate value of 7 involves a tradeoff between efficiency and degeneracy. Opting for larger value of I indeed enhance the testing power. However, an excessively large 7 approaching 1 would result in degeneracy and potential size inflation. This occurs because the normal approximation (4) breaks down under the null hypothesis. It is worth emphasizing that using a relatively small r is generally safer and yields improved power compared to the vanilla splitting- -bated strategies with 0.
To achieve better power while maintaining a reliable size, we propose a simple approach for selecting By (4), the asymptotic normality relies on comparing means from two independent samples D and D each with a size of (1 72 - r). To ensure a favorable normal approximation,
choose the sample size (1 m2 7) such that it meets predetermined "large" sample size, such as ng = 30 or 50. Say, we can specify t = To == (n 2mp)/(n no). In the case of very large n, a truncation may be needed to safeguard against degeneracy. For example, we can set 7 min To, 0.9}. Our numerical experiments show that this selection of r achieves satisfactory performances across wide range of scenarios. For more details on the impact of T, please refer to Section E.T
3 Finite-Sample Experiments
3.1 Synthetic Experiments
3.1.1 Variable Importance Assessment
For illustration, we conduct a simulation study to evaluate the performance of the proposed Zipper method in assessing variables with zero-importance,
of active research. We compare the empirical size and power of Zipper against several benchmark procedures. Firstly, we consider Algorithu 3 proposed in Williamson et al. [5] (referred to as WGSC-3) and the two-split test in Dai et al. (33 (DSP-Split). Both procedures involve an additional splitting of the testing data and can be seen approximate counterparts the proposed Zipper test with 7 O. Another benchmark procedure Algorithm 2 from Williamson et al. (5) (WGSC-2), which can be viewed as a rough include the data perturbation method proposed by Dai et al. (23) (DSP-Pert). For each benchmark procedure, we follow the suggestions of the respective authors to select nuisance parameters. We specify the slider parameter 7 min(To, 0.9)
equivalent to Zipper with 1. Additionally,
with no- 50 as suggested in Section 2.5
We consider two models: one with normal response Y ~ N(X 8, of and another with a binomial response Y ~ (inom(1, logit(X 3)), where logit(t) = 1/(1 + exp(-t)). Both models assume that X ~ (0, Σ), where Σ = (0.2 pxpp For each model, we examine two scenarios. The first scenario is a low-dimensional setting with p E {5, 10) and 3 = (8, 5, 5. 0, 5, T. The second scenario is high- nigh-dimensional setting with p f200, 1000} and / (5, 5, 50.01p. 07.9op-2) where ag represents g-dimensional vector with all entries set to In the normal model, such that the signal-to-noise ratio 3 / 3 by assigning å = 0. The objective is to test whether the first two variables contribute significantly to predictions from a sequence of n E (200, 500, 1000) independent realizations of (Y,X).
specify ož
Let F represent a generally unrestricted model class, subjecting to a degree of sparsity under the high-dimensional settings. Consider Fs C F such that the prediction functions exclude the first two components of the covariates. To test the irrelevance of the first two variables in predictions, we examine Ho ths = 0 in (I). We adopt the squared loss for normal responses and the cross-entropy
7

---

Table 1: Empirical sizes (in percentage) of various testing procedures, with standard deviations in brackets.
Model
p
Zipper
WGSC-3
DSP-Split
WGSC-2
DSP-Pert
5
3.9(0.19)
5.1(0.22)
4.6(0.21)
0.1(0.03)
10.2(0.30)
Normal
1000
4.3(0.20)
6.2(0.24)
5.9(0.24)
16.7(0.37)
35.0(0.48)
5
3.7(0.19)
3.9(0.19)
4.2(0.20)
0.6(0.08)
4.0(0.20)
Binomial
1000
5.6(0.23)
4.8(0.21)
5.1(0.22)
19.9(0.40)
38.6(0.49)
loss for binomial responses. The ordinary least-squares regression and the LASSO are utilized under the low-dimensional and high-dimensional scenarios, respectively. The significance level chosen 5%, and experiments entail 1, 000 replications. These experiments are executed Intel Xeon Gold 5118 18 CPU @ 2.30GHz.
Table displays the empirical sizes for different testing procedures with 500 and p {5, 1000). The results reveal that the Zipper, WGSC-3, and DSP-Split consistently maintain the correct size across all models, anticipated. In contrast, the WGSC-2 exhibits conservative behavior in the low-dimensional setting and inflated sizes the high-dimensione settings, primarily due to the degeneracy phenomenon. In addition, the data perturbation method, DSP-Pert, fails to control the size some cases, particularly in the nigh-dimensional settings. This instability can be attributed to the selection of the amount of perturbation.
Figure 2 depicts the empirical power of various testing methods as func- tion of the magnitude representing variable relevance, when = 500 and E {5, 1000}. As expected, the Zip- per shows substantial improvement in power compared to the vanilla cross- fitting based approaches, WGSC-3 and DSP-Split, with WGSC-3 and DSP-Split demonstrating similar performances. Un- der the high-dimensional settings, the WGSC-2 and DSP-Pert exhibit higher power than Zipper, but at the expense of losing valid size control. For prehensive analysis of the empirical sizes and power across various combinations of n and please refer to Section
p-5
p-1000
0.75
0.26
Power
0.0
0.5
1.0
1.5
2.0
2.5
E22 These additional results consistently support the conclusion that the Zipper method demonstrates reliable empirical size performance and significant power enhancement compared to that methods that utilize non-overlapping splits.
5
Zipper
DSP-spit
Figure 2: Empirical power of various testing methods as function of the magnitude 8 with 500 and p (5. 1000). The dot-dashed horizontal line represents the intercept
5%.
3.1.2 Model Specification Testing
We extend our investigation beyond the variable importance assessment problem include evaluation of the proposed Zipper device in addressing model specification issues. Our analysis focuses on the model defined as Y = X3 E, where X ~ (0, Σ) with Σ = (0.2li-jl) pxx and ~ (0, o²). Here, we assume that ||31|0 = indicating the presence of two nonzero components in 3, while the positions of these components remain unknown. Our objective is to test the model specification hypothesis: Ho 3 = (+, +, 0p-2) versus H, |1,81|o 2 but not Ho, where represents any nonzero value. To generate the data,
consider three scenarios: (i 8 (0.4, 0.4, T (ii) B (0.4, 0, 0.4, op-a) and (iii) / (0, 0, 0.4, 0.4, TT We determine the value of a such that the signal-to-nol ratio 3 E,//0² Subsequently, we generate independent realizations (Y,, X,) for = 1,..., ... with n = 500 and p (5, 10003. To estimate B, we employ ordinary least squares under Ho. and perform the best

---

two subset selection under H1. For the case when 5, we conduct an exhaust search. For the case when p 1000, we utilize the abess algorithm [3] to approximate the solutions. The results are summarized in Table 2 where we observe that under Scenario (i), corresponding to Ho. Zipper maintains correct size control. Furthermore, under Scenarios (ii) and (iii), corresponding H1, Zipper exhibits substantial improvements in power when compared to the vanilla cross-fittigg based approaches, WGSC-3 and DSP-Split.
Table 2: Empirical sizes and powers (in percentage) for the model specification test, with standard deviations in brackets.
P
1000
Scenerio
Zipper
WGSC-3 DSP-Split
WGSC-2
Zipper
WGSC-3 DSP-Split WGSC-2
(i)
4.3(0.20) 6.2(0.22) 5.6(0.20)
0.0(0.00)
4.2(0.19) 5.5(0.20) 6.5(0.21) 16.6(0.36)
(ii)
96.9(0.17) 31.2(0.46) 34.9(0.46)
100.0(0.00)
94.2(0.22) 29.8(0.46) 31.4(0.46) 97.3(0.16)
(iii)
100.0(0.00) 81.4(0.39) 79.3(0.38) 100.0(0.00) 100.0(0.00)
81.3(0.40) 78.1(0.41) 100.0(0.00)
3.2 Real-Data Examples
3.2.1 MNIST Handwritten Dataset
We apply the Zipper method to the widely used MNIST handwritten digit dataset 32 The MNIST dataset consists of size-normalized and center-aligned handwritten digit images, each represented as 28 x 28 pixel grid (resulting p 28 784). For our analysis, we specifically extract subsets of the dataset representing the digits 7 and 9, following Dai et al. [23]. resulting in a total of 14251 images.
In Figure 3 we calculate and graphically represent the average grayscale pixel values for images sharing the same numerals. We divide each image into nine distinct regions, as shown by the blank squares in Figure 3 with the objective of detecting regions that can effectively distinguish between these two digits. To achieve this, we perform sequence of variable importance testing to assess the relevance of each region in making predictions while considering the remaining regions. Given the nature of the data, we employ Convolutional Neural Network (CNN) as the underlying model, leveraging its proven effectiveness in image analysis. In the Zipper approach, we select the slider parameter r such that nig = 50, as recommended in the manuscript. As benchmark, we adopt WGSC-3 (5) (equivalent to DSP-Split (23). which produces valid size, aligning with our approach. We set the predefined significance level for each test as a 0.05/9, applying the Bonferroni correction to account for multiple comparisons. The discovered regions, highlighted in red, are presented in Figure 3 Our findings indicate that the Zipper method outperforms WGSC-3 in identifying critical regions with greater efficacy.
Figure 3: Hypothesis regions (blank squares) and important discoveries (squares filled in red) com- paring the Zipper method (left column) with WGSC- 3 (right column).
3.2.2 Bodyfat Dataset
We expand the application of our Zipper method to the bodyfat dataset [3), which provides estimate of body fat percentages obtained
through underwater weighing, along with various body circumference measurements from a sample of n = 252 men. Our objective is to conduct marginal variable importance tests for each body circumference while considering potential influences from essential attributes such as age, weight, and height. To accurately estimate the relevant regression functions within this dataset, we employ the random forest as our modeling technique. Table 3 presents the resulting p- p-values obtained from the Zipper and WGSC-3 methods. By applying the Bonferroni correction, the Zipper method identifies both Abdomen and Hip significant factors at the significance level of a 0.05/ 10. In contrast, WGSC-3 suggests only Abdomen as important. It is worth noting that a recent study by Zhu et al. as straightforward evaluation index for body fat. Remarkably, our finds align with this fact, further supporting the validity and relevance of our Zipper method in identifying key factors.
34 proposed the formula (Waist+Hip)/Heing

---

Table 3: P-values obtained from the Zipper and WGSC-3 methods for each marginal test regarding the relevance the body circumference.
Body Part Neck Chest
Abdomen
Hip
Thigh Knee Ankle Biceps Forearm
Wrist
Zipper 0.98 0.10
5.48 x 10-10 4.01 x 10 44
0.10
0.03
0.20
0.26
0.35
0,02
WGSC-3 0.12 0.01 9.30 x 10-4
0.29
0.01
0.06
0.36
0.18
0.69
0.05
4 Concluding Remarks
In this paper, we introduce Zipper, a simple yet effective device designed to address the issue of degeneracy in algoritmm/modellaanosiii inference. The mechanism of Zipper involves the recycling of data usage by constructing two overlapping data splits within the testing samples, which holds potential for independent exploration. A key component of Zipper is the slider parameter, which tradeoff. To ensure reliable inference, we propose a simple selection criterion by ensuring large sample size to render asymptotic normality under the null hypothesis. Other ddta-tda strategies are possible and merit further investigation. Moreover, the wwwpeeesesssccsssoooooooooo based framework allows for the utilization of alternative forms of two- sample tests, such as rank-based methods. This capability proves beneficial when dealing with data exhibiting heavy-tailed distributions outliers. Furthermore, incorporating the Zipper device into large-scale comparisons to achieve error rate control warrants additional research. We can conduct a sequence of variable importance tests, each aimed at assessing the relevance of specific variable X, in the predictive model while controlling for a global error rate. This procedure necessitates the fitting of p+ 1 models: one that includes all variables and p null models, each excluding distinct variable. Such process is computationally demanding. Moreover, accurately controlling error rates presents considerable challenge due to complex dependency structures among the p-values.
introduces an seficiincy-anddeeeneeen
Acknowledgements
We would like to acknowledge the anonymous area chair and reviewers for their valuable comments and suggestions. Zou was supported by the National Key R&D Program of China (Grant Nos. 2022YFA1003703 2022YFA1003800) and the National Natural Science Foundation of China (Grant Nos. 11925106, 12231011, 1991101, 12226007, 12326325). Wang was supported by the National Natural Science Foundation of China (Grant No. 12471255) and the Natural Science Foundation of Shanghai (Grant No. 23ZR1419400).
References
[1] Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J. Tibshirani, and Larry Wasserman. Distribution-fies free predictive inference for regression. Journal of the American Statistical Associ- ation, 1136523):0094-11111 2018.
[2] Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duñlo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1):C1-C68, 01 2018.
[3] Aaron Fisher, Cynthia Rudin, and Francesca Dominici. All models wrong, but many useful: Learning a variable's importance by studying an entire class of prediction models simultaneously. Journal of Machine Learning Research, 20(177):1-81, 2019.
[4] Zhanrui Cai, Jing Lei, and Kathryn Roeder. Model-free prediction test with application to genomics data. Proceedings of the National Academy of Sciences. 119(34)ee2205518119 2022.
[5] Brian D. Williamson, Peter B. Gilbert, Noah Simon, and Marco Carone. A general framework for inference on algorithm-agnostic variable importance. Journal of the American Statistical Association, 1185443):1645-1658 2023.
[6] Susanne Schennach and Daniel Wilhelm. A simple parametric model selection test. Journal of the American Statistical Association, 112(520):663-1674 2017.
10

---

[7] Alex Luedtke, Marco Carone, and Mark J. van der Laan. An omnibus non parametrai test of equality distribution for unknown functions. Journal of the Royal Statistical Society Series Statistical Methodology, (1(1):5--9) 2019.
[8] lan Covert, Scott Lundberg, and Su-In Lee. Explaining by removing: A unified framework for model explanation. Journal of Machine Learning Research, 22(209): 2021.
[9] Aaron Hudson. Nonparametric inference on non-negative dissimilarity measures at the boundary of the parameter space. arXiv: 2306.07492, 2023.
[10] Jianqing Fan, Chumming Zhang, and Jian Zhang. Generalized likelihood ratio statistics and Wilks phenomenon. The Annals of Statistics, 19(1):133-193, 2001.
[11] Jing Lei. volidtaali with confidence. Journal of the American Statistical Association, 115(532):1978-19)) 19978, 2020.
[12] Pierre Bayle, Alexandre Bayle, Lucas Janson, and Lester Mackey. Cross-validation confidence intervals for test error. In Advances Neural Information Processing Systems, volume 33, pages 16339-16350. Curran Associates, Inc., 2020.
[13] Alessandro Rinaldo, Larry Wasserman, and Max G'lll.. Bootstrapping and sample splitting for high-dimensional, assumption-leat inference. The Annals of Statistics, 47(())3433- -4688,334,944 2019.
[14] Brian D. Williamson, Peter Gilbert, Marco Carone, and Noah Simon. Nonparametric variable importance assessment using machine learning techniques. Biometrics, 77(1):9-22, 2021.
[15] Lu Zhang and Lucas Janson. Floodgate: Inference for model-free variable importance. arXiv: 2007.01283v5, 2022.
[16] Yue Gao, Abby Stevens. Garvesh Raskutti, and Rebecca Willett. Lazy estimation of variable importance for large neural networks. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 1122-7143. PMLR, 2022.
[17] Rajen D. Shah and Jonas Peters. The hardness of conditional independence testing and the generalised covariance measure. The Annals of Stotistics, 48(3):1514-15388 2020.
[18] Anton Rask Lundborg, Ilmun Kim, Rajen D. Shah, and Richard Samworth. The Projected Covariance Measure for assumption-lean variable significance testing. 2022.
arXiv: 221 120009,
[19] Larry Wasserman, Aaditya Ramdas, and Sivaraman Balakrishnan. Universal inference. Pro- ceedings of the National Academy of Sciences, 11ress,,1111 117(29):1880-16699 2020.
[20] Victor Chernozhukov, Juan Carlos Escanciano, Hidehiko Ichimura, Whitney K. Newey, and James M. Robins. Locally robust semiparametric estimation. Econometrica, 1504):155 2022.
[21] Yuqian Zhang and Jelena Bradic. High-dimensional semi-supervised learning: in search of optimal inference of the mean. Biometrika, 109(2):387-40) 2022.
[22] Isabella Verdinelli and Larry Wasserman. Decorrelated variable importance. Journal of Machine Learning Research, 25(7):1-27, 2024.
[23] Ben Dai, Xiaotong Shen, and Wei Pan. Significance tests of feature relevance for a black-box learner. IEEE Transactions on Neural Networks and Learning Systems, 35(2):1898 1911, 2024.
[24] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Con- ference on Knowledge Discovery and Data Mining, KDD '1, pages 1135-1144. Association for Computing Machinery, 2016.
[25] Ian Covert, Scott M Lundberg, and Su-In Lee. Understanding global feature contributions with additive importance measures. In Advances Neural Information Processing Systems, volume 33, pages 17212-17223, Curran Associates, 2020.
11

---

[26] Ian Covert, Scott Lundberg, and Su-In Lee. Explaining by removing: A unified framework for model explanation. Journal of Machine Learning Research, 22(209):1-90 2021.
[27] I.
Elizabeth Kumar, Suresh Venkatasubramanian, Carlos Scheidegger, and Sorelle Friedler. Prob- lems with mhapley-value-baset explanations as feature importance measures. In Proceedings of the 37th International Conference on Machine Learning Proceedings of Machine Learning Research, pages 5491-5500. PMLR, 2020.
(28) Emmanuel Candès, Yingying Fan, Lucas Janson, and Jinchi Lv. Panning for gold: 'model-X' knockoff's for high dimensional controlled variable selection. Journal of the Royal Statistical Society, Series B. Statistical Methodology, 80(3):551-577, 2018.
[29] Wesley Tansey, Victor Veitch, Haoran Zhang, Raul Rabadan, and David M Blei. The holdout randomization test for feature selection in black box models. Journal of Computational and Graphical Statisties, 31(1):111-16,, 2022.
[30] Scott Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[31] Jin Zhu, Xueqin Wang, Liyuan Hu, Junhao Huang, Kangkang Jiang, Yanhang Zhang, Shiyun Lin, and Junxian Zhu. abess: A fast best-subset selection library in python and r. Journal of Machine Learning Research, 23(202):1 2022.
[32] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied document recognition. Proceedings of the IEFE, 86(11):222) (6(11))2278-2324 1998.
[33] Keith W Penrose, AG Nelson, and AG Fisher. Generalized body composition prediction equation for men using simple measurement techniques. Medicine Science in Sports & Exercise, 17(2):189, 1985.
[34] Yuetong Zhu Hitoshi Maruyama, Ko Onoda, Yue Zhou, Qiuchen Huang, Chunying Hu, Zhongqiu Ye, Bo Li, and Zimin Wang. Body mass index combined with (waist + hip//tighht accurately screened for normal- obesity in chinese young adults. Nutrition, 108: 111:1, 2023.
12

---

Appendix
A Conditions
Assume P E M for a rich class M of distributions. Define the linear space H = f P2) 2 0, Pi,P3 E Mj. For each h E H. let ||/||o sup, [F] - F2(z),. where Fj denotes the distribution function corresponding to P3 for j = 1,2. Consider J as the class of predictions functions endowed with norm || ||x. Let f arg maxjer maxjerC(f, and (f be sequence of estimators of) f. For each < ls S K, define Og, : + C(fk,ns P; P) C(S, P; P) and bk.n + C(fr,n Pe.ni d2 Pk.,) C(Ja,, P; P). The following conditions are imposed on the tuple (P, J,,
(CI) There exists some constant C > 0 such that, for each sequence fi, f2.... E F such that Iff fllr - 0, [C(f) P) - C(f, P)] < C'I - fl for each j large enough;
(C2) There exists some constant > such that for each sequence 61, ...... E R and h, h1, h2,... H satisfying that Ej + and l ho - 0, it holds that
C(f.P+ C(F.P) P:h,) + 0;
sup fer:1lf-fl
Ej
(C3) For each 1 k - K, 1k,n 81|5 =
(C4) For each 1 < k < K, ʃ oh =p=
(C5) For each 1 < & _ K, 67 =p
Condition (C1) pertains to the optimality of the prediction function f. eliminating first-order esti- mation biases. Condition (C2) requires Hadamard diffcrentiability of the predictiveness criterion. Condition (C3) demands accurate estimation of f, rendering second-order terms negligible. Condition (C4) controls the remainder terms. Condition (C5) ensures consistent variance estimators.
B Proofs
B.1 Proof of Theorem 2.1
Lemma B.1. If Conditions (C4) specified in Appendix hold for both tuples (P.F, FI, Jk,n and (P,Fs, fs fkm,ss), then, for each I fo, a, b) and {I....,K} K},
C(fkn Pk.m.) (.,m,, - C(F,P)
Σ
((Z)) + tor((DE,111) and
|Dx,ul t:Z,ED..I
C(famss C(fs, [Da, i.Z.ED.
Σ
es(Zi) oP((DR.11-1/))
Proof. See Theorem 2 in Williamson et al. 5
Denote my = |Dr,A| | = (Dr,Bl Since Da.al = |D..ll = (1 7 r)m and Dr = Tmk we have
my = n/((2 K} due to DKkal | + |D n/ K. By Lemma B.1 we have
CR, - C(f, = T(C(frmm Ph.oo) - C(f, (C Pkmmo) - C(f.
1
Σ ((Z)) 11/2)
m iZ..
13

---

Since the number of folds K is fixed, we have
Ck. C(J.P)
Σ
((Zx) + or(mm 1/2)
K
mk :ZZ
(5)
Σ
o(Z,) + op(n
mi EDz.
Similarly, we conclude that
K
K Ckn. - C(ss,
E
es(2,) + tpp(n
(6)
mik i.Z
k=1
Combining 5
and 66
we obtain
4nn.s -/s
K
1
Σ
s(Z))
E
ds((i) + Σ {o(Z)) - os(2)))
+ +op(n
n/(2- - k=1
i:ZiEDE.o
i:ZEEDE.
By applying the standard central limit theorem,
{(1 ((-T)mi (1-r)n/(2-7))) -))
((Z)) 4 N((, o²), and
K
T T)mk izZiEDka
k=1
((1 (1-TT)n/(2 (
Σ
es(2) 4 0(0,03))
K
m)
as + co. If ths > 0, we have
(Tn/(2 1/2
Σ
(o(Z.) os( 4 (O.
K
Tillk
i.Z,ED,oo
Hence,
(n/(2 -n/(2-))) ///2-)))1/2 188) 4. N(0.
where v3.r = (1-7)(o² + +z²) TT/.
B.2 Proof of Proposition 2.3
It suffices to show that oi 2 o² + 0o. Since
Ö(fkn, Ph - Pk.n) = Clfkm, Pr,ni - Pk.n) C(fk... p: - -P)
+ Ö(fkmn P:.-- - P) -CfC P:o -P)+ Č(f, P: -P),
14

---

we have
1
(C(sk,ns Pa,ni -P.,)) C(Su,n: P;Oz, p]}
7k
i:Z,ED2
Σ
(C(skm; P;dz; -P) -C) P;Oz; P]]2
Σ (C(f,P:8e - P)}
(7)
+2
Σ
{C(fRM: Pk òz. Pk.n)) C(f - -P) P)
+2
Σ
(C(fR, Pūôz. - -P) - P: P)J - P)
+2 Σ
(C(fk,m: Pk,ni 8z. -kk,)) - P;dz, P)}
x (C(fie,; P:òz. P) -P - p1]
For any >> 0, by Markov's inequality and Condition (C5),
Pr
Σ
(C(fk:n: PR.niOn, -PR,)) C(fk P;,z. P)}} 2 D_k
Thk
EZ,ED;
El( Pz.idz, Pk,n) C(Jk, P;ōz, PJ32 [D_A]
(be.n O.
Then, by the law of total expectation, the first term of 7 is (1()). Similarly, the second term is (1). Specifically, by Condition (C4), for any > O,
Pr
Σ
(Clfkm P;óz. - P)
- P)} 2 c D_
ThE
iZZ.ED
< EllC(fa., -
D_n]
I (2) o..
For the third term, by the law of large numbers,
Σ CUSPPóo. PP2 2 E(C(S, -P]] = o², BZZ.D:
TIE
00. For the fourth term, we have
1
Σ
(C(Jem Ph,ni - Ph,n) C(Jkin P;5z, 1: -P) P; P)
Σ (C(f..n Panidz, - Pk. - CifffinnPP... - P132
i.Z,ED,
1/2
x Tuk LIZE
Σ Č(f.P:oz. P)2 P
=o(().
Similarly, both of the last two terms are op (1). Hence, follows that of P o²
By similar arguments, o.n. & o3 and thus v2.s.- B. vs.
15

---

B.2.1 Verification of Condition (C5) in Remark 2.4
Suppose C(f, P) E.pla(Y Then
C(Ju,, Pk,n; - Pk.n) = g(v, [x,
Σ
Jkmi,
Th $:Z,ED.
C(fkm, P) gly fn,n( Eplo(Y, fam))))
By noting that
Ell((JEM; Pr,nide 02 Pk, C(JK, P;O= - P)}2 D_x]
12
=
g(Y,, St,))),
1 - Flg(Y,JE. |
D_
nik
t22,D2
VarlglY, fa (X)3 D_ 40
7k
the conclusion follows.
B.3 Proof of Theorem 2.6
By Theorem 2.1 for any T E [0, 1),
vs) 4 NO, v3,-) (2/(2- - ( 4
oc, where v/ (1 (1- 1 02 E(o?( d3 = E(d3 and n5
(Z( os (2)3 By examining the proof of Proposition 2.3 conclude that Vn,SSr -g S. under H1. Consequently, the power function is
(0)
Pr(T, > 21 Hi)
(0) (n/(2 T)) 18))
(n/(2- 1)31/2/55/ us
S.r
V.S.T 21-a US,T
Pr
HI
Un,ST
VS.T
Un,S,rUS.T
VSS.T 21-a +
(n/(2- - -T)]1/21b
+
+ o(1)
VSSt
1/S.T
=ss (T) + oL).
Given that 1/s = x² + o's (C)o(o()),() (s())) the asymptotic variance v²,r can be expressed as o² + 03 2TCool os Let
(1 (o² 033)
p)(T) =
σ² - -o os)) and
(n/ - 1/
(2(() = (02 + + (=Coo(((Z)
Since 2Coo ( (Z), Øs Z) < 0² o3 and E 10, it follows that (P) and (p2(T) > 0.
When Cov (ooo(o(Z). ()), Øs (2)) 2 0, we observe that (T) is monotonically decreasing, and (2((T) is
monotonically increasing with respect 7. Therefore, the approximate power function G Gs.nn (7) increase with T.
B.3.1 On Remark 2.8
Here we demonstrate the validity of the condition Cou(cb (2)((())))))))) (Z)) > assessing variable
importance when utilizing the square loss. Consider the scenario where Y = E(Y X)+ with the noise e assumed be independent the covariate X. We seek evaluate the significance of specific set of covariates U in predicting the response, where X (UT.VT) Assuming
16

---

B(Y | X) E F and E(Y V) E Fs. the optimal prediction functions within F and Fs are f(X) = E(Y (X) and fs(V) VV), respectively.
Referring to Remark 2.4 the covariance (oo(((2)) ( (2)) can be expressed as
Covld(2)00s(2) = Coel(Y - 1 [(X)) ( [s([}}
= -Var(=") + CoufE -
- fs())) + 2Coufe ((f(() - fs(V)))
It's evident that the first two terms are non-negative. Regarding the third term, since is independent of X and ( V = fv(),, we deduce
-fs(V) (1e-((A)--1S1))))-EEEEEEEEE fs(V))
=E())) - - -
= ((E((())) - fs(V) VA
O.
Thus, Cov(o(Z), oz) 2 follows.
C The Zipper Algorithm
Algorithm The algorithm for the proposed Zipper testing procedure
Input:
Observed data [ number of folds K, and slider parameter T E [0, 1)
Randomly partition {2, ) into K disjoint folds 11....,Dx
for k = =1 K do
Using data in {D,} \ Da. construct estimators fa.n of f and fe,n, of fs Using data in DR. construct estimator PR.n. of P Randomly divide Dr into two splits DA.A and Da.B of roughly equal size meg with an overlap
DR.o such that iD [rma]- Let DR. a D (Droo and Dx.6 = DD. D Using data in DR.I. construct estimators Pk for I a.
Compute Ckn = +((fk, Pk.no)
(1 c]C(f Pkinaa) and on
n EizEe,, (C(fa.n Penidzz - 22 where n& = [Dal Compute Ck.n.S
TC(J.n,ss Pk. + T)D(JRnns Pkond) and ok.. n EiZED, (C(fk,n Pk.ni 5% p2,n))
end for
--1EK Ck,n.s and estimator this Cn Cr.s Compute Cn K- Ex=1 Ckin Cn,s K- Ek=1 of vss
Compute v2 S.S.T (1 - T- -1 Ek=1 (oh + oknns)
Output: P-value - ((nn(2 1/2 //.s /n.. where & denotes the distribution function of (0,
D The Confidence Interval for the Dissimilarity Measure
To construct a valid confidence interval for the dissimilarity measure Vss it is crucial to use consistent variance estimator irrespective of whether Hg holds not. This can be achieved by incorporating an additional plug-in estimator of nīg, which is component of the asymptotic variance v3,T. Let vi,s.r K-1 E ((1 r) +on.ns) + Th....s. where for each k E {I,...,
1
Σ (C(fk,, Pr. PP.n) C(fk.n,s. Pa Pk.n))
Tik
The confidence interval for ths can be formulated
Sn, [n/(2 1/
2n,
Un.s
(2-
which has asymptotic coverage of
17

---

Table 4: Empirical sizes (in percentage) of the Zipper method against different values of 7 with n 500.
Model
P
0
0.2
0.4
0.6
0.8
0.9
0.95
0.99
5
4.6
3.2
3.2
4.4
4.9
4.5
3.7
2.2
Normal
1000
5.9
5.9
5.8
4.2
5.6
7.4
8.2
11.1
5
4.2
3.3
2.4
2.2
3.6
3.1
3.0
3.5
Binomial
1000
5.1
5.7
4.2
5.1
5.6
9.5
11.7
20.9
Table 5: Empirical power (in percentage) of the Zipper method against different values of T with 500. For the normal model,
5 2 for the low-dimensional setting and ō 5 for the high-dimensional setting. For the Binomial model, we set for the low-dimensional setting and
ó 5 for the high-dimensional setting.
T
Model
p
0.2
0.4
0.6
0.8
0.9
5
46.9
54.8
72.0
86.4
98.5
100.0
Normal
1000
87.6
91.0
97.7
99.9
100.0
100.0
89.8
94.7
98.4
99.4
99.9
100.0
Binomial
1000
78.6
85.0
92.7
98.2
99.7
100.0
E
Additional Simulations
E.1
On the Slider Parameter
To assess the influence the slider parameter on the size and power of the proposed Zipper approach, we conduct a small simulation study using the simulation settings outlined in the manuscript. The results of this study summarized in Tables + Notably, observe that the empirical sizes consistently controlled when r falls within appropriate range. However, selecting excessively large values of led slight inflation in size, primarily due to the poor normal approximation under the null hypothesis, as discussed in Section2..5 Furthermore, the empirical power increases as the value of increases, which aligns with our expectations.
E.2 Additional Simulation Experiments
To provide a comprehensive analysis of the empirical sizes and power across various settings, we conducted additional simulations using the simulation settings outlined in the manuscript. We explored different combinations of the sample size n and dimension p. Moreover, we investigated a model with heavy-tailed response by considering Y X3 oye/3, E la, where la represents the !-distribution with 3 degrees of freedom. The results for empirical sizes and power with 5 = 1 summarized in Tables 6[and These additional findings consistently support the conclusion that the Zipper method demonstrates reliable empirical size performance and provides significant power enhancement compared to methods that utilize non-overlapping splits.
F
Comparison between Zipper and Data Perturbation Methods
To evaluate variables with cero-importance. Rinaldo et al. [33 and Dai et al. (33 proposed a data perturbation technique, injecting independent zero-mean noises into empirical influence functions. We describe this method using our own terminology. We begin by partitioning the data randomly into K folds, denoted Di...., Dx, ensuring equitable sizing across folds. For each fold index E (1,..., K). we construct estimators fk.n and fk.n,s for the oracle prediction functions f and fs- correspondingly, utilizing data excluding fold Dj. This method employs the following test statistic:
Sn,S,pert.p
Σ n
Σ iZZ.DD
((Z.) os(Z.) - -Ei
18

---

Table 6: Empirical sizes (in percentage) of various testing procedures.
Model
Zipper
WGSC-3
DSP-Split
WGSC-2
DSP-PE Pert
200
3.6
5.0
3.7
0.4
12.5
500
2.1
4.2
4.7
0.0
9.8
1000
3.0
0.3
10.7
200
3.2
4.3
3.9
1.2
8.9
Normal
500
10
3.7
4.5
4.9
0.3
9.7
1000
2.5
4.0
4.6
0.3
9.6
500
4.7
6.3
4.8
17.1
24.6
200
1000
5.2
5.1
13.6
25.9
1000
1000
4.4
6.3
3,9
15.8
36.6
200
3.5
3,4
3.7
0.3
11.0
500
4.3
4.2
3.8
0.3
8.5
1000
4.3
3.7
4.6
0.3
8.1
200
2.4
1.3
8.5
ts
500
0.0
8.4
1000
4.3
4.0
3.6
0.2
8.6
500
4.9
3.6
5.0
15,4
24.7
200
1000
5.4
3,9
5.3
12.1
27.8
1000
6.4
3.5
34.0
200
3.7
4.9
3.4
0.4
5.9
500
3.3
4,5
3.9
0.2
4.6
1000
2.8
4.1
3.7
0.2
4.4
200
3.1
6.4
3.1
2.0
6.5
Binomial
500
10
2.8
5.2
4.1
0.6
5.0
1000
3.2
0.7
4.1
500
7.0
3.2
6.3
15.2
27.9
200
1000
6.3
3,8
6.0
16.4
24.9
1000
1000
6.3
4.2
5.4
16.9
32.2
Table 7: Empirical power (in percentage) of various testing procedures for
Model
Normal WGSC-3
ts WGSC-3
Binomial WGSC-3
P
Zipper
DSP-Split
Zipper
DSP-Split
Zipper
DSP-Split
200
7.7
6.7
14.0
6.9
6.4
500
5
44.1
10.5
9.6
66.3
17.6
16.4
49.0
12.4
11.5
1000
86.8
13.9
14.9
89.0
21.1
18.4
91.7
20.6
19.6
200
10.3
7.6
7.9
28.6
11.4
13.9
3.7
7.4
2.5
500
10
42.0
11.2
10.4
68.7
18.4
18.8
36.2
12.7
10.3
1000
87.9
15.7
11.6
87.6
18.4
20.5
87.9
19.5
16.0
500
63.6
16.7
14.2
87.0
34.9
32.8
79.2
24.9
23.0
200
1000
96.5
27.3
27.0
93.2
42.1
42.8
99.1
49.1
44.8
1000
1000
10.4
7.7
5.6
22.5
6.2
7.8
22.0
6.7
8.4
where E, tion parameter governing the extent of perturbation. It be established that for any P
N(0, 1) for = 1,...,7 are independent noise, and represents the perturba-
n 1/2 1/2, peettp vs)) 4 N pertt,,))) as - oo, where v V/S,pert,p = n/ + p², and S Elfo(Z) 2s. Following the plug-in principle, employ the normalized test statistic
Tpert.p = 2/ "2./"on.S.pertt.. Un,S. Un.S.pert.p.
which converges in distribution to N N(0, 1) under Ho for any P 0. Here
V2. pert,, K-1 E mhn. p² where for each E fl...., K). n Yı -1-1 ".EEE,, [C(Sk., PR,m; òz Pk.n) Č(Sk,,,s Pk,ni Ōz 22.))) For prespecified significance level (0, 1), this method rejects Hy if T pertt, > 21-a-
19

---

Indeed, a direct correspondence emerges between the slider parameter 7 within our Zipper and the perturbation parameter o employed in the data perturbation technique, thereby facilitating comparative analysis of the two methodologies. To elucidate this connection, we demonstrate that, under H1 vhs 0, the power function corresponding the data perturbation method
Pr(Tpert pert,p 21 H1)
1/2 (20. S... vs))
n 1//
Pr
21-0
H1
VS. perpert,p
V/. perte
n n//2//
+
11-a +
o(1)
US.pert.p/
p.pt (p) + o(1)
First notice that our Zipper method with T = (i.c., the vanilla sample-splitting) has approximate power
Gs,n,a (-21-a+ (2/2) (o2 + 1/2
We establish an upper bound for the perturbation parameter p. namely, p² < 2(02 +03) equality yields na. (p) Gs (0). Should p² > 2( 03) V it becomes evident that Gs.m.o.pert(0 (p) & Gs (0), suggesting that the data perturbation technique may even exhibit lower power.
where
The idea to establish a relationship between 7 and p such that both methods yield similar power, given their valid size control for fixed and p. Employing consistent variance estimator as per Remark 2.9 our Zipper method exhibits an approximate power of
(n/(2 -))1/²(b3
(CD) G. (T) + (-21-a +
VS.T
By setting (CI) G,
(T) Gs (p), obtain
p² = (2- (2-
- () + - 73
(8)
establishing a one-to-one correspondence. Under this correspondence,
(CT) Gs (p) S, (T) Gs,n.o
where the latter denotes the approximate power of our Zipper method utilizing the proposed variance estimation scheme. Thus, our Zipper method can be regarded as a data- data-adaptive perturbation strategy that circumvents external randomization while potentially offering power enhancement owing the variance estimation scheme.
Figure 4 lllustrates the empirical power comparison between Zipper and the data perturbation method, where the perturbation parameter p = p(T) is determined by (8), across various values r. This assessment is conducted according to the low-dimensional normal response scenario outlined in Section 3 particularly with p = 5 and n = 500. As anticipated, Zipper and the perturbation method demonstrate the same power when O. while for E (0, 1), Zipper consistently exhibits superior power compared to the perturbation method.
20

---

8 =
0.8
1.00
0.50
0.25
Rejection Rate
0.00
8
1.6 1.6
8 2.4
1.00
1/
0.50
0.00
0.75
1.000.00
0.25
0.50
Method
Zipper
Perturbation
Figure Empirical size and power comparison of Zipper and data perturbation method function of r. The dot-dashed horizontal line represents the intercept at = 5%.
21

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper's contributions and scope are specifically claimed in the abstract
and introduction, and a single subsection 1.4 fully summarizes our contributions briefly.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss our limitations and further developments in Section 4
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Justification:
We provide our assumptions in Section A and complete and correct proof in Section in appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We fully describe our settings in synthetic experiments and retl-data examples in Section 3 and the information provided is enough to the reproducibility of our main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
23

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We concretely describe our Zipper algorithm in Section C and our sythetic data generating procedure in Section The datasets utilized in real-data analysis are soureed from publicly available materials on the internet. We think they are sufficient to reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.. cc/
cablic/ggides/CCee Coo.eSSuiisiiiooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.c.. cttpppccccccccccccccccccccccclct guwdee/ wwt.ooouuuiiiiinnoooiiioieeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all our training details in Section 3 And we discuss the choice of the hyperparameter 7 Section 2.5
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our work purpose on statistical inference on algorithm/model agnostic goodness-of-fr tests. The experiments conducted in our paper all focus on the type-I or type-II eттоr of statistical tests, and can be dual construct confidence interval in applica- tion. We further represent standard deviation upon repetitions of our synthetic experiments in Table 3.11 and Table 2 and error bars in Figure 2
Guidelines:
The answer NA means that the paper does not include experiments.

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
The computer resources used are briefly provided in Section And the time of execution of our method depend on the algorithm/model chosen fitting the model, varies among different application scenarios.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't it make it into the paper).
paper (e.g., preliminary or failed experiments that NeurlPS Code of Ethics https: http://p.....p..b...c.cc.caeeeea
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the
Answer: [Yes]
Justification: We briefly reviewed of the NeurIPS Code of Ethics, and make sure that our research conducted in the paper conform with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
25

---

Our work conducts general framework of algorithn/model agnostic inference on goodness-of in regression. This is fundamental research on statistical methodology and has no direct societal impacts as far as we currently understand.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The datasets used in real-data examples are all open-source datasets, and we properly cite the original paper that produced the dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
26

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperssi thoode com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
27

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "bKdalPBBA8", "reviewer": "Reviewer_aa86", "summary": "This work presents a test of Performance(f, test_data_1) - Performance(f_subset, test_data_2) where f is the best model in a class F and f_subset is the best model in a subset of F. They focus on how precisely to split your samples between test_data_1 and test_data_2 to avoid degeneracies that can arise when asking questions like whether the performance difference is equal to zero.\n\nThe main proposed solution is to take an underlying set of test data D_k, split it into two possibly overlapping pieces D_A and D_B, with overlap D_o, and to compute Performance(f, test_data_1) as a combination of performance on D_o and D_A \\ D_o, and likewise to compute Performance(f_subset, test_data_2) as a combination of performance on D_o and D_B \\ D_o.\n\nFor example, Performance(f, test_data_1) = \\tau performance(f, D_o) + (1-\\tau) performance(f, D_A \\ D_o) . Here picking \\tau 0<=\\tau<1 avoids the degeneracy.  This idea can be seen as executing a particularly careful and favorable property of sample splitting.\n\nThey show that the test statistic, under the null hypothesis of no difference, is asymptotically linear. They also derive a consistent estimator for the variance of the statistic under the null.  They also provide selection criteria for the zipper \\tau.\n\nUsing these tools they analyze the power/size of the test statistic and run simulations in variable importance, model specification, and more.\n\nAs future work, they leave open some ideas around which other hyperparameters to select data-adaptively, how the test works on large scale data, and how to control errors.", "questions": "Nothing major at the moment, may revise before discussion period.", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- broadly applicable scenario (model/algorithm agnostic inference)\n- solves a known issue of a popular test statistic under the null hypothesis of 0 performance difference\n- does so with minimal intervention to model fitting / testing (just split your data in a certain way, evaluate on carefully defined subsets, and combine)\n- offers guidance on a few hypeparameters\n- provides results including on estimating the variance of the test statistic.\n- suggests fruitful directions for future work.", "weaknesses": "Nothing major at the moment, may revise before discussion period.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our sincere gratitude for your thoughtful and detailed review of our paper. We greatly appreciate your positive evaluation and constructive feedback. We are committed to further refining our manuscript and eagerly welcome any additional comments or questions you may have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mt3AAoRCmn", "reviewer": "Reviewer_2sMb", "summary": "The authors propose a new test statistic they call Zipper for algorithm-agnostic inference on goodness-of-fit testing. While previous solutions suffer from a degeneracy issue (i.e., fails to converge to a non-degenerate distribution under the null hypothesis of equal goodness), the proposed test statistic does not (i.e., it converges asymptotically to a normal distribution under the null). It also improves power over other proposed solutions which also tackle the degeneracy issue (due to Zipper’s effective reuse of data). Zipper could have applications in specification testing, model selection and variable importance assessments.", "questions": "1. Lines 231-236 suggest that it is not only the zipper mechanism that enables power improvements, but also the use of a variance estimator. Could the authors expand on the latter, e.g. provide more intuition, incorporate it into the overall story in the paper outside of these few lines?\n2. When would one prefer this approach to others in, say, variable importance testing (e.g. conditional randomization tests, Shapley values) and why? I understand that the precise goal is slightly different between the options mentioned, but as this work falls under the interpretability / explainability category, it would be useful to situate it explicitly.\n3. Could the authors combine the results of Table 1 and Figure 2 to help visualize how Zipper compares holistically to the other methods?\n4. Could the authors provide both power and size results in Table 2?\n5. Could the authors explain why Figure 3 shows that the proposed method is preferred over the baseline?", "limitations": "The applicability of the method for larger scale settings has not yet been tested. Moreover, for an application such as explanability, GoF testing may not necessarily be the most direct given the test merely tests how well a pre-specified subset performs relative to the full set, rather than locating an important subset.\n\n(As a quick aside, I noticed in the checklist that the authors mention discussing limitations in Section 4, but I do not seem to see any mentions of limitations in that section)", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is very well-written and the motivation is clear.\n2. The benefit of the proposed test statistic over predecessors is clear from the theoretical analysis.\n3. The algorithm for Zipper is also quite straightforward, lending to its practicality.", "weaknesses": "1. The paper could benefit from being more self-contained. For instance:\n- The theorems reference conditions in the appendix. While I understand why it makes sense to place such details in the appendix, some informal discussion of the conditions in the main paper would be helpful for exposition.\n- The paper consistently references WIlliamson et al. without precisely describing what is proposed in the work (beyond the a high-line mention of “Williamson et al. proposed an additional split of the testing data” in lines 97-98). Adding more discussion up front would help with readability / appreciating the paper’s contribution during the analysis.\n2. The paper mentions that this test statistic / goodness-of-fit testing broadly is applicable in many settings but does not compare what is proposed with other methods which are also used in these settings (e.g., other measures of variable importance). It would help to add this discussion to the related work. \n3. The numerical experiments / results could be more comprehensive. Specifically:\n- The results for the numerical experiments do not make it easy to see that Zipper is preferred over the other methods; for instance, Table 1 shows that Zipper has better size than WSGC-1 and DSP-Pert, but Figure 2 shows that the latter have better power.\n- Table 2 is confusing given the caption mentions both size and power but I only see one number for each method. I think based on the text and inferring from the magnitudes that the first row is size while the second and third row are power, but why not include both for all scenarios? \n- Based on Figure 3, it is not obvious to me that the left column (Zipper) is better than the right column (baseline). Could the authors elaborate?\n- The text in 3.2.2 makes it more clear what might be preferable about Zipper, but it is not clear based on Table 3 alone. Perhaps some way of directly showcasing the alignment with a previous study would be helpful?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and constructive feedback on our paper. Your insights are invaluable for refining our manuscript.\n\n**Weaknesses:**\n\n1. _Self-contained discussions:_\n\n- _On conditions:_ Condition (C1) pertains to the optimality of the prediction function $f$, eliminating first-order estimation biases. Condition (C2) requires Hadamard differentiability of the predictiveness criterion. Condition (C3) demands accurate estimation of $f$, rendering second-order terms negligible. Condition (C4) controls the remainder terms. Condition (C5) ensures consistent variance estimators. We will add a detailed discussion of these conditions in the main text to improve readability.\n\n- _On Williamson et al.:_ Williamson et al. introduced a framework for algorithm-agnostic variable importance assessments using cross-fitting and demonstrated the asymptotic linearity of the test statistic. They also advocated additional splitting of the test data for null importance testing. We extend this framework to accommodate a wider range of goodness-of-fit tests, including specification testing and model selection. Our method, Zipper, promotes data reuse, significantly augmenting the testing power. We will elaborate on these points in the Related Work section to strengthen narrative coherence.\n\n2. _Related work:_ For variable importance assessments, in addition to LOCO methods within our framework, Shapley value-based measures are commonly used (Lundberg and Lee, 2017; Covert et al., 2020, 2021; Kumar et al., 2020). These measures, which estimate the incremental predictive accuracy contributed by a specific variable across all possible covariate subsets, reveal complex inter-variable relationships but at a considerable computational expense. Furthermore, conditional randomization tests (Candes et al., 2018; Tansey et al., 2021) offer a robust alternative when covariate distributions are known or can be accurately estimated. These methods are especially beneficial in semi-supervised settings with extensive unlabeled data. Additional methods, such as LIME (Ribeiro et al., 2016) and Floodgate (Zhang and Janson, 2020), will be discussed to expand the comprehensiveness of our manuscript's Related Work section.\n\n3. _Comprehensiveness of numerical experiments:_\n\n- _Preference of Zipper in Figure 2:_ Despite WSGC-2 and DSP-Pert showing superior power in certain settings ($p=1000$), both exhibit considerable size inflation-approximately 0.2 and 0.4 respectively, where the nominal level is 0.05. This size distortion, depicted at $\\delta=0$ in Figure 2, renders these methods less reliable for practical application. Conversely, Zipper maintains robust size control and achieves high power, affirming its utility.\n\n- _Size and power in Table 2:_ To clarify, Scenario (i) represents a specific case of $H_0$, while Scenarios (ii) and (iii) exemplify $H_1$ instances. We will rename Scenario (i) to $H_0^{'}$ and Scenarios (ii) and (iii) to $H_1^{'}$ and $H_1^{''}$, respectively.\n\n- _Improvement over baseline in Figure 3:_ Both Zipper and the baseline method (WGSC-3) demonstrate effective size control. In the MNIST handwriting dataset example, we perform sequential variable importance tests to determine the relevance of each region for prediction. We apply Bonferroni corrections to address multiple testing concerns. Figure 3 indicates that Zipper identifies more significant regions (five regions filled in red) compared to the baseline (two regions).\n\n- _Alignment with a previous study in Table 3:_ In our analysis of the Bodyfat dataset, we perform multiple tests with a Bonferroni correction set at $\\alpha=0.05/10=0.005$. Table 3 reveals that Zipper successfully identifies both the Abdomen and Hip as significant variables (both $\\le 0.005$), whereas WGSC-3 identifies only the Abdomen. Notice that the detection of the Hip variable corroborates results from previous research. We will highlight these outcomes in the revised manuscript.\n\n**Questions**:\n\n1. _Variance estimator:_ The improvement in testing power is attributed to both the overlapping mechanism of Zipper and the variance estimator. This estimator is consistent under $H_0$, thereby ensuring valid size control. Under $H_1$, it underestimates the actual unknown variance, which is advantageous for power enhancement as demonstrated. However, for confidence interval construction-e dual problem of hypothesis testing-a consistent variance estimator under both $H_0$ and $H_1$ is necessary. We provide such an estimator. See Remark 2.9 and Section D.\n\n2. _Preference of Zipper in variable importance testing:_ This is addressed in our response to Weaknesses 2.\n\n3. _Combination of results of Table 1 and Figure 2:_ Please refer to our response to Weaknesses 3, Point 1.\n\n4. _Power and Size in Table 2:_ This is explained in our response to Weaknesses 3, Point 2.\n\n5. _Explanation of Figure 3:_ Please see our response to Weaknesses 3, Point 3.\n\n**Limitations** in _larger scale settings and locating important subsets_: We acknowledge and value this critical feedback. Zipper is specifically designed to evaluate the significance of individual features or subsets in predictions, which paves the way for large-scale comparisons as mentioned at the end of Section 4. Notably, we can conduct a sequence of variable importance tests, each aimed at assessing the relevance of a specific variable $X_j$ in the predictive model while controlling for a global error rate. This procedure necessitates the fitting of $p+1$ models: one that includes all variables and $p$ null models, each excluding a distinct variable. Such a process is computationally demanding. Moreover, accurately controlling error rates presents a considerable challenge due to complex dependency structures among the p-values. We will elucidate this point further in the revised manuscript.\n\nWe appreciate the opportunity to enhance our paper based on your feedback and look forward to further discussions.\n\n####", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QSZ4fNeTyq", "reviewer": "Reviewer_DjJu", "summary": "This paper proposes a method for quantifying model-agnostic goodness of fit to allow better comparison between two models / model classes etc. In particular, this paper deals with the problem of degeneracy under the null hypothesis of equal goodness. Prior work has addressed this by splitting the test set into distinct subsets, however, this reduces the sample size significantly. To overcome this, Zipper splits the test set into overlapping test sets and uses the proportion of overlap of the splits to better aggregate the test statistic. This way, Zipper is able to more effectively use limited test data.", "questions": "N/A", "limitations": "Novelty / Contribution.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method's idea seems sound and useful. Test data is indeed limited in practice and a method like Zipper can enable GoF evaluation with limited test data.", "weaknesses": "1. I'm not an expert in this area, but I'm concerned about the novelty and significance of this contribution. I would urge the authors to further clarify the significance of this change in test data splitting (especially for other expert reviewers).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our sincere gratitude for your dedicated time and thoughtful review of our paper. We would like to further elucidate the novelty and significance of our contribution as follows:\n\n**Broad applicability and compatibility with flexible training techniques:** Our proposed Zipper device is designed for wide-ranging applicability across various goodness-of-fit testing scenarios, including variable importance assessment, specification testing, and model selection. Notably, this device necessitates minimal intervention in the model training process, enhancing its practicality.\n\n**Efficient utilization of test data:** Unlike the approach by Williamson et al., which divides the test data into two non-overlapping parts (thereby halving the sample size for testing), our overlap scheme in the Zipper device allows for a more efficient use of test data. This results in substantial power improvements while maintaining valid size control, as demonstrated in our theoretical results and finite-sample experiments across various contexts.\n\nWe hope these points help to clarify the novelty and significance of our work. We welcome any further comments or questions you may have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper Addresses degeneracy issues in algorithm/model-agnostic goodness-of-fit inference such as assessing variable importance.
It broadly applicable scenario (model/algorithm agnostic inference).
Also, it does so with minimal intervention to model fitting / testing (just split your data in a certain way, evaluate on carefully defined subsets, and combine).
The authors clearly addressed all questions by the reviewers during the rebuttal period please update the paper with the clarifications discussed.
Decision: Accept (spotlight)