Statistical Test for Feature Selection Pipelines by Selective Inference
Authors: Tomohiro Shiraishi
Tatsuya Matsukawa
Shuichi Nishino
Ichiro Takeuchi
Venue: ICML 2025 oral (2025)
Keywords: Data Analysis Pipeline
AutoML
Statistical Test
Selective Inference
Missing Value Imputation
Outlier Detection
Feature Selection
Primary area: theory->probabilistic_methods
TL;DR: We introduce a statistical test for data analysis pipeline in feature selection problems, which allows for the systematic development of valid statistical tests applicable to any pipeline configuration composed of a set of predefined components.

Abstract: A data analysis pipeline is a structured sequence of steps that transforms raw data into meaningful insights by integrating various analysis algorithms. In this paper, we propose a novel statistical test to assess the significance of data analysis pipelines. Our approach enables the systematic development of valid statistical tests applicable to any feature selection pipeline composed of predefined components. We develop this framework based on selective inference, a statistical technique that has recently gained attention for data-driven hypotheses. As a proof of concept, we focus on feature selection pipelines for linear models, composed of three missing value imputation algorithms, three outlier detection algorithms, and three feature selection algorithms. We theoretically prove that our statistical test can control the probability of false positive feature selection at any desired level, and demonstrate its validity and effectiveness through experiments on synthetic and real data. Additionally, we present an implementation framework that facilitates testing across any configuration of these feature selection pipelines without extra implementation costs.

Statistical Test for Feature Selection Pipelines by Selective Inference
Tomohiro Shiraishi 12
Tatsuya Matsukawa
Shuichi Nishino" 12 Ichiro Takeuchi 12
Abstract
useful insights is known as a data analysis pipeline, which plays key role in ensuring the reproducibility and reliabil- ity of data-driven decision-makinn
A data analysis pipeline is structured sequence of steps that transforms raw data into meaning- ful insights by integrating various analysis algo- rithms. In this paper, we propose a novel statis- tical test to assess the significance of data analy- sis pipelines. Our approach enables the system- atic development of valid statistical tests applica- ble to any feature selection pipeline composed of predefined components. We develop this frame- work based on selective inference, a statistical technique that has recently gained attention for data-driven hypotheses. As a proof of concept. we focus on feature selection pipelines for linear models, composed of three missing value imputa- tion algorithms,, threc outlier detection algorithms, and three feature selection algorithms. We theoret- ically prove that our statistical test can control the probability of false positive feature selection at any desired level, and demonstrate its validity and effectiveness through experiments on synthetic and real data. Additionally, we present an im- plementation framework that facilitates testing across any configuration of these feature selection pipelines without extra implementation costs.
In this study, as an example of data analysis pipelines, we consider a class of feature selection pipelines that integrates various missing-valus imputations (MVI) algorithms, out- lier detection (OD) algorithms, and feature selection (FS) algorithms. Figure shows examples of two such pipelines. The pipeline on the left starts with a mean value imputation algorithm, followed by Lı regression based OD algorithm, proceeds with marginal screening to refine feature candi- dates, and concludes by using two F$ searithmss-teppwee feature selection and Lasso-selecting their union as the final features. The pipeline on the right initiates with re- gression imputation, continues with marginal screening to narrow down feature candidates, uses Cook's distance for OD, and applies both stepwise FS and Lasso, ultimately choosing the intersection of their results as the final features.
When a data-driven approach is used for high-stakes decision-making tasks such as medical diagnosis, it is cru- cial to quantify the reliability of the final results by con- sidering all steps in the pipeline. The goal of this study is to develop statistical test for a specific class of feature selection pipelines in linear models, allowing the statistical significance of features obtained through the pipeline to be properly quantified in the form of p-values. The first techni- cal challenge in achieving this is the need to appropriately account for the complex interrelations between pipeline components to determine the overall statistical significance. The second challenge is to develop a universal framework capable of performing statistical tests on arbitrary pipelines (within a given class) rather than creating individual tests for each pipeline.
1. Introduction
In practical data-driven decision-making tasks, integrating various types of data analysis steps is crucial for address- ing diverse challenges. For instance, in genetic research aimed at identifying genes linked to a specific disease, the process often begins with preprocessing tasks such as filling in missing values and detecting outliers. This followed by screeming for potentially related genes using simple de- scriptive statistics and then applying more complex machine learning- based feature selection algorithms. Such system- atic sequence of steps designed to analyze data and derive
To address these challenges, we introduce the concept of selective inference (SI) (Taylor & Tibshirani, 2015; Fithian et al., 2015; Lee & Taylor, 2014), novel statistical infer- ence approach that has gained significant attention over the past decade. The core idea of SI is to characterize the pro- cess of selecting hypotheses from the data and calculate the corresponding p-values using the sampling distribution, con- ditional on this selection process. We propose an approach based on SI that provides valid p-values for any feature selec-
'Equal contribution 'Nagoya University, Aichi, Japan PRIKEN, Tokyo, Japan. Correspondence to: Ichiro Takeuchi makkuuch..chiro.nde uag.l...ggy...........
Proceedings of the nnnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Statistical Test for Feature Selection Pipelines by Selective Inference
Example 1
Example
5:FS lasso
lasso
1:MVI
2:00 boll-podi
4:F5 manginal screaning
6:U-F features
1:MVI regresiion
2:F5 scrvering
4:00 cook ditance
6:1-F
intersaction liuturas
4:F5 stepaise
4:F5 stepaske
Figure 1. Two examples of pipelines within the class considered in this study,
tion pipeline configuration within the aforementioned class. We also introduce modular implementation framework that supports SI for any pipeline configuration within this class without requiring additional implementation efforts. Specifically, with our framework, the statistical significance of features from any pipeline in this class can be quantified as valid p-values when used in linear model, with no extra implementation required beyond specifying the pipeline.
Rügamer et al., 2022). SI is valuable not only for FS in linear models but also for inference across various data- driven hypotheses, including tasks like OD (Chen & Bien, 2020; Tsukurimichi et al.. 2022), segmentation (Tanizaki et al., 2020; Duy et al., 2022; Le Duy et al., 2024), cluster- ing (Lee et al., 2015; Gao et al., 2022), and change-point detection (Duy et al., 2020; Jewell et al., 2022). The core idea of SI is to perform statistical inference using distri- bution conditioned on events of hypothesis selection, with the technical challenge being the characterization of various event selections for different tasks. While studies on SI for various tasks are being conducted, existing research is limited to single tasks, and how to perform inference when integrating multiple tasks into a pipeline remains an open question. Furthermore, existing implementations of SI are developed individually for each task, and there is no unified framework for implementing SI.
We note that our long-term goal beyond this current study is to ensure the reproducibility of data-driven decision- peaiinn by accounting for the entire pipeline from raw data to the final results, with the current study on a class of feature selection pipelines in linear models serving as proof of concept for that goal.
Related Work. Most research data analysis pipelines is concentrated in the field of software engincering rather than machine learning (Sugimura & Hartl, 2018; Hapke & Nelson, 2020; Drori et al., 2021), with a primary focus on the design, implementation, testing, and maintenance of pipeline systems to ensure efficiency, scalability, and robust- ness. Meanwhile, AutoML. has emerged as a related area where researchers are automating the construction of these pipelines, and many companies have developed tools for this purpose (Microsoft, 2018; Amazon, 2019; Google, 2021). However, to the best of our knowledge, there is no existing studies that systematically discusses the reliability of data analysis pipelines. Resampling techniques, such as cross- validation (CV), are commonly used to evaluate the entire data analysis process. However, practical data analysis often includes unsupervised learning tasks like MVIs and ODs, where resampling cannot be used to accurately evaluate the reliability of the entire pipeline. Additionally, dividing the data reduces the sample size, leading to decreased accuracy in hypothesis selection and statistical power.
Contributions. Our contributions in this study are threefold. First, we develop a statistical test for feature selection pipelines composed of various configurations of missing value imputation (MVI), outlier detection (OD), and feature selection (FS) components, based on the SI framework. Second, this study represents the first applica- tion of SI to inference on a combination of multiple analysis components in a unified, systematic manner. Finally, we provide a practical computational framework implemented as the Python package', which facilitates the construction of statistical tests across any pipeline configuration without additional implementation costs. For repro- ducibility, our experimental code is available at https: / htt.://b.cccoottaaaaaaaaattaaaaa
2. Preliminaries
SI has gained attention as statistical inference method for feature selection in linear models (Taylor & Tibshirani, 2015; Fithian et al., 2015). It has been applied to various feature selection algorithms such as marginal screening (Lee & Taylor, 2014), stepwise FS (Tibshirani et al., 2016), and Lasso (Lee et al., 2016), and extended to more complex methods (Yang et al., 2016; Suzumura et al., 2017; Hyun et al., 2018; Rügamer& & Greven, 2020; Das et al., 2022;
Given set of algorithm components, a pipeline is defined by selecting some components from the set and connecting the selected components in an appropriate way. A pipeline can be represented as a directed acyclic graph (DAG) with components as nodes, and the connections as edges. In this study, as an example class of pipelines, we consider a set of
Phttps://pyp .ecillrr///prro wttpetttttc///ssaeeooooosooscess httpojjecc///iipplieeete 4pipeline
2

---

Statistical Test for Feature Selection Pipelines by Selective Inference
algorithms consisting of three MVI algorithms, three OD algorithms, three FS algorithms, as well Intersection and Union operations (specific three algorithms each for MVI, OD, and FS are described later in this section). Figure shows two examples of pipelines within this class. Note that each FS algorithm corresponds to a single node in the DAG, and the FS algorithms are not described as DAGs.
the linear model after imputation of missing values, removal of outliers, and feature selection is expressed as
م (x ( X (+) Similarly, we consider the population least-square solution for the unobservable true value vector j(X) in (I), which is defined as
3 (x X XI - (X.)).
Problem Setting. In this study, we consider the problem of feature selection for linear models from a dataset contain- ing missing values and/or outliers using the aforementioned class of feature selection pipelines. Let us consider linear regression problem with instances and di features. We de- note the observed dataset as (X, y), where X E Rx is the fixed design matrix, while y E R' is the response vector which contains outlying values but excludes missing values (i.e., n' S n). We assume that y is a random realization of the following random response vector
where μ - E ER-1101 is an n n -O- -oi-mmension vector obtained by providing X.OMM M to the unknown true function μ with the missing values imputed with the same MVI algorithm. To quantify the statistical significance of the selected features, we consider the following null hypothesis Ho and the alternative hypothesis H1:
Ho : 83 = 0 v.s. H1 8 + 0, j JEM,
(3)
where, with a slight abuse of notation, 87 and 33 respec- tively indicates the element of 3 and و corresponding to the selected feature j E .M.
Y = (xX)+ t E E~ N
(1)
where (x() E RT is the unknown true value function, while & E I8' is independently normally distributed with variance 0² which is known or estimable from an inddpen- dent d Although we do not pose any functional form on the true value function (xX) for theoretical justifica- tion, we consider a case where the true values (xX) are reasonably approximated by linear model as long as they are non-outliers. This is a common setting in the field of SI, referred to as the saturated model setting. Furthermore, we denote the response vector with imputed missing values as y((+) R". Using the above notations, feature selec- tion pipeline comprising of MVI, OD, and FS algorithm components is represented as mapping:
Missing-Vaie Value Imputation (MVI) Algorithm Compo- nents. In this paper, as three examples of MVI algorithms, we consider mean value imputation, nearest-neighbor im- putation, and regression imputation algorithms (see Ap- pendix A.1). A MVI algorithm component is represented as
fuvi: (X, O,M + ..,M),
where, among the four variables, only y is updated to y but note that this notation is used to uniformly handle all components in the pipeline. It is important to note that these three MVI algorithms are linear algorithm in the sense that, using a matrix Dx E gaxxn that depends on X, the imputed values are written as y Dxy.
" P: Puddd XRR p e (X, + (3) O,,M) E R" x
x2)
(2)
Outlier Detection (OD) Algorithm Components. In this paper, as three examples of OD algorithms, we consider Cook's 's distance-based OD, DFFITS OD, and Ln regression based OD algorithms (see Appendix A.2). A OD algorithm component is represented as
where y (++ E R is the response vector with missing values imputed, O c [n] is the set of detected outliers, and M c
[d] is the set of selected features.
Statistical Test for Pipelines. Given the output of pipeline in (2). the statistical significance of the finally se- lected features can be quantified based on the coefficients of the linear model fitted only with the selected features from a dataset with missing values imputed and outliers re- moved. To formalize this, we denote the design matrix after removing outliers and composed only of the selected fea- tures as X-O.M E |0 and denote the response vector with outliers removed and missing values imputed as y (+) Using these notations, the least squares solution of
Son {X, (t,, ,,,M) - (X) (x),(t))',,, O',
where, O' is the updated set of outliers. Note that, outlier removal and feature selection have not yet been performed, the sets O and M are initialized O and M [d].
Feature Selection (FS) Algorithm Components. In this paper, as three examples of FS algorithms, we consider marginal screening. stepwise feature selection, and Lasso algorithms (see Appendix A.3). A FS algorithm component is represented as
We discuss the robustness of the proposed method when the variance is unknown and the noise deviates from the Gaussian distribution Appendix F.
frs
(X. (.,,,,M
where, M' is the updated set of features.
3

---

Statistical Test for Feature Selection Pipelines by Selective Inference
Union and Intersection Components. When using multi- ple OD/FS algorithms, it is necessary to include components in the pipeline that perform the union/intersection of the de- tected outliers or selected features. Such union/intersection components for OD/SS are respectively written as
3. Selective Inference for Feature Selection Pipelines
To perform statistical test for pipelines, it is necessary to consider how the data influenced the final result through the calculations of each algorithm component of the pipeline and in operations where they are combined with a specified configuration. We address this challenge using the SI frame- work. In the SI, statistical inference is performed based the sampling distribution conditional the process by which the data selects the final result, thereby incorporating the influence of how data is processed in the pipeline.
s: {X, y (Oe))EEE) M3 + {X, y
EEELE M},
M: (X,g(t), O, Meeeeen + (x, ((t, ,0,,,,,,,,,,,, Mej
where is the number of OD/FS algorithms and an operator E indicates either union or intersection of multiple sets.
Selective Inference. In SI, p-values are computed based on the null distribution conditional on an event that certain hypothesis is selected. The goal of SI is to compute a p- value such that
Automatic Pipeline Construction. In this study, we con- sider two cases for pipeline configuration: an option spec- ified by the user and an option determined based on the data. In the first option, the user can select some of the aforementioned data analysis components and specify their own configuration. On the other hand, the second option allows for the selection of the optimal configuration from among multiple pre-defined pipeline configurations based on CV. An important point in the second option is that our statistical test is designed by properly considering the fact that the optimal pipeline configuration has been selected based on the 'ata". For more details on the second option, see $6 and Appendix F.
PH, (p S a a = M, 0) a Va E (0, 1),
(4)
where My and Oy respectively indicate the random set of selected features and detected outliers given the random response vector Y, thereby making the p-value is random variable. Here, the condition part My .M and Or = O in (4) indicates that we only consider response vectors Y yielding a certain feature set . and a certain outlier set O. If the conditional type I error rate can be controlled as in (4) for any possible hypotheses (M, O) E 21d x 2imi, then, by the law of total probability, the marginal type I error rate can also be controlled for any a E (0, 1) because
Selective Inference.
For the statistical test in (3), it is rea- sonable to use B E M as the test statistic. An important point when addressing this statistical test within the SI ap- proach is that the test statistic is represented as linear func- tion of the observed response vector as 8, = =n] y.j E M,, where ny E R²',, .j E M a vector that depends on y only through the detected outlier set O and the selected feature set M 4. In SI, this property utilized perform statisti- cal inference based on the sampling distribution of the test statistic conditional on O and M. More specifically, since follows a normal distribution, it can be derived that the sampling distribution of the test statistic 8 = n, y. E M conditional on O, M, and the sufficient statistic of the nui- sance parameters follows truncated normal distribution. By computing p-values based on this conditional sampling distribution represented as truncated normal distribution, it is ensured that the type I error can be controlled even in finite samples. For more details on SI, please refer to the following sections or literatures such as Taylor& & Tibshirani (2015); Fithian et al. (2015); Lee & Taylor (2014).
HHo (p S a))
PHO
Σ Σ
Pho (pssa a MY M,
M 0e2in)
a.
Therefore, in order to perform valid statistical test, we can employ p-values conditional on the hypothesis selection event. To compute a p-value that satisfies (4), we need to derive the sampling distribution of the test-statistic
T(Y) (My = My, Oy = Oy).
(5)
Selective p-value. To conduct statistical hypothesis test- ing based on the conditional sampling distribution in (5), we introduce an additional condition on the sufficient statistic of the nuisance parameter Φr, defined as
mm7 T ||m|1 Y.
Qr I
(6)
As stated in $1, CV cannot be used for an accurate evaluation of pipeline when it includes unsupervised learning components such as MV1 or OD, However, it possible to compute a valid p-value for a pipeline selected hy CV we properly consider the CV-based pipeline selection as part of the selection event for SI.
This additional conditioning on Qr is standard practice in the SI literature required for computational tractability
* 'Note that the MVT algorithms considered in this paper depend only on X, not on y.
'The nuisance component Qy corresponds the component z in the seminal paper (Lee et al., 2016) (see Sec. 5, Eq. (5.2), and

---

Statistical Test for Feature Selection Pipelines by Selective Inference
Based on the additional conditioning on Qy, the following theorem tells that the conditional p-value that satisfies (4) can be derived by using& truncated normal distribution.
4. Computations: Line Search Interpretation
From the discussion $3, is suffice to identify the one- dimensional subset 2 in (7) to conduct the inference. In this section, we propose a novel line search method to efficiently identify the 2.
Theorem 3.1. Consider a constant design matrix X, ran-
dom response vector Y ~ (u, of Iw) and observed response vector y. Let (My, Ox) and (My. Qy) be the pairs of selected features features and detected outliers, obtained by applying a pipeline process P in the form of (2) to (X, Y) and (X, y), respectively. Let n E R' be a vector depend- ing on (My. Oy), and consider test-statistic in the form of TYY) = =n" Y. Furthermore, define the nuisance param- eter Qy in (6).
4.1. Overview of the Line Search
The difficulty in identifying the Z arises from the fact that the multiple FS/OD algorithms are applied in ar- bitrary complex order. To surmount this difficulty, we pro- pose an efficient search method that leverages parametric- programming and the fact that our pipeline can be concep- tualized as a directed acyclic graph (DAG) whose nodes represent the operations. In a standard analysis pipeline, MM and O are computed and updated along the DAG, How- ever, in framework, intervals for which M and O are constant can also be computed and updated, allowing the computation of the truncation intervals 2.
Then, the conditional distribution
T(Y) I {My = My, Or = Ov. 2x = 2
truncated normal distribution TN with mean n μ., variance o Ill?, and truncation intervals Z, where 2 is defined as
T μ. o² || 2)
In the following, we first discuss how, given a certain compu- tational procedure (combining update rules as discussed in later), the can be identified by garametrie-progrmmminn Then, we summarize the overall procedure compute the selective p-value from the Z. Finally, we describe the up- date rules for each node based on the existing methods of SI for each FS and OD algorithm. Note that DAGs can topologically sortable, so that update rules can be applied in sequence. The overview of the proposed line search method is illustrated in Figure 2.
Z {a E ER|
Matbz = =My, Oatbe O,y,, (7)
a Qy: b=n/ 1n////
The proof of Theorem 3.1 is deferred to Appendix B.1. By using the sampling distribution of the test statistic T(Y) conditional on My = My Oy = Og, and ºr = 2y in Theorem 3.1, we can define the selective p-value as
My -My;
Pselective PHo
[T(Y) [T())
Oy = Oy,
Qr Qy
4.2. Parametric- Proorammig
To identify the truncation intervals Z, we assume that we have a procedure to compute the interval [T.z, U=] for any R. which satisfies
Theorem 3.2. The selective p-value defined in (8) satisfies the property in (4), ie.,
My = =Mg.) Or = =O, ) =a. Va E (0,1), 1).
Vr [L,, Uz], Ma Ma+bz Oa+br Oa+bz.
PHo
Pselective Sa
Then, the truncation intervals Z can be obtained by the union of the intervals [T, Uz] as
Then, the selective p-value also satisfies the following prop- erty of a valid p-value:
= MEBI Oa
U
UZ]
(9)
PHo (Pselective sa)= =a, Va e (0,
The procedure in (9) is commonly referred to parametric- programming. We discuss the details of the procedure to compute the interval [L2, Uz] by defining the update rules for each node in the next subsection.
The proof of Theorem 3.2 is deferred to Appendix B.2. This theorem guarantees that the selective p-value is uniformly distributed under the null hypothesis Hy, and thus can be used to conduct the valid statistical inference in (3). Once the truncation intervals Z is identified, the selective v-luuue in (8) can be easily computed using Theorem 3.1. Thus, the remaining task is reduced to identifying the truncation intervals Z.
4.3. Update Rules
In this subsection, we discuss the computation procedure to obtain the interval [Lg, U2] for any & E R just mentioned
Theorem 5.2) and is used in almost all the SI-related works that we cited.
in $4.2. To compute the interval [Lg, Ug], we consider extending the input of each node in DAG and denote
5

---

Statistical Test for Feature Selection Pipelines by Selective Inference
(i)
5:F5 lasso
1:MVI mean
2:0D
3:F5
6:U-F
regression (solt-ipod!
marginal screening
union
4:FS stepwiss
(iii)
update.
update
update O
update.M
update M
union of of
por porametrioed line
M
M
M
M
Z = {2 R Maste = Mag O D--bs = O0)
truncated regio region
(,4
12, 3]
(1
2, 3,4
(2,3)
21.
2,
(1)
(3.5)
[1,
2,
12,
3)
(2,3)
2g (1,2)
M, (2.3)
Figure Schematic illustration proposed line search method identify truncation intervals The upper part shows the DAG representation of the pipeline and its topological sorting (i). The lower left part shows the operations performed by update rules in sequence (ii). The lower right part shows the identification truncation intervals by taking union some intervals based on paramttri--rogramming (iii).
pair of (X, a. O, where X is the design matrix, b and the currently linear expression of the response vector + be, M and O are the currently selected features and detected outliers, and and are the
not change by changing the inpul
UpdateInterral (P, b,
-[L2, U2], Mal), Oatbz).
currently interval. The input of the first node is initialized to (X, a, b, 2, [d], 0, -cc, oo), where features. We details the update rules for this pair at each node of DAG in Appendix C.
the number of
The proof of Theorem 4.1 is deferred to Appendix B.3.
Algorithm Apply Update Rules in Order of Topological Sorting of DAG (Update Interval)
The overall procedure for computing the interval [L,, Uz] by applying the update rules in the order of the topological sorting of the DAG is summarized in Algorithm 1, where the operation pa receives the index of the target node and returns the indexes its parent nodes, and pa(1) is set to Algorithm satisfies the specifications described in $4.2, i... the following theorem holds.
Require: X, a, b and 1: Converts the pipeline P to topologically sorted graph
(V, F)
2: Initialize the input of the first node Bo as
(X, a, b, [p], 0, -0, oo) (see $4.3)
for each index of node i E {1...., [V|] do
Apply the update rule of the node e; to its input B Bpa(t) obtain the output B; (see $4.3)
Theorem 4.1. Consider pipeline P, design matrix X, and vectors and b representing the linear expression of
end for
the response vector fixed. For any R, let (T.z, U2}, Ma+bz and Oatbz be the output of Algorithm I with P, X, a. b and z as input.
Let the last four components of Biv1 be Ma-bar Oatbr, L, and U., respectively Ensure: [L2, U2], Ma- bz and On-bz
Then, for any r E TTz, U2}, , the output of Algorithm does
6

---

Statistical Test for Feature Selection Pipelines by Selective Inference
5. Implementations: Auto-Conditioning
6. Numerical Experiments
All of the update rules defined in $4.3 are node-specific operations and do not depend on the type of node corre- sponding to the input/output. Then, we can modularize the update rules and apply them sequentially as in Algo- rithm 1, which implementation we call cuulo-conditioiing The auto-conditioning allows one to simply define an arbi- trary pipeline and perform hypothesis testing on it without additional implementation costs. In this section, we show some examples of defining pipelines and performing hy- pothesis testing using the nuto-conniiioninn. The implemen- tation we developed can be interactively executed using the provided Jupyter Notebook (ipynb) file, which is available in the our package repository.
Methods for Comparison. In our experiments, we con- sider the three types of pipelines: opl, op2, and cv. The opl and op2 are defined in Figure 2. The cv is a pipeline selected based on cross- vooss-vlldationn from 16 different param- eters settings each in the op1 and op2 pipelines (i.e., from 32 pipelines in total). For each three types of pipelines, we compare the proposed method (proposed) in terms of type error rate and power with the following three methods:
w/o-pp: An ablation study that excludes the paramet- ric programming technique described in $4.2. This is implemented by replacing the Z in (9) with interval [L1 U2] that contains the observed test statistic T(y).
As an example, Listing shows a code example that defines two pipeline shown in Figure and performs hypothesis testing. based on our package. A similarly simple UI al- lows for easy implementation of other pipeline structures as well as automatic pipeline construction based on the cross-validation. For more examples, please refer to the Appendix G and the our package repository.
naive: This method uses classical 2-test without conditioning, i.e., we compute the naive p-value as Pnaive = =H, ((T(Y)) 2 (T())))-
bonferroni:
This is method to control the type I error rate by using the Bonferroni correction, sim- ple yet widely used method for multiple testing cor- rection. The number of all possible pair of selected features and detected outliers is 2d 2", then we com- pute the Bonferroni corrected p-value as Pbonferroni min(1, 2d 2n Pnaive)-
Listing 1. Code example that defines the pipeline shown in Fig- ure 2. We can create an instance of manager class which handles the desired pipeline simply by specifying each operation in turn. To perform hypothesis testing, we can call the i infernnce method of the manager instance with the input dataset (X, y) and the deviation of the noise a.
Experimental Setup. In all experiments, we set the sig- nificance level a = 0.05. For the experiments to see the type error rate, we change the number of samples n E 200, 300, 400) and set the number of features d to 20. See Appendix D.1 for results when the number of fea- tures d changed, and for the high-dimensional regression setting (i.e., where d >>>> n²). For each configuration, we gen- erated 10,000 null datasets (X, y), where Xij ~ (N(0, 1) for all (i,j) E [n]] x [d] and y ~ (N(, I.). Missing val- ues were introduced by randomly setting each to NaN with a probability of 0.03. To investigate the power, we set n = 200 and 20 and generated dataset (X, y), where X1 ~N 1) for all (i.3) E [n] x [d] and y X,3 + c. The error term followed normal distribution (o Tx), and the coefficient vector / E Rd was constructed such that its first three elements were set to A and the remaining elements were set to O. Missing values were introduced by randomly setting each y; to NaN with a probability of 0.03. We change the true coefficients A E {0.2, 0.4, 0.6, 0.8}. For power evaluation, hypothesis testing was conducted only when the pipeline selected least one truly relevant feature (i.e., one of the first three features), resulting in a to- tal of 10,000 tests. In addition, see Appendix D.2 for results when the missing value probability increased, Appendix D.3 for the computational time of the proposed method for larger datasets and more complex pipelines, and Appendix D.4 for the computer resources used the experiments.
import numpy as np from si4pipcline import *
def optionl () -> PipelincManager:
X, y = initialize wanittaaliee_ataaeeeee dataset ()
y = jaan__aalee_impuaatiin (x, y)
c soft_ipod (x, 0.02)
x, y ferooveoottieerat y, 0)
K = marginai_screent
(x, y, 5)
X extract_Features,
M1 Seepwise_featuree setturrrseeeettoo (x, y, 3)
M2 lasso (X, y, 0.08)
N union M2)
return constrrut__plpallnee.oo
def option2 -> PipelineManager:
x, = initi batiseeddtttttttee
y definite_ def.ndte_rrrression__ppppttte ittpassiiiiimmmttttioinioo (X, y)
M marginal scw._eeeeeennnnneeeee (X, y, 5)
X marginni__srreeningg Feraact_eeaturestee (2, M)
C cook distance (X, y, 3.0)
X, y lemove_outtieerste
y, 0)
M1 -
(x, 3)
X2 lasso (X, y, 0.08]
M - intersection (M1, M2)
return construct_pipell (output-M)
pl= optionl() ()
X = (st. (100, 10))
y np. np.randem. nowma (size=100)
M, p_list - pl. (X, Y, sigma-1. .0
7

---

Statistical Test for Feature Selection Pipelines by Selective Inference
Table 1. Power on eight real-world datasets when changing the sample size n. Fach cell indicates the power of the proposed method (proposed) and the ablation study (w/o-pp), separated by a slash, with the higher value in bold. The proposed method demonstrates significantly higher power than the ablation study method across all datasets and sample sizes. Furthermore, the power of the proposed
method increases with increasing sample size
n
Data
Data2
Data3
Data4
Data5
Data6
Data7
Data8
100
.57/.07
.48/.06
.571.07
.51/.07
.68/.10
.55/.04
.30/005
.25/.06
150
.79/.09
.71.08
.66/.12
.57/.10
.74/.12
.72/.06
.37/.06
.37/.06
200
.91/111
.80/088
.78/.15
.66/.12
.76/.13
.82/.08
.49/.07
.40/.06
Results. The results of type error rate are shown in left side of Figure The proposed, w/o-pp, and bonferroni successfully controlled the type I error rate under the significance level across all settings and pipeline types, whereas the na ve could not. Because the na i Live failed to control the type I error rate, we no longer consider its power. The results of power are shown in right side of Figure 3. Among the methods that controlled the type I error rate, the proposed has the highest power, followed by the w/o-pp, across all settings and pipeline types. The reduced power of the w/ w/o-pp compared to the proposed can be attributed to its inherent conditioning on more information than those defined in (5). This problem is known as over- conditioning in the context of SI. The notably low power of the bonferroni is consistent with the understanding that such classical methods are often too conservative for the large-scale problems considered in this study.
ity of data-driven decision- decision-making by accounting for the entire pipeline from raw data to final results, with this study on a class of feature selection pipelines serving as a proof of concept. To achieve this future goal, there are still limita- tions on the applicable data analysis components, presenting several challenges in extending the proposed framework to more complex data analysis pipelines. Additionally, it is in- teresting to consider extending this framework to scenarios where data analysis pipelines are automatically constructed using state-of-the-art AutoMI. approaches.
1.0
1.0
proposed
proposed
Type Error Rate
wp
w/o woo-pp
bonferroni
bonferroni
Power
0.5
naive
0.5
significance level
0.0
0.0
100
200
300
400
0.2
0.4
0.6
0.8
number of samples
true coefficient
Real Data Experiments. We compared the pr proposed and w/ w/o-pp in terms of power, for the ov pipeline on eight real-world datasets from the UCI Machine Learning Repos- itory (all licensed under the CC BY 4.0; see Appendix D.5 for more details). These experiments were conducted under the implicit assumption that features selected by the fea- ture selection pipeline are truly relevant. This assumption is reasonable because both the proposed and w/o-PP evaluated in this study have been shown to control the type I error rate. From each original dataset, we randomly gen- erated 1,000 sub- sub-sampled datasets with sample sizes of (100, 150, 200]. We then applied both the proposed and w/ pp to assess their powers. The results, presented in Table 1, demonstrate that the proposed method has much higher power than the w/o-pp across all datasets for all sample sizes. Furthermore, the power of the proposed increases with increasing sample size no.
(a) Type Error Rate and Power of op1 pipeline
1.0
1.0
proposed
proposed
Type Error Rate
w/o-pp
w/o-pp
bonferroni
bonferroni
0.5
naive
Power
0.5
signiffeance level
0.0
0.0
100
200
300
400
0.2
0.4
0.6
0.8
number samples
true confficient
(b) Type Error Rate and Power of op2 pipeline
1.0
proposed
proposed
Type Eiror Rate Rate
w/oopp
wlo-pp
bonferreni
Power
bonferroni
0.5
naive
0.5
significance level
0.0
0.0
100
200
300
0.4
number of samplecs
true confficient
(c) Type Error Rate and Power of cv pipeline
7. Conclusions
Figure 3. Type Error Rate when changing the number of samples (left side) and Power when changing the true coefficient (right side). The proposed method (proposec), the ablation study (w/o opp), and the Bonferroni method (bonferron) successfully control the type error rate across all settings and pipeline types. Among the methods that control the type crror rate, the proposed method has the highest power across all settings and pipeline types.
In this study, we introduced a novel framework for testing the statistical significance of feature selection pipelines in linear models, comprising multiple MVI, OD, and FS al- gorithms based on the concept of SI. Our long-term goal extends beyond this current study to ensure the reproducibil-
8

---

Statistical Test for Feature Selection Pipelines by Selective Inference
Acknowledgements
Duy, V. N. L.,, Toda, H., Sugiyama, R., and Takeuchi, I. Computing valid p-value for optimal changepoint by se- lective inference using dynamic programming. In Ad- vances in Neural Information Processing Systems, 2020.
This work was partially supported by JST CREST (IPMICR2ID3, JPMICR22N2), JST Moonshot R&D (JPMUMS2033-05) RIKEN Center for Advanced Intelli- gence Project, and RIKEN Junior Research Associate Pro- gram.
Duy, L., Iwazaki, S., and Takeuchi, I. Quantifying statistical significance of neural network-based image segmentation by selective inference. Advances in Neural Information Processing Systems, 15:31627-3166 2022.
Impact Statement
This work, which focuses on statistical tests for feature se- lection pipelines, aims enhance the reliability of AI and has the potential broadly influence the machine learning community. On the other hand, it does not present signifi- cant ethical concerns or foreseeable societal consequences because this work is theoretical and, as of now, has no direct applications that might impact society or ethical considera- tions.
Fithian, W. Taylor, J., Tibshirani, R., and Tibshirani, R. Selective sequential model selection. arXiv preprint arXiv: 2015.
Gao, L. L.., Bien, J., and Witten, D. Selective inference for hierarchical clustering. Journal of the American Statisti- cal Association, pp. 1-11,202,
Google. Vertex ai, 2021. URL https://cloud. google, watee/coo/eeeeeeeee
References
Hapke, H. and Nelson, C. Building machine learning pipelines. O'Reilly Media, 2020.
Gas Turbine CO and NOx Emission Data Set. UCI Machine Learning Repository, 2019. DOI:
https//cbi.nn//0244////wWwW
Hyun, S., G'sell, M., and Tibshirani, R. J. Exact post- selection inference for the generalized lasso path. Elec- tronic Journal of Statistics, (2())1003-1109) 2018.
Amazon. lot, 2019.
Amazon sagemaker autopi-
URL /etps://docs.. aws.
Jewell, S., Fearnhead, P., and Witten, D. Testing for change in mean after changepoint detection. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84 14):1002-1144, 2022.
amazon. com/ wwwgeaakerlattetteeeeeeea
outopilot-aa automate tettomattemmdel---ee model deveel--dvellppment. html.
Brooks, T., Pope, D., and Marcolini, M. Airfoil Self- Noise. UCI Machine Learning Repository, 1989. DOI: https://ooiorr//024442////VW
Le Duy, V. N., Lin, H.-T., and Takeuchi, I. Cad-da: Con- trollable anomaly detection after domain adaptation by statistical inference. In International Conference on Arti- ficial Intelligence and Statistics, pp. 1828-1836. PMLR, 2024.
Chen, S. and Bien, J. Valid inference corrected for out- lier removal. Journal of Computational and Graphical Statistics, 29(2):323-334, 2020.
Lee, J. D. and Taylor, J. E. Exact post model selection inference for marginal screening. Advances in neural information processing systems, 27, 2014.
Cortez, P., Cerdeira, A., Almeida, F., Matos, T., and Reis, Wine Quality. UCI Machine Learning Repository, 2009. DOI: http:///doi.org/1..444/23 2433//5553T...
Lee, J. D., Sun, Y., and Taylor, J. E. Evaluating the statistical significance of biclusters. Advances in neural information processing systems, 28, 2015.
Das, D., Duy, V. N. L., Hanada, H., Tsuda, K., and Takeuchi, I. Fast and more powerful selective infer- ence for sparse high-order interaction model. Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, 3699):9999-10000 Jun. 2022. doi: 10.1609/ Maaivv36i9.2123.. URL https: awaii..aaaa...m awai.oraggge org/ http:///ggg..aaaaaaaaaa/////////
Lee, J. D., Sun, D. L., Sun, Y., and Taylor, J. E. Exact post- selection inference, with application to the lasso. The Annals of Statistics, 44(3):907-927, 2016.
Microsoft.
Azure automated machine learning. 2018. URL
com/en us
Drori, I., Krishnamurthy, Y., Rampin, R., Lourenco, R. d. P., Ono, J. P., Cho, K., Silva, C., and Freire, J. Al- phad3m: Machine learning pipeline synthesis. arXiv preprint arxiv:211 / 1205008. 2021.
automattedm//oovevvaa
Rügamer, D. and Greven, S. Inference for 2-boosting. Statistics and computing, 30(2):279-28) 2020.

---

Statistical Test for Feature Selection Pipelines by Selective Inference
Rügamer, D., Baumann, P. F., and Greven, S. Selective inference for additive and linear mixed models. Compu- tational Statistics & Data Analysis, 167:107350, 2022.
Sugimura, and Hartl, F. Building a reproducible ma- chine learning pipeline. arXiv preprint arXiv:1810004570, 2018.
Suzumura, S., Nakagawa, K., Umezu, Y., Tsuda, K., and Takeuchi, Selective inference for sparse high-order interaction models. Proceedings of the 34th Interna- tional Conference Machine Sourning-Volumee 70, pp. 3338-3347, JMLR. org, 2017.
Tanizaki, K., Hashimoto, N., Inatsu, Y., Hontani, H., and Takeuchi, Computing valid p-values for image seg- mentation by selective inference. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9553 9553-9562, 2020.
Taylor, J. and Tibshirani, J. Statistical learning and selective inference. Proceedings of the National Academy of Sciences, 112(25)7629-766 -6344, 2015.
Tibshirani, J., Taylor, J., Lockhart, R., and Tibshirani, R.
Exact post-selection inference for sequential regression procedures. Journal of the American Statistical Associa- fion, 111(514):600- -0000 2016.
Tsanas, A. and Xifara, A. Energy Efficiency. UCI Machine Learning Repository, 2012. DOI: https://bb.or//10.244///115
Tsukurimichi, T., Inatsu, Y., Duy, V. N. L., and Takeuchi, Conditional selective inference for robust regres- sion and outlier detection using piecewise-linear homo- topy continuation. Annals of the Institute of Statistical Mathematics, 1476))119-1122 2022. doi: 10.1007/ $10463-022-00884--2 URL https: /ttpiioo//......//////////// 1007/810443-022-008444
Yang. Barber, R. F., Jain, P., and Lafferty, Selective inference for group-sparse linear models. In Advances in Neural Information Processing Systems, pp. 2469-2477, 2016.
Yeh, I.C. Concrete Compressive Strength. UCI
Machine Learning Repository, 1998. DOI:
http::/bbb.rrr///2444///5
Yeh, I.-C. Real Estate Valuation.
UCI Ma-
chine Learning Repository, 2018.
DOI:
https//0bi.oog//044444//33300
10

---

Statistical Test for Feature Selection Pipelines by Selective Inference
A. Pipeline Components
A.1. Missing-Valle Value Imputation (MVI) Algorithm Components
MVI algorithm component is represented as
favI yX (X.
where y E IR' is the response vector which excludes missing values and y E IR" is the vector with imputed missing values. MVI algorithms in this paper are linear algorithm in the sense that, using a matrix Dx E Raxx'' that depends on X are written as y Dxy.
Mean Value Imputation. This method replaces missing values with the mean value observed data and allows for quick and easy imputation of missing values. An example of Dx for y (y1, ya 34) (i.e., y2 is missing value and is:
0
0
1/3
1/3
1/3
Dx
0
1
0
0
0
1
Nearest-Neighbor Imputation. This method replaces missing values with the most similar instance in the dataset. In this method, similarity between instances is measured by some distance between their feature vectors. As distance measures ((::)) .), for example, Euclidean, Manhattan, or Chebyshev distance can be used. An example of Dx for y = (y11 V/ y4) (i.e., Y2 is missing value and 4) is:
0
T
Dx
, J arg min ((22, a;),
1 0
(E(1,3,4
0
where ej is the vector constructed by removing the indices of the missing values (i.e., (2}) from the j-th unit vector in E
Regression Imputation. This method replaces missing values with estimated values based on regression model. We use the observed instances estimate the regression coefficients, and then the estimated coefficients to predict the missing values from its feature vector. We denote the indices of the missing values as NaN, and the indices of the observed values as NaaN. The regression coefficients can be estimated as 3 (X X -NNN.) -1x
and then each imputed
missing value y; (+) NaN can be expressed as y (+) a. G. An example of Dx for = (in Vs. u4) (i.e., 1/2 is missing value and 4) is:
1
X (X X -
XT (1,34))
(1,344).
Dx =
0
0 0
A.2. Outlier Detection (OD) Algorithm Components
OD algorithm component is represented as
fod {X, (X,g
(X, (x.
where O' is the updated set of outliers.
Cook's Distence-based OD, This method identifies instances as outliers when Cook's 's distance, a measure of the influence of a particular instance on entire regression model, exceeds predefined threshold value. Cook 's distance of the i-th instance is defined as
E U Ú5())
D,
d dMS

---

Statistical Test for Feature Selection Pipelines by Selective Inference
where ūن and My(s) are the predicted value of j-th instance from the regression model with and without i-th instance, respectively, and MSF is the mean squared error of the full model. This D; represents the standardized value of the change in predictions for all other instances due to the removal of i-th instance, and the larger D; is, the more it affects the model. By utilizing the leverage value, it can also be represented as
87
his
Dj
MSE (1 h,:)2
where é is the i-th residual, ha the i-th leverage value (i.e., the diagonal component of the matrix X x)-1xT) We identify the i-th instance as an outlier D, > where a predefined threshold value.
DFFITS OD This method has the same concept as Cook's 's distance- OD but uses DFFITS instead of Cook's distance as the measure of influence. DFFITS of the i-th instance is defined
iN De(S)
DFFITS;
(MS
where is and De(s) are the predicted value of the i-th instance from the regression model with and without the i-th instance, respectively, () is the mean squared errоr of the regression model without the i-th instance, and hu is the i-th leverage value. Thus, DFFITS is value that standardizes the difference between the predicted value when excluding and including specific instance, and the larger DFFITS, is, the more it affects the model. By utilizing the external Studentized residual Ti,ex, it can also be represented as
his his
DFFITS, V
Ti,ext-
We identify the i-th instance an outlier if DFFITS; > Ad/(n d) where À is predefined threshold value and usually set to 4.
L1 Regression based OD This method identifies instances as outliers by using L1 regularization for the mean-shift model. In this method, we assume that the unknown true value function ((X) follows the following mean-shift model:
(XX)
X,3 + u.
where u E R" is an outlier term and u, * 0 if the i-th instance an outlier, otherwise u, We estimate (Bx, ú2) by solving the following optimization problem:
(Bx, ūx) = arg min - (+) X, 2
Al u|ll,
BER uER"" 2n
where À is a predefined regularization parameter. We identify the i-th instance as an outlier if ux,; # 0.
A.3. Feature Selection (FS) Algorithm Components
A FS algorithm component is represented as
frs : () (x,
where, M' the updated set of features.
Marginal Screening This method selects the le features that are most correlated with the response variable, where is a predefined number. The correlation is computed as the absolute value of the inner product |ar] ( between the normalized feature vector zj and the response vector y (+).
Stepwise Feature Selection This method selects features by iterating through the steps of adding or deleting the features that best improve the goodness of fit of the regression model. this paper, we deal with forward stepwise feature selection, which only adds features. The residual sum of squares (RSS) of the least squares regression model constructed using the features selected up to the previous step used as the goodness of fit of the model. First, a null model model consisting
12

---

Statistical Test for Feature Selection Pipelines by Selective Inference
of intercept term) is used initial state, and in each step, RSS is calculated from the least squares regression model constructed with the features selected in the previous step and the residual of y After that, select the feature that minimize RSS and update the model. The algorithm terminates if the RSS is not improved by adding any feature, or if the number of selected features reaches a predefined upper limit.
Lasso
This method selects features by using a linear regression model with L1 regularization. We estimate the regression coefficient 3 by solving the following optimization problem:
1
β = arg min BER 2n
lly x3112 + 111
where À is predefined regularization parameter. We select the features for which 3; + 0.
B. Proofs
B.1. Proof Theorem 3.1
According to the conditioning on Qr Qy. we have
Qy Qy + Ln
7 1
Y = =2 + Y = a
|n
where = = T(Y)& E R. Then, we have
(Y RR |M MM Or -,,, Qr Qy
(Y E R" |MY = MM, Ox = O,, = a + bz, 2 E ]R]
{a bz E BR" |Matbr = My. Oa+b: Oy. R}
= (a bz E R²' E}
Therefore, we obtain
T(Y) = My. Or Oy, 2r = 2y} ~ TN(n u, 1=|1m1|, 2).
B.2. Proof of Theorem 3.2
By probability integral transformation, under the null hypothesis, we have
Psolective {My = My, Or = Oy: Qr = 2y} ~ Unif(0, 1),
which leads to
PHo (Pselective S Sal My Or Oy Qr 2y) = a, Va E (0, 1).
For any a E (0, 1). by marginalizing over all the values of the nuisance parameters, we obtain
PHo (Psclective I a My = -My Ox = -O,)
J Pho (Peelective Sa My -My =0y, Qx Qy)
Pho (or 2n |MY -My, Oy O,,) d2n
/ Rm
PHo (Ox = Qy My = My. Or = Oy) dQy a.
Therefore, we also obtain
PHo (Peelective aa)
Σ
Σ
(My, Pt (Peelective salMs Myy Or -00)
My 0g02(n)
=0
Σ
Σ
PHo 0,) =a.
M2 OgE21n)
13

---

Statistical Test for Feature Selection Pipelines by Selective Inference
B.3. Proof of Theorem 4.1
is sufficient to consider only input to Algorithm 1, In addition, as a notation, define Gj the mapping that returns the last four components of B, for i {0, 1, ...., VI), i.e,
g R (Ma+bz: O Oa-oss 1/2 ut) E 2 2 RR? E (0, 10.1,.....1. [V]]
According to the above notation, all we have to show is that (2](22 (r)((r) for any R and any E [!! u We show this by mathematical induction.
In the case i = 0, it is obvious from the definition of Bo in Algorithm 1 that Go(2) = Go(r) = ((p), 0, -00, oo) for any 2 E RR and any r E [22 u = [-00, oo].
any 2 R and any r E [1, ud]. Under this assumption, noting that pa +1 [0, from a property of topological sort, is obvious that G (z) = 1 (r) for any E R and any [t=+], u from the update rule of 1/41 described in $4.3.
Next, we assume that for any fixed i E (O...., V 1}, (y(2) (r(r) for any j E {O.....
i), .
C. Details of the Update Rules
Update Rulu for the Node of MVI. The node of MVI imputes the missing values in the response vector a + bz. All MVI algorithms considered in this study are expressed as linear transformations determined on the basis of X. Thus, let Dx be the linear transformation matrix, the update rule should be follows:
(X, a. b, M,0,1,
Update Rule for the Node of FS. The node of FS selects the features (2) from the dataset (X-O,M, a-o b-oz), which means that feature selection is performed the dataset extracted from (X, bz) based M and O. For all FS algorithms considered in this study, the computation procedure obtain the interval [Ls, us] 2, which satisfies
Vr (12 u.], M'(r) MM(xs),
have been proposed in previous studies (Lee & Taylor, 2014; Tibshirani al., 2016; Lee al., 2016). Utilizing this, the update rule should be as follows:
(X, a, b, z, M,O, (K.. M 0,
min uz)).
Update Rule for the Node of OD. The node of OD detects the outliers O'(z) from the dataset (X-O,M; am by²),, which means that outlier detection is performed on the dataset extracted from (X, a + bz) based on M and O. For all OD algorithms considered in this study, the computation procedure to obtain the interval [1z, uz] D z, which satisfies
Vr E us], O'(r) O'(=;,
have been proposed in previous studies (Chen & Bien, 2020), Utilizing this, the update rule should be as follows:
(X, a, b, M, + (X, a, M, max(l, (2), min(u, (,, u.)).
Update Rule for the Node of Union/Intersection of Features/Outliers. The node computes the union or intersection of selected features or detected outliers. With being the number of input edges, for each selected feature and detected outlier, the update rules should as follows:
a,, O, ue) beEee H bx,
Σ eE[E]
Me, O, max le, min We), eE[E]
eE[E]
4 H (X,
Σ
Oe. max le, min ue);
e eE[E]
eE[E]
where E represents the union or intersection depending on the type of the node.
14

---

Statistical Test for Feature Selection Pipelines by Selective Inference
D. Details of the Experiments
D.I. Additional Type Error Rate Results
We also conducted experiments to investigate the type error rate when the number of features d' is changed, and for the nigh-dimensional regression setting (i.e., where >> n). For the former case, we changed the number of features {10, 20, 30, 40} and set the number of samples and changed the number of features d (400, 800, 1200, 1600). It should be noted that, within this experimental setting, the op2 and op3 pipelines are used, to handle the high-dimensional regression setting. The op3 pipeline is defined by reversing the order of the L1 regression-bad based outlier detection (OD) node and the marginal screening feature selection (FS) node the op1 pipeline. In both cases, we generated the null datasets the same way as in the main experiments ($6), and the results are shown Figure and Figure 5, respectively.
200. For the latter case, set the number of samples
100
1.0
1.0
1.0
proposed
proposed
proposed
Type Error Rate
w/o-pp
Type Error Rate
w/o-pp
Type Error Rate
w/o-pp
bonferroni
bonferroni
bonferroni
0.5
naive
0.5
naive
0.5
naive
significance level
significance level
significance level
0.0
0.0
0.0
10
20
30
40
10
20
30
40
10
20
30
40
number of features
number of features
number of features
op-
(b) op2
Figure 4. Type Error Rate when changing the number of features d. Our proposed method (proposed), the ablation study (v/ (w/o-pp), and the Bonferroni method (bonferroni) successfully control the type error rate across all settings and pipeline types.
1.0
1.0
1.0
proposed
proposed
proposed
Type Error Rate
w/o-pp
Type Error Rate
w/o-pp
Type Error Rate
w/o=pp
bonferroni
bonferrani
bonferroni
0.5
naive
0.5
naive
0.5
naive
significance level
significance level
significance level
0.0
0.0
0.0
400
800
1200
1600
400
800
1200
1600
400
800
1200
1600
number of features
number of features
number of features
(a) op2
(b) op3
(c) cv
Figure Type Error and the Bonferroni method (bonferroni) successfully control the type I error rate across all settings and pipeline types.
high-dimensional regression setting. Our proposed method (proposed), the ablation study (v/ o oppp)
D.2. Effect of Missing Value Probability
We also conducted experiments to investigate the effect of the missing value probability on the type ervor rate and power of the proposed method. In the experiments, we change the missing value probability P E {0.03, 0.12, 0.21, 0.30). For the type error rate, we set the number of samples n 200 and the number of features d = 20. For the power, we set the number of samples n 200, the number of features 20, and the true coefficients A 0.4. In both cases, we generated the datasets the same way as the main experiments ($6), and the results are shown Figure 6.
D.3. Computational Time of the Proposed Method
We also conducted experiments to investigate the computational time our proposed method by applying to three types of pipeline structures (Default, Parallel, and Serial) with large-scale datasets. Default pipeline correspond to the opl pipeline
15

---

Statistical Test for Feature Selection Pipelines by Selective Inference
in $6. Parallel and Serial pipelines are defined as in Figure 8, which clarifies the difference from the Default with components colored in pink. In the experiments, change the number of samples E (400, 800, 1200, 1600) with the number of features d = 80 and the number of features d E {40, 80, 120, 160} with the number of samples datasets in the same way as in the main experiments ($6). Note that in this experiment, we recorded the computational time for a single hypothesis testing (i.e., calculate one p-value) on a single CPU core. The results are shown in Figure 7.
800 generate the null
1.0
1.0
proposed
proposed
Type EEror Rate
w/o-pp
w/o-pp
bonferroni
bonferroni
Power
0.5
naive
0.5
significance level
0.0
0.03
0.12
0.21
0.30
0.03
0.12
0.21
0.30
missing value probability
missing value probability
(a) Type Error Rate
(h) Power
Figure 6. Type Error Rate and Power when changing the missing value probability. The proposed method (proposed), the ablation study (w/ (w/o-pp), and the Bonferroni method (bonferres roni) successfully control the type I error rate across all settings. Among the methods that control the type error rate, the proposed method has the highest power across all settings.
Computationa Time (s)
125
opl
opl_paralic|
Computationa Time
opl parallllll
300
opl_serial
opl_serial
200
100
0
0.-
400
800
1200
1600
40
120 number features
80
160
number samples
(a) Number of Samples
(b) Number of Features
Figure 7. Computational Time when changing the number of samples and features. The results show that computational time exponentially increased the number of samples increases while the number of features has obvious effect. Moreover, it seems that increasing the number of nodes in the pipeline increases the computational time, but how much it increases also depends on the structure.
D.4. Computer Resources
All numerical experiments were conducted on computer with a 96-core 3.60GHz CPU and 512GB of memory.
D.5. Details of the Real Datasets
We used the following eight real datasets from the UCI Machine Learning Repository. All datasets are licensed under the CC BY 4.0 license.
Airfoil Self-Noise (Brooks et al., 1989) for Datal
Concrete Compressive Strength (Yeh, 1998) for Data2
Energy Efficiency (Tsanas & Xifara, 2012) for Data3 (heating load) and Data4 (cooling load)
Gas Turbine CO and NOx Emission Data Set (gas, 2019) for Data5
Real Estate Valuation (Yeh, 2018) for Data6
Wine Quality (Cortez al., 2009) for Data7 (red wine) and Data8 (white wine)
16

---

Statistical Test for Feature Selection Pipelines by Selective Inference
Parallel
4:FS
1:MVI
2:0D regression
3:FS
6:U-F features
9:U-F features
screaning
stepwise
Serial
aass
2:0D (sot=poc)
S:FS lasso
B:U-F union of fentures
value
marginal streering
stepwise
Figure 8. Definition of Parallel and Serial pipelines used in Figure 7.
E. Robustness of Type Error Rate Control
In this experiment, we confirmed the robustness the proposed method for ov pipeline in terms of type error rate control by applying method the two cases: the case where the variance is estimated from the same data and the case where the noise is non-Gaussian.
E.1. Estimated Variance
In the where the variance is estimated from the same data, considered the same two options in type error rate experiments in $6 and Appendix D.I; number of samples and number of features. For each setting, we generated 10,000 null datasets (X, y), where Xiş ~ ~N(01)) 1), v(i,j) E [n)] x [d] and N/0 I.) and estimated the variance 6² as
a² 1 n d y X(x y/
considered three significance levels a 0.05, 0.01, 0.10. The results are shown in Figure 9 and our proposed method can properly control the type I error rate.
0.20
0,20
alpha=0.05
alpha=0.05
alpha-0.01
alpha-0.01
Type Error Rate
0.15
alpha-0.10
Erro Rate
0.15
alpha-0.10
significance levels
significance levels
0.10
0.10
0.05
0.05
8:80
8:80
100
200
300
400
10
20
40
number samples
number of features
(a) Number of Samples
(b) Number of Features
Figure 9. Robustness of Type I Error Rate Control, Our proposed method can robustly control the type error rate even when the variance estimated from the same data.
E.2. Non- Gaussian Noise
non- Gaussian noise, we considered the In the case where the noise is non Gaussian, we set n = 200 and = 20. As non- following five distribution families:
17

---

Statistical Test for Feature Selection Pipelines by Selective Inference
skewnorm: Skew normal distribution family.
exponnorm: Exponentially modified normal distribution family.
gennormsteep: Generalized normal distribution family (limit the shape parameter to be steeper than the normal distribution, i.e., 3 < 2).
gennormflat: Generalized normal distribution family (limit the shape parameter / to be flatter than the normal distribution, i... 3 2).
t: Student's t distribution family.
Note that all of these distribution families include the Gaussian distribution and are standardized the experiment.
To conduct the experiment, each distribution family, for / {0.01, 0.02, 0.03, 0.043. We then generated 10,000 null datasets (X, y), where Xij (0, 1), (i,,j) j) E [n] x [d] and ys vi E [n]] follows the obtained distribution. We considered the two significance levels 0.05, 0.01. The results are shown in Figure 10 and our proposed method can properly control the type I error rate.
first obtained distribution such that the 1-Wasserstein distance from W/, 1) is / in
0.20
0.04
skewnorm
Type Error Rate
0.15
exponnorm
0.03
exponnorm
gennormsteep
gennormsteep
gennormflat
EEror
gennormflat
0.10
significance level
significance level
0.05
0.01
0.01
0.02
0.03
0,04
0.01
0.02
0.03
0,04
Wasserstein Distance
Wasserstein Distance
(a) Significance Level 0.05
(b) Significance Level 0.01
Figure 10. Robustness of Type Error Rate Control, Our proposed method can robustly control the type error rate even when the noise follows non-Gaussian distributions.
F. Automatic Pipeline Construction based on Validaaino
In this section, we discuss cross-validation for pipelines. We consider selecting the pipeline from given set of candidates (Pi...., Ps} where S is the number of candidates. Note that this formulation is general enough to handle many cross- validation targets in unified form. For examples, (i) the case where only changing the regularization strength of lasso node, (ii) the case where changing the method of missing value imputation, and (iii) the case where changing the all structure of the pipeline (i.e., type and order of nodes).
Thereafter, we discuss how statistical inference changes when valisttaii is performed and how cross cross- validation can be formulated. Then, based on above discussion, Algorithm is extended be applicable the case of cross-validation.
F.1. Statistical Inference after Valiiatii
Changes in Statistical Inference As a formulation of statistical inference after cross- validation,, the discussion in $2 and $3 can done in exactly the same way, except with two changes: (i) the procedure for computing M and O (in $2 and $3, M and O are simply the outputs of given mapping representing target pipeline), and (ii) the dependence on the response vector y of which method to use for missing value imputation. This implies that the procedure for computing the truncation intervals Z in $4 can not be directly applied to the case of cross-validation.
Formulation of Cross Validation Procedure We consider the case where K-fold cross- cross-validation is performed. Let (X, y) be the observed data set and ((Ts, ke(K) be the K types of partition of training and validation sets, which
18

---

Statistical Test for Feature Selection Pipelines by Selective Inference
satisfies TR. V& 2|m Th V& 0, and T3 U V
[n]] for any [K]. Then, the cross-validation error (x y) for the
pipeline Ps. is defined
(x,
Σ kE|K]
V4
where Dx is the linear transformation matrix in the missing value imputation of the pipeline P.. (2, (T OT, X N.v,,,.MMM (Dx (D'))) NO.. and (Ma,k: Os,k) is the output of the pipeline Pa with input (Xr,, (D))) In K-fold cross-validation, the pipeline P.- is selected to minimize the cross-validation error (X, y), i.., s" = arg [[ss E, (X,)).
E22 Auto-Cutoo Cato-Conditioning for Cross- Validation
To conduct the statistical inference after cross-validation, it is suffice to have the procedure to compute the interval [L.,, U=} for any E IR which satisfy
Vr E [2, Uz).
arg min E, a + br) = arg min E, a bz)((= s(z));
+E(S)
sE[S]
P Pa(e)(X, br) P (a) [x, bz).
If we have this procedure, for any r E [La, U2]. the selected features and the detected outliers after selecting the pipeline by cross-validation from the data set (X, a + br) are invariant. Therefore, the Pselective can be computed in exactly the same way in $4 only by adding the condition xx(*) D as well as the condition Ma+bz My and Oatbz = Og- Hereafter, we provide the above procedure by extending Algorithm
For implementation of the above procedure, compute two intervals [L USY and [L Usel] for any E E which satisfy
Vr E [4_ [ arg min (,(X, a br) = arg min E, a + bz)(== s(z)), =E(S]
sE(S]
Vr E [Iae] U-el] Pa(e) (X (X, a br) = (P)()((X,,) bz),
respectively, and let L, max LY ) and U. = min U U
To compute the interval [Lg U we use Algorithm repeatedly. For any (8, k) E [S] x [K] and any E R. we compute the interval (=)) [[ which satisfy
Vr E (L(( U( Pa(Xn, (Dxa brbr)),) Po(Xn, (D'a Dzb2)).);
by using Algorithm 1 with input (Ps, Xr,, (D by =). Thus, if we consider the k-th term of the sum in E,(X, a + br) as a function r, then it becomes quadratic in r on the interval (L(**)) _(**)) Therefore, on the interval Oseis] [ke[k] (L," ,u("*)) the cross-validation errors {E, + br) are all quadratic in On this interval Aieis DRE[K] (*M)) U!*.)), the simultaneous inequalities for
(Xte) a + br) S E, a + br), vs E [S].
with s(z) = arg min Ex(X, a ba) become simultancous quadratic inequalities, which can be solved analytically to finally obtain the interval _ UT.).
To compute the interval [Lge]] U. we simply Algorithm with input (Po(s), X, X, b,
G. Examples of Implementations
We show an example of how the pipeline is implemented in our experiments. Listing 2 shows the implementation of the automatic pipeline construction scheme referred to as cv in the experiments ($6). Note that we can specify the candidates of the parameters for each operation and perform cross-validation determine the optimal pipeline by using method.
19

---

Statistical Test for Feature Selection Pipelines by Selective Inference
Listing Code example that defines automatic pipeline construction scheme referred to as cv in experiments. We can create an instance of manager class which handles identically structured pipelines, each with different hyperparameter set, simply by specifying (correpponding option lmmlli and option2_m an2_mllti Manager instances
each operation and its candidates of parameters create new manager instance which handles all the pipelines that each instance handles. To perform hypothesis testing after validaval,,tinnn we can call and inference method of manager instance sequentially.
can use the operator
import numpy as np from si4pipeline import
def optionl_multttt
-> iipelincManage:: x, y = initializee dataset
y irean_vaaue_impuaationn (x, y)
C soft_iped (X, 10.00.0.. 10.02, 0.018]]
X, y remove_ camoeeouutlierstee (X, y, 0)
X marginal serbiaiissrreeinggh y, [3, 5]) M)
X extract_
M1 stepwise feature selection (x, y, 12, 31)
M2 lasso [0.08, 0.12])
M union (M',, M2)
return construct_pipelinns (output-M)
def
-> PipelineManager X, y - _mitiaaline_dataaeeu
y oefiadte__egrrsssooommmmpaaaaooe (8,
X
(X, y, [3, 51)
X astract_featurest M)
C cook_distance (X, y, [2.0, 3.01)
X, y remove_outliers
y, 0)
X1 stepwise feature selection (X, y, 12, 31)
X2 lasso (X, y, 10.08, 0.12])
M intersection (M1, M2)
return constuct_pppplInnssttteee
manager (pticnl_multi)) option2 m)
x, y = np. no.rranom.normalliiis..
10)), pp. ron.radom (size=100)
manager. tune (X, y, num num_folds-2)
M, p_list = manager.inferenee (X, y, sigma=1.0)
20


Reviews:
[{"review_id": "aA77abH3lB", "reviewer": "Reviewer_ewmH", "summary": "This paper presents a statistical testing framework based on selective inference (SI) for assessing the significance of features selected through full feature selection pipelines. These pipelines may include steps such as missing value imputation, outlier detection, and feature selection. The key idea is to compute valid p-values by conditioning on the fact that the features were selected through a specific sequence of data-dependent operations. The proposed framework is modular and can accommodate a range of commonly used pipeline components. Empirical results on both synthetic and real datasets show that the method effectively controls Type I error and reduces false discoveries compared to standard, naive testing approaches—particularly when selection is driven by the data.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your feedback.\n\n> Line 110: \"AD – anomaly detection?\" → consider changing to OD - \"outlier detection\" for consistency with terminology used elsewhere.\n\nThank you for pointing this out. We change the term to OD in the revised manuscript.\n\n> Figure 3: It is difficult to distinguish between the performance of various methods, as many lines are overlapping. Consider using different linestyles, colors, or markers to more clearly show the differences.\n\nWe improve the figure to make it easier to distinguish between the different methods in the revised manuscript.", "claims_and_evidence": "Yes. The claims are supported by both theoretical arguments and empirical evidence. The experimental section demonstrates that naive approaches can yield misleading significance results, while the proposed method performs consistently under various setups.", "methods_and_evaluation_criteria": "Yes. The methods and evaluation criteria are appropriate for the problem. The use of synthetic data for controlled studies and real datasets for practical illustration is well-justified.", "theoretical_claims": "I had a quick look at the proof of Theorem 3.1, which characterizes the distribution of the test statistic under the selective inference framework. The derivation appears correct and follows known techniques in selective inference literature. The modular treatment of pipeline components is a nice extension of this theory.", "experimental_designs_or_analyses": "I have carefully looked at the experimental section in the main part of the paper. The design is clear, comparisons are appropriate, and the analysis effectively demonstrates the strengths of the proposed method.", "relation_to_broader_scientific_literature": "This paper extends recent work in post-selection inference to the setting of feature selection pipelines, where rigorous statistical testing is often lacking. Its main contribution is the development of a framework that supports modular pipelines—composed of multiple data-dependent preprocessing and selection steps—while still enabling valid p-value computation for selected features. This represents a meaningful advance beyond prior work focused on single-step selection procedures.", "essential_references_not_discussed": "None identified.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The application of selective inference to this setting is a compelling and timely contribution.\n\n2. The framework appears practical, modular, and generalizable to many pipelines used in practice.\n\n3. The paper is well written, and the problem motivation is very clear.\n\nWeaknesses:\n\n1. Figures (especially Figure 3) could be improved to make differences between methods more visually accessible.", "other_comments_or_suggestions": "Line 110: \"AD – anomaly detection?\" → consider changing to OD - \"outlier detection\" for consistency with terminology used elsewhere.", "questions_for_authors": "1. Figure 3: It is difficult to distinguish between the performance of various methods, as many lines are overlapping. Consider using different linestyles, colors, or markers to more clearly show the differences.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FM4CQibTSl", "reviewer": "Reviewer_LDea", "summary": "The authors propose an extension of selective inference techniques from single procedures (lasso, marginal screening) to pipelines. They develop a statistical test that they claim to have better power for data analysis pipelines with multiple, data-adaptive decision points.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your feedback.\n\n> It is not clear how well this procedure could truly generalize, if at all, to more complicated ML models or pipelines. The strong assumptions that are made about error normality and a fixed design matrix seem to be quite central to the validity of the statistic.\n\n> Could the authors say more about how this prototype is expected to be extendable to more complicated pipelines and datasets?\n\nConditional selective inference was initially developed targeting the problem of feature selection in linear models, but it has recently been applied to various problems such as clustering and anomaly detection. In this paper, as a proof of concept, we consider a feature selection pipeline for linear models, but we believe that the same concept can be applied to a wide range of other problems as well. All the analysis components considered in this paper are those for which the selection events can be characterized by sets of linear or quadratic inequalities. In addition to the nine specific analysis components in the paper, there are many other analysis components whose selection events can be characterized within this framework. Technicaly, it is easy to incorporate them as additional components of our current pipeline framework.\n\n> Line search is computationally expensive, as evidenced in the appendix fig 6, and for these simple datasets is already quite slow.\n\nAlthough the line search is computationally expensive, we expect that parallelization offers potential for significant time reduction. The vertical axis in Figure 6 shows the computation time needed for a single hypothesis test (i.e., calculate one p-value) on a single CPU core. Our preliminary experiments indicate that using 16 cores reduces this time to roughly one-fifth. We will add an explanation of this in the revised manuscript.", "claims_and_evidence": "The claims of the work are clearly stated and supported mostly by theorems. The data-based evidence relies on a few somewhat out-dated datasets, but is not central to the contribution of the paper. The authors appropriately cast it as having somewhat limited applicability (linear models with normal errors, specific feature selection steps), but with an eye towards generalizing to autoML pipelines, which is a good goal.", "methods_and_evaluation_criteria": "Yes, the proposed methods make sense. The evaluation criteria is also sensible. I like that the authors focused on power since a method like this may be prone to over-conditioning and power loss.", "theoretical_claims": "I checked proofs 3.1, 3.2, and 4.1, although I could have missed some details. Each of them seems clear. \n\n- 3.1 uses a standard selective inference trick to transform $Y$ and obtain the truncated normal distribution $T(Y)$\n- 3.2 is similarly aligned with selective inference theory to establish the uniformity of the p value\n- 4.1 uses some parametric programming arguments to define when selected features and detected outliers remain unchanged (i.e. when the update rules do not trigger a change)\n\nThe proofs appear to correctly apply techniques seen in related works (Lee et al 2016, Tibshirani et al 2016).", "experimental_designs_or_analyses": "As noted above, the only caveat with the experiments are that the datasets are somewhat out-dated. It is kind of an ancillary point to the main value of the paper.", "relation_to_broader_scientific_literature": "The paper is related to selective inference literature, and extends it from single procedures to pipelines. The paper seems clear about its contribution in this regard.", "essential_references_not_discussed": "None noted", "other_strengths_and_weaknesses": "The paper is clearly written and well reasoned. The value proposition is clear. To me it appears to be a good extension of prior work.\n\nThe main weaknesses I see are:\n\n1. It is not clear how well this procedure could truly generalize, if at all, to more complicated ML models or pipelines. The strong assumptions that are made about error normality and a fixed design matrix seem to be quite central to the validity of the statistic. \n2. Line search is computationally expensive, as evidenced in the appendix fig 6, and for these simple datasets is already quite slow.", "other_comments_or_suggestions": "None", "questions_for_authors": "Could the authors say more about how this prototype is expected to be extendable to more complicated pipelines and datasets?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PVnKkky5ed", "reviewer": "Reviewer_sBGD", "summary": "This paper presents a statistical test to assess the significance of data analysis pipelines, which transform raw data by integrating various analysis algorithms. The paper specifically focuses on feature selection pipelines for linear models, which are composed by value imputation algorithms, outlier detection algorithms, and feature selection algorithms. The proposed statistical test builds upon the technique of selective inference, and it is theoretically shown that it can control the probability of false positive feature selection at any desired level. Experiments on synthetic and real data are provided to demonstrate the validity and effectiveness of the proposed statistical test.\n\n## update after rebuttal ##\nThe rebuttal from the authors has supported my initial recommendation to accept the paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your feedback.\n\n> The probability of a missing value was set to 0.03, which seems quite low. How (or better does) such value can affect the performance of the statistical test?\n\nThis probability was set only for experimental convenience. The validity of the proposed method, specifically its control of the Type I Error Rate, is unaffected by how missing values are present in the response vector. However, while a higher proportion of missing values naturally leads to a decrease in statistical power, the proposed method maintains its advantage over the comparison methods.\n\n> I am not entirely sure I understood the real data experiments, and more specifically, why it is necessary to generate random datasets from each original dataset to illustrate the performance of the proposed test?\n\nWe need multiple datasets to evaluate the performances of statistical tests (type I error rate and power). Therefore, for experiments on real datasets, we created multiple datasets by randomly resampling the original dataset. Such an evaluation is a standard approach for assessing statistical tests on real datasets.\n\n> The real-world datasets considered have very few features (<15), so I am wondering how the proposed test will behave in settings with larger number of features.\n\nThe validity of the proposed method and its superior statistical power compared to baseline methods remain consistent even as the number of features increases. Please refer to the results of the synthetic data experiments on high-dimensional data provided in the appendix (see Figure 5 in Appendix D).\n\n> Since the focus of the paper seems to be on feature selection pipelines for the moment, would not it make sense to compare with any of the baselines below?\n> - Knockoff-based feature selection\n> - Meinshausen, Nicolai, and Peter Bühlmann. \"Stability selection.\" Journal of the Royal Statistical Society Series B: Statistical Methodology 72, no. 4 (2010): 417-473.\n\nMethods such as Knockoff-based feature selection and Stability Selection are designed to control the False Discovery Rate (FDR), defined as the expected proportion of falsely selected features within the selected set. In contrast, our proposed method performs hypothesis testing on individual features within a selected feature set. Because they address different inferential goals (set-level FDR vs. individual feature significance), the methods are not directly comparable.", "claims_and_evidence": "The paper provides convincing theoretical and experimental results to support the validity and effectiveness of the proposed statistical test.", "methods_and_evaluation_criteria": "The paper uses both synthetic and real-wold datasets to assess the performance of the statistical test. In addition, it considers three baselines, a version of the proposed test without parametric programming, a classical z-test, and the Bonferroni correction. In general, the proposed methods and evaluation criteria make sense for the problem at hand. Since the focus of the paper seems to be on feature selection pipelines for the moment, would not it make sense to compare with any of the baselines below?\n\n- Knockoff-based feature selection\n- Meinshausen, Nicolai, and Peter Bühlmann. \"Stability selection.\" Journal of the Royal Statistical Society Series B: Statistical Methodology 72, no. 4 (2010): 417-473.", "theoretical_claims": "The proofs of the theoretical claims in the paper seem correct.", "experimental_designs_or_analyses": "Overall, the experimental design seems correct. However, there are certain details that I am unclear about:\n\n1) The probability of a missing value was set to 0.03, which seems quite low. Is it possible that selecting such value to be low can somehow affect the performance of the statistical test?\n2) I am not entirely sure I understood the real data experiments, and more specifically, why it is necessary to generate random datasets from each original dataset to illustrate the performance of the proposed test?\n3) The real-world datasets considered have very few features (<15), so I am wondering how the proposed test will behave in settings with larger number of features?", "relation_to_broader_scientific_literature": "The proposed statistical test builds upon the selective inference statistical technique to provide a new way of assessing the significance of data analysis pipelines. The specific focus is on feature selection pipelines for linear models.", "essential_references_not_discussed": "The paper presents a summary of the most related works in selective inference and AutoML literature.", "other_strengths_and_weaknesses": "Strengths:\n(+) Well written paper that builds upon selective inference to assess statistical significance of data analysis pipelines.\n(+) Includes experiments with both synthetic and real-world data\n(+) The figures provided facilitate the understanding of the proposed approach.\n\nWeaknesses:\n(-) Some parts of the experimental study are unclear.", "other_comments_or_suggestions": "- Line 186: remove \"is\" from \"Making the p-value is a random variable\"\n- Line 246: replace \"can\" with \"are\" in \"Note that DAGs can..\"\n- Line 310: replace \"currently\" with \"current\"\n- Line 312: replace \"details\" with \"details\"\n- The sentence \"Missing values .. of 0.03\" is repeated twice in the Experimental Setup section.\n- In Section 6, \"Methods for Comparison\", for op1 and op2, I believe that the reference should be to Figure 1, not Figure 2.", "questions_for_authors": "I have already stated these questions, but I restate them below:\n\n1) The probability of a missing value was set to 0.03, which seems quite low. How (or better does) such value can affect the performance of the statistical test?\n2) I am not entirely sure I understood the real data experiments, and more specifically, why it is necessary to generate random datasets from each original dataset to illustrate the performance of the proposed test?\n3) The real-world datasets considered have very few features (<15), so I am wondering how the proposed test will behave in settings with larger number of features.\n4) Since the focus of the paper seems to be on feature selection pipelines for the moment, would not it make sense to compare with any of the baselines below?\n- Knockoff-based feature selection\n- Meinshausen, Nicolai, and Peter Bühlmann. \"Stability selection.\" Journal of the Royal Statistical Society Series B: Statistical Methodology 72, no. 4 (2010): 417-473.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers had a very positive impression of this paper: The underlying research question was considered interesting and of high practical relevance, proposed concept of selective inference seems to be well-motivated, clearly described and technically sound, the experiments are (mostly) convincing. There were some minor points of criticism concerning some details of the experimental study or the style of some figures, and one potentially more significant critical comment concerning the need for strong statistical assumptions in the theoretical analysis. In summary, however, to me the positive aspects clearly outweigh the negative ones, so I recommend acceptance of this paper.
Decision: Accept (oral)