ICML 2025

Statistical Test for Feature Selection Pipelines by Selective Inference
Tomohiro Shiraishi * 1 2 Tatsuya Matsukawa * 1 Shuichi Nishino 1 2 Ichiro Takeuchi 1 2
Abstract
A data analysis pipeline is a structured sequence
of steps that transforms raw data into meaning-
ful insights by integrating various analysis algo-
rithms. In this paper, we propose a novel statis-
tical test to assess the significance of data analy-
sis pipelines. Our approach enables the system-
atic development of valid statistical tests applica-
ble to any feature selection pipeline composed of
predefined components. We develop this frame-
work based on selective inference, a statistical
technique that has recently gained attention for
data-driven hypotheses. As a proof of concept,
we focus on feature selection pipelines for linear
models, composed of three missing value imputa-
tion algorithms, three outlier detection algorithms,
and three feature selection algorithms. We theoret-
ically prove that our statistical test can control the
probability of false positive feature selection at
any desired level, and demonstrate its validity and
effectiveness through experiments on synthetic
and real data. Additionally, we present an im-
plementation framework that facilitates testing
across any configuration of these feature selection
pipelines without extra implementation costs.
1. Introduction
In practical data-driven decision-making tasks, integrating
various types of data analysis steps is crucial for address-
ing diverse challenges. For instance, in genetic research
aimed at identifying genes linked to a specific disease, the
process often begins with preprocessing tasks such as filling
in missing values and detecting outliers. This is followed
by screening for potentially related genes using simple de-
scriptive statistics and then applying more complex machine
learning-based feature selection algorithms. Such a system-
atic sequence of steps designed to analyze data and derive
*Equal contribution
...