Knockoff-Inspired Feature Selection via Generative Models
Authors: Marco F. Duarte
Siwei Feng
Venue:  (2020)
Keywords: feature selection
variable selection
knockoff variables
supervised learning

Abstract: We propose a feature selection algorithm for supervised learning inspired by the recently introduced 
knockoff framework for variable selection in statistical regression. While variable selection in statistics aims 
to distinguish between true and false predictors, feature selection in machine learning aims to reduce the 
dimensionality of the data while preserving the performance of the learning method. The knockoff framework 
has attracted significant interest due to its strong control of false discoveries while preserving predictive 
power. In contrast to the original approach and later variants that assume a given probabilistic model for the 
variables, our proposed approach relies on data-driven generative models that learn mappings from data 
space to a parametric space that characterizes the probability distribution of the data. Our approach 
requires only the availability of mappings from data space to a distribution in parametric space and from 
parametric space to a distribution in data space; thus, it can be integrated with multiple popular generative 
models from machine learning. We provide example knockoff designs using a variational autoencoder and 
a Gaussian process latent variable model. We also propose a knockoff score metric for a softmax classifier 
that accounts for the contribution of each feature and its knockoff during supervised learning. Experimental 
results with multiple benchmark datasets for feature selection showcase the advantages of our knockoff 
designs and the knockoff framework with respect to existing approaches.

Under review as a conference paper at ICLR 2020
KNOCKOFF-INSPIRED FEATURE SELECTION VIA GENERATIVE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
We propose a feature selection algorithm for supervised learning inspired by the recently introduced knockoff framework for variable selection in statistical regression. While variable selection in statistics aims to distinguish between true and false predictors, feature selection in machine learning aims to reduce the dimensionality of the data while preserving the performance of the learning method. The knockoff framework has attracted significant interest due to its strong control of false discoveries while preserving predictive power. In contrast to the original approach and later variants that assume a given probabilistic model for the variables, our proposed approach relies on data-driven generative models that learn mappings from data space to a parametric space that characterizes the probability distribution of the data. Our approach requires only the availability of mappings from data space to a distribution in parametric space and from parametric space to a distribution in data space; thus, can be integrated with multiple popular generative models from machine learning. We provide example knockoff designs using variational autoencoder and a Gaussian process latent variable model. We also propose a knockoff score metric for a softmax classifier that accounts for the contribution of each feature and its knockoff during supervised learning. Experimental results with multiple benchmark datasets for feature selection showcase the advantages of our knockoff designs and the knockoff framework with respect to existing approaches.
INTRODUCTION
The availability of digital sensors and low-priced storage, computation, and communication channels is becoming increasingly pervasive. As a result, emerging applications involving extremely large datasets are now feasible. Thus, we now often rely on data-centric inference, in which large amounts of data are concentrated and processed via machine learning (ML) algorithms, based on suitable data models, to extract relevant information from the observations. However, such approaches often suffer from high computational complexity well as limits on available storage, communication, and processing resources. To alleviate such issues, dimensionality reduction (Samet) (2066) is commonly applied at the beginning of the data processing pipeline. Dimensionality reduction also addresses multiple issues with the performance of learning and inference from -igh-nimensional data that are commonly referred to as the curse of dimensionality or Hughes Phenomenon (Jain & Zongker 1997.
Pavlenko 2003; Guyon & Elisseeff 2003).
In the most demanding applications involving real-time data processing or complex sensing archi- tectures (e.g., low-power sensing, streaming data analytics. particle physics, and genetics), it may not be feasible perform the sensing and/or computation required by most existing approaches to dimensionality reduction during data acquisition. Thus, the practitioner is limited in these cases to feature selection (FS) (Guyon & Elisseeff 2003 Liu & Motoda 2008), i.e., to selecting the subset of the data dimensions that most relevant in information extraction. These dimensions are the only ones to be sensed to provide a simple form of dimensionality reduction.
FS has strong connection with the variable selection (VS) problem in regression analysis for statistics (Hastie et al., 2009). where the goal is to find subset of the predictive variables that capture statistical dependence algorithm is to identify as many of the predictive variables on which the response variable is truly dependent (measured by its predictive power) while minimizing the number of selected predictive
the response variable of interest. Informally, the goal of VS

---

Under review as a conference paper at ICLR 2020
variables that are actually unrelated (measured by the false discovery rate); such strong distinction between features is seldom observed in ML, where the focus of FS is to preserve performance while maximizing dimensionality reduction.
The recent development of VS via knockoffs (Barber & Candès 2013
2019: Weinstein et al. 2017
Barber et al. 2019 has attracted significant attention in the statistics community due to its guaranteed control of the false discovery rate, as well as its applicability to arbitrary statistical models for the variables. This new framework relies on the creation of samples of knockoff variables that provide counterparts to the original variables on existing training samples. The knockoff variable design must meet certain requirements: first, knockoff variables must follow the same probability distribution as the original variables, but they must be independent from the labels to be predicted, conditioned on the original variables; second, they should be pairwise uncorrelated with the original variables in order to provide sufficient predictive power; finally, swapping any two corresponding entries of the original and knockoff sample vectors must not change the likelihoods of these vectors. The framework then relies on numerical scores to compare the statistical dependence of original and knockoff' variables with the response variable, discarding those variables for which the scores of these two variable versions (original and knockoff) are indistinguishable. While the original framework focused on jointly Gaussian variable models and Lasso-based linear regression, it has since been extended by new knockoff variable constructions for more sophisticated data models, such Markov models and Bayesian networks (Sesia et al. 2019 Gimenez et al., 2018: Romano et al.) 2019: Fan et al. 2019; Liu & Zheng 2018: Jordon et al. 2019)). as well as proposing additional variable statisties for other regression schemes (Lu et al. (0118); furthermore, technical conditions on the data models and
variable statistics have been simplifted while preserving the original performance guarantees (Candès
et al. 2018).
In this paper, we propose a novel framework for FS in ML inspired by the knockoff framework for VS that is geared towards applications featuring large-scale, high-dimensional datasets. We leverage the flexibility of the framework by designing knockoff feature constructions and scores based on data models that are popular in MI. and driven by large-scale datasets. More specifically, we leverage generative models that provide a data distribution characterization that is compatible with the knockoff VS framework. Examples of knockoff's generated according to methods from the literature and from our proposed method are illustrated in Figure I for multiple samples from several benchmark image datasets, where it is clearly shown that the properties required for knockoff variables are better met by our constructions than by those previously proposed. It is worth remarking that since the analytical framework for VS performance evaluation does not translate FS, the proposed knockoff generation methods and scores are best described as heuristics.
This paper's contributions can be summarized as follows. First, we propose generic algorithm for knockoff variable design that assumes the existence of a generative parameterized data model composed of two mapping functions: one from the data feature space to the space of probability distributions for owwddimensional parameter, and another from the parameter space to the space of probability distributions for the data. Second, we present example implementations of this algorithm using two standard data-dríven models in ML: variational autoencoders and Gaussian process latent variable models. Third, we propose a knockoff- scoring scheme that is designed for FS for softmax classification. Fourth, we provide numerical examples that compare the performance of the proposed FS schemes against multiple baselines, including approaches based on previously proposed knockoff-basse VS designs. Finally, we discuss open questions and directions for future work.
2 BACKGROUND
Feature Selection (FS) (Guyon & Elisseeff 2003 Liu & Motoda 2008) aims to identify subset S2 of the indices {l, N) for the entries of the original feature vector x = ",,,,..,,,]] E R so that the resulting vector xn E [R best preserves the dataset structure relevant for the MI. problem at hand. Three classes of FS approaches have been described in the literature (Guyon & Elisseefi 2003] Liet al. 20117): filter methods, wrapper methods, and embedded methods. Of particular interest to this paper are filter methods, which consider set of J labeled data examples { 1 and compute score for each dimension ((n) based the values zj,n and y, for each data point. The features then ranked by this score and those with highest scores compose the set Q2. Common scores include Fisher's prediction score, the data-to-label correlation, the mutual information [I Yj), etc.
2

---

Under review as a conference paper at ICLR 2020
8
2 6 9
8
2
6 9
8 2 4 9
0
0
9
8
0
7
9
6 7 0 6
6
7
0
6
6
7
0
6
6
7 7 3
0 0 1
6
8
7
3
7
5 1/3 9
5
4 3 7
5
4
3 9
8
9
/
/
3 3 7
3
3
7
3
3 1 7
7
)
7
(a) Original
(b) Gaussian
(c) Deep
(d) VAE
(e) GPLVM
Figure 1: Examples of knockoffs generated from (top to bottom) MNIST; COIL20,, and Yale dataseıs with several data models: (a) Original data, 16 sample images. (b) Multivariate Gaussian model knockoffs (Barber
& Candes 2019) cf. Section 2
(c) Deep knockoff's (Romano et al. 2019). (d) Variational autoencoder (VAE)
model Knockofls, Section
Gaussian process latent variable model (GPLVM) knockoffs, Section 3
Variable Selection (VS) in statistics is similar to FS (Hastie et al. 2009). Consider a set of predictive variables {#1, .N\ } (e.g., the entries of the vector x), and a response variable y (e.g., the label or parameter to be estimated), and assume that the data points and matching labels correspond to independent and identically distributed (i.id.) samples of an underlying joint probability distribution ((x,)). The goal of VS is to determine the set of true predictors {z; jejhenn so that the response is statistically independent of the remaining predictors conditioned on the chosen predictors, i.e., y I Xnc Xx; the variables Xne are often referred to null variables. The selection mechanism should have high power (e.g., high probability of the true predictors Q being selected in a set estimate ②) while minimizing the false discovery rate (FDR) (e.g., the expected fraction El|$ 2 /// of null variables included within the selected variable set). Standard methods, e.g., (Benjamini & Hochberg 1995), require knowledge of the conditional distribution p( (()))
A recent framework for VS based on the construction of knockoff variables (Barber & Candès 2015; 2019; Weinstein et al.) 2017 Barber et al.) 2019 Candès et al., 2018) has been shown to control the FDR while being applicable
wide variety of cases without requiring any knowledge of the conditional response distribution p(y x). This generality of the approach is given by its single prerequisite knowledge of the distribution of the data prior p(x). The knockoff variables x = (ă,.... . } are constructed for each data sample using this prior so that any swap of a variable with its knockoff does not affect the data distribution. More formally, denote by (x, x) swap(()) the pair of vectors (x, x) where the entries of each of the vectors corresponding to the indices has been swapped with the other. The knockoff variables & are designed so that (i) (x, s)) = (x, X), where = denotes equality in distribution (and in practice, over the sample distribution); and (Mi) is independent of y given x (which is trivially met by ignoring y during the design of the knockoffs).
After knockoff samples are created for the VS training dataset, the framework relies on the com- putation of variable statistics V, v. ((X, X), Y) for each predictor variable that may depend on all the original variables, all the knockoff variables, and the label; here X, X and Y denote the set of samples of the original and knockoff variables and the corresponding labels, respec- tively. The variable statistics generator must have a flip-sign property: the sign of the variable must change when the corresponding entries of the vectors are swapped, i.c., v, ((X, {)swap()) Y) = (X,, Y) for each n. Flip-sign variable statistics can be easily obtained by creating variable importance scores ()) ((s1,..., 8N), (81,..,,5 .,)))))))) 1(X, X), Y), measuring the impor-
3

---

Under review as a conference paper at ICLR 2020
tance of the corresponding variables for the set of sample labels Y, that obey the swap property (((X, X) swap(n): Y) (s, (s,5) swap(() for each n; can then define Un Sn šn, VS proceeds by finding magnitude threshold T over the values of v = [e1, uN] that yields a sufficiently small ratio q = (n= Un < // : U/ > H (e.g., the ratio between the number of indices with negative evidence and the number of indices with positíve evidence), since i gives proxy for the
target FDR (Barber& & Candès 2019
Candès et al. 2018).
The promising recent development of model-X knockoff' (Candès et al. 2018 simplifies the original framework above by showing that knockoffs only require distribution invariance to swapping on single variable basis. In other words, knockoff's can be constructed simply by sampling each knockoff variable according to the conditional likelihood of the original variable given the rest of the original variables and the previously obtained knockoff variables, p(z,, [ 1::-11 1im-11 11:-1: x_n): ); therefore, each knockoff variable can be generated independently in particular sequence. The resulting simplicity of the model-X knockoff' framework provides an exciting opportunity for the use of modern data models that aim to learn the distribution of the data x in novel schemes for knockoff generation.
An autoencoder is a neural network that aims to obtain a -ow-dimensional data representation by (i) setting both the input and output of the network to be the data x itself, so that the network learns to recreate the data based on any of the hidden layer representations; and (ii) including a specific hidden layer of low dimensionality whose output activation z provides a compact representation from which the data can be recovered. The neural network is thus naturally divided into encoder and decoder portions. Autoencoders have been used in ML to provide compact data representations that are then fed into modern ML algorithms. However, the mapping obtained by the neural network is deterministic, and does not provide information autoencoders (VAEs) inherit this name because they follow the structure mentioned above. However, the -indeemnsiooal representation for the data follows a prescribed probability distribution p(z)- Consequently, the encoder and decoder aim to learn the forward and backward data distributions p(z|x)
the distribution of the data p(x). Variational
x) and p(x(z) (Doersch) 2016).
More specifically, we assume that there exists a data generating function x - fzz; x that provides map between parameter value and the expected signal conditioned on z, where 9 denotes the various model parameters. Practical VAEs leverage the common Gaussian assumption for the conditional data and parameter distributions. Under such a probabilistic model, the VAE network aims to learn two separate functions in its two halves: the encoder learns an independent-entry multivariate Gaussian approximation Q(z 8) N(zz fx)x,00 Σx)x,e) for the probability distribution P(z|x, 0). Thus, the low -inddimesioonal representation learned by the encoder in its narrowest layer corresponds to the approximated mean Mx|x R E/z|x, 0) and the diagonal of the variance matrix
θ X
Ex/x,0 ~ Var(zix, 0). The decoder learns the mapping x = f(z 9) = E|x/z, 0 (Doersch 2016).
Nonlinear manifold models also aim at learning mappings from the nigh-dimensional space containing the data of interest to -in-dimen representation space, which is usually described in terms of providing a parametrization of the data. As with autoencoders, the large majority of approaches are deterministic, as they aim at preserving the local geometry of the data in the high dimensional space. Furthermore, almost all existing methods for manifold learning do not provide a bidirectional mapping so that new data samples can be obtained from arbitrary representations, lacking a generative model to obtain data from arbitrary points in the parameter space.
In contrast, Gaussian process latent variable models (GPLVMs) (Lawrence) 2003) have been proposed to provide a probabilistic nonlinear mapping between the parameter space and the data space, inspired by probabilistic principal component analysis and kernel methods, and scalable to large-scale datasets (Hensman et al., (013). As in principal component analysis, it is first assumed that each data sample x can be written as x Wz + e, where W is the basis for subspace approximation of the data, z is a low-dimensional parametrization vector, and e is the model error. It is further assumed that z (O I), that (O0 3I), and that each row of W is an i.i.d. zero- Proomman Gaussian vector with covariance aI.
Given a data set X maximum likelihood estimation with the conditional distribution that marginalizes W:
[x] X2....,, the corresponding parameter vectors Z can be obtained via
exp - trace K K-' xx"x))
p 0)
(2T)
DN
K
4

---

Under review as a conference paper at ICLR 2020
Algorithm 1 Knockoff Construction from Arbitrary Probabilistic Generative Model (KпCAPGM)
Input: probability distributions p(x|z, 8), p(z)x, 0), input features
Output: output knockoff features
1: initialize knockoff &
2: for n 1,2, N do
3: marginalize nn from the distribution p(z]x, 0) obtain p(z)x_n= 0)
4: obtain z" as a sample from p(z|X_n. .n.
5: obtain
sample from p(x(z", 1/z22000
6: update knockoff vector x: 3x Z
7: end for
8: return knockoff X
where K = a + I. In order to introduce nonlinearity in the map between z and x, the matrix K is instead rebuilt using kernel space representation (e.g., with radial basis function kernel): Ki,- = k(z, Zz) = a exp (-3
Zj 111 + - .o.j. where i, are indices of the training {ar, 33 7 and via an iterative method that alternatively uses maximum likelihood for 0 and maximum posteriori estimation for Z with the conditional likelihood of the data X given above, respectively (Lawrence) 2003 2005).
dataset X. GPLVM training obtains the parameters
KNOCKOFF FEATURE DESIGN FROM GENERATIVE MODELS
We introduce basic algorithm for the construction of knockoff features using modern generative models from ML for rich data sources that aim to express the underlying distribution of the data. Note that in contrast to the prior work's 's focus on VS with Gaussian and Markov chain models, the models used here have shown be significantly more accurate for datasets of interest, as evident by the significant performance improvements obtained in MI. algorithms that leverage these models.
Our proposed algorithm leverages generative data model that relies on the construction or estimation of two mappings between the data space x E X' and parametric space z E Z, and the probability spaces for their counterparts, respectively, to express or approximate the conditional distributions p(x|z,0) 0) and p(z|x, 0), where 0 represents the model parameters. Our procedure is inspired by the simplified framework of model-X knockoffs, which generates the knockoff sample entries (x,) lexicographically by sampling from the distribution p(r, 11 x.n). We thus proceed iteratively over the data dimensions by (i) marginalizing the parameter space distribution x/z/x.... 0) over the data feature dimension under consideration; (iz) mapping mixed knockoff/original feature data vector to the parameter space, e.g.. drawing a sample z from p(z I I:N
0): (iii) leverag- ing the inverse mapping p(x|z", 0). from the parameter space the data space, infer the missing values for the marginalized dimension x,; (iv) assigning the recovered data values to the knockoff feature entry X,, and preserving for the next iteration of the algorithm. This routine formalized as Algorithm [ while the process described there does not take into account the values of the so- so-far discarded dimensions of the original data X we note that the generative model used compute new knockoffs samples is trained over the original data samples, providing dependence of the new knockoff entry value on the original data distribution and the knockoff entry values computed so far. We proceed by describing two instances of this algorithm that leverage popular generative models.
Knockoffs from VA A VAE trained on a dataset of interest directly provides us with the relevant data probability distributions for Algorithm [
We begin by considering the marginalization p(z) x x_)).
Lemma 1. Assume that x|z ~ N (H 2. z ~ IO, and z|x ~ N ( Expx) (e.g., the VAE model). Then p(z)x_.)) x p(z(x), where x(m] = 0 for = n and x[m] x/m] for m # n.
Proof. We begin by noting that
p(x
p(x)
p(x-n1l
and p(x_n)= m)=
/ /2Txx/2eepp
Mx]]|n]2 20
Cexp
NT P-n [n)
202
5

---

Under review as a conference paper at ICLR 2020
Table 1: Details of datasets and experiments used this paper.
Dataset
Features
Samples
Classes
Type
Training set size
VAE size
BASEHOCK
4862
1988
Text
1024
Caltech101 (VGG19)
4096
3000
100
Image Features
1000
1024
COIL20
1024
1440
20
Image
720
512
Cub (VGG19)
4096
9750
195
Image Features
1950
1024
Isalet
617
1560
26
Audio
780
128
MNIST
784
27200
10
Image
1000
PCMAC
3289
1920
2
Text
960
1024
RELATHE
4322
1426
2
Text
713
1024
Yale
1024
165
15
Image
90
256
where P-n = E[
Mx
[x(n))(xx (x) and C' is normalization constant. We then consider
the marginalization
p(z|x_n) p(x_n/z)p(( p(((2)p(z) C. exp p(x_n)
Mxlu[1]2 C'p(z|xx,
p(x)2 exp
2a
Lemma implies that the distribution learned by the VAE already provides us with a proxy for the necessary marginal, if we set the value of = 0 in the input of the VAE. We then proceed to obtain a sample z" p(z]x_n, f), and use the decoder to obtain new data sample x" drawn from the distribution p(x(z"). We set the knockoff feature H, 2ú which is akin to "inpainting" the marginalized dimension from x according to the learned data distribution p(x[z"). Figure d)) illustrates the outcome of this process for several examples from three image datasets with VAE hidden layer and training dataset sizes given in Table
The GPLVM provides generative probabilistic model that gives dis- tribution for the data vector x given the value of the parameter z and the model parameters 0 trained from X. One show that the underlying distribution is given by p(x|2", 0) X(x|XK-<k* o
Knockoffs from GPLVMs:
here k" has entries ky = x(z,, z where Z; the low-dimensional embedding for the training sample x,, and of = k K K-' While it is difficull to compute the additional mapping p(z|X, 0) required by Algorithm[ [uue the use of the kernel, we propose heuristic that replaces the sampling procedure by maximum posteriori estimate procedure based on the distribution above, i.e., we return the value of z that maximizes p(z) (2) x, f) p(x|z. z. f))((z) ())p(((0), This procedure can be interpreted as out-of-samplee extension of the GPLVM manifold model for the point x, e.g., method that provides an embedding for new (e.g., testing) data points into the low-dimensional pa- rameter space established by the manifold. The independence assumption implicit in the conditional distribution p(x /z", 6) simplifies the marginalization required by Algorithm|] Figure [[e) illustrates the outcome of this process for several examples from three image datasets using GPLVM featuring a 10-dimensional parameter space and with a training set of size given in Table
4 KNOCKOFF-BASER FEATURE SELECTION FOR SUPERVISED LEARNING
Existing importance scores evaluate original and knockoff variables have been limited regression (i.e., estimation) problems. In particular, variable statistics have been developed for logistic regres- sion (Sesia al. 2019 Gimenez al.) 2018). linear regression via lasso and group lasso (Barber &
Candès 2015) 2019 Candès et al.) 2018) Sesia et al. 2019 Gao et al. 2018 Katsevich & Sabatti 2019), Bayesian hierarchical regression (Candes al. 2018), and deep learning estimators (Lu et al.
2018). In contrast, our consideration of knockoff' for FS focuses on general ML problems; as starting point, we will consider standard supervised learning.
Our feature importance scores are designed for the application of softmax classifier, which uses exp(x" TTxxxx"""" w²)
class scores defined as (x) = Σş_, exp(xT w for each of the classes 1,.., C. The sample x is
then assigned the predicted label 6(x) = arg max. lxxx; the parameter weights w are trained using cross-entropy loss between the scores and the sample labels. We propose simplified scoring scheme for FS, reminiscent of the concept in Lu et al. (2018), by expanding the softmax classifier class scores expanded input, consisting of concatenation of the original and knockoff features:

---

Under review as a conference paper at ICLR 2020
1 (x, x) Σeー exp(x w- + w Importance scores for each original and variable knockoff can
exp(x w )
then be obtained as the sums of squares šn = - (vxh²2 and šn E. (1,5², respectively. Similarly to VS, the feature statistics vector is simply obtained as the difference of the original and knockoff feature importance scores V === S.
RELATED WORK
There are several recent contributions to the knockoff literature for VS that are related to our work. Liu & Zheng (2018) proposed knockoff variable construction for regression based autoencoders; however, the design of these knockoffs does not make the value of each knockoff variable independent from the original variable. In contrast, our approach is applicable to many generative models and marginalizes the feature under consideration before starting the construction of the corresponding knockoff. Jordon al. (2019) proposed construction for knockoff variables based generative adversarial network (GAN). The GAN's generative network uses the original variables and random noise as inputs and generates the corresponding knockoff variables, while its discriminative network takes as input a subset-swapped version of the concatenated original -nockofff feature vector and attempts to estimate the set of swapped variables. The approach also uses a mutual information estimate in the training objective function to promote independence between original and knockoff variables. In contrast to the data models considered in this paper, GANs do not aim to obtain a low-dimensional parameterization of the data or its distribution; furthermore, we are not aware of an implementation of this method being available for comparison.
More recently, Romano et al. (2019) proposed the design of knockoff' based on a deep learning network that is also inspired by the GAN architecture. Similarly to the original Gaussian knockoff constructions of Barber & Candès 2015 2019); Candès et al. (2018). the formulation of the objective function for their proposed algorithm is still centered on the matching of moments between the distributions for the original and knockoff variables. We compare the performance of this approach with our own in the sequel.
Separately, [u et al. (2018) proposed framework for the design of variable statisties for estimation based on a deep learning network. The statistics are obtained by augmenting the deep learning network with pair of pre-processing layers, featuring an input of original and knockoff variables, that merge each pair of matching variables (original/knockoff) at the first layer, and assigns trainable weights to each of them; the output of this layer is then mapped directly to the input of the original deep learning network. This straightforward approach allows for easy implementation of variable importance scores in cases where the first action on the input features is a linear projection, as the case with the softmax classifier leveraged in our experiments.
6 EXPERIMENTAL RESULTS
To test the performance of our proposed knockoff designs, we consider several datasets commonly used as benchmarks for FS in ML. (Li et al. 2017 2019). including image, test, and audio data, under softmax classifier setup. Detailed properties of those datasets are summarized in Table [ We combine the variable relevance scores of Section 4 for the aforementioned softmax classifier with the two knockoff construction schemes proposed in Section 3, as well as with two previously proposed and available constructions: second order/Gaussian approximation (Candès et al., 2018) and deep knockoffs (Romano et al., 2019). The knockoff construction schemes proposed this paper were implemented in Python using the Keras and GPy toolboxes (GPy) 2019); we used the implementations available from (Sesia & Romano| 2019) for the two existing constructions tested here. Additionally, we tested the performance of several FS approaches from the literature (Duda et al.,
Nie et al. 2010). using implementations 2017 Li & | 2019). The datasets were available from the feikit- toolbox (Li et al.
2001 Robnik-Siin SabbnikSikonja&& & Kononenko 2003; Wright 1965
randomly split in half into training and testing sets, except in cases where equal representation of
classes requires a smaller training set, or when the majority of the FS algorithm tests lasted more than 30 CPU-days; see Table D for details.
For knockoff construction, we implemented VAES with one hidden layer for both the encoder and decoder of the sizes indicated in Table for each dataset; the low low-dimensional representation is five-dimensional for all datasets as well. The VAE nodes use the ReLU activation function; for the VAE, datasets are normalized to be in the range [0.1]. We also implemented GPLVMs with parameter space dimension 10 for all datasets. The latent dimensionalities of the models tested were selected by evaluation of the quality of the reconstructions obtained from the original data.
7

---

Under review as a conference paper at ICLR 2020
MRMM techit
+ Seutre Occorrt
can Braet
do 22 20
20
175 -00
(a) Basehock
(b) C0 (VGG19)
(c) COIL20
JIGE dicootts
CAuura
AA kow
Qusteh
Guclat's
Pakee
Mo X
(d) Cub (VGG19)
(c) Isolct
(f) MNIST
cna Goclars
Prige
Fibit
STeaker Seinec (h) RELATHE
(g) PCMAC
(i) Yale
Figure 2: Performance of a deep learning softmax classifier for multiple datasets as function of the percentage of features selected by multiple baseline and knockoff-bass algorithms. The algorithms tested for comparison Gaussian knockoff's (Candès et al. 2018). deep knockoff's (Romano al. 2019). Fisher score (Duda et al.
2001). ReliefF (Robnik-Sikonja & Kononenko 2003). F-Score [Wright 1965) and RFS (Nie et al. 2010). For datasets featuring rich training data, knockoff based algorithms outpertorm baselines, with Gaussian knockoffs outperformed their counterparts.
We evaluate the performance of the same softmax classifier trained and tested with the features selected by each of the FS and VS frameworks using the score of Section 4 where the percentage of selected features varies from 2% to 20%. Figure 2 shows that the proposed knockoff- based FS methods outperform the baseline approaches in almost all instances. Note that results are missing for some of the baseline methods because the relevant simulation did not complete after an upper limit of 30 CPU-days. We see that VAE, GPLVM, and deep knockoff's often provide better performance than standard Gaussian knockoffs. In some cases, the number of samples available to fit the data models used in knockoff generation is not sufficient to obtain good performance for knockoff- based algorithms. We also conjecture that the performance of the knockoff- based models studied is dependent on the suitability of the underlying generative model for the dataset being considered. As sanity check, we tested the performance of uniform downsampling of the feature vector, well as the knockoff- based FS method where all knockoff feature scores have zero value (c.g., knockoffss are not considered during feature selection), which we dub baseline in Figure
DISCUSSION AND CONCLUSIONS
In this paper, we have proposed an algorithm for FS based on the knockoff variable framework. Our goal is extend its applicability to general problems in ML, and we focused on the specific problem of classification using softmax activations. We have observed that the performance of FS is dependent on the suitability of the model used in knockoff design for the data being considered and for the number of data points available for training. Given that the data models used by our approach (and by deep knockoff's) are better suited for many modern data-centric problems, we obtain better performance using them the original Gaussian model.
There is additional potential for the knockoffs described here. The fact that knockoff generation does not require knowledge of the data labels (and their joint distribution with the data) implies that knockoffs can be generated for data from unsupervised learning problems. Existing scores for unsupervised FS could be used on knockoff features as well.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
R. Barber, E. J. Candès, and R. Samworth. Robust inference with knockoffs. Ann. Stats., Feb. 2019.
To appear.
R. F. Barber and Candès. Controlling the false discovery rate via knockoffs. Ann. Statist., 43(5):
2055-2085 Oct. 2015.
R. F. Barber and E. J. Candès. A knockoff filter for high-dimensional selective inference. Ann, Statist., 47(5):2504-2555 2537, Oct. 2019.
Y. Benjamini and Y. Hochberg. Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. Royal Statist. Soc. Ser: B, 57(1):289-300, Jan. 1995.
E. J. Candès, Y. Fan, L. Janson, and J. Lv. Panning for gold: Model-X knockoff's for high-dimensional controlled variable selection. J. Royal Statist. Soc. Ser. B, 80(3):51--577, Jan. 2018.
C. Doersch. Tutorial on variational autoencoders. June 2016. Preprint, available at 1eXiv:1606006
R. O. Duda, P. E. Hart, and D. G. Stork. Pattern Classification. John Wiley & Sons, New York City, NY, 2nd edition, 2001.
Y. Fan, J. Lx, M. Sharifvaghefi, and Y. Uematsu. IPAD: Stable interpretable forecasting with knockoffs inference. Am. Statist, Assoc., 2019.
C. Gao, H. Sun, T. Wang, M. Tang, N. I. Bohnen, Martijn L.. T. M. Müller, T. Herman, N. Giladi, A. Kalinin, C. Spino, W. Dauer, J. M. Hausdorff, and I. D. Dinov. Model-based and model-free machine learning techniques for diagnostic prediction and classification of clinical outcomes in Parkinson's 's disease. Scientific Reports, 8(1):7129, May 2018.
Roquero Gimenez, A. Ghorbani, and J. Zou. Knockoffs for the mass: New feature importance statistics with false discovery guarantees. In Int. Conf. Artificial Intelligence and Statistics (AISTATS), pp. 2123-2133, Naha, Japan, Apr. 2018.
GPy. GPy: A Gaussian process framework in Python. GPy May 2019.
I. Guyon and A. Elisseeff. An introduction to variable and feature selection. J. Machine Learning Research, 3:1157-1182, Mar. 2003.
Hastie, R. Tibshirani, and Friedman. The Elements of Statistical Learning. Springer, 2nd edition, Feb. 2009.
Hensman, N. Fusi, and N. Lawrence. Gaussian processes for big data. In Conf. Uncertainty in Artificial Intelligence (UAI), UAI' 13, pp. 282-290, Bellevue, WA, Aug. 2013.
A. Jain and D. Zongker. Feature selection: Evaluation, application, and small sample performance. IEFE Trans. Pattern Anal. Mach. Intell., 19(2):153-158, Feb. 1997.
J. Jordon, J. Yoon, and M. van der Schaar. KnockoffGAN: Generating knockoff' for feature selection using generative adversarial networks. In Int. Conf. Learning Representations (ICLR), New Orleans, LA, May 2019,
E. Katsevich and C. Sabatti. Multilayer knockoff filter: Controlled variable selection multiple resolutions. Ann. Applied Stats., 13(1):1-33, Mar. 2019.
N. D. Lawrence. Gaussian process latent variable models for visualisation of high dimensional data. In Neural Info. Proc. Systems (NeurIPS), pp. 329-336, Vancouver, BC, Dec. 2003.
N. D. Lawrence. Probabilistic non-linear principal component analysis with Gaussian process latent variable models. J. Mach. Learn. Res., 6: 781-16881, Nov. 2005.
Li and K. Cheng. scikit-feature, http://ww....ttooo//ooooooooo//a May 2019.

---

Under review as a conference paper at ICLR 2020
J. Li, K. Cheng, S. Wang, F. Morstatter, R. Trevino, J. Tang, and H. Liu. Feature selection: A data perspective. ACM Comput. Surv., 50(6))94:1 Dec. 2017.
J. Li. K. Cheng, and H. Liu. Feature Selection Datasets. http://featureeeletttio asu edu/ htt///aaaassss... php Jan. 2019.
H. Liu and Motoda (eds.). Computational Methods of Feature Selection. Chapman and Hall/CRC, Boca Raton, FI., 2008.
Y. Liu and C. Zheng. Auto-engoding knockoff generator for FDR controlled variable selection. Sep. 2018. Preprint, available at arXiv:1809.10766
Y. Lu, Y. Fan, J. Lv, and W. S. Noble. DeepPINK: Reproducible feature selection in deep neural networks. In Neural Info. Proc. Systems (NeurPPS), Montreal, QC, Dec. 2018.
Nie, H. Huang, X. Cai, and C. Ding. Efficient and robust feature selection via joint normm Vancouver, BC, Dec.
minimization. In Neural Info. Proc. Systems (NeurIPS), pp. 2010.
Pavlenko, On feature selection, curse-of-dimennioonaity and error probability discriminant analysis. J. Statist. Planning and Inference, 115(2):565-58) Aug. 2003.
M. Robnik-Sikonja and I. Kononenko. Theoretical and empirical analysis of relieff and rrelieff. Machine Learning, 53(1):23 69, Oct 2003.
Y. Romano, M. Sesia, and E. J. Candès. Deep knockoffs. J. Am. Statist. Assoc,, Aug. 2019.
H. Samet. Foundations of Multidimensional and Metric Data Structures. Morgan Kaufman, San Francisco, CA, 2006.
M. Sesia and Y. Romano. Deep Knockoffs Package. http::///iithub.commmsss// deepknockoffs, May 2019.
M. Sesia, C. Sabatti, and E. J. Candès. Gene hunting with knockoffs for hidden Markov models. Biometrika, 106(1):1-18, Mar. 2019.
A. Weinstein, R. Barber, and E. Candès. A power and prediction analysis for knockoff's with lasso statistics. Dec. 2017. Preprint, available at arXiv:1712.06465.
S. Wright. The interpretation of population structure by f-statistics with special regard to systems of mating. Evolution, 19(3):355-43 Sep. 1965.
10


Reviews:
[{"review_id": "SJgD5mIQqH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a way of employing modern generative models like VAEs and GPLVMs within the recently popular “knockoff” framework for variable/feature selection. Roughly speaking, the main idea of knockoffs is to construct a duplicate (“knockoff”) of each regression variable which matches its distributional properties but is independent of the response variable when conditioning on the true input variable the knockoff is based on. Under certain assumptions, fitting a model on top of both original and knocked-off features and observing the difference between the fitted coefficients for the true and the knockoff features can then be used for variable selection with guaranteed false discovery rate. The authors of this paper suggest that many of the current methods for construction of knockoffs may not be appropriate for image, text, and other types of datasets commonly used in modern ML, and suggest a heuristic way of employing VAEs and GPLVMs for this purpose. The paper concludes with an empirical study which shows that their algorithm is competitive with existing feature selections methods in terms of post-selection accuracy of the fitted model.\n\nI am currently leaning towards recommending rejection. While the paper is nicely written and does a good job of reviewing knockoffs, I see two main issues: (1) I am not sure about applications for the proposed algorithm; in particular, the authors allude to use on devices with limited memory and computational power, but do not discuss why the Johnson-Lindenstrauss transform or some of the many low-precision implementations of neural networks (binarised neural networks, xnor-nets, …) cannot be used; furthermore, in the case of images, I am not sure why there is no comparison to simple downsampling to a smaller resolution; (2) The reported results do not show a reliable improvements over existing methods.\n\n\nMajor comments:\n\n- I am not entirely sure lemma 1 is correct. In particular, mu_{z | x} tends to be a function of the whole vector x including x_n. For example, if (a, b) are jointly distributed according to a bivariate normal, then E (b | a) = E(b) + Sigma_{ab} Sigma_{aa}^{-1} (a - E(a)) where Sigma is the corresponding covariance matrix. Hence claiming that the marginal distribution z | x_{-n} is N ( mu_{z|x} , Sigma_{z | x} ) seems wrong as mu_{z | x} will generally depend on x_n (similarly to how E(b | a) depends on “a” above) which cannot be the case when x_n is marginalised. If z depends linearly on x, there is a standard expression for the distribution you seek. However, with the non-linear dependence employed, e.g., within VAE, working out a closed form expression may be quite a challenge. Can you please clarify or drop this result from your paper if it indeed turns out incorrect?\n\n- Can you please clarify why don’t you benchmark against the cited algorithm proposed in Lu et al. (2018)?\n\n- Can you please explain how was the latent dimensionality for the VAEs (5) and GPLVMs (10) selected? Also, for the algorithms where random seed plays a role (e.g., VAEs), how many random seeds were used (are the numbers reported in fig.2 a result of averaging over multiple seeds)? Relatedly, have you tested whether starting from different seeds results in the same subset of variables selected (e.g., when using VAEs)?\n\n\nMinor comments:\n\n- In the 1st sentence of the introduction, “prevalence” can be high or low, increase or decrease, etc. but “becoming increasingly pervasive” does not sound right. Please consider rewording.\n\n- In fig.1e, do you know why GPLVM leads to such a significant mode collapse?\n\n- Just after the 1st display on p.5, “SInce” -> “Since”", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hye4vzrzFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new feature selection method by integrating the knockoff procedure and generative models.\nThe paper is clearly written and easy to read. However, I have the following concerns:\n\n- The motivation is not clear. The advantage of the knockoff procedure is that it can find relevant features (variables) with statistical guarantees such as FDRs, which is clearly discussed in the paper.\n    However, the objective of this paper is to design a better feature selection method for prediction, where the statistical guarantee is usually not important.\n    Hence the advantage of using the knockoff procedure is not clear.\n- In addition to the above issue, the empirical performance of the proposed method is not convincing.\n    In experiments, Figure 2 shows that the proposed approach does not have significant advantage compared to existing methods.\n    It seems that this is a natural consequence as the knockoff procedure is not designed for feature selection.\n    Thus, in its current state, the advantage of the proposed approach is not well presented.\n- Also, the proposed approach is not theoretically analyzed.\n    Since the proposed method is based on the knockoff procedure, it would be interesting if there is some statistical guarantee for the selected features.\n\nMinor comments:\n- P.5, L.-4: \"SInce\" -> \"Since\"\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Hke1qd36Yr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an interesting use of a knockoff framework similar to that of model-X knockoffs (Candes et al., 2018) in generative models like VAE and GPLVM for feature selection in supervised learning. \n\nOn the flip side, it is really hard to tell from the experimental results (Fig. 2) what performance benefits their proposed designs bring over the state of the art. I would encourage the authors to provide a more detailed analysis of the conditions under which their proposed designs would outperform the state of the art (or not). The author should consider presenting their results through other means that can better highlight the performance benefits of their proposed designs.\n\nBesides applying their knockoff designs to a softmax classifier, can the authors provide examples of applications to other supervised learning models?\n \n\nProof of Lemma 1: Without further assumptions from that stated in the lemma, can the authors provide a derivation (or result that they've used) and explanation for the mean vector and covariance matrix of the joint probability p(z,x_n|x_{−n}) = p(z|x)p(x_n|x_{−n})? In particular, why are z and x_n conditionally independent given x_{-n}, as reflected in their zero covariance value?\n\n\nPage 2: The authors say that \"they must be independent from the labels to be predicted\". Shouldn't it be conditionally independent (2nd paragraph, page 3) instead?\n\n\n\nMinor issues\nPage 5: The notation of tilde{bold{x}}_n is ambiguous since the subscript is used to index the data sample (page 2). Is it intended to be without a bold?\n\nStep 3 of Algorithm 1: I would have preferred that it is stated consistently with that in the main text (step i in 2nd paragraph of Section 3).\n\nPage 5: This sentence is difficult to parse: \"the training dataset for the generative models used to compute new knockoffs sample values is trained over the original data samples\".\n\nPage 5: SInce?\n\nPage 6: I cannot find Figure 1(g) and Figure 1(h).", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This manuscript proposes feature selection inspired by knockoffs, where the generative models are implemented using modern deep generative techniques. The resulting procedure is evaluated in a variety of empirical settings and shown to improve performance.

The reviewers and AC agree that the problem studied is timely and interesting, as knockoffs combined with generative models have recently shown promise for inferential problems. However, the reviewers were unconvinced about the motivation of the work, and the strength of the empirical evaluation results. In the option of the AC, this work might be improved by focusing (both conceptually and empirically) on applications where inferential variable selection is most relevant e.g. causal settings, healthcare applications, and so on.
Decision: Reject