Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work considers the controlled feature selection problem, or selecting a small number of important features subject to FDR control. The proposed method, FlowSelect, builds on the existing literature of conditional randomization tests and holdout randomization tests, as well as normalizing flows from the deep generative modeling literature.

In simple terms, FlowSelect uses normalizing flows to learn a data distribution and MCMC to sample individual held-out features from their conditional distribution; it then passes these samples through a single model fit on the original data to generate samples from the null distribution, and it then compares these to the observed test statistic calculated using the observed data (with the rejection threshold set according to Benjamini-Hochberg).

The proposed approach is justified using an asymptotic argument: normalizing flows are theoretically capable of modeling any distribution given enough transforms, and MCMC should be able to sample from the learned distribution given enough samples. The empirical results are quite promising, FlowSelect provides better results than several competing knockoff methods across two types of synthetic tasks, as well as reasonable results on a real scRNA-seq dataset.
strengths_and_weaknesses: ### Strengths

- The method combines known methods for feature selection under FDR control with two new techniques (at least in this subfield): normalizing flows and MCMC to sample from the learned distribution
- The asymptotic argument makes clear why this approach should work, at least in scenarios with a large amount of data, a large flow model, and sufficient MCMC samples
- The empirical results are quite promising
- The ablation study showed that multiple methodological innovations are important for helping FlowSelect outperform the baselines (the normalizing flows + MCMC approach for handling held-out features, as well as the HRT)

### Weaknesses

- The theory is somewhat optimistic: the p-values converge only under two asymptotic results. It's natural to develop asymptotic theory first as this is often easier, but I wonder if there's any way the theory could be refined to more closely reflect how FlowSelect performs in practice, or at least help control whether either of the key steps is working well enough (more on this in the questions section below)


Nits:
- This may not be a mistake, but I wasn't sure of the reason for the +1 in the summation and denominator of eq. 5
- There's a typo on line 252, "to have support in..."
- I may have missed this, but what exactly was the test statistic used in the experiments? The caption of figure 2 says the statistics were calculated using either Lasso or random forest; was it the model's predictive accuracy? 
questions: - Would the authors be able to include additional results showing how FlowSelect (and the baselines) perform given a smaller dataset? I would be curious to see if the normalizing flows approach scales better or worse than the other methods, particularly those that use deep learning
- Based on a quick Google search, it seems there are other papers that have used MCMC to sample from distributions parameterized by normalizing flows (e.g., https://arxiv.org/abs/2105.12603). Are the authors aware of these, and would it make sense to comment on the similarities/differences between this and some existing use cases of this trick?
- As mentioned above, the asymptotic argument for why FlowSelect should work is somewhat optimistic and doesn't quite explain why we should expect this approach to work in the finite regime (finite transformations in the flow model, finite MCMC sampling). Can the authors comment on whether they might be able to refine their analysis, or perhaps include a discussion of how to verify if either part is working "well enough" ? (E.g., metrics for whether the normalizing flow fit the distribution well, established techniques for deciding if MCMC has been run for long enough.)
- Could the authors add a note about why it's important to add Gaussian noise in the scRNA-seq experiment (what the issue is with zero-inflation)?
- This isn't a weakness in FlowSelect specifically, but perhaps in all CRT-based methods that may be helpful to address: it seems like a CRT may fail to identify an important feature that is correlated with another important feature, because removing either from the model should have no impact on the test statistic. Could the authors explain this and why it is/isn't a problem, and if necessary add a note (perhaps in section 2)? 
limitations: A couple of the questions above get at limitations that might be helpful to discuss:
- Ability to identify important features that are strongly correlated with one another (in the worst case, identical)
- Limited applicability of theory to practical usage
- Unclear scaling with dataset size
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new variable selection method called FLOWSELECT to perform controlled feature selection that does not suffer the problems existed in Knockoffs-based methods. Asymptotically,  the proposed method computes valid p-values. Empirically, FLOWSELECT consistently controls the FDR on both synthetic and semi-synthetic benchmarks, whereas competing knockoff-based approaches do not. 
strengths_and_weaknesses: Strength:
+  The paper is well-motivated and well-written.
+  The proposed method has nice theoretical guarantees and sufficient empirical evaluations.


Weakness:
+ In fact, some work on knockoff has improved the two problems highlighted by the author that lead to the failure of traditional knockoffs-based methods, e.g., “Fan Y. et al. RANK: Large-Scale Inference With Graphical Nonlinear Knockoffs.”. However, the authors does not discuss this related literature.

questions: [Fan Y. et al] propose a modified model-X knockoffs method, called graphical nonlinear knockoffs (RANK), to accommodate the unknown covariate distribution. The related theoretical results verify the robustness of the RANK by showing that the false discovery rate (FDR) is asymptotically controlled at the target level and the power is asymptotically one with the estimated covariate distribution. 

+ Does the RANK overcome the first problem often occurred in the knockoff-based method emphasized by the authors? 

+ It would be appreciated if the author could discuss the difference between this method and the above method. 

+  Can rank method be added as a baseline in the numerical experiments？
[Fan Y. et al] Large-Scale Inference With Graphical Nonlinear Knockoffs, JASA.
limitations: All my concerns are presented in “Weakness” and “Questions”. Moreover, I do not have any concern on negative social impact.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper considers the problem of multiple hypothesis testing with the FDR control. The work is situated in the model-X setting and is motivated by the limitations of the existing model-X multiple testing problems, with an emphasis on the knockoff-based methods. The paper proposes using the normalizing flow technique to estimate the conditional distribution of features, i.e., $X_j \mid X_{-j}$, which combine with CRT/HRT yields (approximately) correct p-values for conditional hypothesis testing. The p-values are subsequently fed to the BHq procedure, and a selection set is provided. The proposed method is evaluated with synthetic and semi-synthetic data, showing satisfying empirical results when compared to other methods that rely on inexact knockoff constructions.
strengths_and_weaknesses: Strength: 
1. The paper finds a nice connection between a state-of-art  conditional density estimation method and the model-X feature
selection method.

2. The proposed method performs well empirically.

Weaknesses:
1. I find the motivation and comparison of the proposed method unsuitable. The paper tries to see the proposed method as an alternative method for the knockoff-based method; but at its best (i.e., when the mapping G is exactly known), the proposed method does not guarantee the FDR control due to the dependence between the p-values. Further, the target distribution knockoffs (or inexact knockoff machines) tries to learn is different from what the proposed method tries to learn: the former is a knockoff distribution satisfying the ``swapping invariant'' property while the latter is simply the conditional distribution. I would suggest framing the proposed method as a way of implementing CRT/HRT when the feature distribution is not exactly known. 

2. Following the previous comment, I find the comparison in the simulations a bit unfair: methods w/ different guarantees;
one-bit p-values versus multi-bit p-values. For example, it might be more appropriate to consider using the existing deep method to learn $X_j \mid X_{-j}$ and apply CRT/HRT.

3. The presentation of this paper can be improved. 

a) the notation is not consistent throughout the paper. For example, the dimension is represented as d in line 44, then D in line 45, and then p in line 103. 

b) The description of the existing method is a bit hard to follow. For example, it may help to briefly explain where the test data and training data come from; Theorem 1 involves the index n, and it would help to explain what is this n in the actual flow used (is it M?).



questions: My major questions have been listed in the "strength and weaknesses" part. Some minor questions follow.

1. Line 55: should it be "In situations..."?

2. Line 157: is $\hat{\alpha}_j$ used as the p-value for both hypothesis $j$ and the ordered p-value?

3. Figure 1: what do the x and y-axis represent?

4. Line 212: what is $\Sigma_j$; line 213: what does the index $j$ mean in the definition of $\rho$.

5. Figure 2: it might be helpful to enlarge the fonts.

6. Figure 4: what do the column panels represent?


limitations: My comments on the limitations have been included in the "strengths and weaknesses" part. The authors have adequately addressed the potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors present a method, called FlowSelect based on normalizing flows to select features in a controlled fashion, meaning that the false discovery rate (FDR) is limited. They select features by first approximating the data distribution via the normalizing flow, and then compute a p-value for each feature explaining the data with respect to some model, e.g. a Lasso regression model or a Random Forest. Estimating the p-values involves sampling from the marginal distribution of the flow, which is done via MCMC.

To make sure that the FDR can be controlled, the authors prove that their estimates of the p-values are asymptotically correct, i.e. that the estimates converge to the true value almost surely.

In their experiments, they apply their method to a synthetic dataset, i.e. correlated Gaussians, a semi-synthetic dataset based on gene data, and a real-world dataset where they try to discover genes relevant for the oil content in soy beans. Their method outperforms several baseline procedures based on knockoffs, etc., and they are able to find SNPs relevant for the oil content which have also been confirmed by other studies.
strengths_and_weaknesses: ## Strengths

In general, the paper is well-structured and easy to read. I appreciate the background section, especially, because it guides the reader through the four lines of research the authors take ideas from and combine them to a new method.

FlowSelect combines several known and established methods in a creative way. It is well grounded from a theoretical perspective as the authors prove that their estimator for the p-values converges to the true value almost surely.

FlowSelect was put to test on three different datasets. It should to have high power on the synthetic and semi-synthetic dataset, higher or on par with all of the baselines, while having FDR as high or lower than the threshold, which is not the case for all the baselines. That the authors are also able to identify SNPs, which are relevant for the oil content in soy beans, is impressive.

## Weaknesses

Although the authors compare the SNPs they identify in the soy bean experiment to what has been found in other studies, they do not apply the baselines they used in the previous experiments to the same dataset, which makes their findings slightly less convincing.

Moreover, they give little insight in the computational cost of their method compared to competing procedures. Computing the marginals from the flow is only possible through MCMC, which can be very inefficient if the proposal distribution does not represent the actual distribution well. Especially when dealing with genomic data with 10k+ features this could render the method very inefficient. Unfortunately, the authors give very little insight into how they chose these proposal distributions and ensured that they represent the true marginals well.

## Conclusion

I'm in favor of accepting the article, although I wish that the others could clarify the concerns I raised above.
questions: * What is the computational cost of computing the marginals of the flow for high dimensional systems?
* How do you choose the proposal for the MCMC procedure sampling from the marginals?
* Could you apply the other baseline methods to the genomic soy bean dataset?
limitations: It is not clear how well FlowSelect can be scaled up to very high dimensional datasets such as genomic data with 10k+ features.
review_title: null

Meta-review: metareview: This paper describes how to use normalizing flows for selecting features in a way that controls the type-1 error by using a normalizing flow along with MCMC to sample from the null distribution. The majority of the reviewers were positive, however the most confident reviewer was negative. From taking a look at that reviewers concerns, I tend to agree with most of them.

The paper is titled knockoff-free, which means in the context of this paper that both 1) 1-bit p-values are not used and 2) The full knockoff property is not required, only sampling from complete conditionals are required. Most of the experiments compare knockoff methods to the proposed approach, so it's not clear if 1) 1-bit p-values are not great or 2) the model-X process/complete conditional sampling process is better with normalizing flows. The former point is known and the latter point on the best way to sample from the complete conditionals is really the value. 

If we take the paper as, 

1) complete conditionals are 1-D
2) MCMC can be used to sample from a 1-D unnormalized density
3) Simple MCMC won't be bad because the problem is 1-D

-> Any likelihood based deep generative model can be used to sample complete conditionals

then it's a solid paper.

On the other hand, the belief that flows are the correct choice versus other likelihood-based deep generative models is harder to take as there's only a comparison with a mixture density network used in the original HRT paper. Also from other uses of these models, different models are better in different situations. I'd suggest a heavy discussion in the paper on this point at the minimum. Maybe even a reframing of the paper is needed.

Finally, for the test statistic, the HRT may not be the best choice for work like this paper that studies the problems with estimating X-distribution. The  paper "CONTRA: Contrarian statistics for controlled variable selection" at AISTATS 2021 shows that the HRT test statistic is more sensitive to model-X estimation errors than a simple mixture statistic that doesn't give up much power. The choice of test statistic also merits some discussion in step 3.
confidence: Certain
Decision: Accept