Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors tackle the problem of efficient feature selection on small datasets when having access to related datasets. They propose to encode information about each dataset via a small random sample of instances (e.g. 10-30). The approach then defines a set of parametrised gaussian kernels (one per dimension) where the parameters (the importance of each feature and the scaling for each feature) are learned using a permutation invariant neural network allowing to model non linear dependencies between the features and the target. 
strengths_and_weaknesses: Pos
Given the known HSIC Lasso approach to define the optimization objectives, the authors propose a ANN architecture to learn the few (2) parameters of feature-wise kernels which allows them to be sample efficient.

Neg
1) The introduction and layout of the material could probably be improved to streamline the introduction of the necessary concepts in an orderly fashion. 

2) The methods relies heavily on the capacity of the ANN to generalise le selection of features across related problems. To better clarify the contribution of the approach one should also compare against the baseline of training a permutation invariant (deep) neural network on the task of learning the feature selection mask directly, i.e., running some feature selection approach to obtain which features to select for each dataset. Note that in this case the feature selection performed for each task could fully exploit all the instances in the respective datasets and obtain better feature importance estimates.

3) The ablation study is important and well conceived. It would however be of interest to be able to show which element is yielding significant differences (as each ablation seems to be within the 1% range). Perhaps performing some statistical test using the approach explained in [Janez Demšar. 2006. Statistical Comparisons of Classifiers over Multiple Data Sets. J. Mach. Learn. Res. 7 (December 2006), 1–30] and readily available as a python package in [https://github.com/sherbold/autorank].
questions: The paper would gain strength if the authors could show 
1) a comparison with the baseline proposed in Neg 2), i.e. compare against the baseline of training a permutation invariant (deep) neural network on the task of learning the feature selection mask directly.
2) a statistical test for the ablation study, perhaps the one suggested in neg 3).
limitations: yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors address a challenging and important problem of feature selection. They focus on selecting the most informative features given samples from multiple source tasks, and a few target labeled observations. This is particularly challenging if the number of features is large and if there is a distribution shift between the source and the target data. They use Hilbert-Schmidt Independence Criterion to measure dependencies between features and the label. The proposed solution relies on meta-learning with iterative optimization.  Finally, the authors use synthetic and real data to compare the proposed approach to existing schemes. 
strengths_and_weaknesses: The paper reads well, and the English level is satisfactory. The problem of feature selection with few samples and multiple domains seems unique, and I haven’t seen many works address this regime before. The use of meta-learning feels adequate for this problem, and I like the framework proposed by the authors. Another strength of the method is the use of permutation invariant networks, which probably help the method succeed in a regime of low sample size. One key limitation is the use of HSIC, which is a strong statistical tool, but as I understand doesn’t take into account nonlinear feature interactions. For example, if the predictive function is an XOR of x1 and x2, how would FS mechanism that uses HSIC with univariate feature nonlinearity capture such interaction? I think this is a major drawback, that should at least be mentioned and explained to the reader. 
Another concern is the evaluation criteria, the authors compare all methods using a kernel-based classifier. Since STG is an NN-based embedded method it should be evaluated as such, the features selected by the method are most appropriate for a NN classifier (specifically for the one learned after convergence), so it seems weird to evaluate the accuracy of the model using kernel SVM. 


questions: The first few sentences of the second part of the introduction are not written very well. Specifically, the motivation for supervised fs and its relation to unsupervised fs could be explained by more details and better motivation of the problem
Figure 2 requires clarification, specifically, what features are selected exactly by \alpha, it seems that more than the first two (so the FDR is not low). 
Can you provide a comparison to STG when it is trained end to end with the predictive model, i.e. without kernel SVM?
I might be missing something, but I don’t see where in algorithm 1 you use the small set of labeled samples given from the target dataset, why? 

limitations: The limitations of the proposed work are properly detailed in section F.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper applies the meta-learning to the HSIC feature selection, which can alleviate the problem of the lack of labeled data in some applications. It uses HSIC to measure the dependency of features and the labels, and then uses meta-training to train the proposed model. The experimental results show that the proposed method can outperform some feature selection methods, especially the non-meta-learning version HSIC Lasso.
strengths_and_weaknesses: Strengths:
1. It applies meta-training to alleviate the problem of lack of labeled data, and the experimental results show the effectiveness of meta-learning.
2. The experimental results are good.

Weaknesses:
1. The paper claims that this is the first work of meta-learning for supervised feature selection. However, the idea that applying meta-learning for feature selection has been already proposed before, e.g. [1]. Although the methods of these two papers are quite different, the idea of combining meta-learning and feature selection is not new.
2. The main part of the proposed method except the meta-learning is very similar to HSIC Lasso [2], e.g. the objective function and the interpretation of HSIC are similar. It seems that this paper is an extension of [2] by adding the standard meta-learning. The paper does not show any innovative thing about meta-learning and just applies it to the feature selection.
3. In the experiments, the used data sets are not enough. There are only three real-world data sets. However, in the literature on feature selection, often six to ten data sets are used. 
4. Feature selection is often used to handle high-dimensional data. However, none of the data sets used in this paper are high dimensional.
5. It would be better to show the classification results on all features, which can show whether feature selection is useful for the classification.
6. The Conclusion part is too short. It would be better to summarize the strengths and limitations of the proposed method and introduce some future work.

[1] Meta-Learning Causal Feature Selection for Stable Prediction, in ICME 2021
[2] High-dimensional feature selection by feature-wise kernelized lasso, in Neural Computation 2014.
questions: See above.
limitations: The authors discuss the limitation and social impacts in the Appendix. I think it's good.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a meta-learning model for feature selection. The setup assumes the availability of labeled instances from several related tasks, and at test time, we only have access to a small number of labeled instances from the target task (called support set). A neural network infers kernel functions from the support set, and a Hilbert-Schmidt Information Criterion (a criterion for feature selection) based objective function is defined for optimization. The optimization problem is solved via the Iterative Shrinkage Thresholding (ISTA) algorithm. Results shown on several real-world datasets such as MNISTr, ISOLET, and IOT show the strong performance of the proposed algorithm.
strengths_and_weaknesses: ++ The paper is well written and puts itself nicely in the context of the previous work. The proposed meta-learning approach appears novel, though it involves a complicated optimization procedure.

++ The experimental results are strong and show solid improvements over state-of-the-art baselines. 

-- The paper assumes that only a few labeled observations are available from the support set. It is unclear how a highly parameterized neural network is reliably estimated from so few samples in Equation 4?
questions: How are you able to learn efficient neural network representations for a task from the support set (Equation 4) since by definition you assume you only have access to a few samples only? 
limitations: No potential negative social impacts of the research.
review_title: null

Meta-review: metareview: The  paper studies the problem of feature selection with few labelled samples. The paper develops an optimization framework which applies for both regression and classification: the regression setting has been explored for exposition and the classification details are in the appendix. The  use of permutation invariant Neural networks  and using Multi-task learning are interesting angles which help 
the paper demonstrate that features can be selected even when the number of labelled examples are small.
 


confidence: Less certain
Decision: Accept