Few-shot Learning for Feature Selection with Hilbert-Schmidt Independence Criterion
Authors: Atsutoshi Kumagai
Tomoharu Iwata
Yasutoshi Ida
Yasuhiro Fujiwara
Venue: NeurIPS 2022 Accept (2022)
Keywords: Meta-learning
Few-shot learning
Supervised Feature Selection

Abstract: We propose a few-shot learning method for feature selection that can select relevant features given a small number of labeled instances. Existing methods require many labeled instances for accurate feature selection. However, sufficient instances are often unavailable. We use labeled instances in multiple related tasks to alleviate the lack of labeled instances in a target task. To measure the dependency between each feature and label, we use the Hilbert-Schmidt Independence Criterion, which is a kernel-based independence measure. By modeling the kernel functions with neural networks that take a few labeled instances in a task as input, we can encode the task-specific information to the kernels such that the kernels are appropriate for the task. Feature selection with such kernels is performed by using iterative optimization methods, in which each update step is obtained as a closed-form. This formulation enables us to directly and efficiently minimize the expected test error on features selected by a small number of labeled instances. We experimentally demonstrate that the proposed method outperforms existing feature selection methods.

Few-shot Learning for Feature Selection with Hilbert-Schmidt Independence Criterion
Atsutoshi Kumagai
Tomoharu Iwata
NTT Computer and Data Science Laboratories atsutoshi kumagai ht@hco...... ntt. co. jp
NTT Communication Science Laboratories
tomoharu. iwata. www.tiwwtaaaaa....... gyghco ntt. co jp
Yasutoshi Ida
Yasuhiro Fujiwara
NTT Computer and Data Science Laboratories yasutoshi. iww@ieee..ooooo org
NTT Communication Science Laboratories
yasuhiroo.uujiwaa. www.warr...hhhooooooooooooooooo. kh@hco ntt. co.jp
Abstract
We propose a few-shot learning method for feature selection that can select relevant features given small number of labeled instances. Existing methods require many labeled instances for accurate feature selection. However, sufficient instances are often unavailable. We use labeled instances in multiple related tasks alleviate the lack of labeled instances in a target task. To measure the dependency between each feature and label, we use the Hilbert-Schmide Independence Criterion, which is kernel-hased independence measure. By modeling the kernel functions with neural networks that take few labeled instances in a task input, can encode the task- specific information to the kernels such that the kernels are appropriate for the task. Feature selection with such kernels is performed by using iterative optimization methods, in which each update step is obtained as a closed-form. This formulation enables us directly and efficiently minimize the expected test error on features selected by a small number of labeled instances. We experimentally demonstrate that the proposed method outperforms existing feature selection methods.
Introduction
Feature selection fundamental problem in machine learning that aims find the subset of relevant features 77. By extracting a small subset of features, we can interpret the characteristics of datasets, accelerate the learning processes of subsequent problems such as regression and classification, and climinate the cost of collecting irrelevant or redundant features. Thanks to these beneficial properties, feature selection methods have been used in various applications such as biomarker discovery (52) document categorization [10], disease diagnosis [ and drug development [32].
Many feature selection methods have been proposed. An unsupervised approach aims find relevant features without label information 30 [). Although it is useful, has difficulty finding features that are responsible for predicting responses without labels. On the other hand, supervised approach can find such features by using labeled instances [48] (34 [33] [5] (2)). Therefore, this paper focuses on supervised feature selection. Existing supervised feature selection methods often require many labeled instances to accurately select relevant features [21]. However, sufficient instances can be difficult collect in many applications. For example, although feature selection is used for analysis in healthcare [36] 31. sufficient instances (e.g., patients) for prognosis analysis might be difficult to collect in a hospital. In marketing, when data scientists want analyze sales by using past data in each area, sufficient instances might be difficult to collect from underdeveloped or new areas.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

in source
Support set
(2) creeling kemee fundiions
Optiization
(K,.))
Funclions
problem
HSiC-based
feature
Datasot in source
selectiion
Initil paramelers a,
(1) sampling
(4) backprdpagalsee
(3) ISTA
Datasel
Query set (xxyy))
Test feature selection enror
Adapted cootficient parameters
Source
task
Figure 1: Our meta-learning procedure: (1) For each training iteration, we randomly sample few labeled instances (support set) and test instances (query set) from a randomly selected source task. We assume that all tasks have the same feature space. (2) Kernel functions are inferred by the neural network that takes the support set as input, and the optimization problem for HSIC-based feature selection with the support set is defined.
(3) We solve this problem by using ISTA, in which its initial coefficient parameters are treated as trainable parameters. (4) Test feature selection error in terms of HSIC is calculated with both the query set and adapted coefficient parameters, and can be backpropagated to update both the neural network and initial coefficient parameters.
Even if many labeled instances are difficult to collect in task of interest, called target task, they might be obtainable in different but related tasks, called source tasks. For example, sufficient labeled instances might be obtainable from other hospitals urban areas. When target and source tasks are related, we can transfer useful knowledge in source tasks to target ones 17
In this paper, we propose few-shot learning method for supervised feature selection. Few-shot learning is usually formulated as meta-learning, which learns how to learn from few data using data in multiple tasks. Specifically, the proposed method meta-learns with labeled instances in multiple source tasks and uses the learned knowledge for feature selection with a small number of labeled instances in unseen target tasks. Figure [ illustrates the overview of our meta- veta-learning procedure.
In feature selection, the dependency between each feature and response needs be measured. To this end, we use the Hilbert-Schmice Independence Criterion (HSIC), which is nonpaaamettrcc kernel-based independence measure [15 Since HSIC can measure the non-linear dependencies in a principle way without going through density estimation, it is successfully used for various applications including feature selection (53 I1 47 24). To accurately measure the dependencies from a few labeled instances, the kernel functions need to be suitably designed.
With the proposed method, the kernel functions are modeled by commutation-invariinn neural net- works [56) that take a small number of labeled instances, called a support set, as input. By this modeling, our model can encode the information of the support set to the kernels, and the kernels are meta-learned with labeled instances in source tasks so that our model can accurately select relevant features even from few labeled instances. With such kernels, feature selection problems with the support set can be formulated as non-ne nog-nneative least square problems with an l regularizer, which introduced in a representative feature selection method, HSIC Lasso [53]. To obtain the global optimal solution (coefficient parameters) in this optimization problem, we use the Iterative Shrinkage Thresholding Algorithm (ISTA) S, which each update step can obtained as a flosed-form and is differentiable. This differentiability enables us to efficiently backpropagate the expected test feature selection error in terms of HSIC, which is calculated with testing instances (a query set) drawn from the same task as the support set, after adapting to the support set with gradient-based optimiza- tion methods. The proposed method further treats initial coefficient parameters in the optimization problems as trainable parameters, and they are also meta-learned such that only a few update steps by ISTA lead to appropriate coefficient parameters for each task. Since our neural network and initial coefficient parameters
shared across all tasks, the learned model can be applied to unseen target tasks. As a result, the learned model can accurately select features from a small number of labeled instances in a target task.
Our main contributions are summarized as follows: (1) the best of our knowledge, our work the first attempt shot learning for supervised feature selection. (2) We propose a novel method

---

that combines neural networks with HSIC Lasso in the meta-learning framework. Our formulation enables us to perform accurately few-shot feature selection with the closed- from update steps, which is preferable for the efficient meta-learning (3) We empirically show that the proposed method outperformed existing feature selection methods when there are no sufficient target data in both synthetic and real-word datasets.
2
Related Work
Least Absolute Shrinkage and Selection Operator (Lasso) is representative method that uses a sparse linear model for feature selection [48]. Although this method useful, cannot find non-linear relationship between feature and response. To perform non-linear feature selection, several Kernel-based methods have Among them, HSIC Lasso has performed excellently despite its simple We use this formulation in our framework since it enables effective and efficient support set adaptation to be performed as described later. Neural network- -ased feature selection methods have recently been proposed (54, 29
Many supervised feature selection methods have been proposed 7 6
been proposed [53
47 42
formulation [53 18
3 33 With the high expressive capability of neural networks, they can find non-linear complex interaction between feature and response. However, they cannot perform well due to overfitting when there are insufficient instances (25 [I] Although our model also uses neural networks in kernel functions, they are shared across all tasks and are learned with all instances in source tasks. Thus, the proposed method can accurately select relevant features without overfitting.
Multi-task and transfer feature selection methods improve feature selection performances by sharing instances in several tasks [7 55 () They require instances in
target tasks at the training phase. In contrast, the proposed method does not. Our method can quickly adapt to unseen target tasks given target labeled instances at the testing phase.
Meta-learning has recently been attracting a lot of attention, which trains a model such that it generalizes well after adapting to few instances (support set) [T7, 50 46 4 19]. Although many methods have been proposed, existing methods are not designed for few-shot supervised feature selection and thus they cannot be applied our problems. Some studies use the term "meta-learning" for selecting a feature selection algorithm taking
such as number of classes and number They are not methods for few-shot learning, and thus, the scope of these studies is different to that of our study. One method uses the meta-learning framework to select features in hidden features obtained through CNNs for stable prediction (55). This method cannot perform task-specific feature selection or feature selection from few data. The gradient- -base meta-learning methods such as model-agnostic meta-learning (MAML) [9] adapt to the support set by iteratively updating the whole neural network parameters with gradient descent methods. These methods require second-order derivatives of the parameters of the whole neural network when backpropagating the loss, which imposes considerable computational and memory burdens 4 43). Although the proposed method also iteratively adapts to the support set by ISTA, our formulation can adapt support sets by updating only coefficient parameters for ISTA whose dimension is equal to the feature vector size, and its update step can be obtained as a closed-form. This closed-fome update step enables calculation of the second-order derivative of the parameters to be evaded in an automatic differentiation framework such as Pytorch [40]. which realizes more efficient adaptation. Encoder-decoder meta-learning such as neural processes [12 13 [9] efficiently adapts support set by forwarding the support set to neural networks. They use only neural networks, and do not explicitly minimize the loss of the support when adapting new tasks. Thus, they might have difficulty performing effective adaptation. In contrast, the proposed method explicitly minimize the loss of the support set by solving the optimization problem for feature selection, which achieves accurate adaptation. One few-shot learning method for unsupervised feature selection has recently been proposed that selects features from few unlabeled instances [25]. Unlike our method, it cannot method, which selects features with
of features [45] 22 39
use labels. This method is categorized as an only neural networks. Thus, it might be difficult to perform effective adaptation. We show that our method outperforms this method in our experiments. Some methods use deep kernels, which combine neural networks with kernels, in the meta-learning framework [4 20 49]. They are designed for classification or regression problems, and cannot be used for feature selection. We design novel kernel functions with neural networks that are appropriate for feature selection, which are described in Section|4]
3

---

We explain the interpretation of the problem 2 The first term in Eq. can be rewritten as 3 Preliminary
In this section, we briefly review HSIC, which can measure non-linear dependencies between two random variables without going through density estimation. Let X and Y be two random variables on domains X and y. respectively, and sample (a. y) is drawn from a joint distribution p(a, y). HSIC is given by
HSIC Y) =R N.y K (xx, x' [[y, y + E (x, of B) (T(y,3')
2E. E K (x, x Ey [yy 2 0,
(I)
R and L y y + I are pesftive-define kernels, x' r'(y') ) is i.i.d. copy of a(v), and Ex,,''y.. .ya'" means the expectation over independent pairs (r, y) and (z',y/) drawn from p(z, v). When kernels are characteristic such as RBF kemells, HSIC takes zero and only if and Y are statistically independent [15 47). In other words, large value of HSIC indicates that there is a strong dependency between X and Y. Therefore, HSIC can be used as a dependence measure.
where K X'
Proposed Method
4.1 Problem Formulation
(D))
Let Dr := (xtn: You In be a set labeled instances the t-th task, where Xtn = (xt I'n the D-dimensional feature vector of the n-th instance in the t-th task and you its response. The proposed method can be applied in both regression and classification settings. In this section, for simplicity, explain classification described in the supplemental material (Section B). We assume that each task has the same feature vector size D. but each joint distribution pi (x y) can differ, which is the standard assumption used transfer learning studies (38]. Suppose that T source tasks D == {D, are given the training phase. At the testing phase, we are given small number of labeled instances S : (Xn, Vn N called support set, in target task, which is different from source tasks. Our goal is to select features that are appropriate for the target task from S.
method with regression problems, i.c., Vtn R. Our model for
4.2 Model
We explain our model that outputs the importance of each feature given S. By selecting the top-K features in accordance with the importance values, we can select K relevant features.
Optimization Problem To estimate importance values a = ===a;, (.1,...., .D) E 1R20 consider the following optimization problem with support set S:
min
Ls
Σ d=1
11 +X a I|i, S
aERD D 2N
20
where ||. |/r is the Frobenius norm, |11 is the l norm, X E Rs is regularization parameter, Ls == NN Ls E R.NssNs is the centered Gram matrix, (Ls) == ( Vj) E R is the S-dependent kernel for the response S, IN: := INs NE INe Ne11 NN the centering matrix, INg is the Ns-dimension identity matrix, INs is the Ns-dimension vector whose elements are all one, K = -NN.K K (d) TNN E Ns is the centered Gram matrix for the d-th feature, and (K())) )ij = (S) (d)) IR is the S-dependent kernel for the d-th feature in S. Since kernels (8) and L(S) depend support set S, can reflect information of the support set to the kernels so that our model can accurately select features from the support set. The specific form of
the kernels will be explained later.
D
D | -
Ls
Σ
K
(d)
Y)+ HSIC
Σ
agay (S()s(X(()) (1,xx()()))
xx()))
+ +C,
2N
1=1
d=1
d
(3)

---

where (X = Tr(K's Ls) is the empirical estimate of HSIC with support set Ng S (1). Tr(:) is the trace, and C' is a constant term. If there is a strong dependency between the d-th feature ( and response Y, HSICs(X((d) ()) takes large value and aa should also take large value to minimize Eq. 2 If x is independent of Y, HSTCs (x((), (Y) Y) is close to zero by the regularizer. Thus, can regard og the importance of the d-th feature. Moreover, if X and X(d') are strongly dependent, i.e., redundant features, HSICs(X ((())((d)) takes a large value and thus, either of Oud or o'2 tends to zero. This means that redundant features are automatically removed. Therefore, by minimizing Eq. (2),
and thus Ogg tends to become
select relevant features for predicting the response while removing redundant features. However, when there are insufficient instances, the empirical estimate of HSIC will become inaccurate, and thus, relevant features will be difficult to accurately select.
Support Set-dependent Kernel Functions To alleviate this problem, propose kernel functions appropriate for meta-learning feature selection with a small number of labeled instances. First, we consider obtaining latent representation of task from support set S by using permutation-invariiat neural networks [56]:
2 9
Σ f([x, vl) (x.y)ES
R
(4)
Ns
where and y are any feed-forward neural network, and [:..] is concatenation. Since averaging operation permutation-invarian
the neural network in Eq. 44 outputs the same vector even when
the order of instances in the support set varies. Moreover, this neural network can handle different numbers of instances. Thus, the neural network in Eq. (4) well defined as functions for set inputs. Latent task representation contains information of the empirical distribution of instances in support set S. Although we use averaging operation for simplicity, we can use any other permutation-inss invvriaat function such as summation, max (56), and set transformer (2 to obtain latent task representations.
By using latent task representation z, our kernel functions are defined by
k(a) (z) (d) 3. +00)===== ==:)) (z)exp (-o (z) (c) - (1= (1)))
1 y
L(S) Uy) = exp
(ti 8/)
(5)
where (z)) = (g(1) (z())) (z)...., ( (z)) E So and (z) = (o(1) (Z)))).....
o (2) E R mod- eled by feed-forward neural networks taking z as input, respectively, and o, > 0 is hyperparameter. We use S-independent RBF kernel for y for simplicity although any S-dependent kernel can be used. (()) and o ((d) (z) can vary in accordance with z. Intuitively, we expect to obtain inductive bias appropriate for feature selection in each task, i.c., each feature is easily selected or it is not selected, by learnne (zz) and o (z). Specifically, for example, when ((()) (z) learned zero, both HSTCs(X (d)) Y) and HSICs tends to be made zero by the f regularizer. Similarly, ((1) (z) can be learned to fit the scale of the feature each task. These kernel parameters are meta-learne such that our model can accurately select relevant features from small number of labeled instances, which will be explained in detail in the next subsection. We note that the kernels in Eq. (5) are characteristic if (3d) (z) and ((d) (z) are bounded since RBF kernel characteristic.
(d))) become zero in Eq. 33) Thus, corresponding oa
Support Set Adaptation by ISTA simple and efficient algorithm for solving L regulaarie problems in a mathematically elegant way 88. By using ISTA, we can derive the closed-for update steps for optimization. Specifically, can solve the problem 2 by repeating the following steps:
To solve the optimization problem (2.. we use ISTA, which is
a +
A A (Aa -b),
(6)
N3
at + R = la Au1pb+
(7)
where A (veo(K()) .....
E NoxD b vec(Ls) N3, where vec(.) is the (K pP) [
vectorization operator, RA is non-negative soft- soft-thresholding operator, and [++ ReLUU-). The
5

---

Algorithm Training procedure of our model.
Require: Datasets in source tasks D, support set size Ns, query set size No. step size for ISTA and the number of iterations for ISTA I
Ensure:
Parameters of our model θ repeat
2:
Randomly sample task t from {l..... T}
3: Randomly sample support set S with size Ns from Dr
4: Randomly sample query set Q with size Ne from Dr \ S
5: Calculate latent task representation z from support set by Eq.
6: Calculate kernel matrices (5) with task representation and support set S
7:
for I to do
8: Update coefficient parameters (importance values) or by Eqs. 6 and end for
9:
10: Calculate the loss /(2;S) on query set by Eqs. 9 and 10)
11: Update parameters with the gradients of the loss J ((;;
12:
until End condition is satisfied;
first step (6) minimizes the first term in Eq. (2 by using gradient descent with step size μe > The second step (7) reflects the second term in Eq. (2) and non-negative constants. It shrinks a towards zero while satisfying non-negative constraints. We treat the initial coefficient parameters (importance values) Crp for ISTA as trainable parameters like the MAML (9]. By learning good initial parameters with multiple tasks, we can obtain appropriate parameters for each target task even with a few iterations of ISTA.
4.3 Training
We explain our training procedure. this subsection, symbol used as support set in source tasks. In our model, the parameters to be estimated, θ, are parameters of neural networks f. g. 3, a, initial coefficient parameters ag, and regularization parameter We estimate these parameters by minimizing the expected test error on features selected from support set S, where support set and test instances Q, called the query set, are randomly generated from source tasks:
E.-(I....TT [E(,,e)-, [)]]
(8)
where
D Σ d=1
J(2;S) = 2N Le
(d) e
where a = (a)....,a.) E RD. is coefficient parameters obtained from S by ISTA, Le =
20
Ne is the Gram matrix for the response query Q with size No. Lij 2 = N (d) T E Ne NN is the Gram matrix for the d-th feature on Q, and K!'- := (d) (a: (d) (d)) E R is the kernel for the d-th feature in Q. When query size No is sufficiently large, the empirical estimate HSIC with the RBF kernels becomes accurate 15 Thus, we used the following RBF kernels for all features in Eq. 9 as in 53
N LI No LDNo
L( 9j) E R is the kernel for the response in Q. K (d)
(( (d) (d)) = exp (-
(1 (d) - 1 (1)))
(10)
20
where σx is hyperparameter. For the kernel of the response L, used the same RBF kernel as that for the support set adaptation in Eq. (5). By using the same kernels for all features, we can treat each feature equally. That is, the proposed method can fairly evaluate the importance of each feature obtained from on query set Q.
Algorithm shows the pseudocode for our training procedure. For each iteration, we randomly sample task from source tasks (Line 2). From D., we randomly sample support set and query set Q (Lines 3 and 4). We calculate task representation z from S (Line 5), and the S-dependent kernel matrices (Line 6). We iteratively update coefficient parameters a with support set S by using ISTA (Lines 7 9). Using the adapted coefficient parameter a", calculate loss (Q: S) on query set Q

---

(Line 10). Lastly, the parameters of our model è are updated with the gradient of the loss (Line 11). Since each update step of ISTA (Eqs. O and 7 is obtained as a closed-form that is differentiable. the whole training can be performed by using gradient-based optimization methods. This training explicitly matches the selected features from few labeled instances and those from sufficient labeled instances Q on the basis of HSIC. Thus, the learned model can select relevant features for each target task given a small number of labeled target instances (specifically, by running lines through 9 of Algorithm D with the target instances.)
5
Experiments
In this section, we demonstrate the effectiveness of the proposed method using one synthetic and three real-world datasets.
5.1 Comparison Methods
We compared the proposed method with seven existing feature selection methods: Lasso, HSIC Lasso (HLasso), feature selection with stochastic gate (STG) [44, STG-S, fine-tuning variant of STG (STG-FT), meta-learning method for feature selection based on Gumbel-softnax distribution (MetaGS) 25 and a supervised variant of MetaGS (SMetaGS). Lasso is well-known linear feature selection method. HLasso is kernel-basee non- non-linear feature selection method. STG is neural network-basse based non-linear feature selection method that has been reported to outperform various existing methods in various datasets (54). STG learns a mask vector that determines which features are selected while simultaneously learning predictors. Lasso, Hassoo, and STG uses the target support set for feature selection. STG-S trains the model of STG with labeled instances in source tasks. STG-FT fine-tunes the model trained by STG-S with the target support set. MetaGS a neural network-based encoder-decoder meta-learning method for unsupervised feature selection that uses the reconstruction error for feature selection. SMetaGS is a supervised extension of MetaGS that uses the cross-entropy loss the objective function. (S)MetaGS meta-learns the mask vector for feature selection. Specifically, the task spec-fe mask is modeled by the cotuuutationinvvriann neural network that takes support set (a few data) as input and is meta-learned to improve test feature selection performance. All methods except for MetaGS use label information. The proposed method, STG-FT, MetaGS, and SMetaGS use both the target support set and source instances for feature selection.
5.2 Simulation Experiments
Data We first used simple synthetic regression problems with 100-dimensional feature and one- dimensional response spaces. Response y in the (-th task was generated from the following non-linear function: y Wy1 ( (et/eeepp((2)a))) + (x((a)) where task-specific parameters We1 Wr2, and Wa3 were uniform randomly generated from [-1, 1]. and features (z:(1,, ..... (2000) was generated from 100-dimensional standard Gaussian distribution. Feature indexes (i i22 ii)) represent important feature indexes in the t-th task, and were uniform randomly selected for each task from 10 predefined feature indexes {(1, 2,3), (4,3, .).... (28, 29, 30)}. We generated 300 labeled instances for each task. We standardized the values of response. We randomly created 300 training, 20 validation, and 100 target tasks.
Settings We evaluated the average fraction of correctly selected features on each target task with different numbers of target support instances within {10, 20, 30). For each method, we selected the top-3 features by ranking their importance scores. For the proposed method, we selected the hyperparameter on the basis of mean validation loss. For comparison methods, the best test results are reported from their hyperparameter candidates. The details of the experimental settings such as hyperparameter candidates are described in the supplemental material (Section C).
Results Table I shows average and standard errors of the fraction of correctly selected features with different numbers of target support instances. The proposed method clearly outperformed the other methods. Haaso outperformed Lasso since is the non-linear method that fits on the non-linear synthetic data. Although STG is also non-linear method, it did not perform well since training data was too small to train its neural networks. STG-FT, which the fine-tuning method, improved performance compared with STG-S. Although MetaGS and SMetaGS are meta- meta-learning methods like
7

---

Table 1: Averages and standard errors of fraction of correctly selected features [%] with different
numbers of target support instances Ns on the synthetie dataset.
Ns
Ours
Lasso
HLasso
STG
SIG-S
STG-FT
MetaGS
SMetaGi
10
1.9
22.7-2.3
23.7:11.
11.0=133
11.711.6
14.7H221
7.3.1.4
3.711.0
20
49.7:11.7
30.0=2.1
43.7::177
12.71122
11.71:.6
23.0:2.33
9.311.5
7.3114
30
58.311.00
33.0=2.2
48.34188
15.0=1.2
11.71:6
24.7:2..3
4.01111
4.011.11
Avg.
49.11112
28.6=1.3
38.61122
12.920.9
11.7:0.9
20.8.1.3
6.9.0.88
5.040.7
aipha
3:34
bets
332
feature index
festure index
feature index
Figure 2: One example of feature importance a and kernel parameters, (iz) and (z), obtained by the proposed method in the synthetic dataset with Ng 20. The figures are arranged from left to right in the order a²,, (iz), and (z). True important feature indexes on this example were 1. 2, and 3. Our method accurately assigned the top maximum values of ag to the true features.
the proposed method, it performed worse. This result suggests that task- features are difficult select from few instances only by using neural networks. The proposed method can select features by explicitly solving Eq. (2 with useful knowledge in source tasks, and thus, it worked well.
Figure 2 shows one example of feature importance ax and kernel parameters, B(z) and (z). obtained by the proposed method. As explained in Section 4 (z(z) is expected to indicate how likely the d-th feature is to be selected before solving Eq. ( That is, when (z)z) is small, the d-th feature is less likely to be selected. (z) took small value when d was in {31.... 100}. Since there are no important features within (31.... 100} in all tasks of this synthetic data described above, 3(z) was learned expected. Similarly, (z) tends take different values in the two ranges (1,.... 30) and (31,..., 100). This result means that appropriate scales were automatically learned for feature groups that are easy to select (i.e., d E {l,... .,. .0))) and those that are (i.e., d (31...., 1003). After adaptation to the support set, aº took a large value when was in {1, 2, 3}, which is the true important feature indexes of this example. This result shows that the proposed method worked well as expected. We also confirmed that the proposed method works well on more difficult synthetic data in the supplemental material (Section D.10).
5.3
Real-world Data Experiments
Data We used there real-world datasets: Mnist Isole? and loT which have been widely used in previous studies 3 25 54
Mnistr is derived from Mnist by rotating the images [14. This dataset has six tasks (six rotation angles), and its feature dimension is 256. We created the binary classification problem by regarding even digits as positive and odd digits as negative, respectively. Isolet consists of letters (classes) spoken by 150 speakers, and speakers are grouped into five groups (tasks) by speaking similarity. Each instance is represented as a dim-eimmnii vector. We created the binary classification problem by regarding class labels {1..... 13} as positive and {14,... 263 as negative, respectively. 1oT is real network traffic data, which are gathered from nine IoT devices (tasks) infected by malware [35] Each task has normal and malicious traffic instances, and each instance is represented by 115ddimenion vector. For each task, we randomly used 500 normal and 500 malicious instances. For all datasets, we standardized the values of each response and feature. We randomly select one task for the target task, one task for the validation task, and the rest for the source tasks. For each dataset, we created 10 different target/validation//surre task splits.
Settings
To evaluate the quality of selected features, we trained classifiers by using the target support set with the selected features and evaluated the test classification accuracy, which is the
http:///gihuuu.oo/gghfifiie
hhtp://rrchive.. ies www.uci...dei........... wwtp://m//daanssee//SSOOFF httpss//arhiive.cc. http://wwwww.....eeeeeeeaaaeeeee
of loT botnet_attacks N BaloT

---

Table 2: Averages and standard errors of test accuracy [%] with different numbers of target support instances on the real-world datasets.
Data Mnistr
Ns
Ours
Lasso
HLasso
STG
STG-S
STG-FT
MctaGS
SMetaGS
10
67.1.0..9
60.810.8
62.810.7
61.1::0.9
67.71099
67.8:1.0
59.3=1.0
64,4110.9
72.010.7
65.110.7
67.910.8
67.910.7
70.911.0
70.611.1
63.6=0.9
6..41:..
75.2.10.5
68.54:10
71.810.7
7..341.0
73.421.0
73.4110
69.2-112
73.04:1.0
Avg.
7..410.3
64.810.6
67.310.6
66.540.7
70.7
70.6100.6
64.0-0.7
68.610.7
Isolet
10
6.63
58.1+1.0
58.1+0.8
59.2
37.9=1100
58.341.0
58.7=0.8
61.7++0.8
20
0.73
63.3+.0.7
61.5++0.8
62.0+0.7
66.4-0.9
67.3+..9
60.2=0.9
6..5+
30
0.54
64.2:10.6
61.41006
63.510.77
6..5+1122
65.911.2
63.6=0077
6..01:.0
Avg.
68.010.4
61.910.3
60.3.003
61.110.4
63.3:0.7
63.910.7
6..8
65.710.6
loT
10
71.3++1.3
68.9+0.1
69.0+-.1
69.0+0.1
68.7=0.2
69.5+0088
71.1=1.0
09.5+0.2
20
72.3+1.6
71,9+1.7
68.7+0.1
68.7+0.2
74.0+2.0
74.0+2.0
69.3+0.2
71,0+1.3
30
74.3-12.0
73,442.1
68.420.11
68.440.1
73.9-2.1
73.942.1
70.7=112
71.
Avg.
72.611.0
71.4009
68.710.1
68.740.1
12.2:100
72.3100
7..4
70.8
Table 3: Ablation studies of our model: averages and standard errors of test accuracy [%] over different numbers of target support instances and different numbers of selected features on each real-world datasets.
Data Ours
wo Latent
w/o Feature
w/o LFeature
w/o S-kemel
w/o Initial
w/ Deep
w/o S-adapt
Muistr
0.5
70.310.6
70.010.7
70.1
70.6.0..7
69.410.7
70.6
70.410.7
Isolet
68.0-0.4
65.710.4
66.110.4
65.810.4
65.6.10.4
65.110.4
64.0100.4
64.2.10.5
loT
72.6+1.0
7..0+1.1
71.6+0.9
70.4+0.7
70.6+00.7
71.8+0.9
70.7+-0.7
73.4+1.0
Avg.
70.7=0.4
70.0:0.35
69.320.4
68.820.4
68.920.44
68.820.4
68.420.
69.320.5
Since the proposed method, HLasso, and MetaGS cannot directly classify test instances, we used support vector machines with RBF kernels the classifiers for all methods for a fair comparison. We did not use neural networks as classifiers since they have many hyperparameters to be tuned, and target training instances too small to train without overfitting. The details of the settings such as hyperparameters are described the supplemental material (Section C).
standard evaluation protocol used in the previous feature selection studies [53 47 29 54]
Results Table Tablee shows average and standard errors of test accuracies over different numbers of selected features within {10, 50) when changing the numbers of target support instances within {10,20, 30}. The proposed method achieved the best average accuracies all datasets. Lasso, HLasso, and STG, which used only the target support instances, did not perform well since was difficult to select relevant features from only few labeled instances. STG-FT outperformed these methods for all datasets, which indicates that using information on related tasks is useful to alleviate the lack of target data. MetaGS did not work well all datasets, which indicates the effectiveness of label information. SMetaGS did not perform well with Mnistr and loT since the support set adaptation was difficult with only neural networks. In contrast, the proposed method worked well by appropriately unifying neural networks and kernel methods in meta-learning framework.
Figure 3 shows average and standard errors of test accuracies when changing the number of selected fea- tures K on Isolet. The proposed method performed well across different values. In particular, when K small, the proposed method outperformed the oth- ers by a large margin. In general, when K is small, more important to select informative features im- prove performance. Thus, this result suggests that the proposed method was able assign high importance to informative features.
0.58
Accuracy
0.58
0.64
Fasoo
S10
0.62
0.80
6.58
SMersG
selected features
Figure 3: Average and standard errors of test accuracies over different target support in- stances when changing the selected feature size K on Isolet dataset.
Table 3 shows the results of ablation studies. We compared our model with seven models: w/o Latent, w/o Feature, w/o LFeature, w/o S-kernel, w/o Initial, w/ Deep, and w/o S-adapt. For support set adaptation, w/o Latent uses the kernels without latent task representation z, i.e., it uses the task-invariant feature- specific kernel parameters 3(d) and (d) in Eq.
(5. w/o Feature uses the task-specific

---

(a) Source 1: 3 (b) Source 8 (c) Source 3 (d) Source
(e) Target: 3
Target:
Figure 4: Ten selected features on two source and one target tasks (rotation angles) of Mnistr by the proposed method with Ng = 30. We compared digits '3' and '8", Red represents the selected features (pixels).
kernel parameters (z) and (z) for all features. w/o LFeature uses the task and kernel parameters / and for all features. w/o S-kernel uses the S-independent kernels of Eq. (TO). This method adapts to the support set by only learning initial coefficient parameters. w/o Initial uses same kernels as the proposed method, but does not learn initial coefficient parameters. w/ Deep uses the deep kernels [5I] K (4) (x) ,z)"))= := exp(- z) hh(z,") ²)) where / is three-layered feed- forwwfd neural networks with one output node. w/o S-adapt does not perform support adaptation (i.e., I = 0). This method learns task-invariant initial coefficient parameters for feature selection on any tasks. The proposed method showed the best average results over all datasets. Especially, it outperformed w/o Latent, which indicates the usefulness of inferring task-specific kernels for each feature from the support set. In IoT, w/o Latent worked well. This was probably because each task has similar properties as described in [26). and thus, fask-invariant kernels might be sufficient. w/o Feature and w/o LFeature performed worse than the proposed method, which indicates that feature- -seturef kernel parameters are useful select features. Since w/o S S-kernel and w/o Initial did not work well, we found that both the kernel and initial parameter adaptations need to be considered simultaneously. w/ Deep did not perform well. This result indicates that the form of our kernel function in Eq. (5) is essential for our problems. The proposed method also outperformed w/o S-kernel in Mnistr and isolet, which means that support set adaptation needs to performed to select task task-specific features. Overall, these results indicate the effectiveness of our model design.
We visualized the selected features by the proposed method with Mnistr dataset. Figure shows 10 selected features on each task of Mnistr by the proposed method when Ns = 30. Since the selected features differ between tasks, found that the learned model can select task-specific features by adapting to the support set each task as expected. In addition, the selected features can identify digits "3' and '8' on each task since they captured the bottom left white pixels of digit '8', Therefore, the proposed method can identify relevant features that are responsible for the responses.
6 Conclusion
In this paper, we proposed a shw-shot learning method for feature selection based on Filbert-Schmiae Independence Criterion (HSIC). Experiments showed that the proposed method outperformed existing feature selection methods when there was insufficient data in target tasks. Although the proposed method is useful feature selection method, it cannot train classification regression model that can predict unseen instances simultaneously. As future work, we plan to combine HSIC Lasso and neural network-based clasiffcctionnregression model in meta-learning framework to simultaneously perform feature selection and clasifficationreeresssion model learning from a few labeled data.
References
[1] M. F. Akay. Support vector machines combined with feature selection for breast cancer diagnosis. Expert systems with applications, 36(2):3240-3244 2009.
[2] A. Argyriou, T. Evgeniou, and M. Pontil. Convex multi-task feature learning, Machine learning, 73(3):243-772, 2008.
[3] M. F. Balm,, A. Abid, and Zou. Concrete autoencoders: differentiable feature selection and reconstruction. In ICML, 2019.
10

---

[4] L. Bertinetto, J. F. Henriques, P. H. Torr, and A. Vedaldi. Meta-learinng with differentiable closed-form solvers. In ICLR, 2018.
[5] D. Cai, C. Zhang, and X. He. Unsupervised feature selection for multi-cluster data. In KDD, 2010.
[6] J. Cai, J. Luo, S. Wang, and S. Yang. Feature selection in machine learning: A new perspective. Neurocomputing, 300:70-79, 2018.
[7] G. Chandrashekar and F. Sahin. A survey on feature selection methods. Computers & Electrical Engineering, 40(1):16 16-288 2014.
[8] I. Daubechies, M. Defrise, and C. De Mol. An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Communications on Pure and Applied Mathematics: Journal Issued by the Courant Institute of Mathematical Sciences, 57(1
2004.
[9] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.
[10] G. Forman. Bns feature scaling: improved representaiionn over tf-idf for svm text classification.
In CIKM, 2008.
[11] T. Freidling, B. Poignard, H. Climente-Gonzállez and M. Yamada. Post-selection inference with hsic-lasso. In ICML, 2021.
[12] M. Garnelo, D. Rosenbaum, C. Maddison, Ramalho, D. Saxton, M. Shanahan, Y. W. Teh, D. Rezende, and S. A. Eslami. Conditional neural processes. In ICML, 2018.
[13] M. Garnelo, J. Schwarz, D. Rosenbaum, F. Viola, D. J. Rezende, S. Eslami, and Y. W. Teh. Neural processes. arXiv preprint arXiv: 1807.01622 2018.
[14] M. Ghifary, W. Bastiaan Kleijn, M. Zhang, and D. Balduzzi. Domain generalization for object recognition with multi-task autoencoders. In /CCV, 2015.
[15] A. Gretton, O. Bousquet, A. Smola, and B. Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. In ALT, 2005.
[16] Gu, Z. Wang. J. Kuen, I.. Ma, A. Shahroudy, B. Shuai, T. Liu, X. Wang, G. Wang. J. Cai, et al. Recent advances in convolutional neural networks. Pattern Recognition, 77:354-377, 2018.
[17] T. Hospedales, A. Antoniou, P. Micaelli, and A. Storkey. Meta-learning in neural networks: a survey, arXiv preprint ar (XX::x00040054333 2020.
[18] Q. Huang, M. Yamada, Y. Tian, D. Singh, D. Yin, and Y. Chang. Graphlime: local interpretable model explanations for graph neural networks. arXiv preprint arXiv:2001.062/6 2020.
[19] T. Iwata and A. Kumagai. Meta-learning from tasks with heterogencous attribute spaces. In NeurlPS, 2020.
[20] T. Iwata and Y. Tanaka. Few-shot learning for spatial regression. arXiv preprint arXiv:2010.04360, 2020.
[21] A. Jain and D. Zongker. Feature selection: evaluation, application, and small sample per- formance. IEEE transactions pattern analysis and machine intelligence, 19(2):15--158, 1997.
[22] A. Kalousis and M. Hilario. Feature selection for meta-lcarning. In PAKDD. 2001.
[23] D. P. Kingma and Ba. Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.69888 2014.
[24] K. Koyama, K. Kiritoshi, T. Okawachi, and T. Izumitani. Effective nonlinear feature selection method based on hsic lasso and with variational inference. In AISTATS, 2022.
[25] A. Kumagai, T. Iwaaa, and Y. Fujiwara. Few-shot learning for unsupervised feature selection. arXiv preprint arXiv:2107.000 00000000000..... 2021.

---

[26] A. Kumagai, T. Iwata, and Y. Fujiwara. Meta-learning for relative density-ratio estimation. In NeurIPS, 2021.
[27] Y. LeCun, B. Boser, J. Denker, D. Henderson, R. Howard, W. Hubbard, and L. Jackel. Hand- written digit recognition with pack-propagation network. 1989.
[28] Lee, Y. Lee, Kim, Kosiorek, Choi, and Y. W. Teh. Set transformer: a framework for attention-based permutation-invariant neural networks. In JCML, 2019.
[29] I. Lemhadri, F. Ruan, and R. Tibshirani. Lassonet: neural networks with feature sparsity. In AISTATS, 2021.
[30] J. Li, Tang, and H. Liu. Reconstruction-basee unsupervised feature selection: an embedded approach. IJCAI, 2017.
[3]] B. Liu, Y. Wei, Y. Zhang, and Q. Yang. Deep neural networks for high dimension, low sample size data. In IJCAI, 2017.
[32] Y. Liu. A comparative study on feature selection methods for drug discovery. Journal of chemical information and computer sciences, 1823-18833, 2004.
[33] Y. Lu, Y. Fan, J. Lv, and W. S. Noble. Deeppink: reproducible feature selection in deep neural networks. In NeurIPS, 2018.
[34] M. Masaeli, G. Fung, and J. G. Dy. From transformation-base dimensionality reduction to feature selection. In ICML, 2010.
[35] Y. Meidan, M. Bohadana, Y. Mathov, Y. Mirsky, A. Shabtai, D. Breitenbacher, and Y. Elovici. N- baiot-netwe netlo-- based detection of iot botnet attacks using deep autoencoders. IEEE Pervasive Computing, 17(3):12-25 22, 2018.
[36] M. Z. Nezhad, D. Zhu, X. Li, K. Yang, and P. Levy. Safs: A deep feature selection approach for precision medicine. In BIBM, 2016.
[37] G. Obozinski, B. Taskar, and M. Jordan. Multi-task feature selection. Statistics Department, UC Berkeley, Tech. Rep, 2(2.2), 2006.
[38] S. J. Pan and Q. Yang. A survey on transfer learning. IEEE Transactions on knowledge and Data Engineering. 22(10):1345-1359, 2009.
[39] A. R. S. Parmezan, H. D. Lee, and C. Wu. Metalearning for choosing feature selection algorithms in data mining: Proposal of a new framework. Expert Systems with Applications, 75:1-24, 2017.
[40] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. Vetooo, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiationn in pytorch. 2017.
[41] M. Patacchiola, J. Turner, J. Crowley, M. O' O'Boyle, and A. Storkey. Bayesian meta-leaming for the few-shot setting via deep kernels. In NeurlPS, 2020.
[42] B. Poignard and M. Yamada. Sparse hilbert-schmidt independence criterion regression. In AISTATS, 2020.
[43] A. Rajeswaran, C. Finn, S. M. Kakade, and S. Levine. Meta-learning with implicit gradients. In NeurlPS, 2019.
[44] A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap. Meta-learning with memory-augmented neural networks. In ICML, 2016.
[45] Z. Shen, X. Chen, and J. M. Garibaldi. novel meta learning framework for feature selection using data synthesis and fuzzy similarity. In FUZZ-IEEF, 2020.
[46] Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. NeurlPS, 2017.
12

---

[47] L. Song, A. Smola, A. Gretton, J. Bedo, and K. Borgwardt. Feature selection via dependence maximization. Journal of Machine Learning Research, 13(5), 2012.
[48] R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267-285 2286288,, 1996.
[49] P. Tossou, B. Dura, F. Laviolette, M. Marchand, and A. Lacoste. Adaptive deep kernel learning. arXiv preprint 121111190..12131, 2019.
[50]
Vanschoren. Meta-learning a survey. arXiv preprint arXiv:1880.035500 2018.
[51] A. G. Wilson, Z. Hu, R. Salakhutdinov, and E. P. Xing. Deep kernel learning. In AISTATS, 2016.
[52] E. P. Xing, M. I. Jordan, R. M. Karp, et al. Feature selection for high-dimensional genomic mieroarray data. In ICMI,, 2001.
[53] M. Yamada, W. Jitkrittum, L. Sigal, E. Xing, and M. Sugiyama. High-dimensional feature selection by feature-wise kernelized lasso. Neural computation, 26(1):185- 2014.
[54] Y. Yamada, O. Lindenbaum, S. Negahban, and Y. Kluger. Feature selection using stochastic gates. In ICML, 2020.
[55] 7.. Yuan, X. Peng, X. Wu, B.-k. Bao, and C. Xu. Meta-learning causal feature selection for stable prediction. In 202/ IEEE International Conference on Multimedia and Expo (ICME), pages 1-6. IEEE, 2021.
[56] M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and A. J. Smola. Deep sets. In NewrIPS, 2017.
[57] Y. Zhou, R. Jin, and S. C.-H. Hoi. Exclusive lasso for multi-task feature selection. AISTATS, 2010.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please see Section and Section E.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Please see Section F.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (cither in the supplemental material or as URL)? [No] The code is proprietary. The datasets and experimental procedures are described in Section
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Please see Section 5 and Section C.
(c) Did you report error bars (c.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] We performed experiments multiple times with different Training/valldaaiionnsssng splits and reported average results with standard errors in
Tables 1 2 and
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] Please Section C.
13

---

4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Please see Section 5
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or URL? [No]
(d) Did you discuss whether and how consent obtained from people whose data you'le you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] The datasets used in the paper do not contain such information.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "zzj11C3JuoA", "reviewer": "Reviewer_GdFM", "summary": "This paper applies the meta-learning to the HSIC feature selection, which can alleviate the problem of the lack of labeled data in some applications. It uses HSIC to measure the dependency of features and the labels, and then uses meta-training to train the proposed model. The experimental results show that the proposed method can outperform some feature selection methods, especially the non-meta-learning version HSIC Lasso.", "questions": "See above.", "limitations": "The authors discuss the limitation and social impacts in the Appendix. I think it's good.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. It applies meta-training to alleviate the problem of lack of labeled data, and the experimental results show the effectiveness of meta-learning.\n2. The experimental results are good.\n\nWeaknesses:\n1. The paper claims that this is the first work of meta-learning for supervised feature selection. However, the idea that applying meta-learning for feature selection has been already proposed before, e.g. [1]. Although the methods of these two papers are quite different, the idea of combining meta-learning and feature selection is not new.\n2. The main part of the proposed method except the meta-learning is very similar to HSIC Lasso [2], e.g. the objective function and the interpretation of HSIC are similar. It seems that this paper is an extension of [2] by adding the standard meta-learning. The paper does not show any innovative thing about meta-learning and just applies it to the feature selection.\n3. In the experiments, the used data sets are not enough. There are only three real-world data sets. However, in the literature on feature selection, often six to ten data sets are used. \n4. Feature selection is often used to handle high-dimensional data. However, none of the data sets used in this paper are high dimensional.\n5. It would be better to show the classification results on all features, which can show whether feature selection is useful for the classification.\n6. The Conclusion part is too short. It would be better to summarize the strengths and limitations of the proposed method and introduce some future work.\n\n[1] Meta-Learning Causal Feature Selection for Stable Prediction, in ICME 2021\n[2] High-dimensional feature selection by feature-wise kernelized lasso, in Neural Computation 2014.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "jsH-hUwrie", "reviewer": "Reviewer_mXfb", "summary": "The authors address a challenging and important problem of feature selection. They focus on selecting the most informative features given samples from multiple source tasks, and a few target labeled observations. This is particularly challenging if the number of features is large and if there is a distribution shift between the source and the target data. They use Hilbert-Schmidt Independence Criterion to measure dependencies between features and the label. The proposed solution relies on meta-learning with iterative optimization.  Finally, the authors use synthetic and real data to compare the proposed approach to existing schemes. ", "questions": "The first few sentences of the second part of the introduction are not written very well. Specifically, the motivation for supervised fs and its relation to unsupervised fs could be explained by more details and better motivation of the problem\nFigure 2 requires clarification, specifically, what features are selected exactly by \\alpha, it seems that more than the first two (so the FDR is not low). \nCan you provide a comparison to STG when it is trained end to end with the predictive model, i.e. without kernel SVM?\nI might be missing something, but I don’t see where in algorithm 1 you use the small set of labeled samples given from the target dataset, why? \n", "limitations": "The limitations of the proposed work are properly detailed in section F. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper reads well, and the English level is satisfactory. The problem of feature selection with few samples and multiple domains seems unique, and I haven’t seen many works address this regime before. The use of meta-learning feels adequate for this problem, and I like the framework proposed by the authors. Another strength of the method is the use of permutation invariant networks, which probably help the method succeed in a regime of low sample size. One key limitation is the use of HSIC, which is a strong statistical tool, but as I understand doesn’t take into account nonlinear feature interactions. For example, if the predictive function is an XOR of x1 and x2, how would FS mechanism that uses HSIC with univariate feature nonlinearity capture such interaction? I think this is a major drawback, that should at least be mentioned and explained to the reader. \nAnother concern is the evaluation criteria, the authors compare all methods using a kernel-based classifier. Since STG is an NN-based embedded method it should be evaluated as such, the features selected by the method are most appropriate for a NN classifier (specifically for the one learned after convergence), so it seems weird to evaluate the accuracy of the model using kernel SVM. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "gtrOO7EVvo7", "reviewer": "Reviewer_wGbm", "summary": "The authors tackle the problem of efficient feature selection on small datasets when having access to related datasets. They propose to encode information about each dataset via a small random sample of instances (e.g. 10-30). The approach then defines a set of parametrised gaussian kernels (one per dimension) where the parameters (the importance of each feature and the scaling for each feature) are learned using a permutation invariant neural network allowing to model non linear dependencies between the features and the target. ", "questions": "The paper would gain strength if the authors could show \n1) a comparison with the baseline proposed in Neg 2), i.e. compare against the baseline of training a permutation invariant (deep) neural network on the task of learning the feature selection mask directly.\n2) a statistical test for the ablation study, perhaps the one suggested in neg 3).", "limitations": "yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pos\nGiven the known HSIC Lasso approach to define the optimization objectives, the authors propose a ANN architecture to learn the few (2) parameters of feature-wise kernels which allows them to be sample efficient.\n\nNeg\n1) The introduction and layout of the material could probably be improved to streamline the introduction of the necessary concepts in an orderly fashion. \n\n2) The methods relies heavily on the capacity of the ANN to generalise le selection of features across related problems. To better clarify the contribution of the approach one should also compare against the baseline of training a permutation invariant (deep) neural network on the task of learning the feature selection mask directly, i.e., running some feature selection approach to obtain which features to select for each dataset. Note that in this case the feature selection performed for each task could fully exploit all the instances in the respective datasets and obtain better feature importance estimates.\n\n3) The ablation study is important and well conceived. It would however be of interest to be able to show which element is yielding significant differences (as each ablation seems to be within the 1% range). Perhaps performing some statistical test using the approach explained in [Janez Demšar. 2006. Statistical Comparisons of Classifiers over Multiple Data Sets. J. Mach. Learn. Res. 7 (December 2006), 1–30] and readily available as a python package in [https://github.com/sherbold/autorank].", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "aoQjwFdHLfs", "reviewer": "Reviewer_WvPt", "summary": "The paper proposes a meta-learning model for feature selection. The setup assumes the availability of labeled instances from several related tasks, and at test time, we only have access to a small number of labeled instances from the target task (called support set). A neural network infers kernel functions from the support set, and a Hilbert-Schmidt Information Criterion (a criterion for feature selection) based objective function is defined for optimization. The optimization problem is solved via the Iterative Shrinkage Thresholding (ISTA) algorithm. Results shown on several real-world datasets such as MNISTr, ISOLET, and IOT show the strong performance of the proposed algorithm.", "questions": "How are you able to learn efficient neural network representations for a task from the support set (Equation 4) since by definition you assume you only have access to a few samples only? ", "limitations": "No potential negative social impacts of the research.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "++ The paper is well written and puts itself nicely in the context of the previous work. The proposed meta-learning approach appears novel, though it involves a complicated optimization procedure.\n\n++ The experimental results are strong and show solid improvements over state-of-the-art baselines. \n\n-- The paper assumes that only a few labeled observations are available from the support set. It is unclear how a highly parameterized neural network is reliably estimated from so few samples in Equation 4?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept