MetaFS: An Effective Wrapper Feature Selection via Meta Learning
Authors: Zheyi Pan
Chunyang Li
Songyu Ke
Haoran Xu
Junbo Zhang
Ye Yuan
Yu Zheng
Venue: Submitted to ICLR 2023 (2023)
Keywords: Meta Learning
feature selection

Abstract: Feature selection is of great importance and applies in lots of fields, such as medical and commercial. Wrapper methods, directly comparing the performance of different feature combinations, are widely used in real-world applications. However, selecting effective features meets the following two main challenges: 1) feature combinations are distributed in a huge discrete space; and 2) efficient and precise combinations evaluation is hard. To tackle these challenges, we propose a novel deep meta-learning-based feature selection framework, termed MetaFS, containing a Feature Subset Sampler (FSS) and a Meta Feature Estimator (MetaFE), which transforms the discrete search space into continuous and adopts meta-learning technique for effective feature selection. Specifically, FSS parameterizes the distribution of discrete search space and applies gradient-based methods to optimize. MetaFE learns the representations of different feature combinations, and dynamically generates unique models without retraining for efficient and precise combination evaluation. We adopt a bi-level optimization strategy to optimize the MetaFS. After optimization, we evaluate multiple feature combinations sampled from the converged distribution (i.e., the condensed search space) and select the optimal one. Finally, we conduct extensive experiments on two datasets, illustrating the superiority of MetaFS over 7 state-of-the-art methods.

Under review as a conference paper at ICLR 2023
METAFS: AN EFFECTIVE WRAPPER FEATURE SELECTION VIA META LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Feature selection is of great importance and applies in lots of fields, such as med- ical and commercial. Wrapper methods, directly comparing the performance of different feature combinations, are widely used in real-world applications. How- ever, selecting effective features meets the following two main challenges: feature combinations are distributed in a huge discrete space; and 2) efficient and precise combinations evaluation is hard. To tackle these challenges, we propose a novel deep meta-learning based feature selection framework, termed MetaFS, containing a Feature Subset Sampler (FSS) and Meta Feature Estimator (MetaFF), which transforms the discrete search space into continuous and adopts meta-learning technique for effective feature selection. Specifically, FSS parame- terizes the distribution of discrete search space and applies gradient-based meth- ods to optimize. MetaFE learns the representations of different feature combina- tions, and dynamically generates unique models without retraining for efficient and precise combination evaluation. We adopt a bi-level optimization strategy to optimize the MetaFSS. After optimization, we evaluate multiple feature combina- tions sampled from the converged distribution (i.e., the condensed search space) and select the optimal one. Finally, we conduct extensive experiments on two datasets, illustrating the superiority of MetaFS over 7 state-of-the-ar methods.
INTRODUCTION
Human activities and real-world applications generate a huge amount of data, where the data has a large number of features. Sometimes, we need to identify the effects of different features and select an optimal feature combination for further study. For example, on online shopping websites, marketer analyze the purchase data to study the purchasing preferences of different peoples and con- struct the users' profiling/Allegee et al. (2020), which need to select a few number of representative shopping categories for further study. Supervised feature selection (FS) Cai et al. (2018b), making full use of label (e.g., the group classifications) information, is apposite to solve these problems.
Numerous supervised feature selection frameworks have been proposed, including filter Miao & Niu (2016), embedding [ibshirani (1996); Gui et al. (2019), and wrapper [El Aboudi & Benhlima (2016) frameworks, shown in Figures D Ia. b, and c). Filter frameworks evaluate each feature individ- ually while embedding frameworks greedily weighting features. They both evaluate features instead of combinations and the selection results are less effective. Wrapper frameworks traverse feature combinations and directly score them by wrapped supervised algorithms, e.g., K-nearest neighbors (KNN) and Linear Regression (LR), which significantly improve the effectiveness of feature selec- tion Sharma & Kaur| (202T). However, the number of feature combinations grows exponentially with the number of features. Evaluating such large combinations is computationally intensive, and consequently a small number of evaluations would limit the improvement the final performance. In summary, there are two major problems remaining in wrapper frameworks, which are listed as follows:
Problem 1: How to search feature combinations in a large discrete space? If we select effective features from a feature set with size need to consider (w) possible feature combinations, which forms an exponentially large search space. It's impractical to traverse and compare all feature combinations from such huge discrete space.

---

Under review as a conference paper at ICLR 2023
Fealures
Fealure Subset
Evaluation Fuuction
Peature Score
Subeet Score
A Fealures
A Festures
Feslume
Peatums
UUpdate
(a) Filter Framoworks (b) Embedding Frameworks (c) Wapper Fmmmewrks
(d) Meta Feature Selection
Figure 1: Supervised Feature Selection Frameworks and our MetaFS
How to efficiently and precisely evaluate feature combinations? To evaluate subsets, we usually train an model from scratch for each subset, which is inefficient and time-consuming. To reduce the time consumption, researchers prefer to use some simple model such KNN and LR as the evaluation model Rostami et al. [202T) However, these models are hard to capture the complex correlation among features and are hard to precisely evaluate different subsets (See Section 5.3)
Problem 2:
To address these problems, as shown in Figure I d). we propose a meta-lcarmnng-uusee based feature selec- tion framework (MetaFS). MetaFS first transforms the discrete searching problem into continuous optimization problem over a feature combination distribution, in which good feature combinations own higher sampling probabilities than bad ones. Specifically, represent the probability distribution function, such that it can be easily optimized. Next, we sample feature combinations from our distribution, and adopt deep meta-learning technique, which main- tains the combination representations and generates unique evaluation model for each combinations without re-training, to solve the problem of efficient and precise feature combinations' evaluation. Then, according to the evaluation results, we increase the probabilities of good combinations and decrease bad ones by employing gradient-based method to optimizing the probability distribution function. Finally, we search the optimal combination in condensed search space. The main contri- butions of our work are four folds:
adopt differentiable parameters to
We propose a novel deep meta-learning-bare baaned feature selection framework, entitled MetalSS. It transforms the discrete search space into continuous and applies bi-level optimization, that ternatively condenses the continuous continuous search space and learns meta-learning-basee combination evaluation network for effective feature selection.
We propose Feature Subset Sampler (FSS) to parameterize the distribution of search space. which allows us using gradient-base methods condense the search space, effectively reducing the difficulties in the searching process.
We propose Meta Feature Estimator (MetaFE) to efficiently and precisely evaluate feature com- binations. It learns the representations of different feature combinations and generates unique evaluation model without re-training each feature combination for precise evaluation.
We conduct extensive experiments on two real-world benchmark datasets to verify our framework. The experimental results demonstrate the superiority of MetaFS.
RELATED WORK
Feature Selection Methods Cai et al. (0018) can be classified into supervised, semi-supervised
Yu et al. 20188; Liu al. (2019) and unsupervised methods [iu al. (20IT): Tang et al. (2019).
Unsupervised and semi-supervised methods cannot make full use of label information, having poor performance on supervised tasks.
Filter frameworks, e.g., MI Yang & Pedersen (1997). independently compute the correlation met- rics features. While embedding frameworks Gui et al. (2019); |Singh et al. (2020) estimate the contribution (e.g., attention and weight) of features in the process of model construction. Some embedding methods are proposed recent years. For example, Gui et al. (2019) carries out the feature selection in the learned latent representation space with attention mech- anism, Singh et al. (2020) selects features by an unique loss function and well-designed neural network structure and Wang al. (209)) iterative trains hierarchical neural network for better
2

---

Under review as a conference paper at ICLR 2023
selection. However, they all focus feature scoring, but the greedily chosen features with high scores are usually not the optimal combination Wiegreffe & Pinter (2019); Jain & Wallace (011)).
Many works El Aboudi & Benhlima (2016) divided wrapper frameworks into heuristic- meth- ods SSharma & Kaur (202I): Rostami et al. (02T)) and sequential selection methods Gramitto et al. (2006). Sequential selections consecutively add (or eliminate) features from empty (or full) set to select features. Whereas heuristic methods applies heuristic algorithms to search the best subset, such as particle swarm optimization (PSO) Karasu et al. (2020), genetic algorithms (GA) [Sayed et al. (2019), and reinforcement learning algorithms Fan et al. (2020). However, these mentioned algorithms are still limited by the evaluation efficiency, hardly evaluating huge amounts of subsets during the selection procedure.
Meta Learning for Weight Generation Meta learning provides an alternative paradigm where machine learning model gains experience over multiple learning episodes Hospedales et al. (2020); Vanschoren (2018). For example, [Liu at (2018) learns the representation for different architecture and generate weights to accelerate architecture searching. And Li et al. (2019); Haa et al. (2016) gen- erate weights for different data domain with the shared information. These mentioned meta learning model all focus on the similar datasets, architecture and hyper parameters, which aims to accelerate the convergence and improve the prediction accuracy by learning the shared representation among datasets. However, there is still no work for evaluating feature combinations.
3 PROBLEM FORMULATION
Suppose there are n samples and m features. Let X [x1, .............................. Lm] E gaxm denote the input features, where each 2k I" is the k-th feature in input. Let y [y1..... Unl R" denote labels. Our goal is to find a feature subset = ... ...,. . with its corresponding input data X1 = [ =. 1 Raxr, such that we can achieve the minimum loss on the validation dataset by using the model trained with the training dataset. The optimization target can be formulated as:
① arg
arg min L (xprain .y train θ*)
min Exe
X X yy 01).
s.t.
O
1== (0,1)
where Il represent the set of all subsets, Ai is a binary variable denotes whether subset is selected or not, O₁ denotes the trainable parameters for subset I, and denotes the loss function.
4 METHODOL.OGIES
As there are exponential numbers of subsets and Aι are discrete, Eq. is hard to be optimized. Inspired by Lovász (975), we relax A to be continuous, i.e., A In this way, the subset with the maximum d is equivalent to the optimal result in Eq. 1 and we can apply gradient-based method to continuously optimize the feature selection.
In this paper, we propose MetaFS, consisting of two components: (1) Feature Subset Sampler (FSS) and (2) Meta Feature Estimator (MetaFF). Specifically, FSS parameterizes a subset sampling dis- tribution to model the continuous coefficients d and samples subsets for MetaFF. Next, MetaFF, applying meta learning techniques, efficiently generates unique models without re-training to pre- cisely evaluate sampled subsets. Then, FSS and MetaFE are collaboratively optimized to condense the search space (increase the sampling probabilities of good subsets) and finally find the optimal one. In the following subsections, we'll show the structures of FSS, MetaFE, and the collaborative optimization process.
4.1 FEATURE SUBSET SAMPLER
There are (m) coefficients λ in total, which is too large and cannot be directly maintained. As the constraint of \1 (after relaxation) is similar to that of a probability distribution, we propose Feature Subset Sampler (FSS), using a probability distribution to model these coefficients:
A1 Poet (T), I {ir, {ir..... ........ .......,)) 4 E EN
(2)
3

---

Under review as a conference paper at ICLR 2023
where Peet (I) is the sampling probability of subset I. For each sampling, select features according to features' selecting probabilities without replacement. Suppose p; is the probability for the i-th feature to be select in single sampling, the subset sampling probabilities are formulated
II,, Pi,
Pott (I,) = Exgellligen Pi;
(3)
where II represent the set of all possible subsets. And when the features' selecting probabilities is fixed, e.g., sampling subsets, the denominator is constant and the subset sampling probability is proportional the product of the features' selecting probabilities, i.e., (I, x Ik,, Pi:"
FSS maintains m trainable features' importance с [ci,..., Cm] E R to model the feature se- lected probability Ipı,....PPy Pm.. In this paper, we employ softmax function and define the selected probability of feature i is:
exp(c:)
P.
(4)
k=1 exp(cx)
Then, the sampling probability of subset I, is proportional to the exponent of the sum of correspond- ing features' importance:
Pee (I.)
II PS JEI;
II JEL,
exp(c;) = exp Σ c
(5)
JEL,
In the training process, FSS calculates the feature selecting probability ps according to the main- tained feature importance and samples subsets without replacement for MetaFE. Then, MetaFE calculates the evaluation of sampled subsets and feed backward for FSS to increase the sampiling probabilities of good subsets. (See detailed training stage in Section 4.3
4.2 META FEATURE ESTIMATOR
Conventionally, when we evaluate different subsets, we need to train unique model from scratch for each them, which is time-consuming. Intuitively, similar feature combinations take similar effects, which means that there are inherent correlations among the model parameters that learned from similar subsets. To make full use of such correlations, we propose Meta Feature Estimator (MetaFE), learning the representations of different feature subsets, generate unique model param- eters for each sampled subset without re-training, for efficient and precise subset evaluation.
Suboet
Subset-
MetaFC
Maintained
Extracted
Maintained
Generated
Meta Learner
ReLU
Subsett
de
Wi
Mean
ReLU
28
Subset-
MrtaFC
d,
di dout Meta Parameters
dlout Weights PC
Feature Embeddings
Subset Embedding
(a) MetaFE
(b) MetaFC
(c) W eight Generation Process of Meta L eamnr
Figure Meta Feature Estimator Framework
As shown in Figure aa), MetaFE is composed of activation functions (e.g., ReLU and Sigmoid) and multiple Meta Fully Connected layers (MetaFC). Figure 2 shows the structure of MetaFC, which is a fully connected layer that calculates the output by generated parameters Wr that generated by Meta Learner for subset I, formulating as:
Cout Fin WT b, Wi Learner(T)
Meta Learner maintains different subsets' representations by trainable feature embeddings. And it generates the weight parameters W1 according to the subset representation. As shown in Fig- ure c), let [ei..... cm. m. denote trainable feature embeddings. For each input subset = ii...., }, we suppose that all features have similar impact on model inference, meta learner first extracts the corresponding feature embeddings E ( Rrxd, and fuses them
4

---

Under review as a conference paper at ICLR 2023
e₁ by mean function, shown in Figure (d). Then, meta learner generates the weight parameters Wi E Rududdu by trainable fully connected neural network (FC network) according to the learned subset embedding et. And finally, we apply W1 to compute Wout according to Eq.
into subset embedding e1
Note that instead of training FC weight parameters from scratch, MetaFE generates the parameters from subset embeddings and meta parameters (Figure(p)) this way, MetaFE can generate unique fully connected neural networks for each input subset, and this generating process do not require re-training process, which can significantly improve the evaluation efficiency feature selection.
4.3 SEARCHING PROCEDURE
Feature selection (Eq I can be regarded as a bi-level optimization problem Colson et al. Inspired by Cai et al. 2018a), employ a collaborative optimization procedure to optimize MetaFE and FSS to finally search the best subset, which contains three stages and shown in Figure a).
2007)
Pixed Parameters
Training Parameters
Updating FSS
ppdale importaane
Pre-traiming MetaFe
Puslk) pibset
Partt -Tarthe
Weighted Sum
Data
FSS
MelaFE
Loss
Cailshorntive Optmmeration
Tw culinnd
Uydatin
(b) Updating Feature Subeet Sampler
FES
MesEE
Updating MetaFE
Upda to meta parameters & feature embeddings
7un02 piheet
Sampling Soarching
D
FSS
MetaFE
Sum
1 1osss
Searching Procedure
Updating MetaPE
Figure 3: Training Process of Meta Feature Selection
Pre-training MetaFE. As MetaFE initially cannot compare the effectiveness of subsets without feeding any training data, first pre-train MetaFE by using subsets sampled from uniform distri- bution. In this way, MetaFE could recognize the better feature combination to guide the optimization of FSS. The detailed training process of MetaFF is same that in collaborative learning.
Collaborative learning. FSS and MetaFF are alternately optimized to condense the search space described by FSS and increase the evaluation accuracy of MetaFE, simultaneously:
Updating FSS. FSS is trained to increase the sampling probability of good subsets and decrease the bad ones to condense the search space. As shown in Figure Bb), we first sample batch subsets and evaluate them by MetaFFE, and then calculate the gradient descent direction of features' importance to update FSS. Specifically, for sampled subsets [I.... Is,] with the size number of bg, we select a batch of samples for each of them, and feed these samples into MetaFE to calculate the supervised learning loss (subset evaluation) (4,,..,y Jo.] of different subsets. Finally, we update the feature importance e to increase the probability of good subsets, such that a subset with lower loss need to have higher sampling probability:
c² = argmin Paet (Ix) argmin
Lg exp
Σ Ci.
EEL.
where Poet (IR) is the sampling probability subset I& that exponentially proportional to the sum of corresponding features' importance (Eq, 5). In this work, we employ gradient descent method to update the feature importance c*.
Updating MetaFE. Similar to updating FSS, MetaFE is updated by batch gradient descent. For each update, first samples b, subsets and select a batch of samples [X,,...,, X1,, Xhuur for each of them, where ba is the batch size of samples. Then, we updates MetaFE to increase the accuracy of subsets' evaluation, such that,
min 11
L 2(11, y: O1,),
(8)
5

---

Under review as a conference paper at ICLR 2023
where is the loss function, 11 Ig and all trainable parameters in MetaFE (including subset embeddings and meta parameters). As result, freeze the trainable parameters of FSS and optimize the trainable parameters θ with the following formula:
[[ θ) is the generated parameters that produced by subset
θ arggin I 9: θ)).
Then we can apply gradient descent to optimize all trainable parameters in MetaFE, and the gra- dient of O be easily computed by prok-propagttion according chaining rule.
Sampling and Selecting, when the search space is converged, MetaFS samples and evaluates mul- tiple subsets. Finally, select the optimal subset result.
EXPERIMENTS
In this section, we conduct experiments on two real-world datasets to show our superiority in three aspects: 1) the performance of the selected feature subsets; the efficiency and precision of subset evaluation; 3) the convergence of subset search stages.
5.1
EXPERIMENTS SETTING
5.1.1
DATASETS AND METRICS
The experiments are conducted on two datasets, one of them is public, listed as follows:
This dataset is from online shopping website, which aims to estimate the customs' gender according to 21912 different shopping categories (features). This data is sparse and huge, which has more than million samples. To reduce the time usage, only select 10 thousand samples for baselines to evaluate each subset (use samples in MetaFS).
Shopping:
Gisette Dua & Graff (2017): Gisette is a handwritten digit recognition dataset, which has 7000 samples and 5000 features. And most of the features are highly correlated.
For all datasets, we simply select 50 features in experiments, so that we can artificially analyze them in downstream works. We divide all samples into two parts with ratio of 8:2 for feature selection and testing. For the methods requiring validation dataset, we further divide the samples for feature selection into two parts, where 80% of them for training and 20% of them for validating. To test the performance of the selected subset, we extract the selected features and train fully connected neural network (FCN) from scratch with an early-stop mechanism. And then we evaluate the subset performance on the test dataset. We also such re-training processes 10 times, and take the mean values as the final results on Gisette dataset to reduce the evaluation bias as it has less samples and easy to fall in over-fitting We adopt precision, recall and F1-score to evaluate the results.
5.1.2 BASELINES
We compare our MetaFS with 8 different methods, including four filter and embedding methods,
MI-KBest Yang Pedersen (997): It ranks features by Mutual Information compared with label information. And then select le features with the highest scores.
Lasso Tibshirani [996): It trains linear classification model (logistic regression) via /l penalty, and select the features whose the corresponding weights are non-zero.
AFS Gui al. (2019; It trains neural network and learns the feature importance with attention mechanism. We select the features that the corresponding attention weights are non-zero.
In addition, we compare MetaFS with four wrapper methods, where we apply Decision Tree (DT) the wrapped model, it has good performance subset evaluation (see Section 5.3

---

Under review as a conference paper at ICLR 2023
PSO Chuang et al. (2008): Particle Swarm Optimization (PSO) is heuristic method, which optimizes problem by iteratively trying to improve a candidate solution with regard to a given measure of quality. We use this method to search the best subset.
WOA Mafarja & Mirjalili 2018): Similar to PSO, we apply whale Optimization Algorithm (WOA) to search the subsets, which mimics the hunting mechanism of humpback whales nature.
HHO Too et al. 2 Hybrid High-Order (HHO) is another heuristic method, which searches subsets with several active and time-varying phases of exploration and exploitation.
We also compare a hybrid feature selection method, which is mix of filter and wrapper methods:
PassaC et al. (002)): It a Markov Blanket based feature selection algorithm that selects a subset features by considering their performance both individually as well as a group.
For all baselines, we turn the hyper-parameterss and limit the numbers of selections closing to 50.
Experiments settings In our implementation, we build a MetaFE model with two hidden layers, where the layer sizes set to [64, 32). The size of feature embeddings in MetaFE is simply set to 256. The batch size of samples and subsets are set to 128 and 32, respectively, In the experiments, train MetaFE the training dataset, update FSS and search the optimal subset the validation dataset. We employ cross entropy as a loss function in the experiments, and all experiments are conducted by using single Nvidia 2070 GPU.
5.2 PERFORMANCE COMPARISON
Table shows the evaluation results of the selected features. First, for the filter and embedding methods, i.e., MI-Kbest, Lasso, and AFS, have worse performances on Gisette dataset, as they evaluate features greedily and are hard to capture the cross- cross-correlation among features. Second, for the wrapper and hybrid methods, i.e. RFE, PSO, WHO, HOO, and PPIMBC, which need train plenty of models for evaluating different subsets, have bad performance on the Shopping dataset. The reason is that the Shopping dataset hard learn only with a few samples (see Section 5.3) and the methods easily fall into local minimum with imprecise evaluations.
Table 1: The Evaluation Performance of Selecting Results
Shopping
Gisette
Metrics
(1(%)
Precision($)
Recall(%)
(%)
Precision(%)
Recall (%)
MI-Kbest
0.7012
0.7045
0.7003
0.9456
0.9456
0.9456
Lasso
0.6625
0.6628
0.6624
0.9608
0.9607
0.9608
AFSelect
0.6870
0.6968
0.6865
0.9600
0.9599
0.9602
PSO
0.6782
0.6805
0.6775
0.9615
0.9615
0.9617
WHO
0.6721
0.6813
0.6718
0.9633
0.9632
0.9635
HOO
0.6256
0.6640
0.6336
0,9543
0,9543
0.9545
PPIMBC
0.4500
0.6136
0.5329
0.9725
0.9724
0.9728
MetaFS 0.7114
0.7151
0.7104
0.9726
0,9727
0,9725
Unlike the above methods, MetaFS achieves the best results on the two datasets. The reason that MetaFS, a wrapper framework, can directly compare different feature subsets during the selec- tion procedure. In addition, compared to the other wrapper methods, MetaFS applies meta-learning techniques, efficiently evaluating subsets without re-training. Moreover, the deep-learning-basse -based evaluation model can capture complex correlations among features, improving the evaluation 's pre- cision. With the precise and efficient subset evaluations, MetaFS could quickly filter out the bad subsets to condense the search space and finally tests much more subsets to select the optimal result.
5.3 STUDIES ON FEATURE SUBSET EVALUATION
To study the precision and efficiency of different evaluation methods, we sample 100 different sub- sets from the uniform distribution and converged subset space. Then, we compare the time usage of the evaluation process and the ranking similarities of the evaluating results. The compared eval- uation methods include four popular retrained learning methods, which widely used in wrapper

---

Under review as a conference paper at ICLR 2023
methods, including KNN, Linear Regression (LR), Support Vector Classification (SVC), and Deci- sion Tree (DT). In addition, we compare it with our proposed MetaFE. To illustrate the effectiveness of MetaFE, we also compared it to an FCN model that trained for all subsets without retraining.
Table 2: The Ranking Accuracy of Evaluation Results (RBO)
Retraining DT
Training Once
KNN
LR
SVC
FCN
MetaFE
Random
0.9092
0,9013
0.858233
0.8994
0.7190
0.9146
Gisette
Converaged
0.6371
0.5747
0.6065
0.6313
0.5311
0.6772
Random
0.6883
0.8061
0,7479
0.7600
0.6548
0,8518
Shopping
Converaged
0,5106
0,5971
0,6470
0.6839
0.5265
0.6523
As shown in Table we introduce Rank Biased Overlap (RBO) to evaluate the ranking similarities of different evaluation results, where higher RBO indicates higher similarity. The ground truth is the results from an FCN model re-trained for each subset. For the Gisette dataset, as it is easy to learn, almost all re-trained methods have good performance in the uniform sampling distribution. However, when the distribution converges, these models are hard to distinguish the nuances among good subsets precisely. For the sparse Shopping dataset, the re-trained methods produce worse results in all subset sampling spaces. The reason is that the efficiency of evaluation limits them (see Table 3 and they could only evaluate 10 thousand samples for each subset. Different from the above methods, MetaFF, based on a fully connected neural network, has an excellent ability to capture complex non-linear correlations among features. Moreover, it can almost ignore the time-consuming training stage and can be trained with all visible samples to improve the inference performance. Compared FCN, MetaFE could generate unique model parameters for different subsets, which has good generalization for subset evaluation. As result, MetaFE has precise evaluation performance in experiments, especially for datasets with large number of samples.
Table 3: Time Usage of Evaluating One subset (ms)
Retraining
MetaFE GPU
KNN
DT
LR
SVC
CPU
GPU
Shopping( (10k)
7930.96
3715,88
3722.36
11572.44
375,85
31.26
73.6187
Gisette
533.51
35.65
33.76
366.34
27.92
7.24
13.905
Table 3 shows the time usage of different methods for evaluating one subset, where the re-trained methods only evaluate 10 thousand samples on the Shopping dataset. We also show the time usage of MetaFE under different devices, where the GPU represents the estimated time usage that considers the training process. The re-trained methods cost plenty of time for training models. Worsely, the used time increases with the size of trained samples. Conversely, MetaFE does not need re-training models, costing less time. Based significantly accelerates the calculation speed. Even though taking the training process into account (GPU*), the evaluation process is still much more efficient than re-trained methods. Therefore, MetaFE has excellent advantages in the precision and efficiency on subset evaluation.
neural network, it can fully use GPU parallel technique, which
5.4 CONVERGENCE ANALYSIS
In this subsection, we will analyze the convergence of Metal'S during the training process. We first compare the convergence curve of our MetaFS (FSS +MetaFE) with two variants to show the effectiveness of our training process: 1) FSS+FCN: replacing MetaFE with an FCN model, and 2) Inta+MetaFF:: removing the FSS and sample subsets from an initial uniform distribution.
We take the Gisette dataset as a typical case, and Figure 4 shows the Fi-score varies over time. In the pre-training stage, all models are trained with random subsets (sampled from the uniform distri- bution). MetaFE can capture the inherent correlation of different subsets, having a higher prediction precision than FCN. In the collaborative learning stage, using FSS can increase the probabilities of good subsets and condense the search space. As a result, the estimator (MetaFE or FCN) can be trained with similar subsets and finally improves the precision of model inference. To show the change in model performance during training stages, we compare the difference of the subset evaluations of MetaFE that trained on different training stages. The ground truth is the test from
8

---

Under review as a conference paper at ICLR 2023
Giselte Dalasel
1.0
FSS+MataFE
PSS+IC
0.9
Uni MMttaFE
5 0.8
0.7
Pre-train
Collaborative Learning
0.6
Figure 4: The Convergence Curves in Giseete Dataset
well-trained FCN model, and the subsets are sampled from two different distributions: 1) uniform distribution and 2) converged subset sampling space from well-trained FSS.
Figure 5.4 shows the ranking similarity (RBO) change along the training iteration, where the blue lines are the results of MetaFEs that were trained on different training stages, where the evaluation subsets follow uniform distribution, corresponding to the RBO result on the left axis. The red lines are the results of MetaFEs evaluated with converged subset space, corresponding the right axis.
Gisette Dataset
Shopping Dataset
0.92
0.75
0.8
0.70
(Uniform)
0.50
0.70
ComnerAN
(Uaiform)
0.
0.55
voraged)
0.65
0.88
4.50
0.5
0.35
0.84
Unifera Dhatrbuttia
Conseraged Dnttiivation
0.50
0.3
Conwwagge Derraattion
Vaiform Datributio
0.50
Iteration
Figure 5: Ranking Similarity (RBO) of MetaFF Change along the Training Iteration
First, the RBO curves along the training iteration are unstable. One reason is that, since training FCN models for evaluating subsets is time-consuming, we only sample 100 subsets, which is small and cannot reflect the actual performance. Overall, the general trend of the RBO curve that eval- uates converged subset space is increased with the training iteration. The reason is that Metal is collaboratively trained with the FSS and could specifically increase the precision of evaluating good subsets with the growing training iterations. For the subsets that follow uniform distribution, the performance of MetaFE decreased with the iteration as it is based on a neural network, having a characteristic of forgetting old knowledge Biissialkaa al. 2020). However, rative training stage alternately optimizing the subset sampling space (FSS) and MetaFE, ensuring that MetaFE is always in the optimal state when updating the subset sampling space, can effectively avoid this problem. As result, MetaFS has good performance throughout the training stages.
proposed collabo-
6 CONCLUSION AND FUTURE WORKS
In this paper, we proposed a novel Meta Feature Selection framework (MetaFS) for effective fea- ture selection. MetaFS, consisting of FSS and MetaFE, transforms the discrete search problem into continuous and adopts meta learning techniques to evaluate subsets without retraining. Specifically, the proposed FSS parameterizes the search space and adopts gradient- based methods effectively reduce the search difficulties. Meanwhile, ensuring the evaluation precision, MetaFR could signifi- cantly decrease the time usage of evaluation over the selection procedures. MetaFS adopts bi-level optimization strategies optimize FSS and MetaFE, evaluating multiple subsets sampled from FSS and finally select the optimal one. We conduct extensive experiments on two datasets, demonstrating that MetaFS searches the promising subsets. In the future, will study the hyper parameters of MetaFS and simplify the operation to support rapid deployment in real- world scenarios.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sahar Allegue, Takoua Abdellatif, and Khalil Bannour. Rime: a spending- segmentation. In 2020 IEEE 29th International Conference Enabling Technologies: Infrastructure for Col- laborative Enterprises (WETICE), pp. 165-170. IEEE, 2020.
Magdalena Biesialska, Katarzyna Biesialska, and Marta R Costa-Jussa. Continual lifelong learning in natural language processing: A survey, arXiv preprint arXiv:2012.09823 2020.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint arXiv: 2.00322, 2018a
Jie Cai, Jiawei Luo, Shulin Wang, and Sheng Yang. Feature selection in machine learning: A new perspective. Neurocomputingg 2018b.
Li-Yeh Chuang, AssuhhWWtt Chang, Chung-Jui Tu, and Cheng-Hong Yang. Improved binary pso for feature selection using gene expression data. Computational Biology and Chemistry, 2008.
Benoit Colson, Patrice Marcotte, and Gilles Savard. An overview of bilevel optimization. Annals of operations research, 2007.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.
Naoual El Aboudi and Laila Benhlima. Review on wrapper feature selection approaches. 2016
International Conference Engineering MIS (ICEMIS), 2016.
Wei Fan, Kumpeng Liu, Hao Liu, Pengyang Wang, Yong Ge, and Yanjie Fu. Autofs: Auto- mated feature selection via diversity-awar interactive reinforcement learning. arXiv preprint arXiv:2008. 12001, 2020.
Pablo M Granitto, Cesare Furlanello, Franco Biasioli, and Flavia Gasperi. Recursive feature elim- ination with random forest for ptr-ms analysis of agroindustrial products. Chemometrics and Intelligent Laboratory Systems, 2006.
Ning Gui, Danni Ge, and Ziyin Hu. Afs: An attention-based mechanism for supervised feature selection. In Proc. of AAAI, 2019.
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. arXiv preprint 1oXiv:1609009100 2016.
Atif Hassan, Jiaul H Paik, Swanand Khare, and Syed Asif Hassan. Ppfs: Predictive permutation feature selection. arXiv preprint arXiv:2110.1011 2021.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey, Meta-learning in neural networks: survey, arXiv preprint arXiv:2004. N04452,00.0 2020.
Sarthak Jain and Byron Wallace. Attention is not explanation. arXiv preprint arXiv: /902, 10186, 2019.
Seçkin Karasu, Aytaę Altan, Stelios Bekiros, and Wasim Ahmad. A new forecasting model with prapper-based feature selection approach using multi-objective optimization technique for chaotic crude oil time series. Energy. 2020.
Yiying Li, Yongxin Yang, Wei Zhou, and Timothy Hospedales. Feature-critic networks for hetero- geneous domain generalization. In Proc. of ICML, 2019.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint 1rXiv 09055, 2018.
Keyu Liu, Xibei Yang, Hualong Yu, and et al. Rough set based semi-supervised feature selection via ensemble selector. Knowledge-Be based systems, 2019.
Rong Liu, Robert Rallo, and Yoram Cohen. Unsupervised feature selection using incremental least squares. International Journal of Information Technology & Decision Making, 2011.
László Lovász. On the ratio of optimal integral and fractional covers. Discrete mathematics, 1975.
10

---

Under review as a conference paper at ICLR 2023
Majdi Mafarja and Seyedali Mirjalili. Whale optimization approaches for wrapper feature selection. Applied Soft Computing, 62:441-453, 2018.
Jianyu Miao and Lingfeng Niu. A survey on feature selection. Procedia Computer Science, 2016.
Mehrdad Rostami, Kamal Berahmand, Elahe Nasiri, and Saman Forouzande. Review of swarm intelligence-bart baseee feature selection methods. Engineering Applications of Artificial Intelligence, 2021.
Sabah Sayed, Mohammad Nassef, Amr Badr, and Ibrahim Farag. A nested genetic algorithm for fea- ture selection in high-dimensional cancer microarray datasets. Expert Systems with Applications, 2019.
Manik Sharma and Prableen Kaur. comprehensive analysis of nature-inspired meta-heuristic techniques for feature selection problem. Archives of Computational Methods in Engineering, 2021.
Dinesh Singh, Héctor Climente-González, and al. Fsnet: Feature selection network on high- dimensional biological data. arXiv preprint 20Xi::0011.0822,, 2020.
Chang Tang, Meiru Bian, Xinwang Liu, and et al. Unsupervised feature selection via latent repre- sentation learning and manifold regularization. Neural Networks, 2019.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 1996.
Jingwei Too, Abdul Rahim Abdullah, and Norhashimah Mohd Saad. A new quadratic binary harris hawk optimization for feature selection. Electronics, 8(10):1130, 2019.
Joaquin Vanschoren. Meta-learning: A survey. arXiv preprint arXiv:1810003558.22 2018.
Q. Wang. J. Wan, F. Nie, B. Liu, C. Yan, and X. Li. Hierarchical feature selection for random projection. IEEE Transactions on Neural Networks and Learning Systems, 2019.
Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. arXiv preprint
arXiv: 04626, 2019.
Yiming Yang and Jan Pedersen. comparative study on feature selection text categorization. In Icml, 1997.
En Yu, Jiande Sun, Jing Li, and et al. Adaptive semi-supervised feature selection for cross-modal retrieval. IEEE Transactions on Multimedia, 2018.


Reviews:
[{"review_id": "wCX5q24DuCt", "reviewer": "Reviewer_MBeM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new approach for feature selection, which composes of two components: (1) A feature subset sampler, which learns the distribution of important feature subsets; and (2) a meta-learning based weight generator that quickly constructs a fully connected deep layer for any sampled subset. The method is demonstrated on two datasets. The empirical result is generally positive.", "summary_of_the_review": "Overall, I would recommend a rejection. As stated above, given that this is an empirical work, I believe the reported result is insufficient. The motivation also failed to convince me that there is a practical need for the proposed method.", "strength_and_weaknesses": "Strength: The approach is quite interesting and the result seems positive. The paper is clearly written.\n\nWeakness:\n1. The paper is not well motivated. \n-- For example, the criticism for embedding methods is that \"they all focus on feature scoring, but the greedily chosen features with high scores are usually not the optimal combination\". I do not agree with this. \n\n-- First, all neural networks are considered (embedding) feature extractors, and their features are learned optimally with backpropagation, which is not a greedy method at all. \n\n-- Second, there is no guarantee that the proposed method selects optimal features either. As far as I understand, any subset selection method is just a special case of an embedding layer with output dim = input dim and zero weights given to unimportant features. What would be the value added of making these weights exactly zero?\n\n2. The empirical result is not very convincing. \n-- What is preventing us from learning an end to end neural network to predict given all features? 20000/5000 features are not prohibitively large numbers of features. Maybe such naive approach would perform worse than the proposed method in the end, but the authors should at least try to include such a simple baseline to demonstrate that these datasets truly need feature selection to perform well.\n\n-- Why do we presume that the optimal subset has exactly 50 features? I think there should be a separate study conducted to verify this number. For example, will MetaFE still perform the best when the subset size is larger? Would there be scalability issues?\n\n-- Evaluating the ranking accuracy is a fair approach to understand the optimality of the selected features. However, it seems that the ranking accuracy is quite bad on sample subsets drawn from the converged sampling distribution. I'm not fully convinced that the reason for this is necessarily \"hard to distinguish among good subsets precisely\". Perhaps one way to verify this exactly is to construct a synthetic dataset with many duplicated features and observe if MetaFE can recover one unique copy of each feature.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "f3udTp91rgT", "reviewer": "Reviewer_z1i8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles feature selection problem by using meta-learning. They propose to transform the discrete search problem into continuous one and make use of meta learning to evaluate subsets without retraining. They parameterize the search space and apply gradient-based meta-learning in order to effectively reduce the search difficulties. The experimental results show that the proposed method outperforms many of the existing feature selection methodologies.", "summary_of_the_review": "In summary, I think the paper tackles the important problem of feature selection, also the method seems appealing as well. However, evaluations are a bit limited, so I encourage the authors to evaluate on more datasets and run all the baselines multiple times to judge the statistical significance.\n\nOverall, I think the strength outweigh the weaknesses, so I recommend weak accept.", "strength_and_weaknesses": "Strength\n- Paper is well written and easy to understand\n- The problem is important and applying meta-learning to solve the problem seems appropriate.\n- The empirical results are favorable, covering wide variety of existing baselines.\n\nWeaknesses\n- The authors only use two datasets for evaluation, which is quite limiting. I suggest the authors to use more datasets to properly assess the efficacy of their method.\n- The gap from the baselines are quite marginal, and all the experimental evaluation lacks confidence interval. So one cannot say that the evaluation results are statistically significant.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dkor_LxoGe", "reviewer": "Reviewer_3nfV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors address a challenging and important problem of feature selection. The new approach is a wrapper-type method that relies on meta-learning to select the best subset of features for supervised learning. The idea is to transform the large discrete search space into a relaxed continuous space and use gradient descent to find the optimal subset for classification/regression. ", "summary_of_the_review": "Overall the authors present a new method for an important problem. They present two components that should reduce the computational effort when using wrapper methods for FS. However, a large portion of the paper is not written in a clear way, and the experiments only focus on two binary datasets. One of them is semi-synthetic. Based on such a small evaluation, I can’t recommend accepting the paper.", "strength_and_weaknesses": "The problem is well-studied but still important for the community. The English level is satisfactory. The method itself seems novel, and the authors propose a way to save computational time by exploiting feature correlation and training sharing weights for models that are trained on the same or similar features. The method is quite intuitive, and the continuous relaxation was already demonstrated effective by other works for feature selection, for example:\n“Concrete Autoencoders” or “Feature Selection Using Stochastic Gates”.\n\nSome of the sections in the paper do not read well and are hard to follow. \nThe experimental evaluation is quite poor, the authors only apply the method to two binary data sets (one of them is half synthetic- GISETTE). Also, several leading feature selection baselines are missing.  It is really hard to judge the quality of the approach without additional experiments. The runtime improvements seem promising, but an ablation study is missing from their experiments.\n\n\nIn the following, I detail my comments point by point:\nP1) The authors mention in the introduction that they compare to SOTA in FS, this is not correct, several leading NN baselines are left out, for example:\\\\\n[1] Lemhadri et al. Lassonet: Neural networks with feature sparsity\n[2] Yamada et al. Feature Selection using stochastic gates, 2020\n[3] Balin et al .Concrete autoencoders: Differentiable feature selection and reconstruction\n[4] Singh et al. Fsnet: Feature selection network on high-dimensional biological data\n\nP2) Also, experimenting with two datasets is not considered extensive.\n\nP3) Wrong use of citation style, embed the names in the sentences or use brackets.\n\nP4)Saying that the selection results of filter and embedding method is less effective is wrong and misleading the reader. There are many settings where filter methods excel and save computational time. Furthermore, embedding methods can work with strong NN models and lead to SOTA results.\nConceptually wrapper methods can outperform them, but this might require an unfeasible amount of training time.\n\nP5)Problem 2: an model-> a model\n\nP6) GISETTE is a semi-synthetic data created for the NEURIP 2003 challenge, with most of the features being a nuisance, created artificially. The authors don’t even mention this.\n\nP7) Why do you only select 10K samples for baselines in the Shopping data? And use the full data for your method? This seems unfair.\n\n\nP8) We only select 50 features in experiments….??? What does this sentence mean?\n\n\nP9) You evaluate precision-recall and F1, but you don’t mention what? Namely does can also be evaluated in terms of the ability to identify informative features in GISSETTE (where the set is known). I assume you mean for binary classification, so what threshold do you use to set precision-recall?  \n\nP10) We turn the hyperparameters? Do you mean tune? Also, how do you tune them for all baselines? This is not explained.\n\nP11) The network used is quite narrow and shallow, nowadays theory and practice indicate we need to use wider and deeper models.\n\nP12) Section 5.3 is not clear. For example, “we sample 100 different subsets..” of what? I can try to guess what this means, but a paper should be clearly written for readers.\n\nStrong NN FS methods are missing in the related work and evaluations [1]-[4] above\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "dITOfJEIvwb", "reviewer": "Reviewer_joqh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a meta-learning-based feature selection framework that aims to select effective features for machine learning tasks. Specifically, this wrapper framework consists of two components, i.e., Feature Subset Sampler (FSS) which optimizes the discrete search space in a continuous way, and Meta Feature Estimator (MetaFE) which alleviates the cost of re-training machine learning models.", "summary_of_the_review": "This paper proposed a wrapper framework to solve feature selection problems. The idea of Feature Subset Sampler (FSS) optimizes the discrete space in a continuous way, and the Meta Feature Estimator (MetaFE) avoids the re-training problem. However, the paper is not well polished, the framework ignores feature-feature correlation, and it lacks solutions to deciding selection feature numbers.", "strength_and_weaknesses": "Strengths:\n1. The research problem is important and the idea of MetaFE is meaningful. Wrapper is one of the most widely used strategies in feature selection problems. The major challenge is the repeated and redundant training of the machine learning models. The proposed MetaFE tackle the re-training issue with meta-learning. It could be especially useful for big data.\n2. The FSS transforms an exponential combinatorial optimization problem into a continuous optimization problem, where most existing optimizers in deep learning can be directly adopted. This provides a new perspective for combinatorial optimization.\n\nWeaknesses:\n1. The manuscript is not well polished. There exist many grammar errors, e.g., 'study the purchasing preferences of different peoples', 'construct the users’ profilingAllegue et al.', 'employing gradient-based method to optimizing the probability distribution'.\n2. The correlation between features is ignored, which is one of the most important factors in feature selection problems. Essentially, in this paper, each feature is independently quantified by an importance number $c$. Feature selection needs a joint distribution of all features, instead of one distribution for each feature.\n3. The proposed method can not decide the number of selected features. Besides, there is a lack of analysis of the selected feature number.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject