Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a method for unsupervised feature selection which is based on Dynamic Sparse Training applied to an autoencoder with a single hidden dimension and trained with reconstruction loss. The number of selected features is given as an hyperparameter along with sparsity level of the NN. Fixing the number of epochs to be 10 for a proposed method while the other methods are trained with 100 epochs, produces comparable results in terms of accuracy to both supervised and unsupervised methods. 
strengths_and_weaknesses: Strengths:
+ The method reduces the number of epochs x10 less than other methods
+ The experiments conducted on datasets that come from different domains: image, speech, time series, text, bio
+ Simple method, with clear description, easy to follow
+ Single loss term without additional reguralizations
+ Outperforms almost in all datasets selected for the experiments

Weaknesses:
 - Some related recent works are missed, f.e. DUFS [1] - the method outperforms both CAE and MCFS on real datasets
 - The author has a claim on “robustness of our method in extremely noisy environments and its effectiveness for datasets with very high dimensional feature space and a few training samples”, which is misleading. Noisy environment is defined both in terms of the number of noise features which are unrelated to dataset labels, and also in terms of noise in data samples. This work doesn’t provide any experimental results on datasets with noisy samples. 
- The datasets that were chosen to validate the effectiveness of the method are not challenging since the number of samples in the datasets larger or near the same as the number of features.
- The supervised methods chosen for comparison are mostly classical non-NNet methods so they couldn’t serve as an upper bound (in terms of accuracy) for the proposed unsupervised method, while only a single NN-based method was chosen.
- The number of target selected features should be provided as a hyperparameter. 
- The experiments run only 5 times for each method and dataset, I think it should be at leaset 10 times.

[1] Lindenbaum, Ofir, et al. "Differentiable unsupervised feature selection based on a gated laplacian." Advances in Neural Information Processing Systems 34 (2021): 1530-1542.

questions: * Could you please provide a more detailed comparison to the QS method. Although your method outperforms it, the improvement is marginal in 7 out of 10 datasets (in t.o. accuracy)?
* Is it a typo in Table 2 (AEFS has the maximal accuracy on FashinMNIST)?
* Have you re-implemented other methods by yourself or used them as-is? I didn’t find them in your supplementary.
* Line 230 (Consistency and Stability) - I think it is better to provide a numeric comparison to support your claim. 
* Why are some results missing from Table 2?
* Once you compare your method to classical unsupervised FS, I think it could be interesting to compare the total training time between non-* NNet and NNet-based methods.
* Have you conducted experiments on sparsity level hyper parameter, how does it affects the performance of your method (accuracy/sparsity tradeoff)?

limitations: The limitations are mentioned, also the social impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose an extension to a recent auto-encoder-based sparse network approach for feature selection/extraction. Specifically, they extend the previously proposed drop and grow approach for inducing sparsity by leveraging the contribution of an input or output neuron to the reconstructed output. Experiments on 10 different datasets show that the proposed approach provides statistically significant gains in computational cost compared to the previous approach(es).
strengths_and_weaknesses: Strengths
1. Easy to read and follow paper.
2. Lucid experimental details and crisp presentation.
3. Fair coverage of competing classical approaches.
4. Offer to make the code publicly available.

Growth opportunities
1. The work is an incremental improvement over the previous approach. The proposed approach is intuitively motivated with several judgement calls (on leveraging neuronal weights differently) that are not theoretically justified.
2. The field is moving away from explicit feature selection. Hence, the impact of this work might be limited.
questions: 1. Beyond the classical classification tasks, how would the proposed approach fare against the state-of-the-art in CV, NLP, and graph ML tasks?
2. Why are only the input and output neurons subject to the drop and grow process but not the hidden neurons?
3. I can understand the improvements in efficiency but why does the classification accuracy improve with the use of WAST? Section 5.1 describes this additionally but doesn't explain the root causes.

limitations: Yes, the authors have addressed many of the limitations and societal impact. They might additionally wish to consider the questions above.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents an effective unsupervised method on feature selection via auto-encoders, inspired by the attention mechanism. It acquires the idea of the attention framework and applies it onto the weight-zeroing scheme of the autoencoder recursively, resulting in better training efficiency and promising performance on multiple tasks.
strengths_and_weaknesses: ## Strengths
1. This paper clearly addresses its difference from earlier state-of-the-art level work and identifies its main contributions clearly.
2. The presentation is very clear, with solid statements and references. At least I did not spot any typo or grammatical mistakes there.
3. The novelty is clearly confirmed and proven in the paper, with rigorous mathematical proof.
4. This paper clearly addresses the potential limitation and societal impact.

## Weaknesses
1. The dataset used for benchmarking is a bit out-dated. For example, for spoken digit recognition task, we may want to replace it with a little bit more popular and up-to-date sets such as TIMIT and [yesno](https://www.openslr.org/1/). They are not necessarily harder than the one covered in the paper, but clearly has more work pieces at state-of-the-art level.
2. Also, instead of spoken digit recognition, voice command recognition might be a better choice as tasks, as this paper addresses a lot on real-time application and usage.
3. The work lacks comparison with ad-hoc/hand-crafted features and task-specific features. For example, for spoken digit recognition, one additional experiment with common acoustic features such as [MFCC](https://en.wikipedia.org/wiki/Mel-frequency_cepstrum) may help. But this is trivial.
questions: 1. In equation 3), how did you select the \lamda hyper parameter to balance the two components?

2. In the network propagation, is the output processed with a softmax at the end?

3. What do you think on your work's difference with [Dropconnect](https://proceedings.mlr.press/v28/wan13.html)? I know there definitely are some, but as a popular method, you may want to address it.
limitations: I think the authors have adequately addressed the limitations and potential societal impact of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a novel method for unsupervised feature selection called WAST, which is an adaptation of Quick Selection (Atashgahi et al 2021) to give a more efficient search. The general idea of this and related methods is to pass the features through a simple autoencoder and to keep features which are more important. The main difference between WAST and QS is that, on each epoch, after dropping low-importance weights from the autoencoder, QS regrows random weights, WAST adds weights in a way which takes into account the importance of the weights in the previous epoch.
strengths_and_weaknesses: The paper demonstrates good performance (for K=50) on a large number of benchmarks, often beating supervised baselines. More importantly, the paper demonstrates much better performance than *comparable* benchmarks over many values of K, including QS. The paper also shows that WAST is slightly more efficient than QS (which is already much more efficient than some other related methods). 

The benchmarks in question are evaluated on whether or not they retain accuracy. However, typical supervised variable selection is evaluated on whether it selects the *right* features, on artificial datasets for which the true features are known. While one could argue that the objectives of unsupervised feature selection are not the same, I think it is quite reasonable, given that the method presented here is competitive with supervised methods, to ask whether it can also go farther than just picking out useful features to picking the correct ones. 

The technical advance is somewhat minor, given how close the algorithm is to QS, but on the other hand, the gains are excellent.
questions: QS assigns some importance to the fact that they use denoising autoencoders (because adding noise is supposed to encourage the network to find the most important features), but WAST does not appear to add any noise. Do you have any comment on this? What does this tell us about the approach and why it works?

There is also something quite disturbing about the fact that this method is competitive with supervised methods, given that their objectives are so different. What does this tell us about the approach and why it works?
limitations: The discussion seems well thought out.
review_title: null

Meta-review: metareview: After the rebuttal and discussion, all reviewers recommend acceptance of this paper to some degree. The paper has benefited from a careful review by reviewer U347 and additional experiments and clarifications performed by the authors in response to the reviewer's concerns. All reviewers noted that the paper is clearly written, the proposed method is simple and easily implemented, and that it appears to perform quite well.

confidence: Certain
Decision: Accept