Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: The authors provide a novel combination of known architectures to an important use case of reducing the density of required  measurements in sensor-fusion based temporal multi-class inference tasks. This has implications in energy consumptions of wearable sensors.  but could even generalise to measurement timings in clinical care to make the work of nurses more efficient, and reduce the stress caused by some medical procedures..

The authors represent a way to train consistent policy that predicts the best combination of sensors to estimate the state of the subjects.  They have found that a smaller set of features.  is more explainable than the full set of features.  However, I think that this somewhat of an overpromise.  The trained model gives the optimal density of the measurements and can discern also if old values and features measured are till OK for the inference.  This does not mean that those measurements are not needed at all in the features.  One can only argue that the required features can be estimated from the older measurement. So, the current set of active sensors is not the full set of required measurement values and can not be exclusively used to explain the logic of the system. Even more, the logic of the policy deciding the new  measurement is not discussed in an explainability context. The authors provide no data on this. It may be just an  estimate the derivative of the signal and ignore a new measurement, if it's time  derivative is small enough. 

As a summary , I support publication of the manuscript, provided  the authors modify the message on the interpretable features.
review_title: Very good paper, but some tuning required in its claims.

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents a learning-based binary sampling mechanism for feature selection. It filters salient feature dimensions by sampling from a Gumbel-softmax distribution, which is differentiable and can be trained with other network parameters. The proposed method is evaluated on several Human Activity Recognition (HAR) datasets.

The positive and negative points of this paper can be summarized as following:

pros:
+ This paper is well written and is easy to follow.
+ The experimental evaluations give positive results. 

cons: 
- Important previous works are missing. Learning to generate categorical samples for RNNs is not a fresh idea. Actually, [a] has already employs Gumbel-softmax to sample scales in order to dynamically control the temporal pattern learning; More generally, the topic of this paper is connected to a amount of previous works aiming to adaptively decide how/when to memorize/update the inputs/states, such as [b] and [c]. These works should also been cited by this paper.

- With these missing works taking into account, the novelty of this paper becomes incremental and contribution is trivial. Integrating Gumbel-softmax sampling with RNN cells is very straightforward, and the motivation of applying Gumbel-softmax is very similar to [a]. While [a] is proposed for general sequence tasks, the proposed method seems to work only for HAR with multi-dimensional inputs. 

- Since \tau is the only hyper parameter of Gumbel-softmax, evaluations on how the value of \tau could impact the performance can be important. Yet no such results are reported in the paper. From the original Gumbel-softmax paper we can see a sample can approximate to a one-hot vector when \tau is small and be closed to a uniform distribution when \tau goes large. So it is very likely that the performance will become unstable as \tau changes. Showing such experimental results could be definitely improve the paper quality. I would suggest to report the means and stds of accuracies with different sampling seeds.

Summary:
Considering the concerns listed above, I believe there are problems that outweighs the strengths of this paper. They should be fixed before acceptance.

[a] H Hu, et al. Learning to Adaptively Scale Recurrent Neural Networks. AAAI 2019
[b] V Campos, et al. Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks. ICLR 2018
[c] D Neil, et al. Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences. NIPS 2016

review_title: Although this paper is well written and reported results are positive, the novelty of this paper is quite limited. Besides, several highly-related previous works are missing, and important hyper parameter studies are not reported. These problems prevent me from rating the paper as acceptable.

final_score: rating: 3: Clear rejection
review_content: The authors tackle the important problem of feature selection. They propose to use differentiable gates with an RNN architecture to select different subsets of features for each time point. I think the idea and method are interesting, and the method could be useful. However, I have crucial problems with the way the paper is presented. Most importantly, the authors describe the l_0 relaxation of Bernoulli random variables as if it is their own contribution. They describe existing known results under a section titles “Methodology” as if they are the first to present Bernoulli random variables to feature selection or that they are the first to relax them using the Gumbel Softmax trick. They also use the word: “we derive” (p.3). This is wrong! And misleading! The same relaxation appears in [1] and used for model sparsification, the descriptions are almost identical to what appears in [1] with almost zero credit to the authors in [1] (a citation appears in related work in a different context). Bernoulli relaxation was already used for feature selection, in [2], and [3], these papers were not even mentioned. The reader can think the authors are the first to introduce such relaxation into the problem of feature selection, while this is again, clearly wrong.
The authors are well aware of that this relaxation was presented in [1], and in the experiment section they describe the baseline which solves (4) by citing [1] (citation [18] in their paper), this is again in contradiction to the way they describe the relaxation as if it is their own contribution. 

Putting these CRITICAL comments aside, I think the results are misleading. Specifically, comparing the average number of selected features to the (constant) number of selected features of the non-adaptive method is misleading. You need to compare the union of selected features by your method to the constant number, otherwise, there is no way to infer if this feature selection method can result in any compression of the model or could lead to training or inference speed up.  Given that this is what you measure since you still need all the features to use your model, what are the advantages of the method? Only interpretability?

The authors do not explain how the method is used in the testing phase, is the randomness removed? How exactly?
The authors do not explain how training/ testing is performed, this appears in the appendix but should be moved to the main texts.
The authors should compare the method to the distribution suggested in [1], which seems more suitable for feature selection than the Concrete distribution (used by the authors).
Citations are not in the correct ICLR format.

Some pros: I like the examples used in the paper as well as the comparison to ARM, ST, ST-ARM.
To conclude, I am voting to reject the paper, based on all the reasons mentioned above.

[1] Louizos, Christos, Max Welling, and Diederik P. Kingma. "Learning Sparse Neural Networks through $ L_0 $ Regularization." ICLR, 2018.

[2] Yamada, Y., Lindenbaum, O., Negahban, S., & Kluger, Y.  Feature selection using stochastic gates. ICML, 2020.

[3] Balın, Muhammed Fatih, Abubakar Abid, and James Zou. "Concrete autoencoders: Differentiable feature selection and reconstruction." ICML. 2019.


review_title: Review Dynamic Feature Selection

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes an RNN model for adaptive dynamic feature selection, for efficient and interpretable human activity recognition (HAR). From the intuition that human activity can be predictable by using a small number of sensors, the paper introduces an l0-norm minimization problem with parameter regularization, and provide a logic on formulating a dynamic feature selection model with relaxations. The difficulty of the discrete optimization problem is solved by differentiable relaxation, which is known as Gumbel-Softmax reparameterization techniques. The formulation is naturally led to an RNN model that uses histories as input with an additional sigmoid unit for adaptive feature selection. 

Empirical studies are performed to show the superiority of the adaptive feature selection network. Results are shown on the task of 1) UCI-HAR smartphone dataset with 561 features, 2) UCI Opportunity sensor dataset with 242 features, 3) ExtraSensory dataset with 225 features for multilabel binary classification. In particular, by using the adaptive feature selection technique, the average number of features necessary for HAR prediction can be very small (0.3%, 15.9%, 11.3% among all features) at any given time. Overall, the paper is well written. In particular, analysis results on three datasets are clear and detailed, so that the reader would be available to understand what sensors were necessary for HAR prediction.
 
The key concern about the paper is that the algorithm lacks practicality. To show the adaptive selection algorithm is efficient, it should be shown that the algorithm drastically reduces features that are not necessary for prediction over time, while maintaining the performance even in the lighter feature space. Although the average number of features selected by the adaptive selection algorithm for each snapshot is small, all features are entered as input, which may not help to speed up the algorithm. To claim that the algorithm is efficient, it is required to show that the computation cost can be saved. Also, based on the current experimental results, it is difficult to say that features that were not used in earlier timestamp will not be used in later timestamp with a different context. 
 
Minor comments and questions:
- Can you report the running time of each model? 
- Is this model working in an online setting without tuning? If yes, would you like to clarify? If no, may I think this technique is for maintaining a dashboard that informs important features every time to users by calculating feature importance over time?
- The performance of the adaptive method on the NTU-RGB-D dataset is quite poor. What part of the dataset do you think caused the difficulty in feature selection? Do all features important?
- The technical novelty seems to be low if the proposed model is an RNN with an additional sigmoid layer.
- Figure 2a does not have a ground truth blue line.

review_title: I wonder if it can be applied directly to the online setting, which gradually decreases the number of features.

Meta-review: The authors propose a methodoloy for dynamic feature selection. They use differentiable gates with 
an RNN architecture to select different subsets of features at each time point thus resulting in dynamic selection. 
The reviewers agree that the idea is interesting and the method could be useful and I share their opinion.

The majority vote is towards rejection. The overarching mwssage of the reviews is that the manuscript raises confusion in a number of points. I see this work as one with good potential for impact but its current presentation is confusing. The vivid discussion that it raised is also an indication of it. The authors have done a good job replying to the concerns and the questions raised. However, the reviewers were still unsatisfied with the authors response to their concerns.  I recommend rejection at this time, while encouraging the authors to take seriously the reviewers' requests for a clearer presentation of their approach's contribution in order to strengthen their paper for future submission.


Decision: Reject