Dynamic Feature Selection for Efficient and Interpretable Human Activity Recognition
Authors: Randy Ardywibowo
Shahin Boluki
Zhangyang Wang
Bobak J Mortazavi
Shuai Huang
Xiaoning Qian
Venue:  (2021)
Keywords: dynamic feature selection
human activity recognition
sparse monitoring

Abstract: In many machine learning tasks, input features with varying degrees of predictive capability are usually acquired at some cost. For example, in human activity recognition (HAR) and mobile health (mHealth) applications, monitoring performance should be achieved with a low cost to gather different sensory features, as maintaining sensors incur monetary, computation, and energy cost. We propose an adaptive feature selection method that dynamically selects features for prediction at any given time point. We formulate this problem as an $\ell_0$ minimization problem across time, and cast the combinatorial optimization problem into a stochastic optimization formulation. We then utilize a differentiable relaxation to make the problem amenable to gradient-based optimization. Our evaluations on four activity recognition datasets show that our method achieves a favorable trade-off between performance and the number of features used. Moreover, the dynamically selected features of our approach are shown to be interpretable and associated with the actual activity types.

Under review as a conference paper at ICLR 2021
DYNAMIC FEATURE SELECTION FOR EFFICIENT AND INTERPRETABLE HUMAN ACTIVITY RECOGNITION
Anonymous authors Paper under double-blind review
ABSTRACT
In many machine learning tasks, input features with varying degrees of predic- tive capability are usually acquired some cost. For example, in human activity recognition (HAR) and mobile health (mHealth) applications, monitoring perfor- mance should be achieved with a low cost to gather different sensory features, as maintaining sensors incur monetary, computation, and energy cost. We propose an adaptive feature selection method that dynamically selects features for prediction at any given time point. We formulate this problem as lo minimization problem across time, and cast the combinatorial optimization problem into a stochastic optimization formulation. We then utilize differentiable relaxation to make the problem amenable to gradient- based optimization. Our evaluations on four activity recognition datasets show that our method achieves favorable trade-off between performance and the number of features used. Moreover, the dynamically selected features of our approach are shown be interpretable and associated with the actual activity types.
INTRODUCTION
Acquiring predictive features is critical for building trustworthy machine learning systems, but this often comes daunting cost. Such cost can be in the form of energy needed maintain an ambient sensor (Ardywibowo et al. 2019 Yang et al.) (2020). time needed to complete an experiment (Kiefer 1959), or manpower required to monitor hospital patient (Pierskalla & Brailer, 1994). Therefore, it becomes important not only to maintain good performance in the specified task, but also a low cost to gather these features.
Indeed, existing Human Activity Recognition (HAR) methods typically use fixed set of sensors. potentially collecting redundant features to discriminate contexts (Shen Varshney [013]] Aziz et al. 2016 Brtugrul & Kaya 2017 Cheng et al. 2018). Classic feature selection methods such as the LASSO and its variants can address the performance-cost trade-off by optimizing an objective penalized by a term that helps promote feature sparsity (Tibshirani] 1996; Friedman et al. 2010 2008; Zou & Hastie 2005). Such feature selection formulations are often static, that is, a fixed set of features selected priori. However, different features may offer different predictive power under different contexts. For example, a health worker may not need to monitor a recovering patient patient with the declining condition; an experiment performed twice may be redundant; or a smartphone sensor may be predictive when the user is walking but not when the user car. By adaptively selecting which sensor(s) observe any given time point, one can further reduce the inherent cost for prediction and achieve better trade-off between cost and prediction accuracy.
as frequently compared
In addition to cost-efficiency, an adaptive feature selection formulation can also lead to more in- terpretable and trustworthy predictions. Specifically, the predictions made by the model are only based on the selected features, providing a clear relationship between input features and model predictions. Existing efforts on interpreting models are usually based on some post-analyses of the predictions, including the approaches in (1) visualizing higher level representations or reconstructions of inputs based on them (Li et al. 2016 Mahendran & Vedaldi 2015). (2) evaluating the sensitivity of predictions local perturbations of inputs or the input gradients (Selvaraju et al. 2017 Ribeiro jet al., (2066), and (3) extracting parts of inputs as justifications for predictions (Lei et al, 2016). Another related but orthogonal direction is model compression of training sparse neural networks

---

Under review as a conference paper at ICLR 2021
with the goal of memory and computational efficiency (Louizos et al. 2017; Tartaglione et al. 2018 Han et al. 2015). All these works require collecting all features first and provide post-hoc feature relevance justifications or network pruning.
Recent efforts on dynamic feature selection adaptively assign features based on immediate statis-
ties (Gordon al., 2012 Bloom et al., 2013; Ardywibowo al. 2019 Zappi et al.) 2008). ignoring
the information a feature may have on future predictions. Others treat feature selection as a Markoy
Decision Process (MDP) and use Reinforcement Learning (RL) to solve it (He & Eisner 2012
Karayev et al. 2013 Kolamunna et al. 2016 Spaan & Lima 2009; Satsangi et al.) 2013 Yang
ct 2020)) However, solving the RL objective not straightiorward. Besides being sensitive to hyperparameter settings in general, approximations such as state space discretization and greedy approximations of the combinatorial objective were used to make the RI. problem tractable.
To this end, we propose a dynamic feature selection method that can be easily integrated into existing deep architectures and trained from end to end, enabling task-driven dynamic feature selection. To achieve this, we define feature selection module that dynamically selects which features to use at any given time point. We then formulate a sequential combinatorial optimization that minimizes the trade-off between the learning task performance and the number of features selected at each time point. To make this problem tractable, we cast this combinatorial optimization problem into a stochastic optimization formulation. We then adopt differentiable relaxation of the discrete feature selection variables to make it amenable to stochastic gradient descent based optimization. It therefore can be plugged-in and jointly optimized with state-of-the-art neural networks, achieving task-driven feature selection over time. To show our method's ability to adaptively select features while maintaining good performance, we evaluate it on four time-series activity recognition datasets: the UCI Human Activity Recognition (HAR) dataset (Anguita et al.) 2013), the OPPORTUNITY dataset (Roggen al., (2100), the ExtraSensory dataset (Vaizman et al. 2017). as well as the NTU-RGB-J dataset (Shahroudy et al. 2016
Several ablation studies and comparisons with other dynamic and static feature selection methods demonstrate the efficacy of our proposed method. Specifically, our dynamic feature selection is able to use as low as 0.28% of the sensor features while still maintaining good human activity monitoring accuracy. Moreover, our dynamically selected features are shown to be interpretable with direct correspondence with different contexts and activity types.
METHODOLOGY
2.1 THE NORM MINIMIZATION PROBLEM
Many regularization methods have been developed to solve simultaneous feature selection and model parameter estimation (Tibshirani 1996, Zou & Hastie 20055] Tibshirani] 1997 Sun et al., 2014] Simon et al., 201T). The ideal penalty for the purpose of feature selection is the no-nrrm of the model coefficients for all predictors. This norm is equivalent to the number of nonzero terms in all the model coefficients. Given a dataset containing N independent and identically distributed (iid) input-output pairs {(z1, y1)...., (ø x. DN)} with each æ; containing P features, hypothesis class of predictor functions (): and a loss function L(ū, y) between prediction ý and true output y. the fo-norm regularized optimization problem can be written as follows:
Σ ()(a:))).))) + x||01lo.
min
N
where |101|o = Ef I(0 + 0) penalizes the number of nonzero model coefficients.
In the models that linearly transform the input features o., penalizing the weights relating to each feature in x; enables sparse feature subset selection. However, such a selection is static, as it does not adaptively select features that are appropriate for given context. Moreover, the optimization above is computationally prohibitive as it involves combinatorial optimization to select the subset of nonzero model coefficients corresponding the input features.
In the following, we formulate our adaptive dynamic feature selection problem when learning with multivariate time series. Coupled with training recurrent neural networks, this adaptive feature
2

---

Under review as a conference paper at ICLR 2021
selection problem is transformed into sequential context-dependent feature subset selection problem, to which we devise a stochastic relaxation to make the problem tractable.
2.2 DYNAMIC FEATURE SELECTION VIA SEQUENTIAL CONTEXT-DEPENIEE FEATURE SUBSET SELECTION
Instead of finding a subset of nonzero model coefficients, an equivalent formulation can be derived by directly selecting the feature subset. Without loss of generality, let be binary vector that indicates whether each feature is selected or not. Then, the original fo-norm optimization formulation can be equivalently written follows:
N Σ N
(E
c( O)) + A||=||o-
min 0,2
Compared to the original problem, the penalty on the number of selected features is through the fn-norm of z. This formulation is more flexible, as 2 can be made dependent on corresponding input features, output labels, any contextual information, allowing selection problem when learning with multivariate time series data. Specifically, let the input-output pairs (ar;; yi) be pair of time series data of length T, At each time I, our model predicts the output y, as well as the next feature set to select z,. This optimization problem can be formulated as:
to formulate
dynamic feature
N T:
T:
min 0,2
XE
0--10 o 2000-111 02 :), 21)
A
|2
(3)
i=1
Here, we are tasked to find a set of parameters and feature sets 2t for each sample at each time point to optimize the trade-off between model performance and the number of selected features. The model then uses the parameters and the previously observed features X - 22:tt-1 2; 2 to infer the next output y However, the above formulation remains intractable, as it involves combinatorial optimization select the feature subsets at each time point, in addition the joint optimization of the model parameters and variable selection. Naively, one may also need to solve a separate optimization problem find z for each time point during the run time, In the following section, we derive relaxation based on stochastic optimization parameterizing 2 s to make the above problem tractable.
2.3 RELAXATION THROUGH STOCHASTIC OPTIMIZATION
Instead of finding the exact feature subsets indexed by 2 that achieve the optimal regularized objective, one can treat these 2 s as binary random variables and seek to optimize the distribution π (2)Ø)) that generates these random variables. For the ease of exposition, we first focus on the relaxation of the non-adaptive formulation in ① as follows:
min 0.6
EEDD))> Ex~
[(f(x; z: 0),y,) + Al|=
Note that the solution to this problem is equivalent to the original one, as the original combinatorial problem be recovered by setting (=]())) (e) Bernoulli distribution parameterized by Φ and restricting Φ E {0, 13. Using this relaxation, the regularization term can now be evaluated analytically:
P
E E
||=110|
= (em(())
|1=1lo
Σ (=|0))
Σ 3
J=1
On the other hand, the outer expectation in (4) can be approximated using minibatches. Relaxation of binary random variables has been adopted in Louizos al.] (2017) for network architecture sparsification, and in Yamada et al. (2019): : Balin et (2019) for static feature selection. Here, we extend the above relaxation for time series data, where unlike previous works, the binary random variables are parameterized locally and are context-dependent and features are selected adaptively across time. We first note that our adaptive feature selection formulation in (3) allows each time point to have its own feature selection distribution T/ (21Ø)) A x22|X;-1 Φ) conditioned on previously selected observed features x as defined above. Let ( (==(b) be the set of (2 () for all t E {I...., T.J. The stochastic relaxation of the adaptive feature selection formulation can be written as follows:
min
E ((DDDDD E (x
Σ
Σ
π (z(0)}
(6)
0.0
3

---

Under review as a conference paper at ICLR 2021
2.4 MODEL PARAMETERIZATION AND DIFFERENTIABLE RELAXATION
The difficulty in solving the above problem using gradient descent is that the discrete random variables z/'s not directly amenable stochastic reparameterization techniques. An effective and simple to implement formulation that we adopt is the Gumbel-Softmax reparameterization (Jang et al. 2016 Maddison al., 2016), which relaxes a discrete valued random variable parameterized by @ to a continuous random variable ž. Firstly, we can parameterize т T( 2 Φ) using a vector-valued function o ( x) of the previous observations X with Φ now being the parameters of o (-).
The distribution can now be rewritten π (= Φ) Bern(ar Φ)). With this, the discrete valued random variables 2f can be relaxed into continuous random variables E as follows:
1 + exp (- .)) + L)/T)
Here, L = log u -itt-looe log(1 - u) is logistic distribution, where ~ Uni 1). and T is a temperature parameter. For low values of r. 2 approaches sample of a binary random variable, recovering the original discrete problem, while for high values, E will equal J. With this, we are able to compute gradient estimates of 2 and approximate the gradient of z Vo. v Ve. This enables to backpropagate through the discrete random variables and train the selection parameters along with the model parameters jointly using stochas- tic gradient descent. Meanwhile, at test time, we sample binary random variables from the learned probabilities.
2.5 MODEL SPECIFICATION
To complete our formulation, we specify the model architecture that we use. We have imple- mented our adaptive dynamic feature selection with Gated Recurrent Unit (GRU) (Cho et al.) 2014a), type of Recurrent Neural Network (RNN) Graves et al. 2013), as shown in Fig-
The proposed Gated Recurrent Unit (GRU) based architecture for our adaptive mon- itoring model specification. Here, features any given time are selected based on the previous ob- servations summarized by ht-1-
Figure I:
ure Here, we have the previous observations X; being summarized by the hidden state h!- For adaptive feature selection, the selection distribution is made dependent on h using sigmoid of its linear transformation by weight matrix W as follows: axX Φ) = SIGMOID(Wh!
such that Φ x= [W]. We note that such module can be easily integrated into many existing deep architectures and trained from end to end, allowing for task-driven feature selection. For example, the module can be applied to Recurrent Convolutional Neural Networks (RCNN) (Liang & Hu 2015) to selec- tively determine which convolutional patches/channels to use, to general feedforward networks to selectively deactivate certain neurons/channels to reduce computation. We have demonstrated this ability by applying it to an Independent RNN (Li et al., 2018) benchmarked on the NTU-RGB-D
dataset (Shahroudy et al.) 2016), as detailed in Appendix A.4
With the model specified, our method can be applied to existing human activity recognition datasets. Specifically, we are now able to train a prediction model and dynamic feature selection policy offline, and test withheld testing set. The application of our model online learning is subject to future work.
3 RELATED WORK
Existing HAR systems typically use fixed set of sensors, potentially collecting redundant features for casily discriminated contexts. Methods that attempt to find a fixed static feature set often rank feature sets using metrics such as Information Gain (Shen Varshney 2013), or relevancy ranking through a filtering strategy (Aziz et al. 2016 Ertugrul & Kaya 2017 Cheng al. 2018). However, static feature selection can potentially result in collecting redundant information for highly discriminable contexts.
Work on dynamic feature selection can be divided into Reinforcement Learning (RL) based and non- RI. approaches. Non-RI. based approaches vary from assigning certain features to certain activities
4

---

Under review as a conference paper at ICLR 2021
(Gordon et al. 2012), pre-defining feature subsets for prediction (Bloom et al. 2013 Strubell et al. 2015), optimizing the trade-off between prediction entropy and the number of selected features (Ardywibowo et al. 2019), to building metaclassifier for sensor selection (Zappi et al. 2008). These methods all use immediate rewards to perform feature selection. For predicting long activity sequences, this potentially ignores the information that feature may have on future predictions, or conversely, overestimate the importance of a feature given previous observations.
Among the RL based approaches, some methods attempt build MDP decide which feature to select next or whether to stop acquiring features and make a prediction (He & Eisner] 2012 Karayey al. 2013; Kolamunna al. 2016). These methods condition the choice of one feature on the observation generated by another one, instead of choosing between all sensors simultancously, SSaan Lima (2009) and [Satsangi et al. (2015) formulated Partially Observable MDP (POMDP) using discretization of the continuous state to model the policy. Yang et al., (2020) formulate an RL objective by penalizing the prediction performance by the number sensors used. Although using a desirable objective, the method employs a greedy maximization process to approximately solve the combinatorial optimization. Moreover, they do not integrate easily with existing deep architectures.
Attention is another method worth noting, as it is able to select the most relevant segments of sequence for the current prediction /Vaswani et al. 2017). Attention modules have been recently used for activity recognition (Ma et al., (2019). However, like most attention methods, it requires all of the features to be observed before deciding which features are the most important for prediction. Moreover, the number of instances attended is not penalized. Finally, soft attention methods typically weight the inputs, instead of selecting the feature subset. Indeed, our experiments on naively applying attention for dynamic feature selection show that always selects 100% of the features at all times.
Sparse regularization has previously been formulated for deep models, e.g., Liu al. (2015); Louizos it al. (2017); Frankle & Carbin (2018), but their focus has primarily been in statically compressing model sizes reducing overhtting, instead of dynamically selecting features for prediction. In particular, & regularization is a common method to promote feature sparsity (Tibshirani 1996.
Friedman et al. 2010 2008 Zou& & Hastie 2005).
Selection or skipping along the temporal direction to decide when to memorize vs update model state has been considered in Hu et al. (2019); Campos et al. (2018); Neil et al. (016)). These works either are not context dependent or do not consider energy efficiency ineterpretability. Additionally, skipping time steps may not be suitable for continuous monitoring tasks including HAR, where are tasked to give a prediction at every time step. Nevertheless, our dynamic/adaptive feature selection orthogonal to temporal selection/skipping and we leave exploring the potential integration of these two directions our future research.
Finally, there have been many formulations that propose to solve the issue of backpropagation through discrete random variables (Jang et al.) 2016;; Maddison al. 2016 Tucker et al. [017 Grathwohl et al. 2017; Yin & Zhou, 2018). REBAR (Tucker et al., 2017) and RELAX (Grathwohl et al.) (017) employ REINFORCE and introduce relaxation-based baselines reduce sample variance of the estimator. However, these baseline functions increase the computation and cause potential conflict between minimizing the sample variance of the gradient estimate and maximizing the expectation objective. CeimeentRRIIFFRRCEE
is a self-control gradient estimator that does not need additional baselines (Yin& & Zhou| 2018). It provides unbiased gradient estimates that exhibit low variance, but its direct application to autoregressive or sequential setups is not addressed by Yin & Zhou (2018) and leads to approximate gradients. Moreover, an exact sequential formulation will require prohibitive computation, squared in sequence length forward passes.
4 EXPERIMENTS
Benchmark Datasets and Performance Evaluation We evaluate our model on four different datasets: the UCI Human Activity Recognition (HAR) using Smartphones Dataset (Anguita et al. 2013), the OPPORTUNITY Dataset (Roggen et al.) 2010), the ExtraSensory dataset (Vaizman et al. 2017), and the NTU-RGB-D dataset (Shahroudy et al. 2016). Although there are many other human activity recognition benchmark datasets (Chen et al., 2020). we choose the above datasets to better convey our message of achieving feature usage efficiency and interpretability using our adaptive
5

---

Under review as a conference paper at ICLR 2021
feature selection framework with the following reasons. First, the UCI HAR dataset is clean dataset with no missing values, allowing us to benchmark different methods without any discrepancies in data preprocessing confounding our evaluations. Second, the OPPORTUNITY dataset contains activity labels that correspond to specific sensors. An optimal adaptive feature selector should primarily choose these sensors under specific contexts with clear physical meaning. Finally, the ExtraSensory dataset studies a multilabel classification problem, where two or more labels can be active at any given time, while the NTU-RGB-D dataset is complicated activity recognition dataset with over 60 classes of activities using data from 25 skeleton joints. These datasets allow us to benchmark model performance in complex setting. For all datasets, we randomly split data both chronologically and by different subjects. More details for each dataset and its corresponding experiment setup is provided under its own subheading in the following and also in Appendix [ Due to the page limit, our implementation details and results on the NTU-RGB-D dataset are available in Appendix A and B
We investigate several aspects of our model performance on these benchmarks. To show the effect in prediction accuracy when our selection module is considered, we compare its performance to standard GRU network (Cho et al., 2014b). To show the effect of considering dynamic feature selection, we compare a nonadaptive to formulation that statically selects features by solving (4) (Louizos al. 2017). The performance of our Eo regularized formulation is also benchmarked with an & regularized formulation. To benchmark the performance of our differentiable relaxation- based optimization strategy, we implement the Straight-Through estimator (Hinton et al., 2012) and Segneen-RRINFORCEMMMEE
(ARM) gradient estimates (Yin & Zhou, (018)) as alternative methods to optimize our formulation. As stated in the previous section, the fully sequential application of ARM was not addressed in the original paper, and will be prohibitively expensive to compute exactly. Hence, we combine ARM and Straight-Thron approach to optimize our formulation. More specifically, we calculate the gradients with respect to the Bernoulli variables with ARM, and use the ST estimator to backpropagate the gradients through the Bemoulli variables to previous layers' parameters. We also have tested different values for the temperature hyperparameter in Appendix D] where we observe that the settings with the temperature parameters below 1 generally yield the best results with noticeable performance difference.
(ST) estimator (Hinton et al., (012) as another
To further show the importance of considering the sparse regularized formulation, we compare with an attention-based feature selection, selecting features based the largest attention weights. Because attention yields feature attention weights instead of feature subsets, we select features by using a hard threshold of the attention weights and scaling the selected features by 1 for different values of o. Indeed, without this modification, we observe that an attention- based feature selection would select 100% of the features at all times.
Finally, we have attempted to implement the dynamic feature selection method by Yang et al. (020) as a distinctly different benchmark. However, without any implementation details provided by the authors, we were not able to reproduce their results.
UCI HAR Dataset We first test our proposed method on performing simultaneous prediction and adaptive feature selection on the UCI HAR dataset (Anguita et al., 2013). This dataset consists of 561 smartphone sensor measurements including various gyroscope and accelerometer readings, with the task of inferring the activity that the user performs at any given time. There are six possible activities
100
Walding
Klp
60
Adapthe
Monadapticc
Laytrs
10
20
30
40
(a)
(b)
(c)
Figure 2: UCI HAR Dataset results: (a) Prediction and features selected of the proposed model 1. (b) Feature selection vs. accuracy trade-off curve comparison. (c) Heatmap of sensor feature activations under each activity of the UCI HAR dataset. Only active features are shown out of the 561 features in total.

---

Under review as a conference paper at ICLR 2021
Table 1: Comparison of various optimization techniques for our model on the UCI HAR dataset. Accuracies and average number of features selected are in (%)
Method
Accuracy
Feat. Selected
Gumbel-Softmax ==1
97.18
0.28
ARM À = 1 Yin & Zhou, 2018
95.73
11.67
ST-ARM À [ & Zhou 2018 Hinton et al. 2012
92.79
1.92
Straight Through A
Himfon et al. 2012)
89.38
0.31
L1 Regularization À =
90.43
19.48
100
10
30
50
Featurn Usedd (%)
(b)
(c)
Figure 3: OPPORTUNITY Dataset results: (a) Prediction and features selected of the proposed model
À A= 1. (b) Prediction and features selected of the proposed model on set of activity transitions. (c)
Feature selection vs. Error trade-off curve comparison.
that a subject can perform: walking, walking upstairs, walking downstairs, sitting, standing, and laying.
We first compare various optimization methods, using stochastic gradients by differential relaxation using Gumbel-Softmal reparametrization, ARM, ST-ARM, Throghhh gradients, and an & regularized formulation solve adaptive feature selection. The results are provided in Table [ As shown, Gumbel-SSfftax achieves the best prediction accuracy with the least number of features. Utilizing either the Straight Through estimator, ARM, or ST-ARM for gradient estimation cannot provide better balance between accuracy and efficiency compared with the Gumbel-Softmax relaxation-based optimization. Indeed, the performance the ST estimator is expected, as there is a mismatch between the forward propagated activations and the backward propagated gradients in the estimator. Meanwhile, we attribute the lower performance of the ARM and ST-ARM optimizer to its use in sequential fashion, which was not originally considered. The lower performance of the & regularized formulation is expected, as & regularization is an approximation to the problem of selecting the optimal feature subset. In the following experiments, we have similar trends and only report the results from the Gumbel-Softmax based optimization.
As shown, our adaptive feature selection model is able to achieve competitive accuracy using only 0.28% of the features, or on average about 1.57 sensors at any given time. We also observe that both the attention and our adaptive formulation is able to improve upon the accuracy of the standard GRU, suggesting that feature selection can also regularize the model to improve accuracy. Although the attention-base model yields the best accuracy, this comes at cost of utilizing around 50% of the features at any given time. We also have checked the average accuracy of our model on a time-aligned testing set to show that our model is stable for long-term predictions in Appendix B
Benchmarking results of different models are given in Table 2
We study the effect of the regularization weight À by varying it from À E {1, 0.1, 0.01, 0.005, 0.001 We compare this with the attention model by varying the threshold a used to select features from a E {0.5, 0.9, 0.95, 0.99, 0.995. 0.999), as well as the nonadaptive model by varying its À from X E {1000, 100, 0.01, 0.005, 0.001 A trade-off curve between the number of selected features and the performance for the three models can be seen in Figure 2(b) As shown in the figure, the accuracy of the attention model suffers increasingly with smaller feature subsets, not a formulation specifically tailored to find sparse solutions. On the other hand, the accuracy of our adaptive formulation is unaffected by the number of features, suggesting that selecting around 0.3% of the features on average may be optimal for the given problem. It further confirms that our adaptive formulation selects the most informative features given the context. The performance of
attention is
Table 2:
Comparison of various models for adaptive monitoring three activity recognition datasets.
Accuracyy metrics and average number of features selected are all in (%).
UCT
OPPORTUNITY
ExtraSensory F1
Method
Accuracy
Features
Accuracy
Features
Accuracy
Features
Adaptive (Ours) Attention a = 0.5
97.18
0.28
84.26
15.88
91.14
55.06
11.25
98.38
49.94
83.42
54.20
90.37
53.29
54.73
Nonudaptive À 1 Louizos et al 2017
95.49
14.35
81.63
49,57
91.13
53.18
42.32
selection (GRU) Cho ct al. 20146
96.67
100
84.16
100
91.14
53.53
100
7

---

Under review as a conference paper at ICLR 2021
the nonadaptive model is consistent for feature subsets of size 10% or greater. However, it suffers drop in accuracy for extremely small feature subsets. This shows that for static selection, selecting a feature set that is too large would result in collecting many redundant features for certain contexts, while selecting a feature set that is too small would be insufficient for maintaining accuracy.
An example of dynamically selected features be seen in Figure 2(a) We plot the prediction of our model compared to the true label and illustrate the features that are used for prediction. We also plot heatmap for the features selected under each activity in Figure 2 Although these features alone may not be exclusively attributed as the only features necessary for prediction under specific activities, such visualization useful retrospectively observe the features selected by our model at each time- time-point. Note that mainly 5 out of the 561 features are used for prediction at any given time. Observing the selected features, we see that for the static activities such as sitting, standing, and laying, only sensor feature 52 and 63, features relating to the gravity accelerometer, are necessary for prediction. On the other hand, the active states such as walking, walking up, and walking down requires sensor features: sensor 65, 508, and 556, which are related to both the gravity accelerometer and the body accelerometer. This is intuitively appealing as, under the static contexts, the body accelerometer measurements would be relatively constant, and unnecessary for prediction. On the other hand, for the active contexts, the body accelerometer measurements are necessary reason about how the subject moving and accurately discriminate between the different active states. Meanwhile, we found that measurements relating to the gyroscope were unnecessary for prediction.
UCI OPPORTUNITY Dataset We further test proposed method on the UCI OPPORTUNITY Dataset (Roggen et al., 2010). This dataset consists multiple different label types for human activity, ranging from locomotion, hand gestures, to object interactions. The dataset consists of 242 measurements from accelerometers and Inertial Measurement Units (IMUs) attached to the user, as well as accelerometers attached to different objects with which the can interact.
We use the mid-level gesture activities as the target for our models to predict, which contain gestures related specific objects, such as opening door and drinking from a cup. comparison of the accuracy and the percentage of selected features by different models is given in Table 2 while example predictions and trade-off curve are constructed and shown in Figures 3(a) 3(b] and [(c) with similar trend as the results on the UCI HAR dataset. Notably, the trade-off for the nonadaptive models remains constant for E (0.0001, 0.001,... ...,. 1), with a sharp decrease accuracy for A 2 10.
heatmap for the selected features under each activity shown in Figure 4 Here, the active sensor features across all activities are features 40 and 42, readings of the IMU attached to the subject's s back, feature 82, readings from the IMU attached to the left upper arm (LUA), and features 230 and 239, location tags that estimate the subject's position. We posit that these general sensor features selected to track the subject's 's overall position and movements, as they are also predominantly selected in cases with no labels. Mcanwhile, sensors 5, 6, and 16, readings from the accelerometer attached to the hip, LUA, and back, are specific to activities involving opening/closing doors or drawers.
Interestingly, sensors attached specific objects, such accelerometers doors and cups, unnecessary for prediction. We attribute this to the severe amount of missing values of these sensors. Indeed, the sensors that have the least amount of missing values are the body sensors and the localization tags. We hypothesize that the model prefers these sensors for their consistent discriminative power on multiple activity types compared to the object specific sensors. In addition to these object specific sensors, 5 IMUs, accelerometers, and 2 localization tags can be completely turned off without significantly affecting prediction performance on this task.
open 20or
Dise Door
Open Frege
20 Closs Bheher
0.
Clos Brawer
Close 2rawer
Dipeer
Clas "aolo
Toggle Swithh
2 A Figure 4: Heatmap of sensor feature activations under each activity of the OPPORTUNITY dataset. *Only active features shown out of the 242 features in total.
Number
8

---

Under review as a conference paper at ICLR 2021
vakes
Features (S)
(a)
(b)
Figure 5: ExtraSensory Dataset results: (a) Prediction and features selected of the proposed model. (b) Feature selection vs. Error trade-off curve comparison.
ExtraSensory Dataset We further test our proposed method on the ExtraSensory Dataset (Vaizman al., 2017). This is multilabel classification dataset, where two or more labels can be active at any given time. It consists of 51 different context labels, and 225 sensor features. We frame the problem as multilabel binary classification problem, where have binary output for each label indicating whether it is active. A comparison of the accuracy and selected features by different models tested can seen in Table Our method is again competitive with the standard GRU model using less than 12% of all the features.
trade-off curve is shown in Figure 5(b) where we see a similar trend for both adaptive and attention models. However were unable to obtain a feature selection percentage lower than 25% for the nonadaptive model even with as large as 104. We believe that this because at least 25% of statically selected features catastrophically, similar to the OPPORTUNITY dataset results. A heatmap and detailed discussion of the features that our model dynamically selected can be found in Appendix C
needed; otherwise the nonadaptive model will degrade in performance
The results on these three datasets along with the results on the NTU-RGB-D dataset in Appendix B indicate that adaptive monitoring framework provides the best trade-off between feature efficiency and accuracy, while the features that it dynamically selects are also interpretable and associated with the actual activity types.
5 CONCLUSIONS
We propose novel method for performing adaptive feature selection by sequential context-depender feature subset selection, which cast into a stochastic optimization formulation by modifying the to regularized minimization formulation. To make this problem tractable, we perform stochastic relaxation along with a differentiable reparamaterization, making the optimization amenable to gradient-based optimization with auto-differentiation. We apply this method to human activity recognition by implementing our method to Recurrent Neural Network-based architectures. We benchmark our model on four different activity recognition datasets and have compared with various adaptive and static feature selection benchmarks. Our results show that our model maintains a desirable prediction performance using a fraction of the sensors or features. The features that our model selected were shown to be interpretable and associated with the activity types.
REFERENCES
Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, and Jorge Luis Reyes-Ortiz A public domain dataset for human activity recognition using smartphones. 21th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN), pp. 437-442 CIACO, 2013.
Randy Ardywibowo, Guang Zhao, Zhangyang Wang, Bobak Mortazavi, Shuai Huang, and Xiaoning Qian. Adaptive activity monitoring with uncertainty quantification in switching gaussian process models. arXiv preprint 1901v:1901.2444 2019.
Omar Aziz, Stephen Robinovitch, and Edward J Park. Identifying the number and location of body worn sensors accurately classify walking, transferring and sedentary activities. In 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 5003-5006. IEEE, 2016.

---

Under review as a conference paper at ICLR 2021
Muhammed Fatih Balm, Abubakar Abid, and James Zou. Concrete autoencoders: Differentiable feature selection and reconstruction. In International Conference on Machine Learning, pP- 444-453, 2019.
Victoria Bloom, Vasileios Argyriou, and Dimitrios Makris. Dynamic feature selection for online action recognition. In International Workshop on Human Behavior Understanding. 2013.
Víctor Campos, Brendan Jou, Xavier Giró-i Nieto, Jordi Torres, and Shih-Fu Chang. Skip RNN: Learning to skip state updates recurrent neural networks. In International Conference on Learning Representations, 2018.
Kaixuan Chen, Dalin Zhang, Lina Yao, Bin Guo, Zhiwen Yu, and Yunhao Liu. Deep learning for sensor-based human activity recognition: Overview, challenges and opportunities. arXiv preprint arXiv:2001.07416 2020.
Weihao Cheng, Sarah Erfani, Rui Zhang, and Ramamohanarao Kotagiri. Learning datum-wiss sampling frequency for energy-efficient human activity recognition. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Kyunghyun Cho, Bart Van Merriënhoer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arX9:: /259,, 2014a.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre. Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1078, 2014b.
Öner Faruk Ertugrul and Yilmaz Kaya. Determining the optimal number body-worn sensors for human activity recognition. Soft Computing, 21(17):5053-506 2017.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprini 1803.0363. 2018.
Jerome Friedman, Trevor Hastie, and Robert Tibshirani. Sparse inverse covariance estimation with the graphical lasso. Biostatistics, 9(3):432 441, 2008.
Jerome Friedman, Trevor Hastic, and Robert Tibshirani. A note on the group lasso and a sparse group 2010.
lasso. arXiv preprint
Dawud Gordon, Jurgen Czemy, Takashi Miyaki, and Michael Beigl. Energy-efficient activity recognition using prediction. In International Symposium on Wearable Computers. IEEE, 2012.
Will Grathwohl, Dami Choi, Yuhuai Wu, Geoffrey Roeder, and David Duvenaud. Backpropagation through the void: Optimizing control variates for black-box gradient estimation. arXiv preprint trXiv:1711.00123 2017.
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing, pp. 6645-6649. IEBE, 2013.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 1, NIPS' 15, pp. 1135-1143, 2015.
He He and Jason Eisner. Cost-sensitive dynamic feature selection. In ICML Inferning Workshop. 2012.
Geoffrey Hinton, Nitsh Srivastava, and Kevin Swersky. Neural networks for machine learning. Coursera, video lectures, 264:1, 2012.
Hao Hu, Liqiang Wang, and Guo-Jun Qi. Learning to adaptively scale recurrent neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3822-3829, 2019.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv: 16/ 1.01 144, 2016.
10

---

Under review as a conference paper at ICLR 2021
Sergey Karayev, Mario Fritz, and Trevor Darrell. Dynamic feature selection for classification budget. In International Conference on Machine Learning (ICML): Workshop on Prediction with Sequential Models, 2013.
Jack Kiefer. Optimum experimental designs. Journal of the Royal Statistical Society: Series (Methodological), 21(2):272-304 1959.
Harini Kolamunna, Yining Hu, Diego Perino, Kanchana Thilakarathna, Dwight Makaroff, Xinlong Guan, and Aruna Seneviratne. Afv: Enabling application function virtualization and scheduling in wearable networks. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 981-991, 2016.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. Rationalizing neural predictions. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, pp. 107-117, Austin, Texas, November 2016. Association for Computational Linguistics.
Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. Visualizing and understanding neural models in NLP. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 681-691, San Diego, California, June 2016. Association for Computational Linguistics.
Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao. Independently recurrent neural network (indrnn): Building a longer and deeper rnn. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5457-5466, 2018.
Ming Liang and Xiaolin Hu. Recurrent convolutional neural network for object recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3367-3375, 2015.
Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall Tappen, and Marianna Pensky. Sparse convolu- tional neural networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 806-814, 2015.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1_0 regularization. arXiv preprint 1rXiv:1712.01312 2017.
Haojie Ma, Wenzhong Li, Xiao Zhang. Songcheng Gao, and Sanglu Lu. Attnsense: multi-level attention mechanism for multimodal human activity recognition. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp. 3109-3115, AAAI Press, 2019.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv: 1.000001, 2016.
A. Mahendran and A. Vedaldi. Understanding deep image representations by inverting them. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Daniel Neil, Michael Pfeiffer, and Shih-Chii Liu. Phased LSTM: Accelerating recurrent network training for long or event-basee sequences. Advances in neural information processing systems, 29:3882-388 38900 2016.
William Pierskalla and David J Brailer. Applications operations research in health care delivery. Handbooks in operations research and management science, 6:469-505 1994.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why should trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, pp. 1144,114,, New York, NY, USA, 2016. Association for Computing Machinery.
Daniel Roggen, Alberto Calatroni, Mirco Rossi, Thomas Holleczek, Kilian Förster, Gerhard Tröster, Paul Lukowicz, David Bannach, Gerald Pirkl, Alois Ferscha, et al. Collecting complex activity datasets in highly rich networked sensor environments. In 2010 Seventh international conference networked sensing systems (INSS), pp. 233-240. IEEF, 2010.

---

Under review as a conference paper at ICLR 2021
Yash Satsangi, Shimon Whiteson, and Frans A Oliehoek. Exploiting submodular value functions for faster dynamic sensor selection. In AAAI Conference on Artificial Intelligence, 2015.
R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-Cam: Visual explanations from deep networks via gradient-basee based localization. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618-626, 2017.
Amir Shahroudy, Jun Liu, MaanTssog.aa Ng. and Gang Wang. Ntu rgb+d: A large scale dataset for 3d human activity analysis. In IEEE Conference on Computer Vision and Pattern Recognition, June 2016.
Xiaojing Shen and Pramod K Varshney. Sensor selection based on generalized information gain for target tracking in large sensor networks. IEEE Transactions on Signal Processing. 62(2):363-375 2013.
Noah Simon, Jerome Friedman, Trevor Hastie, and Rob Tibshirami. Regularization paths for cox's proportional hazards model via coordinate descent. Journal of statistical software, 2011.
Matthijs TJ Spaan and Pedro U Lima. A decision-theoretic approach to dynamic sensor selection in camera networks. In Nineteenth International Conference on Automated Planning and Scheduling. 2009.
Emma Strubell, Luke Vilnis, Kate Silverstein, and Andrew McCallum. Learning dynamic feature selection for fast sequential prediction. arXiv preprint 100xv: 2015.
Hokeun Sun, Wei Lin, Rui Feng, and Hongzhe Li. Network-regullrizeed high-dimensional cox regression for analysis of genomic data. Statistica Sinica, 24(3):1433, 2014.
Enzo Tartaglione, Skjalg Lepsøy, Attilio Fiandrotti, and Gianluca Francini. Learning sparse neural networks via sensitivity- regularization. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 3882 3882-3892, 2018
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series (Methodological), 58(1):267-288, 1996.
Robert Tibshirani. The lasso method for variable selection in the cox model. Statistics in medicine, 16(4):385-3955 1997.
George Tucker, Andriy Mnih, Chris Maddison, John Lawson, and Jascha Sohl-Dickstein. Rebar: Low-variance, unbiased gradient estimates for discrete latent variable models. In Advances in Neural Information Processing Systems, pp. 2627-2636. 2017.
Yonatan Vaizman, Katherine Ellis, and Gert Lanckriet. Recognizing detailed human context in the wild from smartphones and smartwatches. JEEE Pervasive Computing, 16(4):62-74, 74, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-59090 6008, 2017.
Yutaro Yamada, Ofir Lindenbaum, Sahand Negahban, and Yuval Kluger. Feature selection using
stochastic gates. 2019.
Xiaodong Yang. Yiqiang Chen, Hanchao Yu. Yingwei Zhang, Wang Lu, and Ruizhe Sun. Instance-wise dynamic sensor selection for human activity recognition. YangX. 2020.
Mingzhang Yin and Mingyuan Zhou. Arm: variable models. arXiv preprint arXiv:1807.11143 2018
gradient for discrete latent
Piero Zappi, Clemens Lombriser, Thomas Stiefmeier, Elisabetta Farella, Daniel Roggen, Luca Benini, and Gerhard Tröster. Activity recognition from on-body sensors: accuracy-power trade-off by dynamic sensor selection. European Conference on Wireless Sensor Networks, pp. 17-33. Springer, 2008
Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the royal statistical society: series B (statistical methodology), 67(2):301-320 2005.
12

---

Under review as a conference paper at ICLR 2021
A IMPLEMENTATION DETAILS
Here, we provide the implementation details for the reported results in each benchmark dataset. In general, the added computation and memory incurred by our adaptive monitoring framework are insignificant, it consists of only an additional fully connected layer used to infer the next feature set. This would only add extra H x P parameters and multiply-ade operations, where H is the number of hidden neurons and P is the number of input features. This additional computational burden is insignificant compared to the memory and computational cost of the main network, which typically of order higher than P)
A.1
UCI HAR DATASET
The UCI HAR dataset consists of a training set and a testing set. To implement our adaptive feature selection and other baseline methods, we divide the training set into a separate validation set consisting 2 subjects. We preprocess the data by normalizing with the mean and standard deviation. We then divide the instances of each subject into segments of length 200.
The base model we utilize is a one-layer GRU with 2800 neurons for the hidden state. We use the cross-entropy of the predicted vs. actual labels as the performance measure. We use temperature of 0.05 for the Humbel-Softman relaxation. We optimize this with a batch size of 10 using the RMMPPro optimizer, setting the learning rate 10-1 and the smoothing constant 0.99 for 3000 epochs. We then save both the latest model and the best model validated on the validation set.
A.2 OPPORTUNITY DATASET
The OPPORTUNITY dataset consists of multiple demonstrations of different activity types. We first extract the instances into segments containing no missing labels for the mid-level gestures. Segments of length smaller than 100 are padded using the observed values at the next time-points in the instance. We then normalize the data such that its values are between -1 and 1. The authors of the dataset recommended removing some features that they believed are not useful, however we find that this does not affect performance and instead use the entire feature set. We have also experimented with interpolating the missing values but also find that does not affect performance compared to imputing the missing values with zeros. Using this, we randomly shuffle the segments and assign 80% for training, 10% for validation, and 10% for testing.
The base model we utilize is a two-layer GRU with 256 neurons for each layer's 's hidden state. The cross-entropy of the predicted vs. actual labels is adopted as the performance measure. We use temperature of 0.05 for the Gumbel-Softmax relaxation. We do not include the cross-entropy loss for the time points with missing labels. We also scale the total performance loss of the observed labels #timepoints
for each batch by #labelled timopoints optimizer, setting the learning rate to 10-4 and the smoothing constant to 0.99 for 3000 epochs. We then save both the latest model and the best model validated on the validation set.
We optimize this loss with a batch size of 100 using the RMSProp
A.3 EXTRASENSORY DATASET
The ExtraSensory dataset consists of multiple demonstrations of human behavior under different activities, where two more activity labels can be active at the same time. We first extract the instances into segments containing no missing labels for the middle level gestures. Segments of length smaller than 70 are padded using the observed values the next time-points in the instance. We then normalize the data such that its values are in between -1 and 1. We have experimented with interpolating the missing values but also find that it does not affect performance compared to imputing the missing values with zeros. Using this, randomly shuffle the segments and assign 70% for training, 10% for validation, and 20% for testing.
The base model we utilize is one-layer GRU with 2240 neurons for its hidden state. We use temperature of 0.05 for the predicted vs. actual labels the performance measure, where the model outputs a binary decision for each label, representing whether each label is active or not. We do not include the performance loss for the missing labels and scale the total performance loss of the observed labels for each batch
relaxation. We use the binary cross-entropy of the
13

---

Under review as a conference paper at ICLR 2021
by #observed lahels labelled timepoints" We optimize this scaled loss with batch size of 100 using the RMSProp optimizer, setting the learning rate to 10-4 and the smoothing constant to 0.99 for 10000 epochs. We then save both the latest model and the best model validated on the validation set.
# timepoints x #total labels
A.4 NTU-RGB-D DATASET
We first preprocess the NTU-RGB-D dataset to remove all the samples with missing skeleton data. We then segment the time-series skeleton data across subjects into 66.5% training, 3.5% validation, and 30% testing sets, The baseline model that we have implemented for the NTU-RGB-D dataset is the Independent RNN (Li et al.) 2018). This model consists of stacked RNN modules with several additional dropout, batch normalization, and fully connected layers in between. Our architecture closely follows the densely connected independent RNN of et al. (2018). To incorporate feature selection using either our adaptive formulation or an attention- based formulation, we add an additional RNN to the beginning of this model. This RNN takes as input the 25 different joint features and is tasked to select the joints to use for prediction further along the architecture pipeline. Since the joints are in the form 3D coordinates, our feature selection method is modified such that it seleets either all 3 of the X, Y, and Z coordinates of particular joint, none at all. Our architecture can be seen in Figure 6
Similar as the baseline method presented in Li al. (218)) we have trained this architecture using batch size of 128 and a sequence length of 20 using the Adam optimizer with a patience threshold of 100 iterations. We then save both the latest model and the best model validated on the validation set.
concat
concat
concat
IndRNN
IndRNN
IndRNN
Batch Norm
Batch Norm
Batch Norm
FC
IndRNN
IndRNN
IndRNN
Batch Norm
Batch Norm
Batch Norm
FC
FC
FC
ho
h1
h2
IndRNN
IndRNN
IndRNN
x
X
Figure Our modified densely connected independent RNN architecture for adaptive feature selection.
B RESULTS AND DISCUSSION OF THE NTU-RGB-D DATASET
We have tested our proposed method on the NTU-RGB-D dataset (Shahroudy et al. 2016). This dataset consists of 60 different activities performed by either single individual or two individuals. The measurements of this dataset are in the form of skeleton data consisting of 25 different 3D coordinates of the corresponding joints of the participating individuals.
We compare our method with three different baselines shown in Table B the standard independent RNN, soft attention baseline, and thresholded attention baseline. We see that our method maintains a competitive accuracy compared to the baseline using less than 50% of the features. On the other hand, because the thresholded attention formulation is not specifically optimized for feature sparsity,
14

---

Under review as a conference paper at ICLR 2021
drink
brushing teeth
brushing hair
00.8
grop
throw
sitting down
standing up
clapeino
reading
writing
wear (acket
take off jacket
0.7
take shoe
wear alase
take off g asses
take off a haticap
cheer up
hand waving sometning
reach into pocket
hopping
jump up
phene
playing th phone
typing on keyboard
pointing
taking sefnte
check time
head/bow
shake head
wipe face
0.5
put the palms together
cross hands front
sneeze
staggoring
falling
touch head
touch back
touch neck
nausea vomiting condition fan
punching other person kicking other person pat on back of other person point finger at the other person hugging other person giving something touch other person pocket
0.4
handsffaking
walking towards walking from each other
base of the spine
nedk
head
left shr ider left elbow
left wrist
right shoulder
right elbow
right wrist
right hand
hip
left knne
left anile
lert toat
right hip
right knee
right ankle
right foet
tip of the left hand
lett chumb
tip of the right highthaand hand
sp
sp
left
right
midel
Figure 7: Heatmap of sensor feature activations under each activity state of the NTU-RGB-D dataset.
we see that it performs significantly worse compared to the other methods. Mcanwhile, the soft- attention slightly improves upon the accuracy of the base architecture. However, as also indicated by our other experiments, soft-attention is not dynamic feature selection method, and tends to select 100% of the features at all times.
heatmap for the features selected under each activity shown in Figure 7 Here, we can see that there are two distinct feature sets used for two different types of interactions: single person interactions and two person interactions. Indeed, since the two person activities require sensor measurements from two individuals, the dynamic feature selection would need to prioritize different features to observe their activities as opposed to single person activities.
Table 3: Comparison of various methods for activity recognition on NTU-RGB-D dataset. "Accrracies and average number of features selected are in (%)
Method
Accuracy
Features Selected
Adaptive
80.54
49.65
Thresholded attention
40.07
52.31
Soft attention
83,28
100
No selection
83.02
100
15

---

Under review as a conference paper at ICLR 2021
C RESULTS AND DISCUSSION OF THE EXTRASENSORY DATASET
S1393
At tome
main warcplare
08
Exii
Sun fing the irteme
Walsen
Adiviyy
With
hand
th courree
dna
Orinding calcodei
-04
passerga"
Sragts
reetaumen
going dove
Lab with
party
Elrada
Phore
000
S
Figure 8: Heatmap of sensor feature activations under each activity state of the ExtraSensory dataset.
A heatmap of the features selected under each activity state can be seen in Figure 8 As shown, there are four groups of sensor features that are used across activities: the phone magnetometer (57-71), watch accelerometer magnitude (85-88), watch accelerometer direction (101-105), and location (138-147). For two particular states, 'on a bus' and 'drinking alcohol', phone accelerometer measurements (5-52) become necessary for prediction. Some states such as 'at home', 'at main workplace', and 'phone in pocket' are notably sparse in sensor feature usage. We believe that these states static, and do not require much sensor usage monitor effectively. Other sensors such as the phone gyroscope, phone state, audio measurements and properties, compass, and various low-frequency sensors are largely unnecessary for prediction in this dataset.
D EFFECTS OF THE HYPERPARAMETER ON MODEL PERFORMANCE
We observe the effects of the temperature hyperparameter in (7) model's performance. To do this, we have tested several hyperparameter values our experiment with the UCI HAR dataset. The results of tests be seen in Figure 9 In general, the settings with the temperature parameters below generally yield the best results with no noticeable performance difference. Once the temperature is set above we observe sharp increase errors. We attribute this to the mismatch between training and testing setups, where in testing, discrete binary values are sampled while training, the samples are reduced to an equal weighting between the features.
E MODEL PERFORMANCE AND STABILITY ACROSS TIME
We show the average accuracy over every 1000 seconds of running the model the testing subjects the UCI HAR dataset in Table 4 Based on the performance of the model across time, the model is shown be stable for long-term predictions. general, there is no clear temporal degradation the testing performance for this dataset. Instead, the change of prediction errors is mostly dependent on the underlying activity types.
Table 4: The average model performance across time averaged across time-aligned testing subjects.
Time
0--999
1000-1999
2000-2999
3000-3999
Error (%)
3.49
2.93
6.46
4.06
Std. Dev.
1.89
1.23
1.05
1.67
16

---

Under review as a conference paper at ICLR 2021
40
30
20
10
10
10"
10.
Temperature
Figure 9: The effect of the temperature hyperparameter the performance of the model.
F UNION OF ALL FEATURES SELECTED BY THE ADAPTIVE MODEL
Here, in addition to showing the average number of selected features, compute the percentage of all features considered by our model across the full time-length. In other words, the results presented here show the union of selected features across the time horizon. In Section 4, we chose present the average number of selected features as it directly reflects the number of required sensors for accurate HAR. Hence, clearly shows the benefits of our proposed dynamic/adaptive feature selection with respect to the power usage for sensor data collection. From Table it is clear that the percentage of all the features considered across the full time-length also significantly low for each of the three benchmark datasets, which further validates the potential of our dynamic feature selection even when additional operational cost of turning on/off sensors needs to be considered.
Table The percentage of the union of selected features across three benchmark datasets.
Dataset
(%) Union
UCI HAR
3.56
OPPORTUNITY
19.83
ExtraSensory
26.66
17

---

Under review as a conference paper at ICLR 2021
DYNAMIC FEATURE SELECTION FOR EFFICIENT AND INTERPRETABLE HUMAN ACTIVITY RECOGNITION
Anonymous authors Paper under double-blind review
ABSTRACT
In many machine learning tasks, input features with varying degrees of predie- tive capability are usually acquired at some cost. For example, in human activity recognition (HAR) and mobile health (mHealth) applications, monitoring perfor- mance should be achieved with a low cost to gather different sensory features, as maintaining sensors incur monetary. computation, and energy cost. We propose an adaptive feature selection method that dynamically selects features for prediction at any given time point. We formulate this problem as an Eo minimization problem across time, and cast the combinatorial optimization problem into a stochastic optimization formulation. We then utilize a differentiable relaxation to make the problem amenable to gradient-basee based optimization. Our evaluations on four activity recognition datasets show that our method achieves a favorable trade-off between performance and the number of features used. Moreover, the dynamically selected features of our approach are shown to be interpretable and associated with the actual activity types.
INTRODUCTION
Acquiring predictive features is critical for building trustworthy machine learning systems, but this often comes at a daunting cost. Such a cost can be in the form of energy needed maintain an ambient sensor [1] (2, time needed to complete an experiment [3], or manpower required to monitor a hospital patient 4 Therefore, it becomes important not only to maintain good performance in the specified task, but also a low cost to gather these features.
Indeed, existing Human Activity Recognition (HAR) methods typically use fixed set of sensors, potentially collecting redundant features to discriminate contexts [5
7 8) Classic feature selection methods such as the LASSO and its variants can address the performance-cost trade-off by optimizing an objective penalized by term that helps promote feature sparsity [9 T0
6
12]. Such feature selection formulations are often static, that is, a fixed set features are selected priori. However, different features may offer different predictive power under different contexts. For example, health worker may not need to monitor a recovering patient as frequently compared to a patient with the experiment performed twice may be redundant; or smartphone sensor may be predictive when the user is walking but not when the user is in car. By adaptively selecting which sensor(s) to observe at any given time point, one can further reduce the inherent cost for prediction and achieve a better trade-off between cost and prediction accuracy.
declining condition;
In addition cost-efficiency, an adaptive feature selection formulation can also lead more inter- pretable and trustworthy predictions. Specifically, the predictions made by the model are only based on the selected features, providing a clear relationship between input features and model predictions. Existing efforts on interpreting models are usually based on some post-analyses of the predictions, including the approaches in (1) visualizing higher level representations or reconstructions of inputs based on them [13] [4], (2) evaluating the sensitivity of predictions to local perturbations of inputs the input gradients [15] [6), and (3) extracting parts of inputs as justifications for predictions [17]. Another related but orthogonal direction is model compression of training sparse neural networks with the goal of memory and computational efficiency [18] 19 20). All these works require collecting all features first and provide post-hoc feature relevance justifications network pruning.
Recent efforts on dynamic feature selection adaptively assign features based on immediate statis- ignoring the information feature may have future predictions. Others treat
ties 21 22
23

---

Under review as a conference paper at ICLR 2021
feature selection a Markov Decision Process (MDP) and use Reinforcement Learning (RL) to solve [24] 25] 26 27 28 2]. However, solving the RL objective is not straightforward. Besides being sensitive hyperparameter settings in general, approximations such state space discretization and greedy approximations of the combinatorial objective were used to make the RI. problem tractable.
To this end, we propose dynamic feature selection method that be easily integrated into existing deep architectures and trained from end to end, enabling task-driven dynamic feature selection. To achieve this, we define feature selection module that dynamically selects which features to use any given time point. We then formulate a sequential combinatorial optimization that minimizes the trade-off between the learning task performance and the number of features selected at each time point. To make this problem tractable, we cast this combinatorial optimization problem into a stochastic optimization formulation. We then adopt differentiable relaxation of the discrete feature selection variables to make it amenable to stochastic gradient descent based optimization. It therefore can be plugged-in and jointly optimized with sate-of-the-an neural networks, achieving task-driven feature selection over time. To show our method's ability to adaptively select features while maintaining good performance, we evaluate it on four time-series activity recognition datasets: the UCI Human Activity Recognition (HAR) dataset [29], the OPPORTUNITY dataset (30,, the ExtraSensory dataset [1). as well as the NTU-RGB-D dataset 32
Several ablation studies and comparisons with other dynamic and static feature selection methods demonstrate the efficacy of our proposed method. Specifically, our dynamic feature selection is able use as low as 0.28% of the sensor features while still maintaining good human activity monitoring accuracy. Moreover, our dynamically selected features are shown to be interpretable with direct correspondence with different contexts and activity types.
2
METHODOLOGY
2.1 THE fo-NORM MINIMIZATION PROBLEM
Many regularization methods have been developed to solve simultaneous feature selection and model parameter estimation 99
The ideal penalty for the purpose of feature selection the lo-norm of the model coefficients for all predictors. This norm is equivalent to the number of nonzero terms in all the model coefficients. Given a dataset D containing
12
33 34 35).
independent and identically distributed (iid) input-output pairs {(x1 yI),..., (æN, ] with each æ; containing P features, hypothesis class of predictor functions f(: 0). and a loss function Lg y) between prediction ý and true output y, the fo-norm regularized optimization problem can be written follows:
min 0
Σ ((((x;)) 2 O||o
where |00||- Efen [10 + 0) penalizes the number of nonzero model coefficients.
In the models that linearly transform the input features x;, penalizing the weights relating to each feature in æ; enables sparse feature subset selection. However, such selection is static, as it does not adaptively select features that are appropriate for given context. Moreover, the optimization above is computationally prohibitive as it involves combinatorial optimization to select the subset of nonzero model coefficients corresponding to the input features.
In the following, we formulate our adaptive dynamic feature selection problem when learning with multivariate time series. Coupled with training recurrent neural networks, this adaptive feature selection problem is transformed into sequential context- -deeeddeededdent feature subset selection problem, to which we devise a stochastic relaxation to make the problem tractable.
2.2 DYNAMIC FEATURE SELECTION VIA SEQUENTIAL SUBSET SELECTION
FEATURE
Instead of finding a subset of nonzero model coefficients, an equivalent formulation can be derived by directly selecting the feature subset. Without loss of generality, let be binary vector that indicates whether each feature is selected or not. Then, the original norm optimization formulation can be equivalently written as follows:
2

---

Under review as a conference paper at ICLR 2021
N Σ c(f(2; oz 0);), + A||=|10-
1 N
min 0,
Compared to the original problem, the penalty the number of selected features is through the lo-norm of z. This formulation is more flexible, as 2 can be made dependent on corresponding input features, output labels, or any contextual information, allowing to formulate our dynamic feature selection problem when learning with multivariate time series data. Specifically, let the input-output pairs (z,, y4) be a pair of time series data of length T. At each time t, our model predicts the output y as well as the next feature set to select 2f. This optimization problem can be formulated as:
min 0.
EE (a (1:tt-111 0:t-1 2,0-11 -0,))())))) vi) +
|z
i=1
Here, we are tasked to find a set of parameters O and feature sets z for each sample at each time point to optimize the trade-off between model performance and the number of selected features. The model then uses the parameters and the previously observed features X - 2: 2::t-1 2, 2 to infer the next output yf. However, the above formulation remains intractable, it involves combinatorial optimization to select the feature subsets at each time point, in addition to the joint optimization the model parameters and variable selection. Naively, one may also need to solve separate optimization problem to find zf for each time point during the run time. In the following section, we derive a relaxation based stochastic optimization parameterizing 2 to make the above problem tractable.
2.3 RELAXATION THROUGH STOCHASTIC OPTIMIZATION
Instead of finding the exact feature subsets indexed by z that achieve the optimal regularized objective, one can treat these z²'s as binary random variables and seek optimize the distribution π (==()) that generates these random variables. For the ease of exposition, we first focus on the relaxation of the adaptive formulation in (I) as follows:
min 0.6
E Bx (E
[(f( z; 0),v, + A||=
Note that the solution this problem is equivalent the original one, as the original combinatorial problem can be recovered by setting (z(Ø)) = Bern(Φ), a Bernoulii distribution parameterized by Φ and restricting & {0, 13. Using this relaxation, the regularization term can now be evaluated analytically:
P Σ (z]0))
E B
[1:/1|| (g-
||=||a|
3
(5)
On the other hand, the outer expectation in can be approximated using minibatches. To extend
the above relaxation for time series data, we first note that our adaptive feature selection formulation
in 3 allows each time point to have its own feature selection distribution (=(Ø) - T= Φ)
conditioned previous observations X Let n,, (2 be the set of (2|()) (=|Ø) for all E {1..... T.).
The stochastic relaxation of the adaptive feature selection formulation can be written as follows:
T,
T.
min 0.
E (x,,))~
Eg (E
C(f(X) 00), +
(6)
2.4 MODEL PARAMETERIZATION AND DIFTERENTIABL RELAXATION
The difficulty in solving the above problem using gradient descent is that the discrete random variables 2 s are not directly amenable to stochastic reparameterization techniques. An effective and simple to implement formulation that we adopt is the Gumbel-SSfttn reparameterization [6] 37 which relaxes discrete valued random variable z parameterized by Φ to a continuous random variable Σ. Firstly, can parameterize x2 Φ) using vector-valued function a oxx
pegg-,ep .)) of the previous observations X.-1 with & now being the parameters of (-)). The distribution can now be rewritten T22|2²- Φ) (e .)).. With this, the discrete valued random variables z!
can be relaxed into continuous random variables ž as follows:
1 + exp (x (-loog x) L)/T)
3

---

Under review as a conference paper at ICLR 2021
Here, L log log(1 - u)) is logistic distribution, where Unif(O, 1). and is temperature parameter. For low values of т, ε; approaches a sample of a binary random variable, recovering the original discrete problem, while for high values, 2 will equal 2. With this, we are able to compute gradient estimates ž and approximate the gradient & as Va. ss V.. This enables us to backpropagate through the discrete random variables and train the selection parameters along with the model parameters jointly using stochas- tic gradient descent. Meanwhile, at test time, we sample binary random variables from the learned probabilities.
2.5 MODEL. SPECIFICATION
To complete formulation, specify the model architecture that we use. We have imple- mented our adaptive dynamic feature selection with Gated Recurrent Unit (GRU) [38]. a type of Recurrent Neural Network (RNN) (39] as shown in Figure D Here, we have the previous observations X being summarized by the hidden state h For adaptive feature selection, the selection distribution is made dependent on h using sigmoid of its linear transformation by a weight matrix W as follows: (x Φ) SIGMOID(Wh; (hhh- such that Φ {W}. We note that such module can be easily integrated into many existing deep architectures and trained from end to end, allowing for task-driven feature selection. For example, the module can be applied to Recurrent Convolutional Neural Networks (RCNN) 44 to selectively determine which convolu- tional patches/channels to use, or to general feedforward networks to selectively deactivate certain neurons/channels reduce computation. We have demonstrated this ability by applying it to an Independent RNN [41 benchmarked on the NTU-RGB-D dataset 32. as detailed in Appendix A.44
The proposed Gated Recurrent Unit (GRU) based architecture for our adaptive mon- itoring model specification. Here, features at any given time are selected based on the previous ob- servations summarized by ht-1-
Figure 1:
3 RELATED WORK
Existing HAR systems typically use fixed set of sensors, potentially collecting redundant features for easily discriminated contexts. Methods that attempt to find a fixed static feature set often rank feature sets using metrics such as Information Gain [5] or relevancy ranking through a filtering strategy (6] 7 g). However, static feature selection can potentially result in collecting redundant information for highly discriminable contexts.
Work on dynamic feature selection can be divided into Reinforcement Learning (RL) based and non-RL approaches. Non-RL based approaches vary from assigning certain features to certain activities (21), pre-defining feature subsets for prediction 22 42,, optimizing the trade-off between prediction entropy and the number of selected features 11. to building metaclassifier for sensor selection 233 These methods all use immediate rewards to perform feature selection. For predicting long activity sequences, this potentially ignores the information that feature may have on future predictions, or conversely, overestimate the importance of feature given previous observations.
Among the RL based approaches. some methods attempt build MDP decide which feature to select next or whether to stop acquiring features and make a prediction 24 225 26). These methods
condition the choice of one feature on the observation generated by another one, instead of choosing between all sensors simultaneously. Spaan and Lima [27] and Satsangi et al. [28] formulated
Partially Observable MDP (POMDP) using a discretization of the continuous state to model the
policy. Yang et al. 2 formulate an RI. objective by penalizing the prediction performance by the number of sensors used. Although using a desirable objective, the method employs a greedy maximization process to approximately solve the combinatorial optimization. Moreover, they do not integrate easily with existing deep architectures.
Attention is another method worth noting, as it is able to select the most relevant segments of sequence for the current prediction 43). Attention modules have been recently used for activity recognition [44). However, like most attention methods, it requires all of the features to be observed before deciding which features are the most important for prediction. Moreover, the number of instances attended to is not penalized. Finally, soft attention methods typically weight the inputs,
4

---

Under review as a conference paper at ICLR 2021
instead of selecting the feature subset. Indeed, our experiments naively applying attention for dynamic feature selection show that always selects 100% of the features at all times.
Sparse regularization has previously been formulated for deep models, c.g., 45 18 46). but their focus has primarily been in statically compressing model sizes or reducing overfitting, instead of dynamically selecting features for prediction. In particular, l regularization a common method to promote feature sparsity 9
12].
Finally, there have been many formulations that propose to solve the issue of backpropagation
through discrete random variables [36 37 47
48 49). REBAR [47] and RELAX (48] employ REINFORCE and introduce relaxation-based baselines reduce sample variance of the estimator.
However, these baseline functions increase the computation and cause potential conflict between minimizing the sample variance of the gradient estimate and maximizing the expectation objective. is self-control gradient estimator that does not need additional base- lines 49 provides unbiased gradient estimates that exhibit low variance, but its direct application to autoregressive or sequential setups is not addressed by Yin and Zhou [49] and leads to approximate gradients. Moreover, an exact sequential formulation will require prohibitive computation, squared in sequence length forward passes.
Beement-REINFORCE.C
4 EXPERIMENTS
Benchmark Datasets and Performance Evaluation We evaluate our model on four different datasets: the UCI Human Activity Recognition (HAR) using Smartphones Dataset [29]. the OPPOR- TUNITY Dataset 30. the ExtraSensory dataset [3T], and the NTU-RGB-D dataset [32 Although there are many other human activity recognition benchmark datasets [50], we choose the above datasets to better convey our message of achieving feature usage efficiency and interpretability using our adaptive feature selection framework with the following reasons. First, the UCI HAR dataset a clean dataset with no missing values, allowing us to benchmark different methods without any discrepancies in data preprocessing confounding dataset contains activity labels that correspond to specific sensors. An optimal adaptive feature selector should primarily choose these sensors under specific contexts with clear physical meaning. Finally, the ExtraSensory dataset studies multilabel classification problem, where two or more labels can be active at any given time, while the NTU-RGB-D dataset is complicated activity recognition dataset with over 60 classes of activities using data from 25 skeleton joints. These datasets allow us to benchmark model performance details and results on the NTU-RGB-D dataset are available in Appendix A and B
evaluations. Second, the OPPORTUNITY
complex setting. Due to the page limit, our implementation
We investigate several aspects of our model performance on these benchmarks. To show the effect in prediction accuracy when our selection module is considered, we compare its performance to standard GRU network [5T). To show the effect of considering dynamic feature selection, we compare nonadaptive to formulation that statically selects features by solving (4) [88] The performance of our εo regularized formulation is also benchmarked with an E regularized formulation. To benchmark the performance of differentiable relaxation-ban based optimization strategy, implement the Straight-Through estimator [52] and Augment-REINe REI-RRRENNOR-EEEE
(ARM) gradient estimates [49] alternative methods to optimize our formulation. As stated in the previous section, the fully sequential application of ARM was not addressed in the original paper, and will be prohibitively
100
Walding
Wel Elp
60
Adapthe
Monadapticc
Laytrs
10
20
30
40
(a)
(b)
(c)
Figure 2: UCI HAR Dataset results: (a) Prediction and features selected of the proposed model 1. (b) Feature selection vs. accuracy trade-off curve comparison. (c) Heatmap of sensor feature activations under each activity of the UCI HAR dataset. Only active features are shown out of the 561 features in total.
5

---

Under review as a conference paper at ICLR 2021
expensive to compute exactly. Hence, we combine ARM and Straight-Throus (ST) estimator [52] as another approach to optimize our formulation. More specifically, we calculate the gradients with respect to the Bernoulli variables with ARM, and use the ST estimator to backpropagate the gradients through the Bernoulli variables to previous layers' parameters.
To further show the importance of considering the sparse regularized formulation, compare with an attention-based feature selection, selecting features based on the largest attention weights. Because attention yields feature attention weights instead of feature subsets, we select features by using hard threshold a of the attention weights and scaling the selected features by 1 a for different values of a. Indeed, without this modification, we observe that an attention-baan based feature selection would select 100% of the features at all times.
Finally, we have attempted to implement the dynamic feature selection method by Yang al. [ as distinctly different benchmark. However, without any implementation details provided by the authors, we were not able reproduce their results.
Table 1: Comparison of various optimization techniques for model on the UCI HAR dataset. * *Accuracies and average
UCI HAR Dataset We first our proposed method on performing si- multaneous prediction and adaptive feature selection on the UCI HAR dataset [29]. This dataset consists of 561 smartphone sensor measurements including various gyroscope and ac- celerometer readings, with the task of inferring the activity that the user per- forms any given time. There are six possible activities that subject can perform: walking, walking upstairs, walking downstairs, sitting, standing, and laying.
number of features selected are in (%)
Method
Accuracy
Feat. Selected
Gumbel-Softtax À
97.18
0.28
ARM
49
95.73
11.67
ST-ARM À = 49 52
92.79
1.92
Straight Through
52
89.38
0.31
1.1 Regularization À
90.43
19.48
We first compare various optimization methods, using stochastic gradients by differential relaxation using Gumbel-Softma reparametrization, ARM, ST-ARM, Straight-Through gradients, and & regularized formulation to solve adaptive feature selection. The results are provided in Table [ As shown, Gumbel-Softmax achieves the best prediction accuracy with the least number of features. Utilizing either the Straight Through estimator, ARM, or ST-ARM for gradient estimation cannot provide better balance between accuracy and efficiency compared with the Gumbel-SSfttma relaxation-based optimization. Indeed, the performance of the ST estimator is expected, there is mismatch between the forward propagated activations and the backward propagated gradients in the estimator. Meanwhile, attribute the lower performance of the ARM and ST-ARM optimizer to its use in a sequential fashion, which was not originally considered. The lower performance of the l regularized formulation is expected, as é regularization is approximation to the problem of selecting the optimal feature subset. In the following experiments, we have seen similar trends and only report the results from the Gumbel-Softmax based optimization.
Benchmarking results of different models are given in Table 2 As shown, our adaptive feature selection model is able to achieve competitive accuracy using only 0.28% of the features, or on average about 1.57 sensors at any given time. We also observe that both the attention and our adaptive formulation is able to improve upon the accuracy of the standard GRU, suggesting that feature selection can also regularize the model to improve accuracy. Although the attention-basee model yields the best accuracy, this comes at a cost of utilizing around 50% of the features at any given time.
We study the effect of the regularization weight À by varying it from À E (1, 0.1, 0.01, 0.005, 0.001) We compare this with the attention model by varying the threshold a used to select features from E {0.5, 0.9, 0.95, 0.99, 0.995, 0.999), well as the nonadaptive model by varying its À from (1000, 100, 0.01, 0.005,0000000 0.001 A trade-off curve between the number of selected features
Table 2: Comparison of various models for adaptive monitoring three activity recognition datasets.
"Accuracy metrics and average number of features selected are all in (%).
UCI HAR
OPPORTUNITY
ExtraSensory
Method
Accuracy
Features
Accuracy
Features
Accuracy
F[
Features
Adaptive (Ours) =
97.18
0.28
84,26
15.88
91.14
55.06
11.25
Attention a = 0.5
98.38
49.94
83.42
54.20
90.37
53.29
54.73
Nonadaptive X=
18
95.49
14.35
81.63
49.57
91.13
53.18
42.32
No selection (GRU)
51
96.67
100
84.16
100
91.14
53.53
100

---

Under review as a conference paper at ICLR 2021
100
10
20
30
50
Feeturms Utadd
(c)
Figure 3: OPPORTUNITY Dataset results: (a) Prediction and features selected the proposed model
À 1. (b) Prediction and features selected of the proposed model on set of activity transitions. (c) Feature selection vs. Error trade-off curve comparison.
and the performance for the three models can be seen in Figure 2(b As shown in the figure, the accuracy of the attention model suffers increasingly with smaller feature subsets, as attention is not formulation specifically tailored to find sparse solutions. On the other hand, the accuracy of our adaptive formulation is unaffected by the number of features, suggesting that selecting around 0.3% of the features on average may be optimal for the given problem. further confirms that our adaptive formulation selects the most informative features given the context. The performance of the nonadaptive model consistent for feature subsets of size 10% or greater. However, it suffers a drop in accuracy for extremely small feature subsets. This shows that for static selection, selecting feature set that too large would result in collecting many redundant features for certain contexts, while selecting feature set that is too small would be insufficient for maintaining accuracy.
We plot the prediction of model compared to the true label and illustrate the features that used for prediction. We also plot a heatmap for the features selected under each activity in Figure 2(c) Note that mainly out of the 561 features are used for prediction at any given time. Observing the selected features, see that for the static activities such as sitting, standing, and laying, only sensor feature 52 and 63, features relating to the gravity accelerometer, are necessary for prediction. On the other hand, the active states such as walking, walking up, and walking down requires sensor features: sensor 65, 508, and 556, which related to both the gravity accelerometer and the body accelerometer. This is intuitively appealing as, under the static contexts. the body accelerometer measurements would be relatively constant, and unnecessary for prediction. On the other hand, for the active contexts, the body accelerometer measurements are necessary to reason about how the subject is moving and accurately discriminate between the different active states. Meanwhile, we found that measurements relating to the gyroscope were unnecessary for prediction.
An example dynamically selected features can be seen in Figure ((a)
UCI OPPORTUNITY Dataset We further test our proposed method on the UCI OPPORTUNITY Dataset 330 This dataset consists of multiple different label types for human activity, ranging from locomotion, hand gestures, to object interactions. The dataset consists of 242 measurements from accelerometers and Inertial Measurement Units (IMUs) attached the user, as well as accelerometers attached to different objects with which the user can interact.
We use the mid-level gesture activities as the target for our models to predict, which contain gestures related to specific objects, such as opening door and drinking from cup. A comparison of the accuracy and the percentage of selected features by different models is given in Table while
example predictions and trade-off curve constructed and shown in Figures 3(a) 3(b) and 3(c]
with similar trend as the results on the UCI HAR dataset. Notably, the trade-off for the nonadaptive models remains constant for À E {0.0001, 0.001, 1}. with sharp decrease in accuracy for Z 10.
opm 2oor
Ocse Door
Open Froge
05
Close Bxhherher
Clss Brawor
close 2
Dipwer
Claan "atlo
Toggle Switch
Number
Figure 4: Heatmap of sensor feature activations under each activity of the OPPORTUNITY dataset. *Only active features shown out of the 242 features in total.
7

---

Under review as a conference paper at ICLR 2021
vakes
Featuoes
(a)
(b)
Figure 5: ExtraSensory Dataset results: (a) Prediction and features selected of the proposed model.
(b) Feature selection vs. Error trade-off curve comparison.
A heatmap for the selected features under each activity is shown in Figure 4 Here, the active sensor features across all activities are features 40 and 42, readings of the IMU attached to the subject's back, feature 82, readings from the IMU attached to the left upper arm (LUA), and features 230 and 239, location tags that estimate the subject's position. We posit that these general sensor features are selected to track the subject's s overall position and movements, they are also predominantly selected in cases with no labels. Meanwhile, sensors 5, 6, and 16, readings from the accelerometer attached to the hip, LUA, and back, are specific to activities involving opening/closing doors or drawers.
Interestingly, sensors attached specific objects, such as accelerometers on doors and cups, are unnccessary for prediction. We attribute this to the severe amount of missing values of these sensors. Indeed, the sensors that have the least amount of missing values are the body sensors and the localization tags. We hypothesize that the model prefers these sensors for their consistent discriminative power on multiple activity types compared to the object specific sensors. In addition these object specific sensors, 5 IMUs, accelerometers, and 2 localization tags can be completely turned off without significantly affecting prediction performance on this task.
ExtraSensory Dataset We further test our proposed method the ExtraSensory Dataset [31 This is multilabel classification dataset, where two or more labels can be active at any given time. It consists of 51 different context labels, and 225 sensor features. We frame the problem as a multilabel binary classification problem, where we have binary output for each label indicating whether it is active. A comparison of the accuracy and selected features by different models tested can be seen in Table 2 Our method is again competitive with the standard GRU model using less than 12% of all the features.
A trade-off curve is shown in Figure 5(b) where we see a similar trend for both adaptive and attention models. However we were unable to obtain a feature selection percentage lower than 25% for the nonadaptive model even with À as large as 10a. We believe that this is because at least 25% of statically selected features are needed; otherwise the nonadaptive model will degrade in performance catastrophically, similar the OPPORTUNITY dataset results. A heatmap and detailed discussion of the features that our model dynamically selected can be found in Appendix
The results these three datasets along with the results the NTU-RGB- D dataset in Appendix B indicate that our adaptive monitoring framework provides the best trade-off between feature efficiency and accuracy, while the features that dynamically selects are also interpretable and associated with the actual activity types.
CONCLUSIONS
We propose novel method for performing adaptive feature selection by sequential context-dependen feature subset selection, which is cast into a stochastic optimization formulation by modifying the to regularized minimization formulation. To make this problem tractable, we perform a stochastic relaxation along with a differentiable reparamaterization, making the optimization amenable to gradient-based optimization with auto-differentiation. We apply this method to human activity recognition by implementing our method to Recurrent Neural Network-based architectures. We benchmark our model on four different activity recognition datasets and have compared with various adaptive and static feature selection benchmarks. Our results show that our model maintains a desirable prediction performance using a fraction of the sensors or features. The features that our model selected were shown to be interpretable and associated with the activity types.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
[1] Randy Ardywibowo, Guang Zhao, Zhangyang Wang, Bobak Mortazavi, Shuai Huang, and Xiaoning Qian. Adaptive activity monitoring with uncertainty quantification in switching gaussian process models. arXiv preprint
2019.
[2] Xiaodong Yang, Yiqiang Chen, Hanchao Yu, Yingwei Zhang, Wang Lu, and Ruizhe Sun. Instance-wise dynamic sensor selection for human activity recognition. http:///a......keeex////////Aaaa Aaa-e///AAAAYAAa..AAA. SaangX.7952.pdee 2020.
[3] Jack Kiefer. Optimum experimental designs. Journal of the Royal Statistical Society: Series B (Methodological), 21 1959.
[4] William P Pierskalla and David Brailer. Applications of operations research in health care delivery. Handbooks in operations research and management science, 6:469-505, -0699-50,,10 1994.
[5] Xiaojing Shen and Pramod K Varshney. Sensor selection based on generalized information gain for target tracking in large sensor networks. IEEE Transactions Signal Processing, 62(2):363-375, 2013.
[6] Omar Aziz, Stephen N Robinovitch, and Edward Park. Identifying the number and location of body worn sensors accurately classify walking, transferring and sedentary activities. In 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pages 5003-5006. IEEE, 2016.
[7] Ömer Faruk Ertuğrul and Yılmaz Kaya. Determining the optimal number body-worn sensors for human activity recognition. Soft Computing, 21(17):5053-500 2017.
[8] Weihao Cheng, Sarah Erfani, Rui Zhang, and Ramamohanarao Kotagiri. Learning datum- date sampling frequency for energy-effiiient human activity recognition. Thirty-Second AAAI Conference Artificial Intelligence, 2018.
[9] Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):667-288, 1996.
[10] Jerome Friedman, Trevor Hastie, and Robert Tibshirani. A note on the group lasso and a sparse group lasso. arXiv preprint arXiv: 2010.
[11] Jerome Friedman, Trevor Hastie, and Robert Tibshirani. Sparse inverse covariance estimation with the graphical lasso. Biostatistics, 9()):432-441, 2008.
[12] Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the royal statistical society: series B (statistical methodology), 67(2):301 2005.
[13] Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. Visualizing and understanding neural models in NLP. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 68 1-19,, San Diego, California, June 2016. Association for Computational Linguistics.
[14] A. Mahendran and A. Vedaldi. Understanding deep image representations by inverting them. In 2015 IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2015.
[15] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-Cam: Visual explanations from deep networks via gradient- braed localization. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 618-626, 2017.
[16] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why should i trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16 page 1135-1144 144, New York, NY, USA, 2016. Association for Computing Machinery.
[17] Tao Lei, Regina Barzilay, and Tommi Jaakkola. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 107-117, Austin, Texas, November 2016. Association for Computational Linguistics.

---

Under review as a conference paper at ICLR 2021
[18] Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1 1_ regularization. arXiv preprint arXiv:1712.01312,, 2017.
[19] Enzo Tartaglione, Skjalg Lepsøy, Attilio Fiandrotti, and Gianluca Francini. Learning sparse neural networks via sensitivity-drive regularization. Proceedings of the 32nd International Conference Neural Information Processing Systems, NIPS'18 18, page 3882-3892, 2018.
[20] Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems- Volume 1, NIPS' 15, page 1135-1143. 2015
[21] Dawud Gordon, Jurgen Czerny, Takashi Miyaki, and Michael Beigl. Energy-efficient activity recognition using prediction. In International Symposium Wearable Computers. IEEE, 2012.
[22] Victoria Bloom, Vasileios Argyriou, and Dimitrios Makris. Dynamic feature selection for online action recognition. In International Workshop on Human Behavior Understanding, 2013.
[23] Piero Zappi. Clemens Lombriser, Thomas Stiefmeier, Elisabetta Farella, Daniel Roggen, Luca Benini, and Gerhard Tröster. Activity recognition from on- on-body sensors: accuracy-power trade-off by dynamic sensor selection. In European Conference on Wireless Sensor Networks, pages 17-33. Springer, 2008.
[24] He He and Jason Eisner. Cost-sensitive dynamic feature selection. In ICML Inferning Workshop. 2012.
[25] Sergey Karayev, Mario Fritz, and Trevor Darrell. Dynamic feature selection for classification on a budget. In International Conference on Machine Learning (ICML): Workshop on Prediction with Sequential Models, 2013.
[26] Harini Kolamunna, Yining Hu, Diego Perino, Kanchana Thilakarathna, Dwight Makaroff, Xinlong Guan, and Aruna Seneviratne. Afv: Enabling application function virtualization and scheduling in wearable networks. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pages 981-991 2016.
[27] Matthijs TJ Spaan and Pedro U Lima. A decision-theoretic approach to dynamic sensor selection in camera networks. In Nineteenth International Conference on Automated Planning and Scheduling, 2009.
(28) Yash Satsangi, Shimon Whiteson, and Frans A Oliehoek. Exploiting submodular value functions for faster dynamic sensor selection. In AAAI Conference on Artificial Intelligence, 2015.
[29] Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, and Jorge Luis Reyes-Ortiz. A public domain dataset for human activity recognition using smartphones. In 21th European Symposium Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN), pages 437-442. CIACO, 2013.
[30] Daniel Roggen, Alberto Calatroni, Mirco Rossi, Thomas Holleczek, Kilian Förster, Gerhard Tröster, Paul Lukowiez. David Bannach, Gerald Pirkl, Alois Ferscha, al. Collecting complex activity datasets in highly rich networked sensor environments. In 2010 Seventh international conference on networked sensing systems (INSS), pages 233-240. IEEE, 2010.
[31] Yonatan Vaizman, Katherine Ellis, and Gert Lanckriet. Recognizing detailed human context in the wild from smartphones and smartwatches. IEEE Pervasive Computing, 16(4):62-74, 2017.
[32] Amir Shahroudy, Jun Liu, Tian-T Tan-TTsng Ng, and Gang Wang. Ntu rgb+d: A large scale dataset for 3d human activity analysis. In IEEE Conference on Computer Vision and Pattern Recognition, June 2016.
[33] Robert Tibshirani. The lasso method for variable selection in the cox model. Statistics in medicine, 16(4):385-395, 1997.
[34] Hokeun Sun, Wei Lin, Rui Feng, and Hongzhe Li. Networ--regularized high-dimensional cox regression for analysis of genomic data. Statistica Sinica, 24(3):1433, 2014.
10

---

Under review as a conference paper at ICLR 2021
[35] Noah Simon, Jerome Friedman, Trevor Hastie, and Rob Tibshirani. Regularization paths for cox's proportional hazards model via coordinate descent. Journal of statistical software, 2011.
[36] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint 161 16100144 2016.
[37] Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint 161 12, 2016.
[38] Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprini arXiv:1409.12555 2014.
[39] Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing, pages
IEEE, 2013.
[40] Ming Liang and Xiaolin Hu. Recurrent convolutional neural network for object recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3367-3375, 2015.
[41] Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao. Independently recurrent neural network (indran): Building longer and deeper mnn In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5457-5466, 2018.
[42]
Emma Strubell. Luke Vilnis, Kate Silverstein, and Andrew McCallum. Learning dynamic feature selection for fast sequential prediction. arXiv preprint arXiv:1505.0610 2015.
[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 1998-6008 2017.
[44] Haojie Ma, Wenzhong Li, Xiao Zhang, Songcheng Gao, and Sanglu Lu. Attssense: multi-level attention mechanism for multimodal human activity recognition. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 3109-3115. AAAI Press, 2019.
[45] Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall Tappen, and Marianna Pensky. Sparse convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 806-814, 2015.
[46] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803. 03635, 2018.
[47] George Tucker, Andriy Mnih, Chris J Maddison, John Lawson, and Jascha Sohl-Dickste Rebar: Low-variance, unbiased gradient estimates for discrete latent variable models. In Advances Neural Information Processing Systems, pages 2627-2636. 2017.
[48] Will Grathwohl, Dami Choi, Yuhuai Wu, Geoffrey Roeder, and David Duvenaud. Backpropa- gation through the void: Optimizing control variates for black-box gradient estimation. arXiv preprint arXiv: 171 1.00/23, 2017.
[49] Mingzhang Yin and Mingyuan Zhou. Arm: Augmment-reinffore-mergg gradient for discrete latent variable models. arXiv preprint arXiv: 2018.
[50] Kaixuan Chen, Dalin Zhang, Lina Yao, Bin Guo, Zhiwen Yu, and Yunhao Liu. Deep learning for sensor- based human activity recognition: Overview, challenges and opportunities. arXiv preprint arXiv:2001.0777 074416 2020.
[51] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcchre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder- decoder for statistical machine translation. arXiv preprint 1406.1:406.101 2014.
[52] Geoffrey Hinton, Nitsh Srivastava, and Kevin Swersky. Neural networks for machine learning. Coursera, video lectures, 264:1, 2012.
11

---

Under review as a conference paper at ICLR 2021
A
IMPLEMENTATION DETAILS
Here, we provide the implementation details for the reported results in each benchmark dataset.
A.1 UCI HAR DATASET
The UCI HAR dataset consists of a training set and a testing set. To implement our adaptive feature selection and other baseline methods, we divide the training set into a separate validation set consisting of 2 subjects. We preprocess the data by normalizing it with the mean and standard deviation. We then divide the instances of each subject into segments of length 200.
The base model we utilize is one-layer GRU with 2800 neurons for the hidden state. We use the cross-entropy of the predicted vs. actual labels as the performance measure. We use a temperature of 0.05 for the numbel-Softmart relaxation. We optimize this with batch size of 10 using the RMSProp optimizer, setting the learning rate to 10-4 and the smoothing constant to 0.99 for 3000 epochs. We then save both the latest model and the best model validated on the validation set.
A.2 OPPORTUNITY DATASET
The OPPORTUNITY dataset consists of multiple demonstrations of different activity types. We first extract the instances into segments containing no missing labels for the mid-level gestures. Segments of length smaller than 100 are padded using the observed values the next time-points in the instance. We then normalize the data such that its values are between -1 and 1. The authors of the dataset recommended removing some features that they believed are not useful, however we find that this does not affect performance and instead use the entire feature set. We have also experimented with interpolating the missing values but also find that does not affect performance compared to imputing the missing values with zeros. Using this, we randomly shuffle the segments and assign 80% for training, 10% for validation, and 10% for testing.
The base model we utilize is a two-layer GRU with 256 neurons for each layer's hidden state. The cross-entropy of the predicted vs. actual labels is adopted as the performance measure. We use a temperature of 0.05 for the Gumbel-Softmax relaxation. We do not include the cross-entropy loss for the time points with missing labels. We also scale the total performance loss the observed labels #timepoints
for each batch by #labelled timepoints optimizer, setting the learning rate 10-4 and the smoothing constant to 0.99 for 3000 epochs. We then save both the latest model and the best model validated on the validation set.
We optimize this loss with a batch size of 100 using the R.MSPro
A.3 EXTRASENSORY DATASET
The ExtraSensory dataset consists of multiple demonstrations of human behavior under different activities, where two more activity labels be active at the same time. We first extract the instances into segments containing no missing labels for the middle level gestures. Segments of length smaller than 70 are padded using the observed values the next time-points in the instance. We then normalize the data such that its values are in between -1 and 1. We have experimented with interpolating the missing values but also find that it does not affect performance compared to imputing the missing values with zeros. Using this, we randomly shuffle the segments and assign 70% for training, 10% for validation, and 20% for testing.
The base model we utilize is one-layer GRU with 2240 neurons for its hidden state. We use temperature of 0.05 for the Gumbel-Softmal relaxation. We use the binary cross-entropy of the predicted vs. actual labels the performance measure, where the model outputs a binary decision for each label, representing whether each label is active or not. We do not include the performance loss for the missing labels and scale the total performance loss of the observed labels for each batch # #timepoints x #total labels
by #observed labels labelled timepoints'
We optimize this scaled loss with batch size of 100 using the RMSProp optimizer, setting the learning rate to 10 and the smoothing constant to 0.99 for 10000 epochs. We then save both the latest model and the best model validated on the validation set.
12

---

Under review as a conference paper at ICLR 2021
A.4 NTU-RGB-D DATASET
We first preprocess the NTU-RGB-D dataset remove all the samples with missing skeleton data. We then segment the time-series skeleton data across subjects into 66.5% training. 3.5% validation, and 30% testing sets. The baseline model that we have implemented for the NTU-RGB-D dataset is the Independent RNN [T) This model consists of stacked RNN modules with several additional dropout, batch normalization, and fully connected layers in between. Our architecture closely follows the densely connected independent RNN of [41]. To incorporate feature selection using either our adaptive formulation or an attention baseed formulation, we add an additional RNN to the beginning of this model. This RNN takes input the 25 different joint features and is tasked to select the joints to use for prediction further along the architecture pipeline. Since the joints are in the form of 3D coordinates, our feature selection method is modified such that it selects either all 3 of the X, Y, and Z coordinates of a particular joint, or none at all. Our architecture can be seen in 6
Similar the baseline method presented in 41, we have trained this architecture using batch size of 128 and a sequence length of 20 using the Adam optimizer with a patience threshold of 100 iterations. We then save both the latest model and the best model validated on the validation set.
V/2
concat
concat
concat
IndRNN
IndRNN
IndRNN
Batch Norm
Batch Norm
Batch Norm
FC
FC
FC
IndRNN
IndRNN
IndRNN
Batch Norm
Batch Norm
Batch Norm
FC
ho
h1
hg
IndRNN
IndRNN
IndRNN
OT
Figure 6: Our modified densely connected independent RNN architecture for adaptive feature selection.
B RESULTS AND DISCUSSION OF THE NTU-RGB-D DATASET
We have tested our proposed method on the NTU-RGB-D dataset [3] This dataset consists of 60 different activities performed by either a single individual or two individuals. The measurements of this dataset are in the form of skeleton data consisting of 25 different 3D coordinates of the corresponding joints of the participating individuals.
We compare our method with three different baselines shown in Table the standard independent RNN, soft attention baseline, and thresholded attention baseline. We see that our method maintains a competitive accuracy compared to the baseline using less than 50% of the features. On the other hand, because the thresholded attention formulation is not specifically optimized for feature sparsity, we see that it performs significantly worse compared to the other methods. Meanwhile, the soft- attention slightly improves upon the accuracy of the base architecture. However, as also indicated by our other experiments, soft-attention is not a dynamic feature selection method, and tends to select 100% of the features at all times.
13

---

Under review as a conference paper at ICLR 2021
A heatmap for the features selected under each activity is shown in Figure 7 Here, that there are two distinct feature sets used for two different types of interactions: single person interactions and two person interactions. Indeed, since the two person activities require sensor measurements from two individuals, the dynamic feature selection would need to prioritize different features observe their activities opposed to single person activities.
Table 3: Comparison of various methods for activity recognition on the NTU-RGB-D dataset. Accuracies and average number features selected are (%).
Method
Accuracy
Features Selected
Adaptive
80.54
49.65
Thresholded attention
40.07
52.31
Soft attention
83.28
100
No selection
83.02
100
drink
eat
brushing teeth
brushing haif
0.8
throw
sitling down
standing up
clapping
wriling tear paper wea ar facket
take off jacket
take a shoe
wear on glasses Lake off glasses put on aaal/cap
hat/cap hand waving
kicking something reach into ket hopping
phone
playing with chone
typing on keyboard
painting to something with finger
laking
check
two hands logether
nod head/bow shake head
wipe
0.5
the palms together
crass hands in front
sheeze
staggering
lallin
touch chest
touch back
touch neck
nausea or vomiting condition
punching other person kicking other person pushing other person pat back of other person point finger the hugging giving som ething other person touch other person's pockel
0.4
person
handshaking
walking towards other
walking
spine
middle of the spine
neck
left shoulder head
left wrist
left hand
right shoulder
right elboww
right hand
left hip
left knee-
left anklee
right hip
right knee
right ankle
spine. right foot
tip of the hand
left thumb
Up of the right hand right thumb
rignt
left
Figure 7: Heatmap sensor feature activations under each activity state NTU-RGB-D dataset.
14

---

Under review as a conference paper at ICLR 2021
Phone table
Indton
home
lying
man worcplace
Sleaping
00
cubsid
A2Dwlty
With
hand
Dressing
calcoren
004
Drve
pusargge
Shoppin
Stais geing dowt
Coanin
Aca
ie the oyn
Phore bag
22
Figure 8: Heatmap of sensor feature activations under each activity state of the ExtraSensory dataset.
C RESULTS AND DISCUSSION OF THE EXTRASENSORY DATASET
heatmap of the features selected under each activity state can be seen in Figure As shown, there are four groups of sensor features that are used across activities: the phone magnetometer (57-71), watch accelerometer magnitude (85-88), watch accelerometer direction (101-105), and location 147). For two particular states, 'on a bus' and 'drinking alcohol', phone accelerometer measurements (5-52) become necessary for prediction. Some states such as 'at home", 'at main workplace', and 'phone in pocket' notably sparse in sensor feature usage. We believe that these states static, and do not require much sensor usage monitor effectively. Other sensors such as the phone gyroscope, phone state, audio measurements and properties, compass, and various low- wwwffregeency sensors are largely unnecessary for prediction in this dataset.
15


Reviews:
[{"review_id": "caIqTnXIjD", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors tackle the important problem of feature selection. They propose to use differentiable gates with an RNN architecture to select different subsets of features for each time point. I think the idea and method are interesting, and the method could be useful. However, I have crucial problems with the way the paper is presented. Most importantly, the authors describe the l_0 relaxation of Bernoulli random variables as if it is their own contribution. They describe existing known results under a section titles “Methodology” as if they are the first to present Bernoulli random variables to feature selection or that they are the first to relax them using the Gumbel Softmax trick. They also use the word: “we derive” (p.3). This is wrong! And misleading! The same relaxation appears in [1] and used for model sparsification, the descriptions are almost identical to what appears in [1] with almost zero credit to the authors in [1] (a citation appears in related work in a different context). Bernoulli relaxation was already used for feature selection, in [2], and [3], these papers were not even mentioned. The reader can think the authors are the first to introduce such relaxation into the problem of feature selection, while this is again, clearly wrong.\nThe authors are well aware of that this relaxation was presented in [1], and in the experiment section they describe the baseline which solves (4) by citing [1] (citation [18] in their paper), this is again in contradiction to the way they describe the relaxation as if it is their own contribution. \n\nPutting these CRITICAL comments aside, I think the results are misleading. Specifically, comparing the average number of selected features to the (constant) number of selected features of the non-adaptive method is misleading. You need to compare the union of selected features by your method to the constant number, otherwise, there is no way to infer if this feature selection method can result in any compression of the model or could lead to training or inference speed up.  Given that this is what you measure since you still need all the features to use your model, what are the advantages of the method? Only interpretability?\n\nThe authors do not explain how the method is used in the testing phase, is the randomness removed? How exactly?\nThe authors do not explain how training/ testing is performed, this appears in the appendix but should be moved to the main texts.\nThe authors should compare the method to the distribution suggested in [1], which seems more suitable for feature selection than the Concrete distribution (used by the authors).\nCitations are not in the correct ICLR format.\n\nSome pros: I like the examples used in the paper as well as the comparison to ARM, ST, ST-ARM.\nTo conclude, I am voting to reject the paper, based on all the reasons mentioned above.\n\n[1] Louizos, Christos, Max Welling, and Diederik P. Kingma. \"Learning Sparse Neural Networks through $ L_0 $ Regularization.\" ICLR, 2018.\n\n[2] Yamada, Y., Lindenbaum, O., Negahban, S., & Kluger, Y.  Feature selection using stochastic gates. ICML, 2020.\n\n[3] Balın, Muhammed Fatih, Abubakar Abid, and James Zou. \"Concrete autoencoders: Differentiable feature selection and reconstruction.\" ICML. 2019.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "aEyOoDp3IsO", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors provide a novel combination of known architectures to an important use case of reducing the density of required  measurements in sensor-fusion based temporal multi-class inference tasks. This has implications in energy consumptions of wearable sensors.  but could even generalise to measurement timings in clinical care to make the work of nurses more efficient, and reduce the stress caused by some medical procedures..\n\nThe authors represent a way to train consistent policy that predicts the best combination of sensors to estimate the state of the subjects.  They have found that a smaller set of features.  is more explainable than the full set of features.  However, I think that this somewhat of an overpromise.  The trained model gives the optimal density of the measurements and can discern also if old values and features measured are till OK for the inference.  This does not mean that those measurements are not needed at all in the features.  One can only argue that the required features can be estimated from the older measurement. So, the current set of active sensors is not the full set of required measurement values and can not be exclusively used to explain the logic of the system. Even more, the logic of the policy deciding the new  measurement is not discussed in an explainability context. The authors provide no data on this. It may be just an  estimate the derivative of the signal and ignore a new measurement, if it's time  derivative is small enough. \n\nAs a summary , I support publication of the manuscript, provided  the authors modify the message on the interpretable features.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Jrf1dulzrls", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an RNN model for adaptive dynamic feature selection, for efficient and interpretable human activity recognition (HAR). From the intuition that human activity can be predictable by using a small number of sensors, the paper introduces an l0-norm minimization problem with parameter regularization, and provide a logic on formulating a dynamic feature selection model with relaxations. The difficulty of the discrete optimization problem is solved by differentiable relaxation, which is known as Gumbel-Softmax reparameterization techniques. The formulation is naturally led to an RNN model that uses histories as input with an additional sigmoid unit for adaptive feature selection. \n\nEmpirical studies are performed to show the superiority of the adaptive feature selection network. Results are shown on the task of 1) UCI-HAR smartphone dataset with 561 features, 2) UCI Opportunity sensor dataset with 242 features, 3) ExtraSensory dataset with 225 features for multilabel binary classification. In particular, by using the adaptive feature selection technique, the average number of features necessary for HAR prediction can be very small (0.3%, 15.9%, 11.3% among all features) at any given time. Overall, the paper is well written. In particular, analysis results on three datasets are clear and detailed, so that the reader would be available to understand what sensors were necessary for HAR prediction.\n \nThe key concern about the paper is that the algorithm lacks practicality. To show the adaptive selection algorithm is efficient, it should be shown that the algorithm drastically reduces features that are not necessary for prediction over time, while maintaining the performance even in the lighter feature space. Although the average number of features selected by the adaptive selection algorithm for each snapshot is small, all features are entered as input, which may not help to speed up the algorithm. To claim that the algorithm is efficient, it is required to show that the computation cost can be saved. Also, based on the current experimental results, it is difficult to say that features that were not used in earlier timestamp will not be used in later timestamp with a different context. \n \nMinor comments and questions:\n- Can you report the running time of each model? \n- Is this model working in an online setting without tuning? If yes, would you like to clarify? If no, may I think this technique is for maintaining a dashboard that informs important features every time to users by calculating feature importance over time?\n- The performance of the adaptive method on the NTU-RGB-D dataset is quite poor. What part of the dataset do you think caused the difficulty in feature selection? Do all features important?\n- The technical novelty seems to be low if the proposed model is an RNN with an additional sigmoid layer.\n- Figure 2a does not have a ground truth blue line.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "CIGP18JiNz", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a learning-based binary sampling mechanism for feature selection. It filters salient feature dimensions by sampling from a Gumbel-softmax distribution, which is differentiable and can be trained with other network parameters. The proposed method is evaluated on several Human Activity Recognition (HAR) datasets.\n\nThe positive and negative points of this paper can be summarized as following:\n\npros:\n+ This paper is well written and is easy to follow.\n+ The experimental evaluations give positive results. \n\ncons: \n- Important previous works are missing. Learning to generate categorical samples for RNNs is not a fresh idea. Actually, [a] has already employs Gumbel-softmax to sample scales in order to dynamically control the temporal pattern learning; More generally, the topic of this paper is connected to a amount of previous works aiming to adaptively decide how/when to memorize/update the inputs/states, such as [b] and [c]. These works should also been cited by this paper.\n\n- With these missing works taking into account, the novelty of this paper becomes incremental and contribution is trivial. Integrating Gumbel-softmax sampling with RNN cells is very straightforward, and the motivation of applying Gumbel-softmax is very similar to [a]. While [a] is proposed for general sequence tasks, the proposed method seems to work only for HAR with multi-dimensional inputs. \n\n- Since \\tau is the only hyper parameter of Gumbel-softmax, evaluations on how the value of \\tau could impact the performance can be important. Yet no such results are reported in the paper. From the original Gumbel-softmax paper we can see a sample can approximate to a one-hot vector when \\tau is small and be closed to a uniform distribution when \\tau goes large. So it is very likely that the performance will become unstable as \\tau changes. Showing such experimental results could be definitely improve the paper quality. I would suggest to report the means and stds of accuracies with different sampling seeds.\n\nSummary:\nConsidering the concerns listed above, I believe there are problems that outweighs the strengths of this paper. They should be fixed before acceptance.\n\n[a] H Hu, et al. Learning to Adaptively Scale Recurrent Neural Networks. AAAI 2019\n[b] V Campos, et al. Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks. ICLR 2018\n[c] D Neil, et al. Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences. NIPS 2016\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a methodoloy for dynamic feature selection. They use differentiable gates with 
an RNN architecture to select different subsets of features at each time point thus resulting in dynamic selection. 
The reviewers agree that the idea is interesting and the method could be useful and I share their opinion.

The majority vote is towards rejection. The overarching mwssage of the reviews is that the manuscript raises confusion in a number of points. I see this work as one with good potential for impact but its current presentation is confusing. The vivid discussion that it raised is also an indication of it. The authors have done a good job replying to the concerns and the questions raised. However, the reviewers were still unsatisfied with the authors response to their concerns.  I recommend rejection at this time, while encouraging the authors to take seriously the reviewers' requests for a clearer presentation of their approach's contribution in order to strengthen their paper for future submission.


Decision: Reject