Dynamic Feature Selection for Efficient and Interpretable Human Activity Recognition

1 INTRODUCTION . Acquiring predictive features is critical for building trustworthy machine learning systems , but this often comes at a daunting cost . Such a cost can be in the form of energy needed to maintain an ambient sensor ( Ardywibowo et al. , 2019 ; Yang et al. , 2020 ) , time needed to complete an experiment ( Kiefer , 1959 ) , or manpower required to monitor a hospital patient ( Pierskalla & Brailer , 1994 ) . Therefore , it becomes important not only to maintain good performance in the specified task , but also a low cost to gather these features . Indeed , existing Human Activity Recognition ( HAR ) methods typically use a fixed set of sensors , potentially collecting redundant features to discriminate contexts ( Shen & Varshney , 2013 ; Aziz et al. , 2016 ; Ertuǧrul & Kaya , 2017 ; Cheng et al. , 2018 ) . Classic feature selection methods such as the LASSO and its variants can address the performance-cost trade-off by optimizing an objective penalized by a term that helps promote feature sparsity ( Tibshirani , 1996 ; Friedman et al. , 2010 , 2008 ; Zou & Hastie , 2005 ) . Such feature selection formulations are often static , that is , a fixed set of features are selected a priori . However , different features may offer different predictive power under different contexts . For example , a health worker may not need to monitor a recovering patient as frequently compared to a patient with the declining condition ; an experiment performed twice may be redundant ; or a smartphone sensor may be predictive when the user is walking but not when the user is in a car . By adaptively selecting which sensor ( s ) to observe at any given time point , one can further reduce the inherent cost for prediction and achieve a better trade-off between cost and prediction accuracy . In addition to cost-efficiency , an adaptive feature selection formulation can also lead to more interpretable and trustworthy predictions . Specifically , the predictions made by the model are only based on the selected features , providing a clear relationship between input features and model predictions . Existing efforts on interpreting models are usually based on some post-analyses of the predictions , including the approaches in ( 1 ) visualizing higher level representations or reconstructions of inputs based on them ( Li et al. , 2016 ; Mahendran & Vedaldi , 2015 ) , ( 2 ) evaluating the sensitivity of predictions to local perturbations of inputs or the input gradients ( Selvaraju et al. , 2017 ; Ribeiro et al. , 2016 ) , and ( 3 ) extracting parts of inputs as justifications for predictions ( Lei et al. , 2016 ) . Another related but orthogonal direction is model compression of training sparse neural networks with the goal of memory and computational efficiency ( Louizos et al. , 2017 ; Tartaglione et al. , 2018 ; Han et al. , 2015 ) . All these works require collecting all features first and provide post-hoc feature relevance justifications or network pruning . Recent efforts on dynamic feature selection adaptively assign features based on immediate statistics ( Gordon et al. , 2012 ; Bloom et al. , 2013 ; Ardywibowo et al. , 2019 ; Zappi et al. , 2008 ) , ignoring the information a feature may have on future predictions . Others treat feature selection as a Markov Decision Process ( MDP ) and use Reinforcement Learning ( RL ) to solve it ( He & Eisner , 2012 ; Karayev et al. , 2013 ; Kolamunna et al. , 2016 ; Spaan & Lima , 2009 ; Satsangi et al. , 2015 ; Yang et al. , 2020 ) . However , solving the RL objective is not straightforward . Besides being sensitive to hyperparameter settings in general , approximations such as state space discretization and greedy approximations of the combinatorial objective were used to make the RL problem tractable . To this end , we propose a dynamic feature selection method that can be easily integrated into existing deep architectures and trained from end to end , enabling task-driven dynamic feature selection . To achieve this , we define a feature selection module that dynamically selects which features to use at any given time point . We then formulate a sequential combinatorial optimization that minimizes the trade-off between the learning task performance and the number of features selected at each time point . To make this problem tractable , we cast this combinatorial optimization problem into a stochastic optimization formulation . We then adopt a differentiable relaxation of the discrete feature selection variables to make it amenable to stochastic gradient descent based optimization . It therefore can be plugged-in and jointly optimized with state-of-the-art neural networks , achieving task-driven feature selection over time . To show our method ’ s ability to adaptively select features while maintaining good performance , we evaluate it on four time-series activity recognition datasets : the UCI Human Activity Recognition ( HAR ) dataset ( Anguita et al. , 2013 ) , the OPPORTUNITY dataset ( Roggen et al. , 2010 ) , the ExtraSensory dataset ( Vaizman et al. , 2017 ) , as well as the NTU-RGB-D dataset ( Shahroudy et al. , 2016 ) . Several ablation studies and comparisons with other dynamic and static feature selection methods demonstrate the efficacy of our proposed method . Specifically , our dynamic feature selection is able to use as low as 0.28 % of the sensor features while still maintaining good human activity monitoring accuracy . Moreover , our dynamically selected features are shown to be interpretable with direct correspondence with different contexts and activity types . 2 METHODOLOGY . 2.1 THE ` 0-NORM MINIMIZATION PROBLEM Many regularization methods have been developed to solve simultaneous feature selection and model parameter estimation ( Tibshirani , 1996 ; Zou & Hastie , 2005 ; Tibshirani , 1997 ; Sun et al. , 2014 ; Simon et al. , 2011 ) . The ideal penalty for the purpose of feature selection is the ` 0-norm of the model coefficients for all predictors . This norm is equivalent to the number of nonzero terms in all the model coefficients . Given a dataset D containing N independent and identically distributed ( iid ) input-output pairs { ( x1 , y1 ) , . . . , ( xN , yN ) } with each xi containing P features , a hypothesis class of predictor functions f ( · ; θ ) , and a loss function L ( ŷ , y ) between prediction ŷ and true output y , the ` 0-norm regularized optimization problem can be written as follows : min θ 1 N ( N∑ i=1 L ( f ( xi ; θ ) , yi ) ) + λ‖θ‖0 , ( 1 ) where ‖θ‖0 = ∑P j=1 I [ θj 6= 0 ] penalizes the number of nonzero model coefficients . In the models that linearly transform the input features xi , penalizing the weights relating to each feature in xi enables sparse feature subset selection . However , such a selection is static , as it does not adaptively select features that are appropriate for a given context . Moreover , the optimization above is computationally prohibitive as it involves combinatorial optimization to select the subset of nonzero model coefficients corresponding to the input features . In the following , we formulate our adaptive dynamic feature selection problem when learning with multivariate time series . Coupled with training recurrent neural networks , this adaptive feature selection problem is transformed into a sequential context-dependent feature subset selection problem , to which we devise a stochastic relaxation to make the problem tractable . 2.2 DYNAMIC FEATURE SELECTION VIA SEQUENTIAL CONTEXT-DEPENDENT FEATURE SUBSET SELECTION . Instead of finding a subset of nonzero model coefficients , an equivalent formulation can be derived by directly selecting the feature subset . Without loss of generality , let z be a binary vector that indicates whether each feature is selected or not . Then , the original ` 0-norm optimization formulation can be equivalently written as follows : min θ , z 1 N ( N∑ i=1 L ( f ( xi ◦ z ; θ ) , yi ) ) + λ‖z‖0 . ( 2 ) Compared to the original problem , the penalty on the number of selected features is through the ` 0-norm of z . This formulation is more flexible , as z can be made dependent on corresponding input features , output labels , or any contextual information , allowing us to formulate our dynamic feature selection problem when learning with multivariate time series data . Specifically , let the input-output pairs ( xi , yi ) be a pair of time series data of length Ti . At each time t , our model predicts the output yti , as well as the next feature set to select z t i . This optimization problem can be formulated as : min θ , z 1 N ( N∑ i=1 Ti∑ t=1 L ( f ( x0 : t−1i ◦ z 0 : t−1 i ; θ ) , y t i ) ) + λ N∑ i=1 Ti∑ t=1 ‖zti‖0 . ( 3 ) Here , we are tasked to find a set of parameters θ and feature sets zti for each sample i at each time point t to optimize the trade-off between model performance and the number of selected features . The model then uses the parameters and the previously observed features X ti , x 0 : t−1 i ◦ z 0 : t−1 i to infer the next output yti . However , the above formulation remains intractable , as it involves combinatorial optimization to select the feature subsets at each time point , in addition to the joint optimization of the model parameters and variable selection . Naively , one may also need to solve a separate optimization problem to find zti for each time point during the run time . In the following section , we derive a relaxation based on stochastic optimization parameterizing zti ’ s to make the above problem tractable . 2.3 RELAXATION THROUGH STOCHASTIC OPTIMIZATION . Instead of finding the exact feature subsets indexed by zti that achieve the optimal regularized objective , one can treat these zti ’ s as binary random variables and seek to optimize the distribution π ( z|φ ) that generates these random variables . For the ease of exposition , we first focus on the relaxation of the non-adaptive formulation in ( 1 ) as follows : min θ , φ E ( xi , yi ) ∼D [ Ez∼π ( z|φ ) [ L ( f ( xi ◦ z ; θ ) , yi ) + λ‖z‖0 ] ] . ( 4 ) Note that the solution to this problem is equivalent to the original one , as the original combinatorial problem can be recovered by setting π ( z|φ ) = Bern ( φ ) , a Bernoulli distribution parameterized by φ , and restricting φ ∈ { 0 , 1 } . Using this relaxation , the regularization term can now be evaluated analytically : Ez∼π ( z|φ ) [ ‖z‖0 ] = Ez∼Bern ( φ ) [ ‖z‖0 ] = P∑ j=1 π ( z|φ ) j = P∑ j=1 φj , ( 5 ) On the other hand , the outer expectation in ( 4 ) can be approximated using minibatches . Relaxation of binary random variables has been adopted in Louizos et al . ( 2017 ) for network architecture sparsification , and in Yamada et al . ( 2019 ) ; Balın et al . ( 2019 ) for static feature selection . Here , we extend the above relaxation for time series data , where unlike previous works , the binary random variables are parameterized locally and are context-dependent , and features are selected adaptively across time . We first note that our adaptive feature selection formulation in ( 3 ) allows each time point to have its own feature selection distribution πti ( z|φ ) , π ( z|X t−1 i , φ ) conditioned on previously selected observed features X t−1i as defined above . Let πi ( z|φ ) be the set of πti ( z|φ ) for all t ∈ { 1 , . . . , Ti } . The stochastic relaxation of the adaptive feature selection formulation can be written as follows : min θ , φ E ( xi , yi ) ∼D [ Ezi∼πi ( z|φ ) [ Ti∑ t=1 L ( f ( X t−1i ; θ ) , y t i ) ] + λ Ti∑ t=1 P∑ j=1 πti ( z|φ ) j ] . ( 6 )

Summary: This paper proposes an RNN model for adaptive dynamic feature selection, for efficient and interpretable human activity recognition (HAR). From the intuition that human activity can be predictable by using a small number of sensors, the paper introduces an l0-norm minimization problem with parameter regularization, and provide a logic on formulating a dynamic feature selection model with relaxations. The difficulty of the discrete optimization problem is solved by differentiable relaxation, which is known as Gumbel-Softmax reparameterization techniques. The formulation is naturally led to an RNN model that uses histories as input with an additional sigmoid unit for adaptive feature selection.