# Joint Learning of Temporal Models to Handle Imbalanced Data for Human Activity Recognition
## 4 Experimental Setup and Evaluation
### Model Interpretability  
In this section, permutation feature importance (PFI) as a useful mechanism of model interpretability [58; 59; 60] is conducted to show more insight of each LSTM and 1D CNN independently. PFI is used to compute and rank input feature importance from Ordonez smart home A and B datasets based on how useful feature are at HAR for LSTM and 1D CNN. PFI reflects how each predictor variable is important from HAR for each LSTM and 1D CNN. Experiments that are based on PFI show that the most and least important features from both datasets are different in HAR for LSTM and 1D CNN. This indicates the most important features in recognition process varies between LSTM and 1D CNN, which has been considered to take advantage of most important features in both models in the proposed joint learning. Hence, the proposed joint temporal model exposes different features from input data into the joint learning to improve the performance of HAR particularly minority classes given the different rank of the important features from both models. The rank of features in LSTM and 1D CNN based on the PFI is different, which is indicative of how much each LSTM and 1D CNN models relies on the features. This illustrates that the joint proposed model takes feature importance in both models into account in the joint learning. Therefore, the joint learning can use a set of the most important features from one model and a different set from the other model to contribute in the recognition to generally improve the performance of HAR particularly minority classes. Algorithm 2 designed based on [60; 61] shows the process of PFI in detail. Firstly, the result scores, i.e., F1-score of each LSTM and 1D CNN, are computed. Next, a single feature from the dataset is randomly shuffled to generated a permuted version of the dataset. This mechanism removes the relationship between the features and the true labels. Subsequently, the LSTM and 1D CNN models are independently applied on the permuted version of the dataset to compute the result score. Finally, we subtract the result score based on permuted data from the result score of the original data. This mechanism measures features' importance by computing the error of the LSTM and 1D CNN after permuting the feature. Moreover, after applying permutation on a feature, the decrease of the f1-score indicates the model depedency on the permuted feature. This mechanism is applied on all of the features to compute feature importance. Tables 7 and 8 show PFI on the Ordonez smart homes A and B datasets and the rank of the importance of each feature. Further, the value of mean and standard deviation of the F1-score for \(N\) runs after permuting the feature are presented. Figures 10 and 11 show the mean results of f1-score with \(N=12\) run for 12 sensors after permuting the sensor features. Figure 9: Joint learning compared with joint LSTM+LSTM and Joint 1D CNN+1D CNN. ```
1:Input: Train model \(M\) on original dataset \(D\), label vector \(\Upsilon\), error measure L(\(\Upsilon\),\(\hat{\Upsilon}\)) (original datasets is the dataset without permutation \(\hat{Y}\) is the predicted label)
2: Compute result score \(RS_{original}(\hat{M})=L(\Upsilon,\hat{M}(D))\) (e.g., F1-score )
3:for each feature \(j\) from \(D\)do
4:for each repetition \(n=1\) to \(N\)do
5: feature permutation on \(D\) to generate \(\hat{D}_{n,j}\) (This removes the relationship between \(D_{j}\) and \(\Upsilon\))
6: Compute result score \(RS_{permuted,nj}(\hat{M})=L(Y,\hat{M}(D_{n,j}))\) (e.g., F1-score on the permuted data)
7:endfor
8: Compute feature importance \(i_{j}\) for feature \(j\), \(i_{j}=RS_{original}(\hat{M})-\frac{1}{n}\sum\limits_{n=1}^{N}RS_{permuted,nj}( \hat{M})\)
9:endfor
10:Output: feature importance \(i\) for all the features
```  
**Algorithm 2**Compute Permutation Feature Importance (PFI)