# Sensors, Techniques, and Future Trends of Human-Engagement-Enabled Applications: A Review
## 5 Metrics  
Metrics play an important role in judging the efficacy of engagement estimators. Comparing estimation results with human performance is a usual means both in performance benchmarking and validation of annotation. [PERSON]'s kappa coefficient, \(\kappa\), is used in several works [13; 56; 57; 58] for this purpose.  
Automatic engagement estimation is regarded as either a classification or a regression task [3] in the vast majority of the literature. Therefore, performance metrics of classification and regression can be used accordingly. For classification, almost all classification-related performance metrics can be applied, including accuracy, precision, recall, F1-score, specificity, sensitivity, Receiver Operating Characteristics (ROCs), Area Under the Curve (AUC), and precision-recall AUC. A confusion matrix can be a useful tool to visualize categorical accuracy and mismatching.  
There are different opinions on engagement classes, which can be a major issue in cross-validation and user acceptance. The UE-HRI dataset [78] specifies four engagement labels: _Sign of Engagement Decrease_ (SED), _Early sign of future engagement BreakDown_ (EBD), _engagement BreakDown_ (BD), and _Temporary Disengagement_ (TD). [PERSON] et al. [56] used _Not engaged at all_, _Nominally engaged_, _Engaged in task_, and _Very engaged_ in their work. The authors of [58; 59] concurred on four levels of engagement: _Not Engaged_, _Barely Engaged_, and _Highly Engaged_. [PERSON] et al. [118] defined six shades of engagement, from _No interest_ up to _Governing/managing discussion_, in their multi-party conversation study. Four states are used in the DAiSEE dataset [57]: _boredom_, _confusion_, _engagement_, and _frustration_, which mix up engagement with sentiments. This is understandable in e-learning scenarios but does not apply to other applications. These distinct labelling schemes are not only difficult to relate to but barely verifiable in cognitive science. Moreover, they  
Figure 2: Three typical DNN architectures for human engagement classification or regression. **Left** takes as input a video containing multiple frames and applies CNN\(+\)LSTM in conjunction with fully connected layers for engagement output, such as [14; 57]. **Top right** evaluates engagement from a single frame with CNN\(+\)MLP as is tested in [131; 138]. **Bottom right** curates a feature vector concatenating sole or multimodal features before learning hidden states with Recurrent Neural Networks (RNNs), such as GRU and LSTM in [13].  
tend to be mutually exclusive and vary from work to work making it harder to adopt, evaluate, and deploy them.  
Formulated as a regression task, engagement levels are often estimated as a continuous value between 0 and 1, such as in [14; 82]. Metrics used for regression include Mean Absolute Error (MAE), MSE, and Root Mean Squared Error (RMSE), which compute the distance from the predicted value to the ground truth. A major underlying problem of using a floating number is its explainability. It is not clear what it means to be, e.g., 50% engaged. Whether human engagement can be linearly projected is uncertain. Similar to those boundary-blurred classification categories, a value between 0 and 1 lacks cognitive grounding to inform real-world design.