# Sensors, Techniques, and Future Trends of Human-Engagement-Enabled Applications: A Review
## 4 Approaches
### Data-Driven Approaches  
Data-driven approaches stem from the need for better accuracy and robustness of engagement estimation across different application scenarios. This process is highly coupled with the blossoming of artificial intelligence (AI) in the past decade. AI-based techniques have dramatically boosted applications in various sectors, including business, education, agriculture, transportation, gaming, and robotics. Affective computing is amongst the main beneficiaries. As a result, understanding human _behavioural_, _emotional_, and _cognitive_ status from raw sensory data is getting easier and more reliable [3]. Engagement estimation can be formulated as a frame-level classification [131] or regression [14] problem or a time series prediction problem [78] by taking as input a set of features over an elapsed time window, as shown in Figure 2. They can be consolidated with input \(X_{t-\tau,t}\), where \(t\) is the current time and \(\tau\) is the observation window length. Engagement is represented as \(y_{t}\). Thus, engagement estimation is expressed as:  
\[y_{t}=\Theta([x_{t-\tau},\ldots,x_{t-2},x_{t-1},x_{t}]) \tag{1}\]  
where \(\Theta\) is the learning-based model containing learnable parameters. In a frame-level classification or regression, \(\tau\) equals 0. A plethora of research adopted handcrafted features with machine learning algorithms to classify engagement state or evaluate engagement level. These handcrafted features include, but are not limited to, facial keypoints of FAUs, head pose, posture, and other nonverbal signs of communication. There are open-source toolkits to help to extract these features, namely Dlib [132], OpenFace [60], OpenPose [18], etc. Support Vector Machines (SVMs) [37; 82; 133] are amongst the most commonly used algorithms. They prove effective in classifying multiparty video conversation [118] realizing a 74% accuracy in six-categorical engagement classification. [PERSON] et al. [56] investigated SVMs, boosting, and Multinomial Logistic Regression (MLR) for engagement classification. All three methods show over 90% accuracy testing on the HBCU dataset. [PERSON] et al. [58] utilized facial and head pose features of videos and compared SVR, Random Forest, and Long Short-Term Memory (LSTM) for engagement regression. They found Random Forest and LSTM are equally competitive, with 0.09 Mean Squared Error (MSE), superior to 0.15 of SVR. [PERSON] et al. [130] noted that SVM, Nearest-Neighbour, Decision Trees, and logistic regression delivered indistinguishable performance in their experiments taking as input facial expression, gaze, and posture of human. Hidden Markov Models act as the foundation of many social interaction models underpinning a wide variety of applications [134; 135]. This extends to multi-party conversational and behavioural engagement modelling [136; 137]. Deep Learning refers to those using Neural Networks (NNs) for representation learning instead of handcrafted features. Deep Learning has the power to learn latent features from raw sensory data such as video, physiological measurement, and sound wave. This gives DNNs an edge over other methods, as a robust representation can be learned in the feature space. This in turn enables transfer learning to exploit pre-trained facial and appearance-based feature extractors for enhanced generalization. [PERSON] et al. [70] contrasted SVMs and Random Forest to their proposed LSTM model for engagement estimation of child-robot interaction. They found that the LSTM model outperformed the other classic machine learning algorithms. [PERSON] et al. [131] presented a bespoke Convolutional Neural Network (CNN) model achieving accurate eye contact detection comparable to human experts. [PERSON] and [PERSON] [16] backed up this finding in comparing CNN\(+\)MLP (Multi-Layer Perceptron) to Generative Adversarial Network (GAN) models. They realized over 80% accuracy on two challenging datasets. [PERSON] et al.