# Sensors, Techniques, and Future Trends of Human-Engagement-Enabled Applications: A Review
## 3 Sensors
### Cameras for Visual Surveillance  
For the past decades, camera technologies experienced rapid advancement, especially in terms of digitization. Low cost, small form factor, high resolution and frame rate, and reliability all promote the utility of cameras in a great diversity of devices and ubiquitous deployment in manifold applications. Coupled with a surge in computer vision and Machine Learning techniques, cameras have become one of the most widely used sensor modalities for scene understanding and monitoring of human activities. Traditionally, cameras refer to photo sensors capturing electromagnetic signals in the spectrum visible by humans. Nowadays, technological advances allow cameras to go beyond the visible spectrum. Infrared (IR) cameras capture data in the infrared spectrum, which is just beyond visible [53]. Thermal cameras can detect heat radiation from objects and generate thermal temperature imaging [54], which is helpful to reveal a human's physiological state. RGB-D (Depth) cameras can provide both RGB imaging and depth information [55], allowing the extraction of distance and shape information. Multispectral cameras capture images across multiple wavelengths. Moreover, event-based cameras, also known as neuromorphic cameras, can adapt and respond to brightness changes at pixel level. Tracking learner engagement with a webcam, a simple RGB camera fitted to a personal computer, produces fruitful results, including large-scale datasets: HBCU [56], DAiSEE [57], EngageWild [58], and EngageNet [59]. Some engagement estimation paradigms rely on extracting facial features, e.g., with OpenFace [60], Dlib [61], etc., to classify emotion prior to deriving engagement level. Popular datasets include AffectNet [62], CK+ [63], and FER2013 [64]. Many studies delve into driver drowsiness and fatigue detection, and also disengagement detection, to facilitate autonomous driving safety assistive systems with in-car cameras [65; 66; 37; 67]. Datasets like YawDD [68], NTHU-DDD [69], and Fatigue-View [38] are broadly adopted in performance benchmarking. Cameras are also widely adopted in other fields such as child engagement analysis [70; 71], ADHD research [72; 73], emotion recognition [62; 63; 64], social robotics design [34; 74], and HRI [13; 14; 15]. Details of the aforementioned datasets are summarized in Table 2.  
Cameras can capture a wide variety of human engagement cues, as can be seen from Table 3. From Facial Action Units (FAUs) revealing human emotion to body posture and gesture, they can capture almost all appearance-based behaviours if placed at the right position. Taking eye tracking for example, gaze and blinks can be computed given a reasonably good resolution at the cropped eye region in the visual feed [75]. Pin-pointing a fixated object in space is possible with cameras if a 3D mapping of the environment is available [32; 76]. Computer-vision-aided head pose estimation has realized superior robustness [60; 77]. In fact, many researchers leveraged vision-based methods to label eye state [14; 70; 78] due to unavailability of eye trackers. Nonverbal communication plays a pivotal role in engagement characterization, and cameras excel at harnessing these personal and sociological cues. Not only can cameras capture body posture, e.g., skeleton-based keypoints [18], but they can detect subtle hand gestures [79], micro-expressions (including involuntary emotional responses) [80], and lip motions which tell who is addressing and controlling the floor. [PERSON] et al. [81] proposed a vision-based method to decode speakers' utterances at sentence level. Note that with calibration or depth-sensing ability, a camera can infer distances between agents. Distance is a vital feature correlating to engagement, especially in HRI [82]. Beyond traditional RGB cameras, applications of RGB-D and thermal cameras are starting to appear. [PERSON] [83] demonstrates the efficacy of using depth information for facial expression estimation.