## 3 Human-Robot Interaction Application for the Reinforcement of ADLs
### Human-Robot Interaction Application: Software Description
#### 3.2.3 AI for Online Action Monitoring of ADLs  
Considering the need to monitor the activity of users, the inclusion of an action detection system in our robot was a no-brainer. As an standard action detection system would require that an entire video of the action be fed into the neural network to recognize the action correctly, we believe that these models are ineligible for monitoring actions in this project. They work offline, in the sense that the action of interest must be completely covered by the video. Instead, for our robotic application, we need an action recognition system that is able to work with partial observations of the actions, or, in other words: to address the problem of localizing actions in untrimmed videos as soon as they happen, which was coined as Online Action Detection (OAD). Therefore, action detections in an OAD technique must be made over video streams, requiring the use of incomplete observations in which the action segments may be more likely to be the exception than the rule relative to the background. Furthermore, this online setting includes a crucial element: the anticipation of the action. For an OAD model, the objective is to anticipate the action even before the action is fully completed, an aspect that is fundamental to attaining an efficient human-robot interaction in the monitoring application designed for this project: LOLA must be able to recognize the actions the users are performing in an online fashion, so as to provide a real-time response. For our robotic platform, we propose a Pytorch implementation of the 3D CNN model described in [72]. See Figure 6 for details of the deep learning architecture deployed on the NVIDIA Jetson Xavier board. Our approach uses a 3D Convolutional Neural Network (CNN) that is trained on the UCF-101 dataset containing 101 distinct categories of actions [73]. As input, our OAD model is given a 16 frame clip. The model is able to cast an action prediction for every clip, classifying the actions as soon as they appear. Using the Jetson board and a live video stream received from the robot's camera, this method produces action estimations at a rate of more than 5 frames per second. Figure 5: For this work, we have built a database with videos containing actors that execute, step by step, a set of actions of daily life that we want to work with patients. These images contain some captures where we can observe two actors performing two different actions. (**a**) The actor is showing the action of writing on the board. (**b**) The actor is showing the action of mopping the floor. Considering that the robot's camera is pointing at users, a recommended distance between users and the robot, in this case, is 1.5-2 m. The actions have been split into two groups according to whether the face needs to be viewed or whether the body needs to be viewed, as it is shown in Table 1. The option of detecting the face and zooming on it is activated or not based on the requested action by the professionals. Whenever the face is required, we have implemented a traditional face detection model [74] with python and the OpenCV library [75]. Technically, our solution performs tracking of the user's face over the captured 16 frames, creates a specific clip with the face only, and feeds it to the OAD model for the recognition of the action category. Figure 7 graphically shows how the implemented pipeline works. In the case of actions that are performed via body movements, the original video frames are prepared for feeding the OAD model. \begin{table}
\begin{tabular}{l c} \hline \hline
**Action Category** & **With Zoom/ Without Zoom** \\ \hline Blow Dry Hair & With zoom \\ Brushing Teeth & With zoom \\ Writing On Board & Without zoom \\ Mopping Floor & Without zoom \\ \hline \hline \end{tabular}
\end{table} 
Table 1: The four categories of actions and their status regarding zooming. Figure 6: This figure illustrates the OAD-3D-CNN model that is used to monitor patients’ daily living activities.