# Deep-cARe: Projection-Based Home Care Augmented Reality System with Deep Learning for Elderly
## 3 Material and Methods
### Pose Estimation Performance  
The joint information and user location can be derived through pose estimation. For immediate use of this application in a real-world environment, an experiment was conducted to measure the estimation performance based on the occlusion of the joint information. The input images of a single user were used. Considering four states: \"stand,\" \"seat,\" \"hide,\" and \"lie,\" 100 frames (25 \(\,\mathrm{fps}\)\(\times\) 4 \(\,\mathrm{s}\)) were used according to each state. The information on 17 joints was estimated. Based on the input images, the ground truth for the joint information was set in advance for each frame. To calculate the precision of the prediction, the percentage of correct keypoints head (PCKh) [36] evaluation metric was used. Based on [EMAIL_ADDRESS], the standard threshold was set to half of the head diameter (the distance between the left and right ears).  
First, in the \"stand\" state in Figure 6, which was the most normal case, at \(r=0.5\), the average precision of all joints was 86.98%. As shown in Figure 6b, the head part exhibited higher precision as the threshold range increased. However, \"leftWrist\" and \"rightHip,\" which were occluded by the user's clothes and other body parts, exhibited low accuracy. Therefore, it was determined that they are highly vulnerable to occlusions.  
Subsequently, the \"seat\" state (Figure 7) demonstrated high precision with an overall precision of 87.47%. In addition, for the \"seat\" state, the face part exhibited high precision because the front side was directly faced, and the precision of the upper body joint detection was 95.65%, which indicates that the joint information was estimated accurately and stably. However, for the hip part, which was occluded by an arm and contact with the sofa, the average precision was 13.04%. Nevertheless, the lower body excluding the hip had the highest precision 100%.  
The \"hide\" state (Figure 8) was applied for performance evaluation in terms of physical occlusions. The average precision of the estimated joints was 93.13%, which is high, although the lower body and left wrist were not estimated at all. Despite the application of deep-learning model, difficulty in joint estimation was experienced because owing to physical occlusions. A \"lie\" state (Figure 9) was  
Figure 6: “Stand” pose estimation. (**a**) Ground truth joint (red point) and PoseNet predicted joint (green point). (**b**) Head part accuracy. (**c**) Upper body accuracy. (**d**) Lower body accuracy.  
Figure 7: “Seat” pose estimation. (**a**) Ground truth joint (red point) and PoseNet predicted joint (green point). (**b**) Head part accuracy. (**c**) Upper body accuracy. (**d**) Lower body accuracy.  
used for pose estimation performance evaluation considering falls that occur among elderly people. However, the results demonstrated that only the face and upper body were estimated. This was because although the pose estimation itself failed for the test subject, the state of no movement was continuously input. Nevertheless, it appears that the joint information can be estimated with high precision through a series of movements.  
By using experiments based on the user state, it was confirmed that the joint estimation for the facial part exhibited high precision. Specifically, the facial part joint that is close to the floor surface or an unusual location is used to determine an abnormal state. The overall observation results of pose estimation using PoseNet in a real-world environment exhibited sensitivity to occlusions, confirming that when a user moves quickly, tracking the detected point cannot be performed immediately. However, when all joint information, namely that of the front and rear sides, was accurately input, high precision was observed. Furthermore, because the estimation precision of the face is particularly high, it is used to determine an abnormal user state.