# Deep-cARe: Projection-Based Home Care Augmented Reality System with Deep Learning for Elderly
## 3 Material and Methods
### Deep-Care System Implementation
#### 3.1.4 Deep-Learning Module  
Realism and immersion can be ensured by providing virtual information to the real space through PAR technology. However, there are several challenges in providing a PAR system to the elderly. There are many difficulties, deep-learning frameworks are applied. This enables recognition of the current user state, identification, and real-world object. Furthermore, external information can be subsequently used. 1. Pose estimation To facilitate monitoring system provision and perform projection on the optimal plane, the user location should be recognized in real time. To estimate the location and pose of the user, PoseNet [29] was used, which is an open-source deep-learning model. PoseNet can track joints and uses 2D images as input. The recognized joint information is converted into 2D coordinates, and the z-axis value (depth information) corresponding to the 2D coordinates is obtained through the depth camera. There are two methods for recognizing specific user states such as falling and tripping using the obtained joint information. First, a study on action recognition in 3D video [30] and a study on skeleton extraction [31] implemented a deep-learning model that  
Figure 4: (**a**) 3D space map reconstruction (**b**) Application registration in optimal planes (red box). can recognize falling. However, to recognize motions, it can be challenging to crop only the target motion and use it as an input. Therefore, although motions can be recognized with high accuracy in a laboratory environment, it can be difficult to apply the model immediately to a real environment. In this study, the proposed system was constructed using a second method -rule-based method-according to the joint information detected by the PoseNet deep-learning model. The rule-based method can configure the system by defining cases where the user's joints are in unusual states. Abnormal states can be recognized by the defined states of the joints and are used for urgent situation alerts such as when the head is lower than other body parts and remains in that state or when most joints stay near the floor (Figure 5 shows that rule-based user state). An abnormal state is identified if the location of the user is the living room or kitchen rather than the bedroom where lying down occurs (Figure 5 shows that space recognition in user). After distinguishing an abnormal state, an alert is sent to the family or rescue team. However, the accuracy of the user's state recognition can be low only with the joint information detected based on the vision information input through the RGB-D depth camera. To prevent misreporting because of erroneous recognition, the accurate state is determined through the feedback UI response of the user after the abnormal state is recognized (Figure 5 shows that user feedback check state)). 2. Face recognition Ordinary elderly people do not suffer severe memory decline, as in dementia or Alzheimer's disease but their ability to identify faces is deteriorated [32]. In particular, they have difficulty identifying people quickly. Erroneous identification of people can result in omission of visiting benefits and delivery services for elderly welfare and may lead to home invasion crimes. To prevent this and support long-term memory, deep learning is applied to the face recognition module. The deep-learning module used was FaceNet [33]. This model can operate at 12 fps near real-time face recognition. An intercom environment was set up to support remote door opening and closing in this scenario. Visitor identification is performed in real time using an image input to the RGB camera attached to the intercom. The images of the faces of likely visitors, such as family, acquaintances, and hospital personnel, are labeled and stored in advance. For previous irregular visitors, such as delivery personnel, the face images are saved as unlabeled data. The face recognition results are largely classified into three types.