# Human Psychophysiological Activity Estimation Based on Smartphone Camera and Wearable Electronics
## 4 Meditation Estimation Approach
### Meditation Estimation Based on Skeleton Detection  
We propose an approach to analyze the meditation process that involves an optical flow utilization to detect the tiniest movement of individual parts of the human body marked with key points on the human skeleton. Optical flow is a vector field showing the movement of each pixel between  
Figure 16: First results of a neural network. Graph (**a**) shows the accuracy of the model prediction. Graph (**b**) shows the value of the loss function. Figure 17: Peak accuracy of neural network predictions (**a**) and loss function (**b**). Figure 18: Examples of meditation pose recognition based on a developed model. two frames. We evaluate the frequency and amplitude of respiration, stoop over time, displacement, and body vibration from side to side by analyzing the optical flow. Furthermore, this information can be used to assess the quality of the meditation technique and identify common mistakes. We chose a section of the meditation video (see Section 4.2) where a person fixes their meditation pose. If a person changes the pose, we implement the new video section analysis. For the first frame of the appropriate video section, the human bounding box is detecting with a convolutional neural network [35], which is based on Mask-RCNN [36] and showed one of the best results on the COCO Detection 2017 (47.7 mAP) and COCO Segmentation 2017 (41.7 mAP) datasets [37]. Then, in the bounding box, the key points of the human body are evaluated using a separate neural network [38], which took second place in the competition associated with the Human 3.6 M dataset [39]. This neural network builds a skeleton of a human and marks key points on the intersection of human body parts. The following key points of the upper part of the body are taken for further analysis: nose, thorax, mid-torso (torso), shoulders, elbows, and wrists (see Figure 19). Then, between the first frame of the video section and all subsequent ones, the optical flow (pixel displacement vectors) is calculated. For this purpose, a separate neural network is used [40], which showed one of the best results on the MPI Sintel [41] dataset (2.040 EPE matched). By calculating the optical flow relative to the first frame, a shift in the human body from the start of meditation is obtained. The graph constructed on the basis of such an optical flow is better visually perceived than the graph of the displacement relative to the previous frame, and the detection of key points of the body is necessary only for the first frame. Potentially, though, the bias can be large (for example, if a person is very stooped); this can affect the accuracy of some algorithms, including the widespread Farneback algorithm [42], but the neural network-based approach [40] that we used is resistant to strong displacements. Such a graph can also be obtained by summing the displacement vectors between pairs of adjacent frames, but summing the individual optical fluxes can lead to an accumulation of error, and it would also require a recalculation of the position of the key points. Then, after calculating the optical flow and the position of the key points of the human skeleton, three motion graphs are formed for each key point and the whole body. These graphs show the displacement of the key point (or the whole body) along the X axis (left/right), the Y axis (up/down), and the absolute length of the displacement vector (takes into account both components). In Y, movement during breathing is better observed; in X, extraneous unwanted body vibrations are detected. Absolute undesirable moving of key points over time can be detected. As a result, 30 charts were received (see Figure 20). Figure 19: Human movement estimation based on skeleton detection. For motion graphs formation of the whole body, the displacement vectors of all the pixels included in the bounding box of a person are averaged.