Patent Publication Number: US-10779739-B2

Title: Contactless-type sport training monitor method

Description:
BACKGROUND OF THE INVENTION 
     1. Field of the Invention 
     The present invention relates to a contactless-type sport training monitor method, particularly to a contactless-type sport training monitor method by using an image method to monitor the heart rate and the facial expression, in order to judge integratedly whether the human body is under overtraining status or not. 
     2. Description of the Prior Art 
     A successful training program aims is to increase athletes&#39; or exercisers&#39; fitness and endurance. This is achieved by the application of the appropriate training protocols for athletes to reach high peak performance status during the competition season. However, overtraining may lead to fatigue, and limit athletes&#39; performance. Sometimes the fatigue may even take the athletes several weeks to recover from. At this time, adjusting the training program becomes a very important topic. 
     Overtraining is characterized by imbalance between stress and recovery after exercising, and the syndrome is characterized by fatigue and low performance. At past, physicians, psychologists and physical therapists have tried to reach diagnosis and consensus regarding overtraining syndrome. The overtraining can be diagnosed by physiological, psychological and behavioral, neuro endocrinal and biochemical indicator, and immunological indicator etc. 
     The athletes&#39; fitness and endurance should be considered during training program. Many clinical decision support systems can assist to monitor the factors of physiological signal, emotion, or the level of fatigue of the athletes&#39; by some wearable devices. In addition, the questionnaire should also be taken into account to produce an assessment report. Such process is cumbersome, and the results are not objective. Furthermore, the person may feel uncomfortable when wearing the devices during the training program. 
     As the abovementioned description, the people expect to estimate the Rating of Perceived Exertion (RPE) without any wearable devices and questionnaires. A camera based heart rate detection algorithm and a fatigue expression feature extractor are fused to estimate the contactless-type Rating of Perceived Exertion (RPE) value. The results show that the heart rate detection algorithm and the fatigue heart rate detection can obtain the similar results of wearable devices. It means the contactless-type sport training monitor method has certain competitiveness. 
     Heart rate and fatigue facial expression are regarded as the index of overtraining. An image based overtraining monitoring system is proposed so that the training Rating of Perceived Exertion (RPE) level of the athlete can be estimated without any wearable devices. Since the questionnaire is not considered, the results would more objective. 
     To address the abovementioned problems, it is necessary to develop an appropriate method to achieve contactless-type monitoring heart rate and facial expression without any wearable devices attached on the athletes&#39; body. This contactless-type method shall also be able to judge whether the athlete is under overtraining status or not by monitoring the fatigue, endurance and performance. The suitable measures shall be adopted to avoid the occurrence of overtraining status. 
     SUMMARY OF THE INVENTION 
     The embodiment of the present invention provides a contactless-type sport training monitor method, comprising: selecting at least an image database to recognize a plurality of expressions in the image database, respectively, in order to train a deep level model; making pre-processing for the plurality of expressions, using a convolutional neural network as a feature point extraction model in order to carry out the feature fusion for the plurality of expressions, in order to extract an expression feature, according to the expression feature to analyze the plurality of expressions as a facial expression image, and store in the deep level model; acquiring a human image; tracking a first target region and a second target region in the specific human facial image, and acquiring a first target region image in the first target region and a second target region image in the second target region, respectively; making chrominance-based rPPG (CHROM rPPG) trace extraction for the first target region image, making Fourier transform for the first target region image; making Peak Selection for the first target region image, to obtain a heart rate value (HR); using the deep level, model to compare the second target region image, to judge the second target region image as a specific facial expression image; and according to the specific expression to judge the heart rate value is a resting heart rate or an exercising heart rate and a lowest heart rate, to calculate a post-exercise heart rate recovery achievement ratio (ΔHRR), according to the post-exercise heart rate recovery achievement ratio to judge the Rating of Perceived Exertion (RPE), and judging whether the human body is under overtraining status or not. 
     In the preferred embodiment, the image database is YouTube, RaFD or ADFES. 
     In the preferred embodiment, training the deep level model includes using the blurring, sharpening, lightened and darkening treatment step to augment the plurality of expressions. 
     In the preferred embodiment, the dimension of the feature point extraction model is 128. 
     In the preferred embodiment, the facial expression image is selected from the group consisting of anger, disgust, fear, happiness, sadness or surprise. 
     In the preferred embodiment, first target region image is the palm skin image or face skin image of the human image. 
     In the preferred embodiment, the second target region image is the specific facial expression image. 
     In the preferred embodiment, the Peak Selection steps comprise a motion noise spectrum stability value is lower than a stability threshold, or the difference between largest frequency and motion frequency is smaller than a frequency threshold. 
     In the preferred embodiment, the stability threshold is −11 bpm, and the frequency threshold is 4.5 bpm. 
     In the preferred embodiment, the motion noise spectrum stability value is 
               10   ⁢     log   ⁡     [       E   1       E   2       ]         ,         
wherein, E 1  is the largest frequency energy, and E 2  is the total signal energy minus E 1 .
 
     In the preferred embodiment, the post-exercise heart rate recovery achievement ratio is 
                   E   ⁡     [     HR   ex     ]       -     HRR   lowest           E   ⁡     [     HR   ex     ]       -     E   ⁡     [     HR   rest     ]           ,         
wherein, HRR lowest  is the lowest heart rate value, E[HR ex ] is the average exercising heart rate value, and HR rest  is the average resting heart rate value.
 
     In order to further understand the features and technological content of the present invention, please refer to the following detailed description and attached figures of the present invention. Nevertheless, the attached figures are used for reference and description, which are not used for limiting the present invention. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The foregoing aspects and many of the attendant advantages of this invention will become more readily appreciated as the same becomes better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings, wherein: 
         FIG. 1  illustrates a flowchart of contactless-type sport training monitor method of the present invention; 
         FIG. 2  illustrates a contactless-type sport training monitor device of the present invention; 
         FIG. 3A  illustrates the heart rate spectrum with high spectrum stability of the present invention; and 
         FIG. 3B  illustrates the heart rate spectrum with low spectrum stability of the present invention. 
     
    
    
     DESCRIPTION OF THE PREFERRED EMBODIMENT 
     In the following context, the specific embodiments are used to describe the contactless-type sport training monitor method of the present invention. The people who are familiar to this art can understand the advantages and efficacies of the present invention easily from the content disclosed in this article. The present invention can also be implemented or applied by other different embodiments. Every detail in this article can also be modified and changed based on different viewpoints and applications without violating the spirit of the present invention. In addition, the figures in the present invention are only brief description, and they are not drawn in actual dimension to reflect the actual size. The following description of preferred embodiment describes the viewpoint of the present invention in more detail, which will not limit the scope of the present invention by any viewpoint. 
     Please refer to  FIG. 1  and  FIG. 2 .  FIG. 1  illustrates a flowchart of contactless-type sport training monitor method of the present invention.  FIG. 2  illustrates a contactless-type sport training monitor device of the present invention. The contactless-type sport training monitor method of the present invention shown in  FIG. 1  comprises the following steps: 
     Firstly, as shown in Step S 102  of  FIG. 1 , selecting an image database  204  to recognize a plurality of expressions in the image database  204 , respectively, in order to train a facial expression feature extraction model. It has to describe that the image database is YouTube, RaFD or ADFES. The image database is used to train a facial expression feature extraction model. The videos are downloaded from the image database, which are cropped with the time interval set to 1, 2 or 3 seconds to produce the facial images. The face images would be filtered out if it is non-face, two-face, or too blurred, or too dark, or the expressions which are hard to be identified. In addition, because the environment condition of YouTube image database is quite bad, the RaFD or ADFES image database is added to train the image database of deep level model in the embodiment. The training images of deep level model would be augmented by blurring, sharpening, lightened and darkening. 
     Please also refer to  FIG. 3 .  FIG. 3  illustrates the Neighbor-center difference image method of the present invention. As shown in  FIG. 3 , it is able to increase the edge efficiency of the plurality of first expression P 1 . 
     As shown in Step S 104  of  FIG. 1  and also refer to  FIG. 2 , a convolutional neural network feature paint is used to extract model  208  to obtain an expression feature from expression P. According to the expression feature, analyze the plurality of expressions as a facial expression image, and store in the facial expression feature extraction model  206  in  FIG. 2 . As an example in  FIG. 3 , when the feature point extraction model  208  analyze and judge the expression feature extracted from expression P as a happy facial expression image, the happy facial expression image is stored in the facial expression feature extraction model  206 . It has to note that the dimension of feature point extraction model is 128. The facial expression image is selected from the group consisting of anger, disgust, fear, happiness, sadness or surprise. 
     Still as shown in Step S 106  of  FIG. 1 , the image device  202  of  FIG. 2  is used to extract the human image I of human body  220 . 
     As shown in Step S 108  of  FIG. 1 , tracking the first target region T 1  and the second target region T 2  in the human image I, and acquiring the first target region image I 1  of the first target region T 1 , and the second target region image I 2  of the second target region T 2  respectively, as shown in  FIG. 2 . It has to note that the first target region image I 1  is the palm skin image or facial skin image of human image I. The second target region image I 2  is the specific facial expression image, as shown in  FIG. 2 . 
     As shown in Step S 110  of  FIG. 1 , making the chrominance-based rPPG (CHROM rPPG) trace extraction for the first target region image I 1 . The CHROM rPPG trace extraction is passing an assumed standard skin color to enhance the robustness of motions and eliminate the noise generated in the motion (such as swing) domain. 
     As shown in Step S 112  of  FIG. 1 , making Fast Fourier Transform (FFT) for the first target region image I 1  to transform to the domain. 
     As shown in Step S 114  of  FIG. 1 , making Peak Selection for the first target region image I 1 , to obtain a heart rate value HR. The Peak Selection is used to reduce the motion noise, so a Spectrum Stability (SS) is defined as: 
               SS   =     10   ⁢     log   ⁡     [       E   1       E   2       ]           ,         
Wherein, E 1  is the energy of the largest frequency, and E 2  is the total signal energy minus E 1 .
 
     As shown in the embodiment of  FIG. 1 , the Peak Selection Step includes the spectrum stability value of motion noise is lower than a stability threshold T h , or the difference between largest frequency f h  and motion frequency f ex  smaller than a frequency threshold T f . 
     As shown in the embodiment of  FIG. 1 , the bandwidth of stability threshold T h  is −11 bpm, and the bandwidth of frequency threshold T f  is 4.5 bpm. 
     Please refer to  FIG. 3A  and  FIG. 3B .  FIG. 3A  illustrates the heart rate spectrum with high spectrum stability of the present invention.  FIG. 3B  illustrates the heart rate spectrum with low spectrum stability of the present invention. 
     As shown in Step S 116  of  FIG. 1 , the deep level model is used to compare the second target region image I 2 , to judge the second target region image I 2  is a specific expression. 
     As shown in Step S 118  of  FIG. 1 , according to the specific expression to judge whether the heart rate value HR is resting heart rate HR rest , or exercising heart rate HR ex , and lowest heart rate HRR lowest , to calculate the post-exercise heart rate recovery achievement ratio ΔHRR. According to the post-exercise heart rate recovery achievement ratio ΔHRR to judge the Rating of Perceived Exertion (RPE), and judge whether the human body is under the overtraining status or not. The post-exercise heart rate recovery achievement ratio ΔHRR is defined as: 
               Δ   ⁢           ⁢   HRR     =         E   ⁡     [     HR   ex     ]       -     HRR   lowest           E   ⁡     [     HR   ex     ]       -     E   ⁡     [     HR   rest     ]                 
Wherein, HRR lowest  is the lowest heart rate value, E[HR ex ] is the average exercising heart rate value, HR rest  is the average resting heart rate value.
 
     By the invention, the people can estimate the Rating of Perceived Exertion (RPE), i.e., a camera based fatigue heart rate detection algorithm and a fatigue facial expression image feature extractor are fused to estimate the contactless-type Rating of Perceived Exertion (RPE) value. Therefore, the invention operating the fatigue heart rate detection, and a fatigue facial expression image feature extractor can obtain the similar results of wearable devices and the questionnaires. 
     Indeed, the invention can achieve the contactless-type monitoring heart rate and the facial expression image without any wearable devices attached on the athletes&#39; body. Therefore, the invention, which is the contactless-type method exactly can judge whether the athlete is under overtraining status or not. 
     It is understood that various other modifications will be apparent to and can be readily made by those skilled in the art without departing from the scope and spirit of the invention. Accordingly, it is not intended that the scope of the claims appended hereto be limited to the description as set forth herein, but rather that the claims be construed as encompassing all the features of patentable novelty that reside in the present invention, including all features that would be treated as equivalents thereof by those skilled in the art to which the invention pertains.