System and method for human motion detection and tracking

A system and method for human motion detection and tracking are disclosed. In one embodiment, a smart device having an optical sensing instrument monitors a stage having a mat. Memory is accessible to a processor and communicatively coupled to the optical sensing instrument. The system captures an image frame from the optical sensing instrument while capturing a data frame from the mat. The image frame is then converted into a designated image frame format, which is provided to a pose estimator. A two-dimensional dataset is received from the pose estimator. The system then converts, using inverse kinematics, the two-dimensional dataset into a three-dimensional dataset, which includes time-independent static joint positions, and then calculates, using the three-dimensional dataset in conjunction with the data frame, the position of each of the respective plurality of body parts in the image frame.

TECHNICAL FIELD OF THE INVENTION

The present disclosure relates, in general, to biomechanical evaluations and assessments, which are commonly referred to as range of motion assessments, and more particularly, to automating a biomechanical evaluation process, including a range of motion assessment, and providing recommended exercises to improve physiological inefficiencies of a user.

BACKGROUND OF THE INVENTION

Human beings have regularly undergone physical examinations by professionals to assess and diagnose their health issues. Healthcare history has been predominantly reactive to an adverse disease, injury, condition or symptom. Increasingly, in modern times, with more access to information, a preventative approach to healthcare has been gaining greater acceptance. Musculoskeletal health overwhelmingly represents the largest health care cost. Generally speaking, a musculoskeletal system of a person may include a system of muscles, tendons and ligaments, bones and joints, and associated tissues that move the body and help maintain the physical structure and form. Health of a person's musculoskeletal system may be defined as the absence of disease or illness within all of the parts of this system. When pain arises in the muscles, bones, or other tissues, it may be a result of either a sudden incident (e.g., acute pain) or an ongoing condition (e.g., chronic pain). A healthy musculoskeletal system of a person is crucial to health in other body systems, and for overall happiness and quality of life. Musculoskeletal analysis, or the ability to move within certain ranges (e.g., joint movement) freely and with no pain, is therefore receiving greater attention. However, musculoskeletal analysis has historically been a subjective science, open to interpretation of the healthcare professional or the person seeking care.

In 1995, after years of research, two movement specialists, Gray Cook and Lee Burton, attempted to improve communication and develop a tool to improve objectivity and increase collaboration efforts in the evaluation of musculoskeletal health. Their system, the Functional Movement Screen (FMS), is a series of seven (7) different movement types, measured and graded on a scale of 0-3. While their approach did find some success in bringing about a more unified approach to movement assessments, the subjectivity, time restraint and reliance on a trained and accredited professional to perform the evaluation limited its adoption. Accordingly, there is a need for improved systems and methods for measuring and analyzing physiological deficiency of a person and providing corrective recommended exercises while minimizing the subjectivity during a musculoskeletal analysis.

SUMMARY OF THE INVENTION

It would be advantageous to achieve systems and methods that would improve upon existing limitations in functionality with respect to measuring and analyzing physiological deficiency of a person. It would also be desirable to enable a computer-based electronics and software solution that would provide enhanced goniometry serving as a basis for furnishing corrective recommended exercises while minimizing the subjectivity during a musculoskeletal analysis. To better address one or more of these concerns, a system and method for human motion detection and tracking are disclosed. In one embodiment, a smart device having an optical sensing instrument monitors a stage having a mat. Memory is accessible to a processor and communicatively coupled to the optical sensing instrument. The system captures an image frame from the optical sensing instrument while capturing a data frame from the mat. The image frame is then converted into a designated image frame format, which is provided to a pose estimator. A two-dimensional dataset is received from the pose estimator. The system then converts, using inverse kinematics, the two-dimensional dataset into a three-dimensional dataset, which includes time-independent static joint positions, and then calculates, using the three-dimensional dataset in conjunction with the data frame, the position of each of the respective plurality of body parts in the image frame. These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter.

DETAILED DESCRIPTION OF THE INVENTION

Referring initially toFIG. 1A, therein is depicted one embodiment of a system for human motion detection and tracking that may be incorporated into an integrated goniometry system, for example, for performing automated biomechanical movement assessments, which is schematically illustrated and designated10. As shown, the integrated goniometry system10includes a smart device12, which may function as an integrated goniometer, having a housing14securing an optical sensing instrument16and a display18. The display18includes an interactive portal20which provides prompts, such as an invitation prompt22, which may greet a crowd of potential users U1, U2, and U3and invite a user to enter a stage24having a mat26, which may include markers27A,27B for foot placement of the user standing at the markers27A,27B to utilize the integrated goniometry system10. The mat26may be a pressure mat that is wirelessly linked to the smart device12. The pressure mat may provide high resolution, time-domain data on the forces exerted by every part of the user's feet and, as a result, reveal subtle changes in balance and acceleration that possibly would not be discernible by solely motion analysis.

The stage24may be a virtual volumetric area28, such as a rectangular or cubic area, that is compatible with human exercise positions and movement. The display18faces the stage24and the optical sensing instrument16monitors the stage24. A webcam17may be included in some embodiments. It should be appreciated that the location of the optical sensing instrument16and the webcam17may vary with the housing14. Moreover, the number of optical sensing instruments used may vary also. Multiple optical sensing instruments or an array thereof may be employed. It should be appreciated that the design and presentation of the smart device12may vary depending on application. By way of example, the smart device12and the housing14may be a device selected from the group consisting of, with or without tripods, smart phones, smart watches, smart wearables, and tablet computers, for example.

Referring now toFIG. 1B, the user, user U2, has entered the stage24and the interactive portal20includes an exercise movement prompt30providing instructions for the user U2on the stage24to be positioned on the mat26with the markers27A,27B providing guidance on foot placement. The exercise movement prompt may also provide instructions to execute a set number of repetitions of an exercise movement, such as a squat or a bodyweight overhead squat, for example. In some implementations, the interactions with the user U2are contactless and wearableless. A series of prompts on the interactive portal20instruct the user U2while the optical sensing instrument16senses body point data of the user U2during each exercise movement. While the optical sensing instrument16of the smart device12senses body point data, simultaneously, the smart device12captures a data frame from the mat26via, for example, wireless communication W1, W2. As shown, the data frame is captured relative to the user U2in the line-of-sight with the optical sensing instrument16while the user U2is on the mat26with feet properly positioned at the markers27A,27B once the user U2turns around as shown by rotation arrow R. As will be discussed in further detail hereinbelow, the data frame may include pressure sensor array data from the mat26. Based on the sensed body point data and the pressure sensor array data, a mobility score, an activation score, a posture score, a symmetry score, or any combination thereof, for example, may be calculated. A composite score may also be calculated. One or more of the calculated scores may provide the basis for the integrated goniometry system10determining an exercise recommendation. As mentioned, a series of prompts on the interactive portal20instruct the user U2through repetitions of exercise movements while the optical sensing instrument16senses body point data of the user U2and the smart device12receives the pressure sensor array data from the mat26. It should be appreciated that the smart device12may be supported by a server that provides various storage and support functionality to the smart device12. Further, the integrated goniometry system10may be deployed such that the server is remotely located in a cloud C to service multiple sites with each site having a smart device.

Referring now toFIG. 2andFIG. 3, respective embodiments of a human skeleton60and body parts identified by the integrated goniometry system10are depicted. Body part data70approximates certain locations and movements of the human body, represented by the human skeleton60. More specifically, the body part data70is captured by the optical sensing instrument16and may include designated body part data72and synthetic body part data74. By way of example and not by way of limitation, designated body part data72may include head data82, neck data84, right shoulder data86, left shoulder data88, right upper arm data90, left upper arm data92, right elbow data94, left elbow data96, right lower arm data99, left lower arm data101, right wrist data98, left wrist data100, right hand data102, left hand data104, right upper torso data106, left upper torso data108, right lower torso data110, left lower torso data112, upper right leg data114, upper left leg data116, right knee data118, left knee data120, right lower leg data122, left lower leg data124, right ankle data126, left ankle data128, right foot data130, and left foot data132. By way of example and not by way of limitation, synthetic body part data74may include right hip140, left hip142, waist144, top of spine146, and middle of spine148. As will be appreciated, the synthetic body part data74may include data captured by the optical sensing instrument16that includes locations in the body in the rear of the person or data acquired through inference.

Referring now toFIG. 4andFIG. 5, graphical heat map representations150of one embodiment of a data frame from the mat26are shown at time taand time tb, for example. The graphical heat map representations150include a horizontal axis152and a vertical axis154which define nodes156representing a multidimensional array of pressure sensors positioned within the mat26. Gradient158, which may be one of several visualization tools provided, may represent a physical quantity that describes the forces of the pressure sensor array data at a particular node of the nodes156. As shown, at time ta, the data frame provides high resolution, time-domain data on forces160with respect to a left foot162and forces164with respect a right foot166. Similarly, as shown, at time tb, the data frame provides high resolution, time-domain data on forces168with respect to the left foot162and forces170with respect to the right foot166.

By way of example, by comparing the data frame at time taand time tb, as reflected in the heat map data by forces160,164,168,170, left and right foot pronation measurements have increased in pressure. Additionally, the left and right supination measurements have decreased in pressure as reflected in the heat map data by comparison of forces160,164,168,170. Such measurement of ground-reaction forces at the feet of the user may enhance the systems and methods presented herein. More particularly, the systems and methods for assessing a user's fitness and mobility by combining captured motion data from a camera with captured ground-reaction force data via a mat during the execution of a known movement provide an enhanced assessment by a detailed time-domain analysis of joint positions and the concurrent forces on the mat from the user's feet. By way of further example, the following table, Table: Detections & Explanations, provides insights into the use of the detailed time-domain analysis of joint positions and concurrent forces as may be gathered wirelessly via the pressure mat.

TABLEDetections & ExplanationsDetectionExplanationHeel Lift Fault DetectionThe heel should remain onthe ground and always havesteady pressure appliedthrough it.Foot Moved Fault DetectionA single foot should notcompletely lose pressuredetectionToe Lift Fault DetectionThe toes should always havepressure through them in thesquatLeft Foot Position Vectorfoot pointed forward, turnedout, or turned in) - duringthe squat, the left footshould be pointed at leastslightly outwardRight Foot Position Vectorfoot pointed forward, turnedout, or turned in) - duringthe squat, the right footshould be pointed at leastslightly outwardFoot Position Vectorleft vs right foot positionSymmetryvector symmetry) - duringthe squat, the left andright feet should be pointedat least slightly outward,and about the same amountLeft/Right Whole Bodyduring the squat the centerCenter of Pressureof pressure should not shiftto the left or the rightForward/Back Center ofduring the squat the centerPressureof pressure should not shiftforward or backwardCenter of Pressure Tracefoot pressure should shiftfrom the mid foot toward theheel and lateral foot duringthis loading phaseLeft vs Right Foot Left toleft foot lateral center ofRight Center of Pressurepressure should be as farTrace Symmetryleft of center body as theright foot lateral center ofpressure is far right ofcenter bodyLeft vs Right Foot Front toleft foot longitudinalBack Center of Pressurecenter of pressure should beTrace Symmetry -as far forward or back ofcenter body as the rightfoot longitudinal center ofpressureLeft Foot Pronationthe measure of the bodyMeasurementweight on the inside of theleft footRight Foot Pronationthe measure of the bodyMeasurementweight on the inside of theright footLeft Foot Supinationthe measure of the bodyMeasurementweight on the outside of theleft foot)Right Foot Supinationthe measure of the bodyMeasurementweight on the outside of theright foot)

Referring now toFIG. 6, image frames associated with a set number of repetitions of an exercise movement by the user U2are monitored and captured by the integrated goniometry system10. As shown, in the illustrated embodiment, the user U2executes three squats and specifically three bodyweight overhead squats at t3, t5, and t7. It should be understood, however, that a different number of repetitions may be utilized and is within the teachings presented herein. That is, N iterations of movement is provided for by the teachings presented herein. At times t1and t9, the user U2is at a neutral position, which may be detected by sensing the body point data within the virtual volumetric area28of the stage24or at t9, an exercise end position which is sensed with the torso in an upright position superposed above the left leg and the right leg with the left arm and right arm laterally offset to the torso.

At times t2, t4, t6, and t8, the user U2is at an exercise start position. The exercise start position may be detected by the torso in an upright position superposed above the left leg and the right leg with the left arm and the right arm superposed above the torso. From an exercise start position, the user U2begins a squat with an exercise trigger. During the squat or other exercise movement, image frames are collected. The exercise trigger may be displacement of the user from the exercise start position by sensing displacement of the body. Each repetition of the exercise movement, such as a squat, may be detected by sensing the body returning to its position corresponding to the exercise start position. By way of example, the spine midpoint may be monitored to determine or mark the completion of exercise movement repetitions.

Referring toFIG. 7, by way of example, an image frame170is captured having data at a time t1. The image frame170includes at each image element, coordinate values that are monoptic and represent two-dimensional coordinate values. Pre-processing occurs to the image frame170to provide a designated image frame format172, which represents the pre-processing of the image frame170. Such pre-processing includes isolation of an object, i.e., the user U2. Next, as shown at a pose estimator174, probability distribution models156generated by a neural network178are applied to the designated image frame format172to identify body parts, such as skeleton points, as shown by two-dimensional dataset180.

The two-dimensional dataset180is then converted via an application of inverse kinematics182into a three-dimensional dataset184prior to the position of each of the respective body parts being determined within the three-dimensional dataset184. More particularly, as shown, the two-dimensional dataset180includes various of the designated body part data72, including the head data82, the neck data84, the right shoulder data86, the left shoulder data88, the right elbow data94, the left elbow data96, the right wrist data98, the left wrist data100, the right knee data118, the left knee data120, the right ankle data126, the left ankle data128, the right hip data140, and the left hip data142, for example. A horizontal axis (designated y) and a vertical axis (designated x) are defined in the two-dimensional dataset180. An intersection is fixed at F between the two-dimensional dataset180and the horizontal axis (designated y) as a start of a kinematic chain186. The intersection corresponds to feet of the user U2.

Then, the smart device12calculates variable joint parameters under assumptions A1, A2, A3, for example, that limb lengths LL1, LL2, LL3, LL4, LL5, LL6, LR1, LR2, LR3, LR4, LR5, LR6, have at least two hinge joints with a component of movement in a depth axis (designated z) perpendicular to the horizontal axis (designated y) and the vertical axis (designated x). In particular, the assumption A1relates to the knees, as represented by the right knee data118and the left knee data120, having a component of movement in a depth axis (designated z); the assumption A2relates to the hips, as represented by the right hip data140and the left hip data142, having a component of movement in a depth axis (designated z); and the assumption A3relates to the elbows, as represented by the right elbow data94and the left elbow data96, having a component of movement in a depth axis (designated z).

The limb length LL1defines the length from the left ankle data128to the left knee data120; the limb length LL2defines the length from the left knee data120to the left hip data142; the limb length LL3defines the length from the left hip data142to the left shoulder data88; the limb length LL4defines the length from the left shoulder data88to the neck data84; the limb length LL5defines the length from the left shoulder data88to the left elbow data96; and the limb length LL6defines the length from the left elbow data96to the left wrist data100. Similarly, the limb lengths LR1, LR2, LR3, LR4, LR5, LR6respectively relate to the segments the right ankle data126to the right knee data118, the right knee data118to the right hip data140, the right hip data140to the right shoulder data86, the right shoulder data86to the neck data84; the right shoulder data86to the right elbow data94, and the right elbow data94to the right wrist data98. The limb length Lcrelates to the length from the neck data84to the head data82.

The smart device12also calculates variable joint parameters with respect to limb lengths of the user U2required to place the ends of the kinematic chain186in a given position and orientation relative to the start of the kinematic chain186at the fixed intersection F. The position of each of the body parts in the image frame170, which were in two dimensions (e.g., xn, yn) is calculated with respect to the image frame150to provide three-dimensional coordinates (e.g. xn, yn, zn) and provide joint positions, for example, such as angle alphan.

Referring now toFIG. 8, as mentioned inFIG. 7, the position of each of the body parts in the image frame170, which were in two dimensions (e.g., xn, yn) is calculated with respect to the image frame150to provide three-dimensional coordinates (e.g. xn, yn, zn) and provide joint positions, for example, such as angle alphan. In some implementations, as also previously mentioned, the system10simultaneously captures a frame of data from the pressure mat that represents the forces on each of the nodes156on the mat26. The three-dimensional dataset for a moment in time is used in conjunction with the pressure mat frame for that same moment in time to deduce the body part positions, as well as accelerations and balances of the user U2being assessed. This enhances the motion analysis, and ultimately scoring188, by measurement of ground-reaction forces. In this manner, the system10may calculate, using the three-dimensional dataset in a non-time domain manner, in conjunction with the pressure sensor array data in a time-domain manner, a position of each of a respective plurality of body parts in the image frame. The pressure sensor array data providing for various deductions into joint positions. As previously discussed, with the addition of the pressure mat-time series data, the system10can detect, by way of example and not by way of limitation, positions of occluded heel joints (heel lift) as well as diagnose weight shifts to the inside or outside of the feet. Further, this data provides deductions into discovering subtle shifts in balance in the time domain during the course of an exercise repetition or across the length of a session.

Referring toFIG. 9, within the housing14of the smart device12, a processor190, memory192, and storage194are interconnected by a busing architecture196within a mounting architecture that also interconnects a network interface198, a camera200, including an image camera input202and/or image camera204, inputs206, which may include a data connection element, outputs208, and the display18. A wireless transceiver210is also provided and connected to the busing architecture196.

The processor190may process instructions for execution within the smart device12as a computing device, including instructions stored in the memory192or in storage194. The memory192stores information within the computing device. In one implementation, the memory192is a volatile memory unit or units. In another implementation, the memory192is a non-volatile memory unit or units. The storage194provides capacity that is capable of providing mass storage for the smart device12. The network interface198may provide a point of interconnection, either wired or wireless, between the smart device12and a private or public network, such as the Internet. The various inputs206and outputs208provide connections to and from the computing device, wherein the inputs206are the signals or data received by the smart device12, and the outputs208are the signals or data sent from the smart device12. The inputs206may support the data connection element, which may provide a wired-connection with the mat26, for example, as an alternative to a wireless-connection with the mat26provided by the wireless transceiver210. The display18may be an electronic device for the visual presentation of data and may, as shown inFIG. 9, be an input/output display providing touchscreen control. The camera200may be enabled by an image camera input202that may provide an input to the optical sensing instrument16, which may be a camera, a point-cloud camera, a laser-scanning camera, an infrared sensor, an RGB camera, or a depth camera, for example, or the camera190may be an image camera194directly integrated into the smart device12. By way of further example, the optical sensing instrument16may utilize technology such as time of flight, structured light, or stereo technology. By way of still further example, in instances where the optical sensing instrument16is a depth camera, an RGB camera, a color camera, a structured light camera, a time of flight camera, a passive stereo camera, or a combination thereof may be employed. Further, it should be appreciated that the optical sensing instrument16may include two or more optical sensing instruments; that is, more than one sensing instrument may be employed. As mentioned, the smart device12and the housing14may be a device selected from the group consisting of (with or without tripods) smart phones, smart watches, smart wearables, and tablet computers, for example. The wireless transceiver210may be internal, external, or a combination thereof to the housing14. Communication between the mat26, or another device, may be enabled by a variety of wireless methodologies employed by the wireless transceiver210, including 802.11, 3G, 4G, Edge, WiFi, ZigBee, near field communications (NFC), Bluetooth low energy and Bluetooth, for example. Also, infrared (IR) may be utilized.

The memory182and storage184are accessible to the processor180and include processor-executable instructions that, when executed, cause the processor180to execute a series of operations. In a first series of operations, the processor-executable instructions cause the processor180to display an invitation prompt on the interactive portal. The invitation prompt provides an invitation to the user to enter the stage prior to the processor-executable instructions causing the processor180to detect the user on the stage by sensing body point data within the virtual volumetric area28. By way of example and not by way of limitation, the body point data may include first torso point data, second torso point data, first left arm point data, second left arm point data, first right arm point data, second right arm point data, first left leg point data, second left leg point data, first right leg point data, and second right leg point data, for example.

The processor-executable instructions cause the processor190to display the exercise movement prompt30on the interactive portal20. The exercise movement prompt30provides instructions for the user to execute an exercise movement for a set number of repetitions with each repetition being complete when the user returns to an exercise start position. The processor190is caused by the processor-executable instructions to detect an exercise trigger. The exercise trigger may be displacement of the user from the exercise start position by sensing displacement of the related body point data. The processor-executable instructions also cause the processor190to display an exercise end prompt on the interactive portal20. The exercise end prompt provides instructions for the user to stand in an exercise end position. Thereafter, the processor180is caused to detect the user standing in the exercise end position.

The processor-executable instructions cause the processor190to calculate one or more of several scores including calculating a mobility score by assessing angles using the body point data, calculating an activation score by assessing position within the body point data, calculating a posture score by assessing vertical differentials within the body point data, and calculating a symmetry score by assessing imbalances within the body point data. The processor-executable instructions may also cause the processor190to calculate a composite score based on one or more of the mobility score, the activation score, the posture score, or the symmetry score. The processor-executable instructions may also cause the processor190to determine an exercise recommendation based on one or more of the composite score, the mobility score, the activation score, the posture score, or the symmetry score.

In a second series of operations, the processor-executable instructions cause the processor190to capture an image frame from the optical sensing instrument16. The image frame may include at each image element, two-dimensional coordinate values including a point related to a distance from the optical sensing instrument16. Then the processor190may be caused to convert the image frame into a designated image frame format. The designated image frame format may include at each image element, coordinate values relative to the image frame. The processor executable instructions may cause the processor190to access, through a pose estimator, and apply multiple probability distribution models to the designated image frame format to identify a respective plurality of body parts. In acquiring the multiple probability distribution models, the probability distribution models may be generated by a neural network. Next, the processor190may be caused to calculate the position of each of the plurality of body parts in the designated image frame format and then calculate the position of each of the plurality of body parts in the image frame.

In a third series of operations, the processor-executable instructions cause the processor190to capture, via the optical sensing instrument, an image frame relative to a user in a line-of-sight with the optical sensing instrument. The image frame may include at each image element monoptic coordinate values. The processor190is then caused by the processor-executable instructions to convert the image frame into a designated image frame format prior to providing the designated image frame format to a pose estimator. The processor190is caused to receive a two-dimensional dataset from the pose estimator and convert, using inverse kinematics, the two-dimensional dataset into a three-dimensional dataset. The processor190then calculates, using the three-dimensional dataset, the position of each of the respective plurality of body parts in the image frame.

In a fourth series of operations, the processor-executable instructions cause the processor190to convert, using inverse kinematics, the two-dimensional dataset into a three-dimensional dataset by first defining a horizontal axis and a vertical axis in the two-dimensional dataset. The processor-executable instructions then cause the processor190to fix an intersection of the two-dimensional dataset and the horizontal axis as a start of a kinematic chain. The intersection may correspond to feet of the user. The processor190is then caused to calculate variable joint parameters under assumptions that the limb lengths have at least two hinge joints with a component of movement perpendicular to the horizontal axis and the vertical axis. Then the processor190calculates the variable joint parameters with respect to limb lengths of the user required to place ends of the kinematic chain in a given position and orientation relative to the start of the kinematic chain.

In a fifth series of operations, the processor-executable instructions cause the processor190to capture, via the optical sensing instrument, an image frame relative to a user in a line-of-sight with the optical sensing instrument. As mentioned, the image frame may inclue at each image element monoptic coordinate values. The user matey be at a known location, the user performing a known movement. The processor-executable instructions then cause the processor190to capture, via the data connection element, a data frame from a mat with the data frame being relative to the user in the line-of-sight with the optical sensing instrument. Further, the user is on a mat and the data frame including pressure sensor array data from the mat. Then, the processor190is caused to convert the image frame into a designated image frame format, provide the designated image frame format to a pose estimator, and receive a two-dimensional dataset from the pose estimator.

The processor-executable instructions then cause the processor190to convert, using inverse kinematics, the two-dimensional dataset into a three-dimensional dataset. As discussed, the three-dimensional dataset includes a time-independent plurality of static joint positions. Then, the processor190is caused to calculate, using the three-dimensional dataset in a non-time domain manner, in conjunction with the pressure sensor array data in a time-domain manner, a position of each of a respective plurality of body parts in the image frame.

The processor-executable instructions presented hereinabove include, for example, instructions and data which cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. Processor-executable instructions also include program modules that are executed by computers in stand-alone or network environments. Generally, program modules include routines, programs, components, data structures, objects, and the functions inherent in the design of special-purpose processors, or the like, that perform particular tasks or implement particular abstract data types. Processor-executable instructions, associated data structures, and program modules represent examples of the program code means for executing steps of the systems and methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps and variations in the combinations of processor-executable instructions and sequencing are within the teachings presented herein.

With respect toFIG. 9, in some embodiments, the system for human motion detection and tracking may be at least partially embodied as a programming interface configured to communicate with a smart device. Further, in some embodiments, the processor190and the memory192of the smart device12and a processor and memory of a server may cooperate to execute processor-executable instructions in a distributed manner. By way of example, in these embodiments, the server may be a local server co-located with the smart device12, or the server may be located remotely to the smart device12, or the server may be a cloud-based server.

Referring toFIG. 10, the mat26may include a body220configured to accept the user thereon. The body may have elastomeric characteristics. A multidimensional array of pressure sensors222is positioned within the body to define the nodes156on the multidimensional grid. A controller224, which may include processor and memory, supported by a busing architecture226, is communicatively located within the body and communicatively disposed with the multidimensional array of pressure sensors222. The controller224captures the data frame including pressure sensor array data from the mat26.

Various inputs230and outputs232, which are interconnected in the busing architecture226, provide connections to and from the mat26, wherein the inputs230are the signals or data received by the mat26, and the outputs232are the signals or data sent from the mat26. The inputs230may support the data connection element, which may provide a wired-connection between the mat26and the smart device12, for example, as an alternative to a wireless-connection with the smart device12provided by a wireless transceiver234. The wireless transceiver234may be internal, external, or a combination thereof to the mat26. Communication between the smart device12, or another device, may be enabled by a variety of wireless methodologies employed by the wireless transceiver234, including 802.11, 3G, 4G, Edge, WiFi, ZigBee, near field communications (NFC), Bluetooth low energy and Bluetooth, for example. Also, infrared (IR) may be utilized.

FIG. 11conceptually illustrates a software architecture of some embodiments of an integrated goniometry application250that may automate the biomechanical evaluation process and provide recommended exercises to improve physiological inefficiencies of a user. Such a software architecture may be embodied on an application installable on a smart device, for example. That is, in some embodiments, the integrated goniometry application250is a stand-alone application or is integrated into another application, while in other embodiments the application might be implemented within an operating system300. In some embodiments, the integrated goniometry application250is provided on the smart device12. Furthermore, in some other embodiments, the integrated goniometry application250is provided as part of a server-based solution or a cloud-based solution. In some such embodiments, the integrated goniometry application250is provided via a thin client. In particular, the integrated goniometry application250runs on a server while a user interacts with the application via a separate machine remote from the server. In other such embodiments, integrated goniometry application250is provided via a thick client. That is, the integrated goniometry application250is distributed from the server to the client machine and runs on the client machine.

The integrated goniometry application250includes a user interface (UI) interaction and generation module252, management (user) interface tools254, data acquisition modules256, image frame processing modules258, image frame pre-processing modules260, a pose estimator interface261, mobility modules262, inverse kinematics modules263, mat interface modules264, graphical processing modules265, calculation/deduction modules266, stability modules267, posture modules268, recommendation modules269, and an authentication application270. The integrated goniometry application250has access to activity logs280, measurement and source repositories284, exercise libraries286, and presentation instructions290, which presents instructions for the operation of the integrated goniometry application250and particularly, for example, the aforementioned interactive portal20on the display18. In some embodiments, storages280,284,286, and290are all stored in one physical storage. In other embodiments, the storages280,284,286, and290are in separate physical storages, or one of the storages is in one physical storage while the other is in a different physical storage.

The UI interaction and generation module250generates a user interface that allows, through the use of prompts, the user to quickly and efficiently perform a set of exercise movements to be monitored, with the body point data collected from the monitoring furnishing an automated biomechanical movement assessment scoring and related recommended exercises to mitigate inefficiencies. Prior to the generation of automated biomechanical movement assessment scoring and related recommended exercises, the data acquisition modules256may be executed to obtain instances of the body point data via the optical sensing instrument16, which is then processed with the assistance of the image frame processing modules258and the image frame pre-processing modules260. The mat interface modules264interface with the mat to collect the pressure sensor array data, which may be processed and presented by the graphical processing modules265. The pose estimator interface261is utilized to provide, in one embodiment, image frame pre-processing files created by the image pre-processing modules260to a pose estimator to derive skeleton points and other body point data. Following the collection of the body point data, the inverse kinematics modules263derives three-dimensional data including joint position data. Then, the mobility modules262, stability modules267, and the posture modules268are utilized to determine a mobility score, an activation score, and a posture score, for example, based on the body part determinations by the calculation/deduction module266as determined by using the three-dimensional dataset in a non-time domain manner, in conjunction with the pressure sensor array data in a time-domain manner. More specifically, in one embodiment, the mobility modules262measure a user's ability to freely move a joint without resistance. The stability modules264provide an indication of whether a joint or muscle group may be stable or unstable. The posture modules266may provide an indication of physiological stresses presented during a natural standing position. Following the assessments and calculations by the mobility modules262, stability modules264, and the posture modules266, the recommendation modules268may provide a composite score based on the mobility score, the activation score, and the posture score as well as exercise recommendations for the user. The authentication application270enables a user to maintain an account, including an activity log and data, with interactions therewith.

In the illustrated embodiment,FIG. 11also includes the operating system300that includes input device drivers302and a display module304. In some embodiments, as illustrated, the input device drivers302and display module304are part of the operating system300even when the integrated goniometry application250is an application separate from the operating system300. The input device drivers302may include drivers for translating signals from a keyboard, a touch screen, or an optical sensing instrument, for example. A user interacts with one or more of these input devices, which send signals to their corresponding device driver. The device driver then translates the signals into user input data that is provided to the UI interaction and generation module252.

FIG. 12depicts one embodiment of a method for integrated goniometric analysis. At block320, the methodology begins with the smart device positioned facing the stage. At block322, multiple bodies are simultaneously detected by the smart device in and around the stage. As the multiple bodies are detected, a prompt displayed on the interactive portal of the smart device invites one of the individuals to the area of the stage in front of the smart device at a mat. At block326, one of the multiple bodies is isolated by the smart device12and identified as an object of interest once it separates from the group of multiple bodies and enters the stage in front of the smart device12. The identified body, a user, is tracked as a body of interest by the smart device.

At block328, the user is prompted to position himself into the appropriate start position on the mat which will enable the collection of a baseline measurement and key movement measurements during exercise. At this point in the methodology, the user is prompted by the smart device to perform the exercise start position and begin a set repetitions of an exercise movement. The smart device collects body point data and pressure sensor array data to record joint angles and positions. At block330, the smart device detects an exercise or movement trigger which is indicative of phase movement discrimination being performed in a manner that is independent of the body height, width, size or shape of the user.

At block332, the user is prompted by the smart device to repeat the exercise movement as repeated measurements provide more accurate and representative measurements. A repetition is complete when the body of the user returns to the exercise start position. The user is provided a prompt to indicate when the user has completed sufficient repetitions of the exercise movement. With each repetition, once in motion, monitoring of body movement will be interpreted to determine a maximum, minimum, and moving average for the direction of movement, range of motion, depth of movement, speed of movement, rate of change of movement, and change in the direction of movement, for example. Additionally, pressure mat-time series data informs the calculations and conclusions by permitting the system10to detect, by way of example and not by way of limitation, positions of occluded heel joints (heel lift) as well as diagnose weight shifts to the inside or outside of the feet. Further, this data provides deductions into discovering subtle shifts in balance in the time domain during the course of an exercise repetition or across the length of a session. At block334, the repetitions of the exercise movement are complete. Continuing to decision block335, if the session is complete, then methodology advances to block336. If the session is not complete, then the methodology returns to the block322. At block336, once the required number of repetitions of the exercise movement are complete, the user is prompted to perform an exercise end position, which is a neutral pose. Ending at block338, with the exercise movements complete, the integrated goniometry methodology begins calculating results and providing the results and any exercise recommendations to the user.

FIG. 13andFIG. 14show the methodology in more detail with elements350through366and elements380through410. Referring now toFIG. 13, the methodology begins with block350and continues to block352where an image frame and pressure mat-time series data are captured. The image frame may be captured by the optical sensing instrument and the pressure mat-time series data may be captured by the wireless transceiver. By way of example and not by way of limitation, image frame acquisition may involve obtaining raw image frame data from the camera. Additionally, in some embodiments, the image frame is captured of a user at a known location performing a known movement, such as a squat. At block354, pre-processing of the image frame occurs. As previously discussed, during pre-processing, the image frame is converted into a designated image frame format such that at each image element monoptic coordinate values are present relative to the image frame. Also, during the pre-processing, the object—the body of the user—may be isolated. At block356, the image frame is converted into the designated image frame format before the submission at block358to a pose estimator for the application of a probability distribution model or models occurs for the body parts. Following the return of the data from the pose estimator, at block360, inverse kinematics as well as deductions based on the pressure mat-time series data are applied to infer three-dimensional data, such as joint positions, from the two-dimensional data.

The three-dimensional dataset may include time-independent static joint positions. It is very common for applications using body pose estimation to be focused on time domain measurements, like attempting to gauge the speed or direction of a movement by comparing joint or limb positions across multiple video frames. Often the goal is to classify the observed movement, for instance using velocity or acceleration data to differentiate falling down from sitting down. In contrast, in some embodiments, the systems and methods presented herein focus on accumulating a dataset of static joint positions that can be used to accurately calculate relevant angles between body parts to assess a session of multiple overhead squats. In these embodiments, the systems and methods know in advance exactly where the user is located in the frame and what the movement will be. It is not necessary to use time domain data to identify the movement being performed or to estimate the speed at which it is performed.

In these embodiments, the assessment utilizes limited time domain data to analyze and score performance of the overhead squats. As previously discussed, pressure mat-time series data informs the calculations and conclusions by permitting the methodology to detect, by way of example and not by way of limitation, positions of occluded heel joints (heel lift) as well as diagnose weight shifts to the inside or outside of the feet. Further, this data provides deductions into discovering subtle shifts in balance in the time domain during the course of an exercise repetition or across the length of a session. The position of each of the body parts may be calculated more accurately at block364before the position of each body part is mapped at block366and the process concludes at block368.

Referring now toFIG. 14, the methodology is initiated with the operation of the camera at block380to ensure the camera is level. At block382, the camera captures an image frame, and the methodology detects a body at decision block384. If a body is not detected, then the methodology returns to block382. On the other hand, if a body is detected, then the position of the body is evaluated at decision block386. If the position of the body has issues, such as the body not being completely in the frame or the body not being square with respect to the frame, then the methodology proceeds to block388, where a correction is presented to assist the user with correcting the error before re-evaluation at decision block386.

Once the position at decision block386is approved, then the methodology advances to posture guidance at block390before the posture is evaluated at decision block392. If the posture of the user is correct, then the methodology advances to decision block394. On the other hand if the user's pose does not present the correct posture, then the methodology returns to block390where posture guidance is provided. At decision block394, if the pose is held long enough then the methodology advances to block396where limb length data is saved. If the pose is not held long enough, then the process returns to decision block392.

At block398, session guidance starts and the session, which presents exercises or poses for the user to complete, continues until completion unless, as shown at decision block400, the session is interrupted or otherwise not completed. If the session is not completed, as shown by decision block400, the methodology returns to decision block384. At block402and block404, the image frame is converted into the processed image frame and recorded two-dimensional skeleton points and limb lengths are utilized with inverse kinematics and deduction based on time-dependent data collected from the mat to calculate relevant angles between body parts. At block406, the scoring algorithm is applied before scores are presented at block408. At decision block410, the scores will be continued to be displayed until the user navigates back to the main screen which returns the methodology to block382.

The order of execution or performance of the methods and data flows illustrated and described herein is not essential, unless otherwise specified. That is, elements of the methods and data flows may be performed in any order, unless otherwise specified, and that the methods may include more or less elements than those disclosed herein. For example, it is contemplated that executing or performing a particular element before, contemporaneously with, or after another element are all possible sequences of execution.