Patent Description:
People make efforts to optimally control their postures that are suitable for the environments inside and outside the body and specific subjects in everyday life, and such posture balance control is necessary and has an important meaning when we independently live.

In particular, movement of the center of gravity of posture balance control abilities can be shown not only in an erect posture, but various activities in everyday line, and the load that is applied in situations such as when a person lifts an object or walks with a bag on his/her shoulder has considerable influence on posture maintenance and balance maintenance.

When excessive load is applied while a person carries an object or when a person walks and carries an object in a wrong way, movement of the center of gravity is influenced by variation of the walking pattern. In normal walking, the center of gravity is regularly and smoothly changed up, down, left, right, and in the walking direction and the legs and arms are harmoniously moved. However, when a person walks with a load in his/her arms or on his/her shoulders, the center of gravity is irregularly moved.

Further, a coping mechanism for maintaining balance by bringing the center line of gravity, of which the position has been changed due to the way of carrying a load, to the center of base of support, for maintaining balance by bringing the center line further forward to move forward, and for reducing energy consumption for walking is activated in a human body.

Gait analysis is used in decision making in clinical tests, evaluation after treatment, evaluation of intention and helping tools, and various biomechanical studies for musculoskeletal and neurologic diseases, does an inclusive evaluation by measuring various features of walking, and includes kinematic analysis, dynamic electromyography, energy expenditure measurement, etc..

In general, walking pattern extraction devices are widely used for rehabilitation of recovering the walking sense of patients with a walking disorder due to injuries to the central nervous system, for walking posture correction training of users who need walking posture correction, or the like.

Meanwhile, according to walking analysis methods of the related art, it is required to attach attachable sensors (for example, inertia sensors such as a gyro sensor and an acceleration sensor) to the body, obtain information from the attachable sensors, and attach such attachable sensors for every measurement, so it is limited to analyze walking in everyday life.

Further, a walking analysis method using a single depth camera has limitation that a range in which walking data can be obtained is small and the accuracy of kinematic information of hidden body parts is deteriorated due to an occlusion phenomenon in which a body part is hidden by another body part.

Accordingly, the present disclosure intends to provide a system for analyzing walking behavior that can solve the problem that the accuracy of kinematic information of body parts is deteriorated due to the occlusion phenomenon.

<CIT> discloses a method for gait analysis of a subject performed periodically over time to detect changes in one or more gait characteristics.

An objective of the present disclosure is to provide a system for analyzing walking behavior that can solve the problems of the related art.

In order to achieve the objectives, a system for analyzing walking behavior according to an embodiment of the present disclosure includes: an identifier configured to identify a walker; a walking information obtainer configured to image a walking motion of the walker in at least one or more directions of a left, a right, and a front on the basis of an identification signal from the identifier, and to extract and provide kinematic information from the taken images; a walking information collector configured to combine and collect kinematic information that is 3D and/or 2D skeleton information of the taken walking images; a walking parameter extractor/classifier configured to extract at least one or more walking parameter on the basis of the combined kinematic information that is 3D and/or 2D skeleton information, and then classify the kind of the walking; and a walking result informer configured to determine whether the at least one or more walking parameters come out of a recommendation reference value, or are included in a walking parameter range of abnormal walking, or are classified as a pathological gait, and then provide the result to the walker or an expert, wherein the walking information obtainer recognizes markers in which three axis points are recorded and that are arranged with regular intervals on a walking path when a walking motion is imaged by a plurality of cameras, and calculates a 3D and/or 2D transformation matrix of each kinematic information that is 3D and/or 2D skeleton information taken on the basis of the three axis points, and transforms axes of coordinates.

By using the system for analyzing walking behavior according to an embodiment of the present disclosure, it is possible to extract and classify walking parameters such as the step, the speed, the waist curvature, the average knee bending angle, etc. of a walker and to provide the classified walking motion of the walker such that the classified walking motion is used as a datum helping an expert propose a medical opinion.

The above and other objectives, features and other advantages of the present invention will be more clearly understood from the following detailed description when taken in conjunction with the accompanying drawings, in which:.

In order to achieve the objectives, a system for analyzing walking behavior according to an embodiment of the present disclosure includes: an identifier configured to identify a walker; a walking information obtainer configured to image a walking motion of the walker in at least one or more directions of a left, a right, and a front on the basis of an identification signal from the identifier, and to extract and provide kinematic information from the taken images; a walking information collector configured to combine and collect kinematic information (3D and/or 2D skeleton information) of the taken walking images; a walking parameter extractor/classifier configured to extract at least one or more walking parameter on the basis of the combined kinematic information (3D and/or 2D skeleton information), and then classify the kind of the walking; and a walking result informer configured to determine whether the at least one or more walking parameters come out of a recommendation reference value, or are included in a walking parameter range of abnormal walking, or are classified as a pathological gait, and then provide the result to the walker or an expert.

In order to achieve the objectives, a system for analyzing walking behavior according to an embodiment of the present disclosure can be used for performing: an identification step of identifying a walker by means of an identifier; a kinematic information extraction step of imaging a walking motion of the walker in at least one or more directions of a left, a right, and a front on the basis of an identification signal from the identifier, and then extracting and providing kinematic information from the taken images by means of a walking information collector; a combination step of combining kinematic information (3D and/or 2D skeleton information) of the walking images taken by a walking information collector; a walking parameter extraction and classification step of a extracting at least one or more walking parameter on the basis of the combined kinematic information (3D and/or 2D skeleton information), and then of classifying a walking type by means of a walking parameter extractor/classifier; and a walking problem determination step of determining a walking problem of a walker on the basis of the at least one or more walking parameters or the result of classifying walking by means of a walking result informer.

In the following description, the structural or functional description specified to exemplary embodiments according to the concept of the present disclosure is intended to describe the exemplary embodiments, so it should be understood that the present disclosure may be variously embodied, without being limited to the exemplary embodiments.

Embodiments described herein may be changed in various ways and various shapes, so specific embodiments are shown in the drawings and will be described in detail in this specification. However, it should be understood that the exemplary embodiments according to the concept of the present disclosure are not limited to the embodiments which will be described hereinbelow with reference to the accompanying drawings.

It will be understood that, although the terms first and/or second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element, from another element. For instance, a first element discussed below could be termed a second element without departing from the right range of the present disclosure. Similarly, the second element could also be termed the first element.

It is to be understood that when one element is referred to as being "connected to" or "coupled to" another element, it may be connected directly to or coupled directly to another element or be connected to or coupled to another element, having the other element intervening therebetween. On the other hand, it should to be understood that when one element is referred to as being "connected directly to" or "coupled directly to" another element, it may be connected to or coupled to another element without the other element intervening therebetween. Further, the terms used herein to describe a relationship between elements, that is, "between", "directly between", "adjacent" or "directly adjacent" should be interpreted in the same manner as those described above.

Terms used in the present disclosure are used only in order to describe specific exemplary embodiments rather than limiting the present disclosure. It will be further understood that the terms "comprises" or "have" used in this specification, specify the presence of stated features, steps, operations, components, parts, or a combination thereof, but do not preclude the presence or addition of one or more other features, numerals, steps, operations, components, parts, or a combination thereof.

Unless defined otherwise, it is to be understood that all the terms used in the specification including technical and scientific terms has the same meaning as those that are understood by those who skilled in the art. It will be further understood that terms defined in dictionaries that are commonly used should be interpreted as having meanings that are consistent with their meanings in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

Hereafter, a system for analyzing walking behavior according to an embodiment of the present disclosure is described in more detail with reference the accompanying drawings.

<FIG> is a block diagram showing a system for analyzing walking behavior according to an embodiment of the present disclosure.

As shown in <FIG>, a system <NUM> for analyzing walking behavior according to an embodiment of the present disclosure includes an identifier <NUM>, a walking information obtainer <NUM>, a walking information collector <NUM>, a walking parameter extractor/classifier <NUM>, a walking result informer <NUM>, and a database <NUM>.

The identifier <NUM> may be a component that identifies a walker. The identifier <NUM> can identify a walker using the facial profile and eyeball information of the walker or can identify a walker using the facial profile, eyeball information, and voice information (voice) of the walker.

For example, when the identifier <NUM> uses eyeball information, the identifier <NUM> may be composed of a light emitter <NUM>-<NUM>, an image obtainer <NUM>-<NUM>, and an identity determiner <NUM>-<NUM> for detecting dilatation information of irises and pupils that respond to the brightness of variable light that is emitted for a predetermined time.

The light emitter <NUM>-<NUM> emits variable light to an eyeball for a predetermined time. The variable light may be emitted in shape of a specific symbol, letter, or number and the sizes of the specific symbol, letter, and number may be changed.

The image obtainer <NUM>-<NUM> detects iris and pupil regions in an eyeball image collected for a predetermined time.

The identity determiner <NUM>-<NUM> can detect an iris region and a pupil region from a plurality of eyeball image frames detected for a predetermined time, calculate and process a color variance according to the variable light into a color coordinate vector, detect expansion and contraction distances of the iris region and the pupil region according to the variable light, detect position information of a specific symbol, letter, or number projected into the iris region and changing in size, and convert the color coordinate vector, the expansion and contraction distances, and the position information into a 2D code.

The identity determiner <NUM>-<NUM> can determine the identity of a walker by determining whether the converted 2D code is the same as a pre-stored 2D code of a walker.

As another example, the identifier <NUM> may be a reader that can identify a walker by recognizing an ID card in which walker information is recorded in a QR code or an NFC tag.

Next, the walking information obtainer <NUM> may be composed of one or a plurality of 2D or depth (3D) cameras. When a plurality of cameras is used, 2D cameras or depth cameras (3D cameras) may be arranged in parallel in lines to be able to take pictures of the left side and the right side of a walker, respectively. Accordingly, two dept (3D) cameras or 2D cameras may be disposed on the same axis to face each other. However, the present disclosure is not limited to the arrangement of two or more cameras on the same axis and cameras may be randomly disposed in a space for measurement. The depth camera may be Kinect, Realsense, Astra, and the like, and an inertia sensor or motion capture equipment that can obtain 3D space information may replace the depth camera.

The sensor or equipment obtains kinematic information (2D or 3D skeleton and joint information) of a walker who is walking.

The kinematic information that is provided by the sensor or equipment can provide skeletal coordinates and skeletal reliability.

Each of the depth cameras can recognize information of the depth to a specific point from at least one or more markers or IR markers attached in a walking path of a walker.

Each of the depth cameras recognize an axis point (coordinate axis) recorded on the marker positioned at a predetermined imaging point.

The axis point is used in a calibration (combination) process of kinetic information by an image combiner to be described below.

The walking information obtainer <NUM> may include a coordinate sensor that traces positions (coordinates) of markers in a set imaging point when the imaging points of a plurality of depth cameras or 2D cameras due to external factors.

The walking information collector <NUM> collects kinematic information (3D and/or 2D skeleton information) according to walking of a target obtained by a plurality of depth cameras, 2D cameras, inertia sensors, or motion capture equipment, classifies in detail kinematic information (3D and/or 2D skeleton information) of cameras positioned in the same axial direction, and calibrates (combines) kinematic information (3D and/or 2D skeleton information) extracted from images taken at positions of the same axis.

The calibration (combination and modification) process may be process of transforming the axis of coordinates of each frame on the basis of three or more axis points obtained through markers, calculating 3D and/or 2D matrixes of the transformed axes of coordinates, and then projecting kinematic information (3D and/or 2D skeleton information and joint information) extracted from a plurality of images to one axis of coordinates.

For example, assuming that a total of four depth cameras are disposed at the left and right sides of a preset walking path and a walker has walked along the walk path for <NUM> ~ <NUM> seconds, when depth camera Nos. <NUM> and <NUM> provide kinematic information of the walker for <NUM> ~ <NUM> seconds and depth camera Nos. <NUM> and <NUM> provide kinematic information of the walker for <NUM> ~ <NUM> seconds, left/right kinematic information of each of (<NUM>, <NUM>) and (<NUM>, <NUM>) may be combined and front/rear kinematic information of (<NUM>, <NUM>) and (<NUM>, <NUM>) may be combined.

As another example, assuming that a total of four dept cameras are arranged with regular intervals at a side of a preset walking path and a walker has walked along the walk path for <NUM> ~ <NUM> seconds, kinematic information of the walker taken by the four cameras at each time section may be combined front and rear.

The walking path may be an actual space having a predetermined length or may be a machine such as a treadmill enabling continuous walking.

Next, the walking parameter extractor/classifier <NUM> may be a component that extracts walking parameters from the calibrated kinematic information (3D and/or 2D skeleton information and then classifies the types of walking using the extracted walking parameters or continuous skeleton information or uses the extracted walking parameters or continuous skeleton information as input data for machine learning.

The walking parameter extractor/classifier <NUM> can classify the extracted walking parameters or continuous skeleton information as any one of types of walking, such as a normal gait, an antlgic gait, an arthrogenic gait, a steppage gait due to weakening of the shin muscles, a lurching gait due to weakening of the gluteus maximus muscle, a trendelenburg gait due to weakening of the gluteus medius muscle, and a pathological gait, using a machine learning model, for example, an algorithm such as CNN, RNN, K-NN, K-MEANS, and RANDOM FOREST. The types of walking described herein are only examples.

For example, when walking is classified as an arthrogenic gait, there is a possibility of a problem with the ankles, the knees, or the pelves, so the classified walking type can greatly help experts propose medical opinions. When walking is classified into a pathological gait, it may be used as information for persuading the walker to visit a hospital.

The walking parameters or continuous skeleton information itself extracted from the walking parameter extractor/classifier may be used as an index for determining a problem with walking. The walking parameters may be a step, a speed, the curvature of a waist, left and right knee angles, an ankle angle, a pelvis twist angle, etc..

The walking speed is calculated as a movement distance/movement time of a body center joint, the step is calculated a z-axial distance between stop sections of both ankles (Z-axial direction: walking direction), the knee bending angle is calculated as the average value of the angles made of thigh joint-knee joint-ankle joint, and the curvature of a waist is calculated as the angle made by a vector connecting the head-body centers during walking in an absolute coordinate system.

The walking result informer <NUM> determines whether the at least one or more walking parameters come out of a recommendation reference value, or are included in a walking parameter range of abnormal walking, or are classified as the pathological gait, and then provides the result to the walker or an expert.

The walking result informer <NUM> determines whether there is a problem with walking of a walker using a recommendation value of each walking parameter.

The walking result informer <NUM> may use, as references for determination, ① how much a walking parameter comes out of a recommendation value, ② how much a walking parameter has changed from a normal state, ③ whether walking of a walker is classified as the pathological gait (whether a classified result is the pathological gait when entire joint information or walking parameters are input to a machine learning model or when walking is the pathological gait when walking parameter themselves are analyzed), etc..

For example, walking may be classified into two classes of a normal gait and a pathological gait, or may be classified into a normal gait, a pathological gait <NUM>, a pathological gait <NUM>,. , and pathological gait n. That is, the range of the normal gait may also be classified into n classes that a user wants, so the reference may depend in situations.

The walking result informer <NUM> can determine various walking problems in accordance with the number of times of walking.

<NUM>) A problem with walking speed; a problem is sensed when the difference from the average walking speed for each age group, <NUM>) a problem with a step size: it is sensed when the difference from a recommendation step (height - <NUM>) is <NUM>% or more, <NUM>) sensing of left-right step unbalance: it is sensed when the difference between a right step and a left step is <NUM> or more, <NUM>) front-rear waist curvature: it is sensed when the front/rear curvature of a waist is <NUM> degrees or more, <NUM>) left-right waist curvature: it is sensed when the left/right curvature of a waist is <NUM> degrees or more, and <NUM>) walking classification: a problem is determined when a classified walking type is a pathological gait, etc..

<NUM>) A problem with walking speed: when a walking speed is different from a normal walking speed by <NUM>% or more or gradually decreases, <NUM>) a problem with a step: when a step is different from a normal step by <NUM>% or more or gradually decreases, <NUM>) sensing of left-right step unbalance: when the difference between left and right steps is <NUM>% or more or gradually increases, <NUM>) front-rear waist curvature: when the front/rear curvature of a waist is different from a normal state by <NUM>% or more or gradually increases, <NUM>) left-right waist curvature: when the left/right curvature of a waist is different from a normal state by <NUM>% or more or gradually increases, and <NUM>) knee bending angle: when the knee bending angle is different from the normal state by <NUM>% or more or gradually decreases, etc..

Meanwhile, the walking information collector <NUM>, the walking parameter extractor/classifier <NUM>, and the walking result informer <NUM> disclosed the present disclosure may be operated in cooperation with the database <NUM>.

The database <NUM> can store the information collected, extracted, and determined by the components described above, and can upload existing stored information.

The database <NUM> may include fields or elements such that the database <NUM> can be implemented to be fitted to the purpose of the present disclosure using a relational database management system (RDBMS) such as Oracle, MYSQL, MSSQL, Infomix, Sybase, and DB2, or an object-oriented database management system (OODBMS) such as Gemston, Orion, and O2, and an XML native database such as Excelon, Tamino, and Sekainu.

Each field or element may be formed as a field or element of a superordinate concept or a subordinate concept.

<FIG> is a flowchart showing a method of analyzing walking behavior by means of a system according to an embodiment of the present disclosure.

Such a method of analyzing walking behavior (S700), first, when a walker is identified (S710), extracts kinematic information of the walker in walking images by means of the walking information obtainer <NUM> (S720).

The walking information collector <NUM> collects kinematic information (3D and/or 2D skeleton information) according to walking of a target obtained by a single or a plurality of depth cameras, 2D cameras, inertia sensors, or motion capture equipment, classifies in detail kinematic information (3D and/or 2D skeleton information) of cameras positioned in the same axial direction, and calibrates (combines) kinematic information (3D and/or 2D skeleton information) extracted from images taken at positions of the same axis (S730).

The calibration (combination and modification) process may be process of transforming the axis of coordinates of each frame on the basis of three axis points obtained through markers, calculating 3D and/or 2D matrixes of the transformed axes of coordinates, and then projecting kinematic information (3D and/or 2D skeleton information and joint information) extracted from a plurality of images to one axis of coordinates.

Meanwhile, the process S730 may include a process of calculating reliability of the coordinates of a skeleton and a process of combining only items of kinematic information of which the reliability of coordinates is a reference value or more.

Thereafter, when the process S730 is finished, the walking parameter extractor/classifier <NUM> extracts and classifies walking parameters on the basis of the calibrated kinematic information (3D and/or 2D skeleton information) (S740).

For example, the process S740 may include a process of classifying the extracted walking parameters or continuous skeleton information as any one of types of walking, such as a normal gait, an antalgic gait, an arthrogenic gait, a steppage gait due to weakening of the shin muscles, a lurching gait due to weakening of the gluteus maximus muscle, a trendelenburg gait due to weakening of the gluteus medius muscle, and a pathological gait, using a machine learning model, for example, an algorithm such as CNN, RNN, K-NN, K-MEANS, and RANDOM FOREST. The types of walking described herein are only examples.

The extracted walking parameters or continuous skeleton information itself may be used as an index for determining a problem with walking or may be input to a machine learning model. The walking parameters may include at least one or more of a walking speed, the height of a walker, movement of hands during walking, the length of a step, a front-rear waist curvature, a left-right waist curvature, a knee bending angle, angles of joints, and the times of a stance phase and a swing phase, but are not limited thereto.

When the process S740 is finished, the walking result informer <NUM> determines whether there is a problem with walking of the walker on the basis of the walking parameters (S750).

The process S750, which is a process of determining whether there is a problem with walking of a walker using a recommendation value for each walking parameter, is a process of determining various walking problems in accordance with the number of times of walking.

The process S750 may use, as references for determination, ① how much a walking parameter comes out of a recommendation value, ② how much a walking parameter has changed from a normal state, ③ whether walking of a walker is classified as the pathological gait (whether a classified result is the pathological gait when entire joint information or walking parameters are input to a machine learning model or when walking is the pathological gait when walking parameter themselves are analyzed), etc..

By using the system for analyzing walking behavior according to an embodiment of the present disclosure, there is an advantage that it is possible to more accurately measure the step, the speed, the waist curvature, the average knee bending angle, etc. of a walker and it is possible to determine whether there is a problem with walking of the walker on the basis of these items of information.

<FIG> is a view showing an exemplary computing environment in which one embodiment disclosed herein can be implemented, in which a system <NUM> including a computing device <NUM> configured to implement the one or more embodiments described above is exemplified. For example, the computing device <NUM> includes a personal computer, a server computer, a hand-held or laptop device, a mobile device (a mobile phone, a PDA, a media player, etc.), a multiprocessor system, a consumer electronic device, a mini computer, a main frame computer, and a distribution computing environment including these systems or devices, but is not limited thereto.

The computing device <NUM> may include at least one processing unit <NUM> and memory <NUM>. The processing unit <NUM>, for example, may include a central processing unit, a graphic processing unit, a microprocessor, an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Arrays (FPGA), etc., and may have a plurality of cores. The memory <NUM> may be a volatile memory (e.g., a RAM, etc.), a nonvolatile memory (e.g., a ROM, a flash memory, etc.), or a combination thereof. The computing device <NUM> may include an additional storage <NUM>. The storage <NUM> includes a magnetic storage, an optical storage, etc., but is not limited thereto. In the storage <NUM>, computer-readable instructions for implementing one or more of embodiments disclosed herein may be stored, and other computer-readable instructions for implementing an operating system, application programs, etc. may also be included. The computer-readable instructions stored in the storage <NUM> can be loaded to the memory <NUM> to be executed by a processing unit <NUM>. The computer device <NUM> may further includes an input device(s) <NUM> and an output device(s) <NUM>.

The input device(s) <NUM>, for example, may include a keyboard, a mouse, a pen, a voice input device, a touch input device, an infrared camera, a video input device, or any other input devices. The output device(s) <NUM>, for example, may include one or more displays, speakers, printers, or any other output devices. The computing device <NUM> may include an input device or an output device of another computer device as the input device(s) <NUM> or the output device (s) <NUM>. The computing device <NUM> may further include a communication connection(s) <NUM> enabling the computing device <NUM> to communicate with another device (e.g., a computing device <NUM>).

The communication connection(s) <NUM> may include a modem, a network interface card (NIC), an integrated network interface, a wireless frequency transmitter/receiver, an infrared port, a USB connection, or other interfaces for connecting the computing device <NUM> to another computing device. The communication connection(s) <NUM> may further include a wired communication or a wireless communication. The components of the computing device <NUM> may be connected by various interconnections such as a bus (e.g., a periphery component interconnection (PCI), a USB, firmware (IEEE <NUM>, an optical bus structure, etc.), and may be interconnected by a network <NUM>. The terms 'component', 'system', etc. used herein generally means hardware, a combination of hardware and software, software, or a computer-related entity that is software that is being executed.

For example, a component may be a process that is being executed by a processor, a processor, an object, an executable matter, an execution thread, a program, and/or a computer, but is not limited thereto. For example, an application that is being executed by a controller and the controller both may be components. One or more components may exist in a process and/or an execution thread, and components may be localized in one computer or may be distributed between two or more computers.

Claim 1:
A system (<NUM>) for analyzing walking behavior, comprising
an identifier (<NUM>) configured to identify a walker;
a walking information obtainer (<NUM>) configured to image a walking motion of the walker in at least one or more directions of a left, a right, and a front on the basis of an identification signal from the identifier (<NUM>), and to extract and provide kinematic information from the taken images;
a walking information collector (<NUM>) configured to combine and collect kinematic information that is 3D and/or 2D skeleton information of the taken walking images;
a walking parameter extractor/classifier (<NUM>) configured to extract at least one or more walking parameter on the basis of the combined kinematic information that is 3D and/or 2D skeleton information, and then classify the kind of the walking using the extracted walking parameter or continuous skeleton information or use the extracted walking parameter or continuous skeleton information as input data for machine learning; and
a walking result informer (<NUM>) configured to determine whether the at least one or more walking parameters come out of a recommendation reference value, or are included in a walking parameter range of abnormal walking, or are classified as a pathological gait, and then provide the result to the walker or an expert,
characterized in that the walking information obtainer (<NUM>)
recognizes markers in which three axis points are recorded and that are arranged with regular intervals on a walking path when a walking motion is imaged by a plurality of cameras, and
calculates a 3D and/or 2D transformation matrix of each kinematic information that is 3D and/or 2D skeleton information taken on the basis of the three axis points, and transforms axes of coordinates.