WALKING TRAINING SYSTEM, NON-TRANSITORY STORAGE MEDIUM STORING CONTROL PROGRAM FOR WALKING TRAINING SYSTEM AND CONTROL METHOD FOR WALKING TRAINING SYSTEM

A walking training system includes a treadmill configured to prompt a trainee to walk, a camera system configured to image the trainee at an angle of view at which at least a part of a gait of the trainee walking on the treadmill is recognizable, a determination unit configured to determine whether abnormal walking occurs in the trainee based on a plurality of predetermined determination criteria, an image processing unit configured to clip, as a video clip, a video in a predetermined period including a timing when the determination unit determines that the abnormal walking occurs from a video captured by the camera system, and perform image processing to emphasize an abnormal part of the abnormal walking determined by the determination unit, and a display control unit configured to cause a display unit to display the video clip subjected to the image processing.

INCORPORATION BY REFERENCE

The disclosure of Japanese Patent Application No. 2019-106943 filed on Jun. 7, 2019 including the specification, drawings and abstract is incorporated herein by reference in its entirety.

BACKGROUND

1. Technical Field

The disclosure relates to a walking training system, a non-transitory storage medium storing a control program for the walking training system, and a control method for the walking training system.

2. Description of Related Art

There is known an apparatus configured to image scenes showing a trainee who undergoes walking training, generate image data corresponding to a motion of a skeleton, and make determination on a walking action of the trainee (see, for example, Japanese Unexamined Patent Application Publication No. 2015-104397 (JP 2015-104397 A)).

SUMMARY

The trainee who undergoes the walking training may want to quickly and intuitively review what is inappropriate in comparison with a normal gait after the training is finished. If the trainee reviews all training scenes in a video, a checking time equal to the training time is required. Even if a fast-forward function is used, a long time is required for a search unless the trainee grasps the point where a video to be reviewed is present.

The disclosure provides a walking training system and the like in which a trainee can quickly and intuitively review training after the training is finished.

A first aspect of the disclosure relates to a walking training system. The walking training system includes a treadmill, a camera system, a determination unit, an image processing unit, and a display control unit. The treadmill is configured to prompt a trainee to walk. The camera system is configured to image the trainee at an angle of view at which at least a part of a gait of the trainee walking on the treadmill is recognizable. The determination unit is configured to determine whether abnormal walking occurs in the trainee based on a plurality of predetermined determination criteria. The image processing unit is configured to clip, as a video clip, a video in a predetermined period including a timing when the determination unit determines that the abnormal walking occurs from a video captured by the camera system, and perform image processing to emphasize an abnormal part of the abnormal walking determined by the determination unit. The display control unit is configured to cause a display unit to display the video clip subjected to the image processing.

The displayed video clip includes a scene at the timing of the determination that the abnormal walking occurs. Moreover, the abnormal part that is the cause of the determination is emphasized. Therefore, the trainee can quickly and intuitively review what is inappropriate.

In the walking training system of the aspect described above, the camera system may include a plurality of cameras configured to image the trainee in different directions. The image processing unit may be configured to determine which of videos captured by the cameras is clipped as the video clip based on a determination criterion determined to be applicable by the determination unit out of the determination criteria. The direction in which the abnormality is clearly observed differs depending on how the walking is abnormal. By determining which of the videos captured by the cameras is used for creating the video clip in association with the determination criterion, the trainee can recognize the abnormal walking more clearly.

In the walking training system of the aspect described above, the image processing unit may be configured such that, when the determination criterion determined to be applicable by the determination unit is a specific determination criterion, videos captured by the cameras are clipped as the video clips. The display control unit may be configured to cause the display unit to display the video clips in synchronization with each other. Depending on the type of the abnormal walking, the scene of the abnormal walking may be well understandable, for example, when a scene from the front and a scene from the side are observed simultaneously. When the videos are displayed in synchronization with each other, the trainee can recognize the scene of his/her abnormal walking more accurately.

In the walking training system of the aspect described above, the image processing unit may be configured to clip, as a comparative video, a video in a normal-walking period in which the determination unit does not determine that the abnormal walking occurs. The display control unit may be configured to cause the display unit to display the comparative video such that the comparative video adjoins the video clip. When the abnormal-walking video and the normal-walking video are displayed adjacent to each other, the trainee can recognize the degree of the abnormal walking more accurately.

In the walking training system of the aspect described above, the image processing unit may be configured to determine a clipping period for the video clip based on the determination criterion determined to be applicable by the determination unit. The time during which the abnormal scene continues differs depending on the type of the abnormal walking. Therefore, it is appropriate that the clipping period be determined based on the determination criterion in order that the trainee can recognize the entire scene of the abnormal walking.

In the walking training system of the aspect described above, the image processing unit may be configured to determine the clipping period for the video clip based on a walking cycle of the trainee. Since the leg motion speed differs among trainees, the trainee can recognize the entire scene of the abnormal walking more securely when the video clip is generated based on the walking cycle of the trainee.

A second aspect of the disclosure relates to a non-transitory storage medium storing a control program for a walking training system. The walking training system includes a treadmill and a camera system. The treadmill is configured to prompt a trainee to walk. The camera system is configured to image the trainee at an angle of view at which at least a part of a gait of the trainee walking on the treadmill is recognizable. The non-transitory storage medium stores instructions that are executable by one or more processors and that cause the one or more processors to perform the following functions. The functions include determining whether abnormal walking occurs in the trainee based on a plurality of predetermined determination criteria, clipping, as a video clip, a video in a predetermined period including a timing when determination is made that the abnormal walking occurs from a video captured by the camera system, performing image processing on the video clip to emphasize an abnormal part of the determined abnormal walking, and causing a display unit to display the video clip subjected to the image processing. With the walking training system in which the control program is executed, the trainee can quickly and intuitively review what is inappropriate.

A third aspect of the disclosure relates to a control method for a walking training system. The walking training system includes a treadmill and a camera system. The treadmill is configured to prompt a trainee to walk. The camera system is configured to image the trainee at an angle of view at which at least a part of a gait of the trainee walking on the treadmill is recognizable. The control method includes determining whether abnormal walking occurs in the trainee based on a plurality of predetermined determination criteria, clipping, as a video clip, a video in a predetermined period including a timing when determination is made that the abnormal walking occurs from a video captured by the camera system, performing image processing on the video clip to emphasize an abnormal part of the determined abnormal walking, and causing a display unit to display the video clip subjected to the image processing.

According to the disclosure, it is possible to provide the walking training system and the like in which the trainee can quickly and intuitively review the training after the training is finished.

DETAILED DESCRIPTION OF EMBODIMENTS

An embodiment of the disclosure is described below. The disclosure according to the claims is not limited to the embodiment described below. All constituent elements described in the embodiment are not essential to solve the problem.

FIG. 1is a schematic perspective view of a walking training apparatus100according to this embodiment. The walking training apparatus100is an example of a walking training system. A trainee900undergoes walking training by using the walking training apparatus100. The trainee900is a hemiplegia patient suffering from paralysis in one leg. The walking training apparatus100mainly includes a control panel133, a treadmill131, and a walking assistance device120. The control panel133is attached to a frame130that constitutes the entire skeleton. The trainee900walks on the treadmill131. The walking assistance device120is attached to an affected leg of the trainee900, which is a paralytic leg.

The frame130is provided upright on the treadmill131installed on a floor. The treadmill131rotates a ring-shaped belt132by using a motor (not illustrated). The treadmill131prompts the trainee900to walk. The trainee900who undergoes walking training stands on the belt132, and attempts to walk in synchronization with movement of the belt132.

The frame130supports the control panel133, a training monitor138, and the like. The control panel133houses an overall control unit210configured to control motors and sensors. For example, the training monitor138is a liquid crystal display panel configured to display information related to execution of training for the trainee900. The training monitor138is installed so that the trainee900can view the training monitor138while walking on the belt132of the treadmill131. The frame130supports a front tension unit135near the front of an area over the head of the trainee900, a harness tension unit112near the area over the head, and a rear tension unit137near the rear of the area over the head. The frame130includes handrails130ato be gripped by the trainee900.

A front camera unit141images the trainee900from the front at an angle of view at which a gait of the trainee900is recognizable. A side camera unit142images the trainee900from the side at an angle of view at which the gait of the trainee900is recognizable. Each of the front camera unit141and the side camera unit142of this embodiment includes a set of a lens and an imaging device to achieve an angle of view at which each of the front camera unit141and the side camera unit142can capture the entire body of the trainee900standing on the belt132, including his/her head. For example, the imaging device is a complementary metal-oxide semiconductor (CMOS) image sensor, and converts an optical image formed on an imaging plane into an image signal. The front camera unit141is installed near the training monitor138to face the trainee900. The side camera unit142is installed on the handrail130ato capture the trainee900from the side.

A front wire134has one end coupled to a reeling mechanism of the front tension unit135, and the other end coupled to the walking assistance device120. The reeling mechanism of the front tension unit135reels or unreels the front wire134in response to a motion of the affected leg by turning ON or OFF a motor (not illustrated). Similarly, a rear wire136has one end coupled to a reeling mechanism of the rear tension unit137, and the other end coupled to the walking assistance device120. The reeling mechanism of the rear tension unit137reels or unreels the rear wire136in response to a motion of the affected leg by turning ON or OFF a motor (not illustrated). Through the cooperative operation of the front tension unit135and the rear tension unit137, a load of the walking assistance device120is balanced so as not to burden the affected leg. Further, the swing of the affected leg is assisted depending on a set level.

An operator910who is a training assistant sets a high assistance level for a trainee suffering from severe paralysis. The operator910is a physical therapist or a doctor having authority to select, change, or add setting items of the walking training apparatus100. When the high assistance level is set, the front tension unit135reels the front wire134with a relatively great force in synchronization with a timing to swing the affected leg. If the training proceeds and no assistance is needed, the operator sets a minimum assistance level. When the minimum assistance level is set, the front tension unit135reels the front wire134with a force for canceling the self-weight of the walking assistance device120in synchronization with a timing to swing the affected leg.

The walking training apparatus100includes a safety device having a safety attachment110, a harness wire111, and the harness tension unit112as main components. The safety attachment110is a belt to be attached around the waist of the trainee900, and is fixed to the waist with, for example, a hook-and-loop fastener. The harness wire111has one end coupled to the safety attachment110, and the other end coupled to a reeling mechanism of the harness tension unit112. The reeling mechanism of the harness tension unit112reels or unreels the harness wire111by turning ON or OFF a motor (not illustrated). The safety device having this structure is configured such that, when the trainee900loses his/her balance significantly, the harness wire111is reeled and the safety attachment110supports the upper body of the trainee900in response to an instruction from the overall control unit210that detects the motion of the trainee900.

A management monitor139is a display device attached to the frame130and used by the operator910for monitoring and operation. For example, the management monitor139is a liquid crystal display panel having a touch panel superimposed on its surface. The management monitor139displays various menu items related to training settings, various parameters during the training, and results of the training. The operator910selects, changes, or adds the setting items via the touch panel, a keyboard (not illustrated), or the like. The management monitor139is installed at a position where the trainee900cannot view the display from a training position on the treadmill131. A support for the management monitor139may have a rotation mechanism configured to flip the display screen when the operator910intends to show the display screen for the trainee900.

The walking assistance device120is attached to the affected leg of the trainee900, and assists walking of the trainee900by reducing loads caused by stretching and bending a knee joint of the affected leg.FIG. 2is a schematic perspective view of the walking assistance device120. The walking assistance device120mainly includes a control unit121, a plurality of frames, and a load sensor222. The frames support respective parts of the affected leg. The load sensor222detects a load on a sole.

The control unit121includes an assistance control unit220and a motor (not illustrated). The assistance control unit220controls the walking assistance device120. The motor generates a driving force for assisting stretching and bending motions of the knee joint. The frames that support respective parts of the affected leg include an upper thigh frame122, a lower thigh frame123, a foot frame124, a front coupling frame127, and a rear coupling frame128. The lower thigh frame123is pivotably coupled to the upper thigh frame122. The foot frame124is pivotably coupled to the lower thigh frame123. The front wire134is coupled to the front coupling frame127. The rear wire136is coupled to the rear coupling frame128. The front coupling frame127extends in a lateral direction in front of an upper thigh, and both ends of the front coupling frame127are connected to the upper thigh frame122. The rear coupling frame128extends in the lateral direction behind a lower thigh, and both ends of the rear coupling frame128are connected to the lower thigh frame123extending in a vertical direction.

The upper thigh frame122and the lower thigh frame123pivot relative to each other about an illustrated hinge axis Ha. The motor of the control unit121rotates in response to an instruction from the assistance control unit220to apply a force so that the upper thigh frame122and the lower thigh frame123are opened or closed relative to each other about the hinge axis Ha. An angle sensor223housed in the control unit121is, for example, a rotary encoder, and detects an angle between the upper thigh frame122and the lower thigh frame123about the hinge axis Ha. The lower thigh frame123and the foot frame124pivot relative to each other about an illustrated hinge axis Hb. An adjustment mechanism126preadjusts an angle range in which the lower thigh frame123and the foot frame124pivot relative to each other.

The upper thigh frame122includes an upper thigh belt129. The upper thigh belt129is provided integrally with the upper thigh frame, and is attached around the upper thigh of the affected leg to fix the upper thigh frame122to the upper thigh. The upper thigh belt129restrains the entire walking assistance device120from being misaligned from the leg of the trainee900.

The load sensor222is embedded in the foot frame124. The load sensor222detects the magnitude and distribution of a vertical load on the sole of the trainee900. For example, the load sensor222is a load detection sheet of a resistance variation detection type, in which electrodes are arranged in a matrix.

Next, the system configuration of the walking training apparatus100is described.FIG. 3is a system configuration diagram of the walking training apparatus100. For example, the overall control unit210is a microprocessor (MPU), and controls the entire apparatus by executing a control program read from a storage unit217. A treadmill driving unit211includes a motor for rotating the belt132, and a drive circuit for the motor. The overall control unit210controls the rotation of the belt132by transmitting a drive signal to the treadmill driving unit211. For example, the rotation speed of the belt132is adjusted depending on a set training level.

An operation receiving unit212includes various input devices configured to receive an input operation from the trainee900or the operator910and transmit an operation signal to the overall control unit210. The touch panel superimposed on the management monitor139is an example of the operation receiving unit212. The trainee900or the operator910operates operation buttons or the touch panel of the operation receiving unit212or an attached remote control to give an instruction to turn ON or OFF the power or an instruction to start training, to input values related to settings, or to select a menu item. The operation receiving unit212may include a voice interaction device or an image recognition device. A display control unit213generates a display video and displays the display video on the training monitor138or the management monitor139in response to a control signal from the overall control unit210. For example, the display control unit213generates a video showing progress of training. Details are described later.

A tensile drive unit214includes a motor for pulling the front wire134, a drive circuit for the motor, a motor for pulling the rear wire136, and a drive circuit for the motor. The overall control unit210controls reeling of the front wire134and reeling of the rear wire136by transmitting drive signals to the tensile drive unit214. In addition to the control on the reeling operations, the overall control unit210controls tensile forces of the respective wires by controlling driving torques of the motors. For example, the overall control unit210determines a timing when the affected leg is switched from a stance phase to a swing phase based on a detection result from the load sensor222, and increases or reduces the tensile forces of the respective wires in synchronization with the timing, thereby assisting the swing of the affected leg.

A harness driving unit215includes a motor for pulling the harness wire111, and a drive circuit for the motor. The overall control unit210controls reeling of the harness wire111and a tensile force of the harness wire111by transmitting a drive signal to the harness driving unit215. For example, when the trainee900loses his/her balance significantly, the overall control unit210reels the harness wire111by a predetermined amount to restrain the trainee from falling down.

An image processing unit216generates image data by performing image processing on image signals received from the front camera unit141and the side camera unit142in response to a control signal from the overall control unit210. The image processing unit216also analyzes specific images by performing image processing on image signals received from the front camera unit141and the side camera unit142in response to an instruction from the overall control unit210. For example, the positions of both shoulders and the position of a hip joint can be detected based on information on an extracted edge. The positional information serves as basic information for evaluating conditions of a body core, legs, feet, and the like during a leg motion of the trainee900. The image processing unit216also generates a video clip by clipping a part of a video in the generated image data and performing image processing. Details are described later.

The storage unit217includes a non-volatile storage medium such as a solid-state drive. The non-volatile storage medium stores not only the control program for controlling the walking training apparatus100, but also various parameters, functions, lookup tables, and a creation criterion table described later for use in control or calculation. The storage unit217further includes a volatile storage medium such as a dynamic random access memory (DRAM). The volatile storage medium functions not only as a working memory to be used when the overall control unit210or the like executes calculation and processing, but also as a temporary memory configured to temporarily store training evaluations and video data.

A camera system140provided in the walking training apparatus100includes the front camera unit141and the side camera unit142as a plurality of camera units configured to image the trainee in different directions. The front camera unit141and the side camera unit142repeat imaging operations and output image signals to the image processing unit216in response to a control signal from the overall control unit210. The overall control unit210also serves as a function executing unit configured to execute various types of control and various types of calculation related to control. An evaluation/determination unit210aevaluates rehabilitation training executed by the trainee900, and makes determination on abnormal walking of the trainee900with reference to predetermined determination criteria. For example, the evaluation/determination unit210afunctions as a determination unit configured to determine whether a walking action is normal or abnormal by applying an analysis result from the image processing unit216to the determination criteria. Specific determination criteria and a specific determination method are described later.

As described above, the walking assistance device120is attached to the affected leg of the trainee900. The walking training apparatus100includes a communication connection interface (IF)219connected to the overall control unit210to give instructions to the walking assistance device120and receive sensor information. The walking assistance device120is provided with a communication connection IF229connected to the communication connection IF219by wire or wireless. The communication connection IF229is connected to the assistance control unit220of the walking assistance device120. Each of the communication connection IFs219and229is a communication interface such as a wireless local area network (LAN) conforming to a communication standard.

For example, the assistance control unit220is an MPU, and controls the walking assistance device120by executing a control program provided from the overall control unit210. The assistance control unit220notifies the overall control unit210of the condition of the walking assistance device120via the communication connection IFs219and229. The assistance control unit220starts or stops the walking assistance device120in response to an instruction from the overall control unit210.

A joint driving unit221includes the motor of the control unit121and a drive circuit for the motor. The assistance control unit220transmits a drive signal to the joint driving unit221to apply a force so that the upper thigh frame122and the lower thigh frame123are opened or closed relative to each other about the hinge axis Ha. This operation assists stretching and bending motions of the knee, and restrains unintended bending on the knee. As described above, the load sensor222detects the magnitude and distribution of a vertical load on the sole of the trainee900, and transmits a detection signal to the assistance control unit220.

The assistance control unit220receives and analyzes the detection signal to determine whether the affected leg is in a swing phase or a stance phase or to estimate switching between the swing phase and the stance phase. As described above, the angle sensor223detects an angle between the upper thigh frame122and the lower thigh frame123about the hinge axis Ha, and transmits a detection signal to the assistance control unit220. The assistance control unit220receives the detection signal, and calculates an open angle of the knee joint.

Next, the display on the training monitor138is described.FIG. 4illustrates a display example of an exercise screen on the training monitor138during execution of training. A status field301is provided at an uppermost part of the training monitor138. Status information in the execution of the training is displayed in the status field301. The status information includes a duration, a walking distance, a training level, and various indicators. The duration is a time from the start of execution, and is measured by using a timer (not illustrated). The walking distance is measured based on a cumulative rotation amount of the belt132rotated by the treadmill driving unit211. The training level indicates difficulty in the execution of the training, and is updated every time the trainee satisfies a preset criterion. The difficulty in the execution of the training is determined based on the rotation speed of the belt132and the assistance amount of the walking assistance device120.

A camera image302is displayed in a field other than the status field301in the training monitor138. The camera image302is an image of the entire trainee900photographed by the front camera unit141or the side camera unit142, and is displayed as a real-time video of, for example, 60 fps. The trainee900can view himself/herself as the real-time video during the execution of the training. When the image captured by the front camera unit141is displayed, the camera image302is preferably a mirror image as in the illustration from the viewpoint of visibility because the trainee900faces the training monitor138.

FIG. 5illustrates a display example of a replay screen on the training monitor138after the execution of the training. The overall control unit210displays a video clip on the training monitor138after the execution of the training. The video clip is generated by clipping a moving image captured during the execution of the training.

In the replay screen, a title310, a video clip320when the gait is appropriate, and a video clip330including a time when an abnormality is detected are superimposed on the continuously displayed camera image302. For example, the title310reads “Replay 1” to indicate a reproduced screen.

The video clip330is generated from a video captured by the camera system140so as to include a time when the evaluation/determination unit210adetermines that any predetermined abnormal-walking determination criterion is satisfied. An emphasis332is superimposed on the clipped video as a computer graphics (CG) image to emphasize an abnormal part determined by the evaluation/determination unit210a.The video clip330is obtained when the evaluation/determination unit210adetermines that a body core tilt occurs in the trainee900. Therefore, CG images of a dashed reference line and a wide body core line are superimposed on the upper body of the trainee as the emphasis332. The video clip is provided in order that the trainee900may review some scenes in the training after the execution of the training. Therefore, the video clip is not displayed as a mirror image like the video during the execution of the training inFIG. 4.

A time indication331is superimposed on the video clip330to indicate a time when the frame images are captured. An attribute icon312and a determination comment314of the video clip330are displayed near the video clip330. The attribute icon312indicates an attribute of the video clip330. Since the video clip330shows a scene of the abnormal walking, a “Bad” icon is selected. The determination comment314is superimposed when the video clip330shows the scene of the abnormal walking. A comment associated with the determination criterion determined by the evaluation/determination unit210ais selected as the determination comment314. Since the evaluation/determination unit210adetermines that the body core tilt occurs in the trainee900, a corresponding comment “Body core has tilted!” is selected.

The video clip320is obtained by clipping, from the video captured by the camera system140, a video in a normal-walking period in which the evaluation/determination unit210adoes not determine that abnormal walking occurs. The video clip320is a comparative video to the video clip330, and therefore preferably adjoins the video clip330. In this case, the video clip320and the video clip330vertically adjoin each other. Since the trainee900may desirably recognize the scene of the abnormal walking, the video clip showing the abnormal walking may be displayed in a size larger than that of the video clip showing the normal walking.

A time indication321is superimposed on the video clip320to indicate a time when the frame images are captured. An attribute icon311of the video clip320is displayed near the video clip320. The attribute icon311indicates an attribute of the video clip320. Since the video clip320shows a scene of the normal walking, a “Good” icon is selected.

The video clip320and the video clip330are simultaneously reproduced so that walking cycles are synchronized to some extent. Since the video clip320and the video clip330are reproduced in this manner in the replay screen together with auxiliary information, the trainee900can accurately recognize what is inappropriate and the degree of inappropriateness in the previous execution of the training in comparison with the appropriate condition.

Next, a clipping operation for obtaining the video clip is described. The camera system140continuously captures images from start to end of the execution of the training, and the image processing unit216generates image data. The video clip is generated by clipping frame images associated with a reproducing period from a set start point to a set end point in the image data.FIG. 6is an explanatory drawing of a reproducing period of an abnormal-walking video clip. A horizontal axis represents an elapse of time.

As in the illustration, it is assumed that the trainee900takes a posture of a reference stance phase at a time t1, swings the leg at a time t2, and stumbles at a time t3. The reference stance phase is a phase immediately before the affected leg is swung. When the evaluation/determination unit210adetects and determines that abnormal walking occurs based on a frame image captured at the time t3, the overall control unit210reads a retrospective time Tuwith reference to the creation criterion table described later, and defines a start point of the video clip at a time tSgoing back from the time t3by Tu. In the creation criterion table, the retrospective time Tuis set so that the time tSgoes back by some length from the time t1when the trainee900takes the posture of the reference stance phase.

The overall control unit210similarly reads a posterior time Tdwith reference to the creation criterion table, and defines an end point of the video clip at a time tEadvancing from the time t3by Td. In the creation criterion table, the posterior time Tdis set so that the time tEat least advances by some length from a time when a next walking cycle may be started.

When a reproducing period (=Tu+Td) from the time tSto the time tEis determined, the image processing unit216generates a video clip by clipping frame images associated with this period. At this time, the image processing unit216also executes image processing to obtain an emphasis associated with a determination criterion serving as a basis for the determination that the abnormal walking occurs at the time t3. In the image processing for obtaining the emphasis, a CG image may directly be embedded in each corresponding frame, or a CG layer to be superimposed on each frame may be created. To be exact, the timing of determination that the abnormal walking occurs differs from the timing of occurrence of the abnormal walking because a calculation time is required from the occurrence of the abnormal walking to the determination that the abnormal walking occurs. The calculation time is substantially negligible, and therefore the timing of the determination that the abnormal walking occurs may be regarded as being coincident with the timing of the occurrence of the abnormal walking. If the performance of the overall control unit210is low and the calculation time is not substantially negligible, the reproducing period may be adjusted in consideration of the calculation time so as to include a frame image at the timing of the occurrence of the abnormal walking.

FIG. 7is an explanatory drawing of a reproducing period of a normal-walking video clip. A horizontal axis represents an elapse of time similarly toFIG. 6. As described above, the normal-walking video clip is generated as the comparative video to the abnormal-walking video clip. Therefore, it is appropriate that the clipping operation for the normal-walking video clip be adjusted so that, when both video clips are reproduced in synchronization with each other, leg motions in both video clips are synchronized to some extent.

When the abnormal-walking video clip is generated as described with reference toFIG. 6, the image processing unit216searches for a predetermined continuous period in which no abnormal walking is detected. From the frame images in this period, the image processing unit216identifies a frame image showing a posture similar to that of the reference stance phase at the time t1inFIG. 6. Assuming that the time of the frame image is t1′, the image processing unit216defines a start point of the normal-walking video clip at a time tS′going back from the time t1′by Tu−(t3−t1). Further, the image processing unit216defines an end point at a time tE′advancing from the time tS′by Tu+Td.

The image processing unit216defines the start point and the end point in this manner, and generates a normal-walking video clip by clipping frame images associated with the period. When simultaneous reproduction is started for the generated normal-walking video clip and the generated abnormal-walking video clip, the frame images of the reference stance phases are displayed at the same timing. In this state, a scene showing appropriate transition of the gait and a scene showing inappropriate transition of the gait are displayed by contrast. Since the reproducing periods of both video clips are Tu+Td, both video clips remain synchronized even if reproduced repeatedly.

In the display example of the replay screen illustrated inFIG. 5, the abnormal-walking video clip is displayed adjacent to the normal-walking video clip associated with the abnormal-walking video clip, but the display example of the replay screen is not limited to the above display example.FIG. 8illustrates a display example of another replay screen on the training monitor138after the execution of the training. The replay screen illustrated inFIG. 8differs from the replay screen illustrated inFIG. 5in that an abnormal-walking video clip340generated from a video captured by the front camera unit141adjoins an abnormal-walking video clip350generated from a video captured by the side camera unit142. The video clips are generated by clipping frame images in the same period from the original videos. That is, the video clips capture scenes showing the trainee900in the same period in different directions.

An emphasis342is superimposed on the video clip340as a CG image to emphasize an abnormal part determined by the evaluation/determination unit210a.The video clip340is obtained when the evaluation/determination unit210adetermines that the trainee900stumbles. Therefore, a CG image of an enclosure line that implies explosion is superimposed as the emphasis342so as to enclose a part of the affected leg of the trainee in contact with the ground. A time indication341is superimposed on the video clip340to indicate a time when the frame images are captured.

An emphasis352is superimposed on the video clip350as a CG image similarly to the emphasis342on the video clip340. A time indication351is superimposed on the video clip350to indicate a time when the frame images are captured. As described above, the video clips340and350are generated by clipping frame images in the same period, and therefore the time indications341and351indicate the same time when simultaneous reproduction is started.

Direction indications317are superimposed near the video clips340and350in the camera image302to indicate imaging directions of the respective video clips. Since the video clip340is the video from the front camera unit141, the corresponding direction indication317reads “Front View”. Since the video clip350is the video from the side camera unit142, the corresponding direction indication317reads “Side View”.

The title310, an attribute icon315, and a determination comment316are superimposed on the camera image302similarly to the replay screen ofFIG. 5. Since both the video clips340and350are the abnormal-walking video clips, one “Bad” icon is displayed as the attribute icon315. The determination comment316is superimposed when the evaluation/determination unit210adetermines that the trainee900stumbles. Therefore, a corresponding comment “Stumbled!” is selected.

Next, the creation criterion table to be used for creating video clips is described.FIG. 9is a diagram illustrating an example of the creation criterion table to be used for creating video clips. The creation criterion table defines how video clips are generated in conjunction with determination criteria. Specifically, the creation criterion table defines whether a video clip is generated by clipping a front video captured by the front camera unit141or a side video captured by the side camera unit142(inFIG. 9, “A” represents a case where the video clip is generated, and “B” represents a case where the video clip is not generated). The creation criterion table defines the retrospective time Tufor determining a clipping start point, and the posterior time Tdfor determining a clipping end point. The creation criterion table defines whether a normal-walking video clip is generated as a comparative video (inFIG. 9, “A” represents a case where the video clip is generated as a comparative video, and “B” represents a case where the video clip is not generated as a comparative video). The creation criterion table defines positions in a frame image and types of CG image related to the image processing for obtaining an emphasis on abnormal walking. The creation criterion table defines determination comments to be superimposed.

For example, the following seven criteria can be employed as the abnormal-walking determination criteria. The first criterion (determination criterion 1) is whether a distance from the hip joint to the ankle along a walking direction is equal to or larger than a reference value when the swing phase is ended and the affected leg is landed. When the distance is smaller than the reference value, determination is made that abnormal walking occurs because the affected leg cannot swing sufficiently. The scene in this case is prominently shown both in the front video and in the side video. Therefore, two video clips are generated by clipping the front video captured by the front camera unit141and the side video captured by the side camera unit142. In this case, both video clips are placed as in the replay screen ofFIG. 8. The retrospective time Tuand the posterior time Tdare determined to be 20 seconds and 10 seconds, respectively, so that a sequence of scenes of the abnormal walking is included. The image processing for obtaining an emphasis is performed at a part of the affected leg in contact with the ground by using a G1 file containing a CG image prepared in the storage unit in advance. The determination comment is “Stumbled!” to indicate the abnormal walking shortly.

The second criterion (determination criterion 2) is whether a load on the sole is detected in the swing phase of the affected leg. When the load is detected, determination is made that abnormal walking occurs because of dragging. The scene in this case is prominently shown both in the front video and in the side video. Therefore, two video clips are generated by clipping the front video captured by the front camera unit141and the side video captured by the side camera unit142. The image processing for obtaining an emphasis is performed at a part of the affected leg in contact with the ground by using a G2 file containing a CG image prepared in the storage unit in advance. The determination comment is “Dragging!” to indicate the abnormal walking shortly.

The third criterion (determination criterion 3) is whether the bending angle of the knee joint is equal to or larger than a reference angle in the stance phase of the affected leg. When the bending angle is smaller than the reference angle, determination is made that abnormal walking occurs because the knee joint lacks a force for supporting the upper body. The scene in this case is prominently shown in the side video, and therefore a video clip is generated by clipping the side video captured by the side camera unit142. The degree of abnormality of this abnormal walking is well understandable in comparison with normal walking, and therefore a normal-walking video clip is generated as well. In this case, both video clips are placed as in the replay screen ofFIG. 5. The image processing for obtaining an emphasis is performed at the position of the knee by using a G3 file containing a CG image prepared in the storage unit in advance. The determination comment is “Knee has bent unintendedly!” to indicate the abnormal walking shortly.

The fourth criterion (determination criterion 4) is whether the distance from the hip joint to the ankle along the walking direction is equal to or larger than a reference value when the stance phase is switched to the swing phase and the affected leg starts to swing. When the distance is smaller than the reference value, determination is made that abnormal walking occurs because the weight of the upper body cannot be shifted freely. The scene in this case is prominently shown in the side video, and therefore a video clip is generated by clipping the side video captured by the side camera unit142. The degree of abnormality of this abnormal walking is well understandable in comparison with normal walking, and therefore a normal-walking video clip is generated as well. The image processing for obtaining an emphasis is performed at a position around the knee by using a G4 file containing a CG image prepared in the storage unit in advance. The determination comment is “Weight shift is insufficient!” to indicate the abnormal walking shortly.

The fifth criterion (determination criterion 5) is whether a forward tilt angle of the body core is equal to or larger than a reference angle in the stance phase of the affected leg. When the forward tilt angle is equal to or larger than the reference angle, determination is made that abnormal walking occurs because of a leaning-forward posture. The scene in this case is prominently shown in the side video, and therefore a video clip is generated by clipping the side video captured by the side camera unit142. The degree of abnormality of this abnormal walking is well understandable in comparison with normal walking, and therefore a normal-walking video clip is generated as well. The image processing for obtaining an emphasis is performed at a body core position by using a G5 file containing a CG image prepared in the storage unit in advance. The determination comment is “Leaning forward!” to indicate the abnormal walking shortly.

The sixth criterion (determination criterion 6) is whether a tilt angle of the body core toward the affected leg is equal to or larger than a reference angle in the stance phase of the affected leg. When the tilt angle is equal to or larger than the reference angle, determination is made that abnormal walking occurs because of large deflection in the lateral direction. The scene in this case is prominently shown in the front video, and therefore a video clip is generated by clipping the front video captured by the front camera unit141. The degree of abnormality of this abnormal walking is well understandable in comparison with normal walking, and therefore a normal-walking video clip is generated as well. The image processing for obtaining an emphasis is performed at the body core position by using a G6 file containing a CG image prepared in the storage unit in advance. The determination comment is “Body core has tilted!” to indicate the abnormal walking shortly.

The seventh criterion (determination criterion 7) is whether the forward tilt angle of the body core is equal to or larger than a reference angle in the swing phase of the affected leg. When the forward tilt angle is smaller than the reference angle, determination is made that abnormal walking occurs because the weight of the upper body cannot be shifted freely and the trainee is leaning backward. The scene in this case is prominently shown in the side video, and therefore a video clip is generated by clipping the side video captured by the side camera unit142. The degree of abnormality of this abnormal walking is well understandable in comparison with normal walking, and therefore a normal-walking video clip is generated as well. The image processing for obtaining an emphasis is performed at the body core position by using a G7 file containing a CG image prepared in the storage unit in advance. The determination comment is “Leaning backward!” to indicate the abnormal walking shortly. In the determination criteria 2 to 7, the retrospective time Tuand the posterior time Tdare determined to have given values in seconds, respectively, so that a sequence of scenes of the abnormal walking is included similarly to the determination criterion 1.

The reference value or the reference angle in each determination criterion may be changed depending on, for example, the age of the trainee900or the training level. The determination criterion is not limited to the determination criteria described above, but other determination criteria may be employed. For example, the determination criterion may be such that gripping of the handrail is recognized, or the harness tension unit112reels the harness wire111. The evaluation/determination unit210adetermines whether any determination criterion is satisfied by using, for example, an analysis result from the image processing unit216or outputs from the sensors and drive signals from the drive units in the walking training apparatus100depending on the determination criterion. If the camera system140includes a camera unit other than the front camera unit141and the side camera unit142, the creation criterion table may define whether a video clip is generated by using a video captured by this camera unit.

Next, description is given of control processing in a series of operations in the execution of the training.FIG. 10is a diagram illustrating a processing flow in the series of operations in the execution of the training. In Step S101, the overall control unit210starts execution of training based on a training menu specified by the operator910or the trainee900. Further, the overall control unit210starts imaging operations of the front camera unit141and the side camera unit142.

When the execution of the training is started, the overall control unit210acquires a training status in Step S102. Specifically, the overall control unit210acquires images captured by the front camera unit141and the side camera unit142, and causes the image processing unit216to perform image processing and also perform analysis processing for analyzing the gait. Further, the overall control unit210acquires output signals from the load sensor222and the like. The processing proceeds to Step S103, and the evaluation/determination unit210adetermines whether the walking action is normal or abnormal by applying the training status acquired in Step S102to the determination criteria. When the evaluation/determination unit210adetermines that the walking action is abnormal, the storage unit217stores the time of determination or frame image information associated with that time.

In Step S104, the overall control unit210determines whether the training is finished as planned. When the overall control unit210determines that the training is not finished, the processing returns to Step S102. When the overall control unit210determines that the training is finished, the processing proceeds to Step S105.

When the processing proceeds to Step S105, the image processing unit216reads, from the storage unit217, the time of abnormality determination or the frame image information associated with that time, which is stored in Step S103. Then, video clips are generated, as described above, by clipping videos in a period including the time of abnormality determination from the videos captured by the front camera unit141and the side camera unit142with reference to the creation criterion table. At this time, the video clips are subjected to image processing to emphasize an abnormal part of the abnormal walking determined in Step S103.

The processing proceeds to Step S106, and the overall control unit210causes, via the display control unit213, the training monitor138to sequentially display the video clips subjected to the image processing as illustrated inFIG. 5orFIG. 8. When the overall control unit210finishes displaying all the generated video clips, the overall control unit210terminates the series of processing steps.

In the embodiment described above, description is given of the example in which the video clips are displayed immediately after the execution of the training. The overall control unit210may store the video clips in the storage unit217, and display the video clips prior to next training to be executed by the same trainee. Even if the video clips are displayed in this manner, the trainee can quickly and intuitively review the previous execution of the training prior to a new execution of the training.

In the embodiment described above, the image processing unit216determines the clipping period for a video clip with reference to the times defined in the creation criterion table in regard to a determination criterion determined to be applicable by the evaluation/determination unit210a.The clipping period is not limited to the clipping period described above. For example, the clipping period may be determined based on a walking cycle of the trainee. For example, the evaluation/determination unit210acan detect the walking cycle of the trainee based on repetition of the posture of the reference stance phase. The image processing unit216determines a clipping period for a video clip based on the walking cycle detected by the evaluation/determination unit210aand defined cycles in the creation criterion table, such as three retrospective cycles and two posterior cycles. When the video clip is generated in this manner, for example, a trainee whose leg motion speed is low can recognize the entire scene of the abnormal walking more securely.

In the embodiment described above, the image processing unit216performs the processing of superimposing a CG image on a frame image as the image processing for emphasizing an abnormal part. The image processing for emphasizing an abnormal part is not limited to this processing. For example, the abnormal part may be trimmed and enlarged, or the outline of the abnormal part may be emphasized by a red line. The camera system140may include not only the camera units configured to capture the entire body of the trainee900, but also a camera unit configured to capture a specific part of the trainee900, such as feet. The image processing unit216may generate a video clip by clipping a video captured by the camera unit configured to capture a specific part.

The emphasizing processing may involve not only displaying an abnormal part, but also displaying suggestion to resolve abnormal walking. For example, when an abnormality of swing difficulty is detected, the toe that is insufficiently lifted off the ground is enclosed by a red circle, and a blue upward arrow is displayed near the ankle to prompt the trainee to imagine the lifting. When the suggestion to resolve the abnormal walking is displayed in a different color, the trainee can intuitively recognize how to resolve the abnormal walking.

In the embodiment described above, description is given of the configuration in which the walking training apparatus100includes the image processing unit216, the storage unit217, and the evaluation/determination unit210a,but those functions may be provided in a server connected to the walking training apparatus via a network. In this case, the server that executes a control program for controlling the walking training apparatus acquires necessary information from the walking training apparatus, determines whether abnormal walking occurs, generates a video clip, emphasizes an abnormal part, and displays the video clip on the monitor of the walking training apparatus.

In the walking training system, all the functional elements are not necessarily integrated into the walking training apparatus100. For example, the function of the evaluation/determination unit210amay be provided in a calculation unit of a server connected to the walking training apparatus100via a network. In this case, the server transmits an abnormal-walking determination result to the walking training apparatus100. The overall control unit210of the walking training apparatus100achieves display similar to that of the embodiment described above by using the transmitted determination result. Thus, the walking training system may include the server and the walking training apparatus100.