Motion information processing device

A motion information processing device for supporting a rehabilitation according to an embodiment includes obtaining circuitry and specification circuitry. The obtaining circuitry obtains image information of a subject carrying out a predetermined motion in the rehabilitation and surroundings of the subject. The specification circuitry specifies motion information of the subject carrying out the predetermined motion on the basis of a predetermined feature in the image information obtained by the obtaining circuitry.

FIELD

Embodiments described herein relate generally to a motion information processing device.

BACKGROUND

In rehabilitation, support has been provided by many experts working in cooperation for the purpose of helping those experiencing mental or physical disabilities due to various causes such as illnesses, injuries, or aging or those having congenital disorders to lead better lives. For example, rehabilitation involves support provided by many experts such as rehabilitation specialists, rehabilitation nurses, physical therapists, occupational therapists, speech-language-hearing therapists, clinical psychologists, prosthetists and orthotists, and social workers working in cooperation.

In the meantime, in recent years, development of motion capture technologies for digitally recording motions of people and objects has been advancing. Examples of systems of the motion capture technologies that are known include optical, mechanical, magnetic, and camera systems. For example, a camera system of digitally recording motions of a person by making the person wear a marker, detecting the marker by a tracker such as a camera, and processing the detected marker is known. For another example, as a system that does not use markers and trackers, a system of digitally recording motions of a person by using an infrared sensor to measure the distance from the sensor to the person and detect the size and various motions of the skeleton of the person is known. Kinect (registered trademark), for example, is known as a sensor using such a system.

DETAILED DESCRIPTION

A motion information processing device for supporting a rehabilitation according to an embodiment includes obtaining circuitry and specification circuitry. The obtaining circuitry obtains image information of a subject carrying out a predetermined motion in the rehabilitation and surroundings of the subject. The specification circuitry specifies motion information of the subject carrying out the predetermined motion on the basis of a predetermined feature in the image information obtained by the obtaining circuitry.

Hereinafter, motion information processing devices according to embodiments will be described with reference to the drawings. Note that the motion information processing devices described below may be used alone or may be embedded in a system such as a medical record system or a rehabilitation department system, for example.

First Embodiment

FIG. 1is a diagram illustrating an example of the configuration of a motion information processing device100according to a first embodiment. The motion information processing device100according to the first embodiment is a device to support rehabilitation in a medical institution, at home, in an office, or the like, for example. Note that “rehabilitation” refers to techniques and methods for developing the potentials of patients with disabilities, chronic diseases, geriatric diseases and the like receiving prolonged treatment, and restoring and promoting their vital functions and also their social functions. Examples of such techniques and methods include functional exercises for restoring and promoting vital functions and social functions. Note that examples of the functional exercises include gait training and range of motion exercise. A person who undergoes rehabilitation will be referred to as a “subject.” Examples of the subject include a sick person, an injured person, an aged person, and a handicapped person. In addition, a person who assists a subject in rehabilitation will be referred to as a “caregiver.” Examples of the caregiver include healthcare professionals such as a doctor, a physical therapist, and a nurse working at medical institutions, and a care worker, a family member, and a friend caring a subject at home, for example. Furthermore, rehabilitation will also be abbreviated as “rehab.”

As illustrated inFIG. 1, in the first embodiment, the motion information processing device100is connected to a motion information collector10.

The motion information collector10detects motion of a person, an object, or the like in a space in which rehabilitation is carried out, and collects motion information representing the motion of the person, the object, or the like. The motion information will be described in detail later in the description of processing performed by motion information generation circuitry14. For the motion information collector10, Kinect (registered trademark) is used, for example.

As illustrated inFIG. 1, the motion information collector10includes color image collection circuitry11, distance image collection circuitry12, speech recognition circuitry13, and the motion information generation circuitry14. Note that the configuration of the motion information collector10illustrated inFIG. 1is only an example, and the embodiment is not limited thereto.

The color image collection circuitry11photographs a subject such as a person, an object, or the like in a space in which rehabilitation is carried out, and collects color image information. The color image collection circuitry11detects light reflected by a surface of the subject by a photodetector, and converts visible light into an electrical signal, for example. The color image collection circuitry11then generates one frame of color image information corresponding to the photographed range by converting the electrical signal into digital data. The color image information of one frame contains photographing time information, and information of pixels contained in the frame and RGB (red, green, and blue) values with which the respective pixels are associated, for example. The color image collection circuitry11takes a moving image of the photographed range by generating multiple successive frames of color image information from visible light detected successively. Note that the color image information generated by the color image collection circuitry11may be output as a color image in which the RGB values of the pixels are arranged in a bitmap. The color image collection circuitry11has a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD), for example, as the photodetector.

The distance image collection circuitry12photographs a subject such as a person, an object, or the like in a space in which rehabilitation is carried out, and collects distance image information. The distance image collection circuitry12irradiates a surrounding area with infrared light and detects with a photodetector a reflected wave that is the irradiation wave reflected by a surface of the subject, for example. The distance image collection circuitry12then obtains the distance between the subject and the distance image collection circuitry12on the basis of a phase difference between the irradiation wave and the reflected wave and on the time from the irradiation to the detection, and generates one frame of distance image information corresponding to the photographed range. The distance image information of one frame contains photographing time information, and information of pixels contained in the photographed range and the distances between the subject and the distance image collection circuitry12with which the respective pixels are associated, for example. The distance image collection circuitry12takes a moving image of the photographed range by generating multiple successive frames of distance image information from reflected waves detected successively. Note that the distance image information generated by the distance image collection circuitry12may be output as a distance image in which shades of colors according to the distances of the pixels are arranged in a bitmap. The distance image collection circuitry12has a CMOS or a CCD, for example, as the photodetector. The photodetector may also be used in common as the photodetector used in the color image collection circuitry11. The unit of a distance calculated by the distance image collection circuitry12is meter [m], for example.

The speech recognition circuitry13collects speech therearound, and carries out determination of the direction of a speech source and speech recognition. The speech recognition circuitry13has a microphone array including multiple microphones, and carries out beamforming. Beamforming is a technique for selectively collecting speech from a particular direction. The speech recognition circuitry13determines the direction of a speech source through beamforming using the microphone array, for example. The speech recognition circuitry13also recognizes words from collected speech by using a known speech recognition technology. Specifically, the speech recognition circuitry13generates information of a word recognized according to the speech recognition technology with which the direction from which the word has been uttered and the time when the word has been recognized are associated, for example, as a speech recognition result.

The motion information generation circuitry14generates motion information indicating a motion of a person, an object, or the like. The motion information is generated by regarding a motion (gesture) of a person as a series of multiple postures (poses), for example. The outline will be explained as follows. The motion information generation circuitry14first obtains coordinates of joints forming a human body skeleton from the distance image information generated by the distance image collection circuitry12by pattern matching using human body patterns. The coordinates of the joints obtained from the distance image information are values expressed in a coordinate system of a distance image (hereinafter referred to as a “distance image coordinate system”). Thus, the motion information generation circuitry14then converts the coordinates of the joints in the distance image coordinate system into values expressed in a coordinate system of a three-dimensional space in which rehabilitation is carried out (hereinafter referred to as a “world coordinate system”). The coordinates of the joint expressed in the world coordinate system constitute skeleton information of one frame. Furthermore, skeleton information of multiple frames constitutes motion information. Hereinafter, processing performed by the motion information generation circuitry14according to the first embodiment will be described more concretely.

FIGS. 2A to 2Care diagrams for explaining processing performed by the motion information generation circuitry14according to the first embodiment.FIG. 2Aillustrates an example of a distance image taken by the distance image collection circuitry12. Note that, inFIG. 2A, an image expressed by line drawing is presented for the purpose of illustration, an actual distance image is an image expressed by color shadings according to the distances, or the like. In this distance image, each pixel has three-dimensional values, which are a “pixel position X” in the horizontal direction of the distance image, a “pixel position Y” in the vertical direction of the distance image, and a “distance Z” between the subject corresponding to the pixel and the distance image collection circuitry12. Hereinafter, coordinate values in the distance image coordinate system will be expressed by the three-dimensional values (X, Y, Z).

In the first embodiment, the motion information generation circuitry14stores human body patterns corresponding to various postures through learning, for example, in advance. Each time distance image information is generated by the distance image collection circuitry12, the motion information generation circuitry14acquires the generated distance image information of each frame. The motion information generation circuitry14then carries out pattern matching on the acquired distance image information of each frame using the human patterns.

Here, the human patterns will be described.FIG. 2Billustrates an example of the human patterns. In the first embodiment, the human patterns are patterns used in pattern matching with the distance image information, and are thus expressed in the distance image coordinate system and have information on the surfaces of human bodies (hereinafter referred to as “human body surfaces”) similarly to a person drawn in a distance image. A human body surface corresponds to the skin or the surface of clothing of the person, for example. Furthermore, a human body pattern has information on joints forming human skeleton as illustrated inFIG. 2B. Thus, in a human pattern, relative positions of a human body surface and the joints are known.

In the example illustrated inFIG. 2B, the human body pattern has information on 20 joints, from a joint2ato a joint2t. The joint2acorresponds to the head, the joint2bcorresponds to the center of the shoulders, the joint2ccorresponds to the waist, and the joint2dcorresponds to the center of the hip. The joint2ecorresponds to the right shoulder, the joint2fcorresponds to the right elbow, the joint2gcorresponds to the right wrist, and the joint2hcorresponds to the right hand. The joint2icorresponds to the left shoulder, the joint2jcorresponds to the left elbow, the joint2kcorresponds to the left wrist, and the joint2lcorresponds to the left hand. The joint2mcorresponds to the right hip, the joint2ncorresponds to the right knee, the joint2ocorresponds to the right ankle, and the joint2pcorresponds to the tarsus of the right foot. The joint2qcorresponds to the left hip, the joint2rcorresponds to the left knee, the joint2scorresponds to the left ankle, and the joint2tcorresponds to the tarsus of the left foot.

While a case in which the human body pattern has information on 20 joints is illustrated inFIG. 2B, the embodiment is not limited thereto, and the positions and the number of joints may be arbitrarily be set by an operator. For example, for capturing only a change in the motion of the limbs, information on the joint2band the joint2cof the joints2ato2dneed not be acquired. For capturing a change in the motion of the right hand in detail, joints of the fingers of the right hand may be newly set in addition to the joint2h. Note that, although the joint2a, the joint2h, the joint2l, the joint2p, and the joint2tinFIG. 2Bare at distal portions of bones and are thus different from what are actually called joints, these points will be referred to as joints for the purpose of explanation since the points are important points for indicating the positions and orientations of the bones.

The motion information generation circuitry14carries out pattern matching with the distance image information of each frame by using such human body patterns. For example, the motion information generation circuitry14carries out pattern matching between the human body surface of the human body pattern illustrated inFIG. 2Band the distance image illustrated inFIG. 2Ato extract a person in a certain posture from the distance image information. In this manner, the motion information generation circuitry14obtains the coordinates of the human body surface of the person drawn in the distance image. Furthermore, as described above, in a human pattern, relative positions of a human body surface and joints are known. The motion information generation circuitry14thus calculates the coordinates of the joints in the person drawn in the distance image from the coordinates of the human body surface of the person. In this manner, as illustrated inFIG. 2C, the motion information generation circuitry14obtains the coordinates of the joints forming the human body skeleton from the distance image information.

Note that the coordinates of the joints obtained here are coordinates in the distance image coordinate system. Note that the motion information generation circuitry14may use information indicating relative positions of the joints supplementarily in carrying out the pattern matching. The information indicating the relative positions of the joints contains connections between joints (“connection between the joint2aand the joint2b,” for example), and the ranges of motion of the joints, for example. A joint is a part connecting two or more bones. The angle between bones changes with a change in posture, and the ranges of range are different for different joints. A range of motion is expressed by the largest value and the smallest value of the angle between bones that the joint connects, for example. In learning a human body pattern, the motion information generation circuitry14also learns the ranges of motion of the joints and stores the learned ranges of motion in association with the respective joints, for example.

Subsequently, the motion information generation circuitry14converts the coordinates of the joints in the distance image coordinate system into values expressed in the world coordinate system. The world coordinate system refers to a coordinate system of a three-dimensional space in which rehabilitation is carried out, such as a coordinate system with the origin at the position of the motion information collector10, the x-axis in the horizontal direction, the y-axis in the vertical direction, and the z-axis in a direction perpendicular to the xy plane. Note that a coordinate value in the z-axis direction may be referred to as a “depth.”

Here, processing of conversion from the distance image coordinate system to the world coordinate system will be described. In the first embodiment, it is assumed that the motion information generation circuitry14stores in advance a conversion formula for conversion from the distance image coordinate system to the world coordinate system.

Coordinates in the distance image coordinate system and an entrance angle of reflected light associated with the coordinates are input to this conversion formula and coordinates in the world coordinate system are output therefrom, for example. The motion information generation circuitry14inputs coordinates (X1, Y1, Z1) of a joint and the entrance angle of reflected light associated with the coordinates to the conversion formula, and converts the coordinates (X1, Y1, Z1) of the joint into coordinates (x1, y1, z1) of the world coordinate system, for example. Note that, since the relation between the coordinates in the distance image coordinate system and the entrance angle of reflected light is known, the motion information generation circuitry14can input the entrance angle associated with the coordinates (X1, Y1, Z1) into the conversion formula. Although a case in which the motion information generation circuitry14converts coordinates in the distance image coordinate system into coordinates in the world coordinate system has been described here, the motion information generation circuitry14may alternatively convert coordinates in the world coordinate system into coordinates in the distance image coordinate system.

The motion information generation circuitry14then generates skeleton information from the coordinates of the joints expressed in the world coordinate system.FIG. 3is a table illustrating an example of the skeleton information generated by the motion information generation circuitry14. The skeleton information of each frame contains photographing time information of the frame and the coordinates of the joints. The motion information generation circuitry14generates skeleton information containing joint identification information and coordinate information associated with each other as illustrated inFIG. 3, for example. Note that the photographing time information is not illustrated inFIG. 3. The joint identification information is identification information for identifying a joint, and is set in advance. For example, joint identification information “2a” corresponds to the head, and joint identification information “2b” corresponds to the center of the shoulders. The other joint identification information data similarly indicate the respective corresponding joints. The coordinate information indicates coordinates of each joint in each frame in the world coordinate system.

In the first row ofFIG. 3, the joint identification information “2a” and the coordinate information “(x1, y1, z1)” are associated. Specifically, the skeleton information ofFIG. 3indicates that the head is present at the position of coordinates (x1, y1, z1) in a certain frame. In addition, in the second row ofFIG. 3, the joint identification information “2b” and the coordinate information “(x2, y2, z2)” are associated. Specifically, the skeleton information ofFIG. 3indicates that the center of the shoulders is present at the position of coordinates (x2, y2, z2) in a certain frame. Similarly for the other joints, the skeleton information indicates that each joint is present at a position expressed by the corresponding coordinates in a certain frame.

In this manner, the motion information generation circuitry14carries out pattern matching on the distance image information of each frame each time the distance image information of each frame is acquired from the distance image collection circuitry12, and converts the coordinates from the distance image coordinate system into those in the world coordinate system to generate the skeleton information of each frame. The motion information generation circuitry14then outputs the generated skeleton information of each frame to the motion information processing device100to store the skeleton information in motion information storage circuitry131, which will be described later.

Note that the processing of the motion information generation circuitry14is not limited to the technique described above. For example, although a technique in which the motion information generation circuitry14carries out pattern matching using human body patterns has been described above, the embodiment is not limited thereto. For example, a technique in which patterns of each part is used instead of or in addition to the human body patterns may be used.

Furthermore, for example, although a technique in which the motion information generation circuitry14obtains coordinates of joints from the distance image information has been described above, the embodiment is not limited thereto. For example, a technique in which the motion information generation circuitry14obtains coordinates of joints by using color image information in addition to the distance image information may be used. In this case, the motion information generation circuitry14carries out pattern matching between a human body pattern expressed in a coordinate system of a color image and the color image information, and obtains coordinates of the human body surface from the color image information, for example. The coordinate system of the color image does not include information corresponding to the “distance Z” in the distance image coordinate system. Thus, the motion information generation circuitry14obtains the information on the “distance Z” from the distance image information, for example, and obtains coordinates of joints in the world coordinate system through a calculation process using these to information data.

The motion information generation circuitry14also outputs color image information generated by the color image collection circuitry11, distance image information generated by the distance image collection circuitry12, and a speech recognition result output from the speech recognition circuitry13, where necessary, to the motion information processing device100to store the color image information, the distance image information, and the speech recognition result in the motion information storage circuitry131, which will be described later. Note that a pixel position in the color image information and a pixel position in the distance image information can be associated with each other in advance according to the positions of the color image collection circuitry11and the distance image collection circuitry12and the photographing direction. Thus, a pixel position in the color image information and a pixel position in the distance image information can also be associated with the world coordinate system calculated by the motion information generation circuitry14. Furthermore, the height and the lengths of body parts (the length of an arm, the length of the abdomen, etc.) can be obtained or the distance between two pixels specified on a color image can be obtained by using the association and a distance [m] calculated by the distance image collection circuitry12. Similarly, the photographing time information in the color image information and the photographing time information in the distance image information can also be associated with each other in advance. In addition, the motion information generation circuitry14can refer to the speech recognition result and the distance image information, and if a joint2ais present about the direction in which a word recognized through speech recognition at certain time has been uttered, can output the word as a word uttered by a person having the joint2a. Furthermore, the motion information generation circuitry14also outputs information indicating relative positions of the joints, where necessary, to the motion information processing device100to store the information in the motion information storage circuitry131, which will be described later.

The motion information generation circuitry14also generates depth image information of one frame corresponding to the photographed range by using a depth that is a coordinate value in the z-axis direction of the world coordinate system. The depth image information of one frame contains photographing time information, and information of pixels contained in the photographed range with which the depths associated with the respective pixels are associated, for example. In other words, the depth image information associates the pixels with depth information instead of the distance information with which the pixels in the distance image information are associated, and can indicate the pixel positions in the distance image coordinate system similar to that of the distance image information. The motion information generation circuitry14outputs the generated depth image information to the motion information processing device100to store the depth image information in the motion information storage circuitry131. Note that the depth image information may be output as a depth image in which shades of colors according to the depths of the pixels are arranged in a bitmap.

Although a case in which motion of one subject is detected by the motion information collector10has been described here, the embodiment is not limited thereto. If multiple subjects are included in the detection range of the motion information collector10, the motion information collector10may detect motions of multiple subjects. If multiple subjects are photographed in distance image information of the same frame, the motion information collector10associates the skeleton information data of the multiple subjects generated by the distance image information of the same frame, and outputs the associated skeleton information data as motion information to the motion information processing device100.

Note that the configuration of the motion information collector10is not limited to the configuration described above. For example, in a case where motion information is generated by detecting motion of a person through another motion capture technology such as an optical, mechanical, or magnetic technology, the motion information collector10need not necessarily include the distance image collection circuitry12. In such a case, the motion information collector10includes a marker to be worn by a human body to detect the motion of a person and a sensor for detecting the marker as a motion sensor. The motion information collector10then detects the motion of the person by using the motion sensor and generates motion information. The motion information collector10also associates pixel positions of the color image information and coordinates of the motion information with each other by using the positions of the marker contained in the image photographed by the color image collection circuitry11, and outputs the association result to the motion information processing device100where necessary. In addition, for example, if the motion information collector10does not output the speech recognition result to the motion information processing device100, the motion information collector10need not have the speech recognition circuitry13.

Furthermore, although the motion information collector10outputs coordinates in the world coordinate system as the skeleton information in the embodiment described above, the embodiment is not limited thereto. For example, the motion information collector10may output coordinates in the distance image coordinate system before conversion, and the conversion from the distance image coordinate system to the world coordinate system may be carried out in the motion information processing device100where necessary.

The description refers back toFIG. 1. The motion information processing device100performs processing for supporting rehabilitation by using the motion information output from the motion information collector10. Specifically, the motion information processing device100prevents false recognition of a subject in supporting rehab by using image information containing the motion information collected by the motion information collector10.

As described above, among various exercises that have been carried out as functional exercises in rehab, in rehab such as gait training that is carried out using a wide space, for example, the motion information collector10described above may collect information on a person or an object (such as a chair and equipment) other than the subject carrying out rehab. In particular, in a small hospital or a clinic (a medical office), it is difficult to allocate a large space reserved for rehab, and the motion information collector10may collect information of a person or an object other than the subject. As a result, the motion information processing device100may carry out processing on the basis of false recognition of information on the person or the object other than the subject as information on the subject, which may interfere with processing in various rehab supports. The motion information processing device100according to the present embodiment is therefore configured to enable prevention of false recognition of a subject.

The motion information processing device100is an information processing device such as a computer or a workstation, for example, and includes output circuitry110, input circuitry120, storage circuitry130, and control circuitry140as illustrated inFIG. 1.

The output circuitry110outputs various information data for supporting rehabilitation. For example, the output circuitry110outputs various information data for supporting rehab by using the motion information of the subject carrying out the rehab. Specifically, the output circuitry110outputs various processing results using the motion information of a subject specified by the control circuitry140, which will be described later. The output circuitry110also displays a graphical user interface (GUI) for an operator who operates the motion information processing device100to input various request by using the input circuitry120, displays display information generated by the motion information processing device100, or outputs an alarm. The output circuitry110is a monitor, a speaker, a headphone, or a headphone part of a headset, for example. The output circuitry110may be a display that is worn on the body of a user such as a spectacle type display or a head mounted display.

The input circuitry120receives input of various information data for supporting rehabilitation. For example, the input circuitry120receives input of various information data for preventing false recognition of a subject. For example, the input circuitry120receives input of various requests (such as a request for setting a predetermined threshold for determining whether or not what is recognized is the subject, a request for selecting various information data, and a measurement request for measurement on the GUI) from the operator of the motion information processing device100, and transfers the received requests to the motion information processing device100. The input circuitry120is a mouse, a keyboard, a touch command screen, a trackball, a microphone, or a microphone part of a headset, for example. The input circuitry120may be a sensor for acquiring biological information such as a sphygmomanometer, a heart rate monitor, or a clinical thermometer.

The storage circuitry130is a storage device such as a semiconductor memory device such as a random access memory (RAM) and a flash memory, a hard disk device, or an optical disk device, for example. The control circuitry140can be an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA), or can be implemented in a predetermined program executed by a central processing unit (CPU).

The configuration of the motion information processing device100according to the first embodiment has been described above. With such a configuration, the motion information processing device100according to the first embodiment prevents false recognition of a subject by the configuration described hereinafter in detail. In the following embodiment, an example in which gait training is carried out as rehab will be described.FIG. 4is a diagram illustrating an example of a detailed configuration of the motion information processing device100according to the first embodiment. First, the storage circuitry130in the motion information processing device100will be described. As illustrated inFIG. 4, in the motion information processing device100, the storage circuitry130includes the motion information storage circuitry131, for example.

The motion information storage circuitry131stores various information data collected by the motion information collector10. Specifically, the motion information storage circuitry131stores motion information generated by the motion information generation circuitry14, and motion information determined to be the motion information of the subject by the control circuitry140, which will be described later, among the motion information generated by the motion information generation circuitry14. More specifically, the motion information storage circuitry131stores skeleton information of each frame generated by the motion information generation circuitry14in the motion information determined to be the motion information of the subject by the control circuitry140. Note that the motion information storage circuitry131can also associate color image information, distance image information, and a speech recognition result output by the motion information generation circuitry14for each frame and store the association result. Note that details of the motion information determined to be the motion information of the subject by the control circuitry140and stored in the motion information storage circuitry131will be described later.

Next, details of the control circuitry140in the motion information processing device100will be described. As illustrated inFIG. 4, in the motion information processing device100, the control circuitry140includes obtaining circuitry141, specification circuitry142, and a display control circuitry143, for example.

The obtaining circuitry141obtains image information of a subject carrying out a predetermined motion and the surroundings of the subject. Specifically, the obtaining circuitry141obtains motion information collected by the motion information collector10and stored in the motion information storage circuitry131. For example, the obtaining circuitry141obtains color image information, distance image information, a speech recognition result, skeleton information, and the like stored for each frame in the motion information storage circuitry131.

Note that the motion information collector10may collect information on a person or an object other than the subject carrying out rehab as described above. In particular, in such a case in which rehab is carried out using almost the entirety of a small space, a person or an object other than the subject is likely to be included in the area from which motion information is collected, and the motion information collector10may collect information on such person or object and store the information in the motion information storage circuitry131. The obtaining circuitry141obtains motion information of each frame containing such information. Specifically, if skeleton information of a person or an object other than the subject has been collected by the motion information collector10, the obtaining circuitry141obtains motion information containing skeleton information of the subject and skeleton information of the person or object other than the subject in each frame. In such a case, information on the subject and information of the person or object other than the subject are also contained in color image information and distance image information, and the obtaining circuitry141can obtain such information.

FIG. 5is a diagram illustrating an example of skeleton information obtained by the obtaining circuitry141according to the first embodiment. Note thatFIG. 5illustrates an example in which the motion information collector10has collected skeleton information of a chair placed in the area from which motion information is collected in addition to skeleton information corresponding to the subject carrying out gait training in the area in the direction indicated by an arrow inFIG. 5. For example, the motion information collector10collects skeleton information containing joints2a1to2t1corresponding to parts of a human body and skeleton information containing joints (2a2,2d2,2e2,2i2,2m2,2n2,2o2,2q2,2r2, and2s2) for each frame as illustrated inFIG. 5, and store the collected skeleton information data in the motion information storage circuitry131.

In such a case, the obtaining circuitry141obtains two type of skeleton information (skeleton information containing the joints2a1to2t1and the joints (2a2,2d2,2e2,2i2,2m2,2n2,2o2,2q2,2r2, and2s2)) each contained in all the frames collected during a series of motions of the subject in gait training. Note that the obtaining circuitry141can also obtain color image information, distance image information and the like in each frame in addition to the skeleton information described above.

The description refers back toFIG. 4, in which the specification circuitry142specifies motion information of the subject carrying out a predetermined motion on the basis of predetermined features in the image information obtained by the obtaining circuitry141. Specifically, the specification circuitry142extracts subject information indicating the subject as the predetermined feature from the image information obtained by the obtaining circuitry141, and specifies the motion information of the subject carrying out the predetermined motion on the basis of the extracted subject information. For example, the specification circuitry142extracts information indicating a moving object contained in the image information as the subject information, and specifies motion information of the subject carrying out the predetermined motion on the basis of the extracted subject information. Note that the specification circuitry142determines an object whose position changes in a three-dimensional space around the subject to be a moving object in the image information obtained over time by the obtaining circuitry141.

FIGS. 6A and 6Bare diagrams for explaining an example of processing performed by the specification circuitry142according to the first embodiment.FIGS. 6A and 6Billustrate an example of processing using the motion information collected by the motion information collector10illustrated inFIG. 5. The specification circuitry142reads out an image of a “T-second” frame and a “(T+t)-second” frame of the motion information collected by the motion information collector10as illustrated inFIG. 6A, for example. The specification circuitry142then extracts a moving object (a moving person or object) in the three-dimensional space from which the motion information has been collected from the read image.

In one example, the specification circuitry142extracts the moving object from a difference between a “T-second image” that is an image of the T-second frame and a “(T+t)-second image” that is an image of the (T+t)-second frame, and sets a predetermined area containing the extracted moving object as a recognition target area R1for specifying the motion information as illustrated inFIG. 6B. If gait training is carried out toward the motion information collector10as illustrated inFIG. 5, the specification circuitry142obtains the difference in depth between the pixels in the “T-second image” and those in the “(T+t)-second image” to extract pixels whose depth information has changed, for example. For an object such as a chair that does not move, the difference will be “0” since the depth information does not change, and the object can be determined to be an object other than the subject and deleted. The specification circuitry142then carries out an area expansion process such as morphological operation on pixels whose depth information has changed, for example, to set the recognition target area R1.

Specifically, the specification circuitry142calculates the difference between the “T-second image” and the “(T+t)-second image” to extract pixels corresponding to the moving object in the image. The specification circuitry142then carries out the area expansion process on the extracted pixels to set the recognition target area R1containing the entire moving object. The specification circuitry142then specifies motion information contained in the set recognition target area R1as the motion information of the subject. For example, the specification circuitry142specifies skeleton information containing the joints2a1to2t1illustrated inFIG. 5as the motion information of the subject carrying out rehab.

Although a case in which the subject carries out gait training toward the motion information collector10has been described in the example above, the embodiment is not limited thereto, and there may be a case in which gait training is carried out in a manner that the subject moves across the motion information collector10, for example. In such a case, the recognition target area can also be set by calculating the difference in depth similarly to the example described above and extracting the pixels whose depth information has been changed.

Furthermore, although a case in which a moving object is extracted on the basis of a change in depth information has been described in the example above, the embodiment is not limited thereto, and there may be a case in which pixel values in color image information are used, for example. In such a case, the recognition target area is set by obtaining the difference between pixel values of a color image of a “T-second” frame and pixel values of a color image of a “(T+t)-second” frame and extracting pixels whose pixel values have been changed, for example.

Alternatively, the recognition target area can also be set by collecting temperature information for each frame through thermography, calculating the difference in the collected temperature information between a “T-second image” and a “(T+t)-second image,” and extracting pixels whose temperature information has been changed, for example.

In the example above, a case in which the moving object is extracted on the basis of the difference between images of two frames, which are the “T-second image” and the “(T+t)-second image” has been described. The embodiment, however, is not limited thereto, and there may be a case in which a moving object is extracted on the basis of differences between images of three or more frames, for example. In one example, a case in which the differences in depth between a “T-second image,” a “(T+t)-second image,” and a “(T+2t)-second image” are obtained, an area in which the depth becomes gradually shallower (or deeper), an area in which the depth changes in one direction over time, or the like is extracted, and the extracted area is set as the recognition target area may be used. As a result, a subject carrying out a regular motion such as gait training can be properly identified even if motion information of a person other than the subject carrying out gait training is collected by the motion information collector10, for example.

The specification circuitry142specifies the motion information (skeleton information) contained in the recognition target area R1set by any of various methods described above as the motion information (skeleton information) of the subject. The specification circuitry142then stores the specified skeleton information in association with color image information, distance image information or the like collected by the motion information collector10into the motion information storage circuitry131.

FIG. 7is a table illustrating an example of motion information stored in the motion information storage circuitry131by the specification circuitry142according to the first embodiment. The specification circuitry142stores motion information in which a name number, a date, and motion information are associated with each name as illustrated inFIG. 7in the motion information storage circuitry131. Note that the “name number” refers to an identifier for uniquely identifying a subject, and is assigned to each name. The “date” refers to the date and time when the subject has carried out rehab (such as gait training). The “motion information” refers to information collected by the motion information collector10.

The specification circuitry142stores “name: A, name number: 1, date: 20120801_1, motion information: color image information, distance image information, speech recognition result, skeleton information,” as illustrated inFIG. 7, for example. The above information indicates that motion information containing “color image information,” “distance image information,” a “speech recognition result,” and “skeleton information” is stored as motion information in the “first” gait training carried out by a person with a “name: A” whose “name number” is “1” on “Aug. 1” in “2012.” Note that the specification circuitry142associates and stores the skeleton information specified as the skeleton information of the subjects as the skeleton information illustrated inFIG. 7. The specification circuitry142also stores the “color image information,” the “distance image information,” the “speech recognition result,” and the “skeleton information” of each of all the frames photographed during the gait training as the motion information illustrated inFIG. 7in association with time in time series.

The specification circuitry142also stores “name: A, name number: 1, date: 20120801_2, motion information: color image information, distance image information, speech recognition result, skeleton information,” as illustrated inFIG. 7, for example. Specifically, the specification circuitry142similarly stores motion information in the “second” gait training carried out by the person with the “name: A” on “Aug. 1” in “2012”. Note that, similarly in the “second” gait training, the skeleton information specified as the skeleton information of the subject is associated and stored.

The specification circuitry142also stores motion information containing “color image information,” “distance image information,” a “speech recognition result,” and “skeleton information” for a person with “name: B, name number: 2,” as illustrated inFIG. 7. In this manner, the specification circuitry142stores motion information in gait training collected for each subject in association with the subject. Note that the motion information illustrated inFIG. 7is merely an example. Specifically, the specification circuitry142can further associate and store information other than the “color image information,” the “distance image information,” the “speech recognition result,” and the “skeleton information” illustrated inFIG. 7, and if the motion information collector10does not include the speech recognition circuitry13, for example, stores the motion information without the speech recognition result.

The “color image information” and the “distance image information” contained in the motion information contain image data in bitmap, JPEG, or other binary formats, a link to the image data, or the like. Furthermore, the “speech recognition result” contained in the motion information may contain speech data or a link to the speech data in addition to the recognition information described above.

The description refers back toFIG. 4, in which the display control circuitry143performs control to display a display image in which information indicating the subject is superimposed on the image information at the position corresponding to the motion information specified by the specification circuitry142. Specifically, the display control circuitry143displays the display image by using the motion information stored in the motion information storage circuitry131by the specification circuitry142.FIG. 8is a diagram illustrating an example of the display image displayed by the display control circuitry143according to the first embodiment. For example, the display control circuitry143performs control to display a display image in which dots and lines indicating the skeleton information specified by the specification circuitry142are superimposed on color image information collected by the motion information collector10on the output circuitry110, as illustrated inFIG. 8.

The display information in which skeleton information is properly superimposed on the subject contained in color image information can be displayed by superimposing the skeleton information specified as the motion information of the subject by the specification circuitry142on the color image information in this manner.

Next, processing of the motion information processing device100according to the first embodiment will be described with reference toFIG. 9.FIG. 9is a flowchart illustrating procedures of the processing performed by the motion information processing device100according to the first embodiment. As illustrated inFIG. 9, in the motion information processing device100according to the first embodiment, the obtaining circuitry141obtains motion information of a subject (step S101). The specification circuitry142then determines whether or not a false recognition preventing function for preventing false recognition of a subject is ON (step S102).

If the false recognition preventing function is ON (Yes in step S102), the specification circuitry142obtains a “T-second image” and a “(T+t)-second image,” and calculates “T-second image—(T+t)-second image” (step S103). The specification circuitry142then carries out the area expansion process to extract the recognition target area (step S104).

Thereafter, the specification circuitry142specifies motion information (skeleton information) in the extracted recognition target area (step S105), and stores the specified motion information (skeleton information) in the motion information storage circuitry131(step S106).

If the false recognition preventing function is not ON in step S102(No in step S102), the motion information is stored in the motion information storage circuitry131without the specification by the specification circuitry142(step S106).

The display control circuitry143then performs control to generate a display image on the basis of the motion information stored in the motion information storage circuitry131and display the display image on the output circuitry110(step S107). In the example of processing described above, a case in which the recognition target area is extracted when the skeleton information is saved and the motion information in the extracted recognition target area is specified has been described. The embodiment, however, is not limited thereto, and there may be a case in which the extraction is performed at arbitrary timing, for example. For example, the extraction may be performed in real time when the motion information is obtained.

As described above, according to the first embodiment, the obtaining circuitry141obtains image information of a subject carrying out a predetermined motion and the surroundings of the subject. The specification circuitry142specifies motion information of the subject carrying out the predetermined motion on the basis of predetermined features in the image information obtained by the obtaining circuitry141. The motion information processing device100according to the first embodiment can therefore specify the motion information of the subject from information collected by the motion information collector10, which can prevent false recognition of the subject.

As a result, the motion information processing device100allows proper rehab support using the motion information (skeleton information) of the subject in rehab support using the motion information collected by the motion information collector10.

Furthermore, according to the first embodiment, the specification circuitry142extracts subject information indicating the subject as the predetermined feature from the image information obtained by the obtaining circuitry141, and specifies the motion information of the subject carrying out the predetermined motion on the basis of the extracted subject information. The motion information processing device100according to the first embodiment can therefore properly extract the motion information of the subject contained in the information collected by the motion information collector10.

Furthermore, according to the first embodiment, the specification circuitry142specifies information indicating a moving object contained in image information as subject information, and specifies motion information of the subject carrying out a predetermined motion on the basis of the extracted subject information. The motion information processing device100according to the first embodiment therefore allows easy extraction of motion information of the subject from information collected by the motion information collector10.

Furthermore, according to the first embodiment, the specification circuitry142determines an object whose position changes in a three-dimensional space around the subject to be a moving object in the image information obtained over time by the obtaining circuitry141. The motion information processing device100according to the first embodiment therefore allows easy extraction of a moving object.

Furthermore, according to the first embodiment, the display control circuitry143performs control to display a display image in which information indicating the subject is superimposed on the image information at a position corresponding to the motion information specified by the specification circuitry142. The motion information processing device100according to the first embodiment therefore allows provision of a display image in which information is properly superimposed on a subject in a color image.

Second Embodiment

In the first embodiment described above, a case in which a moving object is used as a predetermined feature contained in image information is used has been described. In a second embodiment, a case in which a condition according to skeleton information of a human body is used will be described. Note that determination that will be described below may be used additionally in specifying motion information of a subject described in the first embodiment, or may be used alone.

FIG. 10is a diagram illustrating an example of a detailed configuration of the motion information processing device100according to the second embodiment. The motion information processing device100according to the second embodiment illustrated inFIG. 10is different from the motion information processing device100according to the first embodiment illustrated inFIG. 4in that the storage circuitry130additionally includes a recognition information storage circuitry132and in the processing performed by the specification circuitry142. Hereinafter, the description will be focused mainly on these differences.

The recognition information storage circuitry132stores recognition information for specifying skeleton information of a subject among skeleton information contained in motion information collected by the motion information collector10. Specifically, the recognition information storage circuitry132stores information for identifying skeleton information indicating a human body skeleton among skeleton information data contained in motion information.FIGS. 11A and 11Bare tables illustrating examples of recognition information stored by the recognition information storage circuitry132according to the second embodiment.

For example, the recognition information storage circuitry132stores recognition information in which a part and a range are associated as illustrated inFIG. 11A. Note that the “part” presented inFIG. 11Arefers to a part of a human body. Furthermore, the “range” presented inFIG. 11Arefers to a possible range of a human body skeleton. For example, the recognition information storage circuitry132stores recognition information of “part: height [cm], range: 100 cm-200 cm” as illustrated inFIG. 11A. The aforementioned information indicates that skeleton information with a “height” of “100 cm-200 cm” is skeleton information of a human body.

Similarly, the recognition information storage circuitry132stores recognition information of “part: arm length [cm], range: 60 cm-80 cm” as illustrated inFIG. 11A. In this manner, the recognition information storage circuitry132stores recognition information in which a range for identifying a part is set for each of various parts of a human body.

The recognition information storage circuitry132can also store recognition information in which a part, an average, and a variance are associated as illustrated inFIG. 11B. Note that the “part” presented inFIG. 11Brefers to a part of a human body. The “average” presented inFIG. 11Brefers to an average of the associated part as the skeleton of a human body. The “variance” presented inFIG. 11Brefers to a variance of the associated part from the average.

For example, the recognition information storage circuitry132stores recognition information of “part: shoulder length [cm], average: 60 cm, variance: “1σ” as illustrated inFIG. 11B. The aforementioned information indicates that the “average” of the “shoulder length” as the human body skeleton is “60 cm” and the degree of “variance (possible range of the shoulder length)” is “1σ.” Similarly, the recognition information storage circuitry132stores “part: length from head to waist [cm], average: 85 cm, variance: 2σ,” “part: left arm angle, average: 80°, variance: 4σ” and so on as illustrated inFIG. 11B. In this manner, the recognition information storage circuitry132stores recognition information in which an average and a variance for identifying a part are set for each of various parts of a human body.

Note that the examples illustrated inFIGS. 11A and 11Bare merely example, and the recognition information can be arbitrarily set. For example, the part, the range, the average, the variance, and the like can be arbitrarily set by the operator. The recognition information to be used as a condition can be used in combination in various manners. For example, there may be a case in which all of the information data presented inFIGS. 11A and 11Bare essential, or a case in which only the height and the arm length are essential. Alternatively, for example, there may be a case in which either one of the shoulder length and the left arm angle needs to be satisfied.

The description refers back toFIG. 10, in which the specification circuitry142according to the second embodiment extracts information indicating a human body skeleton contained in image information as subject information, and specifies motion information of the subject carrying out a predetermined motion on the basis of the extracted subject information. Note that the specification circuitry142determines a set of joint points according to the human body structure to be information indicating a human body skeleton among information data of joint points contained in image information.FIGS. 12A and 12Bare diagrams for explaining an example of processing performed by the specification circuitry142according to the second embodiment. Note thatFIG. 12Aillustrates processing in a case where the recognition information illustrated inFIG. 11Ais used.FIG. 12Billustrates processing in a case where the recognition information illustrated inFIG. 11Bis used.

First, the processing in the case where the recognition information illustrated inFIG. 11Ais used will be described. The specification circuitry142according to the second embodiment reads out motion information of one frame collected by the motion information collector10and stored in the motion information storage circuitry131as illustrated inFIG. 12A, and carries out processing using the recognition information illustrated inFIG. 11Aon skeleton information contained in the read motion information, for example. In one example, the specification circuitry142refers to the identification information inFIG. 11A“part: height [cm], range: 100 cm-200 cm” and carries out a specification process using the height information. Specifically, the specification circuitry142calculates a height “a” from the y coordinate value of the joint “2a1” corresponding to the head and the y coordinate value of the joint “2t1” corresponding to the tarsus of the left foot illustrated inFIG. 12A, for example. If the calculated height “a” is within the range of “100 cm-200 cm,” the specification circuitry142then specifies skeleton information containing the joint “2a1” and the joint “2t1” as skeleton information of the subject.

The specification circuitry142also calculates a height “b” from the y coordinate value of the joint “2a2” corresponding to the head and the y coordinate of the joint “2s2” corresponding to the left ankle illustrated inFIG. 12A. If the calculated height “b” is not within the range of “100 cm-200 cm,” the specification circuitry142specifies that skeleton information containing the joint “2a1” and the joint “2t1” is not skeleton information of the subject. Although specification using the height is illustrated inFIG. 12A, the embodiment is not limited thereto. Specifically, a specification process using any of the ranges of various parts can be carried out. Furthermore, although a case in which only the height is specified has been described in the example above, the embodiment is not limited thereto, and there may be a case in which motion information of the subject is specified by performing determination in a comprehensive manner using the ranges of two or more parts, for example.

In the example above, a case in which the height “a” and the height “b” are each calculated by using the y coordinates of the joint corresponding to the head and the joints corresponding to the left head has been described. While the motion information processing device100according to the present embodiment can increase the processing speed and allows efficient processing by calculating the heights by the simple method described above, the embodiment is not limited thereto, and the heights can be calculated by using three-dimensional coordinates of the joint corresponding to the head and the joint corresponding to the left foot, for example. For example, there may be a case in which the height “a” is calculated by using the x value, the y value, and the z value of the joint “2a1” corresponding to the head and the x value, the y value, and the z value of the joint “2t1” corresponding to the tarsus of the left foot. As a result, the motion information processing device100according to the present embodiment can achieve more accurate determination.

Next, processing in a case where the recognition information illustrated inFIG. 11Bis used will be described. The specification circuitry142according to the second embodiment reads out motion information of one frame collected by the motion information collector10and stored in the motion information storage circuitry131as illustrated inFIG. 12B, and carries out processing using the recognition information illustrated inFIG. 11Bon skeleton information contained in the read motion information, for example. In one example, the specification circuitry142refers to the identification information inFIG. 11B“part: shoulder length [cm], average: 60 cm, variance: “1σ” and carries out a specification process using the shoulder length information.

Specifically, the specification circuitry142calculates the “shoulder length: 59 cm” from the x coordinate value of the joint “2e1” corresponding to the right shoulder and the x coordinate value of the joint “2i1” corresponding to the left shoulder illustrated inFIG. 12B. The specification circuitry142then refers to the identification information “part: shoulder length [cm], average: 60 cm, variance: “1σ” and calculates the probability that the calculated “shoulder length: 59 cm” is a shoulder length of a human body. Specifically, the specification circuitry142calculates the probability of “99.9%” that the shoulder length is that of a human body as illustrated inFIG. 12B. Although a case in which the shoulder length is calculated by using the x coordinates of the joint corresponding to the right shoulder and the joint corresponding to the left shoulder has been described in the example above, the embodiment is not limited thereto, and there may be a case in which the shoulder length is calculated by using three-dimensional coordinates of the joint corresponding to the right shoulder and the joint corresponding to the left shoulder.

Note that the specification circuitry142may specify skeleton information of a subject on the basis of the calculated probability “99.9%,” or may further calculate the likelihood of being a human body skeleton for a plurality of parts and specify the skeleton information on the basis of the calculated information. For example, the specification circuitry142calculates the “length from head to waist: 100 cm” from the joint “2a1” corresponding to the head and the joint “2d1” corresponding to the waist, and further calculates the “left arm angle: 80°” as illustrated inFIG. 12B.

The specification circuitry142then refers to the recognition information similarly to the case of the shoulder length, and calculates the “probability: 99.8%” of the “length from head to waist,” for example. Similarly, the specification circuitry142refers to the recognition information, and calculates the “probability: 100%” of the “left arm angle,” for example. The specification circuitry142then carries out a specification process of motion information of a subject by using the calculated probability. Note that the specification circuitry142carries out the specification process by comparing three probabilities (those of the shoulder length, the length from the head to the waist, and the left arm angle) with respective thresholds, or may carry out the specification process by comparing a value obtained by multiplying the three probabilities with a threshold.

For example, the specification circuitry142compares “99.7%” obtained by multiplying the “probability of shoulder length: 99.9%,” “probability of length from head to waist: 99.8%” and “probability of left arm angle: 100%” with a predetermined threshold, and specifies the corresponding skeleton information when the value exceeds the predetermined threshold as skeleton information of a subject. Note that the predetermined threshold can be arbitrarily set. In the case where three probabilities are compared with respective thresholds, the corresponding skeleton information may be determined to be skeleton information of a subject if all of the three probabilities exceed the thresholds or the corresponding skeleton information may be determined to be skeleton information of a subject if two out of three probabilities exceed the thresholds. Note that these thresholds can also be arbitrarily set.

The specification circuitry142carries out processing similar to that described above by using skeleton information containing the joints (2a2,2d2,2e2,2i2,2m2,2n2,2o2,2q2,2r2, and2s2) inFIG. 12B. Specifically, the specification circuitry142calculates the shoulder length, the length from the head to the waist, or the like by using coordinates of the joints (2a2,2d2,2e2,2i2,2m2,2n2,2o2,2q2,2r2, and2s2) to obtain the probability of being a human body skeleton. The specification circuitry142the compares the probability with a predetermined threshold to determine whether or not the skeleton information containing the joints (2a2,2d2,2e2,2i2,2m2,2n2,2o2,2q2,2r2, and2s2) is skeleton information of a subject. For example, in the case of the skeleton information “joints (2a2,2d2,2e2,2i2,2m2,2n2,2o2,2q2,2r2, and2s2)” of the chair illustrated inFIG. 12B, the skeleton information is quite different from a human body skeleton, and thus the specification circuitry142determines that the skeleton information is not skeleton information of a subject.

Next, processing of the motion information processing device100according to the second embodiment will be described with reference toFIG. 13.FIG. 13is a flowchart illustrating procedures of the processing performed by the motion information processing device100according to the second embodiment. As illustrated inFIG. 13, in the motion information processing device100according to the second embodiment, the obtaining circuitry141obtains motion information of a subject (step S201). The specification circuitry142then determines whether or not the false recognition preventing function is ON in the motion information collected by the motion information collector10(step S202).

If the false recognition preventing function is ON (Yes in step S202), the specification circuitry142reads out recognition information (step S203), and specifies motion information of a subject on the basis of the read recognition information (step S204). Subsequently, the specification circuitry142stores the specified motion information (skeleton information) in the motion information storage circuitry131(step S205).

If the false recognition preventing function is not ON in step S202(No in step S202), the motion information is stored in the motion information storage circuitry131without the specification by the specification circuitry142(step S205). The display control circuitry143then performs control to generate a display image on the basis of the motion information stored in the motion information storage circuitry131and display the display image on the output circuitry110(step S206).

As described above, according to the second embodiment, the specification circuitry142extracts information indicating a human body skeleton contained in image information as subject information, and specifies motion information of the subject carrying out a predetermined motion on the basis of the extracted subject information. The motion information processing device100according to the second embodiment can therefore extract a human as a moving object, and prevent false recognition of a subject more accurately.

Furthermore, according to the second embodiment, the specification circuitry142determines a set of joint points according to the human body structure to be information indicating a human body skeleton among information data of joint points contained in image information. The motion information processing device100according to the second embodiment therefore enables extraction of a human as a moving object with high accuracy.

Third Embodiment

In a third embodiment, a case in which skeleton information of a subject is specified on the basis of the number of recognized joints among skeleton information data collected by the motion information collector10will be described. The third embodiment is different from the first and second embodiments in the processing of the specification circuitry142. Hereinafter, the description will be focused mainly on this difference.

The specification circuitry142according to the third embodiment determines whether skeleton information is skeleton information of a subject on the basis of the number of joint points collected as the skeleton information among information of joint points contained in image information. In the case of the skeleton information illustrated inFIG. 5as an example, the specification circuitry142compares skeleton information containing 20 joint points (joints2a1to2t1) that have been recognized and skeleton information containing 10 joint points (2a2,2d2,2e2,2i2,2m2,2n2,2o2,2q2,2r2, and2s2) that have been recognized with respective predetermined thresholds to specify skeleton information of a subject. If a threshold is the number of joint points “15,” for example, the specification circuitry142specifies the skeleton information containing 20 joint points (joint2a1to2t1) that have been recognized as skeleton information of a subject.

If the likelihood of the recognition of the joint points is output from the motion information collector10, the specification circuitry142counts the number of joint points with a predetermined likelihood, and compares the counted number with a threshold. If the number of joint points with the predetermined likelihood exceeds a predetermined threshold, the specification circuitry142then specifies the skeleton information with the number of joint points having the predetermined likelihood exceeding the predetermined threshold as the skeleton information of a subject, and stores the specified skeleton information in the motion information storage circuitry131. In this process, the specification circuitry142can store the specified skeleton information in which the joints points are associated with information on the likelihood of recognition.

In this case, the display control circuitry143according to the third embodiment can generate a display image in which the information on the likelihood of the joint points is reflected, and display the generated display image.FIG. 14is a diagram illustrating an example of the display image displayed by the display control circuitry143according to the third embodiment. As illustrated inFIG. 14, for example, the display control circuitry143generates a display image in which joint points having a predetermined likelihood of recognition of joint points are represented by black dots, information indicating bones between the black dots by solid lines is superimposed thereon, joint points not having the predetermined likelihood of recognition of joint points are represented by small white circles, and information indicating bones between the small white circles by broken lines, and displays the generated display image.

As described above, according to the third embodiment, the specification circuitry142specifies skeleton information of a subject on the basis of the number of recognized joint points. The motion information processing device100according to the third embodiment therefore enables specification of skeleton information of a subject by a simple method.

Fourth Embodiment

In the first to third embodiments described above, a case in which skeleton information of a subject is specified from the entire area from which motion information is collected by the motion information collector10has been described. In a fourth embodiment, a case in which a recognition target area is specified by an operator will be described. The fourth embodiment is different from the first to third embodiments in an instruction received by the input circuitry120and in the processing performed by the specification circuitry142. Hereinafter, the description will be focused mainly on these differences.

The input circuitry120according to the fourth embodiment receives a specification operation for specifying a predetermined area in image information. Specifically, the input circuitry120receives an input operation for specifying a recognition target area for recognition of a subject.

The specification circuitry142according to the fourth embodiment uses information contained within a predetermined area received by the input circuitry120to determine motion information of a subject carrying out a predetermined motion to be a predetermined feature from information contained in the predetermined area. Specifically, the specification circuitry142determines skeleton information contained in the recognition target area received by the input circuitry120to be the skeleton information of the subject.

FIG. 15is a diagram for explaining an example of processing performed by the input circuitry120and the specification circuitry142according to the fourth embodiment. The input circuitry120receives an input operation for specifying a recognition target area R2as illustrated in (B) ofFIG. 15on an image illustrated in (A) ofFIG. 15, for example. When the recognition target area R2illustrated in (B) ofFIG. 15is specified, the specification circuitry142then determines skeleton information contained in the specified recognition target area R2to be the skeleton information of the subject. As a result, even when skeleton information corresponding to the chair illustrated inFIG. 15is collected, for example, the skeleton information of the subject can be reliably determined.

As described above, according to the fourth embodiment, the input circuitry120receives a specification operation for specifying a predetermined area in image information. The specification circuitry142then determines motion information of a subject carrying out a predetermined motion to be a predetermined feature within the area received by the input circuitry120. The motion information processing device100according to the fourth embodiment can therefore specify a recognition target area in a simple manner and easily determine skeleton information of a subject.

Fifth Embodiment

In the fourth embodiment described above, a case in which a recognition target area is specified through the input circuitry120has been described. In a fifth embodiment, a case in which a recognition target area is set on the basis of information within an image will be described. The fifth embodiment is different from the first to fourth embodiments in the processing performed by the specification circuitry142. Hereinafter, the description will be focused mainly on this difference.

The specification circuitry142according to the fifth embodiment extracts a set item set for a subject in advance as subject information, and determines motion information of the subject carrying out a predetermined motion on the basis of the extracted subject information. Specifically, first, the subject carrying out rehab wears a thing that functions as a mark and carries out rehab. The specification circuitry142extracts the mark contained in an image, set a recognition target area on the basis of the extracted mark, and determines skeleton information within the set recognition target area to be skeleton information of the subject.

FIG. 16is a diagram for explaining an example of processing performed by the specification circuitry142according to the fifth embodiment.FIG. 16illustrates a case in which marks set in advance are worn on the feet of the subject. The subject carries out gait training with the marks of a predetermined color (red, for example) on his/her feet as illustrated in (A) ofFIG. 16, for example. The specification circuitry142extracts the marks worn on the feet of the subject from color image information collected by the motion information collector10. The specification circuitry142then calculates a midpoint P1that is the center of the width between the extracted marks on the feet as illustrated in (A) ofFIG. 16, and determines a width defining a predetermined distance to the left and to the right from the calculated midpoint P1.

When the distance between the marks on the feet is “15 cm,” for example, the specification circuitry142determines “45 cm” that is three times the distance to be the width of a recognition target area R3. Specifically, the specification circuitry142determines “22.5 cm” to the left and to the right from the midpoint P1to be the width of the recognition target area R3. The specification circuitry142then determines the length from a lower end P2of the marks on the feet to an upper end P3of the color image information to be the length of the recognition target area R3.

Specifically, the specification circuitry142sets the recognition target area R3in the color image information as illustrated in (B) ofFIG. 16, and determines skeleton information contained in the set recognition target area R3to be skeleton information of the subject. Note that the example illustrated inFIG. 16is only an example, and the embodiment is not limited thereto. For example, the marks set on the subject in advance may also be set in various manners other than the color thereof. In one example, there may be a case in which the subject wears things of a predetermined shape.

Note that information on a mark set on the subject in advance is stored in the storage circuitry130in advance, and the specification circuitry142reads out the information on the mark stored in the storage circuitry130and extracts the mark contained in the color image information.

As described above, according to the fifth embodiment, the specification circuitry142extracts a set item set for a subject in advance as subject information, and determines motion information of the subject carrying out a predetermined motion on the basis of the extracted subject information. The motion information processing device100according to the fifth embodiment therefore allows a recognition target area to be set in a simple manner.

Sixth Embodiment

While the first to fifth embodiments have been described above, various different embodiments other than the first to fifth embodiments can be employed.

In the first to fifth embodiments described above, a case in which motion information collected in a state in which rehab is being carried out has been described. The embodiment, however, is not limited thereto, and there may be a case in which various information data are extracted from motion information during rehab on the basis of information before the rehab is carried out, for example.

FIG. 17is a diagram for explaining an example of processing performed by the specification circuitry142according to the sixth embodiment. The specification circuitry142obtains the depth of a “T-second” frame that is image information (motion information) in a state before the rehab is carried out as a reference value as illustrated in (A) ofFIG. 17, for example. The specification circuitry142then obtains the depth of a “(T+t)-second” frame that is image information (motion information) in a state in which the rehab is being carried out as illustrated in (B) ofFIG. 17. In this process, the specification circuitry142can extract a part in which the depth has changed in the “(T+t)-second” frame from that in the “T-second” frame to reduce information to be processed. For example, as illustrated inFIG. 17, the chair that is contained in both images and whose depth information does not change can be ignored.

As a result, objects that are at a shallower or deeper position than the reference value can be easily determined, and objects whose depth has been changed to be shallower, deeper or to the left or right during rehab can be easily extracted, for example. Note that the aforementioned processing can be carried out in combination with the first to fifth embodiments where appropriate.

Furthermore, in the first to fifth embodiments described above, a case in which gain training is carried out as rehab has been described. The motion information processing device100according to the present application, however, can also be applied to rehab exercises other than gait training. For example, the specification circuitry142can also determine skeleton information of a subject carrying out range of motion exercise.

As described above, according to the first to sixth embodiments, the motion information processing device according to the present embodiments can therefore configured prevent false recognition of a subject.

Seventh Embodiment

As described above, in the first to sixth embodiments, a case of supporting rehabilitation by allowing prevention of falsely recognizing a person or object other than the person carrying out a predetermined motion has been described. However, a person who is photographed is not always the subject. Specifically, if a person is within the photographed range of the motion information collector10, the person is photographed by the motion information collector10whether the person is the subject, a caregiver, or even a person not involved in rehab, for example. Thus, in seventh to tenth embodiments below, a case of providing a motion information processing device capable of determining whether or not a digitally recorded motion of a person is that of a subject of rehabilitation will be described. Although the person carrying out a predetermined motion has been referred to as a “subject” in the first to sixth embodiments, a person that is a subject of rehabilitation will be referred to as a “subject” in the seventh to tenth embodiments.

FIGS. 18A and 18Bare diagrams illustrating an example of a distance image taken by the distance image collection circuitry12.FIGS. 18A and 18Billustrates a case in which a person18a(subject) receives assistance from a person18b(caregiver) in rehab. Specifically, the person18a(subject) is supported by the left hand of the person18b(caregiver) on his/her right arm to carry out gait training. Note that, inFIGS. 18A and 18B, a distance image expressed by color shadings according to the distances is expressed by line drawing for the purpose of illustration.

As illustrated inFIG. 18A, the motion information collector10takes a distance image containing the person18a(subject) and the person18b(caregiver) as objects. In generating motion information from the distance image, the motion information collector10recognizes the person18aand the person18bas individual persons, not as a subject and a caregiver and generates motion information of the respective persons as illustrated inFIG. 18B. Thus, for analyzing motion relating to rehab of the subject by using the motion information generated here, it is necessary to specify the person who is the subject. Furthermore, the motion of caregiver may be falsely evaluated when the motion of the subject should be analyzed.

Thus, a motion information processing device200according to the seventh embodiment performs processing described below to determine whether or not a digitally recorded motion of a person is that of a subject of rehabilitation.

Furthermore, the motion information processing device200according to the seventh embodiment may analyze the motion of a caregiver to support the caregiver so as to indirectly support the subject assisted by the caregiver. In such a case, the motion information processing device200according to the seventh embodiment may determine whether or not the digitally recorded motion of a person is that of a caregiver by performing the processing described below.

FIG. 19is a block diagram illustrating a detailed example configuration of the motion information processing device200according to the seventh embodiment. As illustrated inFIG. 19, in the motion information processing device200, storage circuitry230includes motion information storage circuitry231, subject motion feature storage circuitry232, caregiver motion feature storage circuitry233, subject image feature storage circuitry234, caregiver image feature storage circuitry235, and evaluation information storage circuitry236.

The motion information storage circuitry231stores various information data collected by the motion information collector10. For example, the motion information storage circuitry231stores information in which motion information and color image information are associated with each other for a motion of a person. The motion information is skeleton information of each frame generated by the motion information generation circuitry14. Coordinates of joints in the skeleton information and pixel positions in the color image information are associated with each other in advance. Photographing time information in the skeleton information and photographing time information in the color image information are also associated with each other in advance. Furthermore, for example, the motion information and the color image information are stored in the motion information storage circuitry231each time the motion information and the color image information are collected by the motion information collector10.

The motion information storage circuitry231stores the motion information for each rehab such as gait training or range of motion exercise that is carried out, for example. Note that motions of multiple persons may be included in one exercise of rehab. In a specific example, as illustrated inFIG. 4A, when a subject carries out gait training with assistance from a caregiver, one gait training is carried out by combination of the motions of the subject and the caregiver. In such a case, the motion information storage circuitry231associates skeleton information data of multiple persons generated from distance image information of the same frame with one another, and stores the associated information as one motion information item. Thus, the motion information indicates motions of multiple persons at the same time. The motion information storage circuitry231stores motion information in association with photographing start time information on the time when photographing of the motion is started, for example. Although a case in which motion information indicates motions of multiple persons will be described below, the embodiment is not limited thereto, and there may be a case in which the motion information indicates a motion of one person.

The subject motion feature storage circuitry232stores subject motion feature information indicating a feature of the motion of the subject. For example, the subject motion feature storage circuitry232stores information in which a motion ID (identification) and the subject motion feature information are associated with each other. In the information, the motion ID is identification information for identifying a motion, and a number is allotted thereto each time a motion is defined by the designer of the motion information processing device200. The subject motion feature information is information indicating a feature of a motion of a subject, and is defined by the designer of the motion information processing device200in advance, for example.

FIG. 20is a table illustrating an example of information stored in the subject motion feature storage circuitry232. In the first record ofFIG. 20, a motion ID “11” and subject motion feature information “dragging one's leg” are associated. Specifically, the subject motion feature storage circuitry232stores one of the features of the motion of the subject being “dragging one's leg” as the motion with the motion ID “11.” The subject motion feature information “dragging one's leg” is determined according to whether or not the maximum change in the y coordinate of a tarsus (joint2por joint2t) during the motion is smaller than 1 cm, for example. In the second record ofFIG. 20, a motion ID “12” and subject motion feature information “walking with poor posture” are associated. Specifically, the subject motion feature storage circuitry232stores one of the features of the motion of the subject being “walking with poor posture” as the motion with the motion ID “12.” The subject motion feature information “walking with poor posture” is determined according to whether or not the average of an angle between the backbone (a line connecting the joint2band the joint2c) and the vertical direction during the motion is equal to or larger than 3°, for example. In the third record ofFIG. 20, a motion ID “13” and subject motion feature information “walking slowly” are associated. Specifically, the subject motion feature storage circuitry232stores one of the features of the motion of the subject being “walking slowly” as the motion with the motion ID “13.” The subject motion feature information “walking slowly” is determined according to whether or not the maximum value of the moving speed of the waist (joint2c) during the motion is lower than 1 [m/s], for example. As for the other records, the subject motion feature storage circuitry232similarly stores the motion ID and the subject motion feature information in association with each other. Although the subject motion feature storage circuitry232used when gait training is carried out is presented here as an example, the embodiment is not limited thereto, and when range of motion exercise is carried out, a subject motion feature storage circuitry232in which a feature of a motion of a subject carrying out range of motion exercise is stored may be used. Alternatively, the subject motion feature storage circuitry232may store a feature of a motion of a subject carrying out gait training and a feature of a motion of a subject carrying out range of motion exercise without any distinction therebetween.

The caregiver motion feature storage circuitry233stores caregiver motion feature information indicating a feature of a motion of a caregiver. For example, the caregiver motion feature storage circuitry233stores information in which a motion ID and the caregiver motion feature information are associated with each other. The caregiver motion feature information is information indicating a feature of a motion of a caregiver, and is defined by the designer of the motion information processing device200in advance.

FIG. 21is a table illustrating an example of information stored in the caregiver motion feature storage circuitry233. In the first record ofFIG. 21, a motion ID “21” and caregiver motion feature information “supporting subject's arm” are associated. Specifically, the caregiver motion feature storage circuitry233stores one of the features of the motion of the caregiver being “supporting subject's arm” as the motion with the motion ID “21.” The caregiver motion feature information “supporting subject's arm” is determined according to whether or not a hand (the joint2hor the joint2l) of a person is within 5 cm from an arm (a line connecting the joint2eand the joint2for a line connecting the joint2iand the joint2j) of another person for a predetermined time during the motion, for example. In the second record ofFIG. 21, a motion ID “22” and caregiver motion feature information “walking with good posture” are associated. Specifically, the caregiver motion feature storage circuitry233stores one of the features of the motion of the caregiver being “walking with good posture” as the motion with the motion ID “22.” The caregiver motion feature information “walking with good posture” is determined according to whether or not the average of an angle between the backbone (a line connecting the joint2band the joint2c) and the vertical direction during the motion is smaller than 3°, for example. In the third record ofFIG. 21, a motion ID “23” and caregiver motion feature information “walking fast” are associated. Specifically, the caregiver motion feature storage circuitry233stores one of the features of the motion of the caregiver being “walking fast” as the motion with the motion ID “23.” The caregiver motion feature information “walking fast” is determined according to whether or not the maximum value of the moving speed of the waist (joint2c) during the motion is equal to or higher than 1 [m/s], for example. As for the other records, the caregiver motion feature storage circuitry233similarly stores the motion ID and the caregiver motion feature information in association with each other. Although the caregiver motion feature storage circuitry233used when gait training is carried out is presented here as an example, the embodiment is not limited thereto, and when range of motion exercise is carried out, a caregiver motion feature storage circuitry233in which a feature of a motion of a subject carrying out range of motion exercise is stored may be used. Alternatively, the caregiver motion feature storage circuitry233may store a feature of a motion of a subject carrying out gait training and a feature of a motion of a subject carrying out range of motion exercise without any distinction therebetween.

The subject image feature storage circuitry234stores subject feature information indicating a physical feature of a subject or a feature of an object accompanying a subject. For example, the subject image feature storage circuitry234stores subject image feature information indicating a feature of an image of a subject. For example, the subject image feature storage circuitry234stores information in which an equipment ID and subject equipment feature information are associated with each other. In the information, the equipment ID is identification information for identifying equipment, and a number is allotted thereto each time equipment is defined by the designer of the motion information processing device200. The subject equipment feature information is information indicating a feature of equipment of a subject, and is image information of equipment that can be used in pattern matching, for example. The subject equipment feature information is defined in advance by the designer of the motion information processing device200. Note that the subject image feature storage circuitry234is an example of a subject feature storage circuitry.

FIG. 22is a table illustrating an example of information stored in the subject image feature storage circuitry234. In the first record ofFIG. 22, an equipment ID “11” and subject equipment feature information “crutch” are associated. Specifically, the subject image feature storage circuitry234stores one of the features of the image of the subject being image information of a “crutch” as equipment with the equipment ID “11.” In the second record ofFIG. 22, an equipment ID “12” and subject equipment feature information “cast” are associated. Specifically, the subject image feature storage circuitry234stores one of the features of the image of the subject being image information of a “cast” as equipment with the equipment ID “12.” In the third record ofFIG. 22, an equipment ID “13” and subject equipment feature information “wheel chair” are associated. Specifically, the subject image feature storage circuitry234stores one of the features of the image of the subject being image information of a “wheel chair” as equipment with the equipment ID “13.” Although the subject image feature storage circuitry234used when gait training is carried out is presented here as an example, the embodiment is not limited thereto, and when range of motion exercise is carried out, a subject image feature storage circuitry234in which a feature of equipment of a subject carrying out range of motion exercise is stored may be used. Alternatively, the subject image feature storage circuitry234may store a feature of a motion of a subject carrying out gait training and a feature of equipment of a subject carrying out range of motion exercise without any distinction therebetween.

The caregiver image feature storage circuitry235stores caregiver feature information indicating a physical feature of a caregiver or a feature of an object accompanying a caregiver. For example, the caregiver image feature storage circuitry235stores caregiver image feature information indicating a feature of an image of a caregiver. For example, the caregiver image feature storage circuitry235stores information in which an equipment ID and caregiver equipment feature information are associated with each other. In the information, the caregiver equipment feature information is information indicating a feature of equipment of a caregiver, and is image information of equipment that can be used in pattern matching, for example. The caregiver equipment feature information is defined in advance by the designer of the motion information processing device200. Note that the caregiver image feature storage circuitry235is an example of a caregiver feature storage circuitry.

FIG. 23is a table illustrating an example of information stored in the caregiver image feature storage circuitry235. In the first record ofFIG. 23, an equipment. ID “21” and caregiver equipment feature information “stethoscope” are associated. Specifically, the caregiver image feature storage circuitry235stores one of the features of the image of the caregiver being image information of a “stethoscope” as equipment with the equipment ID “21.” In the second record ofFIG. 23, an equipment ID “22” and caregiver equipment feature information “white coat” are associated. Specifically, the caregiver image feature storage circuitry235stores one of the features of the image of the caregiver being image information of a “white coat” as equipment with the equipment ID “22.” In the third record ofFIG. 23, an equipment ID “23” and caregiver equipment feature information “nameplate” are associated. Specifically, the caregiver image feature storage circuitry235stores one of the features of the image of the caregiver being image information of a “nameplate” as equipment with the equipment ID “23.”

The evaluation information storage circuitry236stores information in which evaluation information and a determination result are associated for each subject. In the information, the evaluation information is information for evaluating a motion of a person in rehabilitation. For example, the evaluation information is a posture with which or speed at which a subject or a caregiver walk, or the like. The evaluation information is generated by generation circuitry242, which will be described later. The determination result is a result of determination performed by determination circuitry243, which will be described later, and any one of a subject, a caregiver, and indeterminable is stored, for example.

The description refers back toFIG. 19. In the motion information processing device200, control circuitry240includes obtaining circuitry241, the generation circuitry242, the determination circuitry243, and storage circuitry244.

The obtaining circuitry241obtains motion information to be evaluated. For example, when an input specifying motion information to be evaluated is received from the input circuitry120, the obtaining circuitry241obtains the specified motion information and associated color image information from the motion information storage circuitry231.

In one example, when specification of photographing start time information of motion information to be evaluated is received, the obtaining circuitry241obtains the motion information and color image information associated with the motion information from the motion information storage circuitry231. Note that the motion information may contain skeleton information data of multiple persons generated from distance image information of the same frame or may contain skeleton information of one person.

The generation circuitry242generates evaluation information for evaluating rehabilitation from the motion information obtained by the obtaining circuitry241. For example, the generation circuitry242calculates the postures with which and the speeds at which a subject and a caregiver walk. In a case where the motion information obtained by the obtaining circuitry241contains skeleton information data of multiple persons generated from distance image information of the same frame, the generation circuitry242generates evaluation information for skeleton information of each individual person. The generation circuitry242outputs the generated evaluation information to the storage circuitry244.

FIG. 24is a diagram for explaining processing performed by the generation circuitry242.FIG. 24explains a case in which the posture with which and the speed at which each of the person18aand the person18bwalks are calculated on the basis of motion information collected in the gait training inFIG. 18A. For example, the generation circuitry242extracts the coordinates of the joint2band the coordinates of the joints2cof the person18aof the respective frames from the motion information obtained by the obtaining circuitry241. The generation circuitry242regards a line connecting the extracted joint2band joint2cas the backbone of the person18a, and obtains an angle24abetween the backbone and the vertical direction for each frame. The generation circuitry242then calculates an average value of the angles24ain multiple frames while gait training is carried out as the posture with which the person18awalks. The generation circuitry242also calculates the posture with which the person18bwalks similarly to that of the person18a.

In addition, for example, the generation circuitry242obtains the moving distance [m] that the coordinates of the joint2ccorresponding to the waist of the person18ahave moved at predetermined time intervals (0.5 seconds, for example). The generation circuitry242then calculates the moving speed [m/s] of the person18aat predetermined time intervals on the basis of the moving distance per the predetermined time. The generation circuitry242then calculates an average value of the moving speed of the person18awhile the person18ais carrying out gait training as the speed at which the person18awalks. The generation circuitry242also calculates the speed at which the person18bwalks similarly to that of the person18a.

Although a case in which the posture and the speed of walk during gait training are calculated as evaluation information generated by the generation circuitry242has been described here, the embodiment is not limited thereto. For example, the generation circuitry242may select and calculate other evaluation information as appropriate depending on the content of functional exercises of rehabilitation or the condition of the subject.

The determination circuitry243determines whether or not a person associated with motion information obtained by the obtaining circuitry241is a subject of rehabilitation by using information indicating a feature of a subject. For example, the determination circuitry243refers to the subject motion feature information stored in the subject motion feature storage circuitry232as information indicating a feature of a subject, and determines whether or not the person associated with the motion information obtained by the obtaining circuitry241is a subject.

For example, the determination circuitry243determines whether or not a person associated with the motion information obtained by the obtaining circuitry241is a subject. The determination circuitry243also determines whether or not a person associated with the motion information obtained by the obtaining circuitry241is a caregiver. If skeleton information data of multiple persons generated from distance image information of the same frame are contained in the motion information obtained by the obtaining circuitry241, the determination circuitry243determines whether or not the person is a subject or whether or not the person is a caregiver for skeleton information of each individual person. The determination circuitry243outputs the result of determination to the storage circuitry244. In the following, processing of the determination circuitry243will be concretely described.

First, processing for determining whether or not a person is a subject will be described. The determination circuitry243selects one unprocessed record from records in the subject motion feature storage circuitry232and the subject image feature storage circuitry234, for example. The determination circuitry243then determines whether or not the obtained motion information and color image information satisfy the condition of the selected record.

Here, a case in which the record with the motion ID “11” has been selected from the subject motion feature storage circuitry232will be described. In this case, as illustrated inFIG. 20, the determination circuitry243determines whether or not the motion information obtained by the obtaining circuitry241corresponds to the subject motion feature information “dragging one's leg.” Specifically, the determination circuitry243extracts the y coordinates of a tarsus (joint2por joint2t) from the frames contained in the obtained motion information. The determination circuitry243then calculates the difference between the largest value and the smallest value of the extracted y coordinates as a maximum change amount. If the calculated maximum change amount is smaller than 1 cm, the determination circuitry243determines that the obtained motion information corresponds to the subject motion feature information, that is, the subject is dragging his/her leg.

In addition, a case in which the record with the motion ID “12” has been selected from the subject motion feature storage circuitry232will be described. In this case, as illustrated inFIG. 20, the determination circuitry243determines whether or not the motion information obtained by the obtaining circuitry241corresponds to the subject motion feature information “walking with poor posture.” If the walking posture has been calculated by the generation circuitry242, the determination circuitry243obtains the posture with which a person contained in the obtained motion information walks from the generation circuitry242. If the obtained walking posture is not smaller than 3°, the determination circuitry243then determines that the obtained motion information corresponds to the subject motion feature information, that is, the subject is walking with poor posture. If the walking posture has not been calculated by the generation circuitry242, the determination circuitry243may perform the same processing as that described above with reference to the generation circuitry242to calculate the walking posture.

In addition, a case in which the record with the motion ID “13” has been selected from the subject motion feature storage circuitry232will be described. In this case, as illustrated inFIG. 20, the determination circuitry243determines whether or not the motion information obtained by the obtaining circuitry241corresponds to the subject motion feature information “walking slowly.” If the walking speed has been calculated by the generation circuitry242, the determination circuitry243obtains the moving speed [m/s] at which a person contained in the obtained motion information walks at predetermined time intervals (0.5 seconds, for example) from the generation circuitry242. If the maximum moving speed of the obtained moving speed is lower than 1 [m/s], the determination circuitry243then determines that the obtained motion information corresponds to the subject motion feature information, that is, the subject is walking slowly. If the walking speed has not been calculated by the generation circuitry242, the determination circuitry243may perform the same processing as that described above with reference to the generation circuitry242to calculate the walking speed.

In addition, a case in which the record with the equipment ID “11” has been selected from the subject image feature storage circuitry234will be described. In this case, as illustrated inFIG. 22, the determination circuitry243carries out pattern matching between color image information obtained by the obtaining circuitry241and subject equipment feature information “crutch.” When an image of a crutch is extracted from the color image information as a result of pattern matching, the determination circuitry243determines whether or not the extracted pixel positions of the crutch overlap with coordinates in skeleton information contained in motion information to be evaluated. If the pixel positions of the crutch overlap with the coordinates in the skeleton information, the determination circuitry243determines that the obtained color image information corresponds to the subject equipment feature information, that is, the subject holds a crutch. As for the other records, the determination circuitry243similarly determines whether or not the obtained color image information corresponds to the subject equipment feature information.

As described above, the determination circuitry243determines whether or not the obtained motion information and color image information corresponds to the selected record. If it is determined that the obtained information corresponds to the selected record, the determination circuitry243increments a held subject feature number n by 1. The held subject feature number n represents the number of features as a subject that a person associated with motion information to be evaluated has. As for the other unprocessed records, the determination circuitry243similarly determines whether or not the obtained motion information and color image information correspond to the record. When the held subject feature number n has reached 5, the determination circuitry243determines that the person associated with the motion information to be evaluated to be a subject. If the held subject feature number n does not reach 5 when the determination circuitry243has performed determination on all the records in the subject motion feature storage circuitry232and the subject image feature storage circuitry234, the determination circuitry243determines that the person associated with the motion information to be evaluated is not a subject. Although a case in which the threshold for the held subject feature number n for determining whether or not a person is a subject is 5 has been presented as an example, the embodiment is not limited thereto, and the threshold may be set to any value by the operator.

Furthermore, although a case in which the held subject feature number n is incremented by 1 when information corresponds to a record has been described, the embodiment is not limited thereto, and each record may be weighted, for example.

Next, processing for determining whether or not a person is a caregiver will be described. The determination circuitry243selects one unprocessed record from records in the caregiver motion feature storage circuitry233and the caregiver image feature storage circuitry235, for example. The determination circuitry243then determines whether or not the obtained motion information and color image information correspond to the selected record.

Here, a case in which the record with the motion ID “21” has been selected from the caregiver motion feature storage circuitry233will be described. In this case, as illustrated inFIG. 21, the determination circuitry243determines whether or not the motion information obtained by the obtaining circuitry241corresponds to the caregiver motion feature information “supporting subject's arm.” Specifically, the determination circuitry243extracts the coordinates of a hand (joint2hor joint2l) from the frames contained in the obtained motion information. If an arm (a line connecting the joint2eand the joint2for a line connecting the joint2iand the joint2j) of another person is present within 5 cm from the obtained hand for a predetermined time during gait training, the determination circuitry243then determines that the obtained motion information corresponds to the caregiver motion feature information, that is, the caregiver is supporting the subject's arm.

In addition, a case in which the record with the motion ID “22” has been selected from the caregiver motion feature storage circuitry233will be described. In this case, as illustrated inFIG. 21, the determination circuitry243determines whether or not the motion information obtained by the obtaining circuitry241corresponds to the caregiver motion feature information “walking with good posture.” If the walking posture has been calculated by the generation circuitry242, the determination circuitry243obtains the posture with which a person contained in the obtained motion information walks from the generation circuitry242. If the obtained walking posture is smaller than 3°, the determination circuitry243then determines that the obtained motion information corresponds to the caregiver motion feature information, that is, the caregiver is walking with good posture. If the walking posture has not been calculated by the generation circuitry242, the determination circuitry243may perform the same processing as that described above with reference to the generation circuitry242to calculate the walking posture.

In addition, a case in which the record with the motion ID “23” has been selected from the caregiver motion feature storage circuitry233will be described. In this case, as illustrated inFIG. 21, the determination circuitry243determines whether or not the motion information obtained by the obtaining circuitry241corresponds to the caregiver motion feature information “walking fast.” If the walking speed has been calculated by the generation circuitry242, the determination circuitry243obtains the moving speed [m/s] at which a person contained in the obtained motion information walks at predetermined time intervals (0.5 seconds, for example) from the generation circuitry242. If the maximum moving speed of the obtained moving speed is equal to or higher than 1 [m/s], the determination circuitry243then determines that the obtained motion information corresponds to the caregiver motion feature information, that is, the caregiver is walking fast. If the walking speed has not been calculated by the generation circuitry242, the determination circuitry243may perform the same processing as that described above with reference to the generation circuitry242to calculate the walking speed.

In addition, a case in which the record with the equipment ID “21” has been selected from the caregiver image feature storage circuitry235will be described. In this case, as illustrated inFIG. 23, the determination circuitry243carries out pattern matching between color image information obtained by the obtaining circuitry241and caregiver equipment feature information “stethoscope.” When an image of a stethoscope is extracted from the color image information as a result of pattern matching, the determination circuitry243determines whether or not the extracted pixel positions of the stethoscope overlap with coordinates in skeleton information contained in motion information to be evaluated. If the pixel positions of the stethoscope overlap with the coordinates in the skeleton information, the determination circuitry243determines that the obtained color image information corresponds to the caregiver equipment feature information, that is, the caregiver holds a stethoscope. As for the other records, the determination circuitry243similarly determines whether or not the obtained color image information corresponds to the caregiver equipment feature information.

As described above, the determination circuitry243determines whether or not the obtained motion information and color image information corresponds to the selected record. If it is determined that the obtained information corresponds to the selected record, the determination circuitry243increments a held caregiver feature number m by 1. The held caregiver feature number m represents the number of features as a caregiver that a person associated with motion information to be evaluated has. As for the other unprocessed records, the determination circuitry243similarly determines whether or not the obtained motion information and color image information correspond to the record. When the held caregiver feature number m has reached 5, the determination circuitry243determines that the person associated with the motion information to be evaluated to be a caregiver. If the held caregiver feature number m does not reach 5 when the determination circuitry243has performed determination on all the records in the caregiver motion feature storage circuitry233and the caregiver image feature storage circuitry235, the determination circuitry243determines that the person associated with the motion information to be evaluated is not a caregiver. Although a case in which the threshold for the held caregiver feature number m for determining whether or not a person is a caregiver is 5 has been presented as an example, the embodiment is not limited thereto, and the threshold may be set to any value by the operator. Furthermore, although a case in which the held caregiver feature number m is incremented by 1 when information corresponds to a record has been described, the embodiment is not limited thereto, and each record may be weighted, for example.

As described above, the determination circuitry243determines whether or not a person associated with motion information to be evaluated is a subject, or whether or not the person is a caregiver, and outputs the determination result to the storage circuitry244. If a person associated with motion information to be evaluated is determined not to be a subject nor a caregiver, the determination circuitry243outputs a determination result of indeterminable to the storage circuitry244.

The storage circuitry244outputs the determination result of the determination by the determination circuitry243. For example, the storage circuitry244associates the evaluation information generated by the generation circuitry242and the determination result of determination by the determination circuitry243with each other, and stores the associated information in the evaluation information storage circuitry236. Note that the storage circuitry244is an example of an output control circuitry.

For example, the storage circuitry244displays a screen for allowing confirmation of the determination result of determination by the determination circuitry243on the output circuitry110. When the displayed determination result is confirmed, the storage circuitry244then associates the determination result and the evaluation information generated by the generation circuitry242in association with each other and stores the associated information in the evaluation information storage circuitry236.

FIG. 25is a diagram illustrating an example of a display screen displayed by the storage circuitry244. In the example illustrated inFIG. 25, a case in which a person associated with the motion information to be evaluated is the person18ainFIG. 18Ais illustrated. In addition, a case in which evaluation information of “walking with posture of 15°” is output by the generation circuitry242and a determination result of being a “subject” is output by the determination circuitry243for the person18awill be described.

As illustrated inFIG. 25, the storage circuitry244displays a question message “Person18ais determined to be subject. Correct?” and selection buttons of “Yes/No” for responding to the question message on the output circuitry110. If “Yes” is selected, the storage circuitry244then associates the motion information to be evaluated, the determination result “subject,” and the evaluation information “walking with posture of 15°” with one another and stores the association result in the evaluation information storage circuitry236. Alternatively, for example, the storage circuitry244may enter the evaluation information “body inclination: 15°” in a free entry field of rehabilitation records as illustrated inFIG. 25.

Alternatively, for example, the storage circuitry244may store the determination result without confirmation in the evaluation information storage circuitry236.FIG. 26is a diagram illustrating an example of a display screen displayed by the storage circuitry244. Note thatFIG. 26illustrates an example of a display screen displayed under the same condition as that ofFIG. 25, and further illustrates a case in which the evaluation information “walking with posture of 3°” for the person18bis output.

As illustrated inFIG. 26, the storage circuitry244displays evaluation information “<body inclination>18a: 15°,18b: 3°” for the person18aand the person18b. The storage circuitry244then associates the motion information to be evaluated, the determination result “subject,” and the evaluation information “walking with posture of 15°” with one another and stores the association result in the evaluation information storage circuitry236. Alternatively, for example, the storage circuitry244may display the evaluation information “body inclination: 15°” in a free entry field of rehabilitation records as illustrated inFIG. 26.

In this manner, the storage circuitry244outputs the determination result to the evaluation information storage circuitry236to store the determination result in the evaluation information storage circuitry236, and outputs the determination result to the output circuitry110to display the determination result on the output circuitry110. Note that the output of a determination result from the storage circuitry244is not limited to the above examples. For example, the storage circuitry244may output a determination result to another information processing device or an external storage device.

Next, procedures of processing of the motion information processing device200according to the seventh embodiment will be described with reference toFIG. 27.FIG. 27is a flowchart for explaining an example of procedures of processing performed by the motion information processing device200according to the seventh embodiment.

As illustrated inFIG. 27, when motion information to be evaluated is specified (Yes in step S301), the obtaining circuitry241obtains the motion information to be evaluated, and color image information associated with the motion information (step S302). Note that the obtaining circuitry241is in standby state until the motion information to be evaluated is specified (No in step S301).

The generation circuitry242generates evaluation information from the motion information obtained by the obtaining circuitry241(step S303). For example, the generation circuitry242generates the walking posture and the walking speed as evaluation information from the motion information to be evaluated.

The determination circuitry243performs a determination process of determining whether or not a person associated with the motion information obtained by the obtaining circuitry241is a subject or whether or not the person is a caregiver (step S304).

Here, procedures of processing of the determination circuitry243according to the first embodiment will be described with reference toFIG. 28.FIG. 28is a flowchart for explaining an example of procedures of the determination process performed by the determination circuitry243according to the seventh embodiment.

The determination circuitry243selects one unprocessed record from the subject motion feature storage circuitry232and the subject image feature storage circuitry234(step S401). The determination circuitry243then determines whether or not the obtained motion information and color image information correspond to the selected record (step S402). If the obtained information corresponds to the selected record (Yes in step S402), the determination circuitry243increments the held subject feature number n by 1 (step S403). The determination circuitry243then determines whether or not the held subject feature number n has reached 5 (step S404). If the held subject feature number n has reached 5 (Yes in step S404), the determination circuitry243determines that the person associated with the motion information obtained by the obtaining circuitry241is a subject (step S405).

If the held subject feature number n has not reached 5 (No in step S404), the determination circuitry243determines whether or not an unprocessed record is present in the subject motion feature storage circuitry232and the subject image feature storage circuitry234(step S406). If an unprocessed record is present (Yes in step S406), the determination circuitry243proceeds to the processing in step S401.

If no unprocessed record is present (No in step S406), the determination circuitry243selects one unprocessed record from the caregiver motion feature storage circuitry233and the caregiver image feature storage circuitry235(step S407). The determination circuitry243then determines whether or not the obtained motion information and color image information correspond to the selected record (step S408). If the obtained information corresponds to the selected record (Yes in step S408), the determination circuitry243increments the held caregiver feature number m by 1 (step S409). The determination circuitry243then determines whether or not the held caregiver feature number m has reached 5 (step S410). If the held caregiver feature number m has reached 5 (Yes in step S410), the determination circuitry243determines that the person associated with the motion information obtained by the obtaining circuitry241is a caregiver (step S411).

If the held caregiver feature number m has not reached 5 (No in step S410), the determination circuitry243determines whether or not an unprocessed record is present in the caregiver motion feature storage circuitry233and the caregiver image feature storage circuitry235(step S412). If an unprocessed record is present (Yes in step S412), the determination circuitry243proceeds to the processing in step S407.

If no unprocessed record is present (No in step S412), the determination circuitry243determines that the person associated with the motion information obtained by the obtaining circuitry241is indeterminable (step S413).

The description refers back toFIG. 27. After the determination process is performed by the determination circuitry243, the storage circuitry244displays a confirmation screen for confirmation whether or not the determination result is correct or false on the output circuitry110(step S305). If the input circuitry120has received an input indicating that the determination result is correct (Yes in step S306), the storage circuitry244then stores evaluation information generated by the generation circuitry242together with the determination result in the evaluation information storage circuitry236(step S307).

If the input circuitry120has received an input indicating that the determination result is incorrect (No in step S306), the storage circuitry244waits until a correction instruction input is received by the input circuitry120(No in step S308). If the input circuitry120has received a correction instruction input (Yes in step S308), the storage circuitry244then stores evaluation information in the evaluation information storage circuitry236according to the correction instruction input (step S309), and terminates the processing.

Note that the procedures of processing described above need not necessarily be performed in the order described above. For example, the processing of step S303that is a process of generating evaluation information and the processing of step S304of performing the determination process are not limited to the order described above, and the processing of step S303may be performed after the processing of step S304is performed.

Furthermore, for example, the processing from step S401to step S406that is a process of determining whether or not a person is a subject and the processing from step S407to step S412that is a process of determining whether or not a person is a caregiver are not limited to the order described above. Specifically, the processing from step S401to step S406may be performed after the processing from step S407to step S412is performed.

As described above, the motion information processing device200according to the seventh embodiment stores subject motion feature information indicating a feature of a motion of a subject of rehabilitation. The motion information processing device200then obtains motion information indicating a motion of a person. The motion information processing device200then determines whether or not the person associated with the obtained motion information is a subject by using the subject motion feature information. Thus, the motion information processing device200according to the seventh embodiment can determine whether or not a digitally recorded motion of a person is that of a subject of rehabilitation.

FIG. 29is a diagram for explaining effects of the motion information processing device200according to the seventh embodiment. As illustrated inFIG. 29, for example, the motion information processing device200obtains motion information of each of the person18aand the person18bcarrying out rehabilitation on the basis of a distance image29aphotographed by the motion information collector10. The motion information processing device200then determines whether or not the person associated with the obtained motion information is a subject or whether or not the person is a caregiver by using the motion information. The motion information processing device200then displays the determination result on a display screen29band stores the determination result and evaluation information based on the motion information in the evaluation information storage circuitry236. Thus, the motion information processing device200can evaluate the motion information of a subject, for example.

Furthermore, the motion information processing device200stores caregiver motion feature information indicating a feature of a motion of a caregiver assisting a subject, for example. The motion information processing device200then determines whether or not the person associated with the obtained motion information is a caregiver by using the caregiver motion feature information. Thus, the motion information processing device200according to the seventh embodiment can determine whether or not a digitally recorded motion of a person is that of a caregiver.

Furthermore, for example, the motion information processing device200displays a screen for confirmation of a determination result on a predetermined display, and when the determination result is confirmed, stores the determination result and the generated evaluation information in association with each other in predetermined storage circuitry. Thus, the motion information processing device200can prevent a person to be evaluated from being falsely evaluated, for example.

Furthermore, for example, the motion information processing device200stores subject image feature information indicating a feature of an image of a subject. The motion information processing device200then obtains a person image in which the person is photographed. The motion information processing device200then determines whether or not the person associated with the obtained person image is a subject by using the subject image feature information. It is therefore possible to determine whether or not a digitally recorded motion of a person is that of a subject.

Furthermore, for example, the motion information processing device200stores caregiver image feature information indicating a feature of an image of a caregiver. The motion information processing device200then obtains a person image in which the person is photographed. The motion information processing device200then determines whether or not the person associated with the obtained person image is a caregiver by using the caregiver image feature information. It is therefore possible to determine whether or not a digitally recorded motion of a person is that of a caregiver.

Eighth Embodiment

Although a case in which whether or not a person is a subject is determined by using a feature of a motion and a feature of an image of a subject has been described in the seventh embodiment described above, the embodiment is not limited thereto. For example, the motion information processing device200may determine whether or not a person is a subject by using personal information of the subject. Thus, in the eighth embodiment, a case in which the motion information processing device200determines whether or not a person is a subject by using personal information of the subject will be described.

FIG. 30is a block diagram illustrating an example configuration of the motion information processing device200according to the eighth embodiment. As illustrated inFIG. 30, the motion information processing device200according to the eighth embodiment has a configuration similar to that of the motion information processing device200illustrated inFIG. 19, and differs therefrom in that the storage circuitry230further includes a personal information storage circuitry237and a personal feature information storage circuitry238and in that the control circuitry240further includes extraction circuitry245. In the eighth embodiment, the description will be focused mainly on the differences from the seventh embodiment, and components having the same functions as those described in the seventh embodiment will be designated by the same reference numerals as those inFIG. 19and the description thereof will not be repeated. Note that, in the motion information processing device200according to the eighth embodiment, the storage circuitry230may not include the subject image feature storage circuitry234and the caregiver image feature storage circuitry235.

The personal information storage circuitry237stores identification information identifying a subject and personal information in association with each other. In the information, a subject ID is information for identifying a subject, such as a patient number or the like for identifying a patient in a medical institution, for example. The personal information is information peculiar to the corresponding subject, and contains information such as height, chest circumference, and abdominal circumference, for example. Note that the information stored in the personal information storage circuitry237is information obtained from a medical information system, a personal health record (PHR), or the like, for example, and a disease name, history of hospital visits, and the like in addition to physical features of a subject may be stored.

The personal feature information storage circuitry238stores a personal feature ID and personal feature information. The personal feature ID is information for identifying a feature that can be used to identify a person, and a number is allotted each time a record in the personal feature information storage circuitry238is registered by the designer of the motion information processing device200. The personal feature information refers to an item of personal information that can be used for determining whether or not a person is a subject by using the personal information, and is defined in advance by the designer of the motion information processing device200, for example. Note that the information stored in the personal feature information storage circuitry238is may be obtained from a medical information system, a personal health record (PHR), or the like, for example.

FIG. 31is a table illustrating an example of information stored in the personal feature information storage circuitry238. In the first record ofFIG. 31, a personal feature ID “001” and personal feature information “( ) cm in height” are stored. Specifically, the personal feature information storage circuitry238stores “( ) cm in height,” which is an item of personal information that can be used for determining whether or not a person is a subject by using the personal information, as personal feature information with the personal feature ID “001.” The personal feature information “( ) cm in height” is used for determination after information on the height of the subject is obtained from the personal information storage circuitry237. In the second record ofFIG. 31, a personal feature ID “002” and personal feature information “( ) cm in chest circumference” are stored. Specifically, the personal feature information storage circuitry238stores “( ) cm in chest circumference,” which is an item of personal information that can be used for determining whether or not a person is a subject by using the personal information, as personal feature information with the personal feature TD “002.” In the third record ofFIG. 31, a personal feature TD “003” and personal feature information “( ) cm in abdominal circumference” are stored. Specifically, the personal feature information storage circuitry238stores “( ) cm in abdominal circumference,” which is an item of personal information that can be used for determining whether or not a person is a subject by using the personal information, as personal feature information with the personal feature ID “003.” As for the other records, the personal feature information storage circuitry238stores an item of personal information that can be used for determining whether or not a person is a subject by using the personal information.

The extraction circuitry245receives identification information identifying a subject, and extracts personal information associated with the received identification information from the personal information storage circuitry237. In one example, the extraction circuitry245extracts a physical feature of a subject from information recorded in an electronic medical record of the subject.

For example, when the input circuitry120receives an input of a subject ID of a subject carrying out rehabilitation when the rehabilitation is started, the extraction circuitry245receives the subject ID from the input circuitry120.

Subsequently, the extraction circuitry245extracts personal information associated with the personal feature information in each record in the personal feature information storage circuitry238from the personal information storage circuitry237by using the received subject ID. In the example illustrated inFIG. 31, “( ) cm in height” is stored as the personal feature information in the first record. In this case, the extraction circuitry245refers to the personal information storage circuitry237and extracts “170 cm in height,” for example, as the personal information associated with the received subject ID. The height extracted here is used by the determination circuitry243for determining whether or not a person is a subject by using the first record “( ) cm in height” in the personal feature information storage circuitry238. As for the other records in the personal feature information storage circuitry238, the extraction circuitry245similarly extracts personal information associated with each record in the personal feature information storage circuitry238from the personal information storage circuitry237by using the received subject ID.

Next, procedures of processing of the motion information processing device200according to the eighth embodiment will be described with reference toFIG. 32.FIG. 32is a flowchart for explaining an example of procedures of processing performed by the motion information processing device200according to the eighth embodiment. Note that the input circuitry120receives an input of a subject ID of a subject carrying out rehabilitation when the rehabilitation is started.

As illustrated inFIG. 32, in the motion information processing device200according to the eighth embodiment, since the processing from step S501to step S503is the same as the processing from step S301to step S303, respectively, illustrated inFIG. 27, the description thereof will not be repeated.

After the processing in step S503is terminated, the extraction circuitry245receives the subject ID of a subject carrying out rehabilitation from the input circuitry120(step S504). The extraction circuitry245then extracts personal information associated with the received subject TD from the personal information storage circuitry237(step S505).

The determination circuitry243performs a determination process of determining whether or not a person associated with motion information obtained by the obtaining circuitry241is a subject, or whether or not the person is a caregiver (step S506). Note that the determination circuitry243basically performs the determination process in the procedures of processing as illustrated inFIG. 28, but the determination process is different in that the personal feature information storage circuitry238is used for determination. The difference will thus be described below with reference toFIG. 28.

In step S401, the determination circuitry243selects one unprocessed record from the subject motion feature storage circuitry232and the personal feature information storage circuitry238. Note that, when the motion information processing device200according to the eighth embodiment includes the subject image feature storage circuitry234, the unprocessed record may be selected from the subject image feature storage circuitry234.

Subsequently, in step S402, the determination circuitry243determines whether or not the obtained motion information and color image information correspond to the selected record. In this process, when the record is selected from the personal feature information storage circuitry238, the determination circuitry243performs determination by using the personal information extracted by the extraction circuitry245. For example, when the first record inFIG. 31is selected, the determination circuitry243obtains the personal feature information “( ) cm in height” from the personal feature information storage circuitry238.

Subsequently, the determination circuitry243obtains the personal information “170 cm in height” extracted by the extraction circuitry245as personal information associated with the personal feature information “( ) cm in height.” The determination circuitry243then combines the obtained personal feature information “( ) cm in height” and the personal information “170 cm in height” to generate a condition “170 cm in height.” The determination circuitry243then determines whether or not the motion information and color image information obtained by the obtaining circuitry241satisfy the generated condition “170 cm in height.” Specifically, the determination circuitry243obtains the height from skeleton information contained in the motion information and determines whether or not the value is close to 170 cm.

The determination circuitry243then performs the processing in step S403and subsequent steps. Since the processing in step S403and subsequent steps is the same as that described in the seventh embodiment, the description thereof will not be repeated.

The description refers back toFIG. 32, in which the motion information processing device200performs processing in step S507and subsequent steps. Since the processing from step S507to step S511inFIG. 32is the same as that from step S305to step S309inFIG. 27, respectively, the description thereof will not be repeated.

Note that the procedures of processing described above need not necessarily be performed in the order described above. For example, the processing in step S503that is a process of generating evaluation information may be performed after the processing in step S505of extracting personal information is performed.

As described above, the motion information processing device200according to the eighth embodiment includes the personal information storage circuitry237that stores identification information for identifying a subject and personal information of the subject in association with each other. The motion information processing device200receives identification information, and extracts personal information associated with the received identification information from the personal information storage circuitry237. The motion information processing device200then refers to the personal information extracted by the extraction circuitry245, and determines whether or not a person associated with a person image is the person associated with the personal information. Thus, the motion information processing device200can correctly determine whether or not a digitally recorded motion of a person is that of a subject of rehabilitation.

Although a case in which the motion information processing device200stores the personal information storage circuitry237and the personal feature information storage circuitry238has been described in the eighth embodiment, the motion information processing device200need not necessarily store the personal information storage circuitry237and the personal feature information storage circuitry238. For example, in a case where the motion information processing device200is connected to the medical information system described above via a network, the extraction circuitry245may extract personal information associated with personal feature information in each record in the personal feature information storage circuitry238from the medical information system.

Ninth Embodiment

Although a case in which the determination process is performed by using information capable of identifying a subject has been described in the eighth embodiment described above, the embodiment is not limited thereto. In the ninth embodiment, a case in which a determination process is performed by using information capable of identifying a caregiver will be described, for example.

FIG. 33is a block diagram illustrating an example configuration of a motion information processing device200according to the ninth embodiment. As illustrated inFIG. 33, the motion information processing device200according to the ninth embodiment has a configuration similar to that of the motion information processing device200illustrated inFIG. 19, and differs therefrom in that the storage circuitry230further includes a staff information storage circuitry239. In the ninth embodiment, the description will be focused mainly on the differences from the seventh embodiment, and components having the same functions as those described in the seventh embodiment will be designated by the same reference numerals as those inFIG. 19and the description thereof will not be repeated. Note that, in the motion information processing device200according to the ninth embodiment, the storage circuitry230may not include the subject image feature storage circuitry234and the caregiver image feature storage circuitry235.

The staff information storage circuitry239stores a staff ID and a name in association with each other. The staff ID is information for identifying a staff, such as a number for identifying a person working in a medical institution, for example. The name is information indicating the name of a person corresponding to the staff ID. The information stored in the staff information storage circuitry239is information obtained from a staff personnel system for managing personnel of staffs in a medical institution in which the motion information processing device200is used, for example. Note that the staff includes a caregiver.

FIG. 34is a table illustrating an example of information stored in the staff information storage circuitry239. In the first record ofFIG. 34, a staff ID “1” and a name “AB” are stored. Specifically, the staff information storage circuitry239stores that the name of a staff represented by the staff ID “1” is “AB.” In the second record ofFIG. 34, a staff ID “2” and a name “CD” are stored. Specifically, the staff information storage circuitry239stores that the name of a staff represented by the staff ID “2” is “CD.” In the third record ofFIG. 34, a staff ID “3” and a name “EF” are stored. Specifically, the staff information storage circuitry239stores that the name of a staff represented by the staff ID “3” is “EF.” As for staffs with other staff IDs, the staff information storage circuitry239similarly stores a staff ID and a name in association with each other.

Next, procedures of processing of the motion information processing device200according to the ninth embodiment will be described. The procedures of processing of the motion information processing device200according to the ninth embodiment are similar to those of the motion information processing device200according to the seventh embodiment described with reference toFIG. 27, but differs therefrom in that the staff information storage circuitry239is used for determination. The difference will thus be described below with reference toFIG. 28.

Since the processing from step S401to step S405is the same as that described with reference toFIG. 28, the description thereof will not be repeated.

In step S407, the determination circuitry243selects one unprocessed record from the caregiver motion feature storage circuitry233and the staff information storage circuitry239. Note that, when the motion information processing device200according to the ninth embodiment includes the caregiver image feature storage circuitry235, the unprocessed record may be selected from the caregiver image feature storage circuitry235.

Subsequently, in step S408, the determination circuitry243determines whether or not the obtained motion information and color image information correspond to the selected record. In this process, when the record is selected from the staff information storage circuitry239, the determination circuitry243performs pattern matching on a nameplate from the color image information obtained by the obtaining circuitry241. When an image of the nameplate is extracted from the color image information through pattern matching, the determination circuitry243recognizes characters on the extracted nameplate by a known character recognition technology. When the characters, such as “AB,” are recognized through character recognition, the determination circuitry243then determines whether or not the characters are stored in the staff information storage circuitry239. If the characters recognized through character recognition are stored in the staff information storage circuitry239, the determination circuitry243determines that the obtained color image information corresponds to the personnel information.

The determination circuitry243then performs the processing in step S409and subsequent steps. Since the processing in step S409and subsequent steps is the same as that described in the seventh embodiment, the description thereof will not be repeated.

As described above, the motion information processing device200according to the ninth embodiment includes an identification information storage circuitry that stores identification information for identifying a caregiver. In addition, the motion information processing device200then obtains a person image in which the person is photographed. The motion information processing device200then extracts identification information from the obtained person image, and determines whether or not the extracted identification information corresponds to a person associated with the identification information stored in the identification information storage circuitry. Thus, the motion information processing device200can identify the individual name of a caregiver.

Although a case in which the motion information processing device200determines whether or not a person is a caregiver by using the name of a staff has been described in the ninth embodiment, the embodiment is not limited thereto, and whether or not a person is a caregiver may be determined through face recognition of a staff. In this case, the staff information storage circuitry239stores a staff ID and a face image of the staff in association with each other. The determination circuitry243then recognizes an image of a face from color image information obtained by the obtaining circuitry241, and performs matching on the recognized face image and the face image stored in the staff information storage circuitry239to determine whether or not the obtained color image information corresponds to staff information.

Furthermore, although a case in which the motion information processing device200stores the staff information storage circuitry239has been described in the ninth embodiment, the motion information processing device200need not necessarily store the staff information storage circuitry239. For example, in a case where the motion information processing device200is connected to the staff personnel system described above via a network, the determination circuitry243may access the staff personnel system and perform determination by using information corresponding to that stored in the staff information storage circuitry239.

Tenth Embodiment

While the seventh to ninth embodiments have been described above, various different embodiments other than the seventh to ninth embodiments can be employed.

For example, the configurations described in the seventh to ninth embodiments above are only examples, and all the configurations are not necessarily required. For example, the motion information processing device200may not have the generation circuitry242. Specifically, in the motion information processing device200, the obtaining circuitry241obtains motion information indicating a motion of a person. The determination circuitry243then determines whether or not a person associated with the obtained motion information is a subject by using the subject motion feature information. The storage circuitry244then outputs the determination result of the determination by the determination circuitry243. Thus, the motion information processing device200can determine whether or not a digitally recorded motion of a person is that of a subject of rehabilitation. For example, the motion information processing device200may output the obtained motion information and the determination result in association with each other to another device. Furthermore, for example, in the motion information processing device200, the storage circuitry230may have any of the subject motion feature storage circuitry232, the caregiver motion feature storage circuitry233, the subject image feature storage circuitry234, and the caregiver image feature storage circuitry235.

Furthermore, for example, the configurations described in the seventh to ninth embodiments above are exemplary only, and any one of multiple devices included in a medical information processing system may have the configurations described above. Specifically, a medical information processing system includes obtaining circuitry241, generation circuitry242, determination circuitry243, and storage circuitry244. The obtaining circuitry241obtains motion information indicating a motion of a person. The generation circuitry242generates the evaluation information of rehabilitation from the motion information obtained by the obtaining circuitry241. The determination circuitry243then determines whether or not a person associated with the obtained motion information is a subject by using the subject motion feature information. The storage circuitry244then outputs the evaluation information on the basis of the determination result from the determination circuitry.

Furthermore, for example, although a case in which determination is performed by using a feature of a motion of a person has been described in the seventh to ninth embodiments above, the embodiment is not limited thereto.

In one example, the motion information processing device200may perform determination by using a speech recognition result. Specifically, as a result of concentrating on rehab, a subject tends to talk less or moan. In contrast, for caring for and encouraging a subject, a caregiver tends to utter words such as “are you all right?,” “Right foot next,” “Slowly,” and the like. Thus, the motion information processing device200can determine a subject and a caregiver by using these features.

Here, a case in which determination on whether or not a person is a subject is performed by using the feature that a subject tends to talk less will be described. In this case, the subject motion feature storage circuitry232stores information in which a predetermined motion ID and subject motion feature information “not talking for three minutes or longer” are associated with each other. The obtaining circuitry241then obtains motion information and a speech recognition result associated with each frame of the moLlon information. If a speech recognition result indicating not talking continues for three minutes for the motion information obtained by the obtaining circuitry241, the determination circuitry243then determines that the motion information corresponds to subject motion feature information with the predetermined motion ID, that is, the person is not talking for three minutes or longer. In this manner, the motion information processing device200can determine a subject with high accuracy by using a speech recognition result.

Furthermore, a case in which determination on whether or not a person is a caregiver is performed by using an utterance “Are you all right?” of a caregiver will be described. In this case, the caregiver motion feature storage circuitry233stores information in which a predetermined motion ID and caregiver motion feature information “uttering words ‘Are you all right?’” are associated with each other. The obtaining circuitry241then obtains motion information and a speech recognition result associated with each frame of the motion information. If a speech recognition result indicating the utterance of words “Are you all right?” is provided for the motion information obtained by the obtaining circuitry241, the determination circuitry243then determines that the motion information corresponds to caregiver motion feature information with the predetermined motion ID, that is, the person has uttered the words “Are you all right?” In this manner, the motion information processing device200can determine a caregiver with high accuracy by using a speech recognition result.

In another example, the motion information processing device200may perform determination by using a thermal sensor (thermograph, etc.). Specifically, a subject puts his/her efforts to moves his/her body, and thus the body temperature of the subject rises during rehab. Thus, the motion information processing device200can determine a subject by using this feature. In this case, the motion information processing device200includes a thermal sensor as the input circuitry120, and obtains the body temperature of a person associated with the motion information obtained by the obtaining circuitry241for each frame.

In one aspect, the subject motion feature storage circuitry232stores information in which a predetermined motion ID and subject motion feature information “body temperature being 36.5 degrees or higher” are associated. The obtaining circuitry241then obtains the motion information and the body temperature associated with each frame of the motion information. If the body temperature is 36.5 degrees or higher for the motion information obtained by the obtaining circuitry241, the determination circuitry243then determines that the motion information corresponds to subject motion feature information with the predetermined ID. In this manner, the motion information processing device200can determine a subject with high accuracy by using a thermal sensor.

As described above, according to the seventh to tenth embodiments, the motion information processing device200can determine whether or not a digitally recorded motion of a person is that of a subject of rehabilitation.

Eleventh Embodiment

As described above, in the seventh to tenth embodiments, a case of enabling determination on whether or not a digitally recorded motion of a person is that of a subject of rehabilitation to support rehabilitation has been described. Thus, in eleventh and twelfth embodiments, a case of providing a motion information processing device300capable of providing information with which a motion can be confirmed while hiding personal information of a test subject (subject) will be described. Note that, in the eleventh and twelfth embodiments, the test subject includes a person carrying out a predetermined motion and a person who is a subject of rehabilitation.

FIG. 35is a block diagram illustrating an example configuration of a medical information processing system1according to the eleventh embodiment. The medical information processing system1according to the eleventh embodiment is a system for supporting rehabilitation carried out in a medical institution, at home, in an office, or the like. Although a case in which the motion information processing device300is applied to the medical information processing system1will be described in the eleventh embodiment, the embodiment is not limited thereto. For example, the motion information processing device300can also provide information with which a motion can be confirmed while hiding personal information of a test subject without being applied to the medical information processing system1.

As illustrated inFIG. 35, the medical information processing system1according to the eleventh embodiment includes a motion information collector10, the motion information processing device300, a medical image archiving device20, and a workstation30. Note that the motion information processing device300, the medical image archiving device20, and the workstation30are in state in which the devices can mutually communicate directly or indirectly via a local area network (LAN)40installed in a hospital, for example. For example, in a case where a picture archiving and communication system (PACS) is introduced in the medical information processing system1, for example, the devices transmit and receive medical image information and the like according to the DICOM (Digital Imaging and Communications in Medicine) standard.

The medical image archiving device20archives various information data for supporting rehabilitation. For example, the medical image archiving device20includes a database that archives image information, stores various information data transmitted by the motion information processing device300into the database, and holds the information data. In one example, in a case where an electronic medical record (EMR) is applied to the medical information processing system1, the medical image archiving device20stores information stored on the EMR. The information stored in the medical image archiving device20can be browsed by those authorized to access the medical information processing system1or having a predetermined authority such as an attending doctor or a hospital director, for example. Information stored in the medical image archiving device20will be described later.

Although a case in which the medical image archiving device20archives information in the medical information processing system1in a certain medical institution will be described in the eleventh embodiment, the embodiment is not limited thereto. For example, the medical image archiving device20may be provided on the Internet. In this case, the medical image archiving device20may archive information of multiple medical institutions connected to the Internet for each of the medical institutions, or may archive information of each medical institution in such a manner that the information can be mutually browsed. Furthermore, the medical image archiving device20may archive information data with an access authority set to each of the information data stored in the medical image archiving device20.

The workstation30displays various information data for supporting rehabilitation. For example, the workstation30obtains motion information from the medical image archiving device20, and displays the obtained information.

The motion information processing device300performs various processes for supporting rehabilitation by using motion information output from the motion information collector10. The motion information processing device300is an information processing device such as a computer or a workstation, for example.

In the medical information processing system1, it is considered useful in medical practice to archive motion information in rehab collected by the motion information collector10as information that can be browsed by doctors and physical therapist other than an attending doctor in the medical image archiving device20, for example. For example, a doctor can show information (a moving image or skeleton information) with which a motion in rehab of a test subject can be checked as a good example to another test subject carrying out similar rehab by using the information stored in the medical image archiving device20. This behavior, however, discloses personal information of the test subject presented as a good example to another test subject, which is undesirable in terms of protection of personal information.

Thus, the motion information processing device300according to the eleventh embodiment can provide information with which a motion can be confirmed while hiding personal information of a test subject by performing processing as will be described below.

FIG. 36is a block diagram illustrating a detailed example configuration of the motion information processing device300according to the eleventh embodiment. As illustrated inFIG. 36, in the motion information processing device300, storage circuitry330includes motion information storage circuitry331.

The motion information storage circuitry331stores various information data collected by the motion information collector10. Specifically, the motion information storage circuitry331stores motion information generated by the motion information generation circuitry14. More specifically, the motion information storage circuitry331stores skeleton information of each frame generated by the motion information generation circuitry14. Note that the motion information storage circuitry331can also associate color image information, depth image information, and a speech recognition result output by the motion information generation circuitry14for each frame and store the association result. Note that the color image information is an example of photographed image information.

The motion information storage circuitry331according to the eleventh embodiment stores information similar to that stored in the motion information storage circuitry131illustrated inFIG. 7, for example.

The description refers back toFIG. 36. As illustrated inFIG. 36, in the motion information processing device300, control circuitry340includes obtaining circuitry341, generation circuitry342, and output control circuitry343.

The obtaining circuitry341obtains motion information of a test subject. For example, the obtaining circuitry341obtains motion information at least containing skeleton information and color image information in time series. In one example, each time the motion information collector10and the motion information processing device300are powered on and skeleton information of one frame is stored in the motion information storage circuitry331, the obtaining circuitry341obtains the stored skeleton information and color image information of a frame corresponding to the stored skeleton information from the motion information storage circuitry331. Note that the motion information obtained by the obtaining circuitry341is an example of first motion information.

Although a case in which, each time skeleton information and color image information of each frame are stored in the motion information storage circuitry331by the motion information collector10, the obtaining circuitry341obtains various information data in real time will be described in the eleventh embodiment, the embodiment is not limited thereto. For example, the obtaining circuitry341may obtain a group of frames of skeleton information and color image information stored in the motion information storage circuitry331.

The generation circuitry342generates image information, with which the skeleton of a test subject can be visually confirmed, as medical image information by using the positions of joints contained in the skeleton information, for example.

FIGS. 37A and 37Bare diagrams for explaining processing performed by the generation circuitry342according to the eleventh embodiment.FIG. 37Aillustrates color image information presenting a state in which a test subject is carrying out gait training and skeleton information indicating the positions of joints of the test subject.FIG. 37Billustrates medical image information37agenerated on the basis of the color image information and skeleton information inFIG. 37A.

As illustrated inFIGS. 37A and 37B, the generation circuitry342generates a skeleton model37bindicating the positions of joints2ato2tcontained in the skeleton information obtained by the obtaining circuitry341. Specifically, for example, the generation circuitry342places circles at positions corresponding to pixel positions X and pixel positions Y of the joints2ato2tin a photographed range of the medical image information37a. The generation circuitry342also places lines connecting the circles on the basis of information indicating relative positions of the joints. As a result, the generation circuitry342generates the skeleton model37b.

The generation circuitry342also generates a floor surface image37c. Specifically, for example, the generation circuitry342specifies a pixel corresponding to the position of y=0 in the world coordinate system as a pixel of a floor surface. The generation circuitry342then generates the floor surface image37cby placing parallel lines at intervals of 1 m in the x direction and the z direction among the pixels of the floor surface.

In this manner, the generation circuitry342generates the medical image information37acontaining the skeleton model37band the floor surface image37c. The generation circuitry342then outputs the generated medical image information37ato the output control circuitry343.

Note that the above example is only an example, and the embodiment is not limited thereto. For example, the process in which the generation circuitry342generates the floor surface image37cis not limited to the above example. For example, in a case where coordinates of the floor surface in the world coordinate system can be obtained from the motion information collector10or other sensors, the generation circuitry342may specify the pixels by using the obtained coordinates. The generation circuitry342can generate the floor surface image37cby placing parallel lines at intervals of 1 m in the x direction and the z direction among the specified pixels of the floor surface. Furthermore, alternatively, the generation circuitry342may specify pixels corresponding to positions where the position of a tarsus (the joint2por the joint2t) of a test subject does not change for a predetermined time as pixels of the floor surface, for example. Alternatively, for example, the floor surface image37cneed not necessarily be contained in the medical image information37a.

The output control circuitry343outputs motion information for browsing containing skeleton information and medical image information in time series. In one example, the output control circuitry343associates the skeleton information of the test subject and the medical image information37agenerated by the generation circuitry342in time series, and stores the association result as the motion information for browsing in the medical image archiving device20. Note that the motion information for browsing is an example of second motion information.

FIG. 38is a table illustrating an example of motion information for browsing stored in the medical image archiving device20according to the eleventh embodiment. The medical image archiving device20stores information in which a name number, a date, and motion information for browsing are associated with each name as illustrated inFIG. 38. Note that the “name number” refers to an identifier for uniquely identifying a test subject, and is assigned to each name. The “date” refers to the date and time when the test subject has carried out gait training. The “motion information for browsing” refers to information stored in the medical image archiving device20by the output control circuitry343.

As illustrated inFIG. 38, the medical image archiving device20stores a name “A,” a name number “1,” a date “20120801_1,” motion information for browsing “medical image information, depth image information, speech recognition result, skeleton information” and the like. The above information indicates that motion information for browsing containing “medical image information,” “depth image information,” a “speech recognition result,” and “skeleton information” is stored as motion information in the “first” gait training carried out by a person with a name “A” whose name number is “1” on “Aug. 1” in “2012.” Note that, in the motion information for browsing illustrated inFIG. 38, the “medical image information,” the “depth image information,” the “speech recognition result” and the “skeleton information” for each of all the frames photographed during the gait training are stored in association with time in time series.

In this manner, the output control circuitry343associates the skeleton information of the test subject and the medical image information37agenerated by the generation circuitry342in time series, and stores the association result as the motion information for browsing in the medical image archiving device20. Alternatively, the output control circuitry343may display the motion information for browsing on the output circuitry110.

FIG. 39is a flowchart illustrating procedures of processing performed by the motion information processing device300according to the eleventh embodiment. As illustrated inFIG. 39, the obtaining circuitry341obtains motion information containing skeleton information and color image information (step S601).

Subsequently, the generation circuitry342generates medical image information with which a motion of a test subject can be confirmed without containing personal information of the test subject (step S602). For example, the generation circuitry342generates image information, with which the skeleton of a test subject can be visually confirmed, as medical image information by using the positions of joints contained in the skeleton information.

The output control circuitry343then outputs motion information for browsing containing skeleton information and medical image information in time series (step S603). For example, the output control circuitry343associates the skeleton information of the test subject and the medical image information37agenerated by the generation circuitry342in time series, and stores the association result as the motion information for browsing in the medical image archiving device20.

As described above, the motion information processing device300according to the eleventh embodiment obtains motion information containing skeleton information and color image information. The motion information processing device300then generates medical image information with which a motion of a test subject can be confirmed without containing personal information of the test subject on the basis of at least one of the skeleton information and the color image information. Thus, the motion information processing device300can provide information with which a motion can be confirmed while hiding personal information of a test subject.

Specifically, since the test subject is photographed in color image information, the test subject can be identified by viewing the color image information. It is thus undesirable in terms of protection of personal information to hold the color image information in the medical image archiving device20. Thus, the motion information processing device300according to the eleventh embodiment generates the medical image information37awith which a motion of a test. subject can be confirmed without containing the photographed test subject, and stores the generated medical image information37ainstead of the color image information in the medical image archiving device20. The motion information processing device300according to the eleventh embodiment can therefore provide information with which a motion can be confirmed while hiding personal information of a test subject. As a result, since a test subject is not identified when the medical image information37aof the subject is viewed by others, the motion information processing device300can provide image information that is clinically useful while protecting personal information of the test subject. Specifically, a doctor can show the medical image information37aof a test subject as a good example to another test subject or as a guideline such as “a healthier condition one month later as a result of carrying out the rehab.”

Furthermore, for example, the motion information processing device300according to the eleventh embodiment stores motion information for browsing in the medical image archiving device20in consideration of protection of personal information, which allows processing of motion information for browsing of multiple test subjects. As a result, the motion information processing device300allows feature quantities of each individual or each disease, for example, to be obtained more accurately by statistically processing the motion information for browsing.

Furthermore, as described above, in a case where the medical image archiving device20is provided on a network such as the Internet of a LAN, the motion information processing device300can store clinically-useful image information in the medical image archiving device20on the network while protecting personal information of test subjects. As a result, the motion information processing device300can share information with remote medical institutions. For example, it is possible to share information on a common subject among doctors and physical therapists in regional cooperation, and get advice from a remote doctor, for example.

Modified Example 1 of Eleventh Embodiment

Although a case in which the motion information processing device300generates medical image information37acontaining a skeleton model37bhas been described in the embodiment above, the embodiment is not limited thereto. For example, the motion information processing device300may generate image information in which a part of the image containing features of the test subject included in the color image information is hidden, as the medical image information. Note that the part containing features of the test subject is the head (face), for example. In the following, a case in which the motion information processing device300generates image information in which the image of the head of a test subject contained in color image information is hidden will be described as an example.

In the motion information processing device300according to modified example 1 of the eleventh embodiment, the generation circuitry342generates image information in which at least part of an image of the head of a test subject contained in color image information is hidden, as medical image information, by using the position of the joint at the head contained in the skeleton information.

FIG. 40is a diagram for explaining processing performed by the generation circuitry342according to the modified example 1 of the eleventh embodiment.FIG. 40illustrates medical image information40agenerated on the basis of the color image information and skeleton information inFIG. 37A.

As illustrated inFIG. 40, the generation circuitry342hides an image of the head of a test subject contained in color image information by using the position of the joint at the head of the test subject contained in the skeleton information obtained by the obtaining circuitry341. Specifically, the generation circuitry342specifies a pixel position40bat the center of the head in the color image information by using the coordinates of the joint2aof the head contained in the skeleton information. The generation circuitry342then changes the color of pixels contained in a head area40ccontaining the pixel position40bto a predetermined color (gray, for example). The head area40cis a rectangular area of a predetermined size having the pixel position40bat the center, for example. The generation circuitry342then outputs the color image information resulting from changing the color of the pixels contained in the head area40cas the medical image information40ato the output control circuitry343. As a result, the output control circuitry343associates the skeleton information of the test subject and the medical image information40agenerated by the generation circuitry342in time series, and stores the association result as the motion information for browsing in the medical image archiving device20.

As described above, the motion information processing device300according to the modified example 1 of the eleventh embodiment generates the medical image information40aby hiding the image of the head of the test subject contained in the color image information. Specifically, the motion information processing device300generates medical image information with which a motion of a test subject can be confirmed without containing personal information of the test subject on the basis of at least one of the skeleton information and the color image information. The motion information processing device300then stores the generated medical image information instead of the color image information in the medical image archiving device20. The motion information processing device300according to the eleventh embodiment can therefore provide information with which a motion can be confirmed while hiding personal information of a test subject.

Note that the size of the head area40ccan be changed to any size determined by an operator of the motion information processing device300. Alternatively, the size of the head area40cmay be changed according to the value of the z coordinate of the joint2a. In this case, for example, the motion information processing device300sets the head area40cto be small if the z coordinate value of the joint2ais large (if the test subject is far), or sets the head area40cto be large if the z coordinate value of the joint2ais small (if the test subject is close). Furthermore, although it is preferable that the size of the head area40cbe large enough to hide the entire head, the size is not limited thereto, and the head area40cmay have such a size that only hides the position of the eyes. Furthermore, the shape of the head area40cneed not necessarily be a rectangle. For example, the shape of the head area40cmay be a circle, or the contour of the face may be determined by a face recognition technology with the joint2aat the center, and a range corresponding to the determined contour may be defined as the head area40c.

Furthermore, although a case in which image information in which the image of the head is hidden as one example of the part containing features of a test subject has been described here, the embodiment is not limited thereto. For example, the motion information processing device300may detect a feature of a physical skeleton shape as the part containing features of the test subject, and hide the detected feature. Examples of a feature of the physical skeleton shape include loss of an arm. For example, for a test subject whose left arm is lost, the motion information processing device300generates image information in which a part containing a feature of the test subject is hidden by hiding the position corresponding to the left arm. For example, the generation circuitry342refers to the skeleton information of the test subject, and determines whether or not position information on all of the joints2ato2tis present. If the joints2j,2k, and2lare not present, the generation circuitry342estimates the positions of the joints2j,2k, and2lof the test subject. In one example, the generation circuitry342estimates the positions that are symmetrical with the positions of the joints2f,2g, and2hof the right arm about a center line of the body (a line passing through the joints2band2cto be the positions of the joints2j,2k, and2l. The generation circuitry342then changes the color of the pixels contained in an area (a rectangular area, for example) containing the estimated joints2j,2k, and2lof the left arm to a predetermined color. In this manner, the generation circuitry342generates image information in which the position corresponding to the left arm of the test subject is hidden.

Alternatively, for example, the motion information processing device300may detect a feature appearing on the skin of the test subject as a part containing the feature of the test subject, and hide the detected feature. A feature appearing on the skin is a scar (due to an injury) or the like. For example, for a test subject whose left arm has a scar, the motion information processing device300generates image information in which a part containing a feature of the test subject is hidden by hiding the position corresponding to the left arm. For example, the generation circuitry342generates a pixel histogram for an area of the color image information containing the positions corresponding to skeleton information of a test subject. When a peak is detected from a luminance histogram corresponding to the skin color, the generation circuitry342then changes the color of pixels in the area containing the position corresponding to the peak to a predetermined color. This is because, while a uniform luminance histogram can be obtained for a normal skin, a feature of the skin such as a car is considered present when a peak is detected from the luminance histogram. In this manner, the generation circuitry342generates image information in which the position corresponding to the left arm of the test subject is hidden.

Furthermore, the motion information processing device300is not limited to the examples above, and may generate image information in which an area specified by an operator is hidden. In one example, the generation circuitry342receives specification of an area from the operator via the input circuitry120such as a mouse. The generation circuitry342then generates image information in which the received area is hidden.

Furthermore, the generation circuitry342according to the modified example 1 of the eleventh embodiment may generate image information containing the floor surface image37cdescribed above as medical image information. For example, the generation circuitry342can generate the floor surface image37cas described above and generate image information in which the generated floor surface image37cis superimposed on the medical image information40a.

Modified Example 2 of Eleventh Embodiment

Furthermore, there are cases in which color image information contains a photographed person or object other than test subjects, which should not be disclosed. For example, color image information may contain the face of a caregiver assisting rehab of a test subject or a notice or the like that should not be disclosed. Thus, in addition to the modified example 1 of the eleventh embodiment, the motion information processing device300may generate image information in which an area in which a person or an object other than a test subject is photographed contained in color image information is hidden, as medical image information.

In the motion information processing device300according to modified example 2 of the eleventh embodiment, the generation circuitry342further generates image information in which at least part of an image of a person or an object other than a test subject contained in color image information is photographed is hidden, as medical image information, by using the position of the joints contained in the skeleton information.

FIG. 41is a diagram for explaining processing performed by the generation circuitry342according to the modified example 2 of the eleventh embodiment.FIG. 41illustrates medical image information41agenerated on the basis of the medical image information40ainFIG. 40.

As illustrated inFIG. 41, the generation circuitry342further hides an image in which a person or an object other than a test subject contained in color image information is photographed by using the positions of the joints contained in the skeleton information. Specifically, the generation circuitry342specifies a test subject area41bcontaining a test subject by using the positions of the joints contained in skeleton information. More specifically, the generation circuitry342specifies a joint with the largest x coordinate value of the x coordinates of the joints of the test subject, and determines a line passing through a coordinate obtained by adding a predetermined value to the x coordinate value of the specified joint to be the right side of the test subject area41b. The generation circuitry342also specifies a joint with the smallest x coordinate value of the x coordinates of the joints of the test subject, and determines a line passing through a coordinate obtained by subtracting a predetermined value to the x coordinate value of the specified joint to be the left side of the test subject area41b. The generation circuitry342also specifies a joint with the largest y coordinate value of the y coordinates of the joints of the test subject, and determines a line passing through a coordinate obtained by adding a predetermined value to the y coordinate value of the specified joint to be the upper side of the test subject area41b. The generation circuitry342also specifies a joint with the smallest y coordinate value of the y coordinates of the joints of the test subject, and determines a line passing through a coordinate obtained by subtracting a predetermined value to the y coordinate value of the specified joint to be the lower side of the test subject area41b. The generation circuitry342then changes the color of pixels outside of the specified test subject area41bto a predetermined color (gray, for example). The generation circuitry342then outputs the color image information resulting from changing the color of the pixels outside of the test subject area41bas the medical image information41ato the output control circuitry343. As a result, the output control circuitry343associates the skeleton information of the test subject and the medical image information41agenerated by the generation circuitry342in time series, and stores the association result as the motion information for browsing in the medical image archiving device20.

In this manner, the motion information processing device300according to the modified example 2 of the eleventh embodiment generates medical image information41aby hiding an area in which a person or an object other than a test subject is photographed, and stores the generated medical image information41ainstead of the color image information in the medical image archiving device20. The motion information processing device300according to the eleventh embodiment can therefore provide information with which a motion of a test subject can be confirmed while hiding a photographed person or object other than the test subject, which should not be disclosed. For example, the motion information processing device300can also hide the face of a caregiver assisting rehab of test subject a or a notice or the like that should not be disclosed.

Note that the size of the test subject area41bcan be changed to any size determined by an operator of the motion information processing device300. Furthermore, the shape of the test subject area41bneed not necessarily be a rectangle. For example, the shape of the test subject area41bmay be an ellipse.

Furthermore, the generation circuitry342according to the modified example 2 of the eleventh embodiment may generate image information containing the floor surface image37cdescribed above as medical image information. For example, the generation circuitry342can generate the floor surface image37cas described above and generate image information in which the generated floor surface image37cis superimposed on the medical image information41a.

Modified Example 3 of Eleventh Embodiment

Furthermore, the embodiment is not limited to the embodiments described above, and the motion information processing device300may generate image information presenting only a test substrate area as medical image information, for example.

In the motion information processing device300according to the modified example 3 of the eleventh embodiment, the obtaining circuitry341obtains motion information further containing pixels contained in a photographed range and depth image information in which the pixels and depths are associated with each other in time series. For example, each time the motion information collector10and the motion information processing device300are powered on and depth information of one frame is stored in the motion information storage circuitry331, the obtaining circuitry341obtains the stored depth image information from the motion information storage circuitry331.

In the motion information processing device300according to the modified example 3 of the eleventh embodiment, the generation circuitry342generates image information representing an area of a test subject as medical image information from the depth image information at the time point when the medical image information is processed on the basis of the depth image information at the time point when the medical image information is processed and depth image information at a time point different from the aforementioned depth image information.

FIG. 42is a diagram for explaining processing performed by the generation circuitry342according to the modified example 3 of the eleventh embodiment.FIG. 42illustrates a depth image42aof a frame T, and a depth image42band a binarized image42cof a frame T−1. In the depth image42aof the frame T and the depth image42bof the frame T−1, a solid line represents the position of a test subject in each frame. In addition, in the depth image42aof the frame T, a broken line represents the position of the test subject in the frame T−1. Thus, the depth image42aof the frame T indicates that the test subject has moved from the position of the broken line to the position of the solid line. Note that a “pixel position X” and a “pixel position Y” in the binarized image42ccorrespond to a “pixel position X” and a “pixel position Y” in the distance image coordinate system. In addition, the broken line indicates the position of the test subject in the frame T−1 for convenience, and the position of the test subject in the frame T−1 is not illustrated in the depth image42aof the frame T.

As illustrated inFIG. 42, when the depth image42aof the frame T is obtained by the obtaining circuitry341, the generation circuitry342subtracts the depth of the depth image42bof the frame T−1 from the depth of the depth image42aof the frame T for each pixel. The generation circuitry342then generates the binarized image42cof the frame T by performing binarization of setting pixels with values as a result of subtraction are not smaller than a threshold to white and setting pixels with values smaller than the threshold to black. In the binarized image42c, an area of black pixels represents a position of an object that has moved less than a threshold in the depth direction between the frame T−1 and the frame T, that is, an object such as the floor surface, a wall, a desk, or a chair. An area of white pixels represents a position of a photographed person (a photographed person or object) that has moved equal to or more than the threshold in the depth direction between the frame T−1 and the frame T. Specifically, the generation circuitry342extracts a test subject area42drepresenting the position of a test subject carrying out a walking motion by identifying an object that has not moved and an object that has moved. The generation circuitry342then outputs the binarized image42cfrom which the test subject area42dhas been extracted as medical image information to the output control circuitry343. As a result, the output control circuitry343associates the skeleton information of the test subject and the medical image information generated by the generation circuitry342in time series, and stores the association result as the motion information for browsing in the medical image archiving device20.

In this manner, the motion information processing device300according to the modified example 3 of the eleventh embodiment generates image information indicating only an area of a test subject as medical image information, and stores the medical image information instead of the color image information in the medical image archiving device20. The motion information processing device300according to the eleventh embodiment can therefore provide information with which a motion can be confirmed while hiding personal information of a test subject.

Note that the processing of the generation circuitry342is only an example, and is not limited thereto. For example, although the generation circuitry342extracts the test subject area by using depth image information of successive frames in time series in the above description, the extraction is not limited thereto. For example, the generation circuitry342may calculate the difference in depth for each pixel between depth image information data of frames separated by several frames in time series, and extract a pixel area where the calculated difference is equal to or larger than a threshold as the test subject area. Alternatively, for example, the generation circuitry342may use depth image information in which no person is present as a reference, calculate the difference in depth for each pixel between the depth image information of the frame T and the reference depth image information, and extract a pixel area where the calculated difference is equal to or larger than a threshold as the test subject area.

Furthermore, the generation circuitry342according to the modified example 3 of the eleventh embodiment may generate image information containing the floor surface image37cdescribed above as medical image information. For example, the generation circuitry342can generate the floor surface image37cas described above and generate image information in which the generated floor surface image37cis superimposed on the binarized image42c.

Twelfth Embodiment

Although a case in which the motion information processing device300hides personal information of a test subject when storing motion information for browsing in the medical image archiving device20has been described in the above embodiment, the embodiment is not limited thereto. For example, when motion information for browsing of another test subject is shown as a good example to a test subject at the workstation30, an image of the test subject may be displayed on the basis of the motion of the motion information for browsing that is a good example. As a result, the test subject can view an image in which the test subject himself/herself moves as if the test subject is carrying out a motion of the good example.

A medical information processing system1according to the twelfth embodiment has a configuration similar to that of the medical information processing system1illustrated inFIG. 35, but differs therefrom in part of processing at the workstation30. In the twelfth embodiment, the description will be focused mainly on the difference from the eleventh embodiment, and components having the same functions as those described in the eleventh embodiment will be designated by the same reference numerals as those inFIG. 35and the description thereof will not be repeated.

FIG. 43is a block diagram illustrating a detailed example configuration of the workstation30according to the twelfth embodiment. As illustrated inFIG. 43, the workstation30includes display information storage circuitry31and display control circuitry32.

The display information storage circuitry31stores display information in which test subject image information representing a test subject and the positions of joints of the test subject in the test subject image information are associated.

FIG. 44is a diagram illustrating an example of information stored in the display information storage circuitry31. InFIG. 44, a case in which the test subject is an elderly female will be described. In this case, the display information storage circuitry31stores information in which image information of the test subject and the positions of the joints2ato2tof the test subject are associated as display information44a. Specifically, the display information storage circuitry31stores positional relations of the positions of the joints of the test subject and the positions of the pixels contained in the image information of the test subject.

Although the display information44acontaining a character of an elderly female as test subject image information is presented in the example illustrated inFIG. 44for convenience of explanation, the embodiment is not limited thereto. For example, male and female characters c all age groups such as 0 to 9 years old, 10 to 19 years old, 20 to 29 years old, and so on may be provided in the display information storage circuitry31in such a manner that the characters can be selected according to the age and sex of the test subject. Furthermore, alternatively, the display information storage circuitry31may store the display information44acontaining a photograph of the test subject as test subject image information.

The display control circuitry32obtains motion information for browsing from the medical image archiving device20. The display control circuitry32then performs display control to display an image of the test subject on the basis of the motion information for browsing by mapping image information on the test subject on the medical image information contained in the motion information for browsing on the basis of the positions of joints contained in the display information.

FIG. 45is a diagram for explaining processing performed by the display control circuitry32according to the twelfth embodiment.FIG. 45illustrates a display image45athat is browsed by a test subject at the workstation30. InFIG. 45, a case in which pixels contained in the display information44ainFIG. 44on the medical image information37ainFIG. 37Bwill be explained as an example.

As illustrated inFIG. 45, for example, the display control circuitry32obtains motion information for browsing from the medical image archiving device20. The motion information for browsing contains the medical image information37ainFIG. 37B. The display control circuitry32then performs mapping of image information of the test subject on the medical image information37aon the basis of the positions of the joints contained in the display information44astored in the display information storage circuitry31. Specifically, the display control circuitry32positions the pixels contained in the display information44aon the medical image information37aon the basis of the positional relation between the pixel positions of the pixels and the positions of the joints. More specifically, for example, the display control circuitry32positions the pixels at the position of the joint2ain the display information44aat the position of the joint2aof the head in the medical image information37a. Furthermore, for example, the pixels at the positions of hair in the display information44aare positioned at positions of ΔX and ΔY from the position of the head in the medical image information37aby using the positional relation of the pixels and the position of the joint2aof the head (a distance ΔX in the X direction and a distance ΔY in the Y direction). As a result, the display control circuitry32generates the display image45athat displays the display information44aof an elderly female with the posture of the medical image information37a. The display control circuitry32generates a floor surface image by the same processing as that of the generation circuitry342according to the first embodiment, and superimposes the generated floor surface image on the display image45a. The display control circuitry32then displays the display image45aon which the floor surface image is superimposed in time series to display a moving image appearing as if the elderly female is carrying out the motion of the medical image information37a. Note that the display control circuitry32need not necessarily generates a floor surface image.

As described above, the workstation30according to the twelfth embodiment, the image information of a test subject is displayed on the basis of the motion of the motion information for browsing of another test subject. Thus, the workstation30can display, to a test subject, an image appearing as if the test subject is carrying out a motion of another test subject. As a result, since the image appears to the test subject as if the test subject is carrying out a motion of another person, the test subject can easily and practically visualize himself/herself, in a healthier condition, in a month time after starting rehab. As a result, the motion information processing device300can encourage motivation of the test subject who carries out rehab.

Although a case in which the display control circuitry32displays the display image44aon the basis of the medical image information37ainFIG. 37Bhas been described in the twelfth embodiment, the embodiment is not limited thereto. For example, the display control circuitry32may display the display image44aon the basis of the medical image information40ainFIG. 40or may display the display image44aon the basis of the binarized image42cinFIG. 42. Alternatively, the display control circuitry32may obtain motion information from the motion information processing device300and display the display image44aon the basis of color image information and skeleton information, for example. In this case, the display control circuitry32performs the processing of the display control circuitry32after performing the processing of the generation circuitry342.

As described above, according to the eleventh and twelfth embodiments, the motion information processing device of the present embodiment can provide information with which a motion can be confirmed while hiding personal information of a test subject.

Thirteenth Embodiment

As described above, in the first to sixth embodiments, a case of allowing prevention of falsely recognizing a person or object other than the person carrying out a predetermined motion has been described. Furthermore, in the seventh to tenth embodiments, a case of determining whether or not a digitally recorded motion of a person is that of a subject of rehabilitation has been described. Furthermore, in the eleventh and twelfth embodiments, a case in which a digitally recorded motion of a person provides information with which a motion can be confirmed while hiding personal information of a test subject has been described. Note that the processes described above can be performed as a series of processes. Thus, in a thirteenth embodiment, a case in which the processes described above are performed as a series of processes will be described. As a result, a motion information processing device400according to the thirteenth embodiment can reliably determine motion information of a rehab subject, and further hide personal information of the rehab subject. Note that, in the thirteenth embodiment, a person carrying out a predetermined motion will be referred to as a “subject,” and a subject of rehabilitation will be referred to as a “rehab subject.”

FIG. 46is a block diagram illustrating an example configuration of the motion information processing device400according to the thirteenth embodiment. As illustrated inFIG. 46, the motion information processing device400is an information processing device such as a computer or a workstation, and includes output circuitry110, input circuitry120, storage circuitry430, and control circuitry440. Since the output circuitry110and the input circuitry120are similar to those described in the first embodiment, the description thereof will not be repeated.

The storage circuitry430has a configuration similar to that of the storage circuitry230illustrated inFIG. 19. Specifically, the storage circuitry430includes motion information storage circuitry431, subject motion feature storage circuitry432, caregiver motion feature storage circuitry433, subject image feature storage circuitry434, caregiver image feature storage circuitry435, and evaluation information storage circuitry436. In the thirteenth embodiment, since the motion information storage circuitry431, the subject motion feature storage circuitry432, the caregiver motion feature storage circuitry433, the subject image feature storage circuitry434, the caregiver image feature storage circuitry435, and the evaluation information storage circuitry436are the same as the motion information storage circuitry231, the subject motion feature storage circuitry232, the caregiver motion feature storage circuitry233, the subject image feature storage circuitry234, the caregiver image feature storage circuitry235, and the evaluation information storage circuitry236illustrated inFIG. 19, the description thereof will not be repeated.

The control circuitry440includes obtaining circuitry441, specification circuitry442, first generation circuitry443, determination circuitry444, second generation circuitry445, and output control circuitry446.

The obtaining circuitry441has the same functions as those of the obtaining circuitry141illustrated inFIG. 4. For example, the obtaining circuitry441obtains motion information stored in the motion information storage circuitry431. Note that the obtaining circuitry441need not necessarily obtain motion information containing all of the color image information, the distance image information, the speech recognition result, and the skeleton information. For example, the obtaining circuitry441may obtain at least one of the color image information, the distance image information, the speech recognition result, and the skeleton information, where necessary.

The specification circuitry442has the same functions as those of the specification circuitry142illustrated inFIG. 4. For example, the specification circuitry442specifies motion information of the subject carrying out the predetermined motion on the basis of predetermined features in the image information obtained by the obtaining circuitry441.

The first generation circuitry443has the same functions as those of the generation circuitry242illustrated inFIG. 19. For example, the first generation circuitry443generates evaluation information for evaluating rehabilitation from the motion information specified by the specification circuitry442.

The determination circuitry444has the same functions as those of the determination circuitry243illustrated inFIG. 19. For example, the determination circuitry444determines whether or not a person associated with motion information specified by the specification circuitry442is a subject of rehabilitation by using information indicating a feature of a rehab subject that is a subject of rehabilitation.

The second generation circuitry445has the same functions as those of the generation circuitry342illustrated inFIG. 36. For example, the second generation circuitry445generates medical image information with which a motion of a subject can be confirmed without containing personal information of the subject on the basis of at least one of skeleton information representing a skeleton of a human body, and color image information in which the subject is photographed contained in time series in the motion information specified by the specification circuitry442. Furthermore, for example, the second generation circuitry445generates medical image information of the subject on the basis of at least one the skeleton information and color image information contained in time series in motion information determined to correspond to the subject by the determination circuitry.

The output control circuitry446has the functions of the display control circuitry143illustrated inFIG. 4, the functions of the storage circuitry244illustrated inFIG. 19, and the functions of the output control circuitry343illustrated inFIG. 36. The output control circuitry446performs output control where appropriate.

For example, when the motion information of the subject is specified by the specification circuitry442, the output control circuitry446performs control to display a display image in which information indicating the subject is superimposed on the image information at a position corresponding to the motion information specified by the specification circuitry442.

When determination is performed by the determination circuitry444, the output control circuitry446also output a determination result of determination by the determination circuitry444, for example. Specifically, the output control circuitry446associates the evaluation information generated by the first generation circuitry443and the determination result of determination by the determination circuitry444with each other, and stores the association result in the evaluation information storage circuitry436.

Furthermore, when medical image information is generated by the second generation circuitry445, the output control circuitry446outputs motion information for browsing containing skeleton information and the medical image information in time series, for example. In one example, the output control circuitry446associates skeleton information of a test subject and medical image information generated by the second generation circuitry445in time series, and stores the association result as motion information for browsing in the medical image archiving device20.

FIG. 47is a flowchart illustrating procedures of the processing performed by the motion information processing device400according to the thirteenth embodiment. As illustrated inFIG. 47, in the motion information processing device400according to the thirteenth embodiment, the obtaining circuitry441obtains motion information (step S701). For example, the obtaining circuitry441obtains color image information and skeleton information as the motion information.

Subsequently, the specification circuitry442performs a subject specification process (step S702). The subject specification process corresponds to the processing in steps S103to S105illustrated inFIG. 9. Specifically, the specification circuitry442specifies motion information of the subject by performing the subject specification process.

Subsequently, the first generation circuitry443and the determination circuitry444perform the subject determination process (step S703). The subject determination process corresponds to the processing in steps S303and S304illustrated inFIG. 27. Specifically, the first generation circuitry443generates evaluation information for evaluating rehabilitation from the motion information specified by the specification circuitry442. The determination circuitry444then determines whether or not a person associated with motion information specified by the specification circuitry442is a subject of rehabilitation by using information indicating a feature of a rehab subject that is a subject of rehabilitation.

The second generation circuitry445then performs a medical image information generation process (step S704). The medical image information generation process corresponds to the processing in step S602illustrated inFIG. 39. Specifically, the second generation circuitry445generates medical image information of the subject on the basis of at least one the skeleton information and color image information contained in time series in motion information determined to correspond to the subject by the determination circuitry.

Thereafter, the output control circuitry446performs output control (step S705). For example, the output control circuitry446outputs motion information for browsing containing skeleton information and medical image information in time series.

Note that the procedures of processing illustrated inFIG. 47are not limited to the example described above. For example, although a case in which the motion information processing device400successively performs the subject specification process, the subject determination process, and the medical image information generation process as a series of processes after acquiring the motion information has been described in the example described above, the embodiment is not limited thereto. Specifically, the subject specification process, the subject determination process, and the medical image information generation process may be terminated when the output control circuitry446has stored necessary information as appropriate in the storage circuitry430after these processes have been performed or the obtaining circuitry441may obtain necessary information from the storage circuitry430as appropriate before performing these processes to start the processes.

Furthermore, for example, the subject specification process, the subject determination process, and the medical image information generation process need not necessarily be performed in the order described above. Specifically, these processes may be performed in the order of the medical image information generation process, the subject determination process, and the subject specification process.

Furthermore, for example, the subject specification process, the subject determination process, and the medical image information generation process need not necessarily be performed. Specifically, after performing the subject specification process, the motion information processing device400may perform the subject determination process or the medical image information generation process, and output information resulting from the process as appropriate. Alternatively, the motion information processing device400may sequentially perform the subject determination process and the medical image information generation process, and then output information resulting from the processes as appropriate.

As described above, the motion information processing device400according to the thirteenth embodiment performs the subject specification process, the subject determination process, and the medical image information generation process as a series of processes. As a result, the motion information processing device400according to the thirteenth embodiment can reliably determine motion information of a rehab subject, and further hide personal information of the rehab subject.

Other Embodiments

While the first to thirteenth embodiments have been described above, various different embodiments other than the embodiments described above can be employed.

Application to Service Providing Device

In the first to thirteenth embodiments described above, a case in which the motion information processing device specifies motion information (skeleton information) of a subject carrying out rehab has been described. The embodiment, however, is not limited thereto, and there may be cases where the processes are performed by a service providing device on a network, for example.

FIG. 48is a diagram for explaining an example of application to a service providing device. As illustrated inFIG. 48, a service providing device500is installed in a service center, and connected to terminal devices600installed in a medical institution, at home, and in an office via a network5, for example. The terminal devices600installed in the medical institution, at home, and in the office are each connected with a motion information collector10. The terminal devices600each have a client function of using services provided by the service providing device500.

For example, the service providing device500provides processes similar to those of the motion information processing device100described with reference toFIG. 4as services to the terminal devices600. Thus, the service providing device500has functional units similar to the obtaining circuitry141and the specification circuitry142. The functional unit similar to the obtaining circuitry141obtains image information of a subject carrying out a predetermined motion and the surroundings of the subject. The functional unit similar to the specification circuitry142specifies motion information of the subject carrying out a predetermined motion on the basis of predetermined features in the image information obtained by the functional unit similar to the obtaining circuitry141. For the network5, any type of wired or wireless communication network can be used, such as the Internet and a wide area network (WAN).

Furthermore, the service providing device500has functions similar to those of the motion information processing device200described with reference toFIG. 19, and provides services to the terminal devices600by these functions, for example. Specifically, the service providing device500has functional units similar to the obtaining circuitry241and the determination circuitry243. The functional unit similar to the obtaining circuitry241obtains motion information representing a motion of a person. The functional unit similar to the determination circuitry243determines whether or not the person associated with the motion information obtained by the functional unit similar to the obtaining circuitry241is a subject of rehabilitation by using information representing a feature of the subject. In view of the above, the service providing device500can determine whether or not a digitally recorded motion of a person is that of a subject of rehabilitation.

Furthermore, for example, the service providing device500has functions similar to those of the motion information processing device300described with reference toFIG. 36, and provides services to the terminal devices600by these functions. Specifically, the service providing device500has functional units similar to the obtaining circuitry341, the generation circuitry342, and the output control circuitry343. The functional unit similar to the obtaining circuitry341obtains motion information containing skeleton information representing positions of joints contained in the skeleton of the test subject and photographed image information in which the test subject is photographed in time series. The functional unit similar to the generation circuitry342generates medical image information with which a motion of a test subject can be confirmed without containing personal information of the test subject on the basis of at least one of the skeleton information and the photographed image information.

For example, the service providing device500accepts upload of frames of the motion information to be processed from the terminal devices600. The service providing device500then performs the processes described above to generate motion information for browsing. The service providing device500causes the motion information for browsing to be downloaded at the terminal devices600. The terminal devices600then stores the downloaded motion information for browsing in the medical image archiving device20.

Furthermore, for example, the service providing device500has functions similar to those of the motion information processing device400described with reference toFIG. 46, and provides services to the terminal devices600by these functions. Specifically, the service providing device500have functional units similar to the obtaining circuitry441, the specification circuitry442, the first generation circuitry443, the determination circuitry444, the second generation circuitry445, and the output control circuitry446. The functional unit similar to the obtaining circuitry441obtains image information of a subject carrying out a predetermined motion and the surroundings of the subject. The functional unit similar to the specification circuitry442specifies motion information of the subject carrying out a predetermined motion on the basis of predetermined features in the image information obtained by the functional unit similar to the obtaining circuitry441. The functional unit similar to the determination circuitry444determines whether or not the person associated with the motion information specified by the functional unit similar to the specification circuitry442is a subject of rehabilitation by using information representing a feature of the subject. The functional unit similar to the second generation circuitry445generates medical image information with which a motion of a test subject can be confirmed without containing personal information of the test subject on the basis of at least one of the skeleton information and the photographed image information. The functional unit similar to the output control circuitry446then outputs motion information for browsing containing skeleton information and medical image information in time series.

Note that the configuration of the motion information processing device100according to the first to sixth embodiments is only an example, and the components thereof can be integrated or divided where appropriate. For example, the motion information storage circuitry131and the recognition information storage circuitry132can be integrated, or the specification circuitry142may be divided into calculation circuitry that calculates distances between joints and the like and comparison circuitry that compares calculated values with thresholds.

Furthermore, the configuration of the motion information processing device200according to the seventh to tenth embodiments is only an example, and the components thereof can be integrated or divided where appropriate. For example, the subject motion feature storage circuitry232and the caregiver motion feature storage circuitry233can be integrated, or the determination circuitry243can be divided into processing circuitry that extracts a condition for performing the determination process from the storage circuitry230and processing circuitry that performs determination by using the motion information.

Furthermore, the configuration of the motion information processing device300according to the eleventh and twelfth embodiments is only an example, and the components thereof can be integrated or divided where appropriate. For example, the obtaining circuitry341and the generation circuitry342can be integrated.

Furthermore, the functions of the obtaining circuitry241and the determination circuitry243described in the seventh to tenth embodiments can be implemented by software. For example, the functions of the obtaining circuitry241and the determination circuitry243are achieved by making a computer execute medical information processing programs defining the processes described as being performed by the obtaining circuitry241and the determination circuitry243in the embodiment described above. The medical information processing programs are stored in a hard disk, a semiconductor memory, or the like, and read and executed by a processor such as a CPU and a MPU, for example. Furthermore, the medical information processing program can be recorded distributed on a computer-readable recording medium such as a CD-ROM (Compact Disc-Read Only Memory), a MO (Magnetic Optical disk), or a DVD (Digital Versatile Disc).

Furthermore, the functions of the obtaining circuitry341and the generation circuitry342described in the eleventh and twelfth embodiments can be implemented by software. For example, the functions of the obtaining circuitry341and the generation circuitry342are achieved by making a computer execute medical information processing programs defining the processes described as being performed by the obtaining circuitry341and the generation circuitry342in the embodiment described above. The medical information processing programs are stored in a hard disk, a semiconductor memory, or the like, and read and executed by a processor such as a CPU and a MPU, for example. Furthermore, the medical information processing program can be recorded distributed on a computer-readable recording medium such as a CD-ROM (Compact Disc-Read Only Memory), a MO (Magnetic Optical disk), or a DVD (Digital Versatile Disc).

Note that rehabilitation rule information, recommended status of assistance, and the like presented in the first to ninth embodiments described above may be those provided by various organization in addition to those provided by The Japanese Orthopaedic Association and the like. For example, various regulations and rules provided by associations as follows may be employed: “International Society of Orthopaedic Surgery and Traumatology (SICOT),” “American Academy of Orthopaedic Surgeons (AAOS),” “European Orthopaedic Research Society (EORS),” “International Society of Physical and Rehabilitation Medicine (ISPRM),” and “American Academy of Physical Medicine and Rehabilitation (AAPM&R).” While certain embodiments have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the inventions. Indeed, the novel embodiments described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions and changes in the form of the embodiments described herein may be made without departing from the spirit of the inventions. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the inventions.