Display method, information processing device, and computer-readable recording medium

A display method executed by a processor includes: acquiring an identification result relating to any one of a plurality of elements included in a series of exercise and a position of an athlete that performs the elements based on 3D sensing data that is acquired by sensing the series of exercise by the athlete of a sport scored by a judge and an element dictionary data in which characteristics of an element and a position of the sport scored by a judge are defined; identifying any one of an element and a position that is being displayed in a 3D model video based on the 3D sensing data corresponding to the series of exercise, based on the identification result; and identifying, for the any one of the element and the position being displayed, time information corresponding to a boundary with any one of a next element and a next position.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2018-222339, filed on Nov. 28, 2018, the entire contents of which are incorporated herein by reference.

FIELD

The embodiment discussed herein is related to a display method, and the like.

BACKGROUND

In competitive gymnastics, men compete in six events including the floor exercises, the pommel horse, the rings, the vault horse, the parallel bars, and the horizontal bar, and women compete in four events including the vault, the uneven parallel bars, the balance beam, and the floor. In the events except the vault, plural techniques (e.g., elements) are performed successively to compose a single performance (e.g., exercise) for both men and women.

An exercise score is calculated by totalizing a D (difficulty) score and an E (execution) score. For example, the D score is a score calculated based on whether an element is performed successfully or not. The E-score is a score calculated by the deduction method according the perfection of an element. Whether an element is performed successfully and the perfection of an element are judged by visual inspection of judges based on a rule book in which the code of points is provided.

For the D score, inquiries of the score is allowed, if the inquiries are made immediately after the publication of the score or, at the very latest, before the score of the following gymnast or group is displayed. Only the accredited coaches in the competition area are entitled to make an inquiry. All inquiries need to be examined by the Superior Jury. A recorded video of the gymnast is reviewed, and whether the score is appropriate is deliberated (for example, Japanese Laid-open Patent Publication Nos. 2014-45782, 2015-167676, and 2015-33476).

SUMMARY

According to an aspect of the embodiments, a display method executed by a processor includes: acquiring an identification result relating to any one of a plurality of elements included in a series of exercise and a position of an athlete that performs the elements based on 3D sensing data that is acquired by sensing the series of exercise by the athlete of a sport scored by a judge and an element dictionary data in which characteristics of an element and a position of the sport scored by a judge are defined; identifying any one of an element and a position that is being displayed in a 3D model video based on the 3D sensing data corresponding to the series of exercise, based on the identification result; identifying, for the any one of the element and the position being displayed, time information corresponding to a boundary with any one of a next element and a next position; and highlighting a predetermined time portion from time corresponding to the time information, when information relating to changes over time is displayed for an evaluation index corresponding to a selection selected from among a plurality of selections relating to a plurality of evaluation indexes.

DESCRIPTION OF EMBODIMENTS

When inquiries are made, the judges are to review a recorded video of the gymnast. For example, the elements include one that contains a position holding in which a predetermined posture is kept for a predetermined period (for example, 2 seconds) in a series of movement. The judges determine that the element is failed if the position holding is not kept for the predetermined time, and deduct scores. If the element subject to inquiry includes a position holding, rewinding, fast-forwarding, and the like are repeated to find a part corresponding to the position holding to be reproduced and reviewed. However, the gymnast is performing multiple elements in a series of exercise, and it makes it difficult to review the period of the concerned position holding, and to evaluate the element.

Accordingly, embodiments provide a display method, a display program, and an information processing device to support evaluation of a skill that is subject to be scored by a judge, and that includes a holding position.

Preferred embodiments will be explained with reference to accompanying drawings. Note that the embodiments are not intended to limit the present disclosure.

One performance (e.g., exercise) is composed of multiple techniques (e.g., elements) successively performed by an athlete. Moreover, one element is started from a starting position and ends in a finish position of the element. The finish position of the element is often a holding position in which a predetermined position is held. In the following description, the finish position and holding position in which a predetermined position is held for a predetermined period of time (for example, 2 seconds) are collectively referred to as “holding position”.

FIG. 1is a diagram for describing an example of a starting position and a holding position.FIG. 1illustrates an example in which an athlete5performs an element, “back uprise maltese (2 seconds)” as one example. The athlete5takes a starting position “back uprise” in a period Ta, and takes a holding position, “maltese (2 seconds)”. By combining this starting position, “back uprise” and the holding position, “maltese”, the element, “back uprise maltese (2 seconds)” is composed. As above, when a starting position and a holding position are determined, an element is determined. When the period Tb of a holding position is shorter than 2 seconds, a judge determines that the element, “back uprise maltese (2 seconds)” is failed.

InFIG. 1, an end time of the period Ta (start time of the period Tb) corresponds to a pause of element or position. Moreover, when a next element continues after the element illustrated inFIG. 1, an end time of the period Tb corresponds to a pause of element or position. In the following description, a pause of element of position is simply referred to as “pause of element”.

As described inFIG. 1, one element is determined by combination of a starting position and a holding position, and one exercise is composed by an athlete performing multiple elements as illustrated inFIG. 2.FIG. 2is a diagram illustrating an example of a pause of element included in an exercise. For example, a pause of element corresponds to a boundary with a next element or a next position. As illustrated inFIG. 2, after the element, “back uprise maltese (2 seconds)”, the athlete5performs an element, “back uprise shallow”, and then an element, “Azarian: layout kip and cross with straight arms (2 seconds)”. After “Azarian”, the athlete5performs “back uprise Japanese handstand (2 seconds)”, and then “back uprise handstand (2 seconds)”.

A pause of element of the element, “back uprise maltese (2 seconds)” corresponds to a start time Ta1 and an end time Tb1 of the holding position, “maltese (2 seconds)”. A pause of element of the element, “back uprise shallow (2 seconds)” corresponds to a start time Ta2 and an end time Tb2 of the holding position, “shallow (2 seconds)”.

A pause of element of the element, “Azarian: layout kip and cross with straight arms (2 seconds)” corresponds to a start time Ta3 and an end time Tb3 of the holding position, “cross”. A pause of element of the element, “back uprise Japanese handstand (2 seconds)” corresponds to a start time Ta4 and an end time Tb4 of the holding position, “Japanese handstand (2 seconds)”. A pause of element of the element, “back uprise shallow (2 seconds)” corresponds to a start time Ta5 and an end time Tb5 of the holding position, “handstand”.

Each element is identified by an element number that is a combination of a group and a branch number. For example, the element number of the element, “back uprise maltese (2 seconds)” is “G3 (group)-53 (branch number)”. The element number of the element, “back uprise shallow (2 seconds)” is “G3-70”. The element number of the element, “Azarian: layout kip and cross with straight arms (2 seconds)” is “G2-52”. The element number of the element, “back uprise Japanese handstand (2 seconds)” is “G3-94”. The element number of the element, “back uprise handstand (2 seconds)” is “G1-87”.

Next, a reference technique for achieving the present disclosure is described before escribing the present embodiment. This reference technique is not a related technique.

FIG. 3is a diagram illustrating an example of a display screen that is generated by an information processing device of the reference technique. As illustrated inFIG. 3, a display screen10includes regions10a,11,12,13,14.

The region10ais a region that includes buttons to control reproduction, stop, frame advance, fast-forward, rewind, and the like of a 3D model video. A judge presses respective buttons in the region10ato control reproduction, stop, frame advance, fast-forward, rewind, and the like.

The region11is a region in which a video based on video data is displayed. A video displayed in the region11is reproduced, stopped, frame-advanced, fast-forwarded, rewound, and the like according to a button pressed in the region10a.

The region12is a region in which a 3D model video is displayed from a virtual eyepoint that is specified in advance. The 3D model video displayed in the region12is reproduced, stopped, frame-advanced, fast-forwarded, rewound, and the like according to a button pressed in the region10a.

The region13is a region in which icons of evaluation index related to respective elements are displayed. The evaluation indexes are indexes to determine a score for an element, and the score decreases as the evaluation index diverges from an ideal value. The evaluation index includes an angle formed between one straight line passing through multiple joints of the athlete and another straight line passing through plural straight lines (joint angle), a distance between the one straight line and the other straight line, an angle formed between a reference line (reference plane) and the one straight line, and the like. As an example, the evaluation index includes a knee angle, an elbow angle, a shoulder angle, a distance between knees, a distance between one joint position of the athlete and a normal, and the like.

The judge selects either icon out of the icons displayed in the region13. For example, when an icon131-3corresponding to a shoulder angle is selected, the information processing device displays supplementary information121-3related to the shoulder angle.

The region14is a region in which a graph that indicates information relating to changes over time chronologically for the evaluation index of the selected icon is displayed. The judge refers to a graph in the region14, and thereby determines whether the holding position performed by the athlete satisfies requirements. The requirements of a holding position include holding a position for a predetermined period of time (2 seconds) by an athlete. For example, when the icon131-3corresponding to a shoulder angle is selected, an information processing device70chronologically displays changes of a shoulder angle over time in a graph. When a difference between a maximum value and a minimum value is small, it is determined that the athlete is holding a position (as for a part around the shoulder).

FIG. 4is a diagram illustrating an example of a graph of the reference technique. A vertical axis of a graph15is for a value relating to the evaluation index and, for example, corresponds to a shoulder angle. A horizontal axis of the graph15is for time. When evaluating a holding position of an athlete, the judge refers to a 3D model video in the region12illustrated inFIG. 3, and upon recognizing that the position of the athlete is in a holding position, presses a stop button in the region10a. For example, a timing (a time) when the judge presses the stop button is “current”.

When the stop button is pressed, the information processing device highlights a predetermined time portion15abetween “current” and a predetermined preceding time (for example, 2 seconds before) in the graph15. Moreover, the information processing device displays a marker16athat indicates a maximum value and a marker16bthat indicates a minimum value of the shoulder angle in the predetermined time portion15a. The judge refers to the predetermined time portion15aof the graph15displayed in the region14, and determines whether the position of the athlete is in a holding position for “2 seconds” or more that satisfies the requirement of the holding position. When a difference between the maximum value and the minimum value is large, it is determined that the athlete is not holding the position.

According to the reference technique, a predetermined time is highlighted in the graph based on a time currently displayed in the region11. Therefore, a stop time can be prior to an actual end time of a holding position, and the judge can erroneously determine that a holding period of time in the holding position is not enough. When it is assumed that a period of the holding position that is actually taken by the athlete is time Ts to Te, in the example inFIG. 4, “current” corresponds to a preceding time relative to an end time Ts of the holding position, and a difference between the marker16aand the marker16bis large in the predetermined time portion15a. In this case, the judge can make a judgment that the holding position made by the athlete does not satisfy the requirement.

On the other hand, when it is assumed that a period of the holding position that is actually taken by the athlete is time Ts to Te, because a difference between the maximum value and the minimum value in time Ts to Te is smaller than a threshold, it can be determined that the athlete is holding the position in time Ts to Te. Furthermore, the period of time Ts to Te is 2 seconds or longer, and the requirement for the holding position is satisfied. That is, the reference technique can fail to support judges to evaluate a holding position of an athlete.

Next, an example of a system according to the present embodiment is described.FIG. 5is a diagram illustrating an example of the system according to the present embodiment. As illustrated inFIG. 5, this system includes a 3D laser sensor50, a camera55, an element identification device80, and an information processing device100. A case in which the athlete5performs a gymnastic exercise in front of the 3D laser sensor50and the camera55is described as an example, but it is also applicable to a case in which the athlete5performs other kinds of events scored by judges, similarly.

For example, the other kinds of sporting events scored by judges include trampoline, diving, figure skating, karate forms, ballroom dance, snowboard, skateboard, aerial skiing, and surfing. Furthermore, it may be applied to classical ballet, ski jumping, airs and turns in mogul skiing, form check in baseball or basketball, and the like. Moreover, it may be applied to sports, such as Kendo, judo, wrestling, and sumo.

The 3D laser sensor50is a sensor that performs 3D sensing with respect the athlete5. The 3D laser sensor50outputs 3D sensing data that is a sensing result to an identification device60. In the following description, the 3D sensing data is simply referred to as “sensing data”. The sensing data includes plural frames. A frame includes a frame number and distance information about a distance to respective points on the athlete5. Numbers are assigned to the respective frames in ascending order.

The camera55is a device that acquires video data of the athlete5. The camera55outputs video data to the information processing device100. The video data includes plural frames corresponding to images of the athlete5, and a frame number is assigned to each frame. The frame number of the video data and the frame number of the sensing data are synchronized. Moreover, in the following description, a frame included in the sensing data is referred to as “sensing frame”, and the frame of the video data is referred to as “video frame” as appropriate.

The element identification device80generates 3D model data based on the sensing data that is sensed by the 3D laser sensor50. The element identification device80identifies an event and an element that is performed by the athlete5based on the 3D model data. The element identification device80outputs the 3D model data and identification result data to the information processing device100. The identification result data includes a frame number, an identified event and an element. This identification result data also includes a position identification result.

FIG. 6is a functional block diagram illustrating a configuration of the element identification device according to the present embodiment. As illustrated inFIG. 6, this element identification device80includes a communication unit81, a storage unit82, and a control unit83.

The communication unit81is a processing unit that performs data communication with the information processing device100. The communication unit81corresponds to a communication device.

The storage unit82includes a sensing DB82a, joint definition data82b, a joint position DB82c, a 3D model DB82d, element dictionary data82e, and an element-identification result DB82f. The storage unit82corresponds to a storage device, such as a semiconductor memory device including a random access memory (RAM), a read-only memory (ROM), and a flash memory, and a hard disk drive (HDD).

The sensing DB82ais a DB that stores sensing data acquired from the 3D laser sensor50.FIG. 7is a diagram illustrating an example of a data structure of the sensing DB according to the present embodiment. As illustrated inFIG. 7, this sensing DB82aassociates an exercise identification (ID), a frame number, and a frame with one another. The exercise ID is information uniquely identifying one exercise by the athlete5. The frame number is a number uniquely identifying each sensing frame that is associated with the same exercise ID. The sensing frame is a frame included in the sensing data that is sensed by the 3D laser sensor50.

The joint definition data82bis data that defines each joint position of the athlete5.FIG. 8is a diagram illustrating an example of a data structure of the joint definition data according to the present embodiment. As illustrated inFIG. 8, the joint definition data82bstores information in which respective joints that are identified by a publicly known skeleton model are numbered. For example, as illustrated inFIG. 8, A7 is assigned to a right shoulder joint (SHOULDER_RIGHT), A5 is assigned to a left elbow joint (ELBOW_LEFT), All is assigned to a left knee joint (KNEE_LEFT), and A14 is assigned to a right hip joint (HIP_RIGHT). In the present embodiment, there is a case in which an X coordinate of the right shoulder joint of A8 is expressed as X8, a Y coordinate thereof is expressed as Y8, and a Z coordinate thereof is expressed as Z8.

The joint position DB82cis position data of respective joints of the athlete5that is generated based on the sensing data of the 3D laser sensor50.FIG. 9is a diagram illustrating an example of a data structure of the joint position DB according to the present embodiment. As illustrated inFIG. 9, this joint position DB82cassociates an exercise ID, a frame number, and “X0, Y0, Z0, . . . , X17, Y17, Z17” with one another. Explanation about the exercise ID is the same as the explanation in the sensing DB82a.

InFIG. 9, the frame number is a number uniquely identifying each sensing frame that is associated with the same exercise ID. “X0, Y0, Z0, . . . , X17, Y17, Z17” is XYZ coordinates of the respective joints. For example, “X0, Y0, Z0” is a three-dimensional coordinates of a joint of A0 inFIG. 8.

FIG. 9illustrates chronological changes of the respective joints in the sensing data of an exercise ID “P101”, and it is indicated that the positions of the respective joints are “X0=100, Y0=20, Z0=0, . . . , X17=200, Y17=40, Z17=5” for a frame number “1”. For a frame number “2”, it is indicated that the positions of the respective joints are shifted to “X0=101, Y0=25, Z0=5, . . . , X17=202, Y17=39, Z17=15”.

The 3D model DB82dis a database that stores 3D model data of the athlete5generated based on the sensing data.FIG. 10is a diagram illustrating an example of a data structure of the 3D model DB according to the present embodiment. As illustrated inFIG. 10, the 3D model DB82dassociates an exercise ID, a frame number, skeleton data, and 3D data model with one another. Explanation about the exercise ID and the frame number is the same as the explanation in the sensing DB82a.

The skeleton data is data identifying a skeleton frame of the athlete5estimated by connecting the respective joint positions. The 3D model data is data of the 3D model data of the athlete5estimated based on the information acquired from the sensing data and the skeleton data.

The element dictionary data82eis dictionary data that is used when identifying an element included in an exercise by the athlete5.FIG. 11is a diagram illustrating an example of a data structure of the element dictionary data according to the present embodiment. As illustrated inFIG. 11, this element dictionary data82eassociates an event, an element number, an element name, and a requirement with one another. The event indicates an event of exercise. The element number is information uniquely identifying an element. The element name is a name of an element. The requirement indicates a requirement of an element to be regarded successful. The requirement includes respective joint positions, respective joint angles, movement of respective joint positions, and the like for an element to be regarded successful. When the element includes a holding position, the requirement includes a requirement for the holding position. Although illustration is omitted, information indicating whether a holding position is included may be set for each element number in the element dictionary data82e. Moreover, the requirement may include a requirement of a position. Furthermore, the element dictionary data82emay associate this requirement for a position and a position name with each other.

The element-identification result DB82fis a database that stores an identification result of an element.FIG. 12is a diagram illustrating an example of a data structure of the element-identification result DB according to the present embodiment. As illustrated inFIG. 12, the element-identification result DB82fassociates an event, an exercise ID, an element number, a first start time, a first end time, a flag, a second start time, a second end time, and an element name with one another.

Explanation about the event, the element number, and the element name is the same as explanation in the element dictionary data82e. The first start time indicates a start time of each element. The exercise ID is information uniquely identifying an element. The first end time indicates an end time of each element. The flag is information indicating whether a holding position is included in a concerned element. When a holding position is included in an element, the flag is to be “ON”. When a holding position is not included in an element, the flag is to be “OFF”.

The second start time indicates a start time of a holding position included in an element. The second end time indicates an end time of a holding position included in an element. In the example inFIG. 12, the sequence of elements performed by a performer is “G3-53, G2-52, G1-87, G1-51, G1-52, G3-16, G1-49, G3-69, G1-81, G1-26, G4-41”.

Returning to description ofFIG. 6, the control unit83includes an acquiring unit83a, a model generating unit83b, an element identifying unit83c, and a notifying unit83d. The control unit83can be implemented by a central processing unit (CPU), a micro processing unit (MPU), or the like. Moreover, the control unit83can be implemented also by wired logic, such as an application specific integrated circuit (ASIC) and a field programmable gate array (FPGA).

The acquiring unit83ais a processing unit that acquires sensing data from the 3D laser sensor50. The acquiring unit83astores the acquired sensing data in the sensing DB82a.

The model generating unit83bis a processing unit that generates a 3D model data corresponding to respective frame numbers of respective exercise IDs based on the sensing DB82a. In the following, an example of processing by the model generating unit83bis described. The model generating unit83bcompares a sensing frame in the sensing DB82awith positional relationship of respective joints defined by the joint definition data82b, to identify a type of respective joints included in the sensing frame and three-dimensional coordinates of the joints. The model generating unit83brepeats the above processing for respective frame numbers of each exercise ID to generate the joint position DB82c.

The model generating unit83bgenerates skeleton data by connecting the three-dimensional coordinates of the respective joints stored in the joint position DB82cbased on the connection relationship defined by the joint definition data82b, to generate the skeleton data. Moreover, the model generating unit83bgenerates 3D model data by applying the estimated skeleton data to a skeleton model matched with a body frame of the athlete5. The model generating unit83brepeats the above processing for respective frame numbers of each exercise ID, to generate the 3D model DB82d.

The element identifying unit83cchecks the respective skeleton data stored in the 3D model DB82dsequentially following through the frame numbers, and compares the respective skeleton data with the requirement stored in the element dictionary data82e, to determine whether a requirement is hit. When a requirement is hit, the element identifying unit83cidentifies an event, an element number and an element name corresponding to the hit requirement. Moreover, the element identifying unit83cconverts a starting frame number of a series of frames for which the requirement is hit into the first start time based on predetermined frames per second (FPS), and converts an ending frame number of the series of frame numbers into the first end time. The element identifying unit83cstores the event, the exercise ID, the element number, the first start time, the first end time, and the element name in the element-identification result DB82fin an associated manner. Furthermore, the element identifying unit83cmay further identify a requested position by comparing the respective skeleton data and requirements stored in the element dictionary data82e.

The element identifying unit83cdetermines whether the identified element includes a holding position. When the element does not include a holding position, the element identifying unit83csets the flag in the element-identification result DB82fto “OFF”. On the other hand, when the element includes a holding position, the element identifying unit83csets the flag in the element-identification result DB82fto “ON”. The element-identification result DB82fstores a start time of the holding position as the second start time, and an end time of the holding position as the second end time in the element-identification result DB82f. Note that the “time” herein is an example of time information. For example, the time information may be information including a date and a time, or may be one indicating elapsed time from a start of an exercise.

The element identifying unit83crepeats the above processing each time a requirement is hit, and thereby generate the element-identification result DB82f.

The notifying unit83dis a processing unit that transmits information of the 3D model DB82dand information of the element-identification result DB82fto the information processing device100.

Returning back to description ofFIG. 5, the information processing device100is a processing unit that generates information of a display screen in which a video and the 3D model video are displayed, to cause a display unit (not illustrated) to display it.FIG. 13is a diagram illustrating an example of the display screen that is generated by the information processing device according to the present embodiment. As illustrated inFIG. 13, this display screen20includes regions20a,21,22ato22d, and23.

The region20ais a region that includes buttons to control reproduction, stop, frame advance, fast-forward, rewind, and the like of a 3D model video. A judge presses respective buttons in the region20ato control reproduction, stop, frame advance, fast-forward, rewind, and the like of the video and the 3D model video.

The region21is a region in which a video based on video data is displayed. A video displayed in the region21is reproduced, stopped, frame-advanced, fast-forwarded, rewound, and the like according to a button pressed in the region20a.

The region22is a region in which a 3D model video is displayed from a virtual eyepoint that is specified in advance. The 3D model video displayed in the region22is reproduced, stopped, frame-advanced, fast-forwarded, rewound, and the like according to a button pressed in the region20a.

The region23is a region in which icons of evaluation index related to respective elements are displayed. The evaluation indexes are indexes to determine a score for an element, and the score decreases as the evaluation index diverges from an ideal value. The evaluation index includes an angle formed between one straight line passing through multiple joints of the athlete5and another straight line passing through plural straight lines (joint angle), a distance between the one straight line and the other straight line, an angle formed between a reference line (reference plane) and the one straight line, and the like. As an example, the evaluation index includes a knee angle, an elbow angle, a shoulder angle, a distance between knees, a distance between one joint position of the athlete and a normal, and the like.

The judge selects either icon out of the icons displayed in the region23. For example, when an icon231-3corresponding to a shoulder angle is selected, the information processing device displays supplementary information221-3related to the shoulder angle.

The region24is a region in which a graph that indicates information relating to changes over time chronologically is displayed, for the evaluation index of the selected icon. The judge refers to a graph in the region24, and thereby determines whether the holding position performed by the athlete satisfies requirements. The requirements of a holding position include holding a position for a predetermined period of time (2 seconds) by the athlete5. For example, when the icon231-3corresponding to a shoulder angle is selected, the information processing device100chronologically displays changes of a shoulder angle over time in a graph. When a difference between a maximum value and a minimum value is small, it is determined that the athlete5is holding a position (as for a part around the shoulder).

FIG. 14is a diagram illustrating an example of a graph according to the present embodiment. A vertical axis of a graph25is for a value relating to the evaluation index and, for example, corresponds to a shoulder angle. A horizontal axis of the graph25is an axis corresponding to time. When evaluating a holding position of an athlete5, the judge refers to a 3D model video in the region22illustrated inFIG. 14, and upon recognizing that the position of the athlete is in a holding position, presses a stop button in the region20a. For example, a timing (a time) when the judge presses the stop button, or a time when it is reproduced is “current”.

When the stop button is pressed, the information processing device100identifies an element that is performed by the athlete5at “current”. Regardless of being paused, the information processing device100identifies an element that is performed by the athlete5at the time of reproduction “current”. The information processing device100identifies a second end time “Te” of the holding position included in the identified element, and highlights a predetermined time portion25abetween the second end time and a predetermined preceding time (for example, 2 seconds before).

The information processing device100displays a marker26athat indicates a maximum value and a marker26bthat indicates a minimum value of the shoulder angle in the predetermined time portion25a. The judge refers to the predetermined time portion25aof the graph25displayed in the region24, and determines whether the position of the athlete5is held for “2 seconds” or more that satisfies the requirement of the holding position. When a difference between the maximum value and the minimum value is large, it is determined that the athlete5is not holding the position.

AS illustrated inFIG. 14, the information processing device100according to the present embodiment highlights the predetermined time portion25abetween a second holding time of the holding position included in the element and a predetermined preceding time (for example, 2 seconds before). Unlike the reference technique described inFIG. 4, the information processing device100highlights the predetermined time portion25afrom the second holding time and a predetermined preceding time as long as a holding position is included in an element that is performed by the athlete5at a time when a judge presses the stop button. Therefore, the judge can determine whether the requirement of the holding position is satisfied appropriately. Moreover, compared with the display method of the graph in the reference technique, an erroneous judgment by a judge can be well suppressed and, therefore, the accuracy in determining a score is improved.

Next, a configuration of the information processing device100according to the present embodiment is described.FIG. 15illustrates a configuration of the information processing device according to the present embodiment. As illustrated inFIG. 15, the information processing device100includes a communication unit110, an input unit120, a display unit130, a storage unit140, and a control unit150.

The communication unit110is a processing unit that performs data communication with the camera55and the element identification device80. For example, the communication unit110receives information of the 3D model DB82dand information of the element-identification result DB82ffrom the element identification device80, and outputs the received information of the 3D model DB82dand information of the element-identification result DB82fto the control unit150. Moreover, the communication unit110receives video data from the camera55, and outputs the received video data to the control unit150. The control unit150described later communicates data with the camera55and the element identification device80through the communication unit110.

The input unit120is an input device to input various kinds of information to the information processing device100. For example, the input unit120corresponds to a keyboard, a mouse, a touch panel, and the like. A judge operates the input unit120and selects a button in the region20aon the display screen20illustrated inFIG. 13and the like, and thereby controls reproduction, stop, frame advance, fast-forward, rewind, and the like of a video and a 3D model video. A judge presses respective buttons in the region10ato control reproduction, stop, frame advance, fast-forward, rewind, and the like. Moreover, by operating the input unit120, the judge selects an icon included in the region23of the display screen20illustrated inFIG. 13.

The display unit130is a display device that displays various kinds of information output from the control unit150. For example, the display unit130displays information of the display screen20illustrated inFIG. 13and the like. Furthermore, a judge presses an icon included in the region23, and thereby switches graphs of evaluation indexes to be display in the region24.

The storage unit140includes a video DB140a, a 3D model DB92d, an element-identification result DB92f, and an evaluation index table140d. The storage unit140corresponds to a semiconductor memory device, such as a RAM, a ROM, and a flash memory, and a storage device, such as an HDD.

The video DB140ais a database that stores video frames.FIG. 16is a diagram illustrating an example of a data structure of the video DB according to the present embodiment. As illustrated inFIG. 16, this video DB140aassociates an exercise ID, a frame number, and a video frame with one another. The exercise ID is information uniquely identifying one exercise by the athlete5. The frame number is a number uniquely identifying respective video frames corresponding to the same exercise ID. The video frame is a frame included video data that is acquired by the camera55. The frame number of the sensing frame illustrated inFIG. 7and the frame number of the video frame are synchronized.

The 3D model DB92dis a database that stores information of a 3D model of the athlete5generated by the element identification device80. A data structure of the 3D model DB92dis the same as the data structure of the 3D model DB82ddescribed inFIG. 10.

The element-identification result DB92fis a database that stores information an identification result of respective elements included in a series of exercise generated by the element identification device80. A data structure of the element-identification result DB92fis the same as the data structure of the element-identification result DB82fdescribed inFIG. 12.

The evaluation index table140bis a table that holds various kinds of information relating to evaluation indexes of the icons.FIG. 17is a diagram illustrating an example of a data structure of the evaluation index table according to the present embodiment. As illustrated inFIG. 17, this evaluation index table140bassociates an icon identification number, an icon image, an evaluation index, and a definition with one another.

The icon identification number is information uniquely identifying an icon. The icon image is an image of each icon illustrated in the region23inFIG. 13. The evaluation index is an index to determine a score of an element. The definition indicates definition to acquire an evaluation index of the athlete5from the skeleton data. For example, an evaluation index is defined by a straight line connecting one joint and another joint among plural joints included in the skeleton data, an angle formed by two straight lines (joint angle), and the like.

Returning back to description ofFIG. 15, the control unit150includes an acquiring unit150a, a display control unit150b, and an identifying unit150c. The control unit150can be implemented by hardware logic, such as ASIC and FPGA.

The acquiring unit150aacquires video data from the camera55, and stores the acquired video data in the video DB140a. The acquiring unit150aacquires information of the 3D model DB82dand information of the element-identification result DB82ffrom the element identification device80, and stores the acquired 3D model DB82din the 3D model DB92d. Moreover, the acquiring unit150astores the information of the element-identification result DB82fin the element-identification result DB92f.

The display control unit150bis a processing unit that generates information of the display screen20illustrated inFIG. 13, to display on the display unit130. The display control unit150bsequentially reads video frames from the video DB140a, and reproduces the video in the region21of the display screen20.

The display control unit150bsequentially reads 3D model data from the 3D model DB92d, and reproduces the 3D model video in the region22of the display screen20. The respective 3D model videos displayed in the region22are videos of a 3D model data shot from a virtual eyepoint specified in advance.

The display control unit150bperforms reproduction, synchronizing a time (frame number) of a video displayed in the region21and a time (frame number) of each 3D model video displayed in the region22. When a button displayed in the region20ais pressed by a judge, the display control unit150bperforms reproduction, stop, frame advance, fast-forward, rewind, and the like of the video in the region21and the 3D model video in the region22according to the pressed button.

The display control unit150boutputs information of a reproduction time of the 3D model video that is currently displayed in the region22to the identifying unit150c. In the following description, the reproduction time of the 3D model video currently displayed in the region22is simply referred to as “reproduction time”. This reproduction time corresponds to “current” indicated in the graph inFIG. 14. Note that the present embodiment is described by using the reproduction time, but it may be a display time of the regions21,22.

The display control unit150brefers to the evaluation index table140b, and acquires icon images of icons, and displays the respective icons in the region23of the display screen20. When either icon is selected out of the icons displayed in the region23of the display screen20, the display control unit150bgenerates a graph of an evaluation index corresponding to the selected icon, and displays the graph in the region24.

An example of processing of generating a graph by the display control unit150bis described. The display control unit150bcompares an icon identification number of the icon selected by the judge with the evaluation index table140b, and identifies an evaluation index relating to the selected icon. The display control unit150bcompares definition of the identified evaluation index with the skeleton data of the 3D model DB92d, to calculate a value of changes over time relating to the evaluation index. For example, the display control unit150bidentifies a time Ths predetermined time before the reproduction time and a time The predetermined time after the reproduction time. The display control unit150bcalculates a value of changes over time relating to the evaluation index of the times Ths to The based on the skeleton data of the time Ths to The, to generate a graph.

The display control unit150bacquires identification information from the identifying unit150cdescribed later. The identification information includes a second start time and a second end time of a holding position. The display control unit150bhighlights a predetermined time portion between the second end time and a predetermine preceding time in the graph generated by the above processing. In the graph25illustrated inFIG. 14, the predetermined time portion25ais highlighted. Moreover, the display control unit150bscans values of changes over time of the evaluation index included in the predetermined time portion, identifies a maximum value and a minimum value, and sets markers at portions corresponding to the maximum value and the minimum value. The display control unit150bmay display values of the maximum value and the minimum value included in the predetermined time portion on the display screen.

When either icon is selected by a judge from among the icons (icon images) displayed in the region23of the display screen20, the display control unit150bdisplays supplementary information corresponding to icon identification information of the selected icon, superimposing on the 3D model video in the region22. In the example illustrated inFIG. 13, supplementary information221-3relating to a shoulder angle is displayed. The supplementary information corresponding to the icon identification information is assumed to be set in advance. Moreover, the display control unit150bmay display a value corresponding to the evaluation index calculated when the graph describe above is generated, along with the supplementary information of the 3D model video.

The identifying unit150cis a processing unit that generates identification information based on information about a reproduction time from the display control unit150b, and that outputs the identification information to the display control unit150b. For example, the identifying unit150ccompares the reproduction time with the element-identification result DB92f, to identify an element number associated with the reproduction time. When the flag associated with the element number is “OFF”, the identifying unit150cskips the processing without generating the identification information.

When the flag associated with the element number is “ON”, the identifying unit150cacquires the second start time and the second end time associated with the element number, and generates identification information that includes the acquired second start time and second end time. The identifying unit150coutputs the generated identification information to the display control unit150b. The identifying unit150crepeats the above processing each time information about a reproduction time is received.

Next, an example of a procedure of processing performed by the information processing device100according to the present embodiment is described.FIG. 18is a flowchart illustrating a procedure of processing performed by the information processing device according to the present embodiment. As illustrated inFIG. 18, the acquiring unit150aof the information processing device100acquires video data, information of the 3D model DB82d, and information of the element-identification result DB82f, to store in the storage unit140(step S101).

The display control unit150bof the information processing device100starts reproduction of the video data and the 3D model video (step S102). The display control unit150bdetermines whether a change instruction of a reproduction start time is received (step S103). When the change instruction of the reproduction start time is not received (step S103: NO), the display control unit150bshifts to step S105.

On the other hand, when the change instruction of the reproduction time is received (step S103: YES), the display control unit150bchanges the reproduction time and continues reproduction (step S104), and shifts to step S105.

The identifying unit150cof the information processing device100identifies an element number corresponding to the reproduction time, synchronizing with reproduction (step S105). When the flag associated with the element number is not ON in the element-identification result DB92f(step S106: NO), the identifying unit150cshifts to step S109.

The identifying unit150cidentifies the second end time (and the second start time) of a holding position of the element corresponding to the element number (step S107). The display control unit150bdisplays information of changes of the evaluation index over time corresponding to the selected icon, and chronologically displays it (in graph), and highlights a predetermined time portion between the second end time and a predetermined preceding time (step S108).

The display control unit150bdisplays supplementary information corresponding to the icon, superimposing on the 3D model video (step S109). When the processing is to be continued (step S110: YES), the display control unit150bshifts to step S103. On the other hand, when the processing is not to be continued (step S110: NO), the display control unit150bends the processing.

Next, an effect of the information processing device100according to the present embodiment is described. The information processing device100identifies a time corresponding to a pause of elements or positions that corresponds to a reproduction time. The information processing device100highlights a predetermined time portion from a time to be a pause of elements or positions when information relating to changes over time is chronologically displayed for an evaluation index corresponding to a selection selected from among multiple selections relating to evaluation indexes. Thus, it becomes possible to support evaluation of an element of the athlete5.

For example, as described inFIG. 14, the information processing device100highlights the predetermined time portion25abetween the second start holding time of the holding position included in the element and a predetermined preceding time (for example, 2 second before). Unlike the reference technique described inFIG. 4, the information processing device100highlights the predetermined time portion25abetween the second end time and a predetermined preceding time as long as a holding position is included in an element of the athlete5at a time when a judge presses the stop button. Therefore, the judge can determine whether the requirement of the holding position is satisfied appropriately. Moreover, compared with the display method of a graph in the reference technique, an erroneous judgment by a judge can be well suppressed and, therefore, the accuracy in determining a score is improved.

When highlighting the predetermined time portion25abetween the second holding time and a predetermined preceding time, the information processing device100identifies a maximum value and a minimum value of a value relating to an evaluation index in the predetermined time portion25aand displays markers. Thus, a judge can determine whether the athlete5is holding a position easily by comparing the maximum value and the minimum value.

The information processing device100determines whether an element corresponding to the reproduction time includes a holding position, synchronizing with the reproduction time of the video or the 3D model video. When the element corresponding to the reproduction time includes a holding position, the information processing device100performs the processing of highlighting the predetermined time portion25abetween eh second holding time and a predetermined preceding time. Therefore, it is possible to reduce a processing load of the information processing device100.

The processing of the information processing device100described above is one example, and the information processing device100may perform another processing. For example, the display control unit150bof the information processing device100may highlight a predetermined time portion between the second start time and a predetermined preceding time as long as an element of the athlete5includes a holding position at a time when a judge presses the stop button.

FIG. 19is a diagram for describing another processing performed by the information processing device. A vertical axis in graph35is for a value relating to an evaluation index, and corresponds to, for example, a shoulder angle. A horizontal axis in the graph35is an axis corresponding to time. For example, a timing (a time) when a judge presses the stop button, or a time of reproduction is “current”. The display control unit150bidentifies the second start time “Ts” of a holding position included in an identified element, and highlights a predetermined time portion35abetween the second start time and a predetermined preceding time (for example, 2 seconds before). Moreover, the information processing device100displays a marker36athat indicates a maximum value of the shoulder angle and a marker36bthat indicates a minimum value thereof in the predetermined time portion35a.

At a time when the judge presses the stop button, only if an element of the athlete5includes a holding position, the predetermined time portion35abetween the second start time and a predetermined preceding time is highlighted. Therefore, the judge can determine whether the requirement of the holding position is satisfied appropriately.

FIG. 20is a diagram illustrating an example of another display screen displayed by the information processing device. As illustrated inFIG. 20, this display screen includes regions30a,31,32ato32d, and33.

The region30ais a region that includes buttons to control reproduction, stop, frame advance, fast-forward, rewind, and the like of a video or a 3D model video. The judge presses respective buttons in the region30ato control reproduction, stop, frame advance, fast-forward, rewind, and the like.

The regions32ato32dare regions in which a 3D model video is respectively displayed from a virtual eyepoint that is specified in advance. The 3D model video displayed in the regions32ato32dis reproduced, stopped, frame-advanced, fast-forwarded, rewound, and the like according to a button pressed in the region30a.

The region33is a region in which icons of evaluation index related to respective elements are displayed. The judge selects either icon from among the icons displayed in the region33.

The region34is a region in which a graph that indicates information relating to changes over time chronologically is displayed, for the evaluation index of the selected icon. The judge refers to a graph in the region34, and thereby determines whether the holding position performed by the athlete satisfies requirements. For example, when a holding position is included in an element that is being performed by the athlete5at the production time, the information processing device100may highlight a region between the start time Ts and the end time Te of the holding position.

The case in which the element identification device80and the information processing device100are implemented in separate devices has been explained in the system illustrated inFIG. 5, but it is not limited thereto. The information processing device100may include a function of the element identification device80. For example, the information processing device100may have a function of the control unit83of the element identification device80, and may generates the 3D model DB82dand the element-identification result DB82fbased on the information stored in the storage unit82.

Next, an example of a hardware configuration of a computer that implements similar functions as the element identification device80and the information processing device100is described.FIG. 21is a diagram illustrating an example of a hardware configuration of a computer that implements functions similar to those of the element identification device according to the present embodiment.

As illustrated inFIG. 21, a computer400includes a CPU401that performs various kinds of arithmetic processing, an input device402that receives an input of data from a user, and a display403. Moreover, the computer400includes a reader device404that reads a program and the like from a storage medium, and an interface device405that communicates data with the 3D laser sensor50and the like through a wired or wireless network. The computer400includes a RAM406that temporarily stores various kinds of information, and a hard disk device407. The respective devices401to407are connected to a bus408.

The hard disk device407includes an acquiring program407a, a model generating program407b, an element identifying program407c, a notifying program407d. The CPU401reads the acquiring program407a, the model generating program407b, the element identifying program407c, and the notifying program407dand develops them in the RAM406.

The acquiring program407afunctions as an acquiring process406a. The model generating program407bfunctions as a model generating process406b. The element identifying program407cfunctions as an element identifying process406c. The notifying program407dfunctions as a notifying process406d.

Processing of the acquiring process406acorresponds to the processing of the acquiring unit83a. Processing of the model generating process406bcorresponds to the processing of the model generating unit83b. Processing of the element identifying process406ccorresponds to the processing of the element identifying unit83c. Processing of the notifying process406dcorresponds to the processing of the notifying unit83d.

The respective programs407ato407dare not necessarily requested to be stored in the hard disk device407from the beginning. For example, the respective programs are stored in a “portable physical medium” to be inserted in the computer400, such as a flexible disk (FD), a CD-ROM, a digital versatile disk (DVD), a magneto-optical disk, and an IC card. The respective programs407ato407dmay be read therefrom and executed by the computer400.

FIG. 22is a diagram illustrating an example of a hardware configuration of a computer that implements functions similar to those of the information processing device according to the present embodiment.

As illustrated inFIG. 22, a computer500includes a CPU501that performs various kinds of arithmetic processing, an input device502that receives an input of data from a user, and a display503. Moreover, the computer500includes a reader device504that reads a program and the like from a storage medium, and an interface device505that communicates data with the camera55, the element identification device80, and the like through a wired or wireless network. The computer500includes a RAM506that temporarily stores various kinds of information, and a hard disk device507. The respective devices501to507are connected to a bus508.

The hard disk device507includes an acquiring program507a, a display control program507b, and an identifying program507c. The CPU501reads the acquiring program507a, the display control program507b, and the identifying program507cand develops them in the RAM506.

The acquiring program507afunctions as an acquiring process506a. The display control program507bfunctions as a display control process506b. The identifying program507cfunctions as an identifying process506c.

Processing of the acquiring process506acorresponds to the processing of the acquiring unit150a. Processing of the display control process506bcorresponds to the processing of the display control unit150b. Processing of the identifying process506ccorresponds to the processing of the identifying unit150c.

The respective programs507ato507care not necessarily requested to be stored in the hard disk device507from the beginning. For example, the respective programs are stored in a “portable physical medium” to be inserted in the computer500, such as an FD, a CD-ROM, a DVD, a magneto-optical disk, and an IC card. The respective programs507ato507cmay be read therefrom and executed by the computer500.

It is possible to support evaluation of an element that is subject to be scored by a judge, and that includes a holding position.