Patent Description:
Conventionally, in sports or the like where a player performs a sequence of elements and a judge gives a score, the success or the degree of perfection of a highly difficult element largely affects a high score or a high rank. For example, six events are formed for men, four events for women, and in events other than vault for men and women, one performance is formed of more than ten elements performed in succession. In a scoring rule, criteria are defined for the type, the group, the number, and the execution of elements, and a criterion is set for each element to determine a score. A judge gives a score to a performance in a short time after the player finishes the performance.

Nowadays, there is a known technology of acquiring information about the attitude, the movement, or the like, of a person by using 3D sensing to assist a sport player, and the like. For example, it is used to check the degree of perfection of an element as it acquires the movement and the like of a gymnast when he performs the element during practice.

Patent Literature <NUM>: <CIT>
<CIT> describes methods and systems for utilizing motion capture techniques, for example, video based motion capture techniques, for capturing and modeling the captured 3D movement of an athlete through a defined space. The model is then compared with an intended motion pattern in order to identify deviations and/or form breaks that, in turn, may be used in combination with a scoring algorithm to quantify the athlete's execution of the intended motion pattern to produce an objective score. The methods and systems are said to be anticipated to be particularly useful for training and judging in those sports that have struggled with the vagaries introduced by the subjective nature of human scoring.

It is, however, difficult for even an expert such as a judge to accurately check the joint angle of a player and give a score to the performance in accordance with the scoring rule that is defined for each event or element, and therefore scores vary depending on individual judges.

For example, the body of a gymnast makes a complicated movement in three dimensions, and each gymnast has a different physical size and movement for an element. Therefore, it is not easy for even a judge who is licensed as a judge to give a score after determining, in a short time, that a certain element has been accurately performed in accordance with a scoring rule. For this reason, scores for individual elements sometimes vary depending on a judge and, in some cases, a player expresses an opposition (inquiry) to the judge in a predetermined method so as to make a detailed rejudgement on the score.

According to an aspect, the object is to provide a performance display program, a performance display method, and a performance display apparatus capable of providing information for conducting scoring in the uniform manner in accordance with a scoring rule.

According to an aspect of the present invention, there is provided a performance display program as set out in Claim <NUM>.

According to another aspect of the present invention, there is provided a performance display method as set out in Claim <NUM>.

According to an aspect of the present invention, there is provided a performance display apparatus as set out in Claim <NUM>. Advantageous Effects of Invention.

According to an embodiment, it is possible to provide information for conducting scoring in the uniform manner in accordance with a scoring rule.

There is described herein a non-transitory computer-readable recording medium that stores therein a performance display program that causes a computer to execute a process. The process includes determining a position of a joint of a performer from a sensing result of a sensor; calculating an angle formed by a joint used for scoring an element of an athletic event by using calculation information regarding a calculation formula for calculating the angle and the determined position of the joint of the performer; and displaying a performance image of the performer acquired from a camera and a scoring image having the angle displayed on a 3D model image of the performer generated from the sensing result so as to be compared with each other.

With reference to the drawings, an embodiment of a performance display program, a performance display method, and a performance display apparatus according to the present invention is described below in detail. Furthermore, the present invention is not limited to the embodiment. Moreover, each embodiment may be combined as appropriate as long as the consistency is ensured.

<FIG> is a diagram that illustrates a video reproduction apparatus <NUM> according to a first embodiment. The video reproduction apparatus <NUM> illustrated in <FIG> is an example of a computer, such as a server, a tablet terminal, or a smartphone, and it is an example of a performance display apparatus. The video reproduction apparatus <NUM> uses a 3D (Three Dimensions) laser sensor to determine the movement or the position of a joint of a player in three dimensions and in chronological order and stores it as joint position data in a joint table in advance. Furthermore, the video reproduction apparatus <NUM> obtains, as a rule, the scoring rule for each athletic event and registers it in a database so as to generate, as indexes, the type of element, the joint and the angle used for scoring an element, a point deduction due to angle deviation, or the like.

The above video reproduction apparatus <NUM> is used for the judge to view the video, check the score, and make a rejudgement in the case of an opposition to the score announced after the performance ends. Specifically, the video reproduction apparatus <NUM> stores calculation information regarding a calculation formula for calculating the angle formed by a joint used for scoring an element in an athletic event. Then, the video reproduction apparatus <NUM> determines the position of a joint of the performer from the performance image that is a specific frame designated in the performance video of the performer captured by a video camera. Then, the video reproduction apparatus <NUM> calculates the angle formed by a joint by using the determined position of the joint of the performer and the calculation information and displays the performance image designated in the performance video and the scoring image having the angle displayed on the performance image so as to be compared with each other.

Specifically, the video reproduction apparatus <NUM> not only simply visualizes the movement and the position of each joint during the performance while determining the attitude and the movement of the performer due to 3D sensing with the 3D laser sensor but also presents, to a judge, the information that aids in making a determination for scoring the execution of the element in accordance with a scoring rule. As a result, the video reproduction apparatus <NUM> is capable of providing the information for conducting scoring in the uniform manner in accordance with the scoring rule.

<FIG> is a functional block diagram that illustrates a functional configuration of the video reproduction apparatus <NUM> according to the first embodiment. As illustrated in <FIG>, the video reproduction apparatus <NUM> includes a communication unit <NUM>, a storage unit <NUM>, and a control unit <NUM>.

The communication unit <NUM> is a communication interface, or the like, which controls communications with other devices. For example, the communication unit <NUM> receives a performance video captured by a video camera. Furthermore, the communication unit <NUM> receives joint position data that is a result of 3D sensing with a 3D laser sensor. Moreover, the communication unit <NUM> receives a scoring rule from the administrator's terminal, or the like.

The storage unit <NUM> is an example of a storage device that stores a program and data, and it is, for example, a memory or a hard disk. The storage unit <NUM> stores a joint definition data DB <NUM>, a joint position data DB <NUM>, a video data DB <NUM>, a scoring rule DB <NUM>, and a calculation formula DB <NUM>. Furthermore, the calculation formula DB <NUM> is an example of a scoring storage unit.

The joint definition data DB <NUM> is a database that stores definition information for determining each joint in a skeleton model. The definition information stored here may be measured for each performer during 3D sensing with a 3D laser sensor or may be defined by using a skeleton model of the typical body.

<FIG> is a diagram that illustrates an example of the information stored in the joint definition data DB <NUM>. As illustrated in <FIG>, the joint definition data DB <NUM> stores the information on numbering of each joint determined by using a known skeleton model. For example, as illustrated in <FIG>, number <NUM> is assigned to the right shoulder joint (SHOULDER_RIGHT), number <NUM> is assigned to the left elbow joint (ELBOW_LEFT), number <NUM> is assigned to the left knee joint (KNEE_LEFT), and number <NUM> is assigned to the right hip joint (HIP_RIGHT). Here, according to the embodiment, the X-coordinate of the number-<NUM> right shoulder joint is sometimes described as X7, the Y-coordinate as Y7, and the Z coordinate as Z7. Furthermore, numbers in a dotted line are of a joint, or the like, which are not used for scoring although they are determined from the skeleton model.

The joint position data DB <NUM> is a database that stores the position data on each joint acquired from video data on the performer and temporal changes in the joint position data. <FIG> is a diagram that illustrates an example of the information stored in the joint position data DB <NUM>. As illustrated in <FIG>, the joint position data DB <NUM> stores "record ID, time, X0, Y0, Z0,. , X17, Y17, Z17" in a related manner. "Record ID" stored here is the information for identifying video data, and "time" is the time when the positional information on a joint is determined. "X0, Y0, Z0,. , X17, Y17, Z17" are the XYZ coordinates of each joint, and "X0, Y0, Z0" are the coordinates of the joint with number <NUM> illustrated in <FIG>.

<FIG> illustrates temporal changes in each joint in the video data with number <NUM>, and it indicates that, at the time "<NUM>:<NUM>:<NUM>", the positions of the respective joints are "X0=<NUM>, Y0=<NUM>, Z0=<NUM>,. , X17=<NUM>, Y17=<NUM>, Z17=<NUM>". Furthermore, it indicates that, at the time "<NUM>:<NUM>:<NUM>", the positions of the respective joints have moved to "X0=<NUM>, Y0=<NUM>, Z0=<NUM>,. , X17=<NUM>, Y17=<NUM>, Z17=<NUM>".

The video data DB <NUM> is a database that stores video data on a performer. <FIG> is a diagram that illustrates an example of the information stored in the video data DB <NUM>. As illustrated in <FIG>, the video data DB <NUM> stores "record ID, event, player ID, time, video file" in a related manner. "Record ID" stored here is the information for identifying video data, "event" indicates the event of a performer captured by the video camera, "player ID" is the information for identifying a performer, "time" is the information indicating a performance time, and "video file" is the information for determining video data. The case of <FIG> indicates that the video file "001_pommel horse_1122. mp4" with the record ID "<NUM>" is the video data on "pommel horse" performed by the player with the player ID "<NUM>" and it is the performance data in "<NUM> minutes <NUM> seconds".

The scoring rule DB <NUM> is a database that stores a scoring rule for each event. <FIG> is a diagram that illustrates an example of the information stored in the scoring rule DB <NUM>. As illustrated in <FIG>, the scoring rule DB <NUM> stores "scoring rule ID, event, action, scoring target, threshold (minor fault, intermediate fault, major fault), point deduction (minor fault, intermediate fault, major fault)" in a related manner.

"Scoring rule ID" stored here is the information for identifying a scoring rule. "Event" is the information for determining the target event to be scored. "Action" is the information for determining the target action to be scored. "Scoring target" is the information for determining the element name, the score item, or the like, for which the angle, which is the target to be scored, is calculated. "Threshold (minor fault, intermediate fault, major fault)" is the information indicating a threshold for the angle that is the target for point deduction. "Point deduction (minor fault, intermediate fault, major fault)" is the information indicating the details of point deduction.

In the case of <FIG>, the scoring rule ID "<NUM>" indicates that "angle deviation" in "final phase" of "longitudinal circling" of "pommel horse" is the scoring target. This example indicates that a judgement is made such that point deduction is "<NUM>" in the range of the angle from <NUM>° (<NUM> degrees) to <NUM>°, point deduction is "<NUM>" in the range of the angle from <NUM>° to <NUM>°, and point deduction is "<NUM>" when the angle is more than <NUM>°.

The calculation formula DB <NUM> is a database that stores a calculation formula for the angle between joints, which is the target to be scored, i.e., the angle formed by joints. <FIG> is a diagram that illustrates an example of the information stored in the calculation formula DB <NUM>. As illustrated in <FIG>, the calculation formula DB <NUM> stores "scoring rule ID, calculation formula, reference coordinates, display plane, number of times of measurements" in a related manner. "Scoring rule ID" stored here is the information for identifying a scoring rule. "Calculation formula" is a calculation formula for the target angle to be scored. "Reference coordinates" are coordinates that serve as the reference for displaying the angle. "Display plane" is a plane for displaying the calculated angle. "Number of times of measurements" is the number of times of calculations per scoring.

In the example of <FIG>, the scoring rule ID "<NUM>" indicates that the reference coordinates are of the joint with number <NUM> and the value of "flag" dynamically changes in cases where the direction of circling is in a clockwise direction (CW) and in a counterclockwise direction (CCW). Furthermore, it indicates that, for the display, it is displayed on the XY plane and the number of times of measurements is one.

Furthermore, the scoring rule ID "<NUM>" indicates that the reference coordinates are of the joint with number <NUM> and the difference between the largest value and the smallest value (static attitude) of θR is the scoring target. Furthermore, it indicates that the number of times of measurements is twice as the largest value and the smallest value (static attitude) of θR are calculated and, for the display, it is displayed on the YZ plane.

The control unit <NUM> is a processing unit that controls the overall video reproduction apparatus <NUM>, and it is for example a processor. The control unit <NUM> includes a preprocessing unit <NUM>, a relating unit <NUM>, a comparison display unit <NUM>, and an angle calculating unit <NUM>. Furthermore, the preprocessing unit <NUM>, the relating unit <NUM>, the comparison display unit <NUM>, and the angle calculating unit <NUM> are examples of an electronic circuit such as a processor or examples of a process executed by a processor. Moreover, the angle calculating unit <NUM> is an example of a determining unit and a calculating unit, and the comparison display unit <NUM> is an example of a display control unit.

The preprocessing unit <NUM> is a processing unit that previously acquires definition information on a joint. Specifically, the preprocessing unit <NUM> measures a joint of the performer during 3D sensing with the 3D laser sensor. Then, the preprocessing unit <NUM> numbers the joint of the performer in accordance with the typical skeleton model and stores it in the joint position data DB <NUM>.

Furthermore, the preprocessing unit <NUM> acquires the performance video of the performer captured by using the video camera and stores it as video data in the storage unit <NUM>. Furthermore, the preprocessing unit <NUM> executes 3D sensing with the 3D laser sensor on the performance captured by using the video camera to acquire temporal changes in the joint position data on the performer and stores it in the storage unit <NUM>.

The relating unit <NUM> is a processing unit that relates the performance video with the joint position data during the performance. For example, the relating unit <NUM> assigns the record ID for relating the video data acquired by the preprocessing unit <NUM> and the joint position data on the performer acquired from the video data. Then, the relating unit <NUM> relates the record ID with the information for determining the video data and stores them in the video data DB <NUM> and relates the record ID with the joint position data and stores them in the joint position data DB <NUM>.

The comparison display unit <NUM> is a processing unit that displays the video data on the performance and the joint position data in the video data so as to be compared with each other. Specifically, the comparison display unit <NUM> divides the video data into each frame (performance image) and displays it. Furthermore, the comparison display unit <NUM> acquires the joint position data corresponding to each frame from the joint position data DB <NUM>, applies it to the skeleton model to generate a 3D model, and displays the generated 3D model. Specifically, the comparison display unit <NUM> displays the specific frame selected by the judge or the like from the frames forming the video data alongside of the 3D model of the joint position data on the performer at that point.

The angle calculating unit <NUM> is a processing unit that calculates the angle, which is the target to be scored, with respect to the 3D model of the joint position data on the performer corresponding to the selected frame and displays it on the 3D model. Specifically, the angle calculating unit <NUM> determines the event in the selected frame from the video data DB <NUM> and determines the scoring rule corresponding to the determined event from the scoring rule DB <NUM>. Here, when multiple scoring rules are determined, the angle calculating unit <NUM> prompts the judge or the like to select the scoring rule to be applied. Then, the angle calculating unit <NUM> determines the calculation formula corresponding to the selected scoring rule from the calculation formula DB <NUM> and, by using the determined calculation formula, calculates the target angle to be scored.

For example, in a case where the scoring rule "<NUM>" is selected, the angle calculating unit <NUM> calculates the angle by using the calculation formula in the first line of <FIG>. The angle calculated by the calculation formula is the angle obtained from the positions of the chest and the waist in the "final phase" of the longitudinal circling of pommel horse, and it is calculated based on the coordinates with the joint number <NUM> and the coordinates with the joint number <NUM>. Then, the angle calculating unit <NUM> causes the calculated angle to be displayed on the XY plane of the 3D model.

<FIG> is a diagram that illustrates an example of the calculation of the joint angle in the final phase (front-side support) of pommel horse. In the example illustrated in <FIG>, the longitudinal support is a basis. As illustrated in <FIG>, the angle of the target joint to be scored in the final phase is an angle θ formed between the straight line connecting the chest (the joint number <NUM>) and the waist (the joint number <NUM>) and the horizontal direction of a base A of the pommel horse. The angle θ is calculated by the calculation formula in the first line of <FIG>. Furthermore, it is determined that there is no fault and a point deduction is <NUM> when the angle θ is less than <NUM>° (<FIG>), it is determined that there is a minor fault and a point deduction is <NUM> points when the angle θ is between <NUM>° and <NUM>° (<FIG>), it is determined that there is an intermediate fault and a point deduction is <NUM> points when the angle θ is between <NUM>° and <NUM>° (<FIG>), and it is determined that there is a major fault and a point deduction is <NUM> points when the angle θ is more than <NUM>° (<FIG>).

Another example is described; the angle calculating unit <NUM> calculates the angle by using the calculation formula in the second line of <FIG> in a case where the scoring rule "<NUM>" is selected. The angle calculated by this calculation formula is the difference between the largest angle and the smallest angle (static attitude) of the "shoulder position" from the swing on still rings to the static attitude, and it is calculated based on the coordinates of the joint number <NUM> and the coordinates of the joint number <NUM>. Then, the angle calculating unit <NUM> causes the calculated angle to be displayed on the YZ plane of the 3D model.

<FIG> is a diagram that illustrates an example of the calculation of the joint angle during forward upward circling to support at ring height on still rings. As illustrated in <FIG>, the angle θR of the target joint to be scored during forward upward circling to support at ring height on still rings is the angle formed between the straight line connecting the right shoulder (the joint number <NUM>) and the right hand (the joint number <NUM>) and the direction (Y direction) horizontal to the ground. The angle θR is calculated by the calculation formula in the second line of <FIG>. As illustrated in the second line of <FIG>, the number of times of calculations for the angle θR is twice. This is because the difference between the largest angle and the smallest angle is the target to be scored.

Specifically, the angle calculating unit <NUM> calculates the largest angle θR from the frame that is designated by the judge as the target for determining the largest angle and calculates the angle θR at the stationary time from the frame that is designated by the judge as the target for determining the static attitude. Then, the difference is the target to be scored. For example, it is determined that there is no fault and a point deduction is <NUM> when the difference between the largest angle θR and the angle θR at the stationary time is less than <NUM>°, it is determined that there is a minor fault and a point deduction is <NUM> points when the difference is between <NUM>° and <NUM>°, it is determined that there is an intermediate fault and a point deduction is <NUM> points when the difference is between <NUM>° and <NUM>°, and it is determined that there is a major fault and a point deduction is <NUM> points when the difference is more than <NUM>°.

Next, an example of the screen display presented on the display, or the like, by the comparison display unit <NUM> is described. <FIG> is a diagram that illustrates a display example of the output screen. The comparison display unit <NUM> generates and displays a comparison display screen B illustrated in <FIG>. The comparison display screen B includes information such as the name of a performer, the team he belongs to, or the event performed. The comparison display screen B includes a video area C for displaying the image of a selected frame, a selection area D for displaying a scoring rule in a selectable manner, an area P for displaying each frame, and a scroll bar Q for selecting a frame. Furthermore, the XYZ plane is previously determined on the screen. For example, in the case of pommel horse, it is determined that the horizontal direction of the pommel horse is the X axis, the vertical direction is the Y axis, and the depth direction is the Z axis.

Specifically, the comparison display unit <NUM> displays the video (image) corresponding to the frame selected by the judge on the video area C of the comparison display screen B. Furthermore, the comparison display unit <NUM> displays the image of each frame forming the video data on the area P of the comparison display screen B. Moreover, the comparison display unit <NUM> displays, on the area C, the image of the frame selected from the frames displayed on the area P by a user, such as a judge, moving the scroll bar Q.

Furthermore, the comparison display unit <NUM> displays the skeleton model corresponding to the image of the frame displayed on the video area C alongside of the image in the video area C. Here, the comparison display unit <NUM> displays images of the skeleton model viewed from multiple viewpoints that are previously determined. Furthermore, the angle calculating unit <NUM> uses the calculation formula for the scoring rule selected from the scoring rules displayed on the selection area D to display the angle on each image of the skeleton model displayed. That is, the judge selects the scoring rule displayed on the selection area D so as to display the desired angle.

For example, when the judge selects the frame in <NUM> seconds and a scoring rule D1, the comparison display unit <NUM> displays the image of the frame in <NUM> seconds on the area C and displays the skeleton model for the image. Then, the angle calculating unit <NUM> acquires the joint position data corresponding to the frame from the joint position data DB <NUM>, calculates the target angle to be scored by using the calculation formula corresponding to the scoring rule D1, and then displays it by being superimposed on the displayed image of the skeleton model.

Then, when the judge selects the frame in <NUM> seconds while maintaining the scoring rule D1, the comparison display unit <NUM> displays the image of the frame in <NUM> seconds on the area C and displays the skeleton model for the image. Then, the angle calculating unit <NUM> displays the angle calculated by using the joint position data corresponding to the frame by being superimposed on the displayed image of the skeleton model.

Furthermore, when the judge selects a scoring rule D2 while selecting the frame in <NUM> seconds, the comparison display unit <NUM> maintains the display, and the angle calculating unit <NUM> acquires the joint position data corresponding to the frame from the joint position data DB <NUM>, calculates the target angle to be scored by using the calculation formula corresponding to the scoring rule D2, and then displays it by being superimposed on the displayed image of the skeleton model.

Furthermore, in a case where a single frame is selected, the angle calculating unit <NUM> previously calculates the angle with regard to the previous and the next frames. For example, when the frame in <NUM> seconds is selected, the angle calculating unit <NUM> also calculates the angle using the skeleton model of the frame in <NUM> seconds and the angle using the skeleton model of the frame in <NUM> seconds in addition to the angle using the skeleton model of the frame in <NUM> seconds.

Thus, the previous calculation allows a decrease in the time to calculate the angle due to frame switching and a reduction in the workload of the judge. Furthermore, even in a case where the judge checks angles by frequently switching frames, the checking task may be performed without stress.

Furthermore, the comparison display unit <NUM> may also display, in an enlarged manner, the image of the skeleton model selected by double click or the like from multiple images of the skeleton model. <FIG> is a diagram that illustrates a display example of the enlarged display. As illustrated in <FIG>, in a case where the image of the skeleton model on the top is selected in the state of <FIG>, the comparison display unit <NUM> displays the image of the skeleton model on the top in an enlarged manner. Furthermore, the selection of a frame may be executed by pressing the button displayed under the area B.

Furthermore, the angle calculating unit <NUM> may also change the display format of the angle in accordance with the degree of point deduction. <FIG> is a diagram that illustrates a display example of point deduction. For example, the angle calculating unit <NUM> displays the angle in blue in the case of a minor fault (<FIG>), displays the angle in yellow in the case of an intermediate fault (<FIG>), and displays the angle in red in the case of a major fault (<FIG>). This makes it possible to visually display the degree of point deduction and to reduce the workload of the judge. Furthermore, the colors may be optionally changed.

<FIG> is a flowchart that illustrates the flow of preprocessing. As illustrated in <FIG>, the preprocessing unit <NUM> receives an instruction to start to capture a performance (S101) and then assigns a record ID (S102). Then, the preprocessing unit <NUM> acquires joint position data with the 3D laser sensor (S103) while acquiring video data with the video camera (S104).

Then, the relating unit <NUM> relates the joint position data with the video data and stores them in the joint position data DB <NUM> and the video data DB <NUM>, respectively (S105). Furthermore, the timing in which the record ID is assigned may be optionally changed.

<FIG> is a flowchart that illustrates the flow of a process to register a joint position. As illustrated in <FIG>, the preprocessing unit <NUM> assigns the scoring rule ID (S201), receives, from the administrator or the like, points (action, scoring target, threshold, point deduction, or the like) checked by the gymnastics judge during scoring, and registers them in the scoring rule DB <NUM> for each athletic event (S202).

Then, the preprocessing unit <NUM> determines the joint number related to the skeleton model for each scoring rule ID (S203) and registers the calculation formula for calculating the angle formed by a joint, received from the administrator or the like, in the calculation formula DB <NUM> (S204).

Then, the preprocessing unit <NUM> receives, from administrator or the like, the designation of the plane on which the circular arc representing the angle of the joint is displayed and sets it in the calculation formula DB <NUM>.

Then, in a case where the formed angles at multiple times are used for scoring, the preprocessing unit <NUM> receives, from the administrator or the like, the designation of the number of times of measurements as needed and registers it in the calculation formula DB <NUM> (S206).

<FIG> is a flowchart that illustrates the flow of a comparison display process. As illustrated in <FIG>, the comparison display unit <NUM> retrieves the video data on the performance of the player for a rejudgement from the video data DB <NUM> and displays it (S301). Furthermore, the performance of the player for a rejudgement may be designated by the administrator or the like.

Then, the comparison display unit <NUM> extracts the joint position data related to the record ID of the retrieved video data from the joint position data DB <NUM> (S302). Then, the comparison display unit <NUM> applies the extracted joint position data to a skeleton model to generate a 3D model of the human body and displays it (S303). Then, the comparison display unit <NUM> displays the 3D model in temporal synchronization with the operation to reproduce the video data (S304).

<FIG> is a flowchart that illustrates the flow of a process to display the angle formed by a joint. As illustrated in <FIG>, the angle calculating unit <NUM> acquires the event name with the record ID that is the target to be determined from the video data DB <NUM> (S401) and acquires the scoring rule ID regarding the event from the scoring rule DB <NUM> (S402).

Then, the angle calculating unit <NUM> retrieves the calculation formula for each acquired scoring rule ID from the calculation formula DB <NUM> and calculates the angle formed by a joint at the time when the performance is reproduced (S403). Here, in a case where the number of times of measurements is set to be more than one (S404: Yes), the angle calculating unit <NUM> calculates the difference between the angles at multiple measurement times in accordance with the calculation formula (S405).

Furthermore, in a case where the number of times of measurements is set to be not more than one (S404: No) or S405 is executed, the angle calculating unit <NUM> displays the circular arc indicating the angle on the plane horizontal to the reference plane by using the reference coordinates for the calculation formula DB <NUM> as an origin in the displayed 3D model in conjunction with the selection button (S406). Furthermore, the angle calculating unit <NUM> changes the color of the circular arc in accordance with a threshold (S407).

As described above, as the video reproduction apparatus <NUM> calculates the joint angle of a player for a gymnastics judge in accordance with a calculation formula and displays it, it is possible to provide information for conducting scoring in the uniform manner in accordance with a scoring rule. As a result, a gymnastics judge is capable of scoring a performance in a detailed manner in accordance with a scoring rule defined for each event or element without depending on the individual's habit, capability, experience, or the like.

Although the gymnastics such as pommel horse and still rings are described as examples in the above embodiment, these are not limitations, and other sports are also applicable, in which a player performs a sequence of elements and a judge gives a score. Examples of the other sports are other events of gymnastics, figure skating, rhythmic gymnastics, or cheerleading.

Furthermore, a typical 3D modeling technology may be used as the technology for acquiring the three-dimensional coordinates of each joint from the skeleton model.

Furthermore, all or any of each processing function performed by each device may be implemented by a CPU and a program analyzed and executed by the CPU or may be implemented as a wired logic hardware.

<FIG> is a diagram that illustrates an example of a hardware configuration. As illustrated in <FIG>, the video reproduction apparatus <NUM> includes a communication interface 10a, an HDD (Hard Disk Drive) 10b, a memory 10c, and a processor 10d.

The communication interface 10a is a network interface card, or the like, which controls communications with other devices. The HDD 10b is an example of a storage device that stores a program or data.

Examples of the memory 10c include a RAM (Random Access Memory) such as an SDRAM (Synchronous Dynamic Random Access Memory), a ROM (Read Only Memory), or a flash memory. Examples of the processor 10d are a CPU (Central Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or a PLD (Programmable Logic Device).

Furthermore, the video reproduction apparatus <NUM> operates as an information processing apparatus that reads and executes a program to implement a video reproduction method. Specifically, the video reproduction apparatus <NUM> executes a program for performing the same functions as those of the preprocessing unit <NUM>, the relating unit <NUM>, the comparison display unit <NUM>, and the angle calculating unit <NUM>. As a result, the video reproduction apparatus <NUM> may execute a process for performing the same functions as those of the preprocessing unit <NUM>, the relating unit <NUM>, the comparison display unit <NUM>, and the angle calculating unit <NUM>. Moreover, the program described in this different embodiment is not exclusively performed by the video reproduction apparatus <NUM>. For example, the present invention is also applicable to a case where the program is executed by a different computer or server or a case where the program is executed by them in cooperation.

Claim 1:
A performance display program causing a computer (<NUM>) to execute a process comprising:
displaying (S304) on a display frames of video data of a performance performed by a performer, the video data being captured by a video camera, and synchronously therewith a 3D model image generated based on joint position data of the performer acquired by using a sensor during the performance;
having a user select a frame within the frames;
referring (S402) to a database in a storage unit (<NUM>) registering therein scoring rule for each athletic event of the performance, the database storing a scoring rule ID that identifies a scoring rule, to acquire the scoring rule ID regarding the athletic event;
based on calculation information that is registered in the storage unit and includes the scoring rule ID of the athletic event and, in association therewith, a calculation formula for calculating an angle formed by a joint of the performer and to be scored, and the joint position data corresponding to the selected frame, calculating (S403) the angle; and
displaying the calculated angle superimposed on the 3D model image corresponding to the selected frame,
wherein the calculating includes also calculating an angle based on the calculation information and the joint position data corresponding to a frame previous to the selected frame, and an angle based on the calculation information and the joint position data corresponding to a frame next to the selected frame.