Abstract:
A system and method for receiving positional data corresponding to stages of movement of a person within a time interval, determining from the positional data, at each stage of movement, one of: a position, an orientation, and a pose of the person, and based on the one of the position, the orientation, and the pose of the person determined for each stage of movement, generating a visualization graphic, separate from a display of the person, that illustrates one of the positions, orientations, and poses of the person at the stages of movement within the time interval.

Description:
BACKGROUND  
       [0001]    Viewers of some sporting events may have difficulty following rapidly occurring movement. In some events, such as figure skating, diving and snowboarding, it may be desirable to represent the motion of participants in a graphical manner that may make it easier for viewers to follow rapid actions of the participants. 
       SUMMARY OF THE INVENTION 
       [0002]    A method for receiving positional data corresponding to stages of movement of a person within a time interval, determining from the positional data, at each stage of movement, one of: a position, an orientation, and a pose of the person, and based on the one of the position, the orientation, and the pose of the person determined for each stage of movement, generating a visualization graphic, separate from a display of the person, that illustrates one of the positions, orientations, and poses of the person at the stages of movement within the time interval. 
         [0003]    A system having a data source collecting positional data corresponding to stages of a movement of a person within a time interval and a data server. The data server including a a data receiving module receiving the positional data, a calculation module determining, based on the positional data at each of the stages of movement within the time interval, one of a position, an orientation and a pose of the person and graphical processing module generating, based on the one of the position, the orientation, and the pose of the person determined for each stage of movement a visualization graphic, separate from a display of the person, that illustrates the one of the positions, the orientations and the poses of the person at the stages of movement within the time interval. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0004]      FIG. 1  illustrates an exemplary method for rendering a visualization of the motion of athletes according to an exemplary embodiment. 
           [0005]      FIG. 2  illustrates a first exemplary visualization of the motion of an athlete according to the exemplary method of  FIG. 1 . 
           [0006]      FIG. 3  illustrates a second exemplary visualization of the motion of an athlete according to the exemplary method of  FIG. 1 . 
           [0007]      FIG. 4A  illustrates a third exemplary visualization of the motion of an athlete according to the exemplary method of  FIG. 1 . 
           [0008]      FIG. 4B  illustrates the third exemplary visualization of  FIG. 4A  with additional graphical features. 
           [0009]      FIG. 5  illustrates a fourth exemplary visualization of the motion of an athlete according to the exemplary method of  FIG. 1 . 
           [0010]      FIG. 6  illustrates three different views of a fifth exemplary visualization of the motion of an athlete according to the exemplary method of  FIG. 1 . 
           [0011]      FIG. 7  illustrates an exemplary system for performing the method of  FIG. 1  according to an exemplary embodiment. 
       
    
    
     DETAILED DESCRIPTION 
       [0012]    The exemplary embodiments may be further understood with reference to the following description of exemplary embodiments and the related appended drawings, wherein like elements are provided with the same reference numerals. Specifically, the exemplary embodiments relate to methods and systems for visualizing rapid motion of athletes. 
         [0013]    Viewers of some sporting events may have difficulty following rapidly movement by athletes participating in the sporting events. In some events, such as figure skating, diving and snowboarding, it may be desirable to represent the motion of athletes in a graphical manner that may make it easier for viewers to follow rapid actions of the participants. Appreciating these actions, which occur in three dimensions, may require intense concentration even by fans of the sports, and non-fans may find them incomprehensible. 
         [0014]    Previous techniques for the visualization of such athletes have included slow-motion replays and stutter/strobe illustrations where still images of the athlete at various points in time are displayed concurrently (i.e., as if the athlete is being illuminated with a strobe light and an image is frozen each time the strobe flashes). The exemplary embodiments provide for better visualization of the movement of athletes in three dimensions, particularly where the movement of such athletes may be difficult to follow unaided. Those of skill in the art will understand that the specific implementation details and techniques described herein, including those of the exemplary embodiments, may be implemented using differing techniques. 
         [0015]      FIG. 1  illustrates an exemplary method  100  providing for the visualization of the motion of an athlete in three-dimensional space. In step  110 , positional data is received for the athlete over a given time interval. The time interval may be an appropriate interval for the sport in which the athlete is participating (e.g., for a diver, the time interval may be an entire dive; for a figure skater, the time interval may be the duration of a jump; etc.). The positional data may be sampled periodically during the interval, at a sampling frequency that is appropriate for the sport in which the athlete is participating. The positional data may be sampled several times per second, but does not typically need to be sampled at a video frame rate (e.g., 60 frames per second). The sampling rate may be predetermined or variable, and may typically be selected by a broadcaster of the sporting event to be high enough to capture the details of the athlete&#39;s body movements. 
         [0016]    Positional data may be from a single source or from a plurality of sources. In one exemplary embodiment, multiple video cameras may be used to record video of the athlete, and stereo vision techniques may be used to determine the exact position of the athlete in three-dimensional space; however, such calculations may be processing-intensive. In another exemplary embodiment, sensors may be embedded into the athlete&#39;s uniform, or fixed on the athlete&#39;s body in some other manner; such sensors may record the athlete&#39;s position (e.g., with a GPS receiver) and orientation (e.g., with inertial sensors, accelerometers, etc.), and the sensors may be used to record positional data for the athlete; for example, such sensors may be found in smart phones or other similar mobile computing devices. In another exemplary embodiment, data from sensors such as described above and recorded video may be combined to determine the position of the athlete. The data used in such a combined embodiment may include both video data from external cameras and video data from an on-board camera (e.g., a head-mounted camera, a camera integrated into a sensor package such as a smart phone, etc.). The positional data may be determined in real time as the sporting event is occurring, or may be recorded and exported to a more powerful system for subsequent processing. In one embodiment, the athlete&#39;s positional data may be expressed as [X, Y, Z, yaw, pitch, roll] relative to a venue-specific reference frame; at a minimum, the positional data may be expressed as [X, Y, Z]. 
         [0017]    In step  120 , the positions of two of the athlete&#39;s extremities are determined for each sample of the athlete&#39;s position; in another exemplary embodiment, the results of steps  110  and  120  may be achieved in a single step, such as by directly measuring the location with sensors located at the extremities. In one embodiment, this may be performed by determining a position representing the center of the athlete (e.g., the location of a positional sensor) and determining extremity locations based on the size of the athlete and the orientation as determined by inertial sensors or other similar sensors. In another embodiment, this may be accomplished based on stereovision techniques using solely video data of the athlete. 
         [0018]    In one embodiment, the extremities may be the locations of the athlete&#39;s head and feet, and the exemplary embodiments described herein make reference to the position of the athlete&#39;s head and feet and a graphical representation based on such positions; however, in an alternative embodiment, the positions determined in this step may represent the left and right sides of the athlete rather than the top and bottom (i.e., head and foot). In such an embodiment, the positions may be determined based on the athlete&#39;s orientation and “wingspan” (e.g., of the athlete&#39;s shoulders or arms). Such a representation may be preferable for some sports, such as diving. 
         [0019]    In step  130 , a line segment within three-dimensional space is drawn between the points representing the athlete&#39;s head and feet for each sample of the athlete&#39;s position. This may be performed using standard graphical techniques. In step  140 , a graphic is created and rendered reflecting a continuous path within three-dimensional space, drawn between and through the line segments drawn in step  130 ; this graphic will be referred to hereinafter as a “ribbon”. The ribbon may be drawn using known interpolation techniques; smooth spline interpolation is a standard feature of many rendering engines. 
         [0020]    In step  150 , the ribbon created in step  140  is output for subsequent use or display. Those of skill in the art will understand that the specific nature of this step may vary widely among differing embodiments, and that the specific manners of use to be described herein are only exemplary. In one embodiment, the ribbon may be rendered into a two-dimensional view using standard techniques. For example, the two-dimensional view may be the field of view of a video broadcast of the athlete, such as the view of a television broadcast. The ribbon may be superimposed over one or more still images of the athlete in such a view. In such a view, the ribbon and various athlete positions may be shown in a static manner, or, alternately, the ribbon and various athlete positions may be shown in motion so that, as the athlete performs, a trail of ghost images is left for the viewer to see. 
         [0021]      FIG. 2  illustrates a first exemplary display  200  including an exemplary output of the method of  FIG. 1 . In  FIG. 2 , the athlete shown is a snowboard jumper. The ribbon  210  is shown superimposed over still images of the athlete of  FIG. 2 , who is shown in full saturation in a first position  220 , a second position  222 , and a third position  224 . The athlete is also shown in lighter saturation (e.g., “ghosted”) in ghost positions  230 ,  232 ,  234 ,  236  and  238 . The display  200  also includes angle labels  240  and  242 , indicating the angular progress of the athlete through a spin at the points indicated by the labels. The angle labels  240  and  242  may be manually added by a broadcaster providing the display  200 , or, alternately, may be added automatically, at fixed time intervals (e.g., every half second) or at times of maximal acceleration, angular velocity, or other pertinent physical maxima/minima. In the example where the athlete is a snowboarder, the angle may be determined by analyzing the orientation of the snowboard. 
         [0022]    The superimposition of the ribbon  210  and the labels  240  and  242  over the various positions of the athlete may thereby enable a viewer of the display  200  to follow the progress of the athlete through the jump shown. Additionally, those of skill in the art will understand that while  FIG. 2  is a still image showing still illustrations of the athlete in the various positions described above, the ribbon  210  and the labels  240  and  242  may alternately be superimposed over a video of the athlete moving through the jump being illustrated. In such an embodiment, the projection of the three-dimensional ribbon into two dimensions may be continuously updated as the camera angle follows the athlete. 
         [0023]      FIG. 3  illustrates another exemplary display  300  including an exemplary output of the method of  FIG. 1 . In  FIG. 3 , like in  FIG. 2 , the athlete shown is a snowboard jumper. The display  300  includes a ribbon  310  showing the progress of the athlete through a jump. The display  300  also includes images of the athlete similar to those described above with reference to  FIG. 2 . The display  300  also includes labels  320 ,  321 ,  322 ,  323 ,  324  and  325 , representing the angular progress of the athlete through the jump. The ribbon  310  is subdivided into differently shaded portions  330 ,  331 ,  332 ,  333 ,  334 , and  335 , with each portion representing a portion of the jump. For example, portion  330  represents the portion of the jump from its start through label  320 , representing the athlete&#39;s first 360 degrees of rotation; portion  331  represents the portion of the jump from label  320  through label  321 , representing the athlete&#39;s continued rotation from 360 degrees to 540 degrees. Those of skill in the art will understand that the specific shades used to shade the ribbon  310  are only exemplary, and that varying shades or colors may be used. In one exemplary embodiment, the shades or colors may vary to represent the speed of the athlete&#39;s rotation (e.g., ranging from green representing slow rotation to red representing fast rotation), the athlete&#39;s linear speed (e.g., ranging from green representing slow linear speed to red representing fast linear speed), or the athlete&#39;s acceleration. In another exemplary embodiment, the opacity of the ribbon  310  may vary (e.g., the ribbon may be more transparent where the linear or angular motion of the athlete is slower and more opaque where the linear or angular motion of the athlete is more rapid, or vice versa) in order to assist the viewer in following the motion of the athlete. 
         [0024]      FIG. 4A  illustrates another exemplary display  400  including an exemplary output of the method of  FIG. 1 . In  FIG. 4A , the athlete shown is a figure skater performing a jump. The display  400  includes a ribbon  410  showing the progress of the athlete through the jump. The display also includes images  420 ,  422  and  424  of the athlete at various points during the jump; as discussed above, those of skill in the art will understand that the ribbon  410  may also be shown on a video display of the athlete performing the jump. The ribbon  410  deflects outward toward its bottom at deflected portions  412  and  414 . In deflected portions  412  and  414 , the ribbon does not strictly represent the head and foot positions of the athlete, but, rather, is deflected outward to represent the angular rotation (i.e., spinning) of the athlete. 
         [0025]    This deflection may be understood as similar to the manner in which an item worn by an athlete may deflect outward and upward from the athlete&#39;s body due to centrifugal force. The positioning of the deflected ribbon may therefore be determined by determining the rate of the athlete&#39;s angular rotation, determining an amount of centrifugal force exerted by such rotation, and determining a deflection angle proportional to such centrifugal force. In one embodiment, a predetermined maximum deflection angle may be determined (e.g., 45 degrees from vertical, 60 degrees from vertical, etc.), and the determination of the deflection angle may be made in such a manner such that the deflection angle is the maximum deflection angle at the athlete&#39;s maximum rate of angular rotation, and is correspondingly less at slower rates of angular rotation. Thus, for example, the deflection portion  412 , which is deflected at a greater angle from the vertical than the deflected portion  414 , may indicate a greater rate of angular rotation. Additionally, the length of the ribbon  410  may be truncated from its full length (i.e., the distance from the athlete&#39;s head to foot), in order that the ribbon  410  may be fully opaque but still not obscure the viewer&#39;s ability to see the athlete in the display  400 . 
         [0026]      FIG. 4B  illustrates a display  450  that is a modified version of the display  400  of  FIG. 4A . In the display  450 , the ribbon  460  is not shown with a solid color, but, rather, includes a first pattern  470  and a second pattern  480 , with each pattern corresponding to one side of the ribbon. The use of two patterns may aid the viewer in understanding the direction the athlete is facing at a given point in the ribbon  460 , and also may further aid the viewer in visualizing the rate of angular rotation of the athlete. In one exemplary embodiment, the first pattern  470  and the second pattern  480  may be chosen based on the attire worn by the athlete. Alternately, those of skill in the art will understand that the ribbon  460  may simply be different colors on its two sides, and may provide similar viewer aid to that described above. 
         [0027]    The method  100  was described with reference to a vertical ribbon that spans from a head position and orientation of the athlete to a foot position of the athlete. However, as noted above, in some sports, such as diving, a horizontal ribbon may be preferable for providing information to the viewer. Therefore, in another exemplary embodiment, the endpoints of the lines used to generate the ribbon may represent the left and right sides of the athlete rather than the top and bottom (i.e., head and foot). In such an embodiment, the positions may be determined based on the athlete&#39;s orientation and “wingspan” (e.g., of the athlete&#39;s shoulders or arms).  FIG. 5  illustrates a display  500  including such a ribbon  510 , together with an athlete in four positions  520 ,  522 ,  524  and  526  summarizing the motion generating the ribbon  510 . 
         [0028]    While the embodiments described above have referenced projection of the ribbon into two dimensions for display, such as concurrently with video or still images of an athlete, in other embodiments, the ribbon may be provided to users as a three-dimensional (“3D”) model. For example, if a television broadcaster is generating the ribbon for television display (i.e., projected into two dimensions) during a broadcast of a sporting event, the broadcaster may additionally provide the 3D object model of the ribbon via the Internet to users who are visiting the television network&#39;s website. In such an embodiment, the users may be able to manipulate the ribbon, using standard techniques for the manipulation of 3D, to view it from varying angles, in order to further improve the users&#39; understanding of the sporting event. This function may also be used by a producer of content to further illustrate to viewers, e.g., in an instant replay video, the movement of the athlete. 
         [0029]      FIG. 6  illustrates such a ribbon  610 , shown in three different views. The first view  620  including ribbon  610  illustrates a figure skater performing a jump in the same manner as shown previously in  FIG. 4A . The second view  630  shows the same ribbon  610  from the opposite side of the jump. The third view  640  shows the same ribbon  610  as viewed from behind the athlete&#39;s starting position, facing in the direction that the figure skater is traveling. A user may manipulate the ribbon  610  to produce the varying views  620 ,  630  and  640  by, for example, clicking and dragging a mouse, using keyboard inputs, or any other manner for manipulating the view of a 3D object model that is known in the art. 
         [0030]      FIG. 7  illustrates an exemplary system  700  for generating a visualization of the motion of an athlete  701  using a method such as the method  100  of  FIG. 1 . The system  700  includes data capture devices (e.g., cameras, inertial sensors, detectors, radar guns, LIDAR sensors, etc.)  710 ,  712  and  714  capturing positional data for the athlete  701  (for example, data relating to the center of mass of the athlete, as well as various body parts that can be used to determine the orientation of the athlete in addition to the position). In the system  700 , the data capture device  710  is an onboard data gathering apparatus (e.g., a smart phone) carried or worn by the athlete  701 , and the data capture devices  712  and  714  are video cameras recording the motion of the athlete  701  from two different vantage points. However, as described above, those of skill in the art will understand that, in other systems, different types of data capture devices may be used. For example, in one alternative implementation, no onboard data gathering apparatus may be present, and the athlete&#39;s positional data may be captured solely by external devices, such as video cameras. In another alternative implementation, a differing number of cameras may be present; for example, four cameras may be present rather than two. In a further alternative implementation, another type of data capture device not expressly described herein may be used. 
         [0031]    Data from the data capture devices  710 ,  712  and  714  is exported to a data server  720 . The data server  720  includes a memory  722  which may store the raw data, and a processor  724  that may process the data (e.g., using the exemplary method  100 ) either in real time or after the event being captured by the data capture devices  710 ,  712  and  714  has been completed. The data server  720  may include a data receiving module  730  receiving data at a potentially high rate from the data capture devices  710 ,  712  and  714  and properly formatting it for storage in the memory  722 , a calculation module  740  performing the calculations necessary to convert the athlete positional data into the ribbon as described above with reference to method  100 , and a graphical processing module  750  formatting the ribbon data for display to users, such as projecting the ribbon into two dimensions for display with a still image or video of the athlete. Those of skill in the art will understand that the data receiving module  730 , the calculation module  740 , and the graphical processing module  750  may be may be software modules comprising lines of code, and may be stored in memory  722  and executed by processor  724 . 
         [0032]    The processed data (e.g., the head and foot positions, lines, and ribbon) generated by the processor  724  may also be stored in the memory  722  for subsequent use or retrieval. The data server  720  may also be communicatively coupled to a broadcast server  760 , which may provide the processed data to an end user terminal  770 . As described above, the broadcast server  760  may be adapted for video broadcast (e.g., over-the-air, cable television, satellite television, etc.) or data broadcast (e.g., over the Internet, an intranet, etc.), and the end user terminal  770  may be a type of terminal appropriate for the nature of the broadcast server  760  (e.g., a television, a desktop or laptop computer, a tablet, a smart phone, etc.). 
         [0033]    Therefore, the exemplary embodiments described above may enable viewers of sporting events to better understand and visualize the progress of rapidly moving athletes. The exemplary embodiments may allow for greater accuracy of illustration of the motion of athletes than previous efforts to accomplish the same results, and may allow for data-driven visualization that directly correlates with the athlete&#39;s body. The exemplary embodiments may be implemented in a manner that allows for real-time illustration, or with post-processing with lower computing power requirements. 
         [0034]    Those of skill in the art will understand that the above-described exemplary embodiments may be implemented in any number of matters, including as a software module, as a combination of hardware and software, etc. For example, the exemplary method  100  may be embodied in a program stored in a non-transitory storage medium and containing lines of code that, when compiled, may be executed by a processor. 
         [0035]    It will be apparent to those skilled in the art that various modifications may be made to the exemplary embodiments, without departing from the spirit or the scope of the invention. Thus, it is intended that the present invention cover modifications and variations of this invention provided they come within the scope of the appended claims and their equivalents.