Abstract:
An extraction device includes a processor that executes a procedure. The procedure includes detecting a transition between a close-up captured image and a long distance captured image from captured images obtained by capturing a sports game, and, based on the detected transition, extracting a timing corresponding to a break in play in the sports game, or extracting a captured image corresponding to a break in play in the sports game.

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
       [0001]    This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2014-054208, filed on Mar. 17, 2014, the entire contents of which are incorporated herein by reference. 
       FIELD 
       [0002]    The embodiments discussed herein are related to a recording medium storing an extraction program, an extraction method, and an extraction device. 
       BACKGROUND 
       [0003]    Hitherto, services exist that distribute footage captured of a soccer game both as a live distribution, and as a distribution of Video On Demand (VOD) content. In such services, sometimes there are occasions when the extraction of a particular scene from footage is desired. 
         [0004]    For example, an exciting scene is recognized as being when a sharp rise in sound intensity is detected in audio data corresponding to footage. 
         [0005]    There has also been a proposal to identify highlight segments in video including a frame sequence. 
       RELATED PATENT DOCUMENTS 
       [0006]    Japanese National Publication of International Patent Application No. 2008-511186 
         [0007]    In the extraction of scenes described above, sometimes, for example, the extraction of breaks in play is desired. 
       SUMMARY 
       [0008]    According to an aspect of the embodiments, a non-transitory recording medium stores an extraction program that causes a computer to execute a process. The process includes detecting a transition between a close-up captured image and a long distance captured image from captured images obtained by capturing a sports game, and, based on the detected transition, extracting a timing corresponding to a break in play in the sports game, or extracting a captured image corresponding to a break in play in the sports game. 
         [0009]    The object and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the claims. 
         [0010]    It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention. 
     
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
         [0011]      FIG. 1  is a functional block diagram illustrating a schematic configuration of a footage distribution system according to an exemplary embodiment; 
           [0012]      FIG. 2  is a diagram explaining an example of editing; 
           [0013]      FIG. 3  is a diagram explaining inter-frame differences; 
           [0014]      FIG. 4  is a diagram illustrating an example of a long distance frame; 
           [0015]      FIG. 5  is a diagram illustrating an example of a close-up frame; 
           [0016]      FIG. 6  is a diagram explaining extraction of frames representing the start and end of a single play; 
           [0017]      FIG. 7  is a diagram illustrating an example of a metadata file; 
           [0018]      FIG. 8  is a schematic block diagram illustrating an example of a computer that functions as an extraction device; 
           [0019]      FIG. 9  is a schematic block diagram illustrating an example of a computer that functions as a distribution device; 
           [0020]      FIG. 10  is a flowchart illustrating an example of editing processing; 
           [0021]      FIG. 11  is a flowchart illustrating an example of extraction processing; and 
           [0022]      FIG. 12  is a flowchart illustrating an example of distribution processing. 
       
    
    
     DESCRIPTION OF EMBODIMENTS 
       [0023]    Detailed explanation follows below regarding examples of exemplary embodiments of the technology disclosed herein with reference to the drawings. In each of the exemplary embodiments below, explanation is given regarding an example in which an extraction device of the technology disclosed herein is applied to a footage distribution system that distributes captured footage of a soccer match. 
         [0024]    As illustrated in  FIG. 1 , a footage distribution system  10  according to the present exemplary embodiment includes an extraction device  20  and a distribution device  30 . The extraction device  20  and the distribution device  30  are mutually connected through a network. In the footage distribution system  10 , footage appended with metadata indicating exciting scenes extracted by the extraction device  20  is distributed through the network to a specified terminal by the distribution device  30 . The extraction device  20  includes a detection section  22 , an extraction section  24 , and a generation section  26 . The distribution device  30  includes an editing section  32 , and a distribution section  34 . 
         [0025]    First, detailed description is given regarding each section of the distribution device  30 . 
         [0026]    The editing section  32  acquires captured footage of a soccer match (referred to as “captured footage” hereafter). Note that in the present exemplary embodiment, the footage is media including video data and audio data. The video data included in the captured footage is captured at a frame rate such as 30 fps or 60 fps, and includes plural frames. Each of the frames is associated with time data indicating the elapsed time since capture of the captured footage started. The captured footage includes audio data including audio such as cheering in the match stadium, and audio of commentators, live reports, and the like. The audio data is time sequenced data indicating a sound intensity level for respective sampling points, and at each sampling point, the audio data is associated with time data that synchronizes with time data associated with each frame in the video data, such that the audio data and the video data are synchronized with each other. 
         [0027]    The editing section  32  applies editing instructions, instructed by an operation by an operator using a display device and an input device, not illustrated in the drawings, to each frame in the video data included in the captured footage using image processing. Editing instructions include, for example, adding an overlay  80  displaying a game state as illustrated in  FIG. 2 . Below, footage in which editing on the video data included in the captured footage has been performed by the editing section  32  is referred to as “edited footage”. The editing section  32  transmits the edited footage to the extraction device  20 . 
         [0028]    The distribution section  34  acquires footage to which metadata generated by the extraction device  20  has been appended (referred to as “metadata appended footage” hereafter; detailed description given below). The metadata appended footage is converted to distribution footage, according to specified standards, and distributed to the distribution destination terminal (omitted from illustration in the drawings) by the distribution section  34 . 
         [0029]    Next, detailed description is given regarding each section of the extraction device  20 . 
         [0030]    The detection section  22  acquires edited footage transmitted from the distribution device  30 . The detection section  22  first detects a cut transition based on the video data included in the edited footage. Cut refers to continuous portions in the video data captured at the same angle. Specifically, the detection section  22  computes differences between each frame and the previous frame. The inter-frame difference is, for example, computed according to the sum of the differences in pixel data between pairs of corresponding pixels in the respective frames. As illustrated in  FIG. 3 , the detection section  22  monitors the computed differences for each frame compared to the respective previous frame, and frames for which the difference compared to the previous frame exceeds a specific threshold value are detected as frames representing a cut transition. 
         [0031]    In the consecutive frames  82 ,  84 ,  86 ,  88  in  FIG. 3 , the difference between the frame  84  and the previous frame  82 , and the difference between the frame  86  and the previous frame  84  are comparatively small values. However, the difference between the frame  88  and the previous frame  86  is a large value. Moreover, a cut transition is made between frame  86  and frame  88 . Accordingly, a cut transition can be detected between frame  86  and frame  88  in the example of  FIG. 3 , namely, the frame  86  can be detected as the final frame of a preceding cut, and the frame  88  can be detected as the leading frame of the next cut, based on the difference between the frame  88  and the previous frame  86 . 
         [0032]    Moreover, the detection section  22  may designate the group of frames including from the leading frame until the final frame of the detected cut as a group of frames representing one cut. Then, the detection section  22  determines whether at least one frame included in the group of frames representing one cut is a frame captured at long distance (referred to as a “long distance frame” below), or a frame captured close-up (referred to as a “close-up frame” below). Note that long distance frames and close-up frames are examples of long distance capture video and close-up capture video of the technology disclosed herein. This determination may, for example, be performed by employing an identification model for identifying long distance frames and close-up frames. The identification model may be generated by, for example, training on plural long distance frames like those illustrated in  FIG. 4 , and plural close-up frames like those illustrated in  FIG. 5 . The detection section  22  detects a transition between a long distance frame and a close-up frame based on the determination result. 
         [0033]    The extraction section  24  extracts a break in play based on the transition between the long distance frame and the close-up frame detected by the detection section  22 . Generally, in footage of a soccer game, play is captured at long distance and transition is made to cut to a captured close-up, such as the demeanor of a player on the pitch or on the bench, at respective breaks between single plays. Accordingly, the location where transition is made between a long distance frame and a close-up frame is the start of, or end of, a single play. Note that a single play refers to continuous flow of the ball, from when a ball that was temporarily stopped due to a foul or the ball going out or the like starts to be in play again due to a free-kick, a throw-in, or the like, up until another stop. Note that the meaning of starting and stopping of the ball does not strictly refer to ball movement itself, but rather refers to the status of the ball at recommencement, or suspension, of play. 
         [0034]    Specifically, as illustrated in  FIG. 6 , the extraction section  24  extracts the long distance frame of the transition from the close-up frame to the long distance frame (for example, frame  94  in  FIG. 6 ) as a frame representing the start of a single play. Moreover, the long distance frame of the transition from the long distance frame to the close-up frame (for example, frame  90  in  FIG. 6 ) is extracted as a frame representing the end of a single play. Namely, a group of frames from the frame representing the start of the single play until the frame representing the end of the single play can be extracted as a group of frames representing the single play. 
         [0035]    The generation section  26  generates metadata indicating a break in play based on the group of frames extracted by the extraction section  24 . Specifically, the generation section  26  generates metadata that associates data indicating the start of a single play with time data associated with a frame representing the start of the single play. Moreover, the generation section  26  generates metadata associating data indicating the end of a single play with time data associated with a frame representing the end of the single play. 
         [0036]    The generation section  26  generates a metadata file that stores the plural generated metadata in the sequence of the time data included in the metadata. The metadata file may be generated as a file formatted as, for example, a csv (comma-separated values) format file. An example of a metadata file is illustrated in  FIG. 7 . In the example of  FIG. 7 , each row represents a single item of metadata, and each item of metadata includes time data  96 , and data representing either start or end of a single play (reference numeral  98 ). 
         [0037]    Note that although explanation is given here regarding a case in which the metadata is generated using time data associated with frames, the metadata may be generated using other data that identifies each frame, such as frame numbers. 
         [0038]    The edited footage is appended with the generated metadata file and transmitted to distribution device  30  as metadata appended footage by the generation section  26 . 
         [0039]    The extraction device  20  may, for example, be implemented by a computer  40  as illustrated in  FIG. 8 . The computer  40  includes a CPU  42 , memory  44 , a non-volatile storage section  46 , an input/output interface (I/F)  47 , and a network I/F  48 . The CPU  42 , the memory  44 , the storage section  46 , the input/output I/F  47 , and the network I/F  48  are mutually connected through a bus  49 . 
         [0040]    The storage section  46  may be implemented by a hard disk drive (HDD), flash memory, or the like. An extraction program  50  that causes the computer  40  to function as the extraction device  20  is stored in the storage section  46  that serves as a recording medium. The CPU  42  reads the extraction program  50  from the storage section  46 , expands the extraction program  50  into the memory  44 , and sequentially executes the processes included in the extraction program  50 . 
         [0041]    The extraction program  50  includes a detection process  52 , an extraction process  54 , and a generation process  56 . The CPU  42  operates as the detection section  22  illustrated in  FIG. 1  by executing the detection process  52 . The CPU  42  also operates as the extraction section  24  illustrated in  FIG. 1  by executing the extraction process  54 . The CPU  42  also operates as the generation section  26  illustrated in  FIG. 1  by executing the generation process  56 . The computer  40  executing the extraction program  50  thereby functions as the extraction device  20 . 
         [0042]    The distribution device  30  may be implemented by, for example, a computer  60  illustrated in  FIG. 9 . The computer  60  includes a CPU  62 , memory  64 , a non-volatile storage section  66 , an input/output I/F  67 , and a network I/F  68 . The CPU  62 , the memory  64 , the storage section  66 , the input/output I/OF  67 , and the network I/F  68  are mutually connected through a bus  69 . Moreover, a display device and an input device, not illustrated in the drawings, are connected to the computer  60  through the input/output I/F  67 . 
         [0043]    The distribution device  30  and the extraction device  20  are connected through the network I/F  68  of the distribution device  30 , the network, and the network I/F  48  of the extraction device  20 . 
         [0044]    The storage section  66  may be implemented by a HDD, flash memory, or the like. A distribution program  70  that causes the computer  60  to function as the distribution device  30  is stored in the storage section  66  that serves as a recording medium. The CPU  62  reads the distribution program  70  from the storage section  66  and expands the distribution program  70  into the memory  64 , and sequentially executes the processes included in the distribution program  70 . 
         [0045]    The distribution program  70  includes an editing process  72  and a distribution process  74 . The CPU  62  operates as the editing section  32  illustrated in  FIG. 1  by executing the editing process  72 . The CPU  62  also operates as the distribution section  34  illustrated in  FIG. 1  by executing the distribution process  74 . The computer  60  executing the distribution program  70  thereby functions as the distribution device  30 . 
         [0046]    The extraction device  20  and the distribution device  30  may each be implemented by, for example, a semiconductor integrated circuit, and more specifically by an application specific integrated circuit (ASIC) or the like. 
         [0047]    Explanation next follows regarding operation of the footage distribution system  10  according to the present exemplary embodiment. When the distribution device  30  is input with captured footage, the distribution device  30  executes editing processing illustrated in  FIG. 10 , and outputs edited footage. Next, when the extraction device  20  is input with the edited footage, the extraction device  20  executes extraction processing illustrated in  FIG. 11 , and outputs metadata appended footage. Then, when the distribution device  30  is input with the metadata appended footage, the distribution device  30  executes distribution processing illustrated in  FIG. 12 , and outputs distribution footage. Each type of processing is described in detail below. 
         [0048]    First, at step S 10  of the editing processing illustrated in  FIG. 10 , the editing section  32  acquires captured footage. Next, at step S 12 , the editing section  32  applies editing instructions instructed by operation of an operator using the display device and the input device, not illustrated in the drawings, to each frame of the video data included in the captured footage, using image processing. Next, at step S 14 , the editing section  32  transmits the edited footage to the extraction device  20 , and editing processing ends. 
         [0049]    Next, at step S 20  of the extraction processing illustrated in  FIG. 11 , the detection section  22  acquires the edited footage transmitted from the distribution device  30 . 
         [0050]    Next, at step S 22  the detection section  22  computes differences for each frame compared to the respective previous frame, so as to detect the leading frame and final frame of a cut based on frames for which the difference compared to the previous frame exceeds a specific threshold value. 
         [0051]    Next, at step S 24  the detection section  22  designates a group of frames, including from the leading frame until the final frame of the detected cut, as a group of frames representing a single cut. Then, the detection section  22  determines whether at least one frame included in the group of frames representing the single cut is a long distance frame, or a close-up frame. The detection section  22  detects a transition between a long distance frame and a close-up frame based on the determination result. 
         [0052]    Next, at step S 26  the extraction section  24  extracts a long distance frame of a transition from the close-up frame to the long distance frame as a frame representing the start of a single play. Moreover, the extraction section  24  extracts the long distance frame of a transition from a long distance frame to a close-up frame as a frame representing the end of a single play. Namely, the extraction section  24  extracts a group of frames from the frame representing the start of the single play until the frame representing the end of the single play as the group of frames representing the single play. 
         [0053]    Next, at step S 28 , the generation section  26  generates metadata that associates data indicating the start of the single play to time data associated with the frames representing the start of the single play extracted by the extraction section  24 . Moreover, the generation section  26  generates metadata that associates data representing the end of the single play to time data associated with the frames representing the end of the single play extracted by the extraction section  24 . 
         [0054]    Next, at step S 30 , the generation section  26  generates a metadata file storing the plural metadata generated at step S 28  above in the sequence of the time data included in the metadata. The generation section  26  then appends the generated metadata file to the edited footage, and transmits the edited footage to the distribution device  30  as metadata appended footage, and extraction processing ends. 
         [0055]    Next, at step S 40  of the distribution processing illustrated in  FIG. 12 , the distribution section  34  acquires the metadata appended footage transmitted from the extraction device  20 . Next, at step S 42 , the distribution section  34  converts the metadata appended footage to distribution footage according to the specified standards, and distributes the distribution footage to the distribution destination terminal (omitted from illustration in the drawings), and distribution processing ends. 
         [0056]    As explained above, according to the extraction device  20  of the present exemplary embodiment, from frames of video data included in captured footage of a soccer game, a transition between a long distance frame and a close-up frame is detected, and the transition is extracted as a break in play. This thereby enables extraction of a break in play from captured footage of a soccer game. 
         [0057]    In each of the exemplary embodiments above, easy location of respective scenes of single plays of sports game footage (the captured footage, or the edited footage) based on metadata is enabled when employing metadata appended footage, to which the metadata indicating the start and end of the extracted single play is appended. Moreover during footage distribution, for example, a supplementary service, such as transmission of email to a user, may be performed automatically coordinated with breaks in play based on the metadata. 
         [0058]    Although explanation has been given of examples in which a footage distribution system includes a distribution device and an extraction device in each of the exemplary embodiments above, there is no limitation thereto. Each of the functional sections of the distribution device, and each of the functional sections of the extraction device may be implemented by a single computer. 
         [0059]    Although explanation has been given of cases in which footage appended with metadata, generated by the extraction device, indicating the starts and ends of a single play is distributed by the distribution device in each of the exemplary embodiments above, there is no limitation thereto. For example, metadata appended footage may be saved as a large volume archive, and the respective single plays extracted and output based on the metadata. 
         [0060]    The output from the extraction device may also be employed in applications other than footage distribution systems. 
         [0061]    Although explanation has been given above of modes in which the extraction program  50  is pre-stored (installed) on the storage section  46 , and the distribution program  70  is pre-stored (installed) on the non-volatile storage section  66 , this may be provided in a format recorded on a recording medium such as a CD-ROM or a DVD-ROM. 
         [0062]    One aspect exhibits the advantageous effect of enabling extraction of breaks in play from captured footage of a soccer game. 
         [0063]    All examples and conditional language provided herein are intended for the pedagogical purposes of aiding the reader in understanding the invention and the concepts contributed by the inventor to further the art, and are not to be construed as limitations to such specifically recited examples and conditions, nor does the organization of such examples in the specification relate to a showing of the superiority and inferiority of the invention. Although one or more embodiments of the technology disclosed herein have been described in detail, it should be understood that the various changes, substitutions, and alterations could be made hereto without departing from the spirit and scope of the invention.