Abstract:
A method and apparatus for encoding media content. In the field of video compression, temporal compression generates I-frames and dependent P-frames and B-frames. The placement of I-frames is determined by the presence of scene switches in the video. In order to improve the ability of a user to seek to interesting parts of the video, a video encoder considers semantic data identifying narrative points of interest within the video. The encoding process generates encoded video data having I-frames present at scene transitions and also at narratively interesting parts of the video.

Description:
TECHNICAL FIELD 
       [0001]    The present invention relates to media encoding and in particular to an apparatus for and method of encoding video content making use of semantic data. 
       BACKGROUND 
       [0002]    In recent years, digital media has become a commonplace carrier for delivering information to users. In particular, digital video allows users to obtain information through visual and audio means. 
         [0003]    In its most basic form, digital video is composed of a sequence of complete image frames which are played back to the user at a rate of several frames per second. The quality of the video depends on the resolution of each frame, and also the rate at which frames are displayed. Higher resolution means that more detail can be included in each frame whilst higher frame rates improve the user&#39;s perception of movement in the video. 
         [0004]    Increasing quality of video content results in larger file sizes which is undesirable in many applications. Encoding techniques, and in particular video compression techniques are known which aim to reduce file sizes while minimizing any loss in quality of the video. Video compression techniques generally fall into two groups: spatial compression and temporal compression, with many common video compression formats using a combination of both techniques. 
         [0005]    Spatial compression involves applying compression to each individual image frame, for example in a manner similar to JPEG compression for still images. 
         [0006]    Temporal compression exploits similarities in sequences of consecutive frames to reduce the information storage requirements. In many videos, significant parts of the scene do not change over time. In this case, the scene information from a previous scene can be re-used for rendering the next scene while only information relating to the changed pixels is stored. This can result in significant reductions in file size. Similarly, where the camera pans across a scene, a significant portion of the new frame is identical to the previous scene but offset in the direction of the pan. In this case only the newly viewable pixels would need to be encoded. 
         [0007]    In a video compression such as MPEG-2, complete information frames are called Full Frames or I-frames (Independent frames). These frames are independent of other frames and can therefore be decoded without referring to any information in any other frames of the video. The main compression savings are made by converting the uncompressed video frames into dependent frames. These are frames which are dependent on some information from an adjacent frame in order to be successfully decoded. Dependent frames which are dependent on preceding frames are called Predictive Frames or P-Frames and frames which are dependent on both preceding and following frames are known as B-frames. 
         [0008]    Whilst use of I-frames, P-frames and B-frames provides valuable file size savings, temporal compression techniques can inconvenience the user&#39;s viewing experience. For example, a user may wish to skip to a specific position in the file and begin playback from that position instead of watching the entire video in order. 
         [0009]    If an I-frame is located in the video file at the user&#39;s selected position, then playback can begin from the selected position. However, if an I-frame is not present at the desired location, then in most cases, the video decoder will seek to the nearest I-frame location. The user must then wait for the desired segment of the video file to be played. 
         [0010]    One known way to address the above problem is to insert more I-frames into the compressed video file. In addition to I-frames located at the scene switching points, I-frames are inserted at regular intervals, for example every second, or every 20 frames so that the granularity of the video segments is improved. However, the presence of more I-frames increases the file size of the video. 
         [0011]    The present invention addresses the above problems. 
       STATEMENTS OF INVENTION 
       [0012]    In one aspect the present invention provides a method of encoding media content into a sequence of independent data frames and dependent data frames, the method comprising: analysing the media content to determine where scene changes occur within the media content; generating encoding parameters defining the location of said scene changes; accessing data indicating semantically significant sections of the media content; and updating the encoding parameters so that independent data frames are present at locations indicated by the semantic data. 
         [0013]    In another aspect, the present invention provides an apparatus for encoding media content into a sequence of independent data frames and dependent data frames; the apparatus comprising: means for analysing the visual content of the media content; a configuration data store indicating the location of scene changes in the media content; accessing means for accessing data indicating semantically significant sections of the media content; means for updating the configuration data store to include full-frames at locations indicated by the semantic data. 
         [0014]    In a further aspect, the present invention provides an encoded media file formed of a sequence of independent data frames and dependent data frames, the independent media frames being located at semantically significant parts of the media file. 
         [0015]    Other preferable features are set out in the dependent claims. 
     
    
     
         [0016]    Embodiments of the present invention will now be described with reference to the accompanying figures in which: 
           [0017]      FIG. 1  shows the functional components in an encoding system according to a first embodiment; 
           [0018]      FIG. 2  shows the main functional units within the system illustrated in  FIG. 1 ; 
           [0019]      FIG. 3  shows a flow chart of the operation of the video encoder shown in  FIG. 2 ; 
           [0020]      FIG. 4  schematically shows a coded video file which has been processed in the first pass of the video encoding process to identify scene changes; 
           [0021]      FIG. 5  illustrates the location of the scene change I-frames and the semantic I-frames within the sample video file; 
           [0022]      FIG. 6  shows the contents of the narration data file; 
           [0023]      FIG. 7  shows the functional components of the system in the third embodiment; 
           [0024]      FIG. 8  shows a compressed video file having extra I-frames located after each of the origin I-frames; and 
           [0025]      FIG. 9  shows the contents of a narration data file according to the fifth embodiment. 
       
    
    
     DESCRIPTION 
       [0026]    In the first embodiment, the encoding system processes uncompressed video files to generate corresponding compressed video files having I-frames located at scene changes within the video and P-frames or B-frames for other frames as is conventional. Additionally, the encoder uses semantically significant data such as narrative information to add further I-frames at positions within the video which are not scene changes but are narratively significant. 
         [0027]      FIG. 1  shows the main components in an encoding system  1  according to a first embodiment. The system comprises a central processor  3 , working memory  5 , a data store  7 , a display interface  9  and a display  11 , all connected via a common system bus  13 . The system  1  also includes a dedicated video encoding processor  15  connected to the system bus. 
         [0028]      FIG. 2  shows the main functional units within the system illustrated in  FIG. 1 . The video encoder  15  receives an input video file  21  from the data store  7 . The input video file is in a temporally uncompressed format such that each frame is fully independent of other frames. After encoding, the video encoder  15  stores an output encoded compressed video file  23  in the data store  7 . The video encoder  15  also receives data from a semantically significant data processing unit  25  which is connected to a narration data file  27  which is also stored within the data store  7 . 
         [0029]      FIG. 3  shows a flow chart of the operation of the video encoder shown in  FIG. 2 . In this embodiment the video encoder  17  uses a two-pass video encoding method to generate a compressed video file  23 . 
         [0030]    At step s 1 , the encoder  15  accesses the uncompressed video file  21 . In step s 3 , the encoder  15  performs a first pass of the accessed video  21  to identify where scene changes occur. The locations of the scene changes within the video file  21  are stored in a configuration file  29  stored in the working memory  5 . In this embodiment, the video encoder  15  stores the frame number of each frame where a scene change occurs. For example: 
         [0031]    Frame  0 ; 
         [0032]    Frame  56 ; 
         [0033]    Frame  215 ; 
         [0034]    Frame  394 ; 
         [0035]    Frame  431 ; 
         [0036]    Frame  457 ; 
         [0037]    Frame  1499 . 
         [0038]      FIG. 4  schematically shows a coded video file  23  which has been processed in the first pass of the video encoding process to identify scene changes corresponding to the above example configuration file. Segments within the video file  23  which are based on the same scene are shaded. As a result of the first pass of the video encoding process in step s 3 , the configuration file contains details of where each I-frame is required in the output video file  23 . In this embodiment, the configuration file  29  contains a list of frame numbers where a scene transition occurs. For ease of explanation, in this embodiment, only a single I-frame is created for each scene portion. 
         [0039]    Returning to  FIG. 3 , after the first pass, in step s 5 , the video encoder  15  accesses a semantically important processing unit, which in turn accesses a narration data file  27  in order to identify where additional I-frames are required. At step s 7 , these extra I-frame locations are inserted into the configuration file  29 .  FIG. 5  illustrates the location of the scene change I-frames  31  and the semantic I-frames  33  within the sample video file. 
         [0040]    As shown in  FIG. 3 , once the configuration file has been updated, at step s 9 , the video encoder performs a second pass process to generate a compressed video file in a convention manner. Finally at step s 11 , the video encoder outputs a compressed video file having I-frames  31 ,  33  at locations indicated by the configuration file  29 , with P-frames and B-frames for other frame locations. The output compressed video file  23  is stored in the data store  7 . 
         [0041]    The narration data file  27  and the generation of the narration data file  27  will now be explained.  FIG. 6  shows the contents of the narration data file  27 . The narration data file  27  is very similar to the configuration file  29  generated by the video encoder  15 . The narration data file has a creator field  45  for the creator of the file to identify themselves, and title field  47  to identify the video file  21  to which the narration data file  27  relates. Each entry in the narration data file  27  is a frame number  41  and corresponding time stamp  43  indicating where an additional I-frame is to be present in the compressed video file. Additionally, each entry in the narration data file  27  may have an associated comment  49  indicating the significance of the frame. 
         [0042]    In this embodiment, the narration data file  27  is generated by the user who produces the uncompressed video input  21 . The producer carries out a manual process to mark the start of segments of the video which may be of narrative interest to any end users who view the final video. Examples of narrative interests include: the start of speech by a certain actor, the start of an action sequence, the start of a musical piece etc. The points of narrative interest are not limited to events in the audio track but also include visual events which do not cause a change of scene. For example, a motion freeze, or an actor walking into the scene. 
         [0043]    In the first embodiment, a compressed video file corresponding to an input uncompressed video file is generated having additional I-frames at locations where the video producer has manually specified segments of particular interest. In this way, a user who later views the compressed video has the ability to seek to particularly interesting parts of the video. 
       Second Embodiment 
       [0044]    In the first embodiment, the video encoder produces compressed MPEG-2 videos from an input uncompressed video file using a two-pass encoding scheme. In the second embodiment, the video encoder compresses the input video file using a single pass encoding scheme. 
       Third Embodiment 
       [0045]    In the first and second embodiments, the video encoder processed uncompressed video input and produced compressed video data having I-frames placed in accordance with segments of interest as determined by the video producer as well as the conventional placement of I-frames based on scene changes. 
         [0046]    In the third embodiment, the system allows I-frames representing points of narrative interest to be added to video files which are already compressed. This is useful in cases where a part of the video only becomes of narrative interest once it has been made available for a length of time. 
         [0047]      FIG. 7  shows the functional components of the system in the third embodiment. The physical components are the same as the first embodiment and will not be described again. A compressed video file  51  is processed by the video encoder  53  to produce a new encoded and compressed video file  55 . In processing the input video file  51 , the video encoder  53  builds a configuration file  57  which specifies the location of existing I-frames. It also receives narration data file  61  information via a semantic processing unit  59 . After re-encoding, the new compressed video file  55  contains additional I-frames at locations specified in the narration data file  61  information. In this case, the new video file will be larger than the old compressed video data file because it contains more I-frames. 
       Fourth Embodiment 
       [0048]    In the above embodiments, the video encoder is arranged to produce compressed video files having I-frames located at scene transitions within the video, and also at locations specified in a narration file defined by the producer of the video, or any user who wishes to add I-frames to a video. 
         [0049]    In many videos, when a scene change occurs, there will often be a slight time delay between the start of the new segment of the video and the start of any video content which is narratively significant. In later video editing tasks, for example to merge separate videos, or extracts from a single video into a composite video, it is desirable to filter out the narratively unimportant content. 
         [0050]    In the fourth embodiment, the encoder is further operable to insert additional I-frames after either a scene change I-frame, or a semantic I-frame.  FIG. 8  shows a compressed video file  71  having extra I-frames  73  located after each of the original I-frames  75  and narratively important  77  I-frames. 
       Fifth Embodiment 
       [0051]    In the first to fourth embodiments, the encoder is arranged to insert I-frames at specified locations in the video according narratively important video content in a video file as set out in the narration data file  81 . 
         [0052]    In the fifth embodiment, in addition to inserting I-frames, the video encoder is arranged to emphasize the semantically important sections or an input video by changing the quality of the output video for frames following the inserted I-frame. 
         [0053]    The physical and functional components of the encoding system in the fifth embodiment are similar to those of the previous embodiments, the only differences being in the narration data file  81  and the encoder. 
         [0054]      FIG. 9  shows the contents of a narration data file  81  according to the fifth embodiment. The narration file is similar to the narration data file of previous embodiments, it has a field for the creator&#39;s name  83 , the name of the intended video  85 , the frame number  87  and the time index  89  corresponding to that segment of video and a comment  91 . The narration data file  83  also includes a duration field  93  information specifying the duration of the narratively significant content, in this case the number of frames spanning the narratively significant event. 
         [0055]    The encoder receives information from the narration data file  81  and in response, inserts I-frames and also increases the number of bits allocated to encoding (hereinafter referred to as the bit rate) the sections of the video which are narratively important. 
         [0056]    In some cases, the narratively important sections will coincide with sections which the video encoder would normally consider to require more bits. However, in other situations, for example where a particularly important speech is being delivered but the video background is not changing significantly, the video encoder will allocate a higher bit rate to the sections defined in the narration file. If there are restraints on the allowable bit rate for the video of file size, then the video encoder will allocate more bit rate to the narratively important sections and set a lower bit rate for other parts of the video. 
       Alternatives/Modifications 
       [0057]    In the embodiments, the video encoder generated MPEG2 video files. It will appreciated that any compression format performing temporal frame&#39; compression could be used. For example, WMV or H.264. 
         [0058]    In the first embodiment, the encoder uses a 2-pass encoding scheme. In an alternative, a multi-pass encoding scheme is used. For example a three or four pass encoding scheme. 
         [0059]    In the embodiments, the narration file is manually generated by a user of the system. In an alternative, the narration file is generated without user action. An audio processor analyses the audio stream within the video file to determine when speech occurs and populates the narration file.