Abstract:
A method and device of editing video data are provided for outputting video data with good quality. When some unimportant data or data with poor quality are embedded within a video signal, they would be sifted from the video signal with a trimming or dropping step during editing. The descriptors charactering the video signal are acquired and applied on the trimming or dropping for outputting the video data with good quality.

Description:
FIELD OF THE INVENTION  
       [0001]     The invention relates generally to computer generation of video production. In particular, the invention relates to automatic editing of video production.  
       BACKGROUND OF THE INVENTION  
       [0002]     With the increasing use of video and storage of events and communication via video, video users and managers are confronted with additional tasks of storing, accessing, determining important scenes or frames, and summarizing videos in the most efficient manner.  
         [0003]     In general, techniques exist to automatically segment video into component shots of a video or motion image, typically by finding the large frame differences that correspond to cuts, or shot boundaries. In many applications it is desirable to automatically create a summary or “skim” of an existing video, motion picture, or broadcast. This can be cone by selectively discarding or de-emphasizing redundant information in the video. For example, repeated shots need not be included if they are similar to shots already shown.  
         [0004]     For example, for video summarization, video is partitioned into segments and the segments are clustered according to similarity to each other. The segment closest to the center of each cluster is chosen as the representative segment for the entire cluster. Other video summarization approaches attempt to summarize video using various heuristics typically derived analysis of closed captions accompanying the video. These approaches rely on video segmentation, or require either clustering or training.  
         [0005]     However, some other tools built for browsing the content of a video are known, but only provide inefficient summarization or merely display a video in sequence “as it is”.  
       SUMMARY OF THE INVENTION  
       [0006]     A method and device of editing video data is provided for outputting video production with an easy way. An automatically video construct technology can help users to create video output easily.  
         [0007]     A video outputting with better video quality is provided. With trimming some frames or dropping some shots, each video segment is acquired with good qualities and quantities of the frames or shots.  
         [0008]     A method and device of editing video data to generate video production is provided. With dropping some segments, the video data output with the segments with good qualities.  
         [0009]     Accordingly, one embodiment of the present invention provides a method and device of editing video data for outputting video data with good quality. When some unimportant video segments or frames with poor quality are embedded within a video signal, they would be sifted from the video signal with a dropping or trimming step during editing. The descriptors charactering the video segments and weights based on these descriptors are acquired and applied on the trimming or dropping for outputting the video data with good quality. 
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0010]     The foregoing aspects and many of the attendant advantages of this invention will become more readily appreciated as the same becomes better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings, wherein:  
         [0011]      FIG. 1  is a schematic flow chart illustrating one embodiment in accordance with the present invention;  
         [0012]      FIG. 2  is a schematic block diagram illustrating video data editing system of one embodiment in accordance with this invention; and  
         [0013]      FIG. 3  is a diagram illustrating the video segment versus corresponding segment score in according with the invention. 
     
    
     DETAILED DESCRIPTION OF THE EMBODIMENTS  
       [0014]     Referring to  FIG. 1 , Input signals  20  include one or more pieces of media, which is presented as an input to the system. Supported media types, without limitation, include video, image, slideshow, animation and graphics.  
         [0015]     Video analyzer  11 , extracts the information embedded in media content, like time-code, duration of media, and measures the rate of change and statistical properties of other descriptors, descriptors derived by combining two or more other descriptors, etc. For example, video analyzer  11  measures the probability that a segment of the input video contains a human face, probability that it is a natural scene, etc. In short, video analyzer  11  receives input signals  20  and outputs data with associated descriptors, which describes characteristics of input signals  20 .  
         [0016]     In one embodiment, the data with the associated descriptors are utilized in the next steps in sifting process  12 . First, multitudes of weights are determined based on the associated descriptors. Second, for the acquirement of video production  30  with good quality, the data are adjusted based on at least one of the associated descriptors and weights. Third, the adjusted data are constructed for a video production  30 . All blocks are described in detail as follows.  
         [0017]      FIG. 2  is a schematic block diagram illustrating video data editing system of one embodiment in accordance with this invention. First, the video data editing system  10  receives video input signals  20  and playback control  40 , and generates video production  60 . The term “video input signal” refers to input signal of any video type including video, slideshow, image, animation, and graphics, and inputs as a digital video data file in any suitable standard format, such as DV video format. In an alternate embodiment, an analog video input signal may be converted into a digital video input signal used in the method.  
         [0018]     In one embodiment, video input signals  20 , without limitation, include video input  201 , sideshow  202 , image  203 , etc. In the embodiment, video input  201  is typically unedited raw footage of video, such as video captured from a camera or camcorder, motion video such as a digital video stream or one or more digital video files. Optionally, it may include an audio soundtrack. In the embodiment, the audio soundtrack, such as people dialogue, is recorded simultaneously with video input  201 . Slideshow  202  refers to a video signal including an image sequence, background music and property. Images  203  are typical still images such as digital image files, which are optionally used in addition to motion video.  
         [0019]     In addition to video input signals  20 , other constrains, such as playback control  40 , may be inputted into video data editing system  10  for video production  60  with good quality.  
         [0020]     Next, video data editing system  10  includes video analyzer  11  and sifting process  12 . In one embodiment, video analyzer  11  is configured for generating analyzed data and descriptors  14  by analyzing video input signals  20 . Furthermore, video analyzer  11  is configured for segmenting video input signals  20  according to video descriptors thereof. Video input signals  20  are first parameterized by any typical methods, such as frame-to-frame pixel difference, color histogram difference, and low order discrete cosine coefficient difference. Then video signals  20  are analyzed for acquiring analyzed video data and associated descriptors.  
         [0021]     Typically, various analysis methods to detect segment boundary are used in video analyzer  11 , such as scene change detection, checking similarity of video frames, segments, such as over-exposure, under-exposure, brightness, contrast, video stabilization, motion estimation etc., and determining the importance of video segments, checking skin color and detecting faces, flash (camera flash), dialog attached with video-content, face recognition etc. The analyzed descriptors in video analyzer  11  include typically measures of brightness or color such as histograms, measures of shape, or measures of activity. Furthermore, the analyzed descriptors include durations, qualities, importance and preference descriptors for the analyzed video data. Alternatively, soundtrack derived from the video input  201  can be used as a descriptor for further process. Then, the segmentation performed by video analyzer  11 , for example, is based on scene change detection, camcorder shooting time, or turn on/off from camcorder to improve video segmentation result and generates one or more video segments. The video segment is a sequence of video frames or a part of a clip that is composed one or more shots or scenes.  
         [0022]     It is noted that video input signals  20  with MPEG-7 format contain some video descriptions, such as measures of color including scalable, color layout, dominant color, and measure of motion including motion trajectory and motion activity, camera motion and face recognition, etc. With the descriptions derived from one file in MPEG-7 format, such video input signals  20  may be used for further process, instead of process of video analyzer  11 . Accordingly, the descriptions derived from the file in MPEG-7 format would be used as analyzed video descriptors mentioned in the following processes.  
         [0023]     Next, analyzed data and associated descriptors  14  output to sifting process  12  for determining multitudes of weights, adjusting analyzed data and constructing adjusted data. In one embodiment, without limitation, analyzed data include multitudes of segments, and sifting process  12  includes weighting unit  121 , trimming unit  122 , dropping unit  123  and timeline constructor unit  124 .  
         [0024]     In weighting unit  121 , multitudes of weights (“Wi” for descriptor “i”) are determined with some associated descriptors. In the embodiment, weighting unit  121  determines or assigns one descriptive score such as “frame-based” score (“S(Vi)” for descriptor “i”) to individual associated descriptor related to frames in each analyzed data, without limitation, such as those analyzed descriptors acquired by checking similarity of video frame, dialog analysis or face detection. For example, with face detection for one analyzed data such as one video segment, one or more associated face-characteristic descriptors are assigned or acquired higher scores (“S(Vi)”), respectively. Thus, within one video segment, some frames with more face-area have priorities for video production  60 . On the other hand, weighting unit  121  also determines or assigns another descriptive score such as “segment-based” score to individual associated descriptor related to one analyzed data, without limitation, such as those analyzed descriptors acquired by analyzing video quality, analyzing unsteady segments or face detection. For example, with face detection for analyzed data such as some video segments, one or more associated face-characteristic descriptors are assigned or acquired higher scores (“S(Vi)”), respectively. Thus, within one video signal, one or more video segments with more face-area have priorities for video production  60 .  
         [0025]     Alternatively, with an “attention” curve, weighting unit  121  matches one “duration-based” score for each analyzed data, such as each video segment. In general, when users are trying to capture the attention of an audience, it&#39;s often easier to give them a lot of short video clips instead of attempt to appeal to their artsy side with long, drawn out shots of over 2 minutes long apiece. Shots of 5 to 8 seconds duration often work very well. Thus, in weighting unit  121 , high “duration-based” score is assigned to one analyzed data such as one video segment with segment duration of 5 to 8 seconds. It is understandable one video segment with segment duration too short or too long will acquire lower “duration-based” score. Accordingly, weighting unit  121  determines or assigns scores to the associated descriptors, in which these scores express quality-related or duration-related characteristics for the analyzed data.  
         [0026]     Next, trimming unit  122  is configured to adjust one video segment. Basically, one video segment is adjusting by trimming (excluding) some frames within the video segment. Such adjustment is implemented based on one or more associated descriptors with their “frame-based” scores (“S(Vi)”). In the embodiment, the associated descriptors with their frame-based scores are usually characteristics related to multitudes of frames within the video segment. For one video segment, some frames or clips are trimmed based on the associated descriptors with lower “frame-based” scores. Thus, with trimming adjustment, one video segment consists of frames with good qualities. Furthermore, the trimmed video segment may have a trimmed segment duration different from the original video segment duration. In an alternative embodiment, some frames or shots are trimmed due to constraints by playback control  40 .  
         [0027]     For example, with using soundtrack as a descriptor in trimming unit  122 , some sequential frames, especially in the midst of one “dialog” segment, are with higher “soundtrack” scores, individually. On the other hand, some frames, especially at the beginning or end of the “dialog” segment, are with lower “soundtrack” scores, individually. The frame where the introduction of the soundtrack is can be marked as the beginning of trimming “trim in” , and the frame where the completion of the soundtrack is can be marked as the ending of trimming “trim out”. Those frames positioned between “trim in” and “trim out” are retained. Thus, the frames positioned at the beginning or end of the “dialog” segment will be trimmed in trimming unit  122 . It is noted that a trimmed range for those marked trimmed frames is applied while multitudes of “frame-based” scores are considered. It is due to those marked trimmed frames may be different based on different associated descriptors with “frame-based” scores. Thus, with adjustment of the trimmed range, some marked trimmed frames are determined to trim out.  
         [0028]     On the other hand, in dropping unit  123 , the video segments, with or without frame-based adjustment, can be adjusted based on the associated descriptors with “segment-based” scores, the “duration-based” scores, playback control  40 , or all of them. Dropping unit  123  is configured to adjust some video segments of the analyzed data. Basically, one video segment is wholly dropped (excluded) in dropping unit  123  on the ground that there are the associated descriptors with the lower “segment-based” scores, the lower “duration-based” scores, or both of them.  
         [0029]     In one embodiment, “segment-based” scores are further multiplied by quality-related weights, respectively, and further summarized to acquire one “quality-related” score for each video segment as follows:  
         S   ⁡     (   Qj   )       =       ∑     i   =   1     N     ⁢       Sj   ⁡     (   Vi   )       *   Wi           
 
         [0030]     Where “N” is the total number of descriptors; “i” represents descriptor index; “Vi” is a segment “j” with descriptor “i”; “Wi” represents a quality-related weight for descriptor “i”; “Sj(Vi)” is score of descriptor “i”for one segment “j”; and “S(Qj)” is one “quality-related” score for each video segment “j”.  
         [0031]     Then, multiplied by content-based weight and duration-based weight, respectively, the “quality-related” score and “duration-based” score are summarized to acquire one segment score for each video segment as follows: 
 
 Sj=W ( Q )* S ( Qj )+ W ( T )* S ( Tj ) 
 
         [0032]     Where “S(Tj)” is the original segment duration or a trimmed segment duration for each video segment; “W(T)” means the duration-based weight; and “W(Q)” represents the content-based weight.  
         [0033]     Shown in  FIG. 3 , clip  30  is divided into video segments  301 , 302 , 303 , clip  32  into video segments  321 , 322 , 323 , and clip  34  into video segments  341 , 342 , 343 , 344 . Each video segment has a segment score (Sj). In dropping unit  123 , with a score threshold  35 , some video segments will be dropped, such as video segments  321  and  323 . Accordingly, each segment score for each video segment is characterized by the “quality-related” score and “duration-based” score. Thus, one video segment with higher segment score plays one more important portion for the video production  60 . It is understandable that one video segment with relative lower segment score may be dropped in dropping unit  123 .  
         [0034]     Alternatively, it is noted that the number of dropped video segments is also dependent on a production duration related to the video production  60 . When the summed total duration of the video segments exceeds the production duration, the video segments with relative lower segment scores should be dropped. When the summed total duration of the video segments is less than the production duration, one or more video segments with relative higher segment scores may be repeated to meet the production duration. However, when the summed total duration is near to the production duration, the trimming step may be implemented within any one video segment to adjust the individual duration of one video segment. Additionally, the number of dropped video segments is also just dependent on qualities of the video production  60  without consideration of the predetermined production duration. That is, the summed total duration of the video segment after dropping in view of video qualities is acceptable, when user would like to show up the good quality video production, and do not mind the finial video production duration. Although both of production duration and quality constrain to produce the finial video production is workable.  
         [0035]     Next, the adjusted data output to timeline constructor unit  124  for outputting video production  60 . Timeline constructor unit  124  is configured for constructing the adjusted video data in sequence. Optionally, Timeline constructor unit  124  constructs video data with playback control  40 .  
         [0036]     Normally, video production  60  would be directly viewed and run by users. Of course, with style information template  50 , video production  60  would input into render unit  70  for post processing. In the embodiment, style information  50  is a defined project template, without limitation, which includes descriptors as follows: filters, transition effects, transition duration, title, credit, overlay, beginning video clip, ending video clip, and text.  
         [0037]     It will be clear to those skilled in the art that the invention can be embodied in many kinds of hardware device, including general-purpose computers, personal digital assistants, dedicated video-editing boxes, set-top boxes, digital video recorders, televisions, computer games consoles, digital still cameras, digital video cameras and other devices capable of media processing. It can also be embodied as a system comprising multiple devices, in which different parts of its functionality are embedded within more than one hardware device.  
         [0038]     Although the invention has been described above with reference to particular embodiments, various modifications are possible within the scope of the invention as will be clear to a skilled person.