Abstract:
A method and encoder for compressing a video signal inserts at least one non-referenced frame to coincide with a duration of a flash on detecting the flash in the video signal. On detecting a fade in the video signal, an order in which frames are coded is changed such that referenced frames are positioned immediately before and after the fade and a pyramidal structure of bidirectionally coded frames are used for the duration of the fade.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
       [0001]    This application claims priority from GB 0703889.6 filed Feb. 28, 2007. 
       FIELD OF THE INVENTION 
       [0002]    This invention relates to compression of video signals containing fades and flashes. 
       BACKGROUND OF THE INVENTION 
       [0003]    Motion-compensated predictive video compression exploits a similarity of successive pictures by making predictions from previously coded pictures. The pictures from which the predictions are taken can come from the past (forward prediction) or the future (backward prediction) or a combination of the two (bi-directional prediction) thus enabling the prediction of uncovered areas. Therefore, state-of-the-art video compression engines can cope with most types of motion and critical picture material. 
         [0004]    However, there are two types of effects which can cause severe picture degradation and are worthy of special mitigating treatment: video sequences that include fades and video sequences that contain short, bright flashes such as those that occur when a still camera with a flash apparatus is used in the field of view. During fades a motion estimator often produces random motion vectors due to video luminosity changes. Similarly, predictions taken from pictures with flashes such as camera flashes are poor indications of video behaviour and lead to poor compression performance. 
       SUMMARY OF THE INVENTION 
       [0005]    According to a first aspect of the present invention there is provided a method of compressing a video signal comprising: on detecting a flash in the video signal, inserting at least one non-referenced frame to coincide with a duration of the flash; and on detecting a fade in the video signal, changing an order in which frames are coded such that referenced frames are positioned immediately before and after the fade and a pyramidal structure of bidirectionally coded frames are used for the duration of the fade. 
         [0006]    Preferably, quantisation is increased in a frame in which a flash is detected. 
         [0007]    Preferably, on detection of a fade a search range of a motion estimator is reduced for the duration of the fade. 
         [0008]    Advantageously, detecting a flash in the video signal comprises the steps of: calculating a first average luminance of fields of at least one frame immediately preceding a frame of interest and a second average luminance of fields of at least one frame immediately succeeding the frame of interest; calculating whether the luminance of a top field of the frame of interest exceeds the greater of the first average luminance and the second average luminance by a first predetermined threshold and if so signalling that a flash occurred in the top field of the frame of interest; and calculating whether the luminance of a bottom field of the frame of interest exceeds the greater of the first average luminance and the second average luminance by the first predetermined threshold and if so signalling that a flash occurred in the bottom field of the frame of interest. 
         [0009]    Conveniently, detecting a flash in the video signal comprises the steps of: calculating a first average luminance of the fields of the current frame; calculating a second average luminance of the fields of a frame preceding the current frame by two frames; calculating whether the luminance of a bottom field of the frame preceding the current frame exceeds a greater of the first average luminance and the second average luminance by a first predetermined threshold and if so signalling that a flash occurred in the bottom field of the previous frame; and calculating whether the luminance of a top field of the frame preceding the current frame exceeds a greater of the first average luminance and the second average luminance by a first predetermined threshold and if so signalling that a flash occurred in the top field of the previous frame. 
         [0010]    Advantageously, detecting a fade in the video signal comprises the steps of: calculating an average luminance for each of a first plurality of successive fields; calculating successive differences in the average luminance between each of the first plurality of successive fields; calculating a sum of the successive differences; calculating an average of a second plurality of the sum and such sums of differences of immediately preceding fields; calculating an absolute difference between the sums and their average; and signalling that a fade is detected if the average is greater than a second predetermined threshold and if each of the absolute differences is less than a third predetermined threshold. 
         [0011]    Conveniently, detecting a fade in the video signal comprises the steps of: calculating an average luminance for each of four successive fields; calculating successive differences in the average luminance between each of the four fields; calculating a sum of the successive differences; calculating an average of the sum and two such sums for successive differences of immediately preceding sets of fields; calculating an absolute difference between the three sums and their average; and signalling that a fade is detected if the average is greater than a second predetermined threshold and if each of the absolute differences is less than a third predetermined threshold. 
         [0012]    According to a second aspect of the invention, there is provided an encoder for a video signal comprising: compensating delay means, fade detector means and flash detector means arranged such that a video signal may be input in parallel to the compensating delay means, the fade detector means and the flash detector means; frame re-ordering means having an input connected to an output of the compensating delay means; coding mode means having inputs from the fade detector means and the flash detector means to output a control signal to the frame re-ordering means; compression coding loop means having inputs from the frame re-ordering means and the coding mode means; motion estimator means having inputs from the frame re-ordering means and the coding mode means to output a motion vector to the compression coding loop means and to input a reconstructed video signal from the compression coding loop means; and entropy coding means to receive an input from the compression coding loop and to output a compressed video signal; wherein the frame re-ordering means is arranged such that: on detection of a flash in the video signal by the flash detector means, at least one non-referenced frame is inserted to coincide with a duration of the flash; and on detection of a fade in the video signal by the fade detector means, an order in which frames are coded is changed such that referenced frames are positioned immediately before and after the fade and a pyramidal structure of bidirectionally coded frames are used for the duration of the fade. 
         [0013]    Advantageously, the fade detector means is arranged to: calculate an average luminance for each of a first plurality of successive fields; calculate successive differences in the average luminance between each of the first plurality of fields; calculate a sum of the successive differences; calculate an average of a second plurality of the sum and such sums of differences of immediately preceding fields; calculate an absolute difference between the sums and their average; and signal that a fade is detected if the average is greater than a second predetermined threshold and if each of the absolute differences is less than a third predetermined threshold. 
         [0014]    Conveniently, the fade detector means is arranged to: calculate an average luminance for each of four successive fields; calculate successive differences in the average luminance between each of the four fields; calculate a sum of the successive differences; calculate an average of the last three sums; calculate an absolute difference between the three sums and their average; and signal that a fade is detected if the average is greater than a second predetermined threshold and if each of the absolute differences is less than a third predetermined threshold. 
         [0015]    Advantageously, the flash detector means is arranged to: calculate a first average luminance of fields of at least one frame immediately preceding a frame of interest and a second average luminance of fields of at least one frame immediately succeeding the frame of interest; calculate whether the luminance of a top field of the frame of interest exceeds the greater of the first average luminance and the second average luminance by a first predetermined threshold and if so to signal that a flash occurred in the top field of the frame of interest; and calculate whether the luminance of a bottom field of the frame of interest exceeds the greater of the first average luminance and the second average luminance by the first predetermined threshold and if so to signal that a flash occurred in the bottom field of the frame of interest. 
         [0016]    Conveniently, the flash detector means is arranged to: calculate an average luminance of each field for five field periods; calculate a first average luminance of fields of a frame immediately preceding a frame of interest and a second average luminance of fields of a frame immediately succeeding the frame of interest; calculate whether the luminance of a top field of the frame of interest exceeds the greater of the first average luminance and the second average luminance by a first predetermined threshold and if so to signal that a flash occurred in the top field of the frame of interest; and calculate whether the luminance of a bottom field of the frame of interest exceeds the greater of the first average luminance and the second average luminance by the first predetermined threshold and if so to signal that a flash occurred in the bottom field of the frame of interest. 
         [0017]    Preferably, the compression coding loop is arranged to increase quantisation in a frame in which a flash is detected. 
         [0018]    Preferably, the motion estimation means is arranged to reduce a search range on detection of a fade. 
         [0019]    According to a third aspect of the invention, there is provided a computer program product comprising program code means arranged to perform all the steps of the method described above when that program code means is run on a computer. 
         [0020]    According to a fourth aspect of the invention, there is provided a computer-readable medium embodying a computer program product as described above 
         [0021]    Other aspects and features of the present invention will become apparent to those ordinarily skilled in the art upon review of the following description of specific embodiments of the invention in conjunction with the accompanying figures. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS  
         [0022]    Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which: 
           [0023]      FIG. 1  is illustrates a coding structure with referenced B pictures and non-referenced B pictures; 
           [0024]      FIG. 2  illustrates pyramid B picture coding; 
           [0025]      FIG. 3  is a block diagram of an encoder according to the invention with fade and flash detectors; 
           [0026]      FIG. 4  is a coding structure for a video sequence containing a camera flash; 
           [0027]      FIG. 5  is a flowchart of a method according to the invention of detecting a flash or fade in a video signal and adapting encoding of the video signal accordingly; 
           [0028]      FIG. 6  is a flowchart of the step of detecting a fade of the method of  FIG. 5 ; 
           [0029]      FIG. 7  is an illustration of the step of detecting a fade of the method of  FIG. 5   
           [0030]      FIG. 8  is a flowchart of the step of detecting a flash of the method of  FIG. 5 ; 
           [0031]      FIG. 9  is an example of coding structure used on fade-to-black; and 
           [0032]      FIG. 10  is an illustration of the step of detecting a flash of the method of  FIG. 5 . 
       
    
    
     DETAILED DESCRIPTION OF EMBODIMENTS 
       [0033]    Throughout the description, identical reference numerals are used to identify like parts. 
         [0034]    This disclosure describes methods for detection of video fades and flashes such as camera flashes and mechanisms to improve compression performance under those conditions. The detection is carried out ahead of an actual coding process. Once a video fade or camera flash has been detected, the encoding process can be controlled to provide improved coding with these effects. 
       H.264 Coding Structures 
       [0035]    Compression improvements of video signals with fades or flashes are described herein on a basis of the H.264 compression standard. For the purpose of this explanation four types of picture can be distinguished in H.264: 
         [0036]    Intra-coded pictures (I) coded independently of any other pictures; 
         [0037]    Forward predicted pictures (P) which take predictions from previous I or P pictures; 
         [0038]    Non-referenced bi-directionally predicted pictures (B nr ) which take predictions from past and future I or P pictures but in which no predictions are taken from B nr  pictures; 
         [0039]    Referenced bi-directionally predicted pictures (B r ) which also take predictions from past and future I or P pictures, but in which predictions are also taken from B r  pictures. 
         [0040]    The combination of B nr  and B r  pictures makes it possible to generate a pyramid B coding structure such as shown in  FIG. 1 , in which an I picture  11  is independently coded. A non-referenced B picture  12  is predicted by a prediction  112  from the preceding I picture  11  and by a prediction  132  from a succeeding referenced B picture  13 . Referenced B picture  13  is predicted by a prediction  113  from the preceding I picture  11  and by a prediction  153  from a succeeding P picture  15 . Non-referenced B picture  14  is predicted by prediction  134  from preceding referenced B picture  13  and by prediction  154  from succeeding P picture  15 . P picture  15  is predicted by prediction  115  from preceding I picture  11 . 
         [0041]    Such coding structures are generally more efficient than structures without referenced B pictures, such as MPEG 2. However, if extended to more layers of hierarchical coding such as shown in  FIG. 2 , pyramid B picture coding can be used to improve the coding performance during video fades, provided that the pyramid 
         [0042]    The coding performance of flashes can be improved by ensuring that corresponding pictures are coded as B nr  pictures, that is, pictures that are not involved in the coding of neighbouring pictures. 
         [0043]    In H.264, video frames can be coded as one or two pictures, i.e. one picture for the entire frame or two pictures, one for each of two interlaced fields. Since picture coding mode decisions are usually made for an entire frame, reference will be made hereinafter to I, P, B r  or B nr  frames rather than pictures. 
       Fade Detection 
       [0044]    Three types of fades can be distinguished: fade to black, fade from black and cross-fades from one video signal to another. In the former two cases and, to some extent, in the latter case, a fade can be detected by measuring an average luminosity of a video signal over a number of fields. Referring to  FIGS. 6 and 7 , an example fade detector algorithm based on luminance gradient detection can be summarised as follows: 
         [0045]    1. Calculate, step  61 , an average luminance value for each field, Y av (n) and store for four field periods; 
         [0046]    2. Calculate, step  62 , a difference in average luminance between the last four fields to determine changes in luminance between neighbouring fields, although it will be understood that different numbers of fields could be used in different embodiments of the invention 
         [0000]      diff1 =Y   av ( n )− Y   av ( n− 1) 
         [0000]      diff2 =Y   av ( n− 1)− Y   av ( n− 2) 
         [0000]      diff3 =Y   av ( n− 2)− Y   av ( n− 3); 
         [0047]    3. Calculate, step  63 , a sum of the three field differences and store for three field periods, to provide a measure of the variation in luminance over a series of successive fields, although it will be understood that fewer or more differences could be summed in different embodiments of the invention 
         [0000]      sum( n )=diff1+diff2+diff3; 
         [0048]    4. Calculate, step  64 , an average over the last three sums, to detect any trend in the luminance, although it will be understood that fewer or more sums could be averaged in different embodiments of the invention 
         [0000]      av=(sum( n )+sum( n− 1)+sum( n− 2))/3; 
         [0049]    5. Calculate, step  65 , an absolute difference between the three sums and their average, to isolate any large variations from the average luminance indicative of a change in luminance other than a smooth transition 
         [0000]        d 1=|av−sum( n )| 
         [0000]        d 2=|av−sum( n− 1)| 
         [0000]        d 3=|av−sum( n− 2)|; and 
         [0050]    6. Determine, step  66 , whether (av&gt;threshold  1  AND 
         [0000]      d1&lt;threshold2 AND 
         [0000]      d2&lt;threshold2 AND 
         [0000]      d3&lt;threshold2), if so, then fade is detected 
         [0051]    In step  66  the condition in respect of av detects a significant change in luminance from Y(n−5) to Y(n) and the other conditions in respect of d 1 , d 2  and d 3  detect a relatively smooth transition, for example, without spikes in luminance which might otherwise have a disproportionate effect on the average change av, from a luma level at the start of the fade to that at the end. 
       Flash Detection 
       [0052]    Camera flashes typically last for only one to two fields. A detection algorithm for flashes is much simpler than that for fades. Referring to  FIGS. 8 and 10 , an example algorithm is as follows: 
         [0053]    1. Calculate, step  71 , an average luminance value for each field, Y av (n) and store for five field periods; 
         [0054]    2. At the end of each frame period, calculate, step  72 , an average luminance value of the top and bottom fields of the current frame and those of a frame two frames earlier, although it will be understood that in a different embodiment the averages could be taken over more or fewer fields 
         [0000]      current_av=( Y   av ( n )+ Y   av ( n− 1))/2 
         [0000]      previous_av=( Y   av ( n− 4)+Y av ( n− 5))/2; 
         [0055]    3. If (Yav(n−2)&gt;maximum(current_av,previous_av)+threshold), that is if the luminance of the preceding bottom field is significantly greater than the larger of the average field luminances of the preceding and succeeding frames, then a flash has been detected, step  73 , on a bottom field of the previous frame; and 
         [0056]    4. If (Yav(n−3)&gt;maximum(current_av,previous_av)+threshold), that is if the luminance of the preceding top field is significantly greater than the larger of the average field luminances of the preceding and succeeding frames, then a flash has been detected, step  74 , on the top field of the previous frame. 
         [0057]    Comparing the two fields to the maximum luminance values of the previous and next frame prevents false detection on scene cuts. 
       Compression Improvement 
       [0058]      FIG. 3  shows a block diagram of a system according to an aspect of the invention. An uncompressed video signal is input in parallel to a compensating delay  1 , a fade detector  2  and a flash detector  3 . An output of the compensating delay is input to a frame re-ordering block  4 . Outputs of the fade detector  2  and the flash detector  3  are input to inputs of a coding mode block  5  which generates a coding structure control signal for input to a control input of the frame re-ordering block  4 . A motion search control signal is also output from the coding mode block  5  to a motion estimator  7  and a quantisation control signal is output from the coding mode block  5  to a compression coding loop  6 . Outputs from the frame re-ordering block  4  are also input to the motion estimator  7  and the compression coding loop  6 . Motion vectors are output from the motion estimator  7  to the compression coding loop  6  and a reconstructed video signal is output from the compression coding loop  6  to the motion estimator  7 . An output of the compression coding loop  6  is also output to an entropy coding block  8  to output a compressed video signal. 
         [0059]    In use, the uncompressed video signal is input to the fade detector  2  and the flash detector  3  as well as to the compensating delay  1 . The compensating delay  1  is necessary because the algorithms take several field periods before a fade or flash is detected. It is in the nature of the fade detector and flash detector algorithms that if one property is detected the other one will not be detected. 
       Improvement on Camera Flashes 
       [0060]    Referring to  FIGS. 3 and 5 , the coding mode block  5  changes a manner in which a video signal is encoded. For example, if a flash is detected, step  52 , in a certain field then the coding structure is changed such that the frame on which the flash is detected is coded as a B nr  frame, i.e. a frame from which no other predictions are taken. This can be done by inserting, step  53 , one or more additional B nr  frames between I and P frames.  FIG. 4  shows an example of a sequence  40  with a camera flash  41 . Following a regular coding structure of I B nr  B r  B nr  P B nr  B r  B nr  P frames it can be seen that the frame with the flash would have been coded as a P frame  44 . However, this would give rise to distortions due to poor predictions from the P frame with the flash. The problem is solved by inserting an additional B nr  frame  43  into the coding structure. Furthermore, the compression coding loop  6  is informed, step  54 , about the camera flash so that the frame with the flash can be quantised more coarsely without causing any noticeable picture degradation. 
       Improvement on Fades 
       [0061]    Referring to  FIGS. 3 ,  5  and  8 , if a fade  81  is detected, step  55 , the coding mode block  5  reduces, step  56 , a search range of the motion estimation block  7  in order to reduce a number of otherwise large chaotic motion vectors. Furthermore, the coding mode block  5  changes, step  57 , an order in which frames are coded such that the coding structure consists of a sequence of pyramid B frames  84  across the entire fade as shown in  FIG. 8 . It can be seen in  FIG. 8  that the coding structure is arranged such that there are reference frames  82 ,  83  immediately before and after the fade  81  respectively and a pyramidal coding structure  84  is used during the fade. 
         [0062]    Alternative embodiments of the invention can be implemented as a computer program product for use with a computer system, the computer program product being, for example, a series of computer instructions stored on a tangible data recording medium, such as a diskette, CD-ROM, ROM, or fixed disk, or embodied in a computer data signal, the signal being transmitted over a tangible medium or a wireless medium, for example microwave or infrared. The series of computer instructions can constitute all or part of the functionality described above, and can also be stored in any memory device, volatile or non-volatile, such as semiconductor, magnetic, optical or other memory device. 
         [0063]    Although the present invention has been described with reference to preferred embodiments, workers skilled in the art will recognize that changes may be made in form and detail without departing from the spirit and scope of the invention.