Patent Publication Number: US-8126047-B2

Title: Video compression and transmission system with transmitter side memory restriction

Description:
TECHNICAL FIELD 
     This application relates to video compression and transmission. 
     BACKGROUND 
     The transmission of real-time video over wired/wireless channels is increasingly desirable. The transmission system should provide a maximum visual quality for channels with limited capacity while being implemented in devices with limited memory size (where the total memory size of the transmitter is less than the size of the input picture). An image/video coding standard known as JPEG 2000 is a good solution for this task because it provides rate control with a high accuracy level. (ITU-T and ISO/IEC JTC 1, JPEG 2000 Image Coding System: Core Coding System, ITU-T Recommendation T.800 and ISO/IEC 15444-1, JPEG 2000 Part 1, 2000.) 
     The use of JPEG 2000 may be problematic, however. Where the memory size is restricted (the normal case), the input picture is preferably split into small fragments (tiles), such that each tile may be compressed separately. The original JPEG 2000 rate-control implementation provides approximately an equal bit-size for each compressed tile. But visual quality of the decoded tiles may be rather different. For example, in some implementations, the quality of the reconstructed image may be inconsistent. For example, in the image  50  of  FIG. 4 , some decoded tiles have a “good” visual quality, while others have a “very bad” visual quality. 
     Thus, there is a need for a video compression and transmission scheme to overcome the shortcomings of the prior art. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The foregoing aspects and many of the attendant advantages of this document will become more readily appreciated as the same becomes better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings, wherein like reference numerals refer to like parts throughout the various views, unless otherwise specified. 
         FIG. 1  is a block diagram of a video compression and transmission system, according to some embodiments; 
         FIGS. 2A and 2B  are flow diagrams showing operation of a rate-distortion algorithm used by the video compression and transmission system of  FIG. 1 , according to some embodiments; 
         FIG. 3  is a fragment of test video sequence; 
         FIG. 4  is a fragment of reconstructed test video sequence, compressed prior to transmission using a prior art JPEG 2000 technique, according to some embodiments; 
         FIG. 5  is a fragment of reconstructed test video sequence, compressed prior to transmission using the video compression and transmission system of  FIG. 1 , according to some embodiments; 
         FIG. 6  is a comparative PSNR/frame graph for a channel rate of 56.6 Mbits/sec, according to some embodiments; 
         FIG. 7  is a comparative PSNR/frame graph for a channel rate of 37.7 Mbits/sec, according to some embodiments; and 
         FIG. 8  is a comparative PSNR/frame graph for a channel rate of 28.3 Mbits/sec, according to some embodiments. 
     
    
    
     DETAILED DESCRIPTION 
     In accordance with the embodiments described herein, a video compression and transmission system and method are disclosed. The video compression and transmission scheme is a modification of the JPEG 2000-based video transmission systems for constant bit-rate channels. 
     A novel video compression and transmission system  100  is depicted in  FIG. 1 , according to some embodiments. An input tile  22  is received from a video source  20 , to enter a number of levels detector  24 , which includes two encoders  26  and  28 . Depending on the tile content (desktop or video), the video compression and transmission system  100  chooses a number of levels of wavelet-transform decomposition. As used herein, desktop content refers to non-natural images, text computer graphics, and synthetic images. For example, a desktop icon is a type of desktop content. Video content refers to natural images, such as those found in photographs (e.g., JPG) and streaming video (e.g., movie image). In some embodiments, the number of levels detector  24  uses a simple implementation of a spatial-adaptive wavelet transform. 
     The JPEG2000 tier 1 encoder  26  and the JPEG2000 tier 1 encoder  28  use wavelet discrete transforms (DWT), where “tier 1” refers to lossless compression. In some embodiments, a single level is sufficient for desktop content while three levels is good for video content. Thus, the number of levels detector  24  includes two different encoders, with the encoder  26  having one level DWT and the encoder  28  having three levels DWT. After encoding, the bit stream with the smaller size is chosen. 
     The input tile  22  is also received into a static tile detector  30 , which compares the current tile hash-function value with a hash-function value of a tile from a previous frame of the video source  20 . By comparing the hash function value of the input tile  22  to a hash function value of a previous tile, the static tile detector  30  determines whether the tile is non-static (changing) or static (unchanging). Such information is useful for further processing by the video compression and transmission system  100 . 
     The video compression and transmission system  100  further includes a rate-distortion controller  32 , a transmitter buffer  36 , and a third encoder  34 , a JPEG2000 tier 2 encoder. The rate-distortion controller  32  determines the compression parameters for the current tile  22 , using a rate-distortion algorithm  200 . These parameters depend on the number of bits in the transmitter buffer  48  as well as the result from the static tile detector  30 , that is, whether the tile has been deemed static or non-static (dynamic). The JPEG2000 tier 2 encoder  34  receives the input tile  22  after lossless compression by the number of levels detector  24  and the compression parameters  44  from the rate-distortion controller  32 , packages the tile  22  into data units, called packets, and places a compressed tile  46  representative of the input tile  22  into the transmitter buffer  36 . 
     It is assumed herein that the memory size at the receiver is greater than the number of bits for one input frame of video source  20 . Therefore, the receiver can “copy” and reproduce the corresponding tile from the previous frame. 
     In  FIG. 1 , the video compression and transmission system  100  includes the rate-distortion controller  32 , which executes a rate-distortion algorithm  200 , in some embodiments. The rate-distortion algorithm  200  is described in more detail in the flow diagrams of  FIGS. 2A and 2B , described further below. 
     Each tile is composed of an integer number of code blocks with numbers i=1, 2, 3 . . . . Each block is subsequently composed of an integer number of coding passes. Each additional pass contributes some bytes to the block and, in turn, decreases the overall tile distortion. The task of the rate-distortion controller  32  is to decide which passes are to be included in the resulting code stream, or, more specifically, to find the set of truncation points {n i } that satisfy the above requirements. Consider n={n i } to be a truncation vector, where n i  signifies that i th  block is truncated in the point number, n i . The truncation vector is a compression parameter for the JPEG2000 tier 2 encoder  34 . The overall tile distortion after the truncation is denoted as 
                 d   ⁡     (   n   )       =       ∑   i     ⁢           ⁢     d   i     n   i           ,         
where the quantity, d i   n     i    corresponds to the distortion value of i th  block truncated in the point number, n i . Similarly, the resulting tile rate is denoted as
 
     
       
         
           
             
               r 
               ⁡ 
               
                 ( 
                 n 
                 ) 
               
             
             = 
             
               
                 ∑ 
                 i 
               
               ⁢ 
               
                   
               
               ⁢ 
               
                 
                   r 
                   i 
                   
                     n 
                     i 
                   
                 
                 . 
               
             
           
         
       
     
     In some embodiments, the video compression and transmission system  100  executes a static optimization algorithm for the case when all the tiles in the current frame are static. If all the tiles in the current frame are static, two possible solutions exist. In the first solution, the rate-distortion controller  32  sends a “copy” command to the receiver. In this case, the corresponding tile with distortion d(x t-L ) from the previous frame is reproduced on the receiver side, where L represents the number of tiles in the current frame, x t-L -truncation vector which was used for tile t−L. For this solution, the video compression and transmission system  100  empties the transmitter buffer  36  without buffer accumulation. In the second solution, the tile is transmitted to the rate-distortion controller  32  with d lossless  distortion (which corresponds to visual lossless encoding). Thus, the static optimization algorithm finds a solution for the following optimization task: 
             {             minimize   ⁢           ⁢       ∑     t   =     t   0         t   =       t   0     +   L         ⁢           ⁢     d   t                       b   static     ⁡     (   t   )       &lt;     B   L             ,           
where d t ε{d lossless ,d(x t-L )} represents the distortion for the tile t, b static (t) represents the number of bits in the buffer, and B L  represents the buffer threshold, t 0 —number of first tiles in the frame. As result, for each tile t, there may be a set of distortion values, d t , and a number of bits in the transmitter buffer, b static (t). Before compression of the current frame, the video compression and transmission system  100  finds a solution for the above optimization task, in some embodiments. This solution is used for frame compression with static and non-static tiles (as noted below).
 
       FIGS. 2A and 2B  are flow diagram showing operation of the rate-distortion algorithm  200  of the video compression and transmission system  100  of  FIG. 1 , according to some embodiments. The rate-distortion algorithm  200  begins ( FIG. 2A ) by initializing variables, {circumflex over (d)}(0)=d 0 ,t=0, b(0)=0. (block  202 ). Next, t is incremented, t=t+1, and min(b(t−1), r) bits are transmitted (block  204 ). 
     The rate-distortion algorithm  200  next inquires whether the current tile  22  is the first tile in a frame (block  206 ). If so, the static optimization algorithm, described above, is run (block  208 ). Otherwise, a search is made for truncation vector, x t , so that 
             {             r   ⁡     (     x   t     )       =       min     {   n   }       ⁢           ⁢     r   ⁡     (   n   )                       d   ⁡     (     x   t     )       ≤     d   losless                       
(block  210 ), and a search is made for truncation vector, y t , so that
 
             {             r   ⁡     (     y   t     )       =       min     {   n   }       ⁢           ⁢     r   ⁡     (   n   )                       d   ⁡     (     y   t     )       ≤       d   ^     ⁡     (   t   )                         
(block  212 ). The rate-distortion algorithm  200  inquires whether the current tile  22  is static and max{0,b(t−1)−r}&lt;b static (t) (block  214 ). If so, the solution from running the static optimization algorithm (block  208 ) is used (block  216 ) and the process repeats, starting with incrementing t (block  204 ). Otherwise, the rate-distortion algorithm  200  makes another inquiry, this time determining whether max{0,b(t−1)−r}+r(y t )&gt;B H  (block  218 ). If so, the algorithm  200  proceeds to the next part of the algorithm ( FIG. 2B ), where a bit computation is made (block  232 ), as described below. Otherwise, the compressed tile is appended to the transmit buffer  36  according to a vector, y t  (block  220 ), and the process repeats, starting with incrementing t (block  204 ).
 
     Where max{0,b(t−1)−r}+r(y t )&gt;B H  is true (block  218 ), the rate-distortion algorithm  200  computes the number of bits that are needed to transmit the tile with distortion d empty  by searching for the vector for y t , so that 
             {             r   ⁡     (     y   t     )       =       min     {   n   }       ⁢           ⁢     r   ⁡     (   n   )                       d   ⁡     (     y   t     )       ≤     d   empty                       
(block  228 ). Then, the algorithm  200  searches for z t , so that
 
             {             d   ⁡     (     z   t     )       =       min     {   n   }       ⁢           ⁢     d   ⁡     (   n   )                       r   ⁡     (     z   t     )       ≤     min   ⁡     (         B   0     -     b   ⁡     (   t   )         ,     r   ⁡     (     y   t     )         )                         
(block  232 ). If max(0,b(t−1)−r)=0 (block  234 ), then {circumflex over (d)}(t+1)={circumflex over (d)}(t)+Δd (block  238 ), and the algorithm  200  proceeds to search for y t  again (block  240 ). Otherwise, another inquiry is made, this time whether the current tile  22  is static (block  236 ). If so, the rate-distortion algorithm  200  sends a command to the receiver, “Copy tile from previous frame” (block  240 ). Otherwise, the algorithm  200  appends the compressed tile to the transmitter buffer according to the vector, z t , (block  242 ). Next, t is incremented, t=t+1, and min(b(t−1),r) bits are transmitted, as in block  204  (block  222 ). If the current tile is the first in the frame (block  224 ), the static optimization algorithm is run to produce a solution (block  230 ). Otherwise, the rate-distortion algorithm  200  searches for truncation vector, x t , so that
 
             {             r   ⁡     (     x   t     )       =       min     {   n   }       ⁢           ⁢     r   ⁡     (   n   )                       d   ⁡     (     x   t     )       ≤     d   losless                       
(block  226 ). The above process is then repeated.
 
     The rate-distortion algorithm  200  used by the video compression and transmission system  100  may be advantageous over prior art compression techniques. The reconstructed image quality produced by the video compression and transmission system  100  is better than that of prior art implementations. Further, the video compression and transmission system  100  may be adapted to wireless transmissions. 
     Empirical results using the video compression and transmission system  100  are illustrated in  FIGS. 3-5 , according to some embodiments. The image  40  of  FIG. 3  is a fragment of an original video sequence (1024×768 frame, 30 frames per second). This original video sequence is first compressed with an original JPEG 2000 with a tile size of 256×16 at a channel rate of 37.7 Mbits/sec ( FIG. 4 ). Next, this original video sequence is compressed with the video compression and transmission system  100  with a 50 kB transmit buffer size, with a tile size of 256×16 and a channel rate of 37.7 Mbits/sec ( FIG. 5 ). 
     The graphs of  FIGS. 6-8  demonstrate how the peak signal-to-noise ratio (PSNR) depends on the frame number of the video sequence for different channel rates.  FIG. 6  corresponds to a channel rate of 56.6 Mbits/sec.  FIG. 7  corresponds to a channel rate of 37.7 Mbits/sec.  FIG. 8  corresponds to a channel rate of 28.3 Mbits/sec. PSNR is a metric of visual quality.  FIGS. 6-8  show that the video compression and transmission system  100  is better than the original JPEG2000 rate-control scheme, because the PSNR for all frames is better in the former than in the latter, for different channel throughputs and memory restrictions. 
     Thus, the novelty of the video compression and transmission system  100  is the strategy of controlling the transmit buffer  36  ( FIG. 1 ), which is useful for systems with memory restrictions on the transmitter side. The video compression and transmission system  100  further orients to worst tile quality maximization. 
     While the application has been described with respect to a limited number of embodiments, those skilled in the art will appreciate numerous modifications and variations therefrom. It is intended that the appended claims cover all such modifications and variations as fall within the true spirit and scope of the invention.