Patent Publication Number: US-9900599-B2

Title: High frequency emphasis in decoding of encoded signals

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This patent application is a non-provisional patent application of U.S. provisional patent application Ser. No. 60/520,163, filed on 2003 Nov. 14; this application is a continuation of U.S. patent application Ser. No. 10/988,873 filed on 2004 Nov. 15, of U.S. patent application Ser. No. 13/455,057 filed on 2012 Apr. 24, and of U.S. patent application Ser. No. 14/529,058 filed on 2014 Oct. 30; the benefit of the filing dates of all of the foregoing applications are claimed by this application and all of the foregoing applications are incorporated herein, in their entirety, for all purposes, by this reference. 
    
    
     FIELD OF THE INVENTION 
     The present invention relates to the field of encoding and decoding of digital signals. More specifically, the present invention is related to video and image coding/decoding. 
     BACKGROUND OF THE INVENTION 
     Advances in microprocessor and video related technologies have led to wide spread deployment and adoption of numerous types of audio, video and imaging devices. Examples of such devices include but are not limited to digital cameras, camcorders, MP3 players, digital versatile disk (DVD) players, video enabled laptop and desktop computing devices as well as servers, and so forth. 
     Advances in networking, telecommunication, satellite and other related technologies have also led to increase in on demand and/or real time online delivery of audio and video information, including delivery over public networks, such as the Internet. 
     Whether videos are delivered offline (e.g. from a DVD player) or online (e.g. from a video server), high quality audio or video inherently requires a high volume of data. Thus, audio and video delivery and rendering often involve encoding and decoding to reduce the amount of data to be stored, retrieved and/or transmitted. 
     Additionally, Video Codecs employing schemes like Deblocking Filter and B-Frames have a tendency to attenuate certain spectral information in the video. For example, Deblocking Filter can smoothen out certain frequencies and B-Frame skipped mode can hurt the high frequencies present in Video. Also small transform sizes and/or high quantizers may not be able to capture certain spectral information during video coding. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  illustrates a block diagram of a prior art video processing system. 
         FIG. 2 a    illustrates a frame image without the High Frequency Emphasis (HFE) enhancements. 
         FIG. 2 b    illustrates a frame image with the HFE enhancements with Amplified Basis functions that add texture back to the frame. 
         FIG. 3 a    illustrates a frame image with the HFE enhancements. 
         FIG. 3 b    illustrates a frame image without the HFE enhancements. 
         FIG. 4 a    illustrates a highest frequency determining ZigZag scan practiced in various embodiments. 
         FIG. 4 b    illustrates an exemplary Basis Function C 4  of Strength  2 . 
         FIGS. 5 a -5 c    illustrate transform coefficients of a video frame before quantization, after dequantization without HFE enhancement, and after dequantization with HFE enhancement. 
         FIGS. 6 a -6 c    illustrate distributions of exemplary statistics employed for HFE, in various embodiments. 
         FIGS. 7 a -7 b    illustrate various exemplary statistics determined for an exemplary context, for various embodiments. 
         FIG. 8  illustrates a block diagram of an HFE video processing system, in accordance with various embodiments. 
         FIG. 9  illustrates an example system having a video sender device and a video receiver device incorporated with teachings of the present invention, in accordance with one embodiment. 
         FIGS. 10 a -10 c    illustrate various embodiments of the present invention. 
     
    
    
     DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION 
     Embodiments of the present invention includes a video decoder, devices equipped with these decoders, systems made up of such devices, and methods of operations of these elements, devices and systems, and related subject matters. 
     A prior art video processing system  100  is illustrated in  FIG. 1 . More specifically, the decoding section of video processing system  100  is illustrated. For the illustrated embodiment, Video Codecs employ schemes like Deblocking Filter and 8-Frames. As will be appreciated by those skilled in the art, Video Codecs have a tendency to attenuate certain spectral information in the video. For example Deblocking filter can smoothen out certain frequencies and 8-Frame skipped mode can hurt the high frequencies present in Video. 
     Additionally, as typical of conventional Video Codecs, its inverse transformation and dequantization units operate with small transform sizes and/or high quantizers. Thus, Video Codecs may not be able to capture certain spectral information during video coding. 
     The terms Video Codecs, Deblocking Filter, 8-Frame, 8-Frame skipped mode, transformation, quantizers, high frequency, 1-Frame, P-Frame, quantization parameters, basis function, and so forth, are all used in this specification in accordance with their plain meaning as understood by those of ordinary skill in the art of video signal processing, in particular, consistent with the meaning of these terms as they are used in ITU video signal processing standards. [ITU=International Telecommunication Union.] 
     However, as will be described in more detail below, in various embodiments of the present invention, High Frequency Emphasis (HFE) is employed to correct and further enhance the recovered video. HFE adds back spectral information lost or attenuated in video coding. More specifically, in various embodiments, HFE is implemented as a post filter which, 
     Adds a calculated amount of certain high frequency basis function actually transmitted in the bitstream with the video frames. 
     Adds a calculated amount of high frequency basis functions predicted from coefficients transmitted in the current block or the past or future frame. 
       FIG. 2 a    illustrates a frame image without the HFE enhancements. 
       FIG. 2 b    illustrated a frame image with the HFE enhancements with Amplified Basis functions that add texture back to the frame. 
     As a further example, the HFE enhanced video frame of  FIG. 3 a    has more texture than the unenhanced video frame of  FIG. 3   b.    
     liP Frame HFE 
     In various embodiments, a video frame to be decoded with HFE may be an 1-Frame or a P-Frame, and the liP Frame is decoded and enhanced by performing the followings on one or more blocks (e.g. each block) of the 1/P Frame: 
     The highest transmitted coefficient (HC), i.e., highest frequency, of the block is determined. In various embodiments, the transmitted coefficients of the block is analyzed using the ZigZag scan as shown in  FIG. 4   a.    
     In various embodiments, coefficients C 6 , C 9 , C 13 , and C 14  (shaded boxes) are excluded from consideration, when determining the highest frequency coefficient. The exclusion is motivated by the deblocking filter having a tendency to smooth out the edges of the block and not the center. Resultantly, coefficients C 6 , C 9 , C 13 , and C 14  of  FIG. 4 a    have basis functions which mostly effect the center pixels. 
     In various embodiments, the strength of the correction to be applied is also determined. In some of these embodiments, QP (quantization parameter) is employed to limit its strength. For example, in various embodiment, the strength of the correction may be limited by QP as follows: 
     O&lt;=QP&lt;9: strength=0 
     9&lt;=QP&lt;13: strength=1 
     13&lt;=QP&lt;18: strength=2 
     18&lt;=QP&lt;32: strength=3 
     In various embodiments, the determined basis functions for a block are added to the corresponding block of an intermediate decoded version of the video frame, in view of the determined HC and the Strength.  FIG. 4 b    illustrates an example Basis Function C 4  of Strength  2 . 
     B-Frame HFE 
     In various embodiments, a video frame to be decoded with HFE may be a 8-Frame, and the 8-Frame is decoded and enhanced by performing the followings on one or more blocks (e.g. each block) of the 8-Frame: 
     In various embodiments, if the block has any transmitted coefficient, the same HFE scheme as was used with the liP Frame is used. 
     In various embodiments, in the case of a Skipped MB, there are no transmitted coefficients. The Skipped MB is reconstructed as the weighted average of temporally co-located blocks. More specifically, the HC and QP for the Skipped MB are determined from the nearest temporally co-located blocks (e.g. adjacent ones). 
     In various embodiments, HFE also addresses the fact that Video Codecs inherently involves quantizing some frequency coefficients to zero. Further, high frequencies are quantized to zero more often, because a typical video distribution has high energy in low frequencies and lesser energy in high frequencies. 
       FIGS. 5 a -5 c    illustrate the transform coefficients of an example video frame, the same of a dequantized version of the video frame without HFE, and the same with HFE restoring some or all of the high frequencies quantized to zero. 
     In various embodiments, HFE restores some or all of the high frequencies quantized to zero as follows; 
     It inserts High Frequency Coefficients which statistically look similar to the coefficients which are quantized to zero. 
     In various embodiments, statistics are generated for various contexts in the bitstream, and applied on a context basis. Examples of contexts may include, but are not limited to 
     1&gt;QP 
     2&gt;Macroblock Type 
     3&gt;Last Significant coefficient in a Zig Zag Scan 
     4&gt;Picture Type 
     5&gt;Frequencies in the temporally co-located block. 
       FIGS. 6 a -6 c    illustrate distributions of a number of statistics which may be employed. More specifically,  FIG. 6 a    illustrates a distribution of coefficient levels of Zero Quantized Coeffs, i.e. quantized to zero, whereas  FIG. 6 b    illustrates a distribution of the number of such coefficients quantized to 0 per block.  FIG. 6 c    illustrates distribution of coeff levels quantized to zero of each coefficient represented as Lambda of a ideal Laplacian distribution. 
       FIGS. 7 a -7 b    illustrate distributions of a number of statistics, which may be employed. More specifically,  FIGS. 7 a -7 b    illustrate an Average Distribution of Number of Zero Quantized Coeffs for an exemplary context, which is “given block is not code in an 1-Frame”, in a 1-D and a 2-D view, respectively. 
     Depending on implementations, in various embodiments, one or more statistics is maintained for one or more contexts. An example of another context is the context of DC or Low Frequencies Coded in residue of an 1-Frame. As those skilled in the art would appreciate, embodiments of the present invention may be practiced with one or more selected statistics for one or more selected contexts. 
     Results and Applications: 
     Experience has shown that the use of embodiments of the present invention of HFE with an encoder may increase the average PSNR of the encoder by 0.05 dB and minimum PSNR by 0.15 dB. Since HFE is generally considered to be visually pleasing, stronger HFE schemes may also be practiced in other embodiments. However, the strong HFE schemes may degrade PSNR. [PSNR=Peak Signal-to-Noise Ratio.] 
     In various embodiments, HFE may be used as a post processing stage for decoders, such as ITU H264 compliant decoders. 
     A block diagram of one decoder embodiment  500  of the present invention is illustrated in  FIG. 8 . For the illustrated embodiment, decoder  500  includes similar conventional elements as decoder  100  of  FIG. 1 , e.g. a dequantization unit, an inverse transformation unit, a eblocking filter unit, and so forth. However, decoder  500  further includes a HFE unit  502  adapted to practice the HFE operations described earlier. Further, reconstruct unit  504  is further adapted to additionally enhance an initially decoded video frame to generate the final decoded video frame. in accordance with the HFE specified by HFE unit  502 . 
     While for ease of understanding, HFE and reconstruction units  502 - 504  are each illustrated as a single unit, in alternate embodiments, their functions may be implemented in one or more units. Further, the various units may be implemented in hardware, using ASIC or reconfigurable IC, or in software, in C or other suitable system programming languages. 
     [ASIC=Application Specific Integrated Circuit] 
     The term “initially decoded video frame” refers to the input decoded video frame of the reconstruction unit  504 , whereas the term “finally decoded video frame” refers the output decoded video frame of the reconstruction unit  504 . The input decoded video frame of the reconstruction unit  804  is also referred to as the intermediate decoded video frame. 
     Experience also has shown that HFE post processing, with or without sideband information, can improve an existing codec. The above system can be used to improve the quality of a given codec without requiring modification to the underlying bitstream. The spectral information can be predicted or coded separately as sideband information. 
     Embodiments of the present invention could be an effective scheme to improve upon the codec without changing the underlying bitstream. 
     System 
       FIG. 9  illustrates an overview of an example system incorporated with teachings of the present invention, in accordance with one embodiment. As illustrated, for the embodiment, example system  600  includes video sender  602  and video receiver  604  communicatively coupled to each other as shown. Video sender  602  includes in particular encoder  612 , while video receiver  604  includes in particular decoder  614 . 
     Video sender  602  provides an encoded video signal  606  to video receiver  604 , using encoder  612  to encode the video. Further, decoder  614  advantageously includes hardware and/or software components incorporated with the HFE teachings of the present invention. 
     Video receiver  604  receives the encoded video  606 , decodes the video using decoder  614  with HFE, and renders the decoded video for consumption by one or more users of video receiver  604 . Accordingly, video receiver  604  may also be referred to as a video renderer. For the purpose of the present application, the two terms may be considered interchangeable, unless the context clearly indicates to the contrary. 
     For the purpose of the present application, as described earlier, a slice of a video is an integer number of macroblocks of a video. A video includes a number of pictures, each including a number of fields or frames, with each field or frame including a number of slices of macroblocks. Each macroblock includes a number of blocks of pixels. 
     Except for the fact that decoder  614  is advantageously provided with hardware and/or software components adapted to decode video signals with HFE in accordance with an embodiment of the present invention, video server  602 , including encoder  612 , and video receiver  604  represent a broad range of these elements known in the art or to be designed. Different embodiments may employ different implementations of these elements. 
     Further, video server  602  and video receiver  604  may be communicatively coupled to each other via any one of a number of wire based and/or wireless connections. For examples, the connection may be a bus connection (such as a Universal Serial Bus connection) or a wired local area network connection (such as an Ethernet connection). As a further example, the connection may also be a combined wireless and wired wide area network connection (such as a TMDA or COMA wireless connection between video sender  602  or video receiver  604  and a base station at one end, and a wire based ATM connection from the base station to the other end). [TDMA=Time Division Multiple Access, COMA=Code Division Multiple Access, and ATM=Asynchronous Transfer Mode.] 
     Example Embodiments Including Decoder  500  of  FIG. 8   
       FIGS. 10 a -10 c    illustrate various embodiments incorporated with decoder  500  of  FIG. 8 . More specifically,  FIG. 10 a    illustrates an example digital system incorporated with a software implementation of encoder  712  and decoder  714  (based on decoder  500  of  FIG. 8 ). For the embodiment, example digital system  700  includes processor  702 , memory  704 , mass storage  706 , I/O devices  708 , and communication interface  710  coupled to each other and a (set of bridged) bus(es) as shown. 
     Memory  704  and mass storage  706  may be employed store a working and a persistent copy of a software implementation of encoder  712  and decoder  714 . Except for its usage, memory  704  and mass storage  706  may respectively be any one of a number of storage elements known in the art or to be designed. 
     Processor  702 ,  110  devices  708 , communication interface  710  and the bus(s) represent a broad range of the respective elements known in the art or to be designed. 
     In various embodiments, digital system  700  may be a wireless mobile phone, a personal digital assistant, a palm-sized computing device, a laptop computing device, a tablet computing device, a desktop computing device, a set top box, a server, an entertainment unit, a DVD player, a digital camera, a television, a display monitor, and other computing devices of the like. 
       FIG. 10 b    illustrates an article of manufacture having storage medium  720  and a software implementation of decoder  714  stored therein. Stored decoder  714  may be employed to program and enable an apparatus to decode encoded video with HFE as earlier described. The apparatus may e.g. be digital system  700  of  FIG. 10   a.    
     In various embodiments, article  720  may e.g. be a diskette, a compact disk (CD), a DVD (digital versatile disk) or other computer readable medium of the like. In other embodiments, article  720  may be a distribution server distributing encoder  712  and/or decoder  714  on line, via private and/or public networks, such as the Internet. In one embodiment, article  720  is a web server. 
       FIG. 10 c    illustrates a circuit board embodiment, where circuit board  740  includes an ASIC  760  having a number of hardware implementations of a decoding unit. ASIC  760  may be employed to form e.g. a DVD player, a digital camera, an entertainment unit, a set-top box and so forth. 
     In alternate embodiments, the decoding units may be disposed in multiple ASICs  760  instead. 
     Alternate Embodiments 
     While the present invention has been described with the above illustrative embodiments, including the example video system of  FIG. 9 , and various embodiments of the video decoder of  FIGS. 10 a -10 c   . The invention is not so limited. It may be modified and/or enhanced consistent with the scope of the invention defined by the claims to follow.