Encoding/decoding digital frames by down-sampling/up-sampling with enhancement information

Input digital frames may be down-sampled to create one or more base frames characterized by a lower resolution than the input digital frames. Enhancement information corresponding to a difference between pixel values for the one or more input digital frames and corresponding pixel values of up-sampled versions of the one or more base frames is then created. The one base frames are encoded to form a set of base data and the enhancement information is encoded to form a set of enhancement data. The base data and enhancement data may then be transmitted over a network or stored in a memory.

FIELD OF THE DISCLSOURE

Aspects of the present disclosure are related to encoding and decoding of digital data for streaming applications. In particular, the present disclosure is related to encoding the same video content at multiple resolutions.

BACKGROUND

Digital signal compression (sometimes referred to as video coding or video encoding) is widely used in many multimedia applications and devices. Digital signal compression using a coder/decoder (codec) allows streaming media, such as audio or video signals to be transmitted over the Internet or stored on compact discs. A number of different standards of digital video compression have emerged, including H.261, H.263; DV; MPEG-1, MPEG-2, MPEG-4, VC1; AVC (H.264), and HEVC (H.265). These standards, as well as other video compression technologies, seek to efficiently represent a video frame picture by eliminating the spatial and temporal redundancies in the picture and among successive pictures. Through the use of such compression standards, video contents can be carried in highly compressed video bit streams, and thus efficiently stored in disks or transmitted over networks.

Encoding the same video content at multiple resolutions is often necessary for video streaming over the network and for targeting different decoding hardware with different computational capabilities. For example, it is sometimes necessary to encode video with multiple resolutions due to bandwidth limitations and due to recent advances in the capabilities of video encoding and decoding hardware. With the growing popularity of 4K displays (picture width of 3840 to 4096 pixels), the demand for higher resolution video content has increased. The storage required for 4K video is much higher than for 1080p HD video. In some cases, storing both lower and higher resolution versions of the same video content might be necessary, e.g., for display on different hardware. Having both HD and 4K resolution video streams on a Blu-Ray disc could allow a lower powered video player to display the HD content and a more powerful video player to display the 4K content. However, due to storage limits on physical discs, it might not be possible to fit both resolutions on a single disc. One approach to addressing this issue is to produce both low resolution and high resolution discs for the same content. For video streaming over a network, the higher resolution video could be displayed when the bandwidth is high and the lower resolution could be displayed during network congestion. Fewer bits are needed to transmit lower resolution encoded video (e.g., 1080p HD video) than higher resolution (e.g., 4K) video.

Another approach is to encode the low resolution video and to up-sample it when presenting it on the higher resolution hardware. However, the video quality suffers when the up-sampled low resolution video is played on the high resolution hardware. Yet another approach is to store just the high resolution encoded video on disk. Unfortunately, low powered hardware might have to skip frames because it's not powerful enough to maintain the desired frame rate for the higher resolution video.

Having multiple discs increases the production costs, while storing only one resolution of the video on the disc does not allow the best experience for either the lower powered or the higher powered player. If only the lower resolution video is stored on the disc, the displayed picture will require up-sampling on higher resolution displays. The picture quality of up-sampled video content will be lower than that of content that is originally of higher resolution. If only the higher resolution video is stored on the disc, lower powered devices may not be able to display the content at the original frame rate, and may have to drop frames or lower the frame rate. This too results in a reduced user enjoyment.

While an extension to the Advanced Video Coding (AVC) standard known as Scalable Video Coding (SVC) allows efficient compression for multiple resolutions, the number of devices that support this extension is much lower compared to those that do not. As a result, the common approach to handling multiple resolutions is to resize the video content to multiple resolutions before encoding, and generating separate bitstreams for each resolution that could be decoded independently of each other.

It is within this context that aspects of the present disclosure arise.

DESCRIPTION OF THE DRAWINGS

Introduction

FIG. 1Billustrates an example of conventional encoding multi-resolution video121. An encoder122encodes the high resolution video to generate a high resolution stream124of bits representing encoded high resolution video frames. To accommodate older hardware not configured for high resolution video or to reduce bandwidth required for transmission during bandwidth congestion, the high resolution video is down-sampled, as indicated at123and the resulting down-sampled video121′ is encoded, e.g., by another encoder122′ to generate a stream of bits124′ representing encoded down-sampled video frames.

On the decoder side, illustrated inFIG. 1C, a decoder132receives the high resolution stream124and decodes it to generate high resolution output126in the form of decoded high resolution video frames. Devices not equipped to decode the high resolution stream may ignore the high resolution stream and receive and decode the down-sampled stream124′, e.g., using a different decoder132′.

Aspects of the present disclosure allow for efficient video compression without utilizing extensions to a video coding standard. The approach described herein allows for high picture quality with lower bit usage compared to the existing method of encoding multiple resolutions of the same content as separate bitstreams. Instead of creating a separate bitstream at a higher resolution, the encoder creates an enhancement stream that uses fewer bits. The decoder creates output video for display by combining a lower resolution base stream and the enhancement stream. The extra processing required to generate the output video could be performed efficiently on a graphics processor unit (GPU). The proposed approach is particularly advantageous if the average time needed to generate a high-resolution frame using the proposed is not higher than the time needed to decode a frame using a separate high-resolution bitstream.

According to aspects of the present disclosure, the proposed approach uses a combination of up-sampling of low resolution video and enhancement information. When low resolution video is up-sampled to high resolution some sharpness is lost. As a result, the video looks blurred. “enhancement information” can be combined with up-sampled low resolution video to produce a high quality image for display. The edge enhancement data captures information that is lost when up-sampling low resolution video to high resolution. The edge enhancement information is related to pixel values that correspond to edges within an image. The combination of up-sampling with edge enhancement eliminates the need to store a separate higher resolution video bitstream, and instead only an enhancement stream needs to be stored, which requires fewer bits and therefore much less storage space. This approach is particularly advantageous if the amount of data required for the low resolution video plus edge enhancement information before encoding is less than or equal to the amount of data required for the high resolution video before encoding. Such situations may arise in embodiments wherein high resolution video is down-sampled to low resolution video, and edge enhancement data is created from the high resolution video data before down-sampling. In certain implementations, down-sampling may involve an integer down-sample, e.g., eliminating alternate pixels. An example of an integer down-sample may be down-sampling from 4K (2160P) to 1080P.

In some implementations, the enhancement information may be encoded in an existing video format that a decoder would recognize and an existing encoder would know how to encode. By way of example and not by way of limitation, the enhancement information may be encoded using existing AVC or High Efficiency Video Coding (HEVC) decoders found commonly in devices that support video playback and does not require devices to add support for extensions to the standard. This solution could also be used to reduce the CPU and GPU load for decoding high resolution video bitstreams in devices that include multiple decoders (e.g., both hardware and software-based decoders).

The enhancement stream need not be stored in a video format; however, doing so works quite well with existing hardware. In alternative implementations the edge enhancement information could be encoded, e.g., using JPEG compression or any other arithmetic coding standard.

Up-sampling in conjunction with enhancement information as described herein could also be applied to still images, e.g., where JPEG encoding or some other image compression standard is used to compress both the base and the enhancement information.

Raw video is represented by luminance (intensity) and chrominance (color) values. Many encoders use fewer bits to store the chrominance data so that more bits may be spent on luminance data, as the human eye is more sensitive to differences in luminance than chrominance.

In certain implementations, an enhancement information generation algorithm may analyze the images in video data to find edges within the image. Edge enhancement data may be determined by analyzing the result of comparing an up-sampled version of a low resolution base image to the corresponding original high resolution image and determining the difference between the images. Up-sampling the low resolution image may use a standard algorithm, e.g. bilinear (fastest and lowest quality) or bicubic (better quality but slower). In certain embodiments, this comparison may be performed by the GPU. In alternative embodiments, this comparison may be performed by a CPU. In some cases, there is no edge enhancement information for a frame because there is not a significant difference between the high resolution video and the up-sampled low resolution video. When such a scenario occurs in situations involving high resolution streaming, those frames for which there is not a significant difference could be encoded as original high resolution frames without edge enhancement information. In alternative embodiments, edge enhancement information may be determined as provided below and subsequently compared to a threshold; the result of such a comparison can then be used to determine whether or not to down-sample the frame before encoding.

By way of example, and not by way of limitation, the enhancement information may be generated by determining a difference in pixel values (e.g., chroma values or luma values or both) between the up-sampled low resolution image and the original high resolution image and adding a midpoint for pixel value (e.g., 128 for 8-bit). According to certain aspects of the present disclosure, the enhancement information may be created in such a way as to minimize the arithmetic difference between the input frame and an up-sampled version of the down-sampled frame. As used herein the term ‘difference’ refers to a difference in the mathematical sense, including but not limited to arithmetic difference (i.e., the result of a subtraction). Determining the difference may include other mathematical operations on the pixel values prior to subtraction, such as squaring, taking a square root, multiplying by a scaling factor. Determining the difference may also include mathematical operations on the result of a subtraction. For example, in some implementations negative values resulting from a subtraction may be set to zero, and any values that exceed the maximum value for the number of bits may be set to the maximum value (e.g., for 8-bit pixels, values greater than 255 would be set to 255). Additionally, the same number of bits could be utilized to represent each lower resolution pixel, but fewer bits could be used to represent the edge enhancement data, as a large number of bits might not be needed to represent a small difference. By way of example, and not by way of limitation, a calculated 16-bit difference may have a value reduced to an 8-bit representation. Other examples of generating edge enhancement information include feature and edge detection methods such as using the Sobel operator or Roberts cross operator.

In certain implementations, the difference in the luminance values may be determined without regard for chrominance information that is lost when the up-sampling of the low resolution video is carried out. This frees up computational and memory resources, as no additional chrominance data is saved during the utilization of such a process. This process also increases the efficiency of the encoding process, as computing the differences for the chrominance values is no longer required. By way of example, and not by way of limitation, the luminance information may be stored in the chrominance information in order to manipulate an encoder into encoding some of the luminance information as chrominance information while the rest of the luminance information remains luminance information. On the decoder side, the luminance information stored as chrominance information is moved back to the luminance information and the chrominance information is ignored. Alternatively embodiments allow for the encoding of the chrominance as a flat grey.

In certain alternative embodiments, a filtering stage is added to make the edge enhancement information more suitable for video compression, e.g., by removing noisy pixels. Noisy pixels are, for example, isolated pixels that are of a much different value than surrounding pixels.

In certain implementations, decoding performed on low resolution hardware may involve decoding the low resolution video and taking no action with the enhancement data. In alternative embodiments of the present invention, decoding performed on high resolution hardware may involve decoding the low resolution video and the enhancement data and performing the inverse of the comparison that generated the enhancement data to result in reconstituted high resolution video. The inverse comparison may be performed on either the GPU or the CPU. Up-sampling the data may use bilinear or bicubic according to the algorithm that was used to generate the edge enhancement information.

In alternative embodiments of the present invention, two decoders are utilized. A first decoder may decode low resolution video, and a second decoder may decode the edge enhancement data. In certain embodiments, a hardware decoder may be used for the video and a software decoder may be used for the enhancement data. In alternative embodiments, two instances of a software decoder or two hardware decoders may optionally be utilized. Furthermore, certain alternative embodiments may apply the encoding/decoding processes, methods, and devices described above with respect to audio data.

In other alternative implementations, only one decoder might be utilized. In such cases encoding may be implemented with only one encoder, and the enhancement data may be encoded into the same bitstream as the encoded base video data. The encoded base video data does not reference any frames containing the enhancement data, and can be decoded independently without decoding the enhancement data completely. By way of example, the slice headers could be used to determine if data being decoded corresponds to the base video or the enhancement video, and if the enhancement data is not required, the rest of decoding may be skipped for that frame. In lower powered hardware, only the base video pictures are decoded. In higher powered hardware, all frames are decoded, and the final high resolution frame is reconstructed from the decoded base video and enhancement data.

The above-described processes, methods, and devices may alternatively be used to compress high resolution video for storage, as down-sampling high resolution video and storing the down-sampled video with corresponding edge enhancement data may require less storage space than simply storing the high resolution video.

In certain implementations, the edge enhancement algorithm may be used to determine if any frame has enough detail to be sent at high resolution (e.g., 4K resolution) and, if so, the frame may be encoded at high resolution without down-sampling. By way of example, and not by way of limitation, determining whether a frame has enough detail to be sent at high resolution may use metrics such as variance or a count of the total number of pixels that are not equal to some reference value, e.g., 128, and use thresholds established from empirical data to determine if the enhancement information is significant. If it is determined that the frame does not have enough detail to be sent at 4K resolution, the frame may be down-scaled to a lower resolution (e.g., 1080P), and encoded as a restructured frame containing low resolution pixels surrounded by pixels of uniform chroma and luma values, e.g., flat grey, along with parameters to indicate the section of the frame that contains the lower resolution data. On the decoder side, these restructured frames may be decoded by using the parameters to extract the down-scaled frame and then up-sampling the extracted down-scaled frame. Alternatively, if it is determined that sending the frame at high resolution (e.g., 4K resolution) is inefficient, the frame may be down-scaled to a lower resolution (e.g., 1080P), followed by generation of enhancement information and encoding of base frames and enhancement information. In such implementations, a decoder can up-sample the low resolution frames and combine them with enhancement information and handle the 4K frames normally. By selectively sending some frames as low resolution, these embodiments are capable of reducing the bit stream size.

Before describing up-sampling with edge enhancement in accordance with aspects of the present disclosure, it is useful to understand how digital pictures, e.g., video pictures are encoded for streaming applications. Furthermore, it is useful to understand how encoded streaming digital picture data is decoded, since the manner in which a picture or portion of a picture is encoded affects how it is to be decoded.

By way of example, and not by way of limitation, as shown inFIG. 1A, a single picture100(e.g., a digital video frame) may be broken down into one or more sections. As used herein, the term “section” can refer to a group of one or more pixels within the picture100. A section can range from a single pixel within the picture, up to the whole picture. Non-limiting examples of sections include slices (e.g., macroblock rows)102, macroblocks104, sub-macroblocks106, blocks108and individual pixels110. Other examples include Coding Tree Blocks (CTB) and other section types that are specific to the HEVC (H.265) coding standard. As illustrated inFIG. 1A, each slice102contains one or more rows of macroblocks104or portions of one or more such rows. The number of macroblocks in a row depends on the size of the macroblocks and the size and resolution of the picture100. For example, if each macroblock contains sixteen by sixteen pixels then the number of macroblocks in each row may be determined by dividing the width of the picture100(in pixels) by sixteen. Each macroblock104may be broken down into a number of sub-macroblocks106. Each sub-macroblock106may be broken down into a number of blocks108and each block may contain a number of pixels110. By way of example, and without limitation of the invention, in a common video coding scheme, each 16×16 pixel macroblock104may be broken down into four 8×8 pixel sub-macroblocks106. Each sub-macroblock may be broken down into four blocks108and each block may contain a four by four arrangement of sixteen pixels110.

It is noted that each picture may be either a frame or a field. A frame refers to a complete image. A field is a portion of an image used for to facilitate displaying the image on certain types of display devices. Generally, the pixels in an image are arranged in rows. To facilitate display an image may sometimes be split by putting alternate rows of pixels into two different fields. The rows of pixels in the two fields can then be interlaced to form the complete image. For some display devices, such as cathode ray tube (CRT) displays, the two fields may simply be displayed one after the other in rapid succession. The afterglow of the phosphors or other light emitting elements used to illuminate the pixels in the display combined with the persistence of vision results in the two fields being perceived as a continuous image. For certain display devices, such as liquid crystal displays, it may be necessary to interlace the two fields into a single picture before being displayed. Streaming data representing encoded images typically includes information indicating whether the image is a field or a frame. Such information may be included in a header to the image.

Modern video coder/decoders (codecs), such as MPEG2, MPEG4 and H.264 generally encode video frames as one of three basic types known as Intra-Frames, Predictive Frames and Bipredicitve Frames, which are typically referred to as I-frames, P-frames and B-frames respectively.

An I-frame is a picture coded without reference to any picture except itself. I-frames are used for random access and are used as references for the decoding of other P-frames or B-frames. I-frames may be generated by an encoder to create random access points (to allow a decoder to start decoding properly from scratch at a given picture location). I-frames may be generated when differentiating image details prohibit generation of effective P or B frames. Because an I-frame contains a complete picture, I-frames typically require more bits to encode than P-frames or B-frames. Video frames are often encoded as I-frames when a scene change is detected in the input video.

P-frames require the prior decoding of some other picture(s) in order to be decoded. P-frames typically require fewer bits for encoding than I-frames. A P-frame contains encoded information regarding differences relative to a previous I-frame in decoding order. A P-frame typically references the preceding I-frame in a Group of Pictures (GoP). P-frames may contain both image data and motion vector displacements and combinations of the two. In some standard codecs (such as MPEG-2), P-frames use only one previously-decoded picture as a reference during decoding, and require that picture to also precede the P-frame in display order. In H.264, P-frames can use multiple previously-decoded pictures as references during decoding, and can have any arbitrary display-order relationship relative to the picture(s) used for its prediction.

B-frames require the prior decoding of either an I-frame or a P-frame in order to be decoded. Like P-frames, B-frames may contain both image data and motion vector displacements and/or combinations of the two. B-frames may include some prediction modes that form a prediction of a motion region (e.g., a segment of a frame such as a macroblock or a smaller area) by averaging the predictions obtained using two different previously-decoded reference regions. In some codecs (such as MPEG-2), B-frames are never used as references for the prediction of other pictures. As a result, a lower quality encoding (resulting in the use of fewer bits than would otherwise be used) can be used for such B pictures because the loss of detail will not harm the prediction quality for subsequent pictures. In other codecs, such as H.264, B-frames may or may not be used as references for the decoding of other pictures (at the discretion of the encoder). In H.264, there are two types of B-frame, a reference B-frame and non-reference B-frame. A reference B-frame can be used as a reference frame for B-frame coding and a non-reference B-frame cannot. Some codecs (such as MPEG-2), use exactly two previously-decoded pictures as references during decoding, and require one of those pictures to precede the B-frame picture in display order and the other one to follow it. In other codecs, such as H.264, a B-frame can use one, two, or more than two previously-decoded pictures as references during decoding, and can have any arbitrary display-order relationship relative to the picture(s) used for its prediction. B-frames typically require fewer bits for encoding than either I-frames or P-frames.

As used herein, the terms I-frame, B-frame and P-frame may be applied to any streaming data units that have similar properties to I-frames, B-frames and P-frames, e.g., as described above with respect to the context of streaming video.

By way of example and not by way of limitation, a common bitstream containing encoded base frames and encoded enhancement data may be configured as shown inFIG. 2A. In this example, base frames are coded as I-frames0,13or P-frames1,5,9. The frames for the enhancement data are encoded as B-frames2,3,4,6,7,8,10,11,12,14,15,16. The enhancement frames can use the base frames as reference, as indicated by the arrows, but not vice versa. InFIG. 2A, the I-frames0,13may correspond to the base frame data and the rest of the frames may contain encoded enhancement data. In this example, all of the B-frames are non-reference frames.

In an alternative configuration (not shown) only the I-frames0,13would correspond to the base frame data, while the P-frames1,5,9and B-frames2,3,0,7,8,10,11,12,14,15,16would correspond to the enhancement data. In such an implementation, the P-frames may be used as reference frames for the B-frames.

FIG. 2Billustrates an alternative configuration for a common bitstream in accordance with aspects of the present disclosure. In this example the bitstream is created using a hierarchical-B structure in which certain B-frames2,6,10,14are used as reference for other B-frames. In particular, frame2may be used as a reference for frames3and4, frame6may be used as a reference for frames7and8, frame10may be used as a reference for frames11and12and frame14may be used as a reference for frames15and16. In this example B-frames can be skipped without affecting the decoding of the base frames0,1,5,9,13.

By way of example, and not by way of limitation, digital pictures may be encoded according to a generalized method200as illustrated inFIG. 2C. The encoder receives a plurality of digital images201and encodes each image. Encoding of the digital picture201may proceed on a section-by-section basis. The encoding process for each section may optionally involve padding202, image compression204and motion compensation206. To facilitate a common process flow for both intra-coded and inter-coded pictures, all un-decoded pixels within a currently processing picture201may be padded with temporary pixel values to produce a padded picture, as indicated at202. The padding may proceed, e.g., as described above in U.S. Pat. No. 8,711,933, which is incorporated herein by reference. The padded picture may be added to a list of reference pictures203stored in a buffer. Padding the picture at202facilitates the use of a currently-processing picture as a reference picture in subsequent processing during image compression204and motion compensation206. Such padding is described in detail in commonly-assigned U.S. Pat. No. 8,218,641, which is incorporated herein by reference.

As used herein, image compression refers to the application of data compression to digital images. The objective of the image compression204is to reduce redundancy of the image data for a give image201in order to be able to store or transmit the data for that image in an efficient form of compressed data. The image compression204may be lossy or lossless. Lossless compression is sometimes preferred for artificial images such as technical drawings, icons or comics. This is because lossy compression methods, especially when used at low bit rates, introduce compression artifacts. Lossless compression methods may also be preferred for high value content, such as medical imagery or image scans made for archival purposes. Lossy methods are especially suitable for natural images such as photos in applications where minor (sometimes imperceptible) loss of fidelity is acceptable to achieve a substantial reduction in bit rate.

Examples of methods for lossless image compression include, but are not limited to Run-length encoding—used as default method in PCX and as one of possible in BMP, TGA, TIFF, Entropy coding, adaptive dictionary algorithms such as LZW—used in GIF and TIFF and deflation—used in PNG, MNG and TIFF. Examples of methods for lossy compression include reducing the color space of a picture201to the most common colors in the image, Chroma subsampling, transform coding, and fractal compression.

In color space reduction, the selected colors may be specified in the color palette in the header of the compressed image. Each pixel just references the index of a color in the color palette. This method can be combined with dithering to avoid posterization. Chroma subsampling takes advantage of the fact that the eye perceives brightness more sharply than color, by dropping half or more of the chrominance information in the image. Transform coding is perhaps the most commonly used image compression method. Transform coding typically applies a Fourier-related transform such as a discrete cosine transform (DCT) or the wavelet transform, followed by quantization and entropy coding. Fractal compression relies on the fact that in certain images, parts of the image resemble other parts of the same image. Fractal algorithms convert these parts, or more precisely, geometric shapes into mathematical data called “fractal codes” which are used to recreate the encoded image.

The image compression204may include region of interest coding in which certain parts of the image201are encoded with higher quality than others. This can be combined with scalability, which involves encoding certain parts of an image first and others later. Compressed data can contain information about the image (sometimes referred to as meta information or metadata) which can be used to categorize, search or browse images. Such information can include color and texture statistics, small preview images and author/copyright information.

By way of example, and not by way of limitation, during image compression at204the encoder may search for the best way to compress a block of pixels. The encoder can search all of the reference pictures in the reference picture list203, including the currently padded picture, for a good match. If the current picture (or subsection) is coded as an intra picture, (or subsection) only the padded picture is available in the reference list. The image compression at204produces a motion vector MV and transform coefficients207that are subsequently used along with one or more of the reference pictures (including the padded picture) during motion compensation at206.

The image compression204generally includes a motion search MS for a best inter prediction match, an intra search IS for a best intra prediction match, an inter/intra comparison C to decide whether the current macroblock is inter-coded or intra-coded, a subtraction S of the original input pixels from the section being encoded with best match predicted pixels to calculate lossless residual pixels205. The residual pixels then undergo a transform and quantization XQ to produce transform coefficients207. The transform is typically based on a Fourier transform, such as a discrete cosine transform (DCT).

The transform outputs a set of coefficients, each of which is a weighting value for a standard basis pattern. When combined, the weighted basis patterns re-create the block of residual samples. The output of the transform, a block of transform coefficients, is quantized, i.e. each coefficient is divided by an integer value. Quantization reduces the precision of the transform coefficients according to a quantization parameter (QP). Typically, the result is a block in which most or all of the coefficients are zero, with a few non-zero coefficients. Setting QP to a high value means that more coefficients are set to zero, resulting in high compression at the expense of poor decoded image quality. For a low QP value, more non-zero coefficients remain after quantization, resulting in better decoded image quality but lower compression. Conversely, for a high QP value, fewer non-zero coefficients remain after quantization, resulting in higher image compression but lower image quality.

The inter/intra comparison C, also referred to as a mode decision uses a parameter known as a Lagrange multiplier λ that is related to QP. Basically, a cost function J is computed using a value of λ determined from the value of QP. The encoding mode is determined based on whether the computed cost function J for inter mode coding is above or below a computed cost for intra mode coding. By way of example, the H.264/AVC codec supports a cost function JH, which should be minimized by computing the actual bit consumption R for encoding the overhead (e.g., motion vectors, types) of the section and the reconstruction distortion D (measured, e.g., as a sum of absolute differences, SAD between the original and reconstructed sections). In such a case, the cost function JHis computed according to
JH=D+λ·R.

In alternative implementations, the distortion D may be calculated differently. There are many ways to represent the distortion, for example, sum of squared differences (SSD), sum of absolute transformed differences (SATD), mean absolute difference (MAD), and the like. Those skilled in the art will recognize that for different distortion measures, the cost function has to be modified or re-tuned accordingly.

Under some circumstances, an improper encoding mode decision can trigger an unnecessary IDR or I-frame insertion. Consider an example of steaming video during online video gaming. The encoder tries to meet a target bit rate for the video stream that is generated by a game application. The target bit rate is related to the number of bits per frame. If the game is paused, the video is essentially a stream of still frames. For a still frame, the QP is low in order to meet the target bits for the frame in rate distortion optimization process. When QP is low, the mode decision selects intra coding for most sections (e.g., macroblocks) in a still frame. If the number of intra-coded sections in a frame is above a threshold the codec triggers a scene-change detection and the next frame is coded as an intra frame with extremely low QP, which requires a large number of bits to encode. This is due to the fact that extremely low values of QP (e.g., QP=1, 2) implies nearly lossless coding in this case. By way of example, and not by way of limitation, the threshold for triggering scene change detection may be about 60-80% intra MB in a frame. A sequence of still frames causes in a sequence of scene change detections even though basically the same frame is being repeated. The sequence of intra-frames can cause large and frequent spikes of bit rate usage in a bandwidth-limited communication channel.

Normally relationship between λ and QP is fixed by the codec and is the same for all pictures. According to aspects of the present disclosure, the relationship between λ and QP can be adjusted from picture to picture depending on the number of bits per section in a picture.

According to aspects of the present disclosure, the relation between λ and QP can be adapted based on the number of bits per section so that the encoding mode decision can be configured in a way that reduces the likelihood of unnecessary IDR or I-frame insertion.

According to aspects of the present disclosure, the relationship between λ and QP may be selectively adjusted during encoding, e.g., at the beginning of encoding of a video stream or the beginning of each video frame in a stream in a way that makes it more likely for the section encoding mode decision to result in an “inter” coding decision instead of an “intra” coding mode.

In some implementations it is even possible to change the λ versus QP relationship for each section if there are different size sections in a frame, e.g., as is possible in H.265. This could be beneficial, e.g., in two-pass encoding use cases since the first pass would provide more insight about the content of the picture sections so that better coding mode decisions can be made.

By way of example, and not by way of limitation, adjustment to the relationship between λ and QP may depend on the number of bits in a section (NBS), which generally depends on the target bitrate (e.g., in bits per second), the frame rate (e.g., in frames per second), and the number of sections in a frame. The number of bits in a section NBS can be calculated by dividing the target bitrate BR by the product of the frame rate FR and the number of sections per frame NSF. By way of example, and not by way of limitation, this can be expressed as:
NBS=BR/(FR·NSF)

More generally, the number of bits per section (NBS) may be more broadly expressed as

NBS=(BPF)/(NSF), where BPF is the target number of bits per frame.

This broadened expression allows for the possibility that the value of NBS could be different from frame to frame, depending, e.g., on the target bits allocated by the underlying rate control scheme. In the case of a fixed target number of bits for each frame BPF becomes BR/FR.

The number of sections (e.g., MB) per frame depends on the resolution. The change to the table can be triggered by a combination of resolution, frame rate, and bit rate. E.g., a table change would be triggered for a frame having 960 by 540 resolution, a frame rate of 30 fps, and a target rate of 8-10 Mbps or higher. For a given bitrate and frame rate a table change is less likely to be triggered if the resolution increases. For a given bitrate and resolution a table change is less likely to be triggered if the frame rate increases. For a given frame rate and resolution a table change is less likely to be triggered if the bitrate decreases.

The relationship between λ and QP is typically nonlinear. Generally, when QP is high λ is high and when QP is low λ is low. Examples of relationships between λ and QP are described in U.S. patent application Ser. No. 14/493,238 filed Sep. 22, 2014 and published as U.S. Patent Application Publication Number 2016/0088299, the entire contents of which are been incorporated herein by reference.

The QP value can be adjusted depending on the target bitrate. Since QP controls bit usage in encoding, many encoding programs utilize a rate controller that adjusts QP in order to achieve a desired bitrate. The encoder receives uncompressed source data (e.g., an input video) and produces compressed output. The video coding method typically uses a QP value that affects the bit usage for encoding a video section and therefore affects the bitrate. Generally, lower bit usage results in a higher bitrate. A rate controller determines a QP value based on a demanded bitrate, which may be specified by an external application. The encoder uses the QP value determined by the rate controller and determines the actual resulting bit usage and bit rate. The rate controller can use the actual bit rate to adjust the QP value in a feedback loop.

A relationship between the bitrate and the value of the QP depends partly on the complexity of the image being. The bitrate versus QP relationship can be expressed in terms of a set of curves with different curves for different levels of complexity. The heart of the algorithm implemented by the rate controller is a quantitative model describing a relationship between QP, actual bitrate and some measure of complexity. The relevant bitrate and complexity are generally associated only with the differences between source pixels and predicted pixels (often referred to as residuals) because the quantization parameter QP can only influence the detail of information carried in the transformed residuals.

Complexity generally refers to amount of spatial variation within a picture or part of the picture. On a local level, e.g., block or macroblock level, the spatial variation may be measured by the variance of the pixel values within the relevant section. However, for a video sequence, complexity may also relate to the temporal variation of a scene of a sequence of images. For example, a video sequence consists of one object having substantial spatial variation that translates slowly across the field of view, may not require very many bits because temporal prediction can easily capture the motion using a single reference picture and a series of motion vectors. Although it is difficult to define an inclusive video complexity metric that is also easy to calculate, the Mean Average Difference (MAD) of the prediction error (difference between source pixel value and predicted pixel value) is often used for this purpose.

It is noted that the quantization parameter QP may be determined from multiple factors including, but not limited to the picture type of the source picture, a complexity of the source picture, an estimated target number of bits and an underlying rate distortion model. For example, QP may be determined on a section-by-section basis using a variation for a section of the currently encoding picture, e.g., a section (e.g., MB) variance. Alternatively, QP for a currently encoding section may be determined using an actual bit count for encoding a co-located section (e.g., MB) in a previous frame. Examples of such QP level calculations are described, e.g., in commonly assigned U.S. Patent Application Publication No. 2011/0051806, now U.S. Pat. No. 8,879,623 to Hung-Ju Lee, which is incorporated herein by reference.

Motion search and prediction depend on the type of picture being encoded. Referring again toFIG. 2C, if an intra picture is to be coded, the motion search MS and inter/intra comparison C are turned off. However, in embodiments of the present invention, since the padded picture is available as a reference, these functions are not turned off. Consequently, the image compression204is the same for intra-coded pictures and inter-coded pictures.

The motion search MS may generate a motion vector MV by searching the picture201for a best matching block or macroblock for motion compensation as is normally done for an inter-coded picture. If the current picture201is an intra-coded picture, by contrast, existing codecs typically do not allow prediction across pictures. Instead all motion compensation is normally turned off for an intra picture (e.g., I-frame) and the picture coded by generating transform coefficients and performing pixel prediction. In some implementations, however, an intra picture may be used to do inter prediction by matching a section in the current picture to another offset section within that same picture. The offset between the two sections may be coded as a motion vector MV′ that can be used that for motion compensation at206. By way of example, the encoder may attempt to match a block or macroblock in an intra picture with some other offset section in the same picture then code the offset between the two as a motion vector. The codec's ordinary motion vector compensation for an “inter” picture may then be used to do motion vector compensation on an “intra” picture. Certain existing codecs have functions that can convert an offset between two blocks or macroblocks into a motion vector, which can be followed to do motion compensation at206. However, these functions are conventionally turned off for encoding of intra pictures. In embodiments of the present invention, the codec may be instructed not to turn off such “inter” picture functions for encoding of intra pictures.

As used herein, motion compensation refers to a technique for describing a picture in terms of the transformation of a reference image to a currently processing image. In general, the motion compensation206acts as a local decoder within the encoder implementing the encoding process200. Specifically, the motion compensation206includes inter prediction IP1and (optionally) intra prediction IP2to get predicted pixels PP using the motion vector MV or MV′ from the image compression204and reference pixels from a picture in the reference list. Inverse quantization and inverse transformation IQX using the transform coefficients207from the image compression204produce lossy residual pixels205L which are added to the predicted pixels PP to generate decoded pixels209. The decoded pixels209are inserted into the reference picture and are available for use in image compression204and motion compensation206for a subsequent section of the currently-processing picture201. After the decoded pixels have been inserted, un-decoded pixels in the reference picture may undergo padding202.

In some encoder implementations, if the current picture is intra coded, the inter-prediction portions of motion compensation206are turned off because there are no other pictures that can be used for motion compensation. Alternatively, motion compensation may be performed on any picture201independent of whether a particular picture is to be inter-coded or intra-coded. In some implementations, the encoder implementing may be modified to add the padded picture to the reference picture list203and the inter-prediction portions of the motion compensation206are not turned off, even if the currently processing image is to be intra coded. As a result, the process flow for both inter coded sections and intra coded sections is the same during motion compensation206. The only major difference is the selection of the reference picture to be used for encoding. It is noted that in some implementations, motion-compensation need not be performed on all pictures, and padded pictures need not be added to the reference picture list.

By way of example, and not by way of limitation, in one type of motion compensation, known as block motion compensation (BMC), each image may be partitioned into blocks of pixels (e.g. macroblocks of 16×16 pixels). Each block is predicted from a block of equal size in the reference frame. The blocks are not transformed in any way apart from being shifted to the position of the predicted block. This shift is represented by a motion vector MV. To exploit the redundancy between neighboring block vectors, (e.g. for a single moving object covered by multiple blocks) it is common to encode only the difference between a current and previous motion vector in a bit-stream. The result of this differencing process is mathematically equivalent to a global motion compensation capable of panning. Further down the encoding pipeline, the method200may optionally use entropy coding208to take advantage of the resulting statistical distribution of the motion vectors around the zero vector to reduce the output size.

It is possible to shift a block by a non-integer number of pixels, which is called sub-pixel precision. The in-between pixels are generated by interpolating neighboring pixels. Commonly, half-pixel or quarter pixel precision is used. The computational expense of sub-pixel precision is much higher due to the extra processing required for interpolation and on the encoder side, a much greater number of potential source blocks to be evaluated.

Block motion compensation divides up a currently encoding image into non-overlapping blocks, and computes a motion compensation vector that indicates where those blocks come from in a reference image. The reference blocks typically overlap in the source frame. Some video compression algorithms assemble the current image out of pieces of several different reference images in the reference image list203.

The result of the image compression204and motion compensation206and (optionally) entropy coding208is a set of data211referred to for convenience as a coded picture. The motion vector MV, (and/or intra prediction mode motion vector MV′) and transform coefficients207may be included in the coded picture211. Once a digital picture or other form of streaming data has been encoded, the encoded data may be transmitted and then decoded.

FIG. 3illustrates an example of a possible process flow in a method300for decoding of streaming data301that may be used in conjunction with aspects of the present disclosure. This particular example shows the process flow for video decoding, e.g., using the AVC (H.264) standard. The coded streaming data301may initially be stored in a buffer. Where coded streaming data301(e.g., a video data bitstream) has been transferred over a network, e.g., the Internet, the data301may initially undergo a process referred to as network abstraction layer (NAL) decoding, indicated at302. The Network Abstraction Layer (NAL) is a part of streaming data standards, such as the H.264/AVC and HEVC video coding standards. The main goal of the NAL is the provision of a “network-friendly” representation of streaming data for “conversational” (e.g., video telephony) and “non-conversational” (storage, broadcast, or streaming) applications. NAL decoding may remove from the data301information added to assist in transmitting the data. Such information, referred to as a “network wrapper” may identify the data201as video data or indicate a beginning or end of a bitstream, bits for alignment of data, and/or metadata about the video data itself.

In addition, by way of example, the network wrapper may include information about the data301including, e.g., resolution, picture display format, color palette transform matrix for displaying the data, information on the number of bits in each picture, slice or macroblock, as well as information used in lower level decoding, e.g., data indicating the beginning or ending of a slice. This information may be used to determine the number of macroblocks to pass to each of the task groups in a single section. Due to its complexity, NAL decoding is typically done on a picture and slice level. The smallest NAL buffer used for NAL decoding is usually slice sized. The example illustrated inFIG. 3is described in terms of macroblocks and the AVC (H.265) standard. However, these are not limiting features of aspects of the present disclosure. For example, in the latest H265 (HEVC) standard, there is no macroblock concept. Instead, more flexible Coding Unit (CU), Prediction Unit, (PU), Transform Unit (TU) concepts are introduced. Aspects of the present disclosure may operate in conjunction with such coding standards.

In some embodiments, after NAL decoding at302, the remaining decoding illustrated inFIG. 3may be implemented in three different thread groups or task groups referred to herein as video coded layer (VCL) decoding304, motion vector (MV) reconstruction310and picture reconstruction314. The picture reconstruction task group214may include pixel prediction and reconstruction316and post processing320. In some embodiments of the present invention, these tasks groups may be chosen based on data dependencies such that each task group may complete its processing of all the macroblocks in a picture (e.g., frame or field) or section before the macroblocks are sent to the next task group for subsequent processing.

Certain coding standards may use a form of data compression that involves transformation of the pixel information from a spatial domain to a frequency domain. One such transform, among others, is known as a discrete cosine transform (DCT). The decoding process for such compressed data involves the inverse transformation from the frequency domain back to the spatial domain. In the case of data compressed using DCT, the inverse process is known as inverse discrete cosine transformation (IDCT). The transformed data is sometimes quantized to reduce the number of bits used to represent numbers in the discrete transformed data. For example, numbers 1, 2, 3 may all be mapped to 2 and numbers 4, 5, 6 may all be mapped to 5. To decompress the data a process known as inverse quantization (IQ) is used before performing the inverse transform from the frequency domain to the spatial domain. The data dependencies for the VCL IQ/IDCT decoding process304are typically at the macroblock level for macroblocks within the same slice. Consequently results produced by the VCL decoding process304may be buffered at the macroblock level.

VCL decoding304often includes a process referred to as Entropy Decoding306, which is used to decode the VCL syntax. Many codecs, such as AVC(H.264), use a layer of encoding referred to as entropy encoding. Entropy encoding is a coding scheme that assigns codes to signals so as to match code lengths with the probabilities of the signals. Typically, entropy encoders are used to compress data by replacing symbols represented by equal-length codes with symbols represented by codes proportional to the negative logarithm of the probability. AVC(H.264) supports two entropy encoding schemes, Context Adaptive Variable Length Coding (CAVLC) and Context Adaptive Binary Arithmetic Coding (CABAC). Since CABAC tends to offer about 10% more compression than CAVLC, CABAC is favored by many video encoders in generating AVC(H.264) bitstreams. Decoding the entropy layer of AVC(H.264)-coded data streams can be computationally intensive and may present challenges for devices that decode AVC(H.264)-coded bitstreams using general purpose microprocessors. For this reason, many systems use a hardware decoder accelerator.

In addition to Entropy Decoding306, the VCL decoding process304may involve inverse quantization (IQ) and/or inverse discrete cosine transformation (IDCT) as indicated at308. These processes may decode the headers309and data from macroblocks. The decoded headers309may be used to assist in VCL decoding of neighboring macroblocks.

VCL decoding304may be implemented at a macroblock level data dependency frequency. Specifically, different macroblocks within the same slice may undergo VCL decoding in parallel and the results may be sent to the motion vector reconstruction task group210for further processing.

Subsequently, all macroblocks in the picture or section may undergo motion vector reconstruction310. The MV reconstruction process310may involve motion vector reconstruction312using headers from a given macroblock311and/or co-located macroblock headers313. A motion vector describes apparent motion within a picture. Such motion vectors allow reconstruction of a picture (or portion thereof) based on knowledge of the pixels of a prior picture and the relative motion of those pixels from picture to picture. Once the motion vector has been recovered pixels may be reconstructed at316using a process based on residual pixels from the VCL decoding process304and motion vectors from the MV reconstruction process310. The data dependency frequency (and level of parallelism) for the MV depends on whether the MV reconstruction process310involves co-located macroblocks from other pictures. For MV reconstruction not involving co-located MB headers from other pictures the MV reconstruction process310may be implemented in parallel at the slice level or picture level. For MV reconstruction involving co-located MB headers the data dependency frequency is at the picture level and the MV reconstruction process310may be implemented with parallelism at the slice level.

The results of motion vector reconstruction310are sent to the picture reconstruction task group314, which may be parallelized on a picture frequency level. Within the picture reconstruction task group314all macroblocks in the picture or section may undergo pixel prediction and reconstruction316in conjunction with de-blocking320. The pixel prediction and reconstruction task316and the de-blocking task320may be parallelized to enhance the efficiency of decoding. These tasks may be parallelized within the picture reconstruction task group314at a macroblock level based on data dependencies. For example, pixel prediction and reconstruction316may be performed on one macroblock and followed by de-blocking320. Reference pixels from the decoded picture obtained by de-blocking320may be used in pixel prediction and reconstruction316on subsequent macroblocks. Pixel prediction and reconstruction318produces decoded sections319(e.g. decoded blocks or macroblocks) that include neighbor pixels which may be used as inputs to the pixel prediction and reconstruction process318for a subsequent macroblock. The data dependencies for pixel prediction and reconstruction316allow for a certain degree of parallel processing at the macroblock level for macroblocks in the same slice.

The post processing task group320may include a de-blocking filter322that is applied to blocks in the decoded section319to improve visual quality and prediction performance by smoothing the sharp edges which can form between blocks when block coding techniques are used. The de-blocking filter322may be used to improve the appearance of the resulting de-blocked sections324.

The decoded section319or de-blocked sections324may provide neighboring pixels for use in de-blocking a neighboring macroblock. In addition, decoded sections319including sections from a currently decoding picture may provide reference pixels for pixel prediction and reconstruction318for subsequent macroblocks. It is during this stage that pixels from within the current picture may optionally be used for pixel prediction within that same current picture as described above, independent of whether the picture (or subsections thereof) is inter-coded or intra-coded. De-blocking320may be parallelized on a macroblock level for macroblocks in the same picture.

The decoded sections319produced before post processing320and the post-processed sections324may be stored in the same buffer, e.g., the output picture buffer depending on the particular codec involved. It is noted that de-blocking is a post processing filter in H.264. Because H.264 uses pre-de-blocking macroblock as reference for neighboring macroblocks intra prediction and post-de-blocking macroblocks for future picture macroblocks inter prediction. Because both pre- and post-de-blocking pixels are used for prediction, the decoder or encoder has to buffer both pre-de-blocking macroblocks and post-de-blocking macroblocks. For most low cost consumer applications, pre-de-blocked pictures and post-de-blocked pictures share the same buffer to reduce memory usage. For standards that pre-date H.264, such as MPEG2 or MPEG4 except MPEG4 part10, (note: H.264 is also called MPEG4 part10), only pre-post-processing macroblocks (e.g., pre-de-blocking macroblocks) are used as reference for other macroblock prediction. In such codecs, a pre-filtered picture may not share the same buffer with a post filtered picture.

Thus, for H.264, after pixel decoding, the decoded section319is saved in the output picture buffer. Later, the post processed sections324replace the decoded sections319in the output picture buffer. For non-H.264 cases, the decoder only saves decoded sections319in the output picture buffer. The post processing is done at display time and the post processing output may not share the same buffer as the decoder output picture buffer.

Up-Sampling With Edge Enhancement

Aspects of the present disclosure provide methods to implement up-sampling with enhancement information in encoding and decoding of digital pictures. By way of example, and not by way of limitation,FIG. 4Aillustrates a method400of encoding video and edge enhancement streams in accordance with aspects of the present disclosure. The method400begins by receiving or generating high resolution video410. The high resolution video410may be down-sampled at412, resulting in base video413. From this high resolution video410, and optionally using base video413, enhancement information415may be generated at414. Base video413may then be encoded, resulting in base data417. Base data417may subsequently be stored or transmitted at420. The enhancement information415may be encoded at418, compressed into an enhancement data at419, and then stored or transmitted at422. Depending on the way the enhancement information415is packed, there could be more frames in the enhancement data419than in the base data417or vice versa.

In some implementations, down-sampling the high resolution video at412may include determining whether there is enough detail in a given frame or frames to justify sending them as high resolution frames and, if not, such frames may be encoded as reconstructed frames corresponding to given frames down-sampled to lower resolution down-sampled frames with fewer pixels than the given frames. The restructured frames may include lower resolution pixels of the down-sampled frames surrounded by pixels of uniform pixel values along with parameters indicating which sections of the restructured frames contain the low resolution pixels.

In some implementations, the edge enhancement information415may be compared to a threshold to determine whether generating the enhancement information would be sufficiently advantageous in terms of reduced bit usage. By way of example, it might not be advantageous to down-sample a frame and generate edge enhancement information where it is known that the frame will be decoded by hardware capable of decoding high resolution frames and the bitrate savings gained by down-sampling and edge enhancement are not great enough to justify the computational cost of doing so. If generating the edge enhancement information would not be advantageous for a given frame, that frame may be encoded as an original high resolution frame without down-sampling and generating edge enhancement information.

For the purposes of example single stage of down-sampling412and enhancement information generation414are shown inFIG. 4A. However, aspects of the present disclosure include implementations that use multiple stages of down-sampling and enhancement information generation to generate the base data417and enhancement data419. By way of example and not by way of limitation, multi-stage down-sampling may involve a cascade scheme, in which one or more high resolution input frames are down-sampled to lower resolutions in two or more stages. The down-sampled frame generated at a given stage serves as the input frame for the next stage. Enhancement information can be generated at each stage of down-sampling from a difference between the input frame for that stage and an up-sampled version of the down-sampled frame generated at that stage. The lowest resolution frame or frames generated in the last stage cascade are then encoded to create the set of base data417and the enhancement information created at each stage may be encoded to generate multiple sets of enhancement data419. Encoding for such a cascade scheme may be implemented with multiple encoders, e.g., one or more encoders for encoding the lowest resolution frame and two or more additional encoders, with a separate one for encoding the enhancement information created at each stage of the cascade. The base data419may then be transmitted or stored for later decoding and regeneration of the high resolution image. Network bandwidth limitations may determine which sets of enhancement data (if any) are sent with the base data for subsequent decoding to regenerate higher resolution frames after base data417is decoded.

As a more specific example, a 4K frame may be down-sampled from 4K to 1080P and first stage enhancement information may be generated from a difference between a version of the 1080P frame up-sampled to 4K and the original 4K frame. The resulting 1080P frame may then be down-sampled to 540P and second stage enhancement information may be generated from a difference between a version of the 540P frame up-sampled to 1080P and the 1080P frame. The 540P frame may then be down-sampled to 270P and third stage enhancement information generated from a difference between a version of the 270P frame up-sampled to 540P and the 540P frame. The encoder may then encode the 270P frame to generate the base data and encode the first, second, and third stage enhancement information to generate the enhancement data419.

FIG. 4Billustrates an example method400B of decoding video and edge enhancement streams on low powered hardware in accordance with aspects of the present disclosure. Base data417and compressed enhancement data419may be received by a decoder at424. In this example, the hardware receiving the aforementioned base data417and enhancement data419is low powered, and is unable to effectively display or output high resolution video. Accordingly, the enhancement stream419may be ignored at426. The base data417may then be decoded at428, and the resulting uncompressed base video413B may then be stored or displayed at430.

FIG. 4Cillustrates an example method400C of decoding video and edge enhancement streams on high powered hardware in accordance with aspects of the present disclosure. Base data417and compressed enhancement stream419may be received by a decoder at424. In this example, the hardware receiving the streams417and419is high powered, and is accordingly capable of effectively displaying or outputting high resolution video. Thus, the base data417may be decoded at428to provide uncompressed base video413B, and the compressed enhancement stream419may be decoded at430to provide uncompressed enhancement data415′. The uncompressed base video413B and uncompressed enhancement data415′ may then be reconstructed at432to provide high resolution output433. This high resolution output may then be displayed or stored at434.

As noted above, the enhancement data419may include multiple stages of enhancement data for generating multiple resolutions of a frame. The decoder may determine which sets of enhancement data419to decode and use to regenerate higher resolution output433after decoding the base data419using the reverse process of the cascade scheme described above to generate multiple levels of enhancement information. Specifically, the decoded base data may be up-sampled and combined with enhancement data at two or more stages in a staircase fashion with the result of up-sampling and combination at a given step of the staircase being used as the input for the next step.

Decoding base data417and multi-stage enhancement data419encoded using a cascade scheme, as described above, may be implemented with multiple decoders, e.g., one or more decoders for decoding the base data and two or more additional decoders, with a separate one for decoding the enhancement data created at each stage of the cascade.

As a more specific example, consider the case described above with respect to the multi-stage encoding. Decoding the base data produces a 270P frame, which may be up-sampled to 540P and combined with third-stage enhancement information to generate a 540P frame. The 540P frame may be up-sampled and combined with second stage enhancement information to generate a 1080P frame. The 1080P frame may be up-sampled to 4K and combined with first stage enhancement information to generate a 4K frame. The decoder may decide at which stage to stop the up-sampling and combination.

In some implementations, the base data417may include encoded restructured frames corresponding to original given high-resolution frames down-sampled to lower resolution frames with fewer pixels than the given frames. The number of pixels in the restructured frame is the same as in the original given high resolution frame, but only a portion of the restructured frame contains pixels that correspond to the original high resolution frame. The restructured frames include lower resolution pixels of the down-sampled frames surrounded by pixels of uniform pixel values along with code-specific parameters indicating which sections of the restructured frames contain the low resolution pixels. In such cases, decoding the base data at428may include decoding such restructured frames by using codec specific parameters to extract the low resolution pixels of the down-scaled frame from the encoded restructured frame and then up-sampling the resulting extracted down-scaled frame.

In some implementations, the data417,419, e.g., base data417may include one or more frames encoded as original high resolution frames without down-sampling or enhancement information. Such frames may be decoded as would normally be done for original high resolution frames.

Aspects of the present disclosure are not limited to implementations in which the enhancement information is used for an entire frame. In some implementations, a decoder may decode only a subset of the enhancement data419. The resulting enhancement information may be used to enhance only a portion of the high resolution frame resulting from up-sampling the base frame, e.g., a portion corresponding to a ‘region of interest.’ In another alternative implementation within the scope of the present disclosure the decoder may decode the enhancement data completely, but in the reconstruction stage, the enhancement is only performed for the region of interest.

Aspects of the present disclosure include systems configured to implement up-sampling with edge enhancement in encoding and decoding of digital pictures, as described above. By way of example, and not by way of limitation,FIG. 5illustrates a block diagram of a computer system500that may be used to implement aspects of the present disclosure. According to aspects of the present disclosure, the system500may be an embedded system, mobile phone, personal computer, tablet computer, portable game device, workstation, game console, and the like. The system500generally may include a processor module501and a memory502. The processor module501may include one or more processor cores, e.g., in single core, dual core, quad core, processor-coprocessor, CPU-GPU, or Cell processor architectures.

The memory502may be in the form of an integrated circuit, e.g., RAM, DRAM, ROM, and the like. The memory may also be a main memory that is accessible by all of the processor cores in the processor module501. In some embodiments, the processor module501may have local memories associated with one or more processor cores or one or more co-processors. A codec program503may be stored in the main memory502in the form of processor readable instructions that can be executed on the processor module501. The codec503may be configured to encode digital pictures. By way of example, and not by way of limitation, the codec503may be configured to encode digital pictures or sections of digital pictures as discussed above with respect toFIG. 2A. The codec503may also be configured to decode encoded digital pictures, e.g., as described above with respect toFIG. 3. A down-sampling and enhancement program504may generate base frames505A and enhancement information505B from input high resolution frames507in conjunction with the codec503, as described with respect toFIG. 4A. An up-sampling and enhancement program506may implement decoder side regeneration of high resolution frames507from decoded base frames505A and enhancement information505B, e.g., as discussed above with respect toFIG. 4B. The up-sampling and enhancement program506may alternatively ignore the enhancement information505B, e.g., as described above with respect toFIG. 4C. The codec503and down-scaling and enhancement program504and up-sampling and enhancement program506may be written in any suitable processor readable language, e.g., C, C++, JAVA, Assembly, MATLAB, FORTRAN and a number of other languages.

Input or output high resolution frames507may be stored in memory502. The memory may also store other relevant parameters for encoding and/or decoding frames, such as λ versus QP table that remains fixed during the encoding of a picture or over the course of encoding multiple pictures. During execution of the codec503, programs504,506, portions of program code, base frames505A, enhancement information505B and/or high resolution frames507may be loaded into the memory502or the local stores of processor cores for processing by the processor501. By way of example, and not by way of limitation, the high resolution frames507may include input frames (e.g., video or audio frames), or sections thereof, before encoding or decoding or at intermediate stages of encoding or decoding. In the case of encoding, the data corresponding to the input high resolution frames507may include buffered portions of streaming data, e.g., unencoded frames or portions thereof. In the case of decoding, the base data505A and enhancement data505B may include input data in the form of un-decoded sections, sections that have been decoded, but not post-processed and sections that have been decoded and post-processed. Such input data may include data packets containing data representing one or more coded sections of one or more digital pictures, coded digital audio digital frames, or other coded streaming data. By way of example, and not by way of limitation, such data packets may include a set of transform coefficients and a partial set of prediction parameters. These various sections may be stored in one or more buffers. In particular, decoded and/or post processed sections may be stored in an output picture buffer implemented in the memory502.

The system500may also include well-known support functions510, such as input/output (I/O) elements511, power supplies (P/S)512, a clock (CLK)513and cache514. The apparatus500may optionally include a mass storage device515such as a disk drive, CD-ROM drive, tape drive, or the like to store programs and/or data. The device800may also optionally include a display unit516and user interface unit518to facilitate interaction between the apparatus500and a user. The display unit516may be in the form of a cathode ray tube (CRT) or flat panel screen that displays text, numerals, graphical symbols or images. The user interface518may include a keyboard, mouse, joystick, light pen, or other device that may be used in conjunction with a graphical user interface (GUI). The apparatus500may also include a network interface520to enable the device to communicate with other devices over a network522, such as the interne. These components may be implemented in hardware, software, or firmware, or some combination of two or more of these.

By way of example, and not by way of limitation, the system500may transmit encoded or unencoded streaming data to other devices connected to the network522or receive encoded or unencoded streaming data from such devices via the network interface520. In a particular implementation, encoded streaming data in the form of one or more encoded sections of a digital picture and/or one or more frames of encoded video may be transmitted from the system over the network522. To implement transmitting or receiving streaming data, the processor module may execute instructions implementing a network protocol stack.

By way of example, and not by way of limitation, digital pictures may be generated with a digital camera, which may be part of the user interface518or which may be a separate peripheral coupled to the system500, e.g., via the I/O elements511. According to some aspects, the digital pictures may be generated by a software application executed by the processor module501.

Aspects of the present disclosure allow for lower bit usage than for conventional encoding of high resolution frames. In particular, down-sampling input digital frames, creating enhancement information, encoding the down-sampled frames and enhancement information as described herein generates the base stream and enhancement stream using fewer bits than by down-sampling the same input digital frames and encoding both the down-sampled frames and the one-or more input digital frames into first and second streams, as is conventionally done. Such lower bit usage is particularly advantageous when transmitting frames over networks with limited bandwidth. By way of example and not by way of limitation, encoding high resolution video through down-sampling and then up-sampling with edge enhancement can produce better quality video upon decoding than pure up-sampling.

For some cases, the type of encoding and decoding described herein allows a better quality even compared to the original high resolution video. The graph depicted inFIG. 6shows experimental results indicating that there are certain types of sequences and bitrates where this is true. In the plots that are depicted, the dashed line represents picture quality for 4K pictures that were reconstructed using a down-sampled 1080p stream with 5 Mbps of enhancement data. The solid line represents standard natively encoded 4K pictures. For bitrates where the dashed line is higher than the solid line, the picture quality using up-sampling with edge enhancement is better than the original 4K.

Aspects of the present disclosure provide for reduced bit usage and therefore better usage of available bandwidth in streaming data applications, such as streaming video. Reduced bit usage could also lead to reduced utilization of system resources such as the CPU on the decoder side. Aspects of the present disclosure may be incorporated into systems that produce digital pictures, encode them for transmission over a network, and transmit them over the network.

Aspects of the present disclosure include implementations in which there are multiple sets of base data and multiple sets of enhancement data. By way of example, and not by way of limitation encoding may involve multiple down-sampling stages, with each stage producing sets of data that correspond to progressively lower resolutions. Specifically, each stage may down-sample the same high resolution input frames to generate a different corresponding set of lower resolution base frames and generate a corresponding set of enhancement information. Each set of base frames and corresponding set of enhancement information may then be encoded to produce corresponding sets of base data and enhancement data. The high resolution frames may be reconstructed on the decoder side from multiple sets of base data and enhancement data.