Multi-slice/tile encoder with overlapping spatial sections

Roughly described, pictures are divided into multiple spatial sections to be encoded by multiple section encoders. To address discontinuities of compression decisions across section boundaries, the multiple section encoders encode overlapping regions in a picture.

BACKGROUND

The invention relates to video encoding, and more particularly to the use of parallel encoding techniques to improve speed or reduce artifacts or both.

AVC is defined in ITU-T, Series H: Audiovisual And Multimedia Systems, Infrastructure of audiovisual services—Coding of moving video, Advanced video coding for generic audiovisual services, Recommendation ITU-T H.264 (April 2013), incorporated by reference herein, and HEVC is defined in ITU-T, Series H: Audiovisual And Multimedia Systems, Infrastructure of audiovisual services—Coding of moving video, High efficiency video coding, Recommendation ITU-T H.265 (April 2013), also incorporated by reference herein. As used herein, a system or method is considered to comply with “the AVC standard” so long as it complies with any version of that standard. Similarly, a system is considered to comply with “the HEVC standard” so long as it complies with any version of that standard.

The AVC and HEVC standards both accommodate a variety of profiles, each specifying an image resolution, a scanning type (progressive or interlaced) and a refresh rate, among other things. The profiles include one that has been labeled 1080p, which is in common use today as an HDTV transmission format. The 1080p profile specifies a resolution of 1920×1080 pixels progressively scanned. Another profile specified in both standards is one that has been labeled 4K UltraHD, having a resolution of 3840×2160 pixels progressively scanned, with a 60 Hz refresh rate. There is increasing interest in the industry for making 4K UltraHD widely available.

It is technically very challenging to encode 4K UltraHD video live. Whereas encoders are currently available to encode 1080p live, much more extensive processing power is needed for 4K UltraHD video.

One way to handle this challenge is to divide the video signal into “slices” or “tiles” and to use multiple encoders, one for each slice (tile).FIG. 1shows a picture divided into four slices numbered 1 through 4. Both the AVC and HEVC standards support multiple slice encoding and HEVC also supports multiple tile encoding. As used herein, the term “spatial section” refers to any type of spatial division of a picture, including both slices and tiles. The encoders assigned to handle different spatial sections are sometimes referred to herein as section encoders. The terms “slice” and “tile” have the meanings given to them in the AVC and HEVC standards.

A drawback of multiple slice or multiple tile encoding is that the boundaries between the spatial sections often show compression artifacts. Compression artifacts can arise because each encoder makes its own encoding decisions based on the picture information within its own spatial section. These include decisions related to motion vectors, quantizer indices, deblocking filtering, among others. Since each encoder has different picture information, many encoding decisions will be discontinuous across the boundary, causing the decoded video on either side of the boundary to look different. These differences cause the boundary to be visible as an annoying compression artifact.

A common approach to eliminate these discontinuities is to have the various section encoders share information about their coding decisions near the boundaries. This sharing even can involve sharing many lines of video data, for example to enable motion compensation across section boundaries or to perform in-loop deblocking filtering across the boundaries. This information sharing can require significant bit-rates (many hundreds of Mb/s for 4K UltraHD) and can require very low latency communications. These problems are not limited to 4K UltraHD live video encoding specifically; they apply to any format for which encoding of video at the desired speed is difficult or expensive with then-current technology.

SUMMARY

An opportunity therefore arises to create robust solutions to the problem of encoding video at a desired quality and speed, particularly the encoding of live streams of very high density or resolution video in real time. Better video reproduction, with higher density and more visually appealing images, may result.

Roughly described, the invention involves dividing the pictures into multiple spatial sections to be encoded by multiple section encoders, and to address the discontinuity of compression decisions across section boundaries by having the multiple section encoders encode overlapping regions in a picture. This concept can be used for example to build live 4K UltraHD encoders more simply and easily than with previous approaches.

The above summary of the invention is provided in order to provide a basic understanding of some aspects of the invention. This summary is not intended to identify key or critical elements of the invention or to delineate the scope of the invention. Its sole purpose is to present some concepts of the invention in a simplified form as a prelude to the more detailed description that is presented later. Particular aspects of the invention are described in the claims, specification and drawings.

DETAILED DESCRIPTION

FIG. 2illustrates an example end-to-end system for video capture, transmission and display. It comprises a source digital system200, in which video is captured (or otherwise provided) by a video capture module204, and sent to a video encoder206for encoding. The encoded output from video encoder206is sent to a transmitter208. The encoded video signal is transmitted via medium216to a destination digital system202in which the signal is received by receiver210. The received encoded video signal is then provided to a video decoder212which decodes it and provides it to a display214or other destination device. The video encoder206is required by specification to encode the video from video capture module204at a high speed. Typically the requirement derives from a need to encode the video live, so that images displayed on the display214can keep up with the images being captured by video capture module204. However, even for off-line encoding, where the video capture module204may be for example a pre-recorded movie, business and other practical reasons may drive a need or desire to encode the video at high speed.

FIG. 3is a block diagram of the video encoder206(FIG. 2). In the embodiment ofFIG. 3, the input video signal from video capture module204is provided to an input frame buffer310. Though the input frame buffer310can store the video information in different formats in different embodiments, in one embodiment it is stored as complete pictures, 8 bits per color per pixel. The input frame buffer310may store two or more complete pictures operated as a first-picture-in-first-picture-out buffer.

The output of frame buffer310is provided to a plurality of parallel section encoding units312-1,312-2, . . . ,312-n(collectively312). Each section encoding unit312is connected to retrieve from the input frame buffer310the pixel data for the picture section or sections that it is encoding. The encoding units312in various embodiments represent separate hardware modules, or separate software threads running on a multi-core processor, or separate software processes running on separate processors or servers, or separate software processes running on separate virtual machines, and so on. In some embodiments they can even represent separate tasks sharing a single processor, though the benefits of parallelism might be limited in such an embodiment. The video encoder206may or may not also include communication paths320between section encoding units312as described below. The encoded outputs of the section encoding units312are provided to an output buffer318, and an output unit314merges all the encoded spatial sections into a common encoded output bit stream316conforming to the appropriate AVC or HEVC standard.

The operation of section encoding units312will be described with reference to an example division of a picture for 4K UltraHD as shown inFIG. 4. InFIG. 4, the image is divided into 5 overlapping horizontal strips, sometimes referred to herein as “enlarged” regions, containing lines of the incoming picture as follows:Encoder A encodes lines [0,575]Encoder B encodes lines [448,959]Encoder C encodes lines [832,1343]Encoder D encodes lines [1216,1727]Encoder E encodes lines [1600, 2159]

While each encoder encodes a set of video lines that overlaps with its neighbors, only a set of smaller, non-overlapping regions, sometimes referred to herein as “unenlarged” regions, are used in the ultimate encoded output bit stream316:lines [0,511] come from Encoder Alines [512,895] come from Encoder Blines [896,1279] come from Encoder Clines [1280,1663] come from Encoder Dlines [1664,2159] come from Encoder E

Thus except at the top and bottom picture boundaries, each “enlarged” region encoded by an encoder includes not only its own “unenlarged” region, but also an “extension” region above and below which overlaps into the adjacent “unenlarged” regions above and below, as indicated in the drawing. The top region does not have an extension region above, and the bottom region does not have an extension region below.

The regions are “slices” in this example, though in a different embodiment they can be tiles or other spatial sections. Whether slices or tiles, each region preferably contains an integer number of coding units.

The overlap of regions inFIG. 4allows each section encoder to have information about its neighbor slices for use in motion estimation, rate control, and in-loop deblocking filtering. This information even can be used to help align rate control decisions on opposite sides of each slice boundary. There will be some small differences in coding between how a slice encoder will encode the extension region and how its neighboring slice encoder will encode the portion of its unenlarged region which overlaps with that same extension region, but those differences should be small enough to prevent annoying compression artifacts.

A challenge with this approach is for each slice encoder to generate a legal bitstream only for its output lines and not for the entire enlarged region which it encodes. This can be handled by using the “slice” syntax that is part of the AVC and HEVC standards. Each enlarged region is divided into three sub-regions: the unenlarged region in the center, the extension region above, and the extension region below. (As used herein, a “sub-region” is itself considered to be a “region”) The extension regions are shown inFIG. 4as being double-cross-hatched. In one embodiment each slice encoder encodes its three sub-regions as separate slices, but while encoding the unenlarged sub-region, it is able to look across the sub-region boundaries to the adjacent extension sub-regions in order to take into account image features that occur in the portion of the unenlarged adjacent region which overlaps with the extension sub-regions. Aspects of the encoding process including but not limited to one or more of the following take into account features in adjacent overlapping sub-regions: in-loop deblocking filtering, motion estimation and motion compensation, rate control information, mode decision information, and so on. In an alternative embodiment, each encoder encodes its entire enlarged region as a single slice. In either case, on output, each encoder's top and bottom encoded extension sub-regions are discarded. Only the encoded unenlarged region from the encoder is passed to the encoded output bit stream. Each sub-region preferably is defined so as to have an integer number of Coding Units or macroblocks (henceforth “CU's”). So in the example ofFIG. 4, each of the sub-regions1-13, separately, has a respective integer number of CU's. The sizes of the extension sub-regions need not be constant for all the encoding regions, but preferably they are (except where they are omitted at a picture boundary).

If an HEVC encoder is used tiles can also be used instead of or in addition to slices. Also, whereas inFIG. 4the slices are shown as rectangles extending horizontally all the way to both left-and-right boundaries, in another embodiment this is not necessary. Slices can be shorter than the picture width, and/or they can wrap around to include subsequent lines and can be non-rectangular. Tiles preferably extend entirely across a picture, either vertically or horizontally, but need not in all embodiments. In addition, the splitting of pictures into slices (tiles) for the encoding process typically is fixed for all pictures in a video, but it can be varied for different pictures if desired. In general, the slice (tile) preferably is enlarged for encoding purposes in all directions except where its boundaries abut a picture boundary.

Returning toFIG. 3, with the picture division example ofFIG. 4, encoders A-E preferably correspond to section encoding units312-1through312-5, respectively. Thus encoding unit312-1encodes sub-regions1-3ofFIG. 4; encoding unit312-2encodes sub-regions2-6; encoding unit312-3encodes sub-regions5-9; encoding unit312-3encodes sub-regions8-12; and encoding unit312-5encodes sub-regions11-13. Each section encoding unit312retrieves the pixel data of its assigned subregion(s) from the input frame buffer310when needed. Thus the pixel data for each of the extension subregions will be retrieved twice: once by the encoding unit encoding the above-adjacent slice, and once by the encoding unit encoding the below-adjacent slice. However, preferably none of the encoding units retrieves any pixel data for any regions outside of the enlarged region which it is assigned to encode.

While in the above example each encoder is assigned to encode a respective, fixed one of the image slices, this is not actually a requirement in all embodiments of the invention. Nor is there any actual requirement that the number of encoding units312equals the number of slices (tiles) in a picture. For full utilization of the available units, it is desirable to start a new slice (tile) in each encoding unit312promptly after it finishes the prior one, and slices may be assigned to encoding units in whatever sequence they becomes available. This means the number of encoding units312may be more or less than the number of spatial sections into which pictures are divided in a particular embodiment.

The encoded outputs from the slice (tile) encoding units312are provided to output buffer318. In one embodiment, each section encoding unit312writes into output buffer318only the encoded bit stream for the unenlarged region to which it was assigned. The encoded versions of the extension regions are thus discarded by the section encoding units312, and the output buffer312contains only bit stream data that will be used in the formation of the encoded output bit stream316. In another embodiment, each section encoding unit312writes into output buffer318the encoded bit stream that it formed for its entire enlarged region. In this case the output buffer312will contain more than one encoded version of each of the extension regions. Output unit314then selects into the encoded output bit stream316only the version that is part of an unenlarged region, discarding the version that was encoded only as an extension region. In either case, the output unit314provides on the encoded output bit stream only one version of each region of each picture.

In one embodiment, each section encoding unit312encodes its enlarged region without any picture-by-picture information about coding decisions made for other ones of the enlarged regions. In this case no communication paths320are required among the different section encoding units312. In such an embodiment it is preferable that high level guidance be provided to each of the encoding units312in order to help guide their internal processes. For example, the output bit rates should be fixed (though not necessarily identical) for each of the encoding units312in order that each entire picture (or sequence of pictures) can achieve a desired combined bit rate. As another example, average quantization values can be provided externally to each of the encoding units312, as well as other information for guiding the quantization process.

In another embodiment, each section encoding unit312does communicate certain information with the encoding units encoding adjacent slices. Communication paths320are used for this purpose. The information communicated in this embodiment between encoding units312may include such dynamically developed information as motion vectors calculated for the extension regions, motion vector partitions, coding modes, choice of reference picture, and pixel values reconstructed for in-loop deblocking filtering. The more information that is communicated between the section encoding units312, the better the encoding units should be able to suppress visual artifacts at the slice boundaries. However, limitations in communication bandwidth may make it impossible to share complete information between encoding units. In particular, communication paths320are not used to transmit to a particular encoding unit any raw pixel data from outside the enlarged region that the particular encoding unit is encoding.

The units312do not necessarily have to be all identical, but preferably they are.FIG. 5is a block diagram of one of the section encoding units312usable in some embodiments. TheFIG. 5encoding unit embodiment operates in a conventional manner, such as is described in US Patent Application Pre-Grant Publication No. 2012/0257678, incorporated by reference herein. Another description of a conventional encoder, usable in some embodiments of the invention, can be found in Tudor, P. N., “MPEG-2 Video Compression,” tutorial, Electronics & Communication Engineering Journal, vol. 7, issue 6, pp. 257-264 (December 1995), also incorporated by reference herein.

FIG. 6is a block diagram of one of the section encoding units312in another embodiment. Whereas the embodiment ofFIG. 5illustrates a module-oriented structure, the embodiment ofFIG. 6illustrates a structure in which most of the functionality performed by the modules ofFIG. 5, are performed by software modules running on a processor subsystem instead. The actual functions performed by the software in order to encode spatial sections may be the same as those set forth schematically inFIG. 5, and the software may also be organized into the same or similar modules.

Referring toFIG. 6, encoding unit312includes a CU input interface610which retrieves the picture data that it needs from input frame buffer310and writes it into a local memory subsystem612. The encoding unit312also includes a bit stream output interface614for writing the encoded bit stream from memory subsystem612to the output buffer318. Also in communication with the memory subsystem612is a processor subsystem616which, in an embodiment that includes inter-unit communication paths320, also communicates through an interface618with those paths320.

Local memory subsystem612includes RAM and may also include ROM and/or other data storage components. Local memory subsystem612stores program code and data. The program code, when applied to processor subsystem616, causes the encoding unit312to perform the steps involved in encoding the assigned spatial section. The data includes input pixel data retrieved from input frame buffer310, and encoded output data for writing to the output buffer318, as well as data formed at many intermediate steps in the encoding process.

CU input interface610may in one embodiment retrieve the pixel data by DMA (Direct Memory Access), and then writes the same pixel data, also by DMA, to local memory in the memory subsystem. The processor subsystem616, under control of the program code, programs the DMA controllers with start addresses for reading, start addresses for writing, and an indication of how many data bytes to transfer. In another embodiment the processor subsystem616may retrieve the data from the input frame buffer310directly instead, and/or may write the data into the memory subsystem612directly instead. In all such embodiments only the pixels for the enlarged region that the particular section encoding unit312is encoding, are copied into the memory subsystem612.

Similarly, bit stream output interface614in one embodiment retrieves the encoded output data by DMA from the memory subsystem612, and then writes the same data, also by DMA, to the output buffer318. The processor subsystem616, under control of the program code, programs the DMA controllers with start addresses for reading, start addresses for writing, and an indication of how many data bytes to transfer. In another embodiment the processor subsystem616may retrieve the data from the memory subsystem612directly instead, and/or may write the data into the output buffer318directly instead.

The following documents are incorporated by reference:G. J. Sullivan; J.-R. Ohm; W.-J. Han; T. Wiegand. “Overview of the High Efficiency Video Coding (HEVC) Standard”. IEEE Transactions on Circuits and Systems for Video Technology. (2012 May 25);Ching Chi, Mauricio Alvarez Mesa, Ben Juurlink, Valeri George, and Thomas Schierl, Improving the Parallelization Efficiency of HEVC Decoding, Proceedings of IEEE International Conference on Image Processing (ICIP 2012), Orlando, Fla., USA, September 2012;ITU-T, Series H: Audiovisual And Multimedia Systems, Infrastructure of audiovisual services—Coding of moving video, Advanced video coding for generic audiovisual services, Recommendation ITU-T H.264 (April 2013);ITU-T, Series H: Audiovisual And Multimedia Systems, Infrastructure of audiovisual services—Coding of moving video, High efficiency video coding, Recommendation ITU-T H.265 (April 2013); andU.S. Pre-grant patent publications 2012/0314767 (Wang) and 2012/0257678 (Zhou).

As used herein, a given signal, event or value is “responsive” to a predecessor signal, event or value if the predecessor signal, event or value influenced the given signal, event or value. If there is an intervening processing element, step or time period, the given signal, event or value can still be “responsive” to the predecessor signal, event or value. If the intervening processing element or step combines more than one signal, event or value, the signal output of the processing element or step is considered “responsive” to each of the signal, event or value inputs. If the given signal, event or value is the same as the predecessor signal, event or value, this is merely a degenerate case in which the given signal, event or value is still considered to be “responsive” to the predecessor signal, event or value. “Dependency” of a given signal, event or value upon another signal, event or value is defined similarly.

The foregoing description of preferred embodiments of the present invention has been provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Obviously, many modifications and variations will be apparent to practitioners skilled in this art. Without limitation, any and all variations described, suggested or incorporated by reference in the documents incorporated by reference herein are specifically incorporated by reference into the description herein of embodiments of the invention. In addition, any and all variations described, suggested or incorporated by reference herein with respect to any one embodiment are also to be considered taught with respect to all other embodiments. The embodiments described herein were chosen and described in order to best explain the principles of the invention and its practical application, thereby enabling others skilled in the art to understand the invention for various embodiments and with various modifications as are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the following claims and their equivalents.