Patent Description:
This patent document is directed generally to video and image encoding and decoding.

In the last three decades, a number of different video coding standards have been published for representing digital video into a compressed format by removing some visual redundancy and information in the digital video. During this time, pixel resolution of encoded video content has steadily risen from the early days of Source Input Format (SIF) resolution in MPEG-<NUM> video standard to <NUM> or <NUM> video being currently contemplated. Accordingly, newer video coding standards have adopted more efficient and flexible coding techniques to accommodate increased resolution of video. Non-patent literature <NPL>" discloses (a) setting a minimum size of grid to minmum of CTU size, (b) removing a subpicture grid, signalling subpicture using slices, (c) removing a subpicture grid, signalling subpicture using brick and (d) removing a subpicture grid and signalling subpicture using tiles.

This patent document describes, among other things, techniques for encoding and decoding digital video using techniques that can be used for signaling height of a rectangular slice in a coding tree unit in a subpicture of a video.

In one example aspect, a method of bitstream processing is disclosed. The method includes parsing a portion of a video bitstream at a video unit level for a first field indicative that a slice height is specified for a subpicture partitioning or that the slice height is specified for a tile partitioning at the video unit level;parsing, due to determining that the first field indicates that the slice height is specified for the subpicture partitioning, a preset number of second fields in the portion of the video bitstream to obtain heights for rectangular slices in the video unit, wherein the heights are indicated in multiple of coding tree unit, CTU, heights, and wherein each rectangular slice comprises one or more CTU rows that belong to a same subpicture, wherein the preset number is a positive integer; anddecoding, based on the first field and/or the preset number of second fields, the video bitstream to generate a video.

In another aspect, a video decoder apparatus comprising a processor configured to implement the above-implemented method is disclosed.

In yet another aspect, a method of encoding video is disclosed. The method includes including, in a video bitstream representing a video, a first field indicative of whether a slice height is specified for a subpicture partitioning or for a tile partitioning for encoding a video picture;inserting, due to including the first field indicates that the slice height is specified for the subpicture partitioning, N second fields in the video bitstream indicating heights for rectangular slices in the video picture, wherein the heights are indicated in multiple of coding tree unit (CTU) heights, and wherein each rectangular slice comprises one or more CTU rows that belong to a same subpicture, wherein N is a positive integer; andencoding, by including on the first field and/or the N second fields, the video picture of the video.

In yet another example aspect, a video processing apparatus comprising a processor is disclosed. The processor is configured to implement an encoding or a decoding method described herein.

In yet another example aspect, a computer-program storage medium is disclosed. The computer-program storage medium includes code stored thereon. The code, when executed by a processor, causes the processor to implement a described method.

These, and other, aspects are described in the present document.

Section headings are used in the present document only to improve readability and do not limit scope of the disclosed embodiments and techniques in each section to only that section. Certain features are described using the example of the H. <NUM>/AVC (advanced video coding), H. <NUM>/HEVC (high efficiency video coding) and H. <NUM> Versatile Video Coding (VVC) standards. However, applicability of the disclosed techniques is not limited to only H. <NUM>/AVC or H. <NUM>/HEVC or H. <NUM>/VVC systems.

This patent document relates to video processing and communication, in particular to methods and apparatus for encoding a digital video or picture to generate a bitstream, methods and apparatus for decoding a bitstream to reconstruct a digital video or picture, methods and apparatus for extracting a bitstream to form a sub-bitstream.

Techniques for compressing digital video and picture utilize correlation characteristics among pixel samples to remove redundancy in the video and picture. An encoder may partition a picture into one or more regions containing a number of units. Such region breaks prediction dependencies within a picture, so that a region can be decoded or at least syntax elements corresponding to this region can be correctly parsed without referencing to data of another region in the same picture. Such region introduced in video coding standards is to facilitate resynchronization after data losses, parallel processing, region of interesting coding and streaming, packetized transmission, view-port dependent streaming, and etc. For example, in H. <NUM>/AVC standard, example of such region can be slice and slice group. <NUM>/HEVC standard, example of such region can be slice and tile.

In the development of next-generation video coding standard, MPEG (Moving Picture Experts Group) has mandated that the next-generation video codec should enable efficient extraction of a desired viewport from the compressed bitstream. In addition, in applications involving <NUM>, <NUM> and even higher resolution videos, a viewer may be allowed to select a view-port for rendering on terminal devices, and thus it is useful to enable an efficient extraction of a sub-bitstream from the bitstream corresponding to the original resolution.

In current JVET-P2001 (VVC Draft <NUM>) specification, the video picture partitions are based on subpictures, tiles and slices. The subpictures and the tiles are typically indicated in the bitstream, although through very different methods. While subpictures structure has been given a rather flexible partition arrangement, the tiles structure is specified as a grid system.

In the current text of the standard, if rectangular slices are contained inside a tile, there is a designed way to specify the heights of the rectangular slices in coding tree unit (CTU) size measure. However, the technology provided in VVC presently provides no solution or mechanism to specify heights of rectangular slices that may be positioned to fall inside a subpicture partition. For instance, if the whole video picture is one tile and at the same time there are multiple subpictures in the video picture, there is currently no way to encode the relationship between CTU size and slice heights for slices inside a subpicture. Furthermore, a relationship between heights of such rectangular slices and corresponding CTU size will not be understood by a decoder based on the syntax elements in the bitstream that represents encoded video.

One possible way to indicate rectangular slices heights (in CTU size measure) inside a subpicture is to set the tile's size the same as a CTU size, which would typically result in small-sized tiles. Then, by counting number of such small sized tiles (which are now equivalent in height to CTU), slice height in CTU size measure can be represented using a bitstream format compliant with JVET-P2001 VVC Draft <NUM>. However, this is a sub-optimal solution because it overly constrains the tile size and fails to truly use the benefit of VVC tiles structure design.

In some embodiments according to a solution, for a convenient indication of heights of the rectangular slices in CTU size measure, two new syntax elements and their semantics may be used. The first syntax element is called rect_slice_height_in _subpic_only_flag and may use the format (u(<NUM>)). The second syntax structure is called slice_height_in_ctu_in_subpic_minus1[ i++ ] and may use the format (ue(v)).

The syntax elements may be used as follows.

The first syntax element identifies the context or reference of how slice height is specified. For example, the first synrect_slice_height_in_subpic_only_flag equal to <NUM> specifies that slice height is specified with subpictures partition only. Furthermore,
rect_slice_height_in_subpic_only_flag equal to <NUM> specifies that slices height is specified with tiles partition.

The second syntax element may be used to indicate slice heights in units of CTU rows. For example, slice_height_in_ctu_in_subpic_minus1[ i ] plus <NUM> specifies the height of the i-th rectangular slice in units of CTU rows for the case where the i-th slice contains a subset of CTU rows from a single subpicture. The value of slice_height_in_ctu_minus1[ i ] will be in the range of <NUM> to Ceil( pic_height_max_in_luma_samples / CtbSizeY ) - <NUM>, inclusive.

In current VVC Draft <NUM> specification, a picture partitions can be done as multiple tiles, slices or subpictures. There is a problem for rectangular slices indication: when only one tile for a complete video picture, it cannot indicate the slice height in CTUs within a subpicture. In this contribution, a solution is proposed to allow either slice height in CTUs inside a tile or inside a subpicture.

In current VVC Draft <NUM> specification, the structure of subpictures is indicated in Sequence Parameter Set (SPS) as:.

The highlighted texts in italicized underline in above SPS marked the way that subpictures was defined: in general, all the partitions are based on CTU (Coding Tree Unitbasic unit), by specifying top left X, Y locations plus subpictures width and height information, subpictures can be defined efficiently.

With current subpicture design, the partition of subpictures in <FIG> (each colored block is associated with a subpicture) can be supported. The processing order of all subpictures in <FIG> can be arranged differently (more than one possible way). The key constraint is that when a subpicture is encoded, this subpicture's entire left boundary and top boundary should be available. For instance: (<NUM>) the boundary pixels for the current subpicture are belonging to other subpictures that have already been processed (decoded), or (<NUM>) the relevant boundaries for the current subpicture are picture boundaries. The possible two processing orders for these ten subpictures are given in <FIG>.

<FIG> shows an example of partitioning a picture into tiles, subpictures and slices. <FIG> is another example of a picture with <NUM> by <NUM> luma CTUs that is partitioned into <NUM> tiles and <NUM> raster-scan tiles.

<FIG> shows an example of <NUM> by <NUM> luma CTUs partitioned into <NUM> tiles and <NUM> rectangular slices.

<FIG> shows an example of a picture that is partitioned into <NUM> tiles and <NUM> rectangular slices.

In all these figures, it is clear that the tiles structure in VVC cannot flexibly support the video picture partition as showed in <FIG>.

The tiles structure was described in PPS (Picture Parameter Set) as the following (relevant texts highlighted in "blue" color):.

The highlighted texts in "blue" color showed that the tiles structure can be seen as a specific grid type.

<FIG> shows an example of a video picture with <NUM> tiles, <NUM> subpictures, and <NUM> slices with 20x12 = <NUM> CTUs.

To use tiles concepts in a partition arrangement and at the same time support <FIG> illustrated subpictures structure arrangement. In following <FIG>, it show a possibility: the picture will need to be partitioned as 5x5= <NUM> tiles as displayed. In <FIG>, each colored box is still a subpicture partitioned in this video picture.

According to current spec, the only way to support multiple slices is to use PPS part. In <FIG>'s example, we initially define <NUM> subpictures structure (top left X and Y location, plus subpicture width and height information). And then we define the whole picture has one single tile. Let us see a typical use case, for <FIG> illustrated subpictures partition, we would like to have two slices inside each subpicture, therefore, in total we should have <NUM> slices (for <NUM> subpictures).

In the table below, the new syntax elements that could be added to the current syntax structure of picture parameter set raw byte sequence payload (RBSP) is shown in italicized text.

Under <NUM>. <NUM> "CTB raster scanning, tile scanning, and subpicture scanning processes" in the current document of he JVET specification, the existing texts may be clarified as follows.

The list CtbToSubPicIdx[ ctbAddrRs ] for ctbAddrRs ranging from <NUM> to PicSizeInCtbsY - <NUM>, inclusive, specifying the conversion from a CTB address in picture raster scan to a subpicture index, is derived as follows: (italicized text is to be deleted from the proposed amendment):
<IMG>.

The list NumSlicesInSubpic[ i ] and SliceSubpicToPicIdx[ i ][ k ], specifying the number of rectangular slices in the i-th subpicture and picture-level slice index of the k-th slice in the i-th subpicture, are derived is derived as follows:
<IMG>
<IMG>.

The "italicized" emphasized texts are modified according to "JVET VVC Trac" information #<NUM>.

Furthermore, the following text may be added under <NUM>. <NUM> "Picture parameter set RBSP semantics":.

rect_slice_height_in_subpic_only_flag equal to <NUM> specifies that slices height is specified with subpictures partition only. rect_slice_height_in_subpic_only_flag equal to <NUM> specifies that slices height is specified with tiles partition.

slice_height_in_ctu_in_subpic_minus1[ i ] plus <NUM> specifies the height of the i-th rectangular slice in units of CTU rows for the case where the i-th slice contains a subset of CTU rows from a single subpicture. The value of slice_height_in_ctu_minus1[ i ] shall be in the range of <NUM> to Ceil( pic_height_max_in_luma_samples / CtbSizeY ) - <NUM>, inclusive.

<FIG> is a diagram illustrating a first example device containing at least the example video encoder or picture encoder.

Acquisition unit <NUM> captures video and picture. Acquisition unit <NUM> may be equipped with one or more cameras for shooting a video or a picture of nature scene. Optionally, acquisition unit <NUM> may be implemented with a camera to get depth video or depth picture. Optionally, acquisition unit <NUM> may include a component of an infrared camera. Optionally, acquisition unit <NUM> may be configured with a remote sensing camera. Acquisition unit <NUM> may also be an apparatus or a device of generating a video or a picture by scanning an object using radiation.

Optionally, acquisition unit <NUM> may perform pre-processing on video or picture, for example, automatic white balance, automatic focusing, automatic exposure, backlight compensation, sharpening, denoising, stitching, up-sampling/down sampling, frame-rate conversion, virtual view synthesis, and etc..

Acquisition unit <NUM> may also receive a video or picture from another device or processing unit. For example, acquisition unit <NUM> can be a component unit in a transcoder. The transcoder feeds one or more decoded (or partial decoded) pictures to acquisition unit <NUM>. Another example is that acquisition unit <NUM> get a video or picture from another device via a data link to that device.

Note that acquisition unit <NUM> may be used to capture other media information besides video and picture, for example, audio signal. Acquisition unit <NUM> may also receive artificial information, for example, character, text, computer-generated video or picture, and etc..

Encoder <NUM> is an implementation of the example encoder. Input of encoder <NUM> is the video or picture outputted by acquisition unit <NUM>. Encoder <NUM> encodes the video or picture and outputs generated a video or picture bitstream.

Storage/Sending unit <NUM> receives the video or picture bitstream from encoder <NUM>, and performs system layer processing on the bitstream. For example, storage/sending unit <NUM> encapsulates the bitstream according to transport standard and media file format, for example, e.g. MPEG-<NUM> TS, ISOBMFF, DASH, MMT, and etc. Storage/Sending unit <NUM> stores the transport stream or media file obtained after encapsulation in memory or disk of the first example device, or sends the transport stream or media file via wireline or wireless networks.

Note that besides the video or picture bitstream from encoder <NUM>, input of storage/sending unit <NUM> may also include audio, text, image, graphic, and etc. Storage/sending unit <NUM> generates a transport or media file by encapsulating such different types of media bitstreams.

The first example device described in this embodiment can be a device capable of generating or processing a video (or picture) bitstream in applications of video communication, for example, mobile phone, computer, media server, portable mobile terminal, digital camera, broadcasting device, CDN (content distribution network) device, surveillance camera, video conference device, and etc..

<FIG> is a diagram illustrating a second example device containing at least the example video decoder or picture decoder.

Receiving unit <NUM> receives video or picture bitstream by obtaining bitstream from wireline or wireless network, by reading memory or disk in an electronic device, or by fetching data from other device via a data link.

Input of receiving unit <NUM> may also include transport stream or media file containing video or picture bitstream. Receiving unit <NUM> extracts video or picture bitstream from transport stream or media file according to specification of transport or media file format.

Receiving unit <NUM> outputs and passes video or picture bitstream to decoder <NUM>. Note that besides video or picture bitstream, output of receiving unit <NUM> may also include audio bitstream, character, text, image, graphic and etc. Receiving unit <NUM> passes the output to corresponding processing units in the second example device. For example, receiving unit <NUM> passes the output audio bitstream to audio decoder in this device.

Decoder <NUM> is an implementation of the example decoder. Input of encoder <NUM> is the video or picture bitstream outputted by receiving unit <NUM>. Decoder <NUM> decodes the video or picture bitstream and outputs decoded video or picture.

Rendering unit <NUM> receives the decoded video or picture from decoder <NUM>. Rendering unit <NUM> presents the decoded video or picture to viewer. Rendering unit <NUM> may be a component of the second example device, for example, a screen. Rendering unit <NUM> may also be a separate device from the second example device with a data link to the second example device, for example, projector, monitor, TV set, and etc. Optionally, rendering <NUM> performs post-processing on the decoded video or picture before presenting it to viewer, for example, automatic white balance, automatic focusing, automatic exposure, backlight compensation, sharpening, denoising, stitching, up-sampling/down sampling, frame-rate conversion, virtual view synthesis, and etc..

Note that besides decoded video or picture, input of rendering unit <NUM> can be other media data from one or more units of the second example device, for example, audio, character, text, image, graphic, and etc. Input of rendering unit <NUM> may also include artificial data, for example, lines and marks drawn by a local teacher on slides for attracting attention in remote education application. Rendering unit <NUM> composes the different types of media together and then presented the composition to viewer.

The second example device described in this embodiment can be a device capable of decoding or processing a video (or picture) bitstream in applications of video communication, for example, mobile phone, computer, set-top box, TV set, HMD, monitor, media server, portable mobile terminal, digital camera, broadcasting device, CDN (content distribution network) device, surveillance, video conference device, and etc..

<FIG> is a diagram illustrating an electronic system containing the first example device in <FIG> and the second example device in <FIG>.

Service device <NUM> is the first example device in <FIG>.

Storage medium / transport networks <NUM> may include internal memory resource of a device or electronic system, external memory resource that is accessible via a data link, data transmission network consisting of wireline and/or wireless networks. Storage medium / transport networks <NUM> provides storage resource or data transmission network for storage/sending unit <NUM> in service device <NUM>.

Destination device <NUM> is the second example device in <FIG>. Receiving unit <NUM> in destination device <NUM> receives a video or picture bitstream, a transport stream containing video or picture bitstream or a media file containing video or picture bitstream from storage medium / transport networks <NUM>.

The electronic system described in this embodiment can be a device or system capable of generating, storing or transporting, and decoding a video (or picture) bitstream in applications of video communication, for example, mobile phone, computer, IPTV systems, OTT systems, multimedia systems on Internet, digital TV broadcasting system, video surveillance system, potable mobile terminal, digital camera, video conference systems, and etc..

<FIG> shows an example apparatus <NUM> that may be used to implement encoder-side or decoder-side techniques described in the present document. The apparatus <NUM> includes a processor <NUM> that may be configured to perform the encoder-side or decoder-side techniques or both. The apparatus <NUM> may also include a memory (not shown) for storing processor-executable instructions and for storing the video bitstream and/or display data. The apparatus <NUM> may include video processing circuitry (not shown), such as transform circuits, arithmetic coding/decoding circuits, look-up table based data coding techniques and so on. The video processing circuitry may be partly included in the processor and/or partly in other dedicated circuitry such as graphics processors, field programmable gate arrays (FPGAs) and so on.

The above methods may be implemented by an apparatus as shown in <FIG>.

From the above description, it can be known that techniques that enable signaling of slice heights in subpictures are disclosed. In one advantageous aspect, these techniques may be used by encoders to signal slice height separately and independently for subpictures and tiles, thereby allowing flexible organization of video pictures using subpictures.

It will further be appreciated that, with current VVC Draft <NUM>, if an encoder sets each tile as small as each CTU; then for a subpicture, for coding regions inside it, the slice heights can be specified by the number of tile row heights (in this case: equal to the number of CTU heights). When doing it this way, the tile concept will be no further use as a good partition tool. With VVC Draft <NUM>, if an encoder does not set such a small sized tile, then the encoder does not have the ability to specify the heights (with the number of CTU heights) of the slices inside the subpicture. The technology disclosed in the present document will allow a flexible use of tile partition as a coding or decoding tool and also allow a clear indication for slice heights inside each subpicture.

While this patent document contains many specifics, these should be construed as as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this patent document in the context of separate embodiments can also be implemented in combination in a single embodiment.

Claim 1:
A method of video bitstream processing, comprising:
parsing a portion of a video bitstream at a video unit level for a first field indicative that a slice height is specified for a subpicture partitioning or that the slice height is specified for a tile partitioning at the video unit level;
parsing, due to determining that the first field indicates that the slice height is specified for the subpicture partitioning, a preset number of second fields in the portion of the video bitstream to obtain heights for rectangular slices in the video unit, wherein the heights are indicated in multiple of coding tree unit, CTU, heights, and wherein each rectangular slice comprises one or more CTU rows that belong to a same subpicture, wherein the preset number is a positive integer; and
decoding, based on the first field and/or the preset number of second fields, the video bitstream to generate a video.