Signaling 360-degree video information

This disclosure describes techniques for signaling 360-degree video information in syntax structures. As one example, this disclosure describes examples for signaling projection types and related information for 360-degree video in syntax structures that include one or more syntax elements. In some examples, the same syntax structures may include non-360-degree video information as well. Accordingly, the syntax structure may be used for encoding and decoding a bitstream carrying only non-360-degree video data, only 360-degree video data, or both non-360-degree video data and 360-degree video data.

TECHNICAL FIELD

BACKGROUND

SUMMARY

In general, this disclosure describes techniques for signaling 360-degree video information in syntax structures. As one example, this disclosure describes examples for signaling projection types and related information for 360-degree video in syntax structures that include one or more syntax elements. In some examples, the same syntax structures may include non-360-degree video information as well.

In one example, the disclosure describes a method of decoding video data, the method comprising receiving, as part of a syntax structure, information indicating that a coded bitstream includes 360-degree video, receiving information for the 360-degree video based on the reception of the information that the coded bitstream includes 360-degree video, receiving, as part of the same syntax structure, information for non-360-degree video, and decoding the 360-degree video and the non-360-degree video.

In one example, the disclosure describes a method of encoding video data, the method comprising signaling, as part of a syntax structure, information indicating that a coded bitstream includes 360-degree video, signaling information for the 360-degree video based on the coded bitstream including 360-degree video, signaling, as part of the same syntax structure, information for non-360-degree video, and encoding the 360-degree video and the non-360-degree video.

In one example, the disclosure describes a device for decoding video data, the device comprising a memory configured to store video data and a video decoder comprising one or more processing units implemented in fixed-function or programmable circuitry. The video decoder is configured to receive, as part of a syntax structure stored as video data in the memory, information indicating that a coded bitstream includes 360-degree video, receive information for the 360-degree video based on the reception of the information that the coded bitstream includes 360-degree video, receive, as part of the same syntax structure, information for non-360-degree video, and decode the 360-degree video and the non-360-degree video.

In one example, the disclosure describes a computer-readable storage medium storing instructions thereon that when executed cause one or more processors of a device for decoding video data to receive, as part of a syntax structure, information indicating that a coded bitstream includes 360-degree video, receive information for the 360-degree video based on the reception of the information that the coded bitstream includes 360-degree video, receive, as part of the same syntax structure, information for non-360-degree video, and decode the 360-degree video and the non-360-degree video.

DETAILED DESCRIPTION

Video coding standards include ITU-T H.261, ISO/IEC MPEG-1 Visual, ITU-T H.262 or ISO/IEC MPEG-2 Visual, ITU-T H.263, ISO/IEC MPEG-4 Visual, ITU-T H.264 (also known as ISO/IEC MPEG-4 AVC), including its Scalable Video Coding (SVC) and Multiview Video Coding (MVC) extensions and ITU-T H.265 (also known as ISO/IEC MPEG-4 HEVC) with its extensions. The latest draft of the H.265 specification is: ITU-T H.265, Series H: Audiovisual and Multimedia Systems, Infrastructure of audiovisual services—Coding of moving video, Advanced video coding for generic audiovisual services, The International Telecommunication Union. December 2016, and herein referred to as H.265 or HEVC.

For next generation video coding, a number of methods for coding of 360-degree video are being develop together with methods for coding of non-360-degree video. Encoding and decoding of 360-degree video may be different from that of for non-360-degree video as encoding and decoding of 360-degree video may require different set of coding tools, different pre-processing requirements, and different post processing requirements.

In this disclosure, 360-degree video may refer to examples where there is image content captured in viewing angles all round a video content capture device. For example, the captured image content may be considered as a sphere of image content, and a viewer can view the image content from virtually any angle. In this disclosure, 360-degree video should not be considered limited to full 360-degree video. In general, 360-degree video refers to example encoded and decoded video content that includes and extends beyond the periphery of where a viewer is currently viewing image content (referred to as a viewport). 360-degree video may allow a viewer to look above, below, left, or right of the viewport. The viewport may form a subset of all the video content. Hence, in 360-degree video, the encoding and decoding may include encoding and decoding of video content that goes outside the viewport so that such video content is available to the viewer when he or she determines to view video content outside the viewport.

In this disclosure, non-360-degree video may refer to examples where the encoded and decoded video content is for a fixed viewer perspective. As one example, in rectangular video, the viewer perspective is fixed, and the viewer may not be able to view image content above, below, left, or right of the rectangular video that is being displayed (e.g., in the viewport). Hence, in non-360-degree video, the encoding and decoding may not include encoding and decoding of video content that goes outside the viewport because video content outside of the viewport need not be available to the viewer.

Accordingly, for 360-degree video, additional pre- and post-processing operations may be needed. As one example, encoder pre-processing includes stitching and projecting of video content of the different viewing perspectives, and decoder post-processing may include the corresponding steps for rendering. For instance, processing of 360-degree video may include conversion or projection of video content into certain type of projection type such as equiretangular projection (ERP), cubemap projection (CMP) and its derivative projections such as adjusted cubemap (ACP) and equatorial cylindrical projection (ECP). Such projection information is signaled from a video encoder to a video decoder, which may be used by 360-degree video specific coding tools and/or 360-degree video specific post processing. Also, for 360-degree video, there may be additional signaling of projection metadata and the like.

As one example, “360-degree video data” may include a frame having a plurality of regions corresponding to intersecting, orthogonal planes of a cube map projection. For instance, in some examples, the frame being encoded by the video encoder or being decoded by a video decoder is a flat two-dimensional (2D) image. However, the content of the 2D image includes image content external to the viewport such as regions that form different planes of the example projections. In some examples, “non-360-degree video data” may also a flat 2D image but does not include regions outside the viewport. The above example description of differences in 360-degree video and non-360-degree video are provided simply to assist with understanding and should not be considered limiting. In general, 360-degree video data processing provides for encoding and decoding video data that is external to a current viewport of the viewer allowing the viewer to see video content above, below, in front, behind, to the left, and/or to the right of the current viewport. Non-360-degree video data processing, however, may be limited to the current viewport, possibly with some external periphery information for coding but insufficient to allow the viewer to video content above, below, in front, behind, to the left, and/or to the right of the current viewport.

An example technological problem in the field of video coding (e.g., encoding or decoding) may be that signaling information for 360-degree video should be done in such a way to be compatible with non-360-degree video signaling. With compatibility between signaling 360-degree video and non-360-degree video information, encoding and decoding of 360-degree video information may be performed using many (e.g., most or all) of the coding tools used for non-360-degree video information, and possibly more coding tools. This disclosure describes one or more example technological solutions to technological problems so that 360-degree video information and non-360-degree video information can be signaled in a compatible manner.

As one example, the example techniques may include a method to signal projection types and related information for 360-degree video information in a syntax structure that includes one or more syntax elements. The syntax structure may also include information for non-360-degree video information. Accordingly, the syntax structure may be a syntax structure that used in signaling non-360-degree video information and can also be used to signal 360-degree video information (e.g., the syntax structure forms part of a standard-conforming bitstream regardless of whether the standard-conforming bitstream includes 360-degree video). For example, the syntax structure may be used for encoding and decoding a bitstream carrying only non-360-degree video data, only 360-degree video data, or both non-360-degree video data and 360-degree video data. Examples of the syntax structures include parameter sets; however, other examples of syntax structures exist, such as those described in more detail below.

FIG. 1is a block diagram illustrating an example video encoding and decoding system100that may perform the techniques of this disclosure. The techniques of this disclosure are generally directed to coding (encoding and/or decoding) video data. In general, video data includes any data for processing a video. Thus, video data may include raw, uncoded video, encoded video, decoded (e.g., reconstructed) video, and video metadata, such as signaling data.

In the example ofFIG. 1, source device102includes video source104, memory106, video encoder200, and output interface108. Destination device116includes input interface122, video decoder300, memory120, and display device118. In accordance with this disclosure, video encoder200of source device102and video decoder300of destination device116may be configured to apply the techniques for 360-degree video coding (e.g., encoding by video encoder200and decoding by video decoder300). Thus, source device102represents an example of a video encoding device, while destination device116represents an example of a video decoding device. In other examples, a source device and a destination device may include other components or arrangements. For example, source device102may receive video data from an external video source, such as an external camera or external cameras that capture 360-degree video. Likewise, destination device116may interface with an external display device, rather than including an integrated display device.

In some examples, source device102may output encoded video data to file server114or another intermediate storage device that may store the encoded video generated by source device102. Destination device116may access stored video data from file server114via streaming or download. File server114may be any type of server device capable of storing encoded video data and transmitting that encoded video data to the destination device116. File server114may represent a web server (e.g., for a website), a File Transfer Protocol (FTP) server, a content delivery network device, or a network attached storage (NAS) device. Destination device116may access encoded video data from file server114through any standard data connection, including an Internet connection. This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of both that is suitable for accessing encoded video data stored on file server114. File server114and input interface122may be configured to operate according to a streaming transmission protocol, a download transmission protocol, or a combination thereof.

Although not shown inFIG. 1, in some examples, video encoder200and video decoder300may each be integrated with an audio encoder and/or audio decoder, and may include appropriate MUX-DEMUX units, or other hardware and/or software, to handle multiplexed streams including both audio and video in a common data stream. If applicable, MUX-DEMUX units may conform to the ITU H.223 multiplexer protocol, or other protocols such as the user datagram protocol (UDP).

Video encoder200and video decoder300may operate according to a video coding standard, such as ITU-T H.265, also referred to as High Efficiency Video Coding (HEVC) or extensions thereto, such as the multi-view and/or scalable video coding extensions. Alternatively, video encoder200and video decoder300may operate according to other proprietary or industry standards, such as the Joint Exploration Test Model (JEM) for the Versatile Video Coding (VVC) standard under development. The techniques of this disclosure, however, are not limited to any particular coding standard.

As another example, video encoder200and video decoder300may be configured to operate according to JEM of VVC. According to JEM, a video coder (such as video encoder200) partitions a picture into a plurality of coding tree units (CTUs). Video encoder200may partition a CTU according to a tree structure, such as a quadtree-binary tree (QTBT) structure. The QTBT structure of JEM removes the concepts of multiple partition types, such as the separation between CUs, PUs, and TUs of HEVC. A QTBT structure of JEM includes two levels: a first level partitioned according to quadtree partitioning, and a second level partitioned according to binary tree partitioning. A root node of the QTBT structure corresponds to a CTU. Leaf nodes of the binary trees correspond to coding units (CUs).

In some examples, video encoder200and video decoder300may use a single QTBT structure to represent each of the luminance and chrominance components, while in other examples, video encoder200and video decoder300may use two or more QTBT structures, such as one QTBT structure for the luminance component and another QTBT structure for both chrominance components (or two QTBT structures for respective chrominance components).

In one or more examples, video encoder200may generate syntax structures that each include one or more syntax elements. In some examples, video encoder200may signal, as part of a syntax structure, information for 360-degree video, and signal, as part of the same syntax structure, information for non-360-degree video. Video decoder300may receive, as part of a syntax structure, information for 360-degree video, and receive, as part of the same syntax structure, information for non-360-degree video. As noted, the syntax structure includes one or more syntax elements. The syntax elements may together form one or more of: a sequence parameter set (SPS), a SPS extension, a picture parameter set (PPS), a PPS extension, a video parameter set (VPS), a VPS extension, a picture header, a slice header, a tile header, or a supplemental enhancement information (SEI) message.

A PPS is, for example, a syntax structure that contains syntax elements that apply to zero or more entire coded pictures as determined by a syntax element found in each slice segment header. An SPS is, for example, a syntax structure that contains syntax elements that apply to zero or more entire coded video sequences (CVSs) as determined by the content of a syntax element found in the PPS referred to by a syntax element found in each slice segment header. A VPS is, for example, a syntax structure that contains syntax elements that apply to zero or more entire CVSs as determined by the content of a syntax element found in the SPS referred to by a syntax element found in the PPS referred to by a syntax element found in each slice segment header.

In this manner, video encoder200may generate a bitstream including encoded video data, e.g., syntax elements describing partitioning of a picture into blocks (e.g., CUs) and prediction and/or residual information for the blocks, including syntax structures that include information for both 360-degree video and non-360-degree video. Ultimately, video decoder300may receive the bitstream and decode the encoded video data.

The following describes examples of techniques in accordance with the techniques of this disclosure. The following techniques may be applied independently and/or in combination. As one example, video encoder200may signal 360-degree video specific information in a syntax structure, such as a parameter set (e.g., SPS, SPS extension in HEVC, PPS, PPS extension in HEVC, VPS, VPS extension in HEVC). Additional examples of a syntax structure include a picture header, slice header, tile header, or SEI message.

Solely for ease of description, and in no way limiting, the following is described with respect to the SPS extension in HEVC. The techniques may be applied in a substantially similar manner with other syntax structures. Although not required, in some examples, there may be some benefits of using an extension of a parameters set such as SPS extension. For instance, some examples of video decoder300may not be configured to process and output 360-degree video. By using an extension of a parameter set, video decoder300may be configured to receive all information needed to decode video that is not needed for 360-degree video. For instance, if information for 360-degree video were included before other types of information such as height and width of the picture, some examples of video decoder300may not be able to parse through the 360-degree video syntax elements and generate an error. However, by using an extension, such as SPS extension, video decoder300may receive information needed to decode most types of video.

Moreover, by using a parameter set such as the example parameter sets described in this disclosure, it may be possible to use the same parameter set that is used even where 360-degree video is not included in the bitstream. Accordingly, video encoder200may be configured to signal and video decoder300may be configured to receive, as part of the syntax structure that is used even when 360-degree video is not included, information indicating that the coded bitstream includes 360-degree video. For instance, the syntax structure may be a syntax structure that used in signaling non-360-degree video information and can also be used to signal 360-degree video information. Accordingly, the syntax structure forms part of a standard-conforming bitstream regardless of whether the standard-conforming bitstream includes 360-degree video. For example, the syntax structure may be used for encoding and decoding a bitstream carrying only non-360-degree video data, only 360-degree video data, or both non-360-degree video data and 360-degree video data.

Video encoder200and video decoder300may use extension mechanism such as Sequence Parameter Set (SPS) extension (e.g., SPS extensions bits in H.265) to indicate that the coded bitstream is a 360-degree video bitstream. Using this extension mechanism may allow the 360-degree video and non-360-degree video information to be signaled in the same parameter set.

As one example, when the SPS extension bit indicates that the bitstream is a 360-degree bitstream, projection type of the 360-degree video may be signaled. Examples of the projection type include equiretangular projection (ERP), cubemap projection (CMP) and its derivative projections such as adjusted cubemap (ACP) and equatorial cylindrical projection (ECP). In some examples, for each projection type, further information may be signaled. As one example, a packing arrangement for the geometry projection, or rotation degrees for faces of the geometry projection may be signaled. For example, for cubemap projection, video encoder200may further signal and video decoder300may receive packing arrangement of the cube face (e.g., in 3×4 or 2×3 arrangement) and also rotation degree of each face.

In some examples, instead of or in addition to the SPS or SPS extension, one or more bits may also be signaled in Video Parameter Set (VPS) to indicate 360-degree video bitstream. Such signaling in the VPS may be conveyed using currently reserved bits to avoid interference with non-360-degree video bitstream.

Tables below exhibit example of signaling the 360-degree projection information in SPS extension. As one example, in the example syntax structure shown in Table 1 includes the syntax element: sps_360Video_extension_flag. The sps_360Video_extension_flag is one example of information for 360-degree video (e.g., information indicating whether a coded bitstream includes 360-degree video). As indicated in Table 1, if sps_360Video_extension_flag is true (e.g., the information indicates that the coded bitstream includes 360-degree video), then the example syntax structure includes additional information for 360-degree video (e.g., the syntax elements for sps_360_Video_extension( ). Accordingly, the sps_360Video_extension_flag is an example of a syntax element indicating whether a coded bitstream includes 360-degree video. Based on the coded bitstream including 360-degree video, the syntax structure (e.g., sps_extension) information for the 360-degree video (e.g., the bitstream includes sps_360_Video_extension( ) based on whether sps_360Video_extension_flag is true or false).

geometry_type specifies geometry projection of the 360-degree video.

As indicated in Table 2, if sps_360Video_extension_flag is true, then the bitstream includes sps_360_Video_extension( ) which includes syntax element that specifies the geometry projection. Accordingly, in one or more examples, based on the syntax element indicating that the bitstream includes information for 360-degree video (e.g. based on sps_360Video_extension_flag being true), video encoder200signals and video decoder300receives information indicating a geometry projection (e.g., signals or receives geometry_type).

In the example of Tables 1 and 2, sps_360Video_extension_flag, sps_360_Video_extension( ) and geometry_type are described as being part of the same syntax structure (e.g., SPS extension). However, the example techniques are not limited. In some examples, a syntax element like sps_360Video_extension_flag may be in a first syntax structure, and if true, indicates to receive information from a different syntax structure (e.g., sps_360_Video_extension( ) is in a different syntax structure. Also, it may be possible for a syntax element like geometry_type to be in yet another different syntax structure.

Therefore, video encoder200may signal and video decoder300may receive information indicating the geometry projection, as part of the same syntax structure, as the information for the 360-degree video, or as part of a different syntax structure. Similarly, video encoder200may signal and video decoder300may receive, as part of the same syntax structure or different syntax structure, one or more of information indicating a packing arrangement for the geometry projection, or rotation degrees for faces of the geometry projection. It should be understood that packing arrangement and rotation degrees for faces are merely two non-limiting examples of additional information for 360-degree video that may be signaled and received in various different syntax structures. Other examples are contemplated and included as part of this disclosure.

FIGS. 2A and 2Bare conceptual diagram illustrating an example quadtree binary tree (QTBT) structure130, and a corresponding coding tree unit (CTU)132. The solid lines represent quadtree splitting, and dotted lines indicate binary tree splitting. In each split (i.e., non-leaf) node of the binary tree, one flag is signaled to indicate which splitting type (i.e., horizontal or vertical) is used, where 0 indicates horizontal splitting and 1 indicates vertical splitting in this example. For the quadtree splitting, there is no need to indicate the splitting type, since quadtree nodes split a block horizontally and vertically into 4 sub-blocks with equal size. Accordingly, video encoder200may encode, and video decoder300may decode, syntax elements (such as splitting information) for a region tree level of QTBT structure130(i.e., the solid lines) and syntax elements (such as splitting information) for a prediction tree level of QTBT structure130(i.e., the dashed lines). Video encoder200may encode, and video decoder300may decode, video data, such as prediction and transform data, for CUs represented by terminal leaf nodes of QTBT structure130.

In general, CTU132ofFIG. 2Bmay be associated with parameters defining sizes of blocks corresponding to nodes of QTBT structure130at the first and second levels. These parameters may include a CTU size (representing a size of CTU132in samples), a minimum quadtree size (MinQTSize, representing a minimum allowed quadtree leaf node size), a maximum binary tree size (MaxBTSize, representing a maximum allowed binary tree root node size), a maximum binary tree depth (MaxBTDepth, representing a maximum allowed binary tree depth), and a minimum binary tree size (MinBTSize, representing the minimum allowed binary tree leaf node size).

The root node of a QTBT structure corresponding to a CTU may have four child nodes at the first level of the QTBT structure, each of which may be partitioned according to quadtree partitioning. That is, nodes of the first level are either leaf nodes (having no child nodes) or have four child nodes. The example of QTBT structure130represents such nodes as including the parent node and child nodes having solid lines for branches. If nodes of the first level are not larger than the maximum allowed binary tree root node size (MaxBTSize), they can be further partitioned by respective binary trees. The binary tree splitting of one node can be iterated until the nodes resulting from the split reach the minimum allowed binary tree leaf node size (MinBTSize) or the maximum allowed binary tree depth (MaxBTDepth). The example of QTBT structure130represents such nodes as having dashed lines for branches. The binary tree leaf node is referred to as a coding unit (CU), which is used for prediction (e.g., intra-picture or inter-picture prediction) and transform, without any further partitioning. As discussed above, CUs may also be referred to as “video blocks” or “blocks.”

In one example of the QTBT partitioning structure, the CTU size is set as 128×128 (luma samples and two corresponding 64×64 chroma samples), the MinQTSize is set as 16×16, the MaxBTSize is set as 64×64, the MinBTSize (for both width and height) is set as 4, and the MaxBTDepth is set as 4. The quadtree partitioning is applied to the CTU first to generate quad-tree leaf nodes. The quadtree leaf nodes may have a size from 16×16 (i.e., the MinQTSize) to 128×128 (i.e., the CTU size). If the leaf quadtree node is 128×128, it will not be further split by the binary tree, since the size exceeds the MaxBTSize (i.e., 64×64, in this example). Otherwise, the leaf quadtree node will be further partitioned by the binary tree. Therefore, the quadtree leaf node is also the root node for the binary tree and has the binary tree depth as 0. When the binary tree depth reaches MaxBTDepth (4, in this example), no further splitting is permitted. When the binary tree node has width equal to MinBTSize (4, in this example), it implies no further horizontal splitting is permitted. Similarly, a binary tree node having a height equal to MinBTSize implies no further vertical splitting is permitted for that binary tree node. As noted above, leaf nodes of the binary tree are referred to as CUs, and are further processed according to prediction and transform without further partitioning.

FIG. 3is a block diagram illustrating an example video encoder200that may perform the techniques of this disclosure.FIG. 3is provided for purposes of explanation and should not be considered limiting of the techniques as broadly exemplified and described in this disclosure. For purposes of explanation, this disclosure describes video encoder200in the context of video coding standards such as the HEVC video coding standard and the H.266 video coding standard in development. However, the techniques of this disclosure are not limited to these video coding standards, and are applicable generally to video encoding and decoding.

In the example ofFIG. 3, video encoder200includes video data memory230, mode selection unit202, residual generation unit204, transform processing unit206, quantization unit208, inverse quantization unit210, inverse transform processing unit212, reconstruction unit214, filter unit216, decoded picture buffer (DPB)218, and entropy encoding unit220.

Video encoder200stores reconstructed blocks in DPB218. For instance, in examples where operations of filter unit216are not needed, reconstruction unit214may store reconstructed blocks to DPB218. In examples where operations of filter unit216are needed, filter unit216may store the filtered reconstructed blocks to DPB218. Motion estimation unit222and motion compensation unit224may retrieve a reference picture from DPB218, formed from the reconstructed (and potentially filtered) blocks, to inter-predict blocks of subsequently encoded pictures. In addition, intra-prediction unit226may use reconstructed blocks in DPB218of a current picture to intra-predict other blocks in the current picture.

Video encoder200represents an example of a device configured to encode video data including a memory configured to store video data, and one or more processing units implemented in circuitry and configured to signal, as part of a syntax structure, information for 360-degree video, and signal, as part of the same syntax structure, information for non-360-degree video. For instance, video encoder200may signal, as part of a syntax structure, information indicating that a coded bitstream includes 360-degree video (e.g., the sps_360Video_extension_flag), and signal, as part of the same syntax structure or different syntax structure, information for the 360-degree video based on the bitstream including 360-degree video (e.g., sps_360_Video_extension( ) is signaled based on sps_360Video_extension_flag being true).

Video encoder200may signal, as part of the same syntax structure, information for non-360-degree video (e.g., the syntax elements shown in Table. 1). In some examples, video encoder200may be configured to signal, as part of the syntax structure that is used even when 360-degree video is not included, information indicating that the coded bitstream includes 360-degree video. Accordingly, the syntax structure may be a syntax structure that used in signaling non-360-degree video information and can also be used to signal 360-degree video information. For example, the syntax structure forms part of a standard-conforming bitstream regardless of whether the standard-conforming bitstream includes 360-degree video). For instance, the syntax structure may be used for encoding and decoding a bitstream carrying only non-360-degree video data, only 360-degree video data, or both non-360-degree video data and 360-degree video data.

The syntax structure includes one or more syntax elements that together may form one or more of: a sequence parameter set (SPS), a SPS extension, a picture parameter set (PPS), a PPS extension, a video parameter set (VPS), a VPS extension, a picture header, a slice header, a tile header, or a supplemental enhancement information (SEI) message. As one example, the syntax structure is one or more syntax elements that together form an SPS extension. As one example, the syntax structure is one or more syntax elements that together form a VPS.

In some examples, the information indicating that the coded bitstream includes 360-degree video is an extension bit that is a previously reserved bit of the syntax structure. For instance, in the SPS extension N-bits may be reserved and not assigned for use to any syntax element (e.g., the standard defines N-additional bits that are available within the SPS extension and not assigned to any other syntax element). The information indicating that the coded bitstream includes 360-degree video is one of the N-bits previously reserved for the SPS extension.

In some examples, video encoder200may be configured to signal information indicating geometry projection (e.g., as shown in Table 2), as part of the syntax structure, information for the 360-degree video based on the coded bitstream including 360-degree video. Video encoder200may also signal information indicating one or more of a packing arrangement for the geometry projection and rotation degrees for faces of the geometry projection.

Video encoder200may signal information, as part of the syntax structure, a syntax element indicating whether a bitstream includes additional information for 360-degree video (e.g., signal sps_360Video_extension_flag). If the syntax element indicates that the bitstream includes additional information for 360-degree video (e.g., signal sps_360Video_extension_flag is true), video encoder200may signal information indicating a geometry projection (e.g., geometry_type). In one example, video encoder200may signal information indicating the geometry projection as part of the same syntax structure that includes sps_360Video_extension_flag. In one example, video encoder200may signal information indicating the geometry projection as part of a different syntax structure that includes sps_360Video_extension_flag.

In the example ofFIG. 4, video decoder300includes coded picture buffer (CPB) memory320, entropy decoding unit302, prediction processing unit304, inverse quantization unit306, inverse transform processing unit308, reconstruction unit310, filter unit312, and decoded picture buffer (DPB)314. Prediction processing unit304includes motion compensation unit316and intra-prediction unit318. Prediction processing unit304may include addition units to perform prediction in accordance with other prediction modes. As examples, prediction processing unit304may include a palette unit, an intra-block copy unit (which may form part of motion compensation unit316), an affine unit, a linear model (LM) unit, 360-degree video decoding unit, or the like. In other examples, video decoder300may include more, fewer, or different functional components.

Video decoder300may store the reconstructed blocks in DPB314. As discussed above, DPB314may provide reference information, such as samples of a current picture for intra-prediction and previously decoded pictures for subsequent motion compensation, to prediction processing unit304. Moreover, video decoder300may output decoded pictures from DPB for subsequent presentation on a display device, such as display device118ofFIG. 1.

In this manner, video decoder300represents an example of a video decoding device including a memory configured to store video data, and one or more processing units implemented in circuitry and configured to receive, as part of a syntax structure, information for 360-degree video, and receive, as part of the same syntax structure, information for non-360-degree video. For instance, video decoder300may receive, as part of a syntax structure, information indicating that a coded bitstream includes 360-degree video (e.g., the sps_360Video_extension_flag), and receive, as part of the same syntax structure or different syntax structure, information for the 360-degree video based on the reception of the information that the coded bitstream includes 360-degree video (e.g., sps_360_Video_extension( ) is received based on sps_360Video_extension_flag being true).

Video decoder300may receive, as part of the same syntax structure, information for non-360-degree video (e.g., the syntax elements shown in Table. 1). In some examples, video decoder300may be configured to receive, as part of the syntax structure that is used even when 360-degree video is not included, information indicating that the coded bitstream includes 360-degree video. As described above, the syntax structure may be a syntax structure that used in signaling non-360-degree video information and can also be used to signal 360-degree video information. For example, the syntax structure forms part of a standard-conforming bitstream regardless of whether the standard-conforming bitstream includes 360-degree video. Accordingly, the syntax structure may be used for encoding and decoding a bitstream carrying only non-360-degree video data, only 360-degree video data, or both non-360-degree video data and 360-degree video data.

The syntax structure includes one or more syntax elements that together may form one or more of: a sequence parameter set (SPS), a SPS extension, a picture parameter set (PPS), a PPS extension, a video parameter set (VPS), a VPS extension, a picture header, a slice header, a tile header, or a supplemental enhancement information (SEI) message. As one example, the syntax structure is one or more syntax elements that together form an SPS extension. As one example, the syntax structure is one or more syntax elements that together form a VPS.

In some examples, the information indicating that the coded bitstream includes 360-degree video is an extension bit that is a previously reserved bit of the syntax structure. For instance, in the SPS extension N-bits may be reserved and not assigned for use to any syntax element (e.g., the standard defines N-additional bits that are available within the SPS extension and not assigned to any other syntax element). The information indicating that the coded bitstream includes 360-degree video is one of the N-bits previously reserved for the SPS extension.

In some examples, video decoder300may be configured to receive information indicating geometry projection (e.g., as shown in Table 2), as part of the syntax structure, information for the 360-degree video based on the coded bitstream including 360-degree video. Video decoder300may also receive information indicating one or more of a packing arrangement for the geometry projection and rotation degrees for faces of the geometry projection.

Video decoder300may receive information, as part of the syntax structure, a syntax element indicating whether a bitstream includes additional information for 360-degree video (e.g., signal sps_360Video_extension_flag). If the syntax element indicates that the bitstream includes additional information for 360-degree video (e.g., signal sps_360Video_extension_flag is true), video decoder300may receive information indicating a geometry projection (e.g., geometry_type). In one example, video decoder300may receive information indicating the geometry projection as part of the same syntax structure that includes sps_360Video_extension_flag. In one example, video decoder300may receive information indicating the geometry projection as part of a different syntax structure that includes sps_360Video_extension_flag.

FIG. 5is a flowchart illustrating an example method of encoding video data. Video data memory230(or some other local memory of video encoder200or possibly a memory external to video encoder200) may store video data of syntax elements of a syntax structure. Video encoder200includes one or more processing units implemented in fixed-function or programmable circuitry. Video encoder200is configured to signal, as part of a syntax structure, information indicating that a coded bitstream includes 360-degree video (400). For example, video encoder200may signal the sps_360Video_extension_flag.

In some examples, the syntax structure includes one or more syntax elements that together form one or more of a sequence parameter set (SPS), a SPS extension, a picture parameter set (PPS), a PPS extension, a video parameter set (VPS), a VPS extension, a picture header, a slice header, a tile header, or a supplemental enhancement information (SEI) message. For example, the syntax structure includes one or more syntax elements that together form a sequence parameter set (SPS) extension. As another example, the syntax structure includes one or more syntax elements that together form a video parameter set (VPS).

The information indicating that the coded bitstream includes 360-degree video may be an extension bit that is a previously reserved bit of the syntax structure. For example, the SPS extension may define N-number of reserved bits that are reserved to be assigned for indicating additional information. The information indicating that the coded bitstream includes 360-degree video may utilize one of these N-number of reserved bits.

In some examples, video encoder200may signal, as part of the syntax structure that is used even when 360-degree video is not included, information indicating that the coded bitstream includes 360-degree video. In way, even if video decoder types are not configured to decode 360-degree video, such video decoder types may still be configured to process the syntax structure without outputting an error. Also, if syntax structures like an extension are used, information that video decoder types that do not decode 360-degree video would be received before information for 360-degree video allowing for these video decoder types to operate without error. Accordingly, the syntax structure may be a syntax structure that used in signaling non-360-degree video information and can also be used to signal 360-degree video information (e.g., the syntax structure forms part of a standard-conforming bitstream regardless of whether the standard-conforming bitstream includes 360-degree video). For example, the syntax structure may be used for encoding and decoding a bitstream carrying only non-360-degree video data, only 360-degree video data, or both non-360-degree video data and 360-degree video data.

Video encoder200may be configured to signal information for the 360-degree video based on the coded bitstream including 360-degree video (402). For example, video encoder200may signal information indicating a geometry projection. Also, video encoder200may signal information indicating one or more of a packing arrangement for the geometry projection and rotation degrees for faces of the geometry projection.

In some examples, video encoder200may signal information, in the same syntax structure or a different syntax structure, for the 360-degree video based on the coded bitstream including 360-degree video. For example, the information for the geometry projection may be in the same parameter set as the information indicating that the coded bitstream includes 360-degree video (e.g., both are in the SPS extension). As another example, the information for the geometry projection may be in a different parameter set that the information indicating that the coded bitstream includes 360-degree video (e.g., one is in the SPS extension and the other is in the VPS).

Video encoder200may be configured to signaling, as part of the same syntax structure, information for non-360-degree video (404). For example, as illustrated in Table 1, video encoder200may signal non-360-degree video such as sps_range_extension_flag, sps_multilayer_extension_flag, and their corresponding information when sps_range_extension_flag and/or sps_multilayer_extension_flag are true. Such information for non-360-degree video is in the same parameter set (e.g., SPS extension) as the information indicating that the coded bitstream includes 360-degree video (e.g., sps_360Video_extension_flag).

Video encoder200may encode the 360-degree video and the non-360-degree video data (406). For instance, video encoder200may utilize techniques of the HEVC standard or those of the VVC standard under development. In some examples, video encoder200may determine reference blocks for blocks of video data being encoded, determine residual data between the reference blocks and the blocks being encoded, and signal the residual data along with information to determine the location of the reference blocks.

FIG. 6is a flowchart illustrating an example method of decoding video data. For instance, a memory (e.g., CPB memory320, DPB314, some other local memory of video decoder300, a memory external to video decoder300) is configured to store video data. Video decoder300includes one or more processing units implemented in fixed-function or programmable circuitry. Video decoder300is configured to receive, as part of a syntax structure stored as video data in the memory, information indicating that a coded bitstream includes 360-degree video (500). For example, video decoder300may receive the sps_360Video_extension_flag.

In some examples, the syntax structure includes one or more syntax elements that together form one or more of a sequence parameter set (SPS), a SPS extension, a picture parameter set (PPS), a PPS extension, a video parameter set (VPS), a VPS extension, a picture header, a slice header, a tile header, or a supplemental enhancement information (SEI) message. For example, the syntax structure includes one or more syntax elements that together form a sequence parameter set (SPS) extension. As another example, the syntax structure includes one or more syntax elements that together form a video parameter set (VPS).

The information indicating that the coded bitstream includes 360-degree video may be an extension bit that is a previously reserved bit of the syntax structure. For example, the SPS extension may define N-number of reserved bits that are reserved to be assigned for indicating additional information. The information indicating that the coded bitstream includes 360-degree video may utilize one of these N-number of reserved bits.

In some examples, video decoder300may receive, as part of the syntax structure that is used even when 360-degree video is not included, information indicating that the coded bitstream includes 360-degree video. In way, even if video decoder types are not configured to decode 360-degree video, such video decoder types may still be configured to process the syntax structure without outputting an error. Also, if syntax structures like an extension are used, information that video decoder types that do not decode 360-degree video would be received before information for 360-degree video allowing for these video decoder types to operate without error. As described above, the syntax structure may be a syntax structure that used in signaling non-360-degree video information and can also be used to signal 360-degree video information. For example, the syntax structure forms part of a standard-conforming bitstream regardless of whether the standard-conforming bitstream includes 360-degree video). Accordingly, the syntax structure may be used for encoding and decoding a bitstream carrying only non-360-degree video data, only 360-degree video data, or both non-360-degree video data and 360-degree video data.

Video decoder300may be configured to receive information for the 360-degree video based on the reception of the information that the coded bitstream includes 360-degree video (502). For example, video decoder300may receive information indicating a geometry projection. Also, video decoder300may receive information indicating one or more of a packing arrangement for the geometry projection and rotation degrees for faces of the geometry projection.

In some examples, video decoder300may receive information, in the same syntax structure or a different syntax structure, for the 360-degree video based on the reception of the information that the coded bitstream includes 360-degree video. For example, the information for the geometry projection may be in the same parameter set as the information indicating that the coded bitstream includes 360-degree video (e.g., both are in the SPS extension). As another example, the information for the geometry projection may be in a different parameter set that the information indicating that the coded bitstream includes 360-degree video (e.g., one is in the SPS extension and the other is in the VPS).

Video decoder300may be configured to receive, as part of the same syntax structure, information for non-360-degree video (504). For example, as illustrated in Table 1, video decoder300may receive non-360-degree video such as sps_range_extension_flag, sps_multilayer_extension_flag, and their corresponding information when sps_range_extension_flag and/or sps_multilayer_extension_flag are true. Such information for non-360-degree video is in the same parameter set (e.g., SPS extension) as the information indicating that the coded bitstream includes 360-degree video (e.g., sps_360Video_extension_flag).

Video decoder300may be configured to decode the 360-degree video data and the non-360-degree video data (506). For instance, video decoder300may utilize techniques of the HEVC standard or those of the VVC standard under development. In some examples, video decoder300may receive information for residual blocks and information to determine location of reference blocks. Video decoder300may determine the reference blocks for blocks of video data being decoded and determine the residual data based on the received. Video decoder300may reconstruct the blocks of video data based on the reference blocks and the residual data.