Partitioning of coded point cloud data

Systems and methods for decoding a video stream, a method includes receiving video stream, the video stream including: a frame of a plurality of two-dimensional (2D) pictures that are layers of the frame, each of the plurality of 2D pictures having a respective attribute of a same three-dimensional (3D) representation, frame partition information that indicates the frame is partitioned into a plurality of sub-frames, each of the plurality of sub-frames being a respective combination of a sub-region of each picture of the plurality of 2D pictures, and 3D bounding box information that specifies a 3D position corresponding to a sub-frame of the plurality of sub-frames; the method further includes identifying the sub-frame using the frame partition information; and decoding the sub-frame identified.

FIELD

This disclosure is directed to a set of advanced video coding technologies, more specifically, video based point cloud compression.

BACKGROUND

Advanced three-dimensional (3D) representations of the world are enabling more immersive forms of interaction and communication. They also allow machines to understand, interpret, and navigate our world. Point clouds have been widely used as a 3D representation of the world. Several use cases associated with point cloud data have been identified, and some corresponding requirements for point cloud representation and compression have been developed.

SUMMARY

Some embodiments of the present disclosure provide techniques for signaling of partitioning information in a coded point cloud bitstream. A video-based point cloud compression (V-PCC) scheme of the present disclosure may utilize generic video codecs for point cloud compression. Some embodiments of the present disclosure provide a functionality enabling partial encoding, delivery, and decoding of the coded point cloud bitstream.

In some embodiments, a method for decoding a video stream with at least one processor is provided. The method comprises receiving the video stream, the video stream including: a frame of a plurality of two-dimensional (2D) pictures that are layers of the frame, each of the plurality of 2D pictures having a respective attribute of a same three-dimensional (3D) representation, frame partition information that indicates the frame is partitioned into a plurality of sub-frames, each of the plurality of sub-frames being a respective combination of a sub-region of each picture of the plurality of 2D pictures, and 3D bounding box information that specifies a 3D position corresponding to a sub-frame of the plurality of sub-frames. The method further comprises identifying the sub-frame using the frame partition information; and decoding the sub-frame identified.

In an embodiment, the frame partition information, of the video stream received, includes 2D bounding box information that specifies at least one among a position and boundary of the sub-frame in the frame.

In an embodiment, the 3D bounding box information specifies a 3D position of a 2D bounding box specified in the 2D bounding box information, and the method further comprises, after decoding the sub-frame identified, obtaining a point cloud from the video stream by using the 3D bounding box information.

In an embodiment, the plurality of 2D pictures includes a first picture, a second picture, and a third picture, the first picture being a texture image, the second picture being a geometry image, and the third picture being an occupancy map.

In an embodiment, the method further comprises obtaining, after decoding the sub-frame identified, a point cloud from the video stream by using the 3D bounding box information.

In an embodiment, the method further comprises determining whether the sub-frame is independently coded, wherein the decoding the sub-frame comprises decoding the sub-frame independently from other sub-frames of the plurality of sub-frames, in a case where the sub-frame is determined to be independently coded.

In an embodiment, the video stream received includes a frame parameter set that includes the frame partition information and the 3D bounding box information

in an embodiment, the video stream further includes an additional frame of a plurality of 2D pictures that are layers of the additional frame, each of the plurality of 2D pictures of the additional frame having a respective attribute of a same additional 3D representation, and the method further comprises: identifying a sub-frame of the additional frame using the frame partition information used to identify the sub-frame of the frame, and decoding the sub-frame of the additional frame identified.

In some embodiments, a system for decoding a video stream s provided. The system comprises memory configured to store computer program code; and at least one processor configured to receive the video stream, access the computer program code, and operate as instructed by the computer program code, wherein the video stream includes: a frame of a plurality of two-dimensional (2D) pictures that are layers of the frame, each of the plurality of 2D pictures having a respective attribute of a same three-dimensional (3D) representation, frame partition information that indicates the frame is partitioned into a plurality of sub-frames, each of the plurality of sub-frames being a respective combination of a sub-region of each picture of the plurality of 2D pictures, and 3D bounding box information that specifies a 3D position corresponding to a sub-frame of the plurality of sub-frames, and the computer program code includes: identifying code configured to cause the at least one processor to identify the sub-frame using the frame partition information; and decoding code configured to cause the at least one processor to decode the sub-frame identified.

In an embodiment, the frame partition information, of the video stream, includes 2D bounding box information that specifies at least one among a position and boundary of the sub-frame in the frame.

In an embodiment, the 3D bounding box information specifies a 3D position of a 2D bounding box specified in the 2D bounding box information, and the computer program code further includes obtaining code configured to cause the at least one processor to obtain a point cloud from the video stream by using the 3D bounding box information.

In an embodiment, the plurality of 2D pictures includes a first picture, a second picture, and a third picture, the first picture being a texture image, the second picture being a geometry image, and the third picture being an occupancy map.

In an embodiment, the computer program code further comprises obtaining code configured to cause the at least one processor to obtain, after decoding the sub-frame identified, a point cloud from h video stream by using the 3D bounding box information.

In an embodiment, the computer program code further comprises determining code configured to cause the at least one processor to determine whether the sub-frame is independently coded, and the decoding code is configured to cause the at least one processor to decode the sub-frame independently from other sub-frames of the plurality of sub-frames, in a case where the sub-frame is determined to be independently coded.

In an embodiment, the video stream includes a frame parameter set that includes the frame partition information and the 3D bounding box information.

In some embodiments, a non-transitory computer-readable medium storing computer instructions is provided. The computer instructions, when executed by at least one processor, cause the at least one processor to: identify a sub-frame, from a video stream received by the at least one rocessor, using frame partition information; decode the sub-frame identified; and obtain, after decoding the sub-frame identified, a. point cloud from the video stream by using three-dimensional (3D) bounding box information, wherein the video stream includes: a frame of a plurality of two-dimensional (2D) pictures that are layers of the frame, each of the plurality of 2D pictures having a respective attribute of a same 3D representation, the frame partition information, the frame partition information indicating the frame is partitioned into a plurality of sub-frames, including the sub-frame, each of the plurality of sub-frames being a respective combination of a sub-region of each picture of the plurality of 2D pictures, and the 3D bounding box information, the 3D bounding box information specifying a 3D position corresponding to the sub-frame of the plurality of sub-frames.

DETAILED DESCRIPTION

A point cloud is a set of points in a 3D space, each with associated attributes, e.g. color, material properties, etc. Point clouds can be used to reconstruct an object or a scene as a composition of such points. They can be captured using multiple cameras and depth sensors in various setups and may be made up of thousands up to billions of points in order to realistically represent reconstructed scenes.

Compression technologies are needed to reduce the amount of data required to represent a point cloud. As such, technologies may be needed for lossy compression of point clouds for use in real-time communications and six Degrees of Freedom (Don virtual reality. In addition, technology is sought for lossless point cloud compression in the context of dynamic mapping for autonomous driving and cultural heritage applications, etc. MPEG has started working on a standard to address compression of geometry and attributes such as colors and reflectance, scalable/progressive coding, coding of sequences of point clouds captured over time, and random access to subsets of the point cloud.

The main philosophy behind video-based point cloud compression (V-PCC) is to leverage existing video codecs to compress the geometry, occupancy, and texture of a dynamic point cloud as three separate video sequences. The extra metadata needed to interpret the three video sequences may be compressed separately. A small portion of the overall bitstream is the metadata, which could be encoded/decoded efficiently using software implementation. The bulk of the information may be handled by the video codec.

With reference toFIGS. 1-4, an embodiment of the present disclosure for implementing encoding and decoding structures of the present disclosure are described. The encoding and decoding structures of the present disclosure may implement aspects of V-PCC described above.

FIG. 1illustrates a simplified block diagram of a communication system100according to an embodiment of the present disclosure. The system100may include at least two terminals110,120interconnected via a network150. For unidirectional transmission of data, a first terminal110may code video data at a local location for transmission to the other terminal120via the network150. The second terminal120may receive the coded video data of the other terminal from the network150, decode the coded data and display the recovered video data. Unidirectional data transmission may be common in media serving applications and the like.

FIG. 1illustrates a second pair of terminals130,140provided to support bidirectional transmission of coded video that may occur, for example, during videoconferencing. For bidirectional transmission of data., each terminal130,140may code video data captured at a. local location for transmission to the other terminal via the network150. Each terminal130,140also may receive the coded video data transmitted by the other terminal, may decode the coded data and may display the recovered video data at a local display device.

InFIG. 1, the terminals110-140may be, for example, servers, personal computers, and smart phones, and/or any other type of terminal. For example, the terminals (110-140) may be laptop computers, tablet computers, media players and/or dedicated video conferencing equipment. The network150represents any number of networks that convey coded video data among the terminals110-140including, for example, wireline and/or wireless communication networks. The communication network150may exchange data in circuit-switched and/or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and/or the Internet. For the purposes of the present discussion, the architecture and topology of the network150may be immaterial to the operation of the present disclosure unless explained herein below.

FIG. 2illustrates, as an example of an application for the disclosed subject matter, a placement of a video encoder and decoder in a streaming environment. The disclosed subject matter can be used with other video enabled applications, including, for example, video conferencing, digital TV, storing of compressed video on digital media including CD, DVD, memory stick and the like, and so on.

As illustrated inFIG. 2, a streaming system200may include a capture subsystem213that includes a video source201and an encoder203. The streaming system200may further include at least one streaming server205and/or at least one streaming client206.

The video source201can create, for example, a stream202that includes a 3D point cloud corresponding to a 3D video. The video source201may include, for example, 3D sensors (e.g. depth sensors) or 3D imaging technology (e.g. digital camera(s)), and a computing device that is configured to generate the 3D point cloud using the data received from the 3D sensors or the 3D imaging technology. The sample stream202, which may have a high data volume when compared to encoded video bitstreams, can be processed by the encoder203coupled to the video source201. The encoder203can include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter as described in more detail below. The encoder203may also generate an encoded video bitstream204. The encoded video bitstream204, which may have e a lower data volume when compared to the uncompressed stream202, can be stored on a streaming server205for future use, One or more streaming clients206can access the streaming server205to retrieve video bit streams209that may be copies of the encoded video bitstream204.

The streaming clients206can include a video decoder210and a display212. The video decoder210can, for example, decode video bitstream209, which is an incoming copy of the encoded video bitstream204, and create an outgoing video sample stream211that can be rendered on the display212or another rendering device (not depicted). In some streaming systems, the video bitstreams204,209can be encoded according to certain video coding/compression standards. Examples of such standards include, but are not limited to, ITU-T Recommendation H.265, Versatile Video Coding (VVC), and MPEG/V-PCC.

With reference toFIGS. 3-4, some aspects of V-PCC that may be performed by embodiments of the present disclosure are described below.

FIG. 3illustrates an example functional block diagram of a video encoder203according to an embodiment of the present disclosure.

As illustrated inFIG. 3, the video encoder203may receive a. point cloud. frame(s)350, and generate a geometry image352, a texture image356, and an occupancy map334based on the point cloud frame350. The video encoder203may compress the geometry image352into a compressed geometry image362, the texture image356into a compressed texture image364, and the occupancy map334into a compressed occupancy map372. A multiplexer328of the video encoder203may form a compressed bitstream374that includes the compressed geometry image362, the compressed texture image364, and the compressed occupancy map372.

More specifically, in embodiments, the video encoder203may include a patch generation module302that segments the point cloud frame350into patches. Patches are useful entities of V-PCC. The patch generation process includes decomposing the point cloud frame350into a minimum number of patches with smooth boundaries, while also minimizing the reconstruction error. Encoders of the present disclosure may implement various methods to generate such a decomposition.

The video encoder203may include a patch packing module304that performs a packing process. The packing process includes mapping the extracted patches onto a 2D grid while minimizing the unused space and guaranteeing that every M×M (e.g., 16×16) block of the grid is associated with a unique patch. Efficient patch packing directly impacts the compression efficiency either by minimizing the unused space or ensuring temporal consistency. The patch packing module304may generate the occupancy map334.

The video encoder203may include a geometry image generation module306and a texture image generation module308. In order to better handle the case of multiple points being projected to the same sample, each patch may be projected onto two images, referred to as layers. For example, the geometry image generation module306and the texture image generation module308may exploit the 3D to 2D mapping computed during the packing process of the patch packing module304to store the geometry and texture of the point cloud as images (a.k.a. layers). The generated images/layers may be stored as a video frames) and compressed using a video codec (e.g. HM video codec) according to configurations provided as parameters.

In embodiments, the geometry image generation module306generates the geometry image352and the texture image generation module308generates the texture image356, based on the input point cloud frame350and the occupancy map334. An example of the geometry image352is illustrated inFIG. 5and an example of the texture image356is illustrated inFIG. 6. In an embodiment, the geometry image352may be represented by a monochromatic frame of W×H in YUV420-8 bit format. In an embodiment, the occupancy map334image consists of a binary map that indicates for each cell of the grid whether it belongs to the empty space or to the point cloud. To generate the texture image356, the texture image generation module308may exploit the reconstructed/smoothed geometry358in order to compute the colors to be associated with the re-sampled points.

The video encoder203may include a video compression module322and a video compression module324for compressing the padded geometry image354and the padded texture image360into the compressed geometry image362and the compressed texture image364, respectively.

The video encoder203may include an entropy compression module318for lossless encoding366of the occupancy map334and a video compression module326for lossy encoding368of the occupancy map334.

in embodiments, the video encoder203may include a smoothing module310for generating smoothed geometry358by using a reconstructed geometry image365, provided by the video compression module322, and patch info332. The smoothing procedure of the smoothing module310may aim at alleviating potential discontinuities that may arise at the patch boundaries due to compression artifacts. The smoothed geometry358may be used by the texture image generation module308to generate the texture image356.

The video encoder203may also include an auxiliary patch information compression module312for forming compressed auxiliary patch information370that is provided in the compressed bitstream374by the multiplexer328.

FIG. 4illustrates an example functional block diagram of a video decoder210according to an embodiment of the present disclosure.

As illustrated inFIG. 4, the video decoder210may receive the coded bitstream374from the video encoder203to obtain the compressed texture image362, the compressed geometry image364, the compressed occupancy map372, and the compressed auxiliary patch information370. The video decoder210m,ay decode the compressed texture image362. the compressed geometry image364, the compressed occupancy map372. and the compressed auxiliary patch information370to obtain a decompressed texture image460, a decompressed geometry image462, a decompressed occupancy map464, and decompressed auxiliary patch information466, respectively. Following, the video decoder210may generate a reconstructed point cloud474based on the decompressed texture image460, the decompressed geometry image462, the decompressed occupancy map464, and the decompressed auxiliary patch information466.

In embodiments, the video decoder210may include a demultiplexer402that separates the compressed texture image362, the compressed geometry image364, the compressed occupancy map372, and the compressed auxiliary patch information370of the compressed bitstream374received.

The video decoder210may include a video decompression module404, a video decompression module406, an occupancy map decompression module408. and an auxiliary patch information decompression module410that decode the compressed texture image362, the compressed geometry image364, the compressed occupancy map372, and the compressed auxiliary patch information370, respectively.

The video decoder210may include a geometry reconstruction module412that obtains reconstructed (three dimensional) geometry468based on the decompressed geometry image462, the decompressed occupancy map464, and the decompressed auxiliary patch information466.

The video decoder210may include a smoothing module414that smooths the reconstructed geometry468to obtain smoothed geometry470. The smoothing procedure may aim at alleviating potential discontinuities that may arise at the patch boundaries due to compression artifacts.

The video decoder210may include a texture reconstruction module416for obtaining reconstructed texture472based on the decompressed texture image460and the smoothed geometry470.

The video decoder210may include a color smoothing module418that smooths the color of the reconstructed texture472to obtain a reconstructed point cloud474. Non-neighboring patches in 3D space are often packed next to each other in 2D videos. This implies that pixel values from non-neighboring patches might be mixed up by the block-based video codec. The color smoothing of the color smoothing module418may aim to reduce the visible artifacts that appear at patch boundaries.

As described above, an input cloud may be segmented into several patches and packed into three 2D images, such as an occupancy map, geometry image, and texture image. These images are then compressed by a coding standard such as, for example, HEVC.

It is advantageous for a point cloud compression system to use a region of interest (ROI) in the form of a 3D bounding box. Accordingly, it is advantageous for a content-aware point cloud compression system to fulfill all (or some) of the below features: (1) The ROI is coded with a higher quality than other parts of the point-cloud; (2) the ROI is coded independently from other parts of the point-cloud to facilitate spatial random-access without full-decoding; (3) the independent coding of ROI is harmonized with any system requirements regarding independent (parallel) encoding/decoding; and (4) multiple ROIs is supported.

Some embodiments of the present disclosure may include one or more of the above features. Embodiments of the present disclosure may be used separately or combined in any order. Further, each of the embodiments of the present disclosure, including methods, encoders, and decoders, may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In embodiments, one or more processors may execute a program that is stored in a non-transitory computer-readable medium to perform the functions of methods, encoders, and decoders of the present disclosure.

Embodiments of the present disclosure may accomplish, for example, the following point cloud compression features: (1) Parallel encoding and decoding. Embodiments of the present disclosure may provide a parallel processing implementation with low cost in terms of titrate overhead. (2) Spatial random access. Embodiments of the present disclosure may decode a point cloud corresponding to a region without having to decode an entire bitstream of a video stream.

According to some embodiments, one or more of the above features may be accomplished with a sub-frame design as described below.

With reference toFIG. 7, a video stream.may comprise a coded version of a plurality of frames520wherein each of the frames520corresponds to a respective 3D point cloud frame510that may be obtained by, for example, the video source201, and may be viewable by, for example, the display212. Each of the frames520may include a plurality of two-dimensional pictures that are layers of the frame, wherein each picture of the frame indicates a specific attribute of the corresponding 3D point cloud frame510. For example, with reference toFIG. 8which illustrates a single frame520, the plurality of two-dimensional pictures may include three pictures, such as the geometry image522, the texture image524, and the occupancy map526. Such pictures may be the same or share similarities with the geometry image352, the texture image356, and the occupancy map334described above.

Each frame520may be partitioned into sub-frames, wherein each sub-frame includes a part of each of the plurality of two-dimensional pictures (e.g. a part of the geometry image, a part of the texture image, and a part of the occupancy map), One or more of the sub-frames may correspond to an ROI. For example, with reference toFIG. 8. the frame520may include a suhfrarne A, a subframe B, a subfra e C. a subframe D, and a subframe E. Each of the sub-frames A-E include a portion of the geometry image522, the texture image524, and the occupancy map526. One or more of the sub-frames A-E may correspond to an ROI.

In some embodiments, a sub-fram(e.g. any of subframes A-E) Wray be a rectangular region or a group of tiles. In a case where a sub-frame comprises multiple tiles, the sub-frame may have a rectangular or non-rectangular shape. In an embodiment, a sub-frame may or may not be partitioned into multiple tiles. In a case where the sub-frame is partitioned into multiple tiles, each component of the sub frame (e.g. YUV, XYZ, occupancy map) may have identical tile partitions.

In some embodiments, tiles in the sub-frames can be combined into a rectangular or non-rectangular tile group, but tiles belonging to different sub-frames cannot be grouped. In an embodiment, the tile groups may use the tile group design of VVC.

In some embodiments, 3D bounding box information corresponding to a sub-frame may be signaled or not signaled. The 3D bounding box information may, for example, specify a 3D position of a sub-frame in a point cloud. For example, with reference toFIGS. 7-8, any ones of areas512of the 3D point cloud frame510may correspond to the 3D bounding box and the corresponding position of one of the sub-frames A-E of the frame520within the 3D point cloud frame510.

In some embodiments, any decoding or rendering process (e.g. in-loop filtering, motion compensation) across sub-picture boundaries may be disallowed or allowed. A sub-picture may refer to a sub-portion of a picture (e.g. an area A, B, C. D, or E of a picture522,525, or526illustrated in FIG,8). In some embodiments the boundary of a sub-frame may be extended and padded for motion compensation. In some embodiments, a flag indicating whether the boundary is extended or not is signaled in a Sequence Parameter Set (SPS) of the video bitstream.

In some embodiments, the decoded sub-frame may or may not be resa.mpled to be output. In some embodiments, the spatial ratio between the decoded sub-frame size and the output sub-frame size may be signaled in SPS and may be used to calculate the resampling ratio. In some embodiments, the resampling operations (e.g. adaptive resolution changes) may or may not be applied.

In some embodiments, partitioning information is signaled in a frame parameter set of the video bitstream, which may be activated by a frame. The partitioning information may, for example, indicate whether and how a frame is partitioned into a. plurality of sub-frames. In some embodiments, the partitioning information can be updated within a coded video sequence. In some embodiments, the same partition information may be shared and used by multiple frames, pictures, slices, tile groups, and VCL NAL units with different components.

Provided below is example code of an embodiments of the present disclosure that may be implemented in, for example, V-PCC. As shown below, the code may be provided in a frame parameter set. In an embodiment, the frame parameter set may be provided, in a coded video stream, by an encoder to a decoder.

Semantics of the above code is described below.

“frame_partitioning_enablekflag” equal to 1 specifies that the frame is partitioned into multiple sub-regions (e.g. sub-frames). The sub-bitstream corresponding to each sub-region is accessible and extractable from the entire bitstream. Each region shall be able to be independently decoded. “frame_partitioning_enabled_flag” equal to 0 specifies that the frame may or may not be partitioned into multiple sub-regions (e.g. sub-frames).

“tile_info_present_flag” equal to 1 specifies that each component bitstream contains the tile information, which is aligned with bounding box information for spatial random access and parallel processing. “tile_info_present_flag” equal to 0 specifies that each 2-dimensional bounding box is explicitly signaled on pixel level.

“num_tile_columns_minus1” plus 1 specifies the number of file columns partitioning the frame.

“num_tile_rows_minus1” plus 1 specifies the number of tile rows partitioning the frame.

“uniform_spacing_flag” equal to 1 specifies that tile column boundaries and likewise tile row boundaries are distributed uniformly across the frame.

“uniform_spacing_flag” equal to 0 specifies that tile column boundaries and likewise tile row boundaries are not distributed uniformly across the frame, but signaled explicitly using the syntax elements “column_width_minus1[i]” and “row_height_minus1[i]”.

“column_width_minus1 [i]” plus 1 specifies the width of the i-th tile column in units of CTBs.

“row_height_minus1[i]” plus 1 specifies the height of the i-th tile column in units of CTBs.

“single_file_per_sub_frame_flag” equal to 1 specifies that each 2D bounding box that is indicated in this frame parameter set includes one tile. “single_file_per_sub_frame_flag” equal to 0 specifies that each 2D bounding box that is indicated in this frame parameter set may include more than one tile.

“num_sub_frame_in_frame_minus1” plus 1 specifies the number of two-dimensional bounding boxes in each frame referring to the frame parameter set.

“3D bounding_box_info_present_flag” equal to 1 specifies that three-dimensional bounding box information is present. “3D bounding_box_info_present_flag” equal to 0 specifies that three-dimensional hounding box information is not present.

“rect_sub_frame_flag” equal to 0 specifies that tiles within each sub-frame are in raster scan order. “rect_file_group_flag” equal to 1 specifies that tiles within each sub-frame cover a rectangular region of the frame.

“num_tiles_in_sub_frame_minus1” plus 1, when present, specifies the number of tiles in the non-rectangular sub-picture.

“top_left_tile_idx[i]” specifies the tile index of the tile located at the top-left corner of the i-th 2-D bounding box.

“bottom_right_tile_idx[i]” specifies the tile index of the tile located at the bottom-right corner of the i-th 2-D bounding box.

“3D_bounding_box_x[i]”, “3D_bounding_box_y[i]”, and “3D_bounding_box_z[i]” specify the three dimensional position of the i-th three dimensional bounding box corresponding to the i-th two dimensional bounding box, used for the volumetric representation of the point cloud data.

“sub_frame_x[i]” and “sub_frame_y[i]” specify the two dimensional position of the i-th two dimensional bounding box.

“subframe_dx[i]” and “sub_frame_dy[i]” specify respectively the width and the height of the specific 2D bounding box.

“signalled_bounding_box_id_flag” equal to 1 specifies that the bounding box ID for each bounding box is signalled. “signalled_hounding_box _index_flag” equal to 0 specifies that bounding box IDs are not signalled.

“signalled_bounding_box_id_length_minus1” plus 1 specifies the number of bits used to represent the syntax element bounding_box_id[i]. The value of “signalled_bounding_box_id_length_minus1” shall be in the range of 0 to 15, inclusive.

“bounding_box_id[i]” specifies the bounding box ID of the i-th bounding box. The length of the “bounding_box_id[i]” syntax element is “bounding_box_id_length_minus1”+1 bits.

“Independent_decoding_sub_frame_enabled_flag” equal to 1 specifies that each sub_frame may or may not be independently decoded without inter sub-frame operations. “Independent_decoding_sub_frame_enabled_flag” equal to 0 specifies that each sub_frame cannot be independently decoded without inter sub-frame operations. The inter sub-frame operation includes motion compensation and in-loop filtering across bounaries of sub-frames. If not present, the value of “Independent_decoding_sub_frame_enabled_flag” is inferred to be equal to 0.

“post_processing_across_bounding_box_enabled_flag” equal to 1 specifies that any post-processing after decoding the video bitstreams is enabled across the boundaries of sub_frames. “post_processing_across_bounding_box_enabled_flag” equal to 0 specifies that any post-processing after decoding the video bitstreams is disabled across the boundaries of sub_frames. The post processing may include any operations to generate point cloud data from the decoded video sequences.

With reference toFIG. 9. embodiments of the present disclosure may perform a decoding process of coded point cloud data with sub-frame partition.

After receiving at least a portion of a video stream, frame parameters are parsed (601). Following, individual sub-frame partitions are identified. With respect to one or more of the sub-frames, it may be determined whether the sub-frame is independently coded (603). In a case where an individual sub-frame is determined to be independently coded, the individual sub-frame may be decoded independently (604). In a case where there are no independently coded sub-frames, the entire frame may be decoded (605). In embodiments, the decoding of the sub-frame or frame may performed by the decoder210illustrated in FIG,4.

In embodiments, a device700may comprise memory storing computer program code that, when performed by at least one processor, may cause an at least one processor to perform the functions of the decoders and encoders described above.

For example, with reference toFIG. 10, the computer program code of the device700may comprise identifying code710, decoding code730, obtaining code740, and displaying code750.

The identifying code710may be configured to cause the at least one processor to identify one or more sub-frames using frame partition information provided to the device700. The frame partition information may be, for example, any of the information described above that indicates characteristics (e.g. the number, size, shape, and coding dependencies) of the sub-frames within a frame.

The decoding code730may be configured to cause the at least one processor to decode a sub-frame identified. In embodiments, the decoding code730may be configured to perform the functions of the decompression modules of the decoder210illustrated inFIG. 4to decode the sub-frame.

The obtaining code740may be configured to cause the at least one processor to obtain, after decoding the sub-frame identified, a point cloud by using 3D bounding box information corresponding to the sub-frame identified. In embodiments, the obtaining code740may be configured to perform the functions of the geometry reconstruction module412, the smoothing module414, the texture reconstruction module416, and the color smoothing module418of the decoder210illustrated inFIG. 4to obtain a point cloud.

The displaying code750may be configured to cause the at least one processor to display a 3D image corresponding to the point cloud on a display.

In some embodiments, the computer program code may also include determining code720. The determining code720may be configured to cause the at least one processor to determine whether the sub-frame is independently coded, and the decoding code730may be configured to cause the at least one processor to decode the sub-frame independently from other sub-frames of the plurality of sub-frames, in a case where the sub-frame is determined to be independently coded.

The techniques, described above, can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. For example,FIG. 11shows a computer system900suitable for implementing certain embodiments of the disclosure.

Input human interface devices may include one or more of (only one of each depicted): keyboard901, mouse902, trackpad903, touch screen910, data-glove, joystick905, microphone906, scanner907, camera908.

Computer system900can also include human accessible storage devices and their associated media such as optical media including CD/DVD ROM/RW920with CD/DVD or the like media921, thumb-drive922, removable hard drive or solid state drive923, legacy magnetic media such as tape and floppy disc (not depicted), specialized ROM/ASIC/PLD based devices such as security dongles (not depicted), and the like.

Computer system900can also include interface to one or more communication networks. Networks can for example he wireless, wireline, optical. Networks can further be local, wide-area, metropolitan, vehicular and industrial, real-time, delay-tolerant, and so on. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks to include GSM, 3G, 4G, 5G, LIE and the like, IV wireline or wireless wide area digital networks to include cable IV, satellite TV, and terrestrial broadcast TV, vehicular and industrial to include CANBus, and so forth. Certain networks commonly require external network interface adapters that attached to certain general purpose data ports or peripheral buses949(such as, for example USB ports of the computer system900; others are commonly integrated into the core of the computer system900by attachment to a system bus as described below (for example Ethernet interface into a PC computer system or cellular network interface into a smartphone computer system). Using any of these networks, computer system900can communicate with other entities. Such communication can be urn-directional, receive only (for example, broadcast IV), uni-directional send-only (for example CANbus to certain CANhus devices), or hi-directional, for example to other computer systems using local or wide area digital networks. Such communication can include communication to a cloud computing environment955. Certain protocols and protocol stacks can be used on each of those networks and network interfaces as described above.

Aforementioned humaninterface devices, human-accessible storage devices, and network interfaces954can be attached to a core940of the computer system900.

The core940can include one or more Central Processing Units (CPU)941, Graphics Processing Units (GPU)942, specialized programmable processing units in the form of Field Programmable Gate Areas (FPGA)943, hardware accelerators for certain tasks944, and so forth. These devices, along with Read-only memory (ROM)945, Random-access memory946, internal mass storage such as internal non-user accessible hard drives. SSDs, and the like947, may be connected through a system bus948. In some computer systems, the system bus948can be accessible in the form of one or more physical plugs to enable extensions by additional CPUs, GPU, and the like. The peripheral devices can be attached either directly to the core's system bus948, or through a peripheral bus949. Architectures for a peripheral bus include PCI, USB, and the like. A graphics adapter950may be ncluded in the core940.

CPUs941, GPUs942, FPGAs943, and accelerators944can execute certain instructions that, in combination, can make up the aforementioned computer code. That computer code can be stored in ROM945or RAM946. Transitional data can be also be stored in RAM946, whereas permanent data can be stored for example, in the internal mass storage947. Fast storage and retrieve to any of the memory devices can be enabled through the use of cache memory, that can be closely associated with one or more CPU941, GPU942, mass storage947, ROM945, RAM946, and the like.