Adaptive Filtering of Occupancy Map for Dynamic Mesh Compression

An apparatus comprising circuitry configured to: receive as input a three-dimensional mesh represented as at least one geometry patch of an atlas and at least one texture patch of the atlas; create an occupancy map that indicates which pixels of the at least one geometry patch and the at least one texture patch are occupied having a valid value; wherein the occupancy map is configured to be used to reconstruct the mesh; enter lossy mode; apply, while in lossy mode, an adaptive smoothing filter algorithm to the occupancy map to discard at least one edge of the occupancy map, and to reduce a bitrate for transmission of the occupancy map; store, while in lossy mode, the adaptively smoothed occupancy map and at least one occupancy filter threshold; and encode the adaptively smoothed occupancy map and the at least one occupancy filter threshold into a bitstream.

TECHNICAL FIELD

The examples and non-limiting embodiments relate generally to volumetric video coding, and more particularly, to adaptive filtering of occupancy map for dynamic mesh compression.

BACKGROUND

It is known to perform encoding and decoding of images and video.

SUMMARY

In accordance with an aspect, an apparatus includes: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: receive as input a three-dimensional mesh represented as at least one geometry patch of an atlas and at least one texture patch of the atlas; create an occupancy map that indicates which pixels of the at least one geometry patch and the at least one texture patch are occupied having a valid value; wherein the occupancy map is configured to be used to reconstruct the mesh; enter lossy mode; apply, while in lossy mode, an adaptive smoothing filter algorithm to the occupancy map to discard at least one edge of the occupancy map, and to reduce a bitrate for transmission of the occupancy map; store, while in lossy mode, the adaptively smoothed occupancy map and at least one occupancy filter threshold; and encode the adaptively smoothed occupancy map and the at least one occupancy filter threshold into a bitstream.

In accordance with an aspect, an apparatus includes: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: signal information within a visual volumetric video-based coding extension to indicate a type or format of a content of an atlas, the atlas comprising at least one geometry patch and at least one texture patch used to represent a three-dimensional mesh; wherein the information within the visual volumetric video-based coding extension indicates a bitstream including video content in a domain comprising an adaptively smoothed occupancy map configured to be used to reconstruct the three-dimensional mesh; signal a visual volumetric video-based coding occupancy filter present flag associated with at least one occupancy filter threshold configured to be used to reconstruct the three-dimensional mesh; and signal an occupancy filter threshold syntax element within the visual volumetric video-based coding parameter set extension.

In accordance with an aspect, an apparatus includes: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: decode, from or along a bitstream, an adaptively smoothed occupancy map that indicates which pixels of at least one geometry patch and at least one texture patch are occupied having a valid value; wherein the at least one geometry patch and the at least one texture patch represent an encoded three-dimensional mesh; decode at least one occupancy filter threshold from or along the bitstream; and reconstruct the three-dimensional mesh using the adaptively smoothed occupancy map and the at least one occupancy filter threshold.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

The examples described herein relate to the encoding, signaling and rendering of a volumetric video that is based on mesh coding. The examples described herein focus on methods improving the quality of reconstructed mesh surfaces. The examples described herein relate to methods to improve quality of decoded mesh textures and geometry by using a hierarchical representation of the mesh textures and geometry which as a consequence increases compression efficiency of the encoding pipeline.

Volumetric Video Data

Volumetric video data represents a three-dimensional scene or object and can be used as input for AR, VR and MR applications. Such data describes geometry (shape, size, position in 3D-space) and respective attributes (e.g. color, opacity, reflectance, . . . ), plus any possible temporal transformations of the geometry and attributes at given time instances (like frames in 2D video). Volumetric video is either generated from 3D models, i.e. CGI, or captured from real-world scenes using a variety of capture solutions, e.g. multi-camera, laser scan, combination of video and dedicated depth sensors, and more. Also, a combination of CGI and real-world data is possible. Typical representation formats for such volumetric data are triangle meshes, point clouds, or voxels. Temporal information about the scene can be included in the form of individual capture instances, i.e. “frames” in 2D video, or other means, e.g. position of an object as a function of time.

Because volumetric video describes a 3D scene (or object), such data can be viewed from any viewpoint. Therefore, volumetric video is an important format for AR, VR, or MR applications, especially for providing 6DOF viewing capabilities.

Increasing computational resources and advances in 3D data acquisition devices have enabled reconstruction of highly detailed volumetric video representations of natural scenes. Infrared, lasers, time-of-flight and structured light are all examples of devices that can be used to construct 3D video data. Representation of the 3D data depends on how the 3D data is used. Dense voxel arrays have been used to represent volumetric medical data. In 3D graphics, polygonal meshes are extensively used. Point clouds on the other hand are well suited for applications such as capturing real world 3D scenes where the topology is not necessarily a 2D manifold. Another way to represent 3D data is coding this 3D data as a set of textures and a depth map as is the case in the multi-view plus depth framework. Closely related to the techniques used in multi-view plus depth is the use of elevation maps, and multi-level surface maps.

Selected excerpts from the ISO/IEC 23090-5 Visual Volumetric Video-based Coding and Video-based Point Cloud Compression 2nd Edition standard are referred to herein.

Visual volumetric video, a sequence of visual volumetric frames, if uncompressed, may be represented by a large amount of data, which can be costly in terms of storage and transmission. This has led to the need for a high coding efficiency standard for the compression of visual volumetric data.

The V3C specification enables the encoding and decoding processes of a variety of volumetric media by using video and image coding technologies. This is achieved through first a conversion of such media from their corresponding 3D representation to multiple 2D representations, also referred to as V3C components, before coding such information. Such representations may include occupancy, geometry, and attribute components. The occupancy component can inform a V3C decoding and/or rendering system of which samples in the 2D components are associated with data in the final 3D representation. The geometry component contains information about the precise location of 3D data in space, while attribute components can provide additional properties, e.g. texture or material information, of such 3D data. An example is shown inFIG.1AandFIG.1B.

FIG.1Ashows volumetric media conversion at the encoder, andFIG.1Bshows volumetric media conversion at the decoder side. The 3D media102is converted to a series of 2D representations: occupancy118, geometry120, and attribute122. Additional atlas information108is also included in the bitstream to enable inverse reconstruction. Refer to ISO/IEC 23090-5.

As further shown inFIG.1A, a volumetric capture operation104generates a projection106from the input 3D media102. In some examples, the projection106is a projection operation. From the projection106, an occupancy operation110generates the occupancy 2D representation118, a geometry operation112generates the geometry 2D representation120, and an attribute operation114generates the attribute 2D representation122. The additional atlas information108is included in the bitstream116. The atlas information108, the occupancy 2D representation118, the geometry 2D representation120, and the attribute 2D representation122are encoded into the V3C bitstream124to encode a compressed version of the 3D media102. Based on the examples described herein, V3C patch mesh signaling129may also be signaled in the V3C bitstream124or directly to a decoder. The V3C patch mesh signaling129may be used on the decoder side, as shown inFIG.1B.

As shown inFIG.1B, a decoder using the V3C bitstream124derives 2D representations using an occupancy operation128, a geometry operation130and an attribute operation132. The atlas information operation126provides atlas information into a bitstream134. The occupancy operation128derives the occupancy 2D representation136, the geometry operation130derives the geometry 2D representation138, and the attribute operation132derives the attribute 2D representation140. The 3D reconstruction operation142generates a decompressed reconstruction144of the 3D media102, using the atlas information126/134, the occupancy 2D representation136, the geometry 2D representation138, and the attribute 2D representation140.

Additional information that allows associating all these subcomponents and enables the inverse reconstruction, from a 2D representation back to a 3D representation is also included in a special component, referred to herein as the atlas. An atlas consists of multiple elements, namely patches. Each patch identifies a region in all available 2D components and contains information necessary to perform the appropriate inverse projection of this region back to the 3D space. The shape of such regions is determined through a 2D bounding box associated with each patch as well as their coding order. The shape of these regions is also further refined after the consideration of the occupancy information.

Atlases are partitioned into patch packing blocks of equal size. Refer for example to block202inFIG.2, whereFIG.2shows an example of block to patch mapping. The 2D bounding boxes of patches and their coding order determine the mapping between the blocks of the atlas image and the patch indices.FIG.2shows an example of block to patch mapping with4projected patches (204,204-2,204-3,204-4) onto an atlas201when asps patch precedence order flag is equal to 0. Projected points are represented with dark gray. The area that does not contain any projected points is represented with light grey. Patch packing blocks202are represented with dashed lines. The number inside each patch packing block202represents the patch index of the patch (204,204-2,204-3,204-4) to which it is mapped.

Axes orientations are specified for internal operations. For instance, the origin of the atlas coordinates is located on the top-left corner of the atlas frame. For the reconstruction step, an intermediate axes definition for a local 3D patch coordinate system is used. The 3D local patch coordinate system is then converted to the final target 3D coordinate system using appropriate transformation steps.

FIG.3Ashows an example of an atlas coordinate system,FIG.3Bshows an example of a local 3D patch coordinate system, andFIG.3Cshows an example of a final target 3D coordinate system. Refer to ISO/IEC 23090-5.

FIG.3Ashows an example of a single patch302packed onto an atlas image304. This patch302is then converted, with reference toFIG.3B, to a local 3D patch coordinate system (U, V, D) defined by the projection plane with origin O′, tangent (U), bi-tangent (V), and normal (D) axes. For an orthographic projection, the projection plane is equal to the sides of an axis-aligned 3D bounding box306, as shown inFIG.3B. The location of the bounding box306in the 3D model coordinate system, defined by a left-handed system with axes (X, Y, Z), can be obtained by adding offsets TilePatch3dOffsetU308, TilePatch3DOffsetV310, and TilePatch3DOffsetD312, as illustrated inFIG.3C.

V3C High Level Syntax

Coded V3C video components are referred to herein as video bitstreams, while an atlas component is referred to as the atlas bitstream. Video bitstreams and atlas bitstreams may be further split into smaller units, referred to herein as video and atlas sub-bitstreams, respectively, and may be interleaved together, after the addition of appropriate delimiters, to construct a V3C bitstream.

V3C patch information is contained in an atlas bitstream, atlas_sub_bitstream( ) which contains a sequence of NAL units. A NAL unit is specified to format data and provide header information in a manner appropriate for conveyance on a variety of communication channels or storage media. All data are contained in NAL units, each of which contains an integer number of bytes. A NAL unit specifies a generic format for use in both packet-oriented and bitstream systems. The format of NAL units for both packet-oriented transport and sample streams is identical except that in the sample stream format specified in Annex D of ISO/IEC 23090-5 each NAL unit can be preceded by an additional element that specifies the size of the NAL unit.

NAL units in an atlas bitstream can be divided into atlas coding layer (ACL) and non-atlas coding layer (non-ACL) units. The former is dedicated to carry patch data, while the latter is dedicated to carry data necessary to properly parse the ACL units or any additional auxiliary data.

In the nal_unit_header( ) syntax nal_unit_type specifies the type of the RESP data structure contained in the NAL unit as specified in Table 4 of ISO/IEC 23090-5. nal_layer_id specifies the identifier of the layer to which an ACL NAL unit belongs or the identifier of a layer to which a non-ACL NAL unit applies. The value of nal_layer_id shall be in the range of 0 to 62, inclusive. The value of 63 may be specified in the future by ISO/IEC. Decoders conforming to a profile specified in Annex A of ISO/IEC 23090-5 shall ignore (i.e., remove from the bitstream and discard) all NAL units with values of nal_layer_id not equal to 0.

V3C Extension Mechanisms

While designing the V3C specification it was envisaged that amendments or new editions can be created in the future. In order to ensure that the first implementations of V3C decoders are compatible with any future extension, a number of fields for future extensions to parameter sets were reserved.

For example, the second edition of V3C introduced an extension in VPS related to MIV and the packed video component.

Rendering and Meshes

A polygon mesh is a collection of vertices, edges and faces that defines the shape of a polyhedral object in 3D computer graphics and solid modeling. The faces usually consist of triangles (triangle mesh), quadrilaterals (quads), or other simple convex polygons (n-gons), since this simplifies rendering, but may also be more generally composed of concave polygons, or even polygons with holes.

With reference toFIG.4, objects400created with polygon meshes are represented by different types of elements. These include vertices402, edges404, faces406, polygons408and surfaces410as shown inFIG.4. Thus,FIG.4illustrates elements of a mesh.

Polygon meshes are defined by the following elements:

Vertex (402): a position in 3D space defined as (x,y,z) along with other information such as color (r,g,b), normal vector and texture coordinates.

Edge (404): a connection between two vertices.

Face (406): a closed set of edges404, in which a triangle face has three edges, and a quad face has four edges. A polygon408is a coplanar set of faces406. In systems that support multi-sided faces, polygons and faces are equivalent. Mathematically a polygonal mesh may be considered an unstructured grid, or undirected graph, with additional properties of geometry, shape and topology.

Surfaces (410): or smoothing groups, are useful, but not required to group smooth regions.

Groups: some mesh formats contain groups, which define separate elements of the mesh, and are useful for determining separate sub-objects for skeletal animation or separate actors for non-skeletal animation.

Materials: defined to allow different portions of the mesh to use different shaders when rendered.

UV coordinates: most mesh formats also support some form of UV coordinates which are a separate 2D representation of the mesh “unfolded” to show what portion of a 2-dimensional texture map to apply to different polygons of the mesh. It is also possible for meshes to contain other such vertex attribute information such as color, tangent vectors, weight maps to control animation, etc. (sometimes also called channels).

FIG.5andFIG.6show the extensions to the V-PCC encoder and decoder to support mesh encoding and mesh decoding, respectively, as proposed in MPEG input document [MPEG M47608].

In the encoder extension500, the input mesh data502is demultiplexed with demultiplexer504into vertex coordinates+attributes506and vertex connectivity508. The vertex coordinates+attributes data506is coded using MPEG-I V-PCC (such as with MPEG-I VPCC encoder510), whereas the vertex connectivity data508is coded (using vertex connectivity encoder516) as auxiliary data518. Both of these (encoded vertex coordinates and vertex attributes517and auxiliary data518) are multiplexed using multiplexer520to create the final compressed output bitstream522. Vertex ordering514is carried out on the reconstructed vertex coordinates512at the output of MPEG-I V-PCC510to reorder the vertices for optimal vertex connectivity encoding516.

Based on the examples described herein, as shown inFIG.5, the encoding process/apparatus500ofFIG.5may be extended such that the encoding process/apparatus500signals patch mesh signaling530(e.g. V3C patch mesh signaling) within the output bitstream522. Alternatively, patch mesh signaling530may be provided and signaled separately from the output bitstream522.

As shown inFIG.6, in the decoder600, the input bitstream602is demultiplexed with demultiplexer604to generate the compressed bitstreams for vertex coordinates+attributes605and vertex connectivity606. The input/compressed bitstream602may comprise or may be the output from the encoder500, namely the output bitstream522ofFIG.5. The vertex coordinates+attributes data605is decompressed using MPEG-I V-PCC decoder608to generate vertex attributes612. Vertex ordering616is carried out on the reconstructed vertex coordinates614at the output of MPEG-I V-PCC decoder608to match the vertex order at the encoder500. The vertex connectivity data606is also decompressed using vertex connectivity decoder610to generate vertex connectivity information618, and everything (including vertex attributes612, the output of vertex reordering616, and vertex connectivity information618) is multiplexed with multiplexer620to generate the reconstructed mesh622.

Based on the examples described herein, as shown inFIG.6, the decoding process/apparatus600ofFIG.6may be extended such that the decoding process/apparatus600receives and decodes patch mesh signaling630(e.g. V3C patch mesh signaling), which may be part of the compressed bitstream602. The patch mesh signaling630ofFIG.6may comprise or correspond to the patch mesh signaling530ofFIG.5. Alternatively, patch mesh signaling630may be received and signaled separately from the compressed bitstream602or output bitstream522(e.g. signaled to the demultiplexer604separately from the compressed bitstream602).

Generic Mesh Compression

Mesh data may be compressed directly without projecting it into 2D-planes, like in V-PCC based mesh coding. In fact, the anchor for V-PCC mesh compression call for proposals (CfP) utilizes off-the shelf mesh compression technology, Draco (https://google.github.io/draco/), for compressing mesh data excluding textures. Draco is used to compress vertex positions in 3D, connectivity data (faces) as well as UV coordinates. Additional per-vertex attributes may be also compressed using Draco. The actual UV texture may be compressed using traditional video compression technologies, such as H.265 or H.264.

Draco uses the edgebreaker algorithm at its core to compress 3D mesh information. Draco offers a good balance between simplicity and efficiency, and is part of Khronos endorsed extensions for the glTF specification. The main idea of the algorithm is to traverse mesh triangles in a deterministic way so that each new triangle is encoded next to an already encoded triangle. This enables prediction of vertex specific information from the previously encoded data by simply adding delta to the previous data. Edgebreaker utilizes symbols to signal how each new triangle is connected to the previously encoded part of the mesh. Connecting triangles in such a way results on average in 1 to 2 bits per triangle when combined with existing binary encoding techniques.

MPEG 3DG (ISO/IEC SC29 WG7) has issued a call for proposals (CfP) on the integration of MESH compression into the V3C family of standards (ISO/IEC 23090-5). During the work on the CfP response of Applicant of the instant disclosure, the Applicant has identified that transmitting the occupancy map in a lossless mode requires large bitrates. Switching to lossy mode does not bring direct advantages, because of the binary nature of the occupancy data, which is not well compressed with DCT-based video codecs.

Further distortion (quantization) of the occupancy map using a 2D video encoder resulted in strong non-linear artifacts in 3D objects (holes, broken surfaces, false faces, spikes) in Applicant's experiments. Efficient compression of occupancy information thus remains unsolved.

Some approaches, such as those that implement lossy compression of point cloud occupancy maps, describe smoothing without adaptivity to smooth out small details and merge some parts of a 3D mesh (or point cloud). Described herein is an approach capable of maintaining such details.

Described herein are methods for improving quality and coding efficiency of occupancy information for 3D meshes in a V3C coding framework. This is achieved by adaptive filtering of the occupancy map to prepare it for lossy video compression.

The main encoder embodiment includes coding efficiency improvement of 3D dynamic meshes by using adaptive filtering of an occupancy map. The main signaling embodiment includes i) per-sequence signaling of occupancy filtering and a base threshold; ii) per-patch delta threshold information (optional threshold+−8); and iii) occupancy map and geometry packing information. The main decoder embodiment includes receiving occupancy filter thresholding information to reverse adaptive filtering during 3D reconstruction.

1. Detailed Problem Description

In V3C coding of 3D data individual frames of dynamic 3D meshes are represented as geometry (GEO) and texture (TEX) patches. For efficient compression by a video encoder, GEO and TEX atlases of patches are padded to generate smooth transition areas between patches and eliminate strong patch borders. This improves compression efficiency dramatically, but also implies the use of an occupancy map to indicate which pixels of patches are occupied (have real values) and should be used to reconstruct the 3D mesh. Therefore as part of the geometry information, the occupancy map is created, coded losslessly and transmitted.

It can be said that the information of the actual patch border was moved from GEO and TEX components into the occupancy map. The occupancy map still requires a lot of bits for transmitting, especially as the occupancy map is typically encoded losslessly.

2. Herein Described Solution

The general idea of the examples described herein is to compress efficiently the occupancy map by switching to lossy mode and applying an adaptive smoothing filter algorithm before passing the occupancy map to a video encoder. The herein described method reduces the bitrate required for occupancy map transmission in lossy mode and improves objective and subjective quality of the reconstructed dynamic 3D mesh.

Advantages and technical effects of this method include bitrate reduction because of smoother frames, better inter-prediction of occupancy map atlases, and accurate reconstruction because of the adaptive nature of the filtering.

Encoder Embodiments

In an embodiment, the encoder performs the adaptive filtering of the occupancy map in such a way that the decoder can decode the occupancy map correctly using simple thresholding on occupancy values. This threshold is transmitted in the V3C bitstream. In the encoder, an analysis of the occupancy map is performed using a multiscale filtering approach.

In one embodiment, several smoothing levels of the occupancy map are generated by a Gaussian pyramid. Other multiscale filtering approaches can be used, and are not restricted to, a discrete wavelet transform (DWT) analysis or a multi-resolution set of bank filters. Although there is a need for adaptivity, the goal of the filtering is to smooth occupancy edges to reduce bitrate cost in the video encoding step. Hence adaptive filters that preserve edges should be discarded.

Once a set of smoothing levels has been generated, an analysis is performed at the pixel level. For every pixel of the original occupancy map, the smoothing level which gives the possibility to reconstruct the original occupancy value by simple thresholding is selected. If such level is not found, the pixel of the occupancy map is not filtered and left unchanged.

FIG.7Ashows an original occupancy map710, andFIG.7Bshows an adaptively filtered occupancy map720that has been adaptively filtered based on the methods described herein.

In one embodiment, the smoothing level threshold is selected adaptively based on bitdepth requirements. Also, the threshold could be fine-tuned iteratively after per-pixel adaptive filtering. Tuning the threshold is important, because a more accurate threshold generates better occupancy map quality.

2.2 Signaling on the V3C Level

In one embodiment to differentiate the subbitstreams containing video content in a different domain new signaling information is added to the V3C extension that indicates the type and/or format of a content of a given atlas. An excerpt of a VPS syntax table with the new extension is provided below and inFIG.8(refer to items802and804).

vps_occupancy_filter_present_flag (item802) equal to 1 specifies that the vps_occupancy_filter_extension( ) syntax structure is present in the v3c_parameter_set( ) syntax structure. vps_occupancy_filter_present_flag (item802) equal to 0 specifies that this syntax structure is not present. When not present, the value of vps_occupancy_filter_present_flag (item802) is inferred to be equal to 0.

FIG.9(item902) and below shows signaling that indicates a base threshold used to reconstruct an occupancy map in an atlas having an ID.

vti_occupancy_filter_threshold[atlasID] indicates a threshold value which should be used to reconstruct an occupancy map in an atlas with atlas ID equal atlasID. For example, pixels in occupancy with values below vti_occupancy_filter_threshold are defined to indicate unoccupied pixels, and pixels with values higher than or equal to vti_occupancy_filter_threshold indicate occupied values.

In another embodiment the occupancy filter threshold may be signaled as part of a common atlas sequence parameter set, an atlas sequence parameter set, a common atlas frame parameter set, an atlas frame parameter set, or as an SEI message.

Furthermore an additional level of flexibility may be added by enabling signaling of the occupancy filter threshold per patch in a patch data unit.

Decoder Embodiments

In an embodiment, the decoder receives a V3C bitstream containing occupancy filter threshold information to reverse the adaptive filtering during the 3D reconstruction. The smoothed occupancy map could be directly translated into a binary occupancy map by applying the specified threshold. Alternatively, probability-based restoration could be used. In 3D mesh encoding, the occupancy map contains mostly straight lines of face edges. So, a high probability of line pattern should be used as criteria for proper thresholding.

V3C bitstreams with this feature have additional signaling values. Once the signaling is enabled, the special form of the occupancy map may be like that depicted inFIG.7B.

The idea described herein, or part of the idea described herein, is to be part of Applicant's response to the mesh coding CfP (where Applicant is the Applicant of the herein described disclosure) and is to be contributed to standardization in SC29/WG7.

FIG.10is an apparatus1000which may be implemented in hardware, configured to implement adaptive filtering of an occupancy map for dynamic mesh compression, based on any of the examples described herein. The apparatus comprises a processor1002, at least one memory1004(memory1004may be non-transitory, transitory, non-volatile or volatile) including computer program code1005, wherein the at least one memory1004and the computer program code1005are configured to, with the at least one processor1002, cause the apparatus to implement circuitry, a process, component, module, function, coding, and/or decoding (collectively1006) to implement adaptive filtering of an occupancy map for dynamic mesh compression, based on the examples described herein. The apparatus1000is further configured to provide or receive signaling1007, based on the signaling embodiments described herein. The apparatus1000optionally includes a display and/or I/O interface1008that may be used to display an output (e.g., an image or volumetric video) of a result of coding/decoding1006. The display and/or I/O interface1008may also be configured to receive input such as user input (e.g. with a keypad, touchscreen, touch area, microphone, biometric recognition etc.). The apparatus1000also includes one or more communication interfaces (I/F(s))1010, such as a network (NW) interface. The communication I/F(s)1010may be wired and/or wireless and communicate over a channel or the Internet/other network(s) via any communication technique. The communication I/F(s)1010may comprise one or more transmitters and one or more receivers. The communication I/F(s)1010may comprise standard well-known components such as an amplifier, filter, frequency-converter, (de)modulator, and encoder/decoder circuitry(ies) and one or more antennas. In some examples, the processor1002is configured to implement item1006and/or item1007without use of memory1004.

The apparatus1000may be a remote, virtual or cloud apparatus. The apparatus1000may be either a writer or a reader (e.g. parser), or both a writer and a reader (e.g. parser). The apparatus1000may be either a coder or a decoder, or both a coder and a decoder (codec). The apparatus1000may be a user equipment (UE), a head mounted display (HMD), or any other fixed or mobile device.

The memory1004may be implemented using any suitable data storage technology, such as semiconductor based memory devices, flash memory, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The memory1004may comprise a database for storing data. Interface1012enables data communication between the various items of apparatus1000, as shown inFIG.10. Interface1012may be one or more buses, or interface1012may be one or more software interfaces configured to pass data within computer program code1005. For example, the interface1012may be one or more buses such as address, data, or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics or other optical communication equipment, and the like. In another example, interface1012is an object-oriented software interface. The apparatus1000need not comprise each of the features mentioned, or may comprise other features as well. The apparatus1000may be an embodiment of and have the features of any of the apparatuses shown inFIG.1A,FIG.1B,FIG.5, and/orFIG.6.

FIG.11is a method1100to implement adaptive filtering of an occupancy map for dynamic mesh compression, based on the examples described herein. At1110, the method includes receiving as input a three-dimensional mesh represented as at least one geometry patch of an atlas and at least one texture patch of the atlas. At1120, the method includes creating an occupancy map that indicates which pixels of the at least one geometry patch and the at least one texture patch are occupied having a valid value. At1130, the method includes wherein the occupancy map is configured to be used to reconstruct the mesh. At1140, the method includes entering lossy mode. At1150, the method includes applying, while in lossy mode, an adaptive smoothing filter algorithm to the occupancy map to discard at least one edge of the occupancy map, and to reduce a bitrate for transmission of the occupancy map. At1160, the method includes storing, while in lossy mode, the adaptively smoothed occupancy map and at least one occupancy filter threshold. At1170, the method includes encoding the adaptively smoothed occupancy map and the at least one occupancy filter threshold into a bitstream. Method1100may be performed with apparatus500or apparatus1000.

FIG.12is a method1200to implement adaptive filtering of an occupancy map for dynamic mesh compression, based on the examples described herein. At1210, the method includes signaling information within a visual volumetric video-based coding extension to indicate a type or format of a content of an atlas, the atlas comprising at least one geometry patch and at least one texture patch used to represent a three-dimensional mesh. At1220, the method includes wherein the information within the visual volumetric video-based coding extension indicates a bitstream including video content in a domain comprising an adaptively smoothed occupancy map configured to be used to reconstruct the three-dimensional mesh. At1230, the method includes signaling a visual volumetric video-based coding occupancy filter present flag associated with at least one occupancy filter threshold configured to be used to reconstruct the three-dimensional mesh. At1240, the method includes signaling an occupancy filter threshold syntax element within the visual volumetric video-based coding parameter set extension. Method1200may be performed with apparatus500or apparatus1000.

FIG.13is a method1300to implement adaptive filtering of an occupancy map for dynamic mesh compression, based on the examples described herein. At1310, the method includes decoding, from or along a bitstream, an adaptively smoothed occupancy map that indicates which pixels of at least one geometry patch and at least one texture patch are occupied having a valid value. At1320, the method includes wherein the at least one geometry patch and the at least one texture patch represent an encoded three-dimensional mesh. At1330, the method includes decoding at least one occupancy filter threshold from or along the bitstream. At1340, the method includes reconstructing the three-dimensional mesh using the adaptively smoothed occupancy map and the at least one occupancy filter threshold. Method1300may be performed with apparatus600or apparatus1000.

The following examples 1-29 are described herein.

Example 1: An apparatus includes at least one processor; and at least one memory including computer program code; wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to: receive as input a three-dimensional mesh represented as at least one geometry patch of an atlas and at least one texture patch of the atlas; create an occupancy map that indicates which pixels of the at least one geometry patch and the at least one texture patch are occupied having a valid value; wherein the occupancy map is configured to be used to reconstruct the mesh; enter lossy mode; apply, while in lossy mode, an adaptive smoothing filter algorithm to the occupancy map to discard at least one edge of the occupancy map, and to reduce a bitrate for transmission of the occupancy map; store, while in lossy mode, the adaptively smoothed occupancy map and at least one occupancy filter threshold; and encode the adaptively smoothed occupancy map and the at least one occupancy filter threshold into a bitstream.

Example 2: The apparatus of example 1, wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus at least to: generate several smoothing levels of the occupancy map using a multiscale filtering approach.

Example 3: The apparatus of example 2, wherein the multiscale filtering approach comprises: a Gaussian pyramid; a discrete wavelet transform; or a multi-resolution set of bank filters.

Example 4: The apparatus of any of examples 2 to 3, wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus at least to: determine, for a pixel of the occupancy map, one of the smoothing levels that allows reconstruction of an original value of the occupancy map; determine to filter the pixel, in response to determining one of the smoothing levels that allows reconstruction of the original value of the occupancy map; and determine not filter the pixel, in response to not being able to determine one of the smoothing levels that allows reconstruction of the original value of the occupancy map.

Example 5: The apparatus of any of examples 1 to 4, wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus at least to: determine the at least one occupancy filter threshold adaptively based on at least one bitdepth parameter.

Example 6: The apparatus of any of examples 1 to 5, wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus at least to: signal information within a visual volumetric video-based coding extension to indicate a type or format of a content of the atlas, and further to indicate that the bitstream includes video content in a domain comprising the adaptively smoothed occupancy map.

Example 7: The apparatus of any of examples 1 to 6, wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus at least to: signal a visual volumetric video-based coding occupancy filter present flag.

Example 8: The apparatus of example 7, wherein: the visual volumetric video-based coding occupancy filter present flag having a value of one specifies that a visual volumetric video-based coding occupancy filter extension syntax structure is present within a visual volumetric video-based coding parameter set syntax structure; and the visual volumetric video-based coding occupancy filter present flag having a value of zero specifies that the visual volumetric video-based coding occupancy filter extension syntax structure is not present within the visual volumetric video-based coding parameter set syntax structure.

Example 9: The apparatus of any of examples 1 to 8, wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus at least to: signal an occupancy filter threshold syntax element within a visual volumetric video-based coding parameter set extension; wherein the occupancy filter threshold syntax element indicates the at least one occupancy filter threshold configured to be used to reconstruct the occupancy map within an atlas with a given atlas identifier.

Example 10: The apparatus of any of examples 1 to 9, wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus at least to: signal the at least one occupancy filter threshold as part of: a common atlas sequence parameter set; an atlas sequence parameter set; a common atlas frame parameter set; an atlas frame parameter set; or a supplemental enhancement information message.

Example 11: The apparatus of any of examples 1 to 10, wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus at least to: signal the at least one occupancy filter threshold per patch in a patch data unit.

Example 12: An apparatus includes at least one processor; and at least one memory including computer program code; wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to: signal information within a visual volumetric video-based coding extension to indicate a type or format of a content of an atlas, the atlas comprising at least one geometry patch and at least one texture patch used to represent a three-dimensional mesh; wherein the information within the visual volumetric video-based coding extension indicates a bitstream including video content in a domain comprising an adaptively smoothed occupancy map configured to be used to reconstruct the three-dimensional mesh; signal a visual volumetric video-based coding occupancy filter present flag associated with at least one occupancy filter threshold configured to be used to reconstruct the three-dimensional mesh; and signal an occupancy filter threshold syntax element within the visual volumetric video-based coding parameter set extension.

Example 13: The apparatus of example 12, wherein: the visual volumetric video-based coding occupancy filter present flag having a value of one specifies that a visual volumetric video-based coding occupancy filter extension syntax structure is present within a visual volumetric video-based coding parameter set syntax structure; and the visual volumetric video-based coding occupancy filter present flag having a value of zero specifies that the visual volumetric video-based coding occupancy filter extension syntax structure is not present within the visual volumetric video-based coding parameter set syntax structure.

Example 14: The apparatus of any of examples 12 to 13, wherein the occupancy filter threshold syntax element indicates the at least one occupancy filter threshold configured to be used to reconstruct the occupancy map within an atlas with a given atlas identifier.

Example 15: The apparatus of any of examples 12 to 14, wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus at least to: signal the at least one occupancy filter threshold as part of: a common atlas sequence parameter set; an atlas sequence parameter set; a common atlas frame parameter set; an atlas frame parameter set; or a supplemental enhancement information message.

Example 16: The apparatus of any of examples 12 to 15, wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus at least to: signal the at least one occupancy filter threshold per patch in a patch data unit.

Example 17: An apparatus comprising: at least one processor; and at least one memory including computer program code; wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to: decode, from or along a bitstream, an adaptively smoothed occupancy map that indicates which pixels of at least one geometry patch and at least one texture patch are occupied having a valid value; wherein the at least one geometry patch and the at least one texture patch represent an encoded three-dimensional mesh; decode at least one occupancy filter threshold from or along the bitstream; and reconstruct the three-dimensional mesh using the adaptively smoothed occupancy map and the at least one occupancy filter threshold.

Example 18: The apparatus of example 17, wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus at least to: translate the adaptively smoothed occupancy map into a binary occupancy map, using the at least one occupancy filter threshold.

Example 19: The apparatus of any of examples 17 to 18, wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus at least to: reconstruct the three-dimensional using probability based restoration; wherein the at least one occupancy filter threshold comprises at least one probability of a line pattern of the occupancy map.

Example 20: The apparatus of any of examples 17 to 19, wherein the adaptively smoothed occupancy map has been smoothed in lossy mode using an adaptive smoothing filter algorithm to discard at least one edge of the occupancy map, and to reduce a bitrate for transmission of the occupancy map.

Example 21: A method includes receiving as input a three-dimensional mesh represented as at least one geometry patch of an atlas and at least one texture patch of the atlas; creating an occupancy map that indicates which pixels of the at least one geometry patch and the at least one texture patch are occupied having a valid value; wherein the occupancy map is configured to be used to reconstruct the mesh; entering lossy mode; applying, while in lossy mode, an adaptive smoothing filter algorithm to the occupancy map to discard at least one edge of the occupancy map, and to reduce a bitrate for transmission of the occupancy map; storing, while in lossy mode, the adaptively smoothed occupancy map and at least one occupancy filter threshold; and encoding the adaptively smoothed occupancy map and the at least one occupancy filter threshold into a bitstream.

Example 22: A method including signaling information within a visual volumetric video-based coding extension to indicate a type or format of a content of an atlas, the atlas comprising at least one geometry patch and at least one texture patch used to represent a three-dimensional mesh; wherein the information within the visual volumetric video-based coding extension indicates a bitstream including video content in a domain comprising an adaptively smoothed occupancy map configured to be used to reconstruct the three-dimensional mesh; signaling a visual volumetric video-based coding occupancy filter present flag associated with at least one occupancy filter threshold configured to be used to reconstruct the three-dimensional mesh; and signaling an occupancy filter threshold syntax element within the visual volumetric video-based coding parameter set extension.

Example 23: A method includes decoding, from or along a bitstream, an adaptively smoothed occupancy map that indicates which pixels of at least one geometry patch and at least one texture patch are occupied having a valid value; wherein the at least one geometry patch and the at least one texture patch represent an encoded three-dimensional mesh; decoding at least one occupancy filter threshold from or along the bitstream; and reconstructing the three-dimensional mesh using the adaptively smoothed occupancy map and the at least one occupancy filter threshold.

Example 24: An apparatus includes means for receiving as input a three-dimensional mesh represented as at least one geometry patch of an atlas and at least one texture patch of the atlas; means for creating an occupancy map that indicates which pixels of the at least one geometry patch and the at least one texture patch are occupied having a valid value; wherein the occupancy map is configured to be used to reconstruct the mesh; means for entering lossy mode; means for applying, while in lossy mode, an adaptive smoothing filter algorithm to the occupancy map to discard at least one edge of the occupancy map, and to reduce a bitrate for transmission of the occupancy map; means for storing, while in lossy mode, the adaptively smoothed occupancy map and at least one occupancy filter threshold; and means for encoding the adaptively smoothed occupancy map and the at least one occupancy filter threshold into a bitstream.

Example 25: An apparatus includes means for signaling information within a visual volumetric video-based coding extension to indicate a type or format of a content of an atlas, the atlas comprising at least one geometry patch and at least one texture patch used to represent a three-dimensional mesh; wherein the information within the visual volumetric video-based coding extension indicates a bitstream including video content in a domain comprising an adaptively smoothed occupancy map configured to be used to reconstruct the three-dimensional mesh; means for signaling a visual volumetric video-based coding occupancy filter present flag associated with at least one occupancy filter threshold configured to be used to reconstruct the three-dimensional mesh; and means for signaling an occupancy filter threshold syntax element within the visual volumetric video-based coding parameter set extension.

Example 26: An apparatus includes means for decoding, from or along a bitstream, an adaptively smoothed occupancy map that indicates which pixels of at least one geometry patch and at least one texture patch are occupied having a valid value; wherein the at least one geometry patch and the at least one texture patch represent an encoded three-dimensional mesh; means for decoding at least one occupancy filter threshold from or along the bitstream; and means for reconstructing the three-dimensional mesh using the adaptively smoothed occupancy map and the at least one occupancy filter threshold.

Example 27: A non-transitory program storage device readable by a machine, tangibly embodying a program of instructions executable with the machine for performing operations is described and provided, the operations comprising: receiving as input a three-dimensional mesh represented as at least one geometry patch of an atlas and at least one texture patch of the atlas; creating an occupancy map that indicates which pixels of the at least one geometry patch and the at least one texture patch are occupied having a valid value; wherein the occupancy map is configured to be used to reconstruct the mesh; entering lossy mode; applying, while in lossy mode, an adaptive smoothing filter algorithm to the occupancy map to discard at least one edge of the occupancy map, and to reduce a bitrate for transmission of the occupancy map; storing, while in lossy mode, the adaptively smoothed occupancy map and at least one occupancy filter threshold; and encoding the adaptively smoothed occupancy map and the at least one occupancy filter threshold into a bitstream.

Example 28: A non-transitory program storage device readable by a machine, tangibly embodying a program of instructions executable with the machine for performing operations is described and provided, the operations comprising: signaling information within a visual volumetric video-based coding extension to indicate a type or format of a content of an atlas, the atlas comprising at least one geometry patch and at least one texture patch used to represent a three-dimensional mesh; wherein the information within the visual volumetric video-based coding extension indicates a bitstream including video content in a domain comprising an adaptively smoothed occupancy map configured to be used to reconstruct the three-dimensional mesh; signaling a visual volumetric video-based coding occupancy filter present flag associated with at least one occupancy filter threshold configured to be used to reconstruct the three-dimensional mesh; and signaling an occupancy filter threshold syntax element within the visual volumetric video-based coding parameter set extension.

Example 29: A non-transitory program storage device readable by a machine, tangibly embodying a program of instructions executable with the machine for performing operations is described and provided, the operations comprising: decoding, from or along a bitstream, an adaptively smoothed occupancy map that indicates which pixels of at least one geometry patch and at least one texture patch are occupied having a valid value; wherein the at least one geometry patch and the at least one texture patch represent an encoded three-dimensional mesh; decoding at least one occupancy filter threshold from or along the bitstream; and reconstructing the three-dimensional mesh using the adaptively smoothed occupancy map and the at least one occupancy filter threshold.

In the figures, arrows between individual blocks represent operational couplings there-between as well as the direction of data flows on those couplings.

The following acronyms and abbreviations that may be found in the specification and/or the drawing figures are defined as follows:2D or 2d two-dimensional3D or 3d three-dimensional3DG 3D graphics coding group6DOF six degrees of freedomACL atlas coding layerAR augmented realityASIC application-specific integrated circuitasps atlas sequence parameter setCD committee draftCfP call for proposal(s)CGI computer-generated imageryDCT discrete cosine transformDWT discrete wavelet transformFPGA field programmable gate arrayGEO geometry data of meshglTF graphics language transmission formatH.264 advanced video coding video compression standardH.265 high efficiency video coding video compression standardHMD head mounted displayid or ID identifierIdx indexIEC International Electrotechnical CommissionI/F interfaceI/O input/outputISO International Organization for Standardizationmiv or MIV MPEG immersive videoMPEG moving picture experts groupMPEG-I MPEG immersiveMR mixed realitynal or NAL network abstraction layerNW networkpdu patch data unitRBSP raw byte sequence payloadSC subcommitteeSEI supplemental enhancement informationTEX texture data of meshu(n) unsigned integer using n bits, e.g. u(1), u(2)UE user equipmentue(v) unsigned integer exponential Golomb coded syntax element with the left bit firstUV coordinate texture, where “U” and “V” are axes of a 2D textureV3C visual volumetric video-based codingVPCC or V-PCC video-based point cloud coding/compressionvps or VPS V3C parameter setVR virtual realityWG working group