Patent ID: 12256098

This specification includes references to “one embodiment” or “an embodiment.” The appearances of the phrases “in one embodiment” or “in an embodiment” do not necessarily refer to the same embodiment. Particular features, structures, or characteristics may be combined in any suitable manner consistent with this disclosure.

“Comprising.” This term is open-ended. As used in the appended claims, this term does not foreclose additional structure or steps. Consider a claim that recites: “An apparatus comprising one or more processor units . . . ” Such a claim does not foreclose the apparatus from including additional components (e.g., a network interface unit, graphics circuitry, etc.).

“Configured To.” Various units, circuits, or other components may be described or claimed as “configured to” perform a task or tasks. In such contexts, “configured to” is used to connote structure by indicating that the units/circuits/components include structure (e.g., circuitry) that performs those task or tasks during operation. As such, the unit/circuit/component can be said to be configured to perform the task even when the specified unit/circuit/component is not currently operational (e.g., is not on). The units/circuits/components used with the “configured to” language include hardware—for example, circuits, memory storing program instructions executable to implement the operation, etc. Reciting that a unit/circuit/component is “configured to” perform one or more tasks is expressly intended not to invoke 35 U.S.C. § 112(f), for that unit/circuit/component. Additionally, “configured to” can include generic structure (e.g., generic circuitry) that is manipulated by software and/or firmware (e.g., an FPGA or a general-purpose processor executing software) to operate in manner that is capable of performing the task(s) at issue. “Configure to” may also include adapting a manufacturing process (e.g., a semiconductor fabrication facility) to fabricate devices (e.g., integrated circuits) that are adapted to implement or perform one or more tasks.

“First,” “Second,” etc. As used herein, these terms are used as labels for nouns that they precede, and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.). For example, a buffer circuit may be described herein as performing write operations for “first” and “second” values. The terms “first” and “second” do not necessarily imply that the first value must be written before the second value.

“Based On.” As used herein, this term is used to describe one or more factors that affect a determination. This term does not foreclose additional factors that may affect a determination. That is, a determination may be solely based on those factors or based, at least in part, on those factors. Consider the phrase “determine A based on B.” While in this case, B is a factor that affects the determination of A, such a phrase does not foreclose the determination of A from also being based on C. In other instances, A may be determined based solely on B.

DETAILED DESCRIPTION

As data acquisition and display technologies have become more advanced, the ability to capture three-dimensional (3D) volumetric content has increased. Also, the development of advanced display technologies, such as virtual reality, augmented reality, cross reality, etc. has increased potential uses for 3D volumetric content. However, 3D volumetric content files are often very large and may be costly and time-consuming to store and transmit. Also, such large files may be computationally intensive to render at display devices. For example, such 3D volumetric content may require generating and rendering a large number of vertices which may overwhelm computational capabilities of a given rendering device and/or may slow down the rendering process.

In some embodiments, an encoder may be used to generate a compressed version of the 3D volumetric content to reduce costs and time associated with storing and transmitting large 3D volumetric content files. In some embodiments, a system may include an encoder that compresses attribute and/or spatial information of a volumetric point cloud or immersive video content file such that the file may be stored and transmitted more quickly than non-compressed volumetric content and in a manner such that the compressed volumetric content file may occupy less storage space than non-compressed volumetric content. In some embodiments, such compression may enable 3D volumetric content to be communicated over a network in real-time or in near real-time, or on-demand in response to demand from a consumer of the 3D volumetric content.

In some embodiments, a system may include a decoder that receives encoded 3D volumetric content comprising video encoded attribute information and video encoded geometry information via a network from a remote server or other storage device that stores or generates the volumetric content files. For example, a 3-D display, a holographic display, or a head-mounted display may be manipulated in real-time or near real-time to show different portions of a virtual world represented by 3D volumetric content. In order to update the 3-D display, the holographic display, or the head-mounted display, a system associated with the decoder may request data from the remote server based on user manipulations (or anticipated user manipulations) of the displays, and the data may be transmitted from the remote server to the decoder in a form of encoded 3D volumetric content (e.g. video encoded attribute patch images and video encoded depth patch images/depth maps). The displays may then be updated with updated data responsive to the user manipulations, such as updated views.

However, instead of rendering a mesh representing the 3D object or scene that includes a vertex for each pixel included in the depth patch images/depth maps, a decoding computing device (e.g. decoder) performs real-time mesh simplification. For example, the decoder models a set of pixels corresponding to a block of an image frame comprising the depth patch images/depth map as a sub-mesh comprising a set of mesh surfaces such as triangles that are oriented in a way that approximates the depth values of the set of pixels of the depth map for that block, but without generating a vertex for each pixel. For example, for a four pixel by four pixel block (which includes 16 pixels total), instead of rendering a mesh comprising 16 vertices-one for each pixel in the depth map—the decoder may instead render a sub-mesh comprising two triangles and four vertices total, wherein the surfaces of the triangles approximate the depth values of the internal pixels of the block. Furthermore, the decoder may compare the depth values approximated by the sub-mesh to the actual depth values signaled in the depth map and, if it is determined that the error is greater than a threshold amount, the decoder may further sub-divide the block into sub-blocks. The decoder may then carry out a similar process for the sub-blocks to determine if sub-meshes approximating the sub-blocks result in error less than the threshold, or whether the sub-blocks should be further sub-divided. As can be seen such a simplification process may considerably reduce a number of vertices to be rendered at the decoding device as compared to rendering a mesh with a same number of vertices as pixels signaled in the depth map.

Also, in some embodiments, the threshold error value may be dynamically determined. For example, based on available computational capacity of the decoding device to render vertices, the decoding device may dynamically adjust the acceptable error threshold up or down. Also, in some embodiments, metadata may be signaled to the decoder with the encoded video image frames comprising the packed attribute patch images and the packed depth patch images. For example, the metadata may identify important portions of the 3D volumetric content, such as depth discontinuities or a region of interest, such as a person's face. Additionally, the metadata may define alternative error thresholds to be used for these portions of the 3D volumetric content. For example, less error in depths may be tolerated for portions of a rendered mesh representing the person's face than may be tolerated on a portion of the rendered mesh representing the person's arm or leg, as an example.

Also, in some embodiments, the error threshold may be dynamically determined for a given block, set of blocks based on a gaze or viewing direction of a user device that is to render the 3D volumetric content. For example, portions of the mesh in a direct line of sight of the gaze of the user device may be assigned a dynamically determined error threshold that tolerates less depth error in the region directly in the line of sight than may be acceptable for other regions of the mesh that are only in a periphery of the line of sight. Moreover, as the user device is manipulated to change its gaze direction, the error threshold may be dynamically updated based on the updated gaze direction. For example, in some embodiments, the user device rendering the 3D volumetric content may include an inertial measurement unit or other type of sensor that is configured to determine an orientation of the device in relation to viewing the mesh.

In some embodiments, as part of generating the 3D volumetric content sensors may capture attribute information for one or more points, such as color attributes, texture attributes, reflectivity attributes, velocity attributes, acceleration attributes, time attributes, modalities, and/or various other attributes. For example, in some embodiments, an immersive video capture system, such as that may follow MPEG immersive video (MIV) standards, may use a plurality of cameras to capture images of a scene or object from a plurality of viewing angles and/or locations and may further use these captured images to determine spatial information for points or surfaces of the object or scene, wherein the spatial information and attribute information is encoded using video-encoded attribute image patches and video encoded depth patch images/depth maps as described herein.

Generating 3D Volumetric Content

In some embodiments, 3D volumetric content that is to be encoded/compressed and decoded/decompressed, as described herein, may be generated from a plurality of images of an object or scene representing multiple views of the object or scene, wherein additional metadata is known about the placement and orientation of the cameras that captured the multiple views.

For example,FIG.1Aillustrates an object (person102) for which multiple images are being captured representing multiple views of the object, when viewed from cameras located at different locations and viewing angles relative to the object.

InFIG.1Acameras104,106,108,110, and112view person102from different camera locations and/or viewing angles. For example, camera112captures a front center (FC) view of person102, camera108captures a left side (LS) view of person102, camera110captures a right side (RS) view of person102, camera104captures a front left (FL) view of person102, and camera106captures a front right (FR) view of person102.

FIG.1Billustrates additional cameras that may be located behind person102. For example, camera118captures a back center (BC) view of person102, camera114captures a back left (BL) view of person102, camera116captures a back right (BR) view of person102, etc.

FIG.1Cis a top view illustrating the cameras shown inFIGS.1A and1Bthat are located at different locations and viewing angles relative to person102. Note that the camera positions and camera angles shown inFIGS.1A-1Care given as an example configuration and in some embodiments other camera configurations may be used. For example, in some embodiments, when capturing images for a scene, the cameras may face outward towards the scene as opposed to pointing inward towards an object, as shown inFIG.1C. Also, in some embodiments, the cameras may not necessarily be arranged in a circular configuration, but may instead be arranged in other configurations, such as a square, rectangle, grid pattern, etc.

FIG.1Dillustrates images that may have been captured via cameras104-118as shown inFIGS.1C-1D. For example image120shows a front center (FC) view, image122shows a back center (BC) view, image124shows a left side (LS) view, image126shows a right side (RS) view, image128shows a front right (FR) view, image130shows a front left (FL) view, image134shows a back right (BR) view, and image134shows a back left (BL) view.

In some embodiments, metadata is associated with each of the views as shown inFIG.1D, wherein the metadata (e.g. source camera parameters) indicate locations and camera angles for the respective cameras104-118that were used to capture images120-134. In some embodiments, this metadata may be used to determine geometry information for the object or scene that is being captured by the respective cameras, such as X, Y, and Z coordinates of points of the object or scene (or other types of spatial information).

For example, a component of an encoder, such as an atlas constructor410(as shown inFIG.4) may use source camera parameters (e.g. metadata indicating source camera parameters402, such as camera location and orientation) along with the images captured from the cameras to determine distances to surfaces in the captured images from the cameras at the known locations with the known orientations. In turn, spatial information indicating locations in space for the surfaces may be determined using the determined distances from the cameras and the known locations and orientations of the cameras.

For example, as shown inFIG.1E, source camera parameters402may indicate locations and orientations for right side camera110and front right camera106that both capture images of a portion of a shoulder of person102. Moreover, the atlas constructor410may determine that the cameras106and110are both capturing images comprising a same surface of the object (e.g. the portion of the person's shoulder). For example, pixel value patterns in the images may be matched to determine that images from both cameras106and110are capturing the same portion of the person102's shoulder. Using the source camera parameters402and knowing points in the captured images that are located at a same location in 3D space, the atlas constructor410may determine a location in 3D space of the matching portions of the captured images (e.g. the portion of person102's shoulder). Based on this determination using the known locations and orientations of cameras106and110, the atlas constructor510may determine geometry/spatial information for the portion of the object, such as X, Y, and Z coordinates for points included in the matching portion of the person102's shoulder.

Furthermore, the spatial/geometry information may be represented in the form of a depth map (also referred to herein as a depth patch image). For example, the spatial information for the person's shoulder, e.g. points with coordinates X1, Y1, Z1; X2, Y2, Z2; and X3, Y3, Z3, may be projected onto a flat plane of a depth map, wherein the X and Y spatial information is represented by a location of a given point in the depth map. For example, X values may be represented by locations of the points along a width of the depth map (e.g. the “U” direction) and Y values may be represented by locations of the points along the height of the depth map (e.g. the “V” direction). Moreover, the Z values of the points may be represented by pixel values (“pv”) associated with the points at locations (U, V). For example, a first point with coordinates in 3D space of X1, Y1, Z1may be represented in the depth map at pixel (U1, V1) which has pixel value pv1, wherein darker pixel values indicate lower Z values and lighter pixel values indicate greater Z values (or vice versa).

In some embodiments, depth maps may only be generated for views that are to be included in an atlas. For example, depth maps may not be generated for redundant views or redundant portions of views that are omitted from the atlas. Though, in some embodiments, image data and source camera parameters of all views may be used to generate the depth maps, but the redundant views may not be included in the generated depth maps. For example, whereas cameras106and110capture redundant information for the person102's shoulder, a single depth map may be generated for the two views as opposed to generating two redundant depth maps for the person's shoulder. However the images captured from cameras106and110that redundantly view the person's shoulder from different locations/camera viewing angles may be used to determine the spatial information to be included in the single depth map representing the person's shoulder.

Encoding 3D Volumetric Content

FIG.2illustrates a flowchart for an example process for generating an atlas from the captured views, wherein redundant information already included in a given view already included in the atlas is omitted from other views that are to be included in the atlas, according to some embodiments.

At block202, a view optimizer (such as view optimizer406of the encoder shown inFIG.4) receives source views comprising both attribute and depth information, such as source views comprising views120-134illustrated inFIG.1D. The view optimizer also selects one of the received views as a main view. In some embodiments, the view optimizer may also receive source camera parameters, such as source camera parameters402, which indicate locations and orientations of the cameras that captured the source views.

The view optimizer may select one or more main views and tag the selected views as main views. In order to determine a ranking (e.g. ordered list of the views) at block204the view optimizer then re-projects the selected one or more main views into remaining ones of the views that were not selected as main views. For example, the front center view (FC)120and the back center view (BC)122may be selected as main views and may be re-projected into the remaining views, such as views124-134. At block206, the view optimizer determines redundant pixels, e.g. pixels in the remaining views that match pixels of the main views that have been re-projected into the remaining views. For example, portions of front right view128are redundant with portions of front center view120, when pixels of front right view128are re-projected into front center view120. In the example, these redundant pixels are already included in the main view (e.g. view120from the front center (FC)) and are omitted from the remaining view (e.g. view128from the front right (FR)).

The view optimizer (e.g. view optimizer406) may iteratively repeat this process selecting a next remaining view as a “main view” for a subsequent iteration and repeat the process until no redundant pixels remain, or until a threshold number of iterations have been performed, or another threshold has been met, such as less than X redundant pixels, or less than Y total pixels, etc. For example, at block208the re-projection is performed using the selected remaining view as a “main view” to be re-projected into other ones of the remaining views that were not selected as “main views” for this iteration or a previous iteration. Also, at block212redundant pixels identified based on the re-projection performed at210are discarded. At block214the process (e.g. blocks208-212) are repeated until a threshold is met (e.g. all remaining views comprise only redundant pixels or have less than a threshold number of non-redundant pixels, etc.). The threshold may be measured also be based on all of the remaining views having empty pixels (e.g. they have already been discarded) or all of the remaining views have less than a threshold number of non-empty pixels.

The ordered list of views having non-redundant information may be provided from the view optimizer (e.g. view optimizer406) to an atlas constructor of an encoder (e.g. atlas constructor410as shown inFIG.4). Additionally, the source camera parameters402may be provided from the view optimizer406to the atlas constructor410.

The atlas constructor410may prune the empty pixels from the respective views (e.g. the pixels for which redundant pixel values were discarded by the view optimizer406). This may be referred to as “pruning” the views as shown being performed in atlas constructor410. The atlas constructor410may further aggregate the pruned views into patches (such as attribute patch images and geometry patch images) and pack the patch images into respective image frames.

For example,FIG.3illustrates an atlas comprising packed attribute patch images representing views included in the atlas, wherein redundant information has been omitted and also illustrates a corresponding atlas/depth map comprising depth patch images that correspond with the attribute patch images included in the adjacent attribute patch image atlas, according to some embodiments.

Attribute patch images304and306for main views120and122are shown packed in the atlas302. Also, patch images308and310comprising non-redundant pixels for views124and126are shown packed in atlas302. Additionally, attribute patch images312,314,316, and318comprising non-redundant pixels for remaining views128,130,132, and134are shown packed in atlas302.

Atlas320/depth map320comprises corresponding depth patch images322-336that correspond to the attribute patch images304-318packed into attribute atlas302.

FIG.4illustrates a block diagram for an encoder configured to encode three-dimensional (3D) volumetric content using video encoded attribute patch images and simplified mesh-based representations that are mesh encoded, according to some embodiments.

As discussed above, source camera parameters402indicating location and orientation information for the source cameras, such as cameras104-118as illustrated inFIGS.1A-1Care provided to the view optimizer406. Also source views404which, include both attributes (e.g. colors, textures, etc.) and depth information are provided to view optimizer406. The view optimizer406determines main views and remaining views as discussed in regard toFIG.2. The view optimizer406and/or the pruner of atlas410may further disregard redundant pixels as described inFIG.2. For example, the view optimizer may mark redundant pixels as empty and the pruner of atlas constructor410may prune the empty pixels. Note, the main views and remaining views along with camera lists comprising source camera parameter metadata comprising location and orientation information for the cameras that captured the main and remaining views are provided to atlas constructor410. As shown inFIG.4, the atlas constructor410prunes the views (main and remaining) to remove empty pixels. The atlas constructor410further aggregates the pruned views into patches and packs the patches into a 2D video image frame. For example, in atlas302redundant/empty pixels have been pruned from views128,130,132, and134. Also as shown in atlas302for views128,130,132, and134, the remaining (non-pruned) portions of these views have been aggregated into attribute patch images312,314,316, and318. These attribute patch images have further been packed into atlas302, which may have a same size/resolution as the video image frame comprising the attribute patch images (e.g. atlas302). It is worth pointing out that white space has been included in atlas302for ease of illustration. However, in at least some embodiments, the non-redundant portions of the views may be more closely packed into smaller patch images with less open space than what is shown inFIG.3.

Packed atlas302may be provided to encoder416which may video encode the attribute patch images and video encode the depth patch images.

Additionally, atlas constructor410generates an atlas parameters lists412, such as bounding box sizes and locations of the patch images in the packed atlas. The atlas constructor410also generates a camera parameters list408. For example, atlas constructor410may indicate in the atlas parameters list412that an attribute patch image (such as attribute patch image304) has a bounding box size of M×N and has coordinates with a bottom corner located at the bottom left of the atlas. Additionally, an index value may be associated with the patch image, such as that it is a 1st, 2ndetc. patch image in the index. Additionally, camera parameter list408may be organized by or include the index entries, such that camera parameter list includes an entry for index position1indicating that the camera associated with that entry is located at position X with orientation Y, such as camera112(the front center FC camera that captured view120that was packed into patch image304).

Metadata composer414may entropy encode the camera parameter list508and entropy encode the atlas parameter list412as entropy encoded metadata. The entropy encoded metadata may be included in a compressed bit stream long with video encoded packed image frames comprising attribute patch images that have been encoded via encoder416and along with video encoded depth patch images/depth map that have been video encoded via encoder416.

Decoding 3D Volumetric Content with Real-Time Mesh Simplification

FIG.5illustrates a block diagram for a decoder configured to use video encoded attribute patch images and video encoded depth map to generate a reconstructed version of encoded 3D volumetric content, according to some embodiments.

The compressed bit stream may be provided to a decoder, such as the decoder shown inFIG.5. The entropy encoded metadata may be directed to a metadata parser504and the video encoded image frames comprising attribute patch images packed in the image frames and also the depth patch images packed in the same image frame or an additional image frame may be provided to decoder502, which may video decode the attribute image frames and the depth image frames (e.g. depth maps). The decoded atlas (or set of complimentary decoded atlases) comprising attribute patch images and depth patch images may be provided to reference renderer508along with atlas patch occupancy maps that have been generated by atlas patch occupancy map generator506using the entropy decoded atlas parameter list. Also, the camera view metadata included in the entropy decoded metadata may be provided to reference renderer508. For example, camera parameter list metadata may be used by reference renderer508to select a given view of the 3D volumetric content to render based on a user manipulation of the viewport (e.g. viewing position and viewing orientation information received by the reference renderer508).

The reference renderer508includes a real-time mesh simplification module510configured to reduce an amount of vertices to be included in synthesized meshes generated to render the 3D volumetric content such that the synthesized meshes have fewer vertices than un-simplified meshes but also limit a degree to which the meshes are simplified such that distortion or errors resulting from the simplification of the meshes is less than a threshold level of acceptable error. In some embodiments, the threshold level of acceptable error may be dynamically determined.

For example,FIG.6illustrates a block diagram showing additional processes performed by a reference renderer, such as reference renderer508, to render three-dimensional (3D) volumetric content based on views, according to some embodiments.

The reference renderer (such as reference renderer508illustrated inFIG.5) receives viewing and position orientation information606from a rendering device that is to display the rendered 3D volumetric content. For example the rendering device may include an inertial measurement unit (IMU) as illustrated inFIG.15that includes gyroscopes that determine an orientation of the rendering device in relation to the 3D volumetric content to be rendered. Additionally, the reference renderer receives camera parameter list604, which is provided by metadata parser504(e.g. camera view metadata “D” as shown inFIG.5). Additionally, the reference renderer receives a video decoded atlas comprising attribute patch images (e.g. colors, textures, etc.) and also receives a video decoded complementary atlas comprising depth patch images (e.g. a depth map). Also, the reference renderer508receives occupancy maps608for the atlases. Note that in some embodiments the attribute atlas and the depth atlas/depth map may be video encoded in a same image frame or using separate image frames. These are shown inFIG.6as atlas or view representations602, which have been video decode by decoder502as shown inFIG.5. Based on the viewing position and orientation for the viewport indicated in viewing and position and orientation information606, a view selection component610of the reference renderer508selects particular ones of the views included in the atlases that are to be rendered.

Based on the selected views, occupancy map update pass1(614) identifies portions of the atlases (e.g. decoded video image frames comprising packed attribute patch images and packed depth patch images) that include patches comprising a main view to be rendered. The occupancy map update pass1(614) provides these patches, which may include view components packed into a bounding box of a patch for the main view, to the synthesizer616for pass1. For example, the occupancy map update module614may identify depth patch322as shown inFIG.3as a patch for a main view to be synthesized.

However, instead of generating a mesh vertex for each pixel shown in the main view patch, such as depth patch image322, the synthesizer may further perform a real-time mesh simplification (e.g. using real-time mesh simplification module510). The real-time mesh simplification process is further described in detail inFIGS.7-14. The result of the synthesizer pass1(616) is the generation of a simplified sub-mesh representing the portion of the 3D volumetric content shown in the main view. For example, the front center (FC) of the person as shown in view120illustrated inFIG.1Dthat was the basis for attribute patch image304and depth patch image322shown inFIG.3. Thus a mesh representing the front center view of the person is generated, wherein the generated mesh is simplified in the sense that it includes less vertices than would have been the case if a vertex was generated for each pixel value indicating a depth of the front center view of the person as included in depth patch image322.

Additionally, the reference renderer508projects the attribute values for the front center view of the person onto the generated sub-mesh for the front center view of the person. For example, attribute values indicated in attribute patch image304are projected onto the simplified mesh generated using the depth information included in depth patch image322. Thus, a synthesized sub-mesh representing the front center (FC) view of the person102is synthesized.

A similar process is carried out for other views. For example, occupancy map update pass2(618) may identify depth patch image328as corresponding to a next view to be synthesized, wherein depth patch image328corresponds to a remaining view representing a right side (RS) view of the person, e.g. view126as shown inFIG.1D. Synthesizer pass2(620) generates a simplified sub-mesh representing the portion of the 3D volumetric content shown in the remaining view and projects the attribute values from the corresponding attribute patch image310onto the generated simplified sub-mesh. For example, the right side (RS) of the person as shown in view126illustrated inFIG.1Dthat was the basis for attribute patch image310and depth patch image328shown inFIG.3is synthesized. Then, at622the reference renderer merges the two generated sub-meshes together.

This process is then repeated for each of the other views included in the views selected via view selection610to be rendered. For example the selected views may be selected based on a viewport view of the person to be rendered in the view port that views the person from the front and right side. Such that the front center (FC) view is visible, the right side (RS) view is visible, and the front right (FR) view is visible. Thus the next view to be synthesized may be the front right (FR) view. Thus, at occupancy map update pass N (624), the reference renderer may identify depth patch image312as corresponding to a next view to be synthesized, wherein depth patch image330corresponds to a remaining view representing a front right (FR) view of the person, e.g. view128as shown inFIG.1D. Synthesizer pass N (620) generates a simplified sub-mesh representing the portion of the 3D volumetric content shown in the remaining view. For example, the front right (FR) side of the person as shown in view128illustrated inFIG.1Dthat was the basis for attribute patch image312and depth patch image330shown inFIG.3is synthesized. Then, at622the reference renderer merges the generated sub-mesh with the other merged sub-meshes. Note, this process may be repeated for any number of views visible in a target view that is to be rendered.

Next the reference renderer508performs an in-painting process628to fill in any gaps in the merged sub-meshes by using a linear interpolation to fill the gaps based on values of the sub-meshes on sides surrounding the gap.

Finally, the target view (630) is rendered in the viewport. Note that as the viewport is manipulated, additional views may be rendered following the process as shown inFIG.6, wherein the meshes to be rendered are simplified in real-time as described inFIGS.7-14.

For example, in order to discuss the simplification of a mesh for a given view, take the front center view as an example. The depth map/depth patch image322provides a starting point for the example. In order to simplify the mesh, the bounding box of the depth map/depth patch image322may be divided into overlapping blocks.

For example,FIGS.7A-7Dillustrate an example of overlapping blocks being determined for a portion of an image frame comprising depth map information, according to some embodiments.

For example,FIG.7Aillustrates a first block (e.g. block1) being determined for a portion of an image frame702. For example, portion702may correspond to a bounding box size for depth patch image322and the pixels of portion702may be the pixels of atlas/image frame320that are covered by the bounding box for depth patch image322.

In some embodiments, the blocks may be selected as (M+1)×(N+1) blocks, wherein M=N=4. Thus the blocks may include columns and rows with 4 pixels and1edge pixel. For exampleFIG.7Billustrates a second block that overlaps the first block by one pixel on the edge adjacent to the first block.FIG.7Cthen shows a third block that overlaps the first block and also overlaps the second block at a corner pixel.FIG.7Dshows an additional block (Nth block) that overlaps both the second and third block and also the first block at a corner pixel.

The use of blocks may allow for utilization of GPU and/or CPU parallelization to increase a speed of simplifying a given sub-mesh to be rendered, because each block may be evaluated in parallel.

The use of blocks may also allow for the mesh resolution to be adjusted to the local variability of the depth information (e.g. depth map) such that the mesh resolution can be significantly reduced in local regions where the reduction does not introduce considerable error, but at the same time refraining from reducing (or reducing to a lesser degree) the mesh resolution in local regions where a greater reduction in the mesh resolution would appreciably affect the visual quality of the rendered content.

Also, the error threshold can be varied block by block, region by region, view by view, etc. providing a simple control over the simplification process. Also the error threshold may be dynamically adjusted based on other parameters, such as available computational resources to render the mesh, a current gaze of the viewport, etc. In some embodiments the error threshold may be computed as a local decimation error (€) that is computed as a local maximum error, a local average error, a local median error, a local n-th percentile error, etc. In some embodiments, an error value may be determined for each pixel of a block as a difference between the signaled depth for that pixel, (e.g. the pixel value for the given pixel indicating a depth value or an inverse depth that can be converted into a depth value) and a depth of the simplified mesh surface at the given pixel value, wherein a bi-linear interpolation is used to compute the surface depth value of the sub-mesh at the given location corresponding to a given pixel.

For example, in order to measure approximation error resulting from the use of the simplified sub-mesh for a given block (or sub-block), interpolated depth values are generated for boundary and internal pixels of the block (or sub-block) based on the values of the four corner pixels in the block. For example, a bilinear interpolation may be used. However, various interpolation techniques could also be used. Note,FIG.10illustrates a block and shows corner, pixels, boundary pixels, and internal pixels. The block illustrated inFIG.10may be any of the overlapping blocks shown inFIGS.7A-7D. The decimation error is measured between the original depth values and the interpolated ones. Various metrics could be used for the error such as sum of square errors (SSE), sum of absolute differences (SAD), max error, median error, etc.

For example,FIG.8Aillustrates a sub-mesh comprising two triangles that is generated for a given block (or sub-block) of a depth map, according to some embodiments. Also,FIG.8Billustrates errors being determined for depths represented by the sub-mesh in comparison to depth-values of the depth map for the block or sub-block, wherein the errors are used to determine whether or not the block or sub-block is to be further divided into sub-blocks, according to some embodiments.

If the approximation error is higher than the threshold then the block is subdivided into four sub-blocks of size (M/2+1)×(N/2+1) as depicted inFIGS.9A-9C. The same analysis is then applied to the sub-blocks.

For example,FIGS.9A-9Cillustrate a block determined for a depth map being sub-divided into sub-blocks in response to a determination that distortion introduced when modelling the block as a single sub-mesh (e.g. two triangles) exceeds a distortion threshold, according to some embodiments.

Due to the overlapping nature of the blocks, the algorithm needs to make sure that pixels values on block boundaries are consistent between neighboring blocks, in order to avoid generating a mesh with cracks. To achieve this, the following rules are enforced:Approximated depth values on boundary pixels assigned by bigger blocks cannot be overwritten by smaller blocks.Interpolated values for boundary pixels use only their associated corner pixels.The interpolation used for approximating sub-mesh depths for boundary pixels when determining error (in a decision whether or not to further sub-divide a block or sub-block) matches the interpolation used when rendering the sub-mesh for the given block.

In some embodiments, quantized inverse depth values are stored in the pixels of the depth map. Here, the decimation process is applied, while considering the inverse depth in order to give more importance to foreground objects (vs. background). Since the mesh will be finally rendered in the depth space, the linear interpolation needs to guarantee a crack free mesh in that space.

In such embodiments, a pixel (u, v) with a quantized inverse depth & could be unprojected to a 3D point P(M) as follows:

P-1(u,v,δ)=M=[xyz]=[z×(u-u⁢0)fuz×(v-v⁢0)fvf⁢a⁢r×n⁢e⁢a⁢rnear+δ×qs×(f⁢a⁢r-n⁢e⁢a⁢r)]

Where,fu, fv, u0, v0 are intrinsic camera parametersfar, near and qs are parameters used for depth quantization.

Projecting a point P(M) to generate a pixel (u, v) with a inverse depth δ is given by the following formula:

P⁡(M)=P⁡([xyz])=[uvδ]=[x×fuz+u⁢0y×fvz+v⁢01z-1far1near-1far⁢qs]

Consider a boundary pixel b(uB, vB, δB) and the two associated corner pixels C0(u0, V0, δ0) and c1(u1, v1, δ1). Let B(xB, yB, zB), C0(x0, y0, Z0) and c1(x1, y1, Z1) be the unprojected 3D points associated with b, c0and c1:B(xB, yB, zB)=P−1(b)C0(x0, y0, Z0)=P−1(c0)C1(x1, y1, Z1)=P−1(c1)

Since B is located on the line segment [C0, C1], it could be represented as follows:
B=C0+a(C1−C0)  [1]
xB=x0+a(x1−x0)
zB=Z0+a(z1−z0)

On the other hand, B should verify:

P⁡(B)=b[2]uB=xB×fuzB+u⁢0(uB-u⁢0)⁢(z0+α⁡(z1-z0))=(x0+α⁡(x1-x0))×fuα⁢{(uB-u⁢0)⁢(z1-z0)-fu(x1-x0)}=fu⁢x0-z0(uB-u⁢0)α=fu⁢x0-z0(uB-u⁢0)(uB-u⁢0)⁢(z1-z0)-fu(x1-x0)[3]

Equation (3) makes it possible to compute zBand therefore δBcould be computed as follows:

δB=1zB-1far1near-1far⁢q⁢s

Note: the above calculations suppose that (uB-u0)=0, which is the case of boundary pixels 0, 1, 2, 9, 10 and 11 in as shown inFIG.10. For the remaining boundary pixels inFIG.10(i.e., pixels 3, 4, 5, 6, 7 and 8), (VB-v0)+0. Therefore, the same formulas can be applied while replacing u with v.

More generally, if the projection function is highly non-linear and it is hard to directly solve the system, other approaches like dichotomy, gradient descent or other suitable techniques may be used to solve the non-linear systems.

Avoiding Cracks by Using Re-Meshing

FIGS.11A and11Billustrate examples of re-meshing that may be performed to increase a triangle count in particular regions of a sub-mesh corresponding to a given block (or sub-block), according to some embodiments.

In some embodiments, an additional re-meshing procedure may be performed on the simplified mesh comprising the merged sub-meshes. In some embodiments, such a re-meshing may be performed as follows:During a first pass, information about the retained subdivision structure for each (M+1)×(N+1) block (e.g., 4×4 block or four 2×2 blocks, or three 2×2 blocks and four 1×1 blocks . . . ) is determined.During a second pass, each block is triangulated based on its neighboring blocks (or sub-blocks), while making sure that all boundary points introduced by the neighboring blocks are taken into account.FIG.11A-11Bshows examples of how this could be achieved for two possible configurations.

In the second pass, multiple triangulations are possible. Choosing the best triangulation may consider different criteria such as: shape of the generated triangles; number of generated triangles; decimation error, etc.

Also, in some embodiments, different strategies for the re-meshing may be used based on the available time budget, such as finding the best triangulation based on the above criteria with different weightings assigned to the different criteria, e.g. shape of the generated triangles; number of generated triangles; decimation error, etc. Another strategy that may be used is a fixed choice per configuration or heuristics to achieve the best possible results (according the criteria defined above) for a fixed time budget.

In some embodiments, a decimation error threshold ε could be adaptively set for different regions based on various criteria (e.g., position with respect to the camera, confidence, importance, etc.)

In some embodiments, given a triangle budget, the decimation error threshold ε could be adaptively updated in order to achieve a number of triangles lower or equal to the budget. Prediction techniques from previous frames and intra frame prediction could be used to help with this process. The techniques used for rate control in video coding could be transposed.

In some embodiments, generation of interpolated values could be obtained by leveraging hardware-accelerated mipmap generation. Otherwise, it could highly benefit from single instruction multiple data (SIMD) operations.

In some embodiments, the decimation technique could consider directly depth values, or any quantity related to depth such as disparity, inverse of depth, or any function of depth.

FIG.12illustrates a flowchart for encoding three-dimensional (3D) volumetric content using video encoded 2D image frames, according to some embodiments.

At1202, an encoding computing device (such as the encoder shown inFIG.4, e.g. an “encoder”) receives images of a 3D scene or object captured from a plurality of camera locations and/or viewing angles, such as the views shown inFIGS.1Dcaptured by the cameras shown inFIGS.1A-1C. At1204, the encoder generates attribute patch images and depth patch images for the 3D object or scene, wherein a re-projection process (as described inFIG.2) is used to discard redundant information. At1206the encoder packs the attribute patch images and the depth patch images (e.g. the atlas and depth map) into 2D image frames, such as frames of a video image that are to be video encoded. At1208, the encoder generates metadata for the packed attribute and depth patch images packed into the image frames, such as auxiliary information indicating bounding box sizes and locations for bounding boxes for the respective patches (e.g. attribute patch images and corresponding depth patch images/depth maps). At1210, the encoder video encodes the packed image frames and, at1212, the encoder signals the metadata. For example the video encoded image frames and entropy encoded metadata may be included in an encoded bit stream for the 3D volumetric content.

FIGS.13A-13Billustrate a flowchart for determining blocks for portions of a packed image frame comprising depth patch images and determining error amounts for the blocks if a corresponding sub-mesh is generated for the blocks without further subdividing the respective blocks. The flowchart further illustrates the blocks being further subdivided if the determined error exceeds an error threshold, wherein the blocks and sub-blocks are iteratively sub-divided until the error threshold is met, according to some embodiments.

At1302a decoding computing device (e.g. decoder) receives video encoded image frames comprising packed attribute patch images and packed depth patch images (e.g. depth maps). At1304, the decoder video decodes the video-encoded image frames and at1306selects a first (or next) image frame comprising depth patch images to evaluate.

At1308, the decoder determines a plurality of blocks residing within the selected image frame being evaluated, such as the blocks shown inFIGS.7A-7D. The blocks may be evaluated in parallel by the decoder. Thus at1310,1312, and1314the decoder selects blocks of the selected image frame to evaluate. At1316,1318, and1320the decoder generates a sub-mesh for each of the respective blocks, such as the sub-mesh illustrated inFIGS.8A and8B. At1322,1324, and1326the decoder determines an error amount between approximated depth values that approximate a surface of the sub-mesh and actual depth values signaled for the block in the depth map. For example, as shown inFIG.8B. At1328,1330, and1332the decoder determines if the error amounts are greater than an error threshold. If so, then the blocks are further sub-divided at1336,1338, or1342(note that each block is evaluated separately so that some blocks may be sub-divided while others are not further sub-divided). For example, the blocks may be subdivided as shown inFIGS.9A-9C. If the determined error is less than the threshold, then the sub-mesh corresponding to the block(s) (without further sub-division) is selected for use in rendering the 3D volumetric content, e.g. this done at1334,1338, and/or1342. For the blocks that were further sub-divided an updated sub-mesh is generated for each sub-block of the further subdivided block, this is done at1346,1348,1350,1352,1354, and1356. Error values are determined for the further subdivided blocks and the process is repeated (at1358) until error for the sub-meshes for the current level of sub-blocks is less than the threshold. At1360the polygons to be used to render the 3D volumetric content are selected based on the sub-division process taking into account acceptable error.

FIG.14illustrates a flowchart for steps that may be included in the process shown inFIGS.13A/13B to exclude empty blocks from further consideration, according to some embodiments.

In some embodiments, blocks that do not include any occupied pixels (e.g. depth values) may be omitted from further consideration. For example, prior to evaluating a given block at1310,1312, or1314, the process shown inFIG.14may be performed, wherein at1402it is determined whether or not the block is empty. If the block is not empty, the process proceeds to1310,1312, or1314(as shown in1404ofFIG.14). If the block is empty, the block is omitted from further evaluation (1406ofFIG.14).

Example Inertial Measurement Unit (IMU)

FIG.15illustrates an example inertial measurement unit (IMU) that may be included in a decoding device, according to some embodiments.

Inertial measurement device1500includes accelerometer1502aligned with a Z-axis and configured to measure acceleration in the Z-direction, accelerometer1504aligned with a X-axis and configured to measure acceleration in the X-direction, and accelerometer1506aligned with a Y-axis and configured to measure acceleration in the Y-direction. Inertial measurement device1500also includes gyroscope1508configured to measure angular motion (Y) about the Z-axis, gyroscope1510configured to measure angular motion (0) about the X-axis, and gyroscope1512configured to measure angular motion (Q) about the Y-axis. In some embodiments, an inertial measurement device, such as inertial measurement device1500may include additional sensors such as magnetometers, pressure sensors, temperature sensors, etc. The accelerometers and gyroscopes of an inertial measurement device, such as accelerometers1502,1504, and1506, and gyroscopes1508,1510, and1512, may measure both translation motion and angular motion in multiple directions and about multiple axis. Such measurements may be used by one or more processors to determine motion of an object mechanically coupled to an inertial measurement device in three-dimensional space, such as decoding computing device.

Example Computer System

FIG.16illustrates an example computer system1600that may implement an encoder or decoder or any other ones of the components described herein, (e.g., any of the components described above with reference toFIGS.1-15), in accordance with some embodiments. The computer system1600may be configured to execute any or all of the embodiments described above. In different embodiments, computer system1600may be any of various types of devices, including, but not limited to, a personal computer system, desktop computer, laptop, notebook, tablet, slate, pad, or netbook computer, mainframe computer system, handheld computer, workstation, network computer, a camera, a set top box, a mobile device, a consumer device, video game console, handheld video game device, application server, storage device, a television, a video recording device, a peripheral device such as a switch, modem, router, or in general any type of computing or electronic device.

Various embodiments of an encoder or decoder, as described herein may be executed in one or more computer systems1600, which may interact with various other devices. Note that any component, action, or functionality described above with respect toFIGS.1-15may be implemented on one or more computers configured as computer system1600ofFIG.16, according to various embodiments. In the illustrated embodiment, computer system1600includes one or more processors1610coupled to a system memory1620via an input/output (I/O) interface1630. Computer system1600further includes a network interface1640coupled to I/O interface1630, and one or more input/output devices1650, such as cursor control device1660, keyboard1670, and display(s)1680. In some cases, it is contemplated that embodiments may be implemented using a single instance of computer system1600, while in other embodiments multiple such systems, or multiple nodes making up computer system1600, may be configured to host different portions or instances of embodiments. For example, in one embodiment some elements may be implemented via one or more nodes of computer system1600that are distinct from those nodes implementing other elements.

In various embodiments, computer system1600may be a uniprocessor system including one processor1610, or a multiprocessor system including several processors1610(e.g., two, four, eight, or another suitable number). Processors1610may be any suitable processor capable of executing instructions. For example, in various embodiments processors1610may be general-purpose or embedded processors implementing any of a variety of instruction set architectures (ISAs), such as the x86, PowerPC, SPARC, or MIPS ISAs, or any other suitable ISA. In multiprocessor systems, each of processors1610may commonly, but not necessarily, implement the same ISA.

System memory1620may be configured to store compression or decompression program instructions1622and/or sensor data accessible by processor1610. In various embodiments, system memory1620may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), nonvolatile/Flash-type memory, or any other type of memory. In the illustrated embodiment, program instructions1622may be configured to implement an image sensor control application incorporating any of the functionality described above. In some embodiments, program instructions and/or data may be received, sent or stored upon different types of computer-accessible media or on similar media separate from system memory1620or computer system1600. While computer system1600is described as implementing the functionality of functional blocks of previous Figures, any of the functionality described herein may be implemented via such a computer system.

In one embodiment, I/O interface1630may be configured to coordinate I/O traffic between processor1610, system memory1620, and any peripheral devices in the device, including network interface1640or other peripheral interfaces, such as input/output devices1650. In some embodiments, I/O interface1630may perform any necessary protocol, timing or other data transformations to convert data signals from one component (e.g., system memory1620) into a format suitable for use by another component (e.g., processor1610). In some embodiments, I/O interface1630may include support for devices attached through various types of peripheral buses, such as a variant of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard, for example. In some embodiments, the function of I/O interface1630may be split into two or more separate components, such as a north bridge and a south bridge, for example. Also, in some embodiments some or all of the functionality of I/O interface1630, such as an interface to system memory1620, may be incorporated directly into processor1610.

Network interface1640may be configured to allow data to be exchanged between computer system1600and other devices attached to a network1685(e.g., carrier or agent devices) or between nodes of computer system1600. Network1685may in various embodiments include one or more networks including but not limited to Local Area Networks (LANs) (e.g., an Ethernet or corporate network), Wide Area Networks (WANs) (e.g., the Internet), wireless data networks, some other electronic data network, or some combination thereof. In various embodiments, network interface1640may support communication via wired or wireless general data networks, such as any suitable type of Ethernet network, for example; via telecommunications/telephony networks such as analog voice networks or digital fiber communications networks; via storage area networks such as Fibre Channel SANs, or via any other suitable type of network and/or protocol.

Input/output devices1650may, in some embodiments, include one or more display terminals, keyboards, keypads, touchpads, scanning devices, voice or optical recognition devices, or any other devices suitable for entering or accessing data by one or more computer systems1600. Multiple input/output devices1650may be present in computer system1600or may be distributed on various nodes of computer system1600. In some embodiments, similar input/output devices may be separate from computer system1600and may interact with one or more nodes of computer system1600through a wired or wireless connection, such as over network interface1640.

As shown inFIG.16, memory1620may include program instructions1622, which may be processor-executable to implement any element or action described above. In one embodiment, the program instructions may implement the methods described above. In other embodiments, different elements and data may be included. Note that data may include any data or information described above.

Those skilled in the art will appreciate that computer system1600is merely illustrative and is not intended to limit the scope of embodiments. In particular, the computer system and devices may include any combination of hardware or software that can perform the indicated functions, including computers, network devices, Internet appliances, PDAs, wireless phones, pagers, etc. Computer system1600may also be connected to other devices that are not illustrated, or instead may operate as a stand-alone system. In addition, the functionality provided by the illustrated components may in some embodiments be combined in fewer components or distributed in additional components. Similarly, in some embodiments, the functionality of some of the illustrated components may not be provided and/or other additional functionality may be available.

Those skilled in the art will also appreciate that, while various items are illustrated as being stored in memory or on storage while being used, these items or portions of them may be transferred between memory and other storage devices for purposes of memory management and data integrity. Alternatively, in other embodiments some or all of the software components may execute in memory on another device and communicate with the illustrated computer system via inter-computer communication. Some or all of the system components or data structures may also be stored (e.g., as instructions or structured data) on a computer-accessible medium or a portable article to be read by an appropriate drive, various examples of which are described above. In some embodiments, instructions stored on a computer-accessible medium separate from computer system1600may be transmitted to computer system1600via transmission media or signals such as electrical, electromagnetic, or digital signals, conveyed via a communication medium such as a network and/or a wireless link. Various embodiments may further include receiving, sending or storing instructions and/or data implemented in accordance with the foregoing description upon a computer-accessible medium. Generally speaking, a computer-accessible medium may include a non-transitory, computer-readable storage medium or memory medium such as magnetic or optical media, e.g., disk or DVD/CD-ROM, volatile or non-volatile media such as RAM (e.g. SDRAM, DDR, RDRAM, SRAM, etc.), ROM, etc. In some embodiments, a computer-accessible medium may include transmission media or signals such as electrical, electromagnetic, or digital signals, conveyed via a communication medium such as network and/or a wireless link.

The methods described herein may be implemented in software, hardware, or a combination thereof, in different embodiments. In addition, the order of the blocks of the methods may be changed, and various elements may be added, reordered, combined, omitted, modified, etc. Various modifications and changes may be made as would be obvious to a person skilled in the art having the benefit of this disclosure. The various embodiments described herein are meant to be illustrative and not limiting. Many variations, modifications, additions, and improvements are possible. Accordingly, plural instances may be provided for components described herein as a single instance. Boundaries between various components, operations and data stores are somewhat arbitrary, and particular operations are illustrated in the context of specific illustrative configurations. Other allocations of functionality are envisioned and may fall within the scope of claims that follow. Finally, structures and functionality presented as discrete components in the example configurations may be implemented as a combined structure or component. These and other variations, modifications, additions, and improvements may fall within the scope of embodiments as defined in the claims that follow.