MULTI-FORMAT REPRESENTATION AND CODING OF VISUAL INFORMATION

System and methods are provided for using multi-format representation and coding of visual information. The system accesses an image data that comprises a texture data and a depth map; decomposes the depth map into a plurality of component depth maps (CDMs); and generates multiple focal planes (MFPs) comprising a plurality of focal planes. Each respective focal plane is based on the texture data and a respective CDM of the plurality of CDMs. The system selects a data subset including one or more of: (a) the texture data, (b) the depth map, (c) the plurality of CDMs, or (d) the plurality of focal planes; generates encoded data based on the selected data subset; and transmits, over a communication network, the encoded data to a client device to cause the client device to: generate for display or for further processing an image based on the encoded data.

BACKGROUND

This disclosure is generally directed to techniques for encoding video frames.

SUMMARY

A video plus depth (V+D) format (also referred to as texture and depth format) may be used for content capture and delivery and for supporting various stereoscopic 3D (S3D) and 3D applications. The V+D format may be used in multiple focal plane (MFP) display solutions to support natural eye focus/accommodation. However, the V+D format has various limitations which may hinder its use in 3D video coding and streaming. The format may have various limitations in particular w.r.t the dynamics and quality of the depth signal. For example, coding and transmitting high dynamic data from modern depth sensors may need improvements. Further, in current and future 3D applications, supporting arbitrary viewpoints may require better solutions for depth (distance) based quality adjustment.

When using V+D format, allocating bits between video and depth components may considerably affect the coding quality. An optimal format for coding quality may depend on the content and purpose of the use of the data. Thus, it may be desirable for a format to have more flexibility and precision to allocating bits.

In one approach, the average differences (e.g. mean squared error (MSE)) of original and coded pixel values may be used to make choices for coding quality. This distortion measure may work well for coding video (i.e. texture) sequences. However, it may not work well for coding depth signals (sequences of depth maps) due to different properties of video and depth signals. Thus, it may be desirable to not directly use video coding methods for coding depth data.

In one approach, video coding may be developed and used for viewing TV type of content, i.e., to be seen from a fixed viewpoint (e.g., without having a natural 3D viewpoint). In such cases, (e.g., similar for the human eye), the accuracy of the captured content may be inversely proportional to the square of the viewing distance. Further, in such use, knowledge on the pixel (voxel) distances (i.e. depth information) may not be needed. For example, stereoscopic (S3D) perception may be supported without depth data by capturing two views with a small disparity. However, this approach may have problems when content quality depends on a viewer's viewpoint and distance from the content (renderings). For example, problems may happen when a video plus depth signal is used for 3D reconstruction, and a user views content from varying viewpoints either by the user moving physically or by moving the content as a 3D model on a computer screen. In such cases, the content may be expected to be seen with a same or similar quality from any viewpoint and the accuracy cannot follow the inverse square of the viewing distance.

The requirements for content accuracy and resolution may be heavily dependent on a user's interactions and viewpoints to a rendering. Thus, it may be desirable that future compression coding methods provide better flexibility to support distance or depth dependent coding.

The present disclosure helps to address the problems described above, by, for example, providing systems and methods that improve and extend a V+D format by using distance (depth blending) based decompositions to produce additional content components.

The V+D format supports two different content types, i.e. image texture and depth information. The V and D formats may represent opposite ends of a texture-depth continuum, texture image being a depth-agnostic format, and depth being correspondingly a texture-agnostic component. However, the disclosed approach may use intermediate formats between these two, which may bring various benefits to 3D applications. The disclosed approach may extend the V+D format by a set of formats that may contain fine-grained mixtures of texture and depth information. The set of formats may be formed using a depth blending approach for a chosen number of depth ranges. The images generated in the set of formats (additional content components) may be referred to as multiple-format images. Source images may refer to images that may be chosen as representative images. For example, the images of the V+D format (e.g., texture image and depth map) may be referred to as source images (e.g., original source images). In some embodiments, texture data is selected as a default component (representative image) and the additional content components along with texture data may be referred to as source images. In addition to texture and depth map images, the set of multiple-format images may include component depth map images (CDMs), multiple focal planes (MFPs), and component depth map focal planes (CDMFPs).

In some embodiments, in addition to a texture image, one of several other source images may be chosen as a representative image for a depth range. In some embodiments, CDMFP images are not chosen as a representative image. By not choosing from CDMFP images, only all but one depth range (e.g., out of n depth ranges, only n−1 depth ranges) may need to be represented to reconstruct the original texture plus depth images from the representative images. An image for the one unrepresented range may be reconstructed indirectly, based on the relations of the decomposed images and the complementary properties of depth blending functions (e.g., partition of unity).

In some embodiments, a texture image is chosen as a default for delivery (e.g., transmission with other selected source images). The default texture image may enable the use of partition of unity property to recover the MFP for an unrepresented depth range. The depth map may not be transmitted by default, and a similar recovery may not be possible for a missing CDMFP image. Correspondingly, the whole format of CDMFPs may be discarded from being used as representative images.

The flexibility of choosing representative images may enable emphasizing desired properties of each input (V+D) image on the texture-depth continuum. The disclosed approach may enable a choice of a representative image for each depth range, free allocation of bits between the representatives, and which coding approach is used for each of the representatives. For example, depth signal properties may differ greatly at different distances from a depth sensor. In a receiver, any or all decomposed images may be formed from the decoded representatives, which may support a flexible variety of use cases for the output. In some embodiments, the receiver is a server that may support thin clients.

In some embodiments, a system (e.g., using a codec application) accesses image data that comprises texture data and a depth map. The system may decompose the depth map into a plurality of component depth maps (CDMs). The system may generate multiple focal planes (MFPs) comprising a plurality of focal planes. Each respective focal plane may be based on the texture data and a respective CDM of the plurality of CDMs.

In some embodiments, the system selects a data subset including one or more of: (a) the texture data, (b) the depth map, (c) the plurality of CDMs, or (d) the plurality of focal planes. For example, the plurality of CDMs may be a set of n CDMs, and the data subset may comprise (a) the texture data and (b) n−1 selected CDMs. As another example, the plurality of focal planes may be a set of n focal planes, and the data subset may comprise (a) the texture data and (b) n−1 selected focal planes. In one example, the data subset may comprise (a) the depth map and (b) the plurality of focal planes. As another example, the data subset may comprise (a) the plurality of CDMs and (b) the plurality of focal planes.

In some embodiments, the plurality of CDMs is a set of n CDMs, and the plurality of focal planes is a set of n focal planes. Each respective CDM of the plurality of CDMs and each respective focal plane of the plurality of focal planes may correspond to a respective depth range of the depth map. In some embodiments, the data subset comprises (a) the texture data and (b) n−1 selected CDMs and selected focal planes, and each one of the selected CDMs and the selected focal planes may correspond to a different depth range of the depth map. In some embodiments, the data subset comprises (a) the texture data and (b) n selected CDMs and selected focal planes, and each one of the selected CDMs and the selected focal planes may correspond to a different depth range of the depth map.

In some embodiments, the system generates encoded data based on the selected data subset. The system may transmit, over a communication network, the encoded data to a client device to cause the client device to generate for display an image based on the encoded data.

In some embodiments, the system decodes the encoded data to generate (a) decoded texture data and (b) n−1 decoded selected CDMs and decoded selected focal planes. The system may generate n reformed CDMs and n reformed focal planes from the decoded texture data and the n−1 decoded selected CDMs and decoded selected focal planes. The n reformed CDMs and n reformed focal planes may contain omitted CDMs and omitted focal planes that were not included in the selected data subset. In some embodiments, the system generates a reformed depth map by inverse depth blending the n reformed CDMs.

In some embodiments, the system generates a reformed CDM of the n reformed CDMs by: summing n−1 reformed CDMs of the n reformed CDMs to generate a partial CDM sum; and subtracting the partial CDM sum from a saturated matrix to generate the reformed CDM. A saturated matrix may be a fully saturated image (e.g., saturated image, each pixel is a maximum value of the pixel). For example, in an 8-bit image, each pixel may have 256 possible numerical values from 0-255, and in a saturated matrix or a fully saturated image of an 8-bit image, each pixel value may be the maximum value of 255.

In some embodiments, the system generates a reformed focal plane of the plurality of focal planes by: summing n−1 reformed focal planes of the n reformed focal planes to generate a partial MFP sum; and subtracting the partial MFP sum from the decoded texture data to generate the reformed focal plane.

In some embodiments, the system generates, from a decoded CDM of the n reformed CDMs, a corresponding reformed focal plane of the n reformed MFPs by: multiplying the decoded texture data by the decoded CDM to generate the corresponding reformed focal plane of the n reformed focal planes.

In some embodiments, the system transmits, over a communication network, the encoded data to an intermediary server to cause the intermediary server to: generate reformed data based on the encoded data, the reformed data containing omitted data not included in the selected data subset; and serve a subset of the reformed data to a thin client.

The disclosed multi-format approach may enable assigning bits and quality to desired features of the content to be coded. In addition to texture or depth based features, the variety of multi-format options may enable emphasizing/weighting features along more fine-grained texture-depth continuum. In effect, choosing an intermediate image format for coding may mean modifying an error measure from conventional error measures (e.g. SNR or PSNR). This may be effective for use in applications with changing user positions and viewpoints.

As a result of the use of these techniques, 3D media content may be efficiently encoded for storage and/or for transmission.

DETAILED DESCRIPTION

A video plus depth content format may be a versatile format for various S3D applications. The disclosed multi-format approach may improve and extend this format further by using distance (depth blending) based decompositions to produce additional content components. A small subset of multi-format components may enable adjusting quality on desired properties in the encoder and may be adequate to reconstruct a whole set of formats in the receiver. The disclosed approach may provide quality improvements and improve flexibility for various existing and future 3D applications.

A sequence of texture and depth map images, i.e. video plus depth (V+D) format may be a simple and versatile format for various multimedia applications. In addition to stereoscopic (S3D) renderings it may support even accommodation (natural eye focus) when decomposed into multiple focal planes (MFPs). The V+D format may support synthesizing of 3D viewpoints e.g. when producing stereoscopic (cf. depth image based rendering, DIBR) and motion parallax effects. Further, using various reconstruction algorithms (Simultaneous Localization and Mapping (SLAM), Truncated Signed Distance Function (TSDF), etc.), video plus depth streams may be used to produce 3D reconstructions of physical spaces.

However, in emerging 3DoF, 3DoF+ and 6DoF applications, a user may make more arbitrary and large viewpoint changes, which may challenge the possibilities of being served by the video plus depth format. Viewpoints may namely change to such orientations, that the quality of more far away objects and surfaces may not be enough. This type of need may depend largely on the use case and content and may require better flexibility than what the basic video plus depth format can provide. In particular, possibilities to adjust content quality depending on a camera's and viewer's varying distances to a scene may need to be improved.

Video plus depth (V+D) may be a common format for content capture and delivery. However, the format may have various limitations which may hinder its use in more advanced 3D applications. In particular, these limitations may relate the maximum supported dynamics and accuracy for the depth signal, and the possibilities for allocating bits (quality) depending on the distances of captured data. The disclosed approach may support a larger set of content formats. The formats may enable emphasizing/optimizing content properties in a more fine-grained way between the texture (depth agnostic) and depth map (texture agnostic) extremes.

The disclosed approach may use formats formed and supported by depth blending. For example, depth blending may enable conversions between different formats and the formation of various output formats in a receiver. Depth blending may be an approach to produce depth-based decompositions using video plus depth (texture and depth map) images. These images may form a continuum between the texture and depth map representation.

FIG.1Ashows an example of generating additional content components from texture image and a depth map using depth blending and selecting a subset of data for a multi-format representation of the texture image and the depth map, in accordance with some embodiments of this disclosure.

In some embodiments, a system receives, at step102, a texture data and a depth map. For example, a system may receive as an input (or access) an image in texture and depth (or V+D) format, such as example texture data120and a depth map121shown inFIG.1A. For example, image data may comprise a frame of a video (e.g., 3D video) or other media content (e.g., 3D video game, 3D video conference, etc.). In another example, image data may be received from a camera or a sensor. In some embodiments, image data may include a texture of an image and depth data (depth map) for the image data. For example, the texture data may be a table comprising luminance and/or color (e.g., YCbCr, RGB, or any suitable color format) matrix, where each cell in the matrix represents brightness and color of a single pixel. The depth data may be a matrix (of the same size as the texture matrix) that defines depth for each of the pixels in the texture matrix. While V+D may be a common format for content capture and delivery, this format may have limitations for use in advanced 3D applications. In steps104to108, the system may generate a set of additional content components (also referred to as source images and multi-format images) which may be used to support a more flexible coding and processing of content.FIG.1Ashows examples of decompositions formed using depth blending. The decomposed images may be the result of a multiple focal plane (MFP) formation process by depth blending. For example, MFPs may be formed by traditional linear depth blending (e.g., as described in Akeley, Kurt, et al. “A stereo display prototype with multiple focal distances.” ACM transactions on graphics (TOG) 23.3 (2004): 804-813, which is herein incorporated by reference in its entirety). The decomposed images inFIG.1Ainclude component depth maps (CDMs), multiple focal planes (MFPs), and component depth map focal planes (CDMFPs).

The system may decompose, at step104, the depth map into component depth maps (CDMs). For example, the system may decompose the depth map121into three component depth maps a first CDM130, a second CDM131, and a third CDM132. In this example, the depth map121is decomposed into three CDMs, but the depth map may be decomposed to any suitable number of CDMs (e.g., 3, 4, 5, etc. CDMs). For example, the system may decompose a depth map into n CDMs, where n is an integer greater than 1. The system may decompose the depth map into component depth maps by using a depth blending approach. For example, the system may apply linear (tent) blending functions to the depth map121to generate CDMs. In some embodiments, CDMs are referred to as weight planes.

In the example shown inFIG.1A, the depth map121may be a depth matrix d(x,y). For example, the three CDMs130,131, and132may be three matrices d1(x,y) (where d1(x,y) is a first CDM for a first depth range, d2(x,y) is a second CMD for second depth range, etc.). The CDM may have been generated by mapping (replacing) each pixel depth d(x, y) by a depth dependent value wi(d), where e.g., i=1 . . . 3. Similar technique may be used for any number of CDMs for any number of depth ranges, e.g., value of i may vary between 1 and 5, 1 and 10, 1 and 20, etc.

In some embodiments, depth blending is forming depth-range-dependent image components (planes) by weighting (blending) pixels in neighboring depth ranges. Blending may create a stack of image planes with smooth transitions and partial overlap between their contents. Depth blending may be used to form multiple focal planes (MFPs), which can be used to support natural eye-focus/accommodation by rendering the MFPs optically into a viewer's viewing frustum, at distances that the planes are formed to represent.

MFPs or other depth blending based decompositions may be formed at regular, dioptric or even arbitrary distances. In the example ofFIG.1A, and other examples in this disclosure, MFPs or other depth blending based decompositions (additional content components) may be given with fixed separations (e.g., separated at regular distances); however, the MFPs or additional content components may be formed at any suitable separation (e.g., regular, irregular, etc.) from each other. Details regarding depth blending can be found in descriptions ofFIG.2.

A property of the decompositions formed by depth blending may be their reciprocity, i.e., that an original source image can be reconstructed back from its decompositions (component images). In some embodiments, operations for completing a hierarchy of decomposed multiformat images using a chosen set of representative images are 1) depth blending, 2) inverse depth blending, and 3) partition of unity. Partition of unity may be implied by depth blending, and may be based on functions fulfilling this specific property. Details regarding depth blending using linear blending functions and inverse depth blending can be found in descriptions forFIGS.3A-3D.

The system may generate, at step106, multiple focal planes (MFPs) associated with the CDMs. MFPs may be a mixture of texture and depth information at specified depth ranges. For example, the system may multiply (e.g., by pixelwise multiplication) the texture data120with a respective CDM130,131, and132, to generate a respective MFP140,141, and142. The system may multiply texture data120with a first CDM130to generate a first MFP140. The system may multiply texture data120with a second CDM131to generate a second MFP141. The system may multiply texture data120with a third CDM132to generate a third MFP142. In some embodiments, the system generates n CDMs from a depth map, and the system generates n corresponding MFPs for each respective CDM.

The system may generate, at step108, multiple focal planes (MFPs) associated with the CDMs, referred to as component depth map MFPs (CDMFPs). For example, the system may multiply (e.g., by pixelwise multiplication) the depth map121with a respective CDM130,131, and132, to generate a respective CDMFP150,151, and152. The system may multiply the depth map121with a first CDM130to generate a first CDMFP150. The system may multiply depth map121with a second CDM131to generate a second CDMFP151. The system may multiply depth map121with a third CDM132to generate a third CDMFP152. In some embodiments, step108of generating the CDMFPs is an optional step.

The system may select, at step110, a data subset including one or more of the texture data, the depth map, the CDMs, or the MFPs. For example, the system may select texture image120, CDM130, and MFP142to represent the input image comprising the texture data120and the depth map102. In some embodiments, the system may have texture data120as a default and select the CDM130and MFP142from the additional content components (source images) as representative images. In some embodiments, the system selects a small subset of multi-format components. The selection of a small subset of multi-format components may enable adjusting quality on desired properties in the encoder and may be adequate to reconstruct a whole set of formats in the receiver.

FIG.1Bshows an example table of hierarchy/relations of content formats on a texture-depth continuum, in accordance with some embodiments of this disclosure. The example table175shows the relationship of content formats and their positioning on a texture-and-depth continuum. The first column indicates that processing advances in a downward direction, the middle columns represent the texture-depth continuum, and the last column provides comments. In the second row, the texture-depth continuum is represented by labels “Only Texture,” “Texture and Depth,” and “Only Depth.” As indicated by the comments, the third row shows examples of basic formats of “Only Texture” and “Only Depth.” For example, “Texture” or texture data may be a format that represents only texture in the texture-depth continuum, and a “Depth Map” may be a format that represents only depth in the texture-depth continuum. Advancing downwards to the fourth row, an initial process step may be decomposition of a depth map, which may generate CDMs, and “CDM” is a format that represents only depth in the texture-depth continuum. Advancing to the fifth row, and as indicated by the comments a next process step may be generating focal planes of a texture data. For example, “CDM” and “Texture” may generate the focal planes, and “MFP” is a format that represents texture and depth in the texture-depth continuum.

An example selection of three multi-format representations may be “Texture”, “CDM” and “MFP,” as shown in the shaded cells of the table. In some embodiments, texture may be a default representative format.

Conversions between formats may be made using depth blending as indicated by the arrows in the table. For example, a depth map may be used to form CDMs. A texture and CDM formats may be used to form MFP formats. A CDM format may be used for form CDMFP format. For example, a CDM format may be used to reconstruct a depth map format, and a depth map format may be multiplied (e.g., pixelwise multiplication) by the CDM format to generate a CDMFP format. In another example, although not shown for simplicity inFIG.1B, a depth map format may be multiplied by CDM format to generate CDMFPs.

In addition to video and depth signals, a set of formats supporting flexible distance specific coding and processing may be those obtained using a depth blending approach. For increasing flexible content generation, coding, and usage, use of a depth map or a sequence of depth maps may be replaced by using CDMs, which may be replaced with MFPs, formed using CDMs. MFPs may be a mixture of texture and depth information at specified ranges. The result may be a set of source images, from which a smaller subset may be coded and transmitted. In some embodiments, the system optimizes coding and usage of the content by selecting a subset based on desired depth properties and quality. In some embodiments, the system flexibly varies the content representation based on the viewpoints of the camera and a viewer, and their distances from the view or rendering.

For example, by using CDMs and MFPs instead of a depth map, a system can support flexibility in adjusting (depth dependent) quality and to enable multiple output formats. Both depth-dependent components may be formed from a texture and depth map image pair using depth blending. By omitting the depth map format, the resulting multi-format presentation may not unduly increase redundancy over the V+D format.

Texture (i.e. color) information may be needed for every pixel (voxel) of a view. Therefore, texture may be selected as a default data component, and may be coded and transmitted in any compilation of multi-format images. In some embodiments, where depth map is not selected as a default data component, CDMFPs may not be used for representative images. Note that a depth map may enable one depth range without explicit data (by exploiting the partition of unity property). CDMFPs may be formed in the receiver based on the supported multi-format options.

Any one component from each depth range may carry information both for texture and depth (cf. MFP) or just for the depth (cf. CDM). The granularity for the components can be adjusted by the number of depth blending functions (depth ranges) over the whole span of distances. Thus, in case e.g. more fine-grained depth-dependent adjustment for quality is desired, or e.g. more accuracy in accommodative rendering by MFPs in the receiver, the number of depth ranges may be increased. Correspondingly, the total number of content components and their combinations may be increased (i.e., options for source images for a multi-format representation).

In the multi-format decomposition into three depth ranges shown inFIG.1A, there may be a total of seven source images (a texture image120plus six options for multi-format components three CDMs130,131,132and three MFPs140,141,142). More examples and a general rule can be found below for the number of representative images as the function of the depth ranges produced by depth blending (c.f., columns in decompositions).

In some embodiments, increasing the number of depth ranges may increase the number of components to be reconstructed in the receiver. Although operations for the reconstruction may be straightforward per-pixel operations, the increase in reconstruction of components may be a drawback for thin clients. In some embodiments, a server performs reconstructions and desired formats may be transmitted from the server to thin clients. Details regarding an embodiment with a server can be found in the description ofFIG.10.

FIG.2depicts an example of depth blending techniques, in accordance with some embodiments of this disclosure.

For example, in a case where there are only two planes (L1and L2) that generate blended voxels, the depth blending200between the two focal planes may be determined according to the following equation:

where w1and w2are depth-weighted fusing functions. The perceived depth z of the fused voxel may be considered as a function of the depth-weighted fusing functions: z=f (w1, w2), where a simple approximation function may be the weighted sum of the depths of the two focal planes: z=W1(z)z1+W2(z)z2. The meaning of symbols of the equation above are demonstrated byFIG.1B. In particular, A refers to aperture of the eye, L1and L2refer to blended luminance values on focal planes at distances z1and z2, L0refers to the luminance value of the voxel to be blended at distance z. The symbol (2 refers to view angle from the eye through distances z1and z2.

FIGS.3A-Cdepict an example of depth blending techniques, in accordance with some embodiments of this disclosure. In particular,FIGS.3A-Cillustrate an example function that may be used to generate CDMs by decomposition block712,812ofFIG.7,FIG.8, respectively.

FIG.3Ashows a graph302of an example tent function. The “tent” is defined by a first line that intersects the x-axis at point a and a second line that that intersects the x-axis at point c. The lines further intersect at point b along the x axis. As shown, the x-axis corresponds to possible depth values. While depth values between 0 and 255 are shown for 8-bit matrices, any other ranges of depth values may be handled in a similar fashion.

FIG.3Bshows a graph304of an example set of tent functions. Multiple (e.g.,5or any number i) tent functions may be each be defined by respective points ai, bi, and ci. As shown, five tent functions are shown for (e.g., for creating 5 CDMs118). For each depth value x, two depth function values wj1(x) and wj2(x) can be computed by image processing application, each contributing to different component depth map in different depth ranges. Consequently, each depth value x in a respective position in depth matrix d will contribute to values in the respective position in two CDMs.

FIG.3Cshows a formula306for computing each depth function value wi(e.g., for i in range 1-5) based on tent functions304inFIG.3B. As shown the blending function306maps each depth value to at most two adjacent CDMs, and CDM values at other CDMs are zeroes. The formula306is reversible, as explained below.

FIG.3Ddepicts an example formula308for reversing depth blending (e.g., depth blending in accordance with formula306), in accordance with some embodiments of this disclosure. In particular, depth data d is recovered based on depth function values wj1(x) and Wj2(x) that were computed, e.g., as shown inFIG.3C.

While tent functions were shown inFIGS.3A-3C, any suitable blending function may be used instead (e.g., sinusoid functions).

In some embodiments, blending functions result in values that typically add up to the maximum depth value (255, when using 8 bit/pixel or to 2{circumflex over ( )}n−1 when using n bit/pixel). This is a property known as “partition of unity”, referring to expressing blending weights scaled between 0-1 (for any bit/pixel). This requirement can be fulfilled with an infinite number of suitable functions. In addition to linear (tent) functions, various other blending functions may be used, including polynomial, (piecewise) sinusoidal functions, spline functions, sc. bump functions (and their complements), blending by smooth transition functions, and different variations approximating partition of unity (e.g., using Friedrich's mollifiers).

FIG.4shows an illustrative example of selection options for a two-range multi-format presentation when texture data is selected as a default, in accordance with some embodiments of this disclosure. Example configurations a), b), c), and d) show a texture plus depth (V+D) image (top row) and its decomposition to two CDMs (2ndrow) and MFPs (3rdrow) and CDMFPs (lowest row). The grayed out images (e.g., unselected images) can be reconstructed using the chosen (selected) representative components.

Texture image401may be chosen as a default component and may not increase the number of configurations of the two decomposed multi-format components (MFPs and CDMs).FIG.4shows four options (circled) for decomposing a texture+depth map (V+D) image into a three component (e.g., CDM, MFP, and CDMFPs) multi-format presentation. Note that the images are schematic and illustrations of principle (e.g., not necessarily drawn to scale, etc.).

In some embodiments, original source images comprising texture image401(e.g., texture data) and a depth map402are decomposed into a two-level decomposition. Two CDMs may be formed by decomposing the depth map402into CDM1 and CMD2. Two MFPs may be formed by combining each CDM with the texture image401(e.g., multiplying by pixelwise multiplication, scaling, weighting, combining, or any combinations thereof). MFP1 may be formed by multiplying CDM1 by texture image401, and MFP2 may be formed by multiplying CDM2 by the texture image401. Two CDMFPs may be formed by combining each CDM with the depth map402(e.g., multiplying by pixelwise multiplication, scaling, weighting, combining, or any combinations thereof). CDMFP1 may be formed by multiplying CDM1 by the depth map, and CDMFP2 may be formed by multiplying CDM2 by the depth map.

In some embodiments, texture image401is used as a default source image, and one CDM or one MFP is selected in the different configurations (e.g., selected CDM or MFP in each configuration is as follows: a) CDM1, b) CDM2, c) MFP1, and d) MFP2). From the texture image401and the selected CDM or MFP in each configuration, the unselected images can be reconstructed. For example, in configuration a) the depth map402, CDM2, MFP1, MFP2, CDMFP1, and CDMFP2, can be reconstructed using the texture image401and CDM1. Details regarding reconstruction of the unselected images of the example ofFIG.4in configuration a) can be found in descriptions ofFIGS.9and17.

In the example forFIG.4, there are four compilations of representative images when two depth ranges are in use, and the total number of possible compilations for the representative images is four. Two different formats (CDM and MFPs) support four combinations with two depth ranges (columns). For example, inFIG.4there are two depth ranges and each configuration has two columns of images. The components CDM1, MFP1 and CDMFP1 in a first column may correspond to a first depth range, and the components CDM2, MFP2, and CDMFP2 in a second column may correspond to a second depth range. The depth ranges may correspond to columns in each configuration.

Table 1 shows a number of required multi-format images for coding and transmission (number of ranges/columns from2to5are shown) in accordance with some embodiments of this disclosure. For example, the required multi-format images for coding may refer to a minimum number of multi-format images to be selected as representative images (when texture image is chosen as a default) to be able to reconstruct the unselected images. Table 1 lists the number of possible components for a number of depth ranges (columns) (e.g., 2, 3, 4, and 5). In some embodiments, the texture image is chosen as a default (e.g., “1” in second column of Table 1 “Texture images”). In some embodiments, in addition to a default image texture, the need for representative components is one less than the supported number of depth ranges/columns (e.g., 1, 2, 3, and 4 in third column of Table 1). The total number of required multi-format images may be the sum of second column and third column in Table 1 (e.g., 2, 3, 4, 5 in fourth column of Table 1).

The total number of alternative multi-format representatives (cf. the last column in Table 1) may be determined as follows:

For two formats/rows (for CDMs and MFPs) and L ranges/columns:

For example, for L=4, Total number of options=4×8=32 (cf. Table 1).
The number of optional configurations may increase fast by increasing the number of depth ranges/columns.

FIG.5shows an illustrative example of a selection option for a three-level decomposition when texture data is selected as a default, in accordance with some embodiments of this disclosure. The example shown inFIG.5is one example of selecting an adequate number of supported multi-format images from a three-range decomposition. The example ofFIG.5shows chosen source images (=representative images) for a three-component multi-format presentation using a texture plus depth (V+D) image (top row), and its CDM decomposition (middle row) and MFP decomposition (bottom row).

In the example ofFIG.5, with three depth ranges, there may be a total of twelve adequate compilations of source images (cf. Table 2). Using texture as a default component, the system may select any source image from all but one column in order to form a complete multi-format presentation. For example, the source images to be selected from may be: first CDM530, second CDM531, third CDM532, first MFP540, second MFP541, and third MFP542. With three depth ranges,FIG.5shows three columns501,502, and503. Each column may represent a depth range (e.g., column501includes first CDM530and first MFP540corresponding to a first depth range, column502includes second CDM531and second MFP541corresponding to a second depth range, etc.). The system may select a source image from all but one column (e.g., select a source image from two of the three columns501,502, and503) as representative images to form a complete multi-format presentation. In the example ofFIG.5, the first CDM530from the first column501and the third MFP542from the third column503are selected as representative images.

Missing components can be reconstructed based either on the complementary properties of depth blended components (implied by the partition of unity property by the depth blending functions), or by using knowledge of the relation of depth blended components and using the complementary properties. In the example ofFIG.5, missing components (e.g., unselected components, omitted components) may include the second CDM531, the third CDM532, the first MFP540, the second MFP541.

Complementary Properties May be:

A missing CDM can be formed knowing that the sum of CDMs is a fully saturated (white) image (cf. partition of unity)A missing MFP can be formed if n−1 MFPs are known (by applying partition of unity and the knowledge of the texture image)

Relations of Depth Blended Components May be:

A missing MFP can be formed by the corresponding CDM (provided it exists) (e.g., a decoded or reformed CDM) and the texture image (i.e. by the depth blending process)A depth map can be formed by reverse depth blending after first deriving a full set of MFPs or CDMs

Both sets of reconstruction rules may be implications of the way decompositions are made, i.e. depth blending with functions fulfilling partition of unity. An example process for forming multi-format components in the receiver can be found in the descriptions ofFIGS.9and17.

In some embodiments, CDMFPs may be a format that can be reconstructed. For example, CDMFPs may be formed by decomposing a depth map to CDMs with depth blending functions, and by weighting (multiplying) the depth map with each CDM.

A procedure for forming MFPs, CDMs and CDMFPs may be depth blending. Depth blending may be used for depth-based decomposition of textures for supporting natural accommodation (natural eye-focus). Forming CDMs may be a pre-step for forming MFPs, i.e. (V+D)→CDMs→MFPs. CDMs can be used to form focal planes for depth maps, i.e. CDMFPs. In some embodiments, CDMFPs are not used as representation images, but CDMFPs can be formed in the receiver for an output and further use.

FIG.6shows a summary of the decompositions by depth blending with examples of use, in accordance with some embodiments of this disclosure. For example, a depth data (depth map) may be decomposed to CDMs, and CDMs may be used to form CDMFPs. CDMFPs may be used for viewpoint synthesis to 3D depth surfaces. CDMFPs may be used to improve predictive depth coding. CDMs may be used for depth dependent compression and quality adjustment. CDMs may be used for decomposition for MFP presentations. Video and CDMs may be used to form MFPs. MFPs may be used to support for accommodation and for viewpoint synthesis to textured depth maps (cf. DIBR). Video, CDMs, and MFPs may be used for all the above-mentioned purposes forFIG.6. Video, CDMs, and MFPs may be used for depth dependent compression and quality adjustment, Video, CDMs, and MFPs may be used for general flexibility.

Source formats may have a property that they can be formed from or converted between each other. Source images may form a hierarchy or matrix of images, in which every depth range except one may be represented by one image in any of the supported formats. The one depth range without a representative image can be completed (generated, reconstructed) using the complementary properties of depth blending functions (i.e., partition of unity, and reversibility of depth blending operations).

The disclosed multi-format approach may replace depth maps with multi-format images/decompositions obtained by depth blending. Component images may represent image properties on a chosen number of depth ranges. Corresponding image content may be reduced w.r.t the original depth map and/or texture (e.g., if there are no objects on a certain depth range, there may be no content either).

Although the number of images may be increased from the original (cf. the conventional pair of texture and depth images), increasing the number of components may not increase redundancy or overhead. Instead, the component images may be complementary, more compact, and straightforward to encode, and may produce less bits than the original texture-plus-depth image pair.

Although the luminance distributions of component images may differ from the original texture or depth map components, the results may appear to be similar and used as an option to be coded with existing video coding methods.

FIG.7shows an illustrative example of a simplified schematic block diagram for using a multi-format representation for coding, transmission, and decoding texture and depth data, in accordance with some embodiments of this disclosure. The system700may include a transmitter701and a receiver702. The transmitter701may access or receive as an input image data (e.g., texture data and a depth map) and may output coded selected data for transmission. The receiver702may receive the coded selected data as input and may generate a reformed image data (e.g., reformed texture data and reformed depth map) from decoded selected data. In some embodiments, the system700may include additional or different entities. For example, although not shown inFIG.7, system700may include a sync streaming block to synchronize multiple outputs from the coding block718. For example, sync streaming block may receive input signals and syncs the received input signals to generate synced output signals. In some embodiments, the output of the transmitter may include data that is synchronized (e.g., in a container file, or sequentially but with markings that allow them to be re-synchronized). For example, markings may be made in stream headers of associated data that allows for synchronization.

The transmitter701may include a decomposition block712, an MFP formation block714, a CDMFP formation block715, a selection block716, and a coding block718. In some embodiments, the CDMFP formation block715is optional. In some embodiments, transmitter701may include different or additional blocks.

The decomposition block712may decompose a depth map into component depth maps. The input to the decomposition block712may be an input a depth map from the image data. The output of the decomposition block712may be a set of component depth maps CDMs. In some embodiments, the decomposition block712generates CDMs from the depth map using depth blending techniques. For example, decomposition block712generates the component depth maps using pixel-by-pixel processing of the depth map using the depth blending functions. The decomposition block712may generate n CDMs from a depth map, where n is an integer greater than 1.

The MFP formation block714may generate MFPs from texture image and a depth map. The input to the MFP formation block714may be texture data and a set of CDMs (e.g., n CDMs, where n is an integer greater than 1). The MFP formation block714may generate a set of MFPs from the set of CDMs and the texture data. Each CDM of the set of CDMs may be multiplied (e.g., pixelwise multiplication) by a texture data to generate a respective MFP. The output of the MFP formation block may be a set of MFPs.

The CDMFP formation block715may generate CDMFPs from CDMs and a depth map. The input to the CDMFP formation block715may be a depth map and a set of CDMs (e.g., n CDMs). The CDMFP formation block715may generate a set of CDMFPs from the set of CDMs and the depth map. Each CDM of the set of CDMs may be multiplied (e.g., pixelwise multiplication) by a depth map to generate a respective CDMFP. The output of the MFP formation block may be a set of CDMFPs (e.g., n CDMFPs, where n is an integer greater than 1).

The selection block716may select a subset of data from the texture data, depth map, CDMs, MFPs, and CDMFPs. The input of the selection block716may be a texture image, a depth map, CDMs, MFPs, and CDMFPs. The selection block716may select the subset of data such that the selected subset of data may reconstruct the texture image, the depth map, and the source images in the configuration (e.g., full set of CDMs, MFPs, and CDMFPs). In some embodiments, the selection block716may select texture image as a default, and select additional components (e.g., selected CDMs and MFPs) as source images representative of the original source image (texture and data format) that may be used to reconstruct the texture image and depth map (and the full set of source images). The output of the selection block716may be a subset of data from the texture data, depth map, CDMs, MFPs, and CDMFPs.

In some embodiments, the selection block716may select a subset of data based on selection information data. The selection information data may include information relating to application type and/or user preferences. Application type information may include the type of application the transmitted information will be used for (e.g., TV, S3D, 6DoF, etc.). User preference information may be a setting that may indicate a selection preference of the user. For example, a user preference may be for a preference for a particular level or quality of images the user would like to view.

Selection information data may be obtained from a device705. The device705may be a user device including user preferences. The device705may be a storage device (e.g., storage database) which may include information including user preferences or other selection information data. InFIG.7, the device705is included in the transmitter701; however in some embodiments the device705is separate from the transmitter701. Details regarding selection information data and selection of source images based on selection information data can be found in descriptions ofFIG.8.

In some embodiments, selection block716may select a minimum number of images for reconstructing a set of output formats. For example, with n depth ranges (e.g., n CDMs and n MFPs), the selection block716may select a set of n source images (e.g., texture data and n−1 selected CDMs and MFPs, texture data and n−1 selected CDMs, or texture data and n−1 selected MFPs).

In some embodiments, selection block716may select more than a minimum number of images for reconstructing a set of output formats. For example, with n depth ranges (e.g., n CDMs and n MFPs), the selection block716may select a set of N+1 source images (e.g., texture data and n selected CDMs and MFPs, texture data and n selected CDMs, or texture data and n selected MFPs). Selecting additional source images may add redundancy, address (assign) and improve quality, increase the number of multi-format options for coding and transmission, and ease up forming and using output compositions/formats.

In some embodiments, selection block716may select depth map as a default selected source image. In some embodiments, selection block716may not select any default components (texture or depth).

In some embodiments, selection block716may select multiformat images so that the texture can be reconstructed in the decoder (e.g., receiver702, reconstruction block724, etc.) when it is not selected as a source image (e.g., despite not coding and sending texture explicitly). The selection block716may select multi-format components so that texture from all depth ranges is used when forming the multi-format components. For example, the selection block716may select MFPs from all distances (e.g., select each MFP from a set of n MFPs). The selection block716may select other depth-dependent components for adjusting quality and/or forming a corresponding depth map.

The coding block718may code data and output coded data. Coded data may be data that is no longer in a form of an image such as a frame of pixels. The coding block718may replace the input signal values with (statistically) shorter codes to reduce bits. The input of the coding block718may be the subset of selected data. For example, the subset of selected data may be the texture data and selected additional components (e.g., selected CDMs and MFPs). In some embodiments, as a default, source images in various formats may be coded separately, and due to their specific properties may get individual and largely independent errors when coded. In some embodiments, source images may be jointly coded. By jointly coding several components, coding efficiency may be increased (producing less bits). For example, some components may not be totally independent, as an object or surface can extend over several components (e.g., CDMs, MFPs). In such cases, the coding block718may apply a predictive method for jointly coding between several (neighboring) components. In some embodiments, there may be multiple coding blocks. For example, there may be a separate coding block for texture data in embodiments where texture data is selected as a default.

The receiver702may include a decoding block722and a reconstruction block724. In some embodiments, receiver702may include different or additional blocks.

The decoding block722decodes received data. The decoding block722may decode received data by replacing statistically shorter codes with corresponding expanded bits. The input to the decoding block722may be encoded selected data transmitted by the transmitter701. The decoding block722may decode the encoded selected data to generate decoded selected data. In some embodiments, there may be multiple decoding blocks. For example, there may be a separate decoding block for texture data in embodiments where texture data is selected as a default. In some embodiments, decoding block722may receive a container of encoded data, and the decoding block722may decode the container to retrieve the decoded selected data.

The reconstruction block724may generate reformed data based on the decoded selected data. For example, the reconstruction block724may use the decoded selected data to generate reformed data (e.g., texture data, depth map, CDMs, MFPs, and/or CDMFPs that were not selected). In some embodiments, reformed data may include decoded data and reconstructed data. For example, the reconstruction block724may generate reformed image data which may include reformed texture data and reformed depth map. In some embodiments, the reformed texture data may be decoded texture data, and the reformed depth map may be a reconstructed depth map. As another example, the reconstruction block724may generate n reformed CDMs and n reformed MFPs. The n reformed CDMs may include decoded CDMs (e.g. corresponding to selected CDMs) and reconstructed CDMs (e.g., corresponding to missing or omitted CDMs). The n reformed MFPs may include decoded MFPs (e.g. corresponding to selected MFPs) and reconstructed MFPs (e.g., corresponding to missing or omitted MFPs). The reconstruction block724may output reformed data. In some embodiments, the reconstruction block724may output the set or a subset reformed texture data, depth map, CDMs, MFPs, and CDMFPs. In some embodiments, after forming and converting coded source images to unified formats, coding results may be averaged to improve coding quality.

FIG.8shows an illustrative example of a schematic block diagram for using a multi-format representation including texture data for coding, transmission, and decoding video and texture data, in accordance with some embodiments of this disclosure. The system800includes a transmitter801and a receiver802. In some embodiments, the system800may include additional or different entities.FIG.8shows texture signal x1and depth signals d1(e.g., texture image and depth map) as inputs to the transmitter801. In some embodiments, transmitter801corresponds to (e.g., is the same as, similar to) transmitter701except that in the example ofFIG.8, texture data is selected as a default and encoded separately and that additional selection of data is made from source images (e.g., CDMs, MFPs, and CDMFPs). In some embodiments, receiver802corresponds to (e.g., is the same as, similar to) receiver702except that in the example ofFIG.8, texture data may be received and decoded separately from the selected source images.

InFIG.8, the decomposed components for each depth range may be numbered from1to n, or may be indicated by a number n, beside a connection for n parallel signals. Signal x1may refer to the texture (video) signal, and signal x1′ may correspond to the decoded version of x1. Signals d1and d1′ may refer to the original and decoded depth signals, respectively. However,FIG.8does not show a decoded depth signal d1′ and instead shows a reconstructed depth signal d2′. Subscript2for output signal x2′ may emphasize that signal x1′ may be improved by the received multiformat components. The subscript2for depth signal d2′ may emphasize that the signal may not be directly generated from a coded version of d1. A benefit of the disclosed approach may be that policies for adjusting coding quality can be varied for the chosen multi-format components. These policies may be managed by the source selection block816inFIG.8.

The transmitter801includes a texture coding block811, decomposition block812, MFP formation block814, CDMFP formation block815, source selection block816, and multi-format coding block818. In some embodiments, the decomposition block812, MFP formation block814, and CDMFP formation block815ofFIG.8correspond to (e.g., is the same as or similar to) decomposition block712, MFP formation block714, and CDMFP formation block715, respectively, ofFIG.7. In some embodiments, transmitter801may include different or additional blocks.

In some embodiments, the texture coding block811ofFIG.8corresponds to (e.g., is the same as or similar to) coding block711ofFIG.7, except the inputs and outputs differ. For example, the coding block711may have as input and code any of texture data, depth map, CDMs, MFPs, or CDMFPs, while the texture coding block811ofFIG.8may have as an input and code only texture data. Texture coding block811may code texture data. The input to the texture coding block811may be a texture (video) signal x1. The texture coding block811may encode the texture signal x1to generate encoded texture signal for transmission by the transmitter801.

Decomposition block812may decompose a depth map into a stack of CDMs. The input to the decomposition block812may be a depth data d1, and the output may be a set of n CDMs (e.g., n is an integer greater than 1). Although not shown inFIG.8for simplicity, the output of decomposition block812(e.g., n CDMs) may be an input to the multi-format coding block818.

MFP formation block814may generate MFPs from CDMs. The input to the MFP formation block814may be a set of n CDMs, and the output may be a set of n MFPs.

CDMFP formation block815may generate CDMFPs from CDMs and depth data. For example, the input to the MFP formation block814may be a set of n CDMs and the depth signal d1and the output may be a set of n MFPs. Although not shown inFIG.8for simplicity, the depth signal d1may be an input of the CDMFP formation block815.

In some embodiments, a source selection block816ofFIG.8corresponds to (e.g., is the same as or similar to) a selection block716ofFIG.7except the inputs and outputs differ. For example, the source selection block816may have as an input selection information data and may output an indication of which component source images (e.g., CDMs, MFPs, or CDMFPs) are selected, while the selection block716may have as an input selection information data and a full set of source images (e.g., texture data, depth data, CDMs, MFPs, and CDMFPs) and output the selected source images.

The source selection block816may select a subset of source images. The selection may be based on selection information data. The input to the source selection block816may be selection information data. The selection information data may include information relating to application type and/or user preferences. Application type information may include the type of application the transmitted information will be used for (e.g., TV, S3D, 6DoF, etc.). User preference information may be a setting that may indicate a selection preference of the user. For example, a user preference may be for a preference for a particular level or quality of images the user would like to view.

The selection information data may be obtained from a device805. In some embodiments device805corresponds to (e.g., is the same as, similar to) device705. The device805may be a user device including user preferences. The device805may be a storage device (e.g., storage database) which may include information including user preferences or other selection information data. InFIG.8, the device805is included in the transmitter801; however in some embodiments the device805is separate from the transmitter801.

The source selection block816may choose the set of source images and their quality based on the application or a user preference. For example, if a user preference indicates that the user prefers high quality images, source selection block816may select additional data to be sent to a client device associated with the user (e.g., instead of selecting n source images, selecting N+1 source images or an additional source image, or one or more additional source images to be sent). Having a representative component for every depth range instead of n−1 range may add some redundancy to a minimum representation and address (assign) and improve quality.

The source selection block816may select a quality of image based on an application type. For example source selection block816may choose high quality source images for particular application types (e.g., 6DoF applications). The source selection block816may send the multi-format coding block818an indication of what type of quality version of images may be used. For example, the multi-format coding block818may have different quality version multi-format images, and the source selection block816may send an indication to the multi-format coding block818to select a particular quality version of multi-format images.

In some embodiments, the source selection block816may choose the compilation of source images to increase or reduce the quality at different depth ranges (distances). For example, the source selection block816may select a CDM at a greater distance, using bits for coding a CDM at a greater distance so that far away depth properties are emphasized, and select a closer MFP, using bits for coding a closer MFP so that both nearby texture and depth are emphasized for quality.

By increasing the number of layers in the CDM and MFP decomposition, the selectiveness of the above-described weighting of properties may be increased (cf. Table 1).

Depending on the application or user's needs, the output can be chosen from many possible options. For example, either outputs with accommodation support (i.e., MFPs) or without such support (e.g., V+D for stereoscopic 3D rendering) can be chosen, depending on display capabilities. While V+D format may provide comparable benefits for output flexibility, the choice multi-format components (e.g., allocation of bits between multi-format components) may adjust the used distortion measure, which may enable assigning different quality to chosen features (represented by multi-format components).

Allocating bits (quality) between multi-format components may mean adjusting the used distortion measure. By choosing one of the several optional formats for coding the distortion metrics may be changed for the coding. When coding a texture, the metrics is depth agnostic. By choosing/coding an MFP, the metrics is texture-plus-depth-dependent on a corresponding chosen distance range. By coding a CDM, the metrics is based only on distances.

Normal metrics may not necessarily comply well with the perceived quality. The quality depends on the contents and the use of the content (e.g. viewing from different distances and synthesized viewpoints). By enabling choosing between various texture and depth dependent content components, the disclosed approach may offer flexibility to mitigate this problem. Using an altered distortion measure may not necessarily show any increase in traditional signal to noise ratio (SNR or PSNR). On the other hand, e.g. the possibility to separately weight depth ranges by quality suggests improvements in specific use cases. For example, being able to present certain depth ranges with better quality may improve synthesizing content or 3D viewpoints for those areas in 3DoF or 6DoF type of 3D applications.

The output of the source selection block816may be selector data. The selector data may include information indicating which source images are selected. The selector data may indicate which source images are selected.

In some embodiments, the selector data (e.g., where texture is selected as a default) may include a table, a matrix, or any suitable format for representing whether a source image is selected. For example, a matrix of size 2×2 (e.g., two rows and two columns) may be used to represent a selector data for the example ofFIG.4with CDMs and MFPs representing two depth ranges. Each row may represent a type of component image (e.g., first row may represent CDM, second row may represent MFP) and each column may represent a different depth range (e.g., first column may represent a first depth range, second column may represent a second depth range). As another example, a matrix of size 2×3 (e.g., 2 rows and 3 columns) may be used to represent a selector data for the example ofFIG.5with CDMs and MFPs representing three depth ranges. Each entry in the matrix may indicate whether a source image is selected. In the example ofFIG.4, the first row and first column of the matrix may correspond to CDM1 (component depth map corresponding to a first depth range). To represent this option a) ofFIG.4, the selector data may have an entry of “1” in the first column and first row, indicating that CDM1 is a selected source image. The other entries of the table (e.g., row 1, column 2; row 2, column 1; row 2, column 2) may be “0” to indicate that CDM2, MFP1, and MFP2 are not selected.

Multi-format coding block818may code a selected subset of multi-format components. The input to the multi-format coding block818may include a selector data. The source selector data may indicate which multi-format components to select to be coded. The multi-format coding block818accesses the components based on the selector data and encodes the selected components. Although not shown inFIG.8, in some embodiments the output of multi-format coding block818may transmit more than n selected images and the multi-format decoding block822may receive more than n encoded selected images.

Sync streaming block820receives input signals and syncs the received input signals to generate synced output signals. For example, the input to the sync streaming block820may be encoded texture data and n encoded multi-format components. The texture data may be encoded and ready for transmission before the multi-format components are generated, selected, and encoded for transmission. The sync streaming block820may delay streaming the encoded texture data until the encoded selected multi-format components are ready so that the encoded texture data and the encoded selected multi-format components are streamed together. The sync streaming block820may output synced encoded texture data with n encoded multi-format components. In some embodiments, the encoded texture data and the n encoded multi-format components may be transmitted in a synchronized manner (e.g., in a container file, or sequentially but with markings that allow them to be re-synchronized). For example, markings may be made in stream headers of associated data that allows for synchronization.

In some embodiments, the multi-format coding block818may select and encode n−1 multi-format components. The sync streaming block820may sync the encoded texture data with n−1 encoded multi-format components. In some embodiments, the encoded texture data and the n−1 encoded multi-format components may be transmitted in a synchronized manner (e.g., in a container file, or sequentially but with markings that allow them to be re-synchronized). For example, markings may be made in stream headers of associated data that allows for synchronization.

In some embodiments, the texture decoding block821ofFIG.8corresponds to (e.g., is the same as or similar to) decoding block722ofFIG.7, except the inputs and outputs differ. For example, the decoding block722may have as input and decode any of encoded texture data, depth map, CDMs, MFPs, or CDMFPs, while the texture decoding block821ofFIG.8may have as an input and decode only encoded texture data. Texture decoding block821may decode the encoded texture data. For example, the input to texture decoding block821may be encoded texture data, and the output may be decoded texture data x1′.

In some embodiments, the multi-format decoding block822ofFIG.8corresponds to (e.g., is the same as or similar to) decoding block722ofFIG.7, except the inputs and outputs differ. For example, the decoding block722may have as input and decode any of encoded texture data, depth map, CDMs, MFPs, or CDMFPs, while the multi-format decoding block822ofFIG.8may have as an input and decode only encoded multi-format data (e.g., CDMs, MFPs, and/or CDMFPs). Multi-format decoding block822may decode the selected subset of multi-format components. For example, multi-format decoding block822may decode n multi-format components. In some embodiments, multi-format decoding block822may decode a different number of multi-format components (e.g., n−1 multi-format components).

In some embodiments, the post-processing block824ofFIG.8corresponds to (e.g., is the same as or similar to) reconstruction block724ofFIG.7. Post-processing block824may reconstruct “missing” components from decoded texture and multi-format components. For example the post-processing block824may reconstruct missing depth map, CDMs, MFPs, and CDMFPs using depth blending, inverse depth blending, and partition of unity. In some embodiments, the post-processing block824may generate reformed texture data x2′ by summing (e.g., pixel by pixel) the set of decoded or reformed MFPs. The reformed texture data x2′ may be of higher quality than a decoded texture data x1′. The output of the post-processing block824may be a reformed texture image x2′, reformed depth map d2′, and reformed n additional components (e.g., n CDMs or n MFPs). In some embodiments, the output of post-processing block824may the set or a subset of reformed texture data, depth map, CDMs, MFPs, and CDMFPs. In some embodiments, the post-processing block824may choose an output format as defined by a user or application. For example, the post-processing block824may choose outputs with accommodation support (i.e., MFPs) or without such support (e.g., V+D for stereoscopic 3D rendering), as defined by a user or application (e.g., user preference, 3DoF, 3DoF+, 6DoF applications, etc.).

Display/further processing block826displays and/or performs further processing (e.g., further editing, or manipulation) on the texture data, the depth map, and the multi-format components. For example, the input into display/further processing block826may be a reformed texture image x2′, reformed depth map d2′, and reformed n additional components. In some embodiments, the input may include other components. The display/further processing (e.g., further filtering of image data) block826may display an image based on the reformed texture image x2′, reformed depth map d2′, and reformed n additional components. In some embodiments, further processing may be e.g., computer analysis for detecting and tracking objects at different distances (depth) of the scene.

FIG.9shows an illustrative example of generating reformed additional content components, in accordance with some embodiments of this disclosure.FIG.9illustrates an example of processes for forming the whole set of output images in the receiver, using two received (coded) multi-format images. The example choice for multi-format images of texture data901and CDM1 inFIG.9corresponds to option a) of texture data401and component depth map CDM1 inFIG.4. Note that the images are schematic and illustrations of principle (e.g., not necessarily drawn to scale, etc.).FIG.9is one example, and other cases may be processed in a similar way by following basic rules and processes of1) the partition of unity and 2) depth blending and its inverse. There may be alternative ways and order of the processes to result in the same or similar multi-format component outputs. For example, there may be two alternative ways to derive a missing MFP (e.g., using depth blending or partition of unity). In some embodiments, deriving all output formats may not be necessary depending on the use case. In some embodiments, a thin client may derive a subset of the multi-format components.

In the example ofFIG.9, a receiver (e.g., receiver702, receiver802) may receive encoded data representing a texture data901and a component depth map CDM1. The receiver (e.g., decoding block720, texture decoding block821) may decode the encoded texture data to generate decoded texture data based on a corresponding coding method (e.g., of coding block718, texture coding block811). The receiver (e.g., decoding block720, multi-format decoding block822) may decode the encoded CDM1 to generate decoded CDM1.

The receiver may generate a reformed CDM based on the partition of unity. For example, the receiver may generate a reformed CDM2 by subtracting (e.g., by pixelwise subtraction) the decoded CDM1 from a fully saturated image. The computation of the reformed CDM2 is based on the partition of unity that a set of CDMs sum up to a fully saturated image903. For example, with two depth ranges, two CDMs sum up to a fully saturated image.

The receiver may generate the reformed MFPs by depth blending of the texture image. For example, the receiver may generate the reformed MFPs by multiplying (e.g., by pixelwise multiplication) the texture data901by the respective CDMs. For example, the receiver may generate reformed MFP1 by multiplying texture data901by CDM1, and reformed MPF2 by multiplying texture data by the reformed CDM2. In some embodiments, the receiver may generate a reformed MFP using the partition of unity principle. For example, the receiver may generate a reformed MFP2 by subtracting (e.g., by pixelwise subtraction) a reformed MFP1 (generated by multiplying texture data901by CDM1) from the texture data901.

The receiver may generate a reformed depth map by inverse depth blending CDMs. For example, the receiver may generate depth map902by inverse depth blending decoded CDM1 and reformed CDM2.

The receiver may generate reformed CDMFPs by depth blending of the depth map image. For example, the receiver may generate reformed CDMFPs by multiplying (e.g., by pixelwise multiplication) the depth map902by respective CDMs. For example, the receiver may generate CDMFP1 by multiplying the depth map902by decoded CDM1, and CDMFP2 by multiplying the depth map902by reformed CDM2 (generated by subtracting decoded CDM1 from a fully saturated image). In some embodiments, a missing CDMFP may be generated by using the partition of unity property. For example, CDMFP2 may be generated by subtracting a reformed CDMFP1 (generated by pixelwise multiplication of the depth map by CDM1) from the reformed depth map902.

FIG.10shows an illustrative example of a schematic block diagram for using a multi-format representation including texture data with a server and thin client, in accordance with some embodiments of this disclosure. The system1000includes a transmitter1001, a server1003, and one or more thin client devices1004. In some embodiments, the system1000may include additional or different entities. In some embodiments, the transmitter1001and sync streaming block1020ofFIG.10corresponds to (e.g., is the same as, is similar to) the transmitter801and sync streaming block820, respectively, ofFIG.8.

In some embodiments, processing for reconstructing multi-format components is performed on a server for low-end (thin) client devices. A network server/content dispatcher may reconstruct the multi-format components. The server (e.g. on the network edge) may service several local clients by reconstructing (up to) a complete set of multi-format components. The server may support clients with less processing power. The server may keep track and manage the individual needs of multiple clients. In some embodiments, a fully individual service e.g. for enabling client-specific content quality is provided. In some embodiments, basic content is broadcast to the server(s) with only one quality level, from which various client streams are parsed.

The server1003includes texture decoding block1021, multi-format decoding block1022, post-processing block1024, client manager block1030, and multi-format coding and multiplexing block1032. The texture decoding block1021, multi-format decoding block1022, and post-processing block1024ofFIG.10correspond to (e.g., is the same as, is similar to) texture decoding block821, multi-format decoding block822, and post-processing block824, respectively, ofFIG.8. In some embodiments, the server1003may include different or additional blocks.

The client manager block1030may keep track and manage needs of multiple clients. The input to the client manager block1030may be control signals from thin client device(s)1004indicating user and application preferences. The output of the client manager block1030may be a client selection signal indicating which components are to be selected and coded for the client (e.g., a particular client).

The multi-format coding and multiplexing block1032may select which signals to code based on the client selection signal from the client manager block1030. For example, the input into multi-format coding and multiplexing block1032may be a reformed texture image x2′, reformed depth map d2′, and reformed n additional components. In some embodiments, the input may include other components. The multi-format coding and multiplexing block1032may code the selected signals. The output of the multi-format coding and multiplexing block1032may be the coded selected data which is transmitted from the server1003. In some embodiments, the client manager block1030ofFIG.10may correspond (e.g., is the same as, similar to) to the source selection block816ofFIG.8, except the inputs and outputs may differ. For example, client manager block1030may receive as input and manage multiple client selection signals from different client devices instead of receiving a selection signal from one device.

Thin client(s)1004may be user devices which may not have a lot of computing power. Thin client(s)1004may include a multi-format decoding block1041. In some embodiments, the thin client(s)1004may include different or additional blocks. Device1005may provide user and application preferences data1042to thin client(s)1004. The server1003(e.g., client manager1030) may send and receive control signals based on the user and application preferences data1042. The client manager1030may use these control signals to determine the subset of data to select.

Device1005may be a storage with user and application preferences. In some embodiments device1005corresponds to (e.g., is the same as, similar to) device705or device805. The device1005may be a user device including user preferences. The device1005may be a storage device (e.g., storage database) which may include information including user preferences or other selection information data.

Multi-format decoding block1041may decode received data input. For example, the input to multi-format decoding block1041may be client input data. The multi-format decoding block1041may output decoded client input data as client output data. For example, client output data may be decoded selected data. In some embodiments, the multi-format decoding block1041may receive as an input, user and application preferences data1042. The multi-format decoding block1041may decode the client input based on the user and application preferences data1042. For example, the client input data from server1003may include texture data, depth map, set of CDMs and/or set of MFPs, etc., and the multi-format decoding block1041may decode a portion of client input data (e.g., set of MFPs) as client output based on the user application and preferences data1042(e.g., indicating use of MFPs to be displayed simultaneously e.g., using a stack of transparent displays and/or using spatial light modulators (SLMs) that are capable of rendering image content to varying distances, controlled by tailored phase functions).

Because all the component images can be generated from texture and depth map (and texture is given), showing that the depth map D(x,y) can be generated from any combination of the representatives would prove that all component images can be generated.

Case 1: k=n−1: all representatives are CDMs.

Because ΣiCi(x,y)=255 (e.g., fully saturated 8-bit image, maximum value is 255) and n−1 CDMs are known, the nthCDM can be calculated. When all CDMs are known, depth map can be calculated using inverse blending function. Value of 255 is used as an example for 8 bit data, full saturation for n-bit data would be a value of 2{circumflex over ( )}n−1.

Case 2: k=0: all representatives are MFPs.

Because ΣiMi(x, y)=T(x, y), T(x,y) and n−1 MFPs are known, the nthMFP can be calculated. When all MFPs are known, all CDMs can be calculated (Ci(x, y)=T(x, y)/Mi(x, y)) and the depth map can be calculated using inverse blending function.

Case 3: k≠0 and k≠n−1: there are k CDM representatives and n-k-1 MFP representatives.
Because all the representatives are from different columns ({i1, i2, . . . ik}∩{j1, j2, . . . jn-k-1}=Ø), the missing CDMs (Cj(x,y), where j={j1, j2, . . . jn-k-1}) can be calculated using Cj(x,y)=T(x,y)/Mj(x,y). Then n−1 CDMs are known (nthCDM can be calculated, all CDMs are known) and depth map can be calculated using inverse blending function.

FIGS.12-13depict illustrative devices, systems, servers, and related hardware for image encoding/decoding.FIG.12shows generalized embodiments of illustrative user equipment devices1200and1201. For example, user equipment device1200may be a smartphone device, a tablet, a virtual reality or augmented reality device, or any other suitable device capable of processing video data. In another example, user equipment device1201may be a user television equipment system or device. User television equipment device1201may include set-top box1215. Set-top box1215may be communicatively connected to microphone1216, audio output equipment (e.g., speaker or headphones1214), and display1212. In some embodiments, display1212may be a television display or a computer display. In some embodiments, set-top box1215may be communicatively connected to user input interface1210. In some embodiments, user input interface1210may be a remote-control device. Set-top box1215may include one or more circuit boards. In some embodiments, the circuit boards may include control circuitry, processing circuitry, and storage (e.g., RAM, ROM, hard disk, removable disk, etc.). In some embodiments, the circuit boards may include an input/output path.

Each one of user equipment device1200and user equipment device1201may receive content and data via input/output (I/O) path (e.g., circuitry)1202. I/O path1202may provide content (e.g., broadcast programming, on-demand programming, Internet content, content available over a local area network (LAN) or wide area network (WAN), and/or other content) and data to control circuitry1204, which may comprise processing circuitry1206and storage1208. Control circuitry1204may be used to send and receive commands, requests, and other suitable data using I/O path1202, which may comprise I/O circuitry. I/O path1202may connect control circuitry1204(and specifically processing circuitry1206) to one or more communications paths (described below). I/O functions may be provided by one or more of these communications paths, but are shown as a single path inFIG.12to avoid overcomplicating the drawing. While set-top box1215is shown inFIG.12for illustration, any suitable computing device having processing circuitry, control circuitry, and storage may be used in accordance with the present disclosure. For example, set-top box1215may be replaced by, or complemented by, a personal computer (e.g., a notebook, a laptop, a desktop), a smartphone (e.g., device1200), a tablet, a network-based server hosting a user-accessible client device, a non-user-owned device, any other suitable device, or any combination thereof.

Control circuitry1204may be based on any suitable control circuitry such as processing circuitry1206. As referred to herein, control circuitry should be understood to mean circuitry based on one or more microprocessors, microcontrollers, digital signal processors, programmable logic devices, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), etc., and may include a multi-core processor (e.g., dual-core, quad-core, hexa-core, or any suitable number of cores) or supercomputer. In some embodiments, control circuitry may be distributed across multiple separate processors or processing units, for example, multiple of the same type of processing units (e.g., two Intel Core i7 processors) or multiple different processors (e.g., an Intel Core i5 processor and an Intel Core i7 processor). In some embodiments, control circuitry1204executes instructions for the codec application stored in memory (e.g., storage1208). Specifically, control circuitry1204may be instructed by the codec application to perform the functions discussed above and below. In some implementations, processing or actions performed by control circuitry1204may be based on instructions received from the codec application.

In client/server-based embodiments, control circuitry1204may include communications circuitry suitable for communicating with a server or other networks or servers. The codec application may be a stand-alone application implemented on a device or a server. The codec application may be implemented as software or a set of executable instructions. The instructions for performing any of the embodiments discussed herein of the codec application may be encoded on non-transitory computer-readable media (e.g., a hard drive, random-access memory on a DRAM integrated circuit, read-only memory on a BLU-RAY disk, etc.). For example, inFIG.12, the instructions may be stored in storage1208, and executed by control circuitry1204of a device1200.

In some embodiments, the codec application may be a client/server application where only the client application resides on device1200, and a server application resides on an external server (e.g., server1304and/or server1316). For example, the codec application may be implemented partially as a client application on control circuitry1204of device1200and partially on server1304as a server application running on control circuitry1311. Server1304may be a part of a local area network with one or more of devices1200or may be part of a cloud computing environment accessed via the internet. In a cloud computing environment, various types of computing services for performing searches on the internet or informational databases, providing encoding/decoding capabilities, providing storage (e.g., for a database) or parsing data are provided by a collection of network-accessible computing and storage resources (e.g., server1304and/or edge computing device1316), referred to as “the cloud.” Device1200may be a cloud client that relies on the cloud computing capabilities from server1304to determine whether processing (e.g., at least a portion of virtual background processing and/or at least a portion of other processing tasks) should be offloaded from the mobile device, and facilitate such offloading. When executed by control circuitry of server1304or1316, the codec application may instruct control circuitry1311or1318to perform processing tasks for the client device and facilitate the encoding/decoding.

Control circuitry1204may include video generating circuitry and tuning circuitry, such as one or more analog tuners, one or more MPEG-2 decoders or other digital decoding circuitry, high-definition tuners, or any other suitable tuning or video circuits or combinations of such circuits. Encoding circuitry (e.g., for converting over-the-air, analog, or digital signals to MPEG signals for storage) may also be provided. Control circuitry1204may also include scaler circuitry for upconverting and downconverting content into the preferred output format of user equipment1200. Control circuitry1204may also include digital-to-analog converter circuitry and analog-to-digital converter circuitry for converting between digital and analog signals. The tuning and encoding circuitry may be used by user equipment device1200,1201to receive and to display, to play, or to record content. The tuning and encoding circuitry may also be used to receive video data for encoding/decoding data. The circuitry described herein, including for example, the tuning, video generating, encoding, decoding, encrypting, decrypting, scaler, and analog/digital circuitry, may be implemented using software running on one or more general purpose or specialized processors. Multiple tuners may be provided to handle simultaneous tuning functions (e.g., watch and record functions, picture-in-picture (PIP) functions, multiple-tuner recording, etc.). If storage1208is provided as a separate device from user equipment device1200, the tuning and encoding circuitry (including multiple tuners) may be associated with storage1208.

Control circuitry1204may receive instruction from a user by way of user input interface1210. User input interface1210may be any suitable user interface, such as a remote control, mouse, trackball, keypad, keyboard, touch screen, touchpad, stylus input, joystick, voice recognition interface, or other user input interfaces. Display1212may be provided as a stand-alone device or integrated with other elements of each one of user equipment device1200and user equipment device1201. For example, display1212may be a touchscreen or touch-sensitive display. In such circumstances, user input interface1210may be integrated with or combined with display1212. In some embodiments, user input interface1210includes a remote-control device having one or more microphones, buttons, keypads, any other components configured to receive user input or combinations thereof. For example, user input interface1210may include a handheld remote-control device having an alphanumeric keypad and option buttons. In a further example, user input interface1210may include a handheld remote-control device having a microphone and control circuitry configured to receive and identify voice commands and transmit information to set-top box1215.

Audio output equipment1214may be integrated with or combined with display1212. Display1212may be one or more of a monitor, a television, a liquid crystal display (LCD) for a mobile device, amorphous silicon display, low-temperature polysilicon display, electronic ink display, electrophoretic display, active matrix display, electro-wetting display, electro-fluidic display, cathode ray tube display, light-emitting diode display, electroluminescent display, plasma display panel, high-performance addressing display, thin-film transistor display, organic light-emitting diode display, surface-conduction electron-emitter display (SED), laser television, carbon nanotubes, quantum dot display, interferometric modulator display, or any other suitable equipment for displaying visual images. A video card or graphics card may generate the output to the display1212. Audio output equipment1214may be provided as integrated with other elements of each one of device1200and equipment1201or may be stand-alone units. An audio component of videos and other content displayed on display1212may be played through speakers (or headphones) of audio output equipment1214. In some embodiments, audio may be distributed to a receiver (not shown), which processes and outputs the audio via speakers of audio output equipment1214. In some embodiments, for example, control circuitry1204is configured to provide audio cues to a user, or other audio feedback to a user, using speakers of audio output equipment1214. There may be a separate microphone1216or audio output equipment1214may include a microphone configured to receive audio input such as voice commands or speech. For example, a user may speak letters or words that are received by the microphone and converted to text by control circuitry1204. In a further example, a user may voice commands that are received by a microphone and recognized by control circuitry1204. Camera1218may be any suitable video camera integrated with the equipment or externally connected. Camera1218may be a digital camera comprising a charge-coupled device (CCD) and/or a complementary metal-oxide semiconductor (CMOS) image sensor. Camera1218may be an analog camera that converts to digital images via a video card.

The codec application may be implemented using any suitable architecture. For example, it may be a stand-alone application wholly-implemented on each one of user equipment device1200and user equipment device1201. In such an approach, instructions of the application may be stored locally (e.g., in storage1208), and data for use by the application is downloaded on a periodic basis (e.g., from an out-of-band feed, from an Internet resource, or using another suitable approach). Control circuitry1204may retrieve instructions of the application from storage1208and process the instructions to provide encoding/decoding functionality and perform any of the actions discussed herein. Based on the processed instructions, control circuitry1204may determine what action to perform when input is received from user input interface1210. For example, movement of a cursor on a display up/down may be indicated by the processed instructions when user input interface1210indicates that an up/down button was selected. An application and/or any instructions for performing any of the embodiments discussed herein may be encoded on computer-readable media. Computer-readable media includes any media capable of storing data. The computer-readable media may be non-transitory including, but not limited to, volatile and non-volatile computer memory or storage devices such as a hard disk, floppy disk, USB drive, DVD, CD, media card, register memory, processor cache, Random Access Memory (RAM), etc.

In some embodiments, the codec application is a client/server-based application. Data for use by a thick or thin client implemented on each one of user equipment device1200and user equipment device1201may be retrieved on-demand by issuing requests to a server remote to each one of user equipment device1200and user equipment device1201. For example, the remote server may store the instructions for the application in a storage device. The remote server may process the stored instructions using circuitry (e.g., control circuitry1204) and generate the displays discussed above and below. The client device may receive the displays generated by the remote server and may display the content of the displays locally on device1200. This way, the processing of the instructions is performed remotely by the server while the resulting displays (e.g., that may include text, a keyboard, or other visuals) are provided locally on device1200. Device1200may receive inputs from the user via input interface1210and transmit those inputs to the remote server for processing and generating the corresponding displays. For example, device1200may transmit a communication to the remote server indicating that an up/down button was selected via input interface1210. The remote server may process instructions in accordance with that input and generate a display of the application corresponding to the input (e.g., a display that moves a cursor up/down). The generated display is then transmitted to device1200for presentation to the user.

FIG.13is a diagram of an illustrative system1300for encoding/decoding, in accordance with some embodiments of this disclosure. User equipment devices1307,1308,1310(e.g., which may correspond to one or more of computing device may be coupled to communication network1306). Communication network1306may be one or more networks including the Internet, a mobile phone network, mobile voice or data network (e.g., a 5G, 4G, or LTE network), cable network, public switched telephone network, or other types of communication network or combinations of communication networks. Paths (e.g., depicted as arrows connecting the respective devices to the communication network1306) may separately or together include one or more communications paths, such as a satellite path, a fiber-optic path, a cable path, a path that supports Internet communications (e.g., IPTV), free-space connections (e.g., for broadcast or other wireless signals), or any other suitable wired or wireless communications path or combination of such paths. Communications with the client devices may be provided by one or more of these communications paths but are shown as a single path inFIG.13to avoid overcomplicating the drawing.

Although communications paths are not drawn between user equipment devices, these devices may communicate directly with each other via communications paths as well as other short-range, point-to-point communications paths, such as USB cables, IEEE 1394 cables, wireless paths (e.g., Bluetooth, infrared, IEEE 702-11x, etc.), or other short-range communication via wired or wireless paths. The user equipment devices may also communicate with each other directly through an indirect path via communication network1306.

System1300may comprise media content source1302, one or more servers1304, and one or more edge computing devices1316(e.g., included as part of an edge computing system). In some embodiments, the codec application may be executed at one or more of control circuitry1311of server1304(and/or control circuitry of user equipment devices1307,1308,1310and/or control circuitry1318of edge computing device1316). In some embodiments, a data structure transmitted by transmitter701ofFIG.7may be stored at database1305maintained at or otherwise associated with server1304, and/or at storage1322and/or at storage of one or more of user equipment devices1307,1308,1310.

In some embodiments, server1304may include control circuitry1311and storage1314(e.g., RAM, ROM, Hard Disk, Removable Disk, etc.). Storage1314may store one or more databases. Server1304may also include an input/output path1312. I/O path1312may provide encoding/decoding data, device information, or other data, over a local area network (LAN) or wide area network (WAN), and/or other content and data to control circuitry1311, which may include processing circuitry, and storage1314. Control circuitry1311may be used to send and receive commands, requests, and other suitable data using I/O path1312, which may comprise I/O circuitry. I/O path1312may connect control circuitry1311(and specifically control circuitry) to one or more communications paths.

Control circuitry1311may be based on any suitable control circuitry such as one or more microprocessors, microcontrollers, digital signal processors, programmable logic devices, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), etc., and may include a multi-core processor (e.g., dual-core, quad-core, hexa-core, or any suitable number of cores) or supercomputer. In some embodiments, control circuitry1311may be distributed across multiple separate processors or processing units, for example, multiple of the same type of processing units (e.g., two Intel Core i7 processors) or multiple different processors (e.g., an Intel Core i5 processor and an Intel Core i7 processor). In some embodiments, control circuitry1311executes instructions for an emulation system application stored in memory (e.g., the storage1314). Memory may be an electronic storage device provided as storage1314that is part of control circuitry1311.

Edge computing device1316may comprise control circuitry1318, I/O path1320and storage1322, which may be implemented in a similar manner as control circuitry1311, I/O path1312and storage1324, respectively of server1304. Edge computing device1316may be configured to be in communication with one or more of user equipment devices1307,1308,1310and server1304over communication network1306, and may be configured to perform processing tasks (e.g., for encoding/decoding) in connection with ongoing processing of video data. In some embodiments, a plurality of edge computing devices1316may be strategically located at various geographic locations, and may be mobile edge computing devices configured to provide processing support for mobile devices at various geographical regions.

FIG.14is a flowchart of a detailed illustrative process1400for multi-format processing including texture data for the transmitter, in accordance with some embodiments of this disclosure. In various embodiments, the individual steps of process1400may be implemented by one or more components of the devices and systems ofFIGS.7-8,10, and12-13. Although the present disclosure may describe certain steps of process1400(and of other processes described herein) as being implemented by certain components of the devices and systems ofFIGS.7-8,10, and12-13, this is for purposes of illustration only, and it should be understood that other components of the devices and systems ofFIGS.7-8,10, and12-13may implement those steps instead.

At step1402, control circuitry (e.g., control circuitry1311, control circuitry1318, or control circuitry of any of devices1307,1308, or1310) reads a texture and a depth image (x1and d1). For example, control circuitry may read (or access) a texture image and depth map from storage (e.g., storage1305, or at storage1322and/or at storage of one or more of user equipment devices1307,1308,1310). In some embodiments, control circuitry may be coupled to a sensor device and receive a stream of texture and depth data (e.g., texture image and a depth map) from the sensor device.

At step1404, the control circuitry decomposes the depth map into (chosen number of) n component depth maps (CDMs) using depth blending functions. For example, the control circuitry may generate the component depth maps using pixel-by-pixel processing of the depth map using the depth blending functions.

At step1406, the control circuitry forms n multiple focal planes (MFPs). For example, the control circuitry may generate n MFPs by pixelwise multiplication of each CDM of the n CDMs with the texture image (x1).

At step1408, the control circuitry forms n focal planes for component depth maps (CDMFPs) using CDMs. For example, the control circuitry may generate n CDMFPs by pixelwise multiplication of each CDM of the n CDMs with the depth map. Step1408may be an optional step.

At step1410, the control circuitry encodes the texture image x1with a chosen quality. For example, a choice for coding quality may be based on the average differences (e.g. MSE) of original and coded pixel values.

At step1412, the control circuitry selects and encodes n−1 other source images with chosen qualities. For example, the control circuitry may increase or reduce the quality at different depth ranges (distances) by using bits on (e.g., selecting and encoding) a CDM at a greater distance so that far away depth properties are emphasized, and using bits on (e.g., selecting and encoding) a closer MFP so that both nearby texture and depth are emphasized for quality.

At step1414, the control circuitry multiplexes and synchronizes all images and sends the data to the decoder(s). For example, the control circuitry may multiplex and synchronize the encoded texture image and n−1 encoded source images to the decoder(s). In some embodiments, input/output circuitry (e.g., input/output circuitry1311ofFIG.13) connected to the control circuitry transmits the data.

FIG.15is another flowchart of a detailed illustrative process for multi-format processing including texture data for the receiver, in accordance with some embodiments of this disclosure. In various embodiments, the individual steps of process1500may be implemented by one or more components of the devices and systems ofFIGS.7-8,10, and12-13. Although the present disclosure may describe certain steps of process1500(and of other processes described herein) as being implemented by certain components of the devices and systems ofFIGS.7-8,10, and12-13, this is for purposes of illustration only, and it should be understood that other components of the devices and systems ofFIGS.7-8,10, and12-13may implement those steps instead. In some embodiments, process1500may be performed to decode data that was encoded and transmitted using process1400.

At step1502, input/output circuitry (e.g., input/output circuitry1311ofFIG.13) connected to control circuitry (e.g., control circuitry1311, control circuitry1318, or control circuitry of any of devices1307,1308, or1310) receives encoded data. For example, input/output control circuitry may receive encoded texture image and n−1 encoded selected source images (e.g., from steps1410-1414ofFIG.14).

At step1504, the control circuitry (e.g., control circuitry1311, control circuitry1318, or control circuitry of any of devices1307,1308, or1310) demultiplexes the data. For example, control circuitry may demultiplex the received encoded data (e.g., encoded texture image and n−1 encoded selected source images).

At step1506, the control circuitry decodes the texture image x1′. For example, the control circuitry may decode encoded texture image to generate decoded texture image x1′.

At step1508, the control circuitry decodes chosen and received multi-format images. For example, the control circuitry may decode the n−1 encoded selected source images to generate n−1 decoded selected source images.

At step1510, the control circuitry forms ‘missing’ multi-format components by the inverse processes for depth blending and using the partition of unity rule. For example, the control circuitry may form source images that were not received in step1502(e.g., source images that were not selected and encoded in step1412and transmitted in step1414ofFIG.14).

At step1512, the control circuitry sums up source images in unified formats to average out coding errors in the multi-format presentation. In some embodiments, step1512is an optional step.

At step1514, the control circuitry forms an enhanced texture image x2′ and an enhanced depth map d2′. For example, the texture image x2′ and the depth map d2′ may be improved by the received multiformat components. In some embodiments, the set of MFPs may be summed up pixel by pixel to generate an enhanced texture image x2′. In some embodiments, a set of reformed CDMFPs may be summed up to generate an enhanced depth map. In some embodiments, a set of CDMs may be inverse depth blended to generate an enhanced depth map.

At step1516, the control circuitry uses the result for display or other purposes. In some embodiments, other purposes may refer to further image processing, e.g., computer analysis for detecting and tracking objects at different distances (depth) of the scene. For example, the control circuitry may use the result (e.g., set of MFPs) for accommodation support. For example, the control circuitry may use the result (e.g., enhanced texture image and depth map) for stereoscopic 3D rendering.

FIG.16is a flowchart of a detailed illustrative process for using a multi-format representation for coding and transmission for the transmitter, in accordance with some embodiments of this disclosure. In various embodiments, the individual steps of process1600may be implemented by one or more components of the devices and systems ofFIGS.7-8,10, and12-13. Although the present disclosure may describe certain steps of process1600(and of other processes described herein) as being implemented by certain components of the devices and systems ofFIGS.7-8,10, and12-13, this is for purposes of illustration only, and it should be understood that other components of the devices and systems ofFIGS.7-8,10, and12-13may implement those steps instead.

At step1602, control circuitry (e.g., control circuitry1311, control circuitry1318, or control circuitry of any of devices1307,1308, or1310) accesses an image data comprising a texture data and a depth map. In some embodiments, control circuitry accesses or reads a texture image and depth map from storage (e.g., storage1305, or at storage1322and/or at storage of one or more of user equipment devices1307,1308,1310). In some embodiments, control circuitry may be coupled to a sensor device and receive a stream of texture and depth data (e.g., texture image and a depth map) from the sensor device.

At step1604, the control circuitry decomposes the depth map into a plurality of component depth maps (CDMs). In some embodiments, the plurality of CDMs is a set of n CDMs. Each respective CDM of the plurality of CDMs may correspond to a respective depth range of the depth map.

At step1606, the control circuitry generates multiple focal planes (MFPs) comprising a plurality of focal planes, wherein each respective focal plane is based on the texture data and a respective CDM of the plurality of CDMs. In some embodiments, the plurality of focal planes is a set of n focal planes. Each respective focal plane of the plurality of focal planes may correspond to a respective depth range of a depth map.

At step1608, the control circuitry selects a data subset including one or more of: (a) the texture data, (b) the depth map, (c) the plurality of CDMs, or (d) the plurality of focal planes. For example, the data subset may comprise the texture data and n−1 selected CDMs. As another example, the data subset may comprise the texture data and n−1 selected MFPs. In one example, the data subset may comprise the texture data and n−1 selected CDMs and MFPs, where each one of the selected CDMs and the selected focal planes may correspond to a different depth range of the depth map. For example, the data subset may comprise the depth map and the plurality of focal planes. As an example, the data subset may comprise the plurality of CDMs and the plurality of focal planes. For example, the data subset may comprise the texture data and n selected CDMs and selected focal planes, where each one of the selected CDMs and the selected focal planes may correspond to a different depth range of the depth map.

At step1610, the control circuitry generates encoded data based on the selected subset. For example, the data subset may comprise texture data and n−1 selected CDMs and MFPs, and control circuitry may generate encoded texture data and n−1 encoded selected CDMs and MFPs.

At step1612, input/output circuitry (e.g., input/output circuitry1311ofFIG.13) connected to the control circuitry transmits, over a communication network, the encoded data to a client device to cause the client device to: generate for display an image based on the encoded data. In some embodiments, step1612is optional, and a system may perform steps1602-1610and1614.

At step1614, input/output circuitry (e.g., input/output circuitry1311ofFIG.13) connected to the control circuitry transmits, over a communication network, the encoded data to an intermediary server to cause the intermediary server to: generate reformed data based on the encoded data, the reformed data containing omitted data not included in the selected data subset; and serve a subset of the reformed data to a thin client. In some embodiments, step1614is optional, and a system may perform steps1602-1612.

FIG.17is a flowchart of a detailed illustrative process in the receiver, in accordance with some embodiments of this disclosure. In various embodiments, the individual steps of process1700may be implemented by one or more components of the devices and systems ofFIGS.7-8,10, and12-13. Although the present disclosure may describe certain steps of process1700(and of other processes described herein) as being implemented by certain components of the devices and systems ofFIGS.7-8,10, and12-13, this is for purposes of illustration only, and it should be understood that other components of the devices and systems ofFIGS.7-8,10, and12-13may implement those steps instead.

At step1702, control circuitry (e.g., control circuitry1311, control circuitry1318, or control circuitry of any of devices1307,1308, or1310) starts the process. In some embodiments, steps1704-1716may correspond to an example ofFIG.4option a) and steps taken to decodeFIG.4option a) inFIG.9. For example, references to CDM(1), CDM(2), MFP(1), MFP(2), CDMFP(1) and CDMFP(2) ofFIG.17may correspond to (e.g., is the same as, similar to) CDM1, CDM2, MFP1, MFP2, CDMFP1, and CDMFP2, respectively, ofFIGS.4and9.

At step1704, input/output circuitry (e.g., input/output circuitry1311ofFIG.13) connected to the control circuitry receives coded data. The control circuitry decodes texture image with a decoder. The control circuitry decodes the component depth map CDM(1) with a decoder.

At step1706, the control circuitry forms a missing CDM(2) as partition of unity (partition of a maximum value) complement by subtracting the decoded CDM(1) pixel-by-pixel from a fully saturated image.

At step1708, the control circuitry reconstructs a coded depth map by using an inverse depth blending algorithm using CDM(1) and CDM(2).

At step1710, the control circuitry forms the missing MFP(1) image by multiplying (e.g., by pixelwise multiplication) the decoded texture image by the corresponding decoded CDM(1). The control circuitry forms the missing MFP(2) image by multiplying the reconstructed texture image by the corresponding reconstructed CDM(2).

At step1712, the control circuitry forms two focal planes CDMFP(1) and CDMFP(2) for the reconstructed depth map using the two CDMs.

At step1714, the control circuitry uses any of the set of output images (texture, depth map, CDMs, MFPs, or CDMFPs) to support chosen end use.

At step1716, the control circuitry renders reconstructed texture plus depth frames for display or use decomposed output options for corresponding purposes (i.e., MFPs for accommodation support and MFPs and CDMFPs to viewpoint synthesis to texture and depth, correspondingly).

At step1718, the control circuitry checks whether all images are processed. If all images are not processed, control circuitry proceeds to step1704. If all images are processed, control circuitry proceeds to step1720to end the process.

In some embodiments, if high quality coding is used for the coded CDM, its accuracy is inherited also by a ‘complement’ CDM derived using the partition of unity property. A complete depth map may thus also be reconstructed with high accuracy. Similarly, the increased accuracy may be inherited by the reconstructed ‘missing’ MFP. A texture image received as their sum may correspondingly be of higher quality than the directly received texture image. Thus, in some embodiments, a system may reconstruct the texture image for use, even if the texture data has been received as a default component (cf. output signals x1′ and x2′).

FIG.18is a flowchart of a detailed illustrative process for using a multi-format representation for decoding video and texture data, in accordance with some embodiments of this disclosure. In various embodiments, the individual steps of process1800may be implemented by one or more components of the devices and systems ofFIGS.7-8,10, and12-13. Although the present disclosure may describe certain steps of process1800(and of other processes described herein) as being implemented by certain components of the devices and systems ofFIGS.7-8,10, and12-13, this is for purposes of illustration only, and it should be understood that other components of the devices and systems ofFIGS.7-8,10, and12-13may implement those steps instead.

At step1802, control circuitry (e.g., control circuitry1311, control circuitry1318, or control circuitry of any of devices1307,1308, or1310) decodes the encoded data to generate (a) decoded texture data and (b) n−1 decoded selected CDMs and decoded selected focal planes, wherein the plurality of CDMs is a set of n CDMs, the plurality of focal planes is a set of n focal planes and each respective focal plane of the plurality of focal planes corresponds to a respective depth range of the depth map.

At step1804, the control circuitry generates n reformed CDMs and n reformed focal planes from the decoded texture data and the n−1 decoded selected CDMs and decoded selected focal planes, wherein the n reformed CDMs and n reformed focal planes contain omitted CDMs and omitted focal planes that were not included in the selected data subset. In some embodiments, the system generates a reformed depth map by inverse depth blending the n reformed CDMs.

In some embodiments, the system generates a reformed CDM of the n reformed CDMs by summing n−1 CDMs of the n reformed CDMs to generate a partial CDM sum; and subtracting the partial CDM sum from a saturated matrix to generate the reformed CDM. A saturated matrix may be a fully saturated image.

In some embodiments, the system generates a reformed focal plane of the n reformed focal planes by: summing n−1 reformed focal planes of the n reformed focal planes to generate a partial MFP sum; and subtracting the partial MFP sum from the decoded texture data to generate the reformed focal plane.

In some embodiments, the system generates, from a decoded CDM of the n reformed CDMs, a corresponding reformed focal plane of the n reformed focal planes by: multiplying the decoded texture data by the decoded CDM to generate the corresponding reformed focal plane of the n reformed focal planes.

The disclosed multi-format approach may improve and extend a video plus depth format further by using distance (depth blending) based decompositions to produce additional content components. These new components, intermediate on a texture—depth continuum, may improve setting, producing, and perceiving desired depth-dependent content properties.

Further, the disclosed approach may provide improved flexibility for the output formats, which—in addition to the traditional video plus depth format—may now include various options based on depth-dependent decompositions. These outputs may support directly e.g. eye accommodation and synthesizing of 3D viewpoints. Using decompositions, the approach may further enable improved overall dynamics and quality, although each component may be coded with less bits and dynamics.

In some embodiments, depth and/or video coding may be used for coding source images. In some embodiments, a bit budget may be used to emphasize chosen signal properties. In some embodiments, using decompositions may enable higher dynamics and accuracy in coding.

In some embodiments, the number of source images to be coded may not necessarily increase redundancy. In some embodiments, depth blending may split (sample) the image information into multiple depth ranges, i.e., into different component images with little/no redundancy. This property may bring the benefit of adjusting bit budget between depth ranges as desired, in application or case-specific way—e.g., on those distances of interest in a surveillance view (cf. respecting the privacy of other targets). Choosing between depth and texture components for coding may mean setting weight on specific content properties (e.g., higher quality for distant objects/features, or e.g., more quality when coding close, textured regions). In some embodiments, coding more component images may enable increasing the overall quality (dynamics) over coding fewer components.

In some embodiments, an output format may be selected/changed based on application/user need (to be reconstructed). A reconstructed format (e.g., selected output format) may be changed on-the-fly (e.g., for supporting accommodation by MFPs, should a client device support it). In some embodiments, a network server (e.g., on the edge) may service multiple clients with individual contents and needs by parsing several outputs from one multi-format content. In some embodiments, a server may include support for client/case dependent quality control (e.g., emphasizing quality of certain distances or textures). The operations for depth blending and its inverse may be simple per pixel operations.