Patent ID: 12225178

DETAILED DESCRIPTION OF THE EMBODIMENTS

The invention will be described with reference to the Figures.

It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of the apparatus, systems and methods, are intended for purposes of illustration only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the apparatus, systems and methods of the present invention will become better understood from the following description, appended claims, and accompanying drawings. It should be understood that the Figures are merely schematic and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the Figures to indicate the same or similar parts.

Methods of encoding and decoding depth data are disclosed. In an encoding method, depth values and occupancy data are both encoded into a depth map. The method adapts how the depth values and occupancy data are converted to map values in the depth map. For example, it may adaptively select a threshold, above or below which all values represent unoccupied pixels. By adapting how the depth and occupancy are encoded, based on analysis of the depth values, the method can enable more effective encoding and transmission of the depth data and occupancy data. The encoding method outputs metadata defining the adaptive encoding. This metadata can be used by a corresponding decoding method, to decode the map values. Also provided are an encoder and a decoder for depth data, and a corresponding bitstream, comprising a depth map and its associated metadata.

It would be desirable to compress depth data using known image and video compression algorithms. It would be particularly desirable to be able to compress the depth data using standardized algorithms. Suitable hardware and software for encoding and decoding according to standardized codecs is widely available, and often highly optimized in terms of both speed, quality and power consumption. However, most video compression is lossy, in order to achieve bit rates that are practical for transmission and storage. Therefore, it cannot generally be guaranteed that a depth map compressed using video compression techniques will be reconstructed perfectly at the decoder. Errors will be introduced both by the compression and potentially also by the transmission and/or storage of the bitstream.

One basic approach to combine encoding of depth values and occupancy data in a depth map (not according to the invention) would be to set a fixed threshold to distinguish between unoccupied pixels and valid depth values. For example, it may be desired to encode depth data using HEVC Main 10 Level 5.2, meaning that the maximum bit depth is 10. This implies that 1024 levels (from 0 to 1023) are available to encode the depth and occupancy data. The coding scheme for depth data may define that all levels from 0 to 63 indicate an unoccupied pixel. Only levels 64 to 1023 are used to encode depth values. This implies that over 6% of the available range is given over to encode the occupancy. This may be appropriate for some content but inefficient for other content. It is difficult to choose a single fixed threshold that will be suitable generally for all types of content.

FIG.1illustrates an encoding method according to the first embodiment of the present invention.FIG.2is a schematic block diagram of an encoder for carrying out the method ofFIG.1.

The encoder300comprises an input310; an analyzer320; a depth value encoder330; and an output340. In step110, the input310receives source data comprising depth values. In the present embodiment, the source data is immersive video data comprising a plurality of source views. Each source view comprises texture values and depth values. Encoding of the texture values is outside the scope of the present invention and will not be discussed further here.

In step120, the depth value encoder330defines a depth map comprising an array of map values. Each map value takes one of a plurality of levels. For example, if the maximum bit depth is 10, there would be 1024 levels.

The input310is coupled to the analyzer320. In step130, the analyzer analyzes the depth values to determine how best to encode the depth values into the plurality of levels. In the present embodiment, map values below a threshold will be used to represent unoccupied pixels and map values above the threshold will be used to encode depth values. Therefore, the task of the analyzer is to choose the threshold (step140) based on the analysis of the depth values. Further details of how to choose the threshold will be discussed later below. For now, it is noted that the threshold (T) may be chosen to be a power of two (T=2n). This may be advantageous since it can allow a simple check, at the decoder, to establish whether a given map value is above or below the threshold. Rather than comparing the map value with a specific threshold value, the decoder can simply check the most significant bits (MSBs) of the map value. For example, if the threshold T=256=28, then the decoder can check the two most significant bits of the 10-bit representation. If both of these bits are 0, the value is below the threshold; otherwise, if either of the bits is 1, the value is above the threshold.

In step150, the depth value encoder330populates the depth map. For pixels that are unoccupied, the depth map is populated with one or more map values below the selected threshold. For each pixel that is occupied, the depth value encoder330converts the depth value to a respective map value lying above the threshold.

The depth value encoder330provides the populated depth map, containing the encoded map values, to the output340. Meanwhile, the analyzer320provides metadata to the output340. The metadata includes information defining how the depth values are encoded. In particular, the metadata includes information about the threshold chosen. The metadata may also include information about the mapping of depth values to map values in the range above the threshold. However, this may not be necessary in some embodiments as of the mapping may be defined explicitly in the coding scheme. For example, all depth values may be normalized disparity values in the range [0,1], and the mapping may be a linear mapping to map values above the threshold.

The output340generates and outputs a bitstream comprising at least the depth map. It also outputs the metadata, either as part of the same bitstream or separately from the bitstream.

FIG.3is a flowchart illustrating a method of decoding an encoded depth map according to a second embodiment of the invention.FIG.4is a schematic block diagram of a decoder for carrying out the method ofFIG.3.

The decoder400comprises an input410; a depth value decoder420; and an output430. Optionally, it may also comprise a renderer440.

In step210, the input410receives a bitstream comprising a depth map. The input also receives metadata describing the bitstream. The metadata may be embedded in the bitstream or may be separate. The depth map in this example is one created according to the method ofFIG.1described above. Note that the depth map input to the decoder400will typically be a version of the depth map output by the encoder300that has subsequently been subjected to lossy video compression (and possibly error-prone communication through a transmission channel).

In step220, the depth value decoder420decodes the depth map. This involves identifying map values above the threshold and converting them back to depth values. As discussed above, the threshold is included in the metadata. The proper conversion function may be agreed between the encoder and decoder in advance (for example, defined as part of a standardized coding scheme). Alternatively, if not defined/agreed in advance, the conversion function may be embedded in the metadata and the decoder may extract it from the metadata.

The depth value decoder420provides the decoded depth values to the output430. The output430outputs the depth values (step230). The depth value decoder may also output an occupancy map, indicating the pixels of the depth map where the map value was below the threshold.

If the decoder400includes the optional renderer440, the depth value decoder420may provide the decoded depth values to the renderer, which reconstructs one or more views from the depth data. In this case, the renderer430may provide the reconstructed view to the output430, and the output430may output this reconstructed view (for example, to a frame buffer).

There are various ways in which the map values can be dynamically assigned to encode the depth (and respectively occupancy) data. Some of these ways will now be discussed in more detail—along with the corresponding analysis to be performed by the analyzer320.

In some embodiments, analyzing the depth values comprises determining a dynamic range of the depth values. If the dynamic range is small (that is, if the depth values are all around the same value and the differences between them are not significant, then a small number of bits can be used to encode occupancy. For example, if all cameras are sufficiently close to an object, and the dynamic range of the depth map is not critical then one bit may be used to encode the occupancy map. That is, for a 10-bit depth map, the threshold level would be T=512=29. This still leaves 512 levels for encoding the depth data, which may be sufficient in close-up scenarios.

When a patch, view, frame or video is determined to have full occupancy, then the threshold may be set to 0 indicating that all pixels are occupied. This maximizes the number of levels available to encode the depth data.

In some embodiments, the method may comprise measuring or predicting the extent of visible errors in the decoded depth data that would be caused by encoding the depth values in a particular way. For example, the analyzer320may study the camera parameters associated with the source views in order to determine how to encode the depth values. If two cameras have a wide angular spacing (for instance, 90°), then depth errors in one view will be readily apparent as a shift to the left or right in the other view. In these circumstances, it would be advantageous to encode the depth values as accurately as possible. On the other hand, if two cameras have a small angular spacing (for instance, 5°), then errors in the depth values are much less likely to be perceptible.

In some embodiments, the analysis may comprise encoding the depth values, compressing the depth values, decompressing and decoding the depth values, and synthesizing a test view from the decoded depth values. The synthesized test view can be compared with a reference view derived from the original source data, to produce an error/difference image. This may be repeated for different configurations of the first subset and the second subset of levels. The configuration leading to the smallest error/difference may be chosen for encoding the depth values.

Depth values may be stored as normalized disparities (1/Z) with a near and far depth corresponding to the highest and lowest depth level, respectively. One model assumes that the chosen threshold level corresponds to the far depth and 1023 to the near depth (for 10-bit data).

When specifying the depth range, there are various ways to specify the occupancy coding.

FIG.5illustrates an example of depth values encoded into map values by a linear function. For a given depth value on the x-axis the function defines a corresponding map value on the y-axis. It is assumed that the depth values are provided as normalized disparities; hence, the depth value ranges from a minimum value of 1/Zmaxto a maximum value of 1/Zmin. In the example ofFIG.5, the maximum depth is assumed to be at infinity (Z=∞); therefore, the minimum value of the normalized disparity is 0. The y-axis intercept of the line gives the threshold level. T. Map values below T will be used to represent unoccupied pixels. The maximum depth value (1/Zmin) will be converted to the maximum map value (1023, in the case of 10-bit encoding). The map values for depth values between 0 and 1/Zmincan be obtained by linear interpolation. The conversion function may be defined in the metadata in various ways. For example, the metadata may include the threshold T and the minimum depth value Zmin. The maximum and minimum map values may be known implicitly from the bit-depth (1023 and 0, respectively, for 10-bits, for example); therefore, it may be unnecessary to encode these explicitly in the metadata.

FIG.6shows another exemplary linear conversion function. Here, the maximum depth, Zmax, is less than infinity; therefore, there is a nonzero 1/Zmax. The linear conversion function runs from the coordinates (1/Zmax, T) to the coordinates (1/Zmin, 1023). Again, map values below T are used to represent unoccupied areas. This conversion function can be defined in the metadata by three parameters: the threshold T; the minimum depth value Zmin; and the maximum depth value Zmax.

A potential problem can arise if compression or transmission errors are introduced in normalized disparity values that are very close to 1/Zmax. The map value may cross the threshold T as a result of an error, meaning that a pixel at a far depths incorrectly replaced with an unoccupied pixel.FIG.7shows a conversion function that seeks to prevent this by introducing guard levels between the levels that are used to encode the depth data and the levels that are used to indicate unoccupied pixels. The lowest map value used to encode depth values is given by a first threshold, T. But only map values below a second threshold, TG, are used to represent unoccupied pixels. This can equally be seen as introducing a guard band in the depth values, between 1/Zmaxand 1/ZG. It may be described in the metadata in either way. In other words, one additional threshold is defined in the metadata, which may be a threshold in map value or a threshold in depth value.

FIG.8illustrates another way to define a linear conversion function. Here, the metadata comprises a positive depth value 1/Zmin, and a negative depth value, −d. The negative depth value has no real, physical meaning (since normalized disparities are always non-negative). It serves as an implicit definition of the threshold T (at the y-axis intercept) below which map values are used to indicate unoccupied pixels. The example ofFIG.8has no guard levels, but these could be introduced by specifying an additional threshold, similarly to the approach taken inFIG.7.

FIG.9shows a piecewise linear conversion function. The piecewise linear function has three segments, which may be defined in the metadata by the coordinates of their endpoints. The map value defining the y-axis coordinate of the first endpoint implicitly defines the threshold T. In other words, the (x, y) coordinates of the first endpoint are (1/Zmax, T). The first linear segment has a relatively shallow slope, meaning that relatively few map values are being assigned to depth values in this range. The middle linear segment has a steeper slope, indicating that relatively more map values are being assigned to depth values in this range. The third linear segment again has a shallower slope, with fewer levels assigned to its map values. Such a profile may be appropriate, for example, when most of the depth values are clustered in the middle distance. In this case, errors in pixels that are very near to the camera or very far away may be less disruptive—that is, more acceptable to the viewer.

FIG.10shows another piecewise linear conversion function, with guard levels. The guard levels may be understood (and potentially encoded in the metadata) as a vertical step in the piecewise linear function. The guard levels are the map values in the range of the step, from TGto T. The unoccupied pixels are indicated only by map values less than TG.

The examples inFIGS.5-10are not mutually exclusive and conversion functions may be provided that combine attributes of several of these examples. For instance, one or more pieces of a piecewise linear function (like the one inFIG.9) may be defined by a notional negative depth value, as inFIG.8. A third subset of levels, consisting of guard levels, may be introduced in any of the examples.

In each of the examples above, the second subset of levels (indicating unoccupied pixels) are separated from the first subset of levels (indicating valid depth values) by one or two thresholds, with the first subset of levels being higher than the second subset of levels. It will be understood that this is not essential. In other embodiments, the levels may be allocated in different ways. For example, the analysis in step130may reveal that the depth data consists of a cluster of depth values close to the camera, and a cluster of depth values very far from the camera, with no pixels having depth values in the middle distance. In this case, it may make sense to allocate a set of levels in the middle of the range of map values for denoting unoccupied pixels. Such a range could be defined by a start threshold and an end threshold. In the case of encoding using a piecewise linear function, these thresholds may be implicit in the coordinates of the endpoints of the linear segments.

Although examples described above have used piecewise linear functions, it is of course possible that other functions could be used to convert between depth values and map values. Such functions could include (but are not limited to): quadratic functions, higher order polynomial functions, exponential functions, and logarithmic functions. The functions may be used in their entirety, or piecewise, combined with other piecewise functions.

Embodiments of the present invention rely on the use of metadata describing the encoding process when decoding the map values. Since the metadata is important to the decoding process, it may be beneficial if the metadata is encoded with additional error detecting or error correcting codes. Suitable codes are known in the art of communications theory.

The encoding and decoding methods ofFIGS.1and3, and the encoder and decoder ofFIGS.2and4, may be implemented in hardware or software, or a mixture of both (for example, as firmware running on a hardware device). To the extent that an embodiment is implemented partly or wholly in software, the functional steps illustrated in the process flowcharts may be performed by suitably programmed physical computing devices, such as one or more central processing units (CPUs) or graphics processing units (GPUs). Each process—and its individual component steps as illustrated in the flowcharts—may be performed by the same or different computing devices. According to embodiments, a computer-readable storage medium stores a computer program comprising computer program code configured to cause one or more physical computing devices to carry out an encoding or decoding method as described above when the program is run on the one or more physical computing devices.

Storage media may include volatile and non-volatile computer memory such as RAM, PROM, EPROM, and EEPROM. Various storage media may be fixed within a computing device or may be transportable, such that the one or more programs stored thereon can be loaded into a processor.

Metadata according to an embodiment may be stored on a storage medium. A bitstream according to an embodiment may be stored on the same storage medium or a different storage medium. The metadata may be embedded in the bitstream but this is not essential. Likewise, metadata and/or bitstreams (with the metadata in the bitstream or separate from it) may be transmitted as a signal modulated onto an electromagnetic carrier wave. The signal may be defined according to a standard for digital communications. The carrier wave may be an optical carrier, a radio-frequency wave, a millimeter wave, or a near field communications wave. It may be wired or wireless.

To the extent that an embodiment is implemented partly or wholly in hardware, the blocks shown in the block diagrams ofFIGS.2and4may be separate physical components, or logical subdivisions of single physical components, or may be all implemented in an integrated manner in one physical component. The functions of one block shown in the drawings may be divided between multiple components in an implementation, or the functions of multiple blocks shown in the drawings may be combined in single components in an implementation. Hardware components suitable for use in embodiments of the present invention include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs). One or more blocks may be implemented as a combination of dedicated hardware to perform some functions and one or more programmed microprocessors and associated circuitry to perform other functions.

Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure and the appended claims. In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. If a computer program is discussed above, it may be stored/distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. If the term “adapted to” is used in the claims or description, it is noted the term “adapted to” is intended to be equivalent to the term “configured to”. Any reference signs in the claims should not be construed as limiting the scope.