Patent Description:
Some computers configured to generate color video images of objects transmit the color video images over a network using a compression scheme. The technical problem is to transmit a color video image of an object over the network using as small a bitrate as possible. Conventional approaches to solving this technical problem include associating a delta quantization parameter (ΔQP) value to respective macroblocks (e.g., <NUM> X <NUM> pixels) of each video frame. In this way, a user can selectively increase or decrease the quality of the video in particular spatial regions of the frame.

The proposed solution relates to a method as defined by independent claim <NUM>, to a computer program product as defined by independent claim <NUM>, and to an electronic apparatus as defined by independent claim <NUM>.

Some color video systems achieve real-time transmission of color video by exploiting hardware video encoding and decoding of multiple streams of video, for example three depth views and four color views. These views are fused in a receiver to create low-latency left/right views based on the receiver's tracked eyes.

For such systems, a general problem is reducing a network transmission bitrate required by the video streams, in particular by the plurality of color views. Conventional approaches to reducing the network transmission bitrate includes selectively increasing or decreasing the quality of the video in particular spatial regions of the frame. For example, in a teleconferencing scenario, such conventional approaches may be used to keep greater detail on a user's face while allowing other parts of the frame to have reduced quality. Other conventional approaches attempt to preserve the quality of each of multiple color images; this is done by adapting the quality of view images (e.g., by compressing the compression quality spatially) with the aim of allowing a high-quality rendering of the final object as a combination of the compressed views.

A technical problem involved in the above-described conventional approaches to the network transmission bitrate is that such approaches are not effective in the presence of multiple color and depth views. For example, there is substantial overhead present in the redundancy of the color views, i.e. the fact that a point in the environment is often visible in several of the color views. Such redundancy is not addressed in the conventional approaches.

In contrast to the conventional approaches to solving the above-described technical problem, a technical solution to the above-described technical problem includes computing a delta quantization parameter (ΔQP) for the color images based on a similarity between the depth image surface normal and the view direction associated with a color image. For example, upon receiving a frame having an image with multiple color and depth images, a computer finds a depth image that is closest in orientation to a color image. For each pixel of that depth image, the computer generates a blend weight based on an orientation of a normal to a position of the depth image and the viewpoints from which the plurality of color images were captured. The computer then generates a value of ΔQP based on the blend weight and determines a macroblock of color image corresponding to the position, the macroblock being associated with the value of ΔQP for the pixel. The computer then performs a compression operation on that color image, a compression ratio of each of the plurality of macroblocks of the color image being based on the value of ΔQP with which the macroblock is associated.

A technical advantage of the above-described technical solution is that the technical solution allows for more efficient compression of color video images with multiple color and depth images because redundant information is reduced or eliminated. This efficiency is achieved by identifying content in the input color images that will not contribute to the final blended rendering, and degrading the compression quality of that content.

In a color video system improved according to the technical solution, left/right views rendered in a receiver are created by (i) geometrically fusing the multiple depth views to create a surface in 3D and (ii) blending the multiple color views over this fused surface. The color views are blended together at each surface point according to blend weights, where a blend weight associated with each view is a function of (i) whether the surface point is visible in the view (i.e. not occluded or outside the field of view) and (ii) a similarity between the surface normal and the view direction.

Note that a pixel in a color view may contribute only a small blend weight in the color reconstruction of the scene surface because either (i) the view ray through the pixel does not intersect the scene surface and instead sees the background or (ii) the ray intersects a surface obliquely such that another view provides a better (more head-on) coverage at that surface point. Accordingly, if a block of pixels in a color view all contribute small blend weights, one may compress that block with lower quality (using a higher QP value) without affecting the quality of the final reconstruction.

<FIG> is a diagram that illustrates an example electronic environment <NUM> in which the above-described improved techniques may be implemented. As shown, in <FIG>, the example electronic environment <NUM> includes a computer <NUM>.

The computer <NUM> is configured to compress color images. The computer <NUM> includes a network interface <NUM>, one or more processing units <NUM>, and memory <NUM>. The network interface <NUM> includes, for example, Ethernet adaptors, and the like, for converting electronic and/or optical signals received from a network to electronic form for use by the computer <NUM>. The set of processing units <NUM> include one or more processing chips and/or assemblies. The memory <NUM> includes both volatile memory (e.g., RAM) and non-volatile memory, such as one or more ROMs, disk drives, solid state drives, and the like. The set of processing units <NUM> and the memory <NUM> together form control circuitry, which is configured and arranged to carry out various methods and functions as described herein.

In some embodiments, one or more of the components of the computer <NUM> can be, or can include processors (e.g., processing units <NUM>) configured to process instructions stored in the memory <NUM>. Examples of such instructions as depicted in <FIG> include a depth image manager <NUM>, a color image manager <NUM>, a depth image identification manager <NUM>, a blend weight manager <NUM>, a quantization parameter manager <NUM>, a macroblock manager <NUM>, and a compression manager <NUM>. Further, as illustrated in <FIG>, the memory <NUM> is configured to store various data, which is described with respect to the respective managers that use such data.

The depth image manager <NUM> is configured to receive depth image data <NUM>. The depth image manager <NUM> receives the depth image data <NUM>, in some implementations, over a network via the network interface <NUM>. In some implementations, the depth image manager <NUM> receives the depth image data <NUM> from a local storage device, e.g., a hard drive, a flash drive, a storage disk, and so on.

The depth image data <NUM> represents a plurality of depth images <NUM>(<NUM>). <NUM>(N) of an object. Each depth image, e.g., depth image <NUM>(<NUM>), represents a map of distances along a line from a camera to pixels on the surface of the object. The camera is oriented with respect to the object at an angle indicated by a viewpoint from which the depth image is captured. In some implementations, each of the plurality of depth images <NUM>(<NUM>). <NUM>(N) has a plurality of pixels, where each pixel is associated with a depth value. In some implementations, each depth image represents the depth value (z) of the surface point in the reference frame of an image capture device (e.g., a camera).

The color image manager <NUM> is configured to receive color image data <NUM>. The color image manager <NUM> receives the color image data <NUM>, in some implementations, over a network via the network interface <NUM>. In some implementations, the color image manager <NUM> receives the color image data <NUM> from a local storage device, e.g., a hard drive, a flash drive, a storage disk, and so on.

The color image data <NUM> represents a plurality of color images <NUM>(<NUM>). <NUM>(N) of an object. Each depth image, e.g., color image <NUM>(<NUM>), represents a map of distances - or depths - along a line from a camera to pixels on the surface of the object. The camera is oriented with respect to the object at an angle indicated by a viewpoint from which the depth image is captured.

The depth image identification manager <NUM> is configured to identify a depth image of the plurality of depth images associated with that color image. In some implementations, the depth image identification manager <NUM> identifies, as the depth image associated with a color image (e.g., color image <NUM>(<NUM>)), a depth image (e.g., depth image <NUM>(<NUM>)) having a center of projection closest to a center of projection of the color image <NUM>(<NUM>). Such an identification is further described with regard to <FIG>.

The blend weight manager <NUM> is configured to determine a position of the color image based on a depth value of the depth image associated with a pixel of the depth image and generate a blend weight corresponding to the determined position to produce blend weight data. In some implementations, the position is a three-dimensional world-space position. In some implementations, the blend weight manager <NUM> is configured to normalize the blend weights across the plurality of color images.

The blend weight data <NUM> represents the respective blend weight computed for each pixel of each depth image. In some implementations, the blend weight for the pixel of the depth image is based on an inner product of a normal of the surface of the object at the position and the viewpoint from which the color image is captured. In some implementations, the blend weight for that pixel is proportional to the inner product raised to the power of a specified exponent. In some implementations, the blend weight for that pixel is further based on whether a point at the position is visible along the viewpoint from which the color image is captured.

The quantization parameter (QP) manager <NUM> is configured to generate, as the QP data <NUM>, a value of a delta quantization parameter (ΔQP) for each of the plurality of pixels based on the blend weight for that pixel. In some implementations, ΔQP is based on normalized weights.

The QP data <NUM> represents the ΔQP values across pixels of each color image and, ultimately, across macroblocks across each color image. It is understood that a QP determines a step size for associating coefficients of a discrete cosine transform (DCT) of a color image with a finite set of steps. Large values of QP represent big steps that crudely approximate the transform, so that most of the signal can be captured by only a few coefficients. Small values of QP more accurately approximate the block's spatial frequency spectrum, but at the cost of more bits. <NUM>, each unit increase of QP lengthens the step size by <NUM>% and reduces the bitrate by roughly <NUM>%.

The macroblock manager <NUM> is configured to determine a macroblock of the plurality of macroblocks of the color image corresponding to the position to produce macroblock data <NUM>. Ultimately, the macroblock is associated with an aggregate (e.g., mean, median, maximum, minimum) value of ΔQP for the pixel of the depth image (e.g., is based on the value of ΔQP for the pixel of the depth image). In some implementations, the macroblock manager <NUM> is configured to project a point at the position of the surface into a color image plane in which a camera is situated to determine the macroblock associated with the value ΔQP for the pixel.

In some implementations, the macroblock manager <NUM> is configured to perform a rasterization operation on the point into a framebuffer associated with the color image to store a new value of ΔQP in a pixel of a plurality of pixels of the framebuffer. The framebuffer also includes a plurality of tiles, each of the plurality of tiles including a set of pixels of the framebuffer. In this case, the macroblock manager <NUM> is further configured to perform a read operation on a tile of the frame buffer corresponding to a macroblock to produce an aggregated value of ΔQP. The aggregated value of ΔQP is based on the new values of ΔQP of the tile.

The macroblock data <NUM> represents matrices of values of ΔQP over the macroblocks of each color image. If the color image is easily viewed from the depth image surface, then the weight will be large and the value of ΔQP for the macroblock to which the point associated with the pixel is projected will be accordingly small, and vice-versa.

In some implementations, the macroblock manager <NUM> can update the macroblock data <NUM> based on the regions of the depth image projected to each macroblock. For example, the values of ΔQP for macroblocks identified with a face of a person may be decreased while other macroblocks associated with other parts of the person may have increased values of ΔQP.

The compression manager <NUM> is configured to perform a compression operation on each color image according to the macroblock data <NUM> to produce the compression data <NUM>. The compression data <NUM> is transmitted to a receiver, where it is decompressed (decoded) and fused together to create a single color video image.

<FIG> is a flow chart depicting an example method <NUM> of compressing color images. The method <NUM> may be performed by software constructs described in connection with <FIG>, which reside in memory <NUM> of the user device computer <NUM> and are run by the set of processing units <NUM>.

At <NUM>, the depth image manager <NUM> receives depth image data <NUM> representing a plurality of depth images of an object, each of the plurality of depth images being a depth image of the object captured from a respective viewpoint, and the color image manager <NUM> receives color image data <NUM> representing a plurality of color images, each of the plurality of color images (a) being a color image of the object captured from a respective viewpoint and (b) having a plurality of macroblocks, each of the macroblocks corresponding to a respective region of the color image.

At <NUM>, the depth image identification manager <NUM> identifies, for each of the plurality of color images, a depth image of the plurality of depth images associated with that color image. In some implementations, the associated depth image is chosen to be one with values of view parameters closest to those of the color image.

At <NUM>, the blend weight manager <NUM>, for each pixel of the depth image, determines a position of a surface of the object based on a depth value of the depth image associated with that pixel.

At <NUM>, the blend weight manager <NUM>, for each pixel of the depth image, generates a blend weight corresponding to the determined position, the blend weight being based on a similarity between a surface normal of the depth image at the position and the viewpoint from which the depth image was captured. A blend weight is computed for each color image based on the similarity between the surface normal and the direction from the surface point towards the color image viewpoint. These blend weights are then normalized (e.g. so they sum to unity), and the normalized blend weight associated with the current color image whose ΔQP value is computed is kept.

At <NUM>, the quantization manager <NUM>, for each pixel of the depth image, generates a value of a delta quantization parameter (ΔQP) based on the blend weight. As discussed above, each such value of ΔQP is stored in a pixel of a framebuffer.

At <NUM>, the macroblock manager <NUM>, for each pixel of the depth image, determines a macroblock of the plurality of macroblocks of the color image corresponding to the position, the macroblock being associated with the value of ΔQP for the pixel. As discussed above, the value of ΔQP for a macroblock is an aggregate of the values of ΔQP over all pixels in a tile of the framebuffer corresponding to the macroblock.

At <NUM>, the compression manager <NUM> performs a compression operation on that color image, a compression ratio of each of the plurality of macroblocks of that color image being based on the value of ΔQP with which the macroblock is associated.

<FIG> is a diagram that illustrates an example geometry <NUM> for generating blend weights for a position <NUM> of the surface <NUM> of a depth image. <FIG> illustrates a pair of color image viewpoints <NUM> and <NUM> with respective orientations <NUM> and <NUM> as well as the viewpoint <NUM> of the depth image with orientation <NUM>.

The point <NUM> is, as discussed above, determined from the depth value of the depth map corresponding to a pixel of the depth map. That is, the determination of the point <NUM> of the surface <NUM> is repeated for each pixel of the depth image and accordingly over the surface <NUM>.

Also shown in <FIG> is a normal <NUM> to the surface <NUM> at the point <NUM>. Denote the direction of the normal as n, the orientation <NUM> as c<NUM> and the orientation <NUM> as c<NUM>. The weight corresponding to the color image with viewpoint <NUM> is given by w<NUM> = (c<NUM> · n)e and the weight corresponding to the color image with viewpoint <NUM> is given by w<NUM> = (c<NUM> · n)e, where e is an exponent. In some implementations, the exponent e is specified. In some implementations, the exponent e is determined empirically based on target compression ratios.

As shown in <FIG>, the point <NUM> is better seen from viewpoint <NUM> than from viewpoint <NUM>. Accordingly, w<NUM> > w<NUM>. The color near the point <NUM> is kept at a high quality in the color image having the viewpoint <NUM>. Further, when these color images are received at a receiver and decompressed, the color images will be combined according to the weights w<NUM> and w<NUM>. For example, the normalized blend weights are <MAT> <MAT> In some implementations, when there are more than two color views, then the denominators of the normalized blend weights are the sum of the blend weights over the color views. The normalized blend weight <MAT> is then used to compute the value of ΔQP.

In some implementations, the normal <NUM> is computed according to the depth values in a neighborhood <NUM> of the point <NUM>. In some implementations, the neighborhood <NUM> includes a grid of pixels including the pixel corresponding to the point <NUM>. In some implementations, the grid is <NUM> X <NUM>.

<FIG> is a diagram that illustrates an example geometry <NUM> for generating blend weights for a position <NUM> of the surface <NUM> of a depth image. <FIG> illustrates the pair of color image viewpoints <NUM> and <NUM> with respective orientations <NUM> and <NUM> as well as the viewpoint <NUM> of the depth image with orientation <NUM>.

The point <NUM> is, as discussed above, determined from the depth value of the depth map corresponding to another pixel of the depth map. That is, the determination of the point <NUM> of the surface <NUM> is repeated for each pixel of the depth image and accordingly over the surface <NUM>.

As shown in <FIG>, the point <NUM> is better seen from viewpoint <NUM> than from viewpoint <NUM>. Accordingly, w<NUM> > w<NUM>. The color near the point <NUM> is kept at a high quality in the color image having the viewpoint <NUM>. Further, when these color images are received at a receiver and decompressed, the color images will be combined according to the weights w<NUM> and w<NUM>.

<FIG> is a diagram that illustrates an example that illustrates example projections between surface points (e.g., points <NUM> and <NUM>) and macroblocks <NUM> of a color image associated with the depth image (e.g., depth image <NUM> having orientation <NUM>). Note that each macroblock <NUM> is mapped from multiple pixels of the depth image so that each macroblock <NUM> contains a portion of the depth image.

Each macroblock <NUM> is associated with a value of ΔQP, which in turn is determined from the respective blend weight of each pixel of the depth image. In some implementations, the value of ΔQP for a macroblock <NUM> depends on the values of the weights for all pixels mapped to the macroblock <NUM>. In some implementations, the value of ΔQP for a macroblock <NUM> is based on a mean of the weights of the pixels mapped to the macroblock <NUM>. In some implementations, the value of ΔQP for a macroblock <NUM> is based on a minimum of the weights of the pixels mapped to the macroblock <NUM>. In some implementations, the value of ΔQP for a macroblock <NUM> is based on a maximum of the weights of the pixels mapped to the macroblock <NUM>. In some implementations, the value of ΔQP for a macroblock <NUM> is based on a median of the weights of the pixels mapped to the macroblock <NUM>.

In some implementations, the pixels of the depth or color image include identifiers identifying whether the portion of the depth image containing the pixels is associated with a region of importance (e.g., a face of a person). In such an implementation, the value of ΔQP is decreased for macroblocks <NUM> containing pixels having such an identifier. Conversely, in some implementations, the value of ΔQP is increased for macroblocks <NUM> containing pixels not having such an identifier.

<FIG> illustrates an example of a generic computer device <NUM> and a generic mobile computer device <NUM>, which may be used with the techniques described here.

As shown in <FIG>, computing device <NUM> is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device <NUM> is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, and other similar computing devices.

Thus, for example, expansion memory <NUM> may be provided as a security module for device <NUM>, and may be programmed with instructions that permit secure use of device <NUM>.

Returning to <FIG>, in some implementations, the memory <NUM> can be any type of memory such as a random-access memory, a disk drive memory, flash memory, and/or so forth. In some implementations, the memory <NUM> can be implemented as more than one memory component (e.g., more than one RAM component or disk drive memory) associated with the components of the compression computer <NUM>. In some implementations, the memory <NUM> can be a database memory. In some implementations, the memory <NUM> can be, or can include, a non-local memory. For example, the memory <NUM> can be, or can include, a memory shared by multiple devices (not shown). In some implementations, the memory <NUM> can be associated with a server device (not shown) within a network and configured to serve the components of the compression computer <NUM>.

The components (e.g., modules, processing units <NUM>) of the compression computer <NUM> can be configured to operate based on one or more platforms (e.g., one or more similar or different platforms) that can include one or more types of hardware, software, firmware, operating systems, runtime libraries, and/or so forth. In some implementations, the components of the compression computer <NUM> can be configured to operate within a cluster of devices (e.g., a server farm). In such an implementation, the functionality and processing of the components of the compression computer <NUM> can be distributed to several devices of the cluster of devices.

The components of the computer <NUM> can be, or can include, any type of hardware and/or software configured to process attributes. In some implementations, one or more portions of the components shown in the components of the computer <NUM> in <FIG> can be, or can include, a hardware-based module (e.g., a digital signal processor (DSP), a field programmable gate array (FPGA), a memory), a firmware module, and/or a software-based module (e.g., a module of computer code, a set of computer-readable instructions that can be executed at a computer). For example, in some implementations, one or more portions of the components of the computer <NUM> can be, or can include, a software module configured for execution by at least one processor (not shown). In some implementations, the functionality of the components can be included in different modules and/or different components than those shown in <FIG>.

Although not shown, in some implementations, the components of the computer <NUM> (or portions thereof) can be configured to operate within, for example, a data center (e.g., a cloud computing environment), a computer system, one or more server/host devices, and/or so forth. In some implementations, the components of the computer <NUM> (or portions thereof) can be configured to operate within a network. Thus, the components of the computer <NUM> (or portions thereof) can be configured to function within various types of network environments that can include one or more devices and/or one or more server devices. For example, a network can be, or can include, a local area network (LAN), a wide area network (WAN), and/or so forth. The network can be, or can include, a wireless network and/or wireless network implemented using, for example, gateway devices, bridges, switches, and/or so forth. The network can include one or more segments and/or can have portions based on various protocols such as Internet Protocol (IP) and/or a proprietary protocol. The network can include at least a portion of the Internet.

In some embodiments, one or more of the components of the computer <NUM> can be, or can include, processors configured to process instructions stored in a memory. For example, the depth image manager <NUM> (and/or a portion thereof), the color image manager <NUM> (and/or a portion thereof), the depth image identification manager <NUM> (and/or a portion thereof), the blend weight manager <NUM> (and/or a portion thereof), the QP manager <NUM> (and/or a portion thereof), the macroblock manager <NUM> (and/or a portion thereof), and the compression manager <NUM> (and/or a portion thereof) can be a combination of a processor and a memory configured to execute instructions related to a process to implement one or more functions.

A number of embodiments have been described. Nevertheless, it will be understood that various modifications may be made within the scope of the appended claims, which define the invention.

It will also be understood that when an element is referred to as being on, connected to, electrically connected to, coupled to, or electrically coupled to another element, it may be directly on, connected or coupled to the other element, or one or more intervening elements may be present. In contrast, when an element is referred to as being directly on, directly connected to or directly coupled to another element, there are no intervening elements present. Although the terms directly on, directly connected to, or directly coupled to may not be used throughout the detailed description, elements that are shown as being directly on, directly connected or directly coupled can be referred to as such.

While certain features of the described implementations have been illustrated as described herein, many modifications, substitutions, changes and equivalents will now occur to those skilled in the art. It should be understood that they have been presented by way of example only, not limitation, and various changes in form and details may be made. Any portion of the apparatus and/or methods described herein may be combined in any combination, except mutually exclusive combinations. The implementations described herein can include various combinations and/or sub-combinations of the functions, components and/or features of the different implementations described.

Claim 1:
A method, comprising:
receiving (i) depth image data (<NUM>) representing a plurality of depth images (<NUM>(<NUM>) - <NUM>(N)) of an object, each of the plurality of depth images (<NUM>(<NUM>) - <NUM>(N)) being a depth image of the object captured from a respective viewpoint (<NUM>, <NUM>, <NUM>), and (ii) color image data (<NUM>) representing a plurality of color images (<NUM>(<NUM>) - <NUM>(M)), each of the plurality of color images (<NUM>(<NUM>) - <NUM>(M)) (a) being a color image of the object captured from a respective viewpoint (<NUM>, <NUM>, <NUM>) and (b) having a plurality of macroblocks (<NUM>), each of the macroblocks (<NUM>) corresponding to a respective region of the color image;
for a color image from the plurality of color images (<NUM>(<NUM>) - <NUM>(M)):
identifying a depth image of the plurality of depth images (<NUM>(<NUM>) - <NUM>(N)) associated with the color image;
characterized in that
the method further comprises:
for a set of pixels of the depth image, wherein the set of pixels comprises all pixels of the depth image mapped to a macroblock (<NUM>) of the associated color image, each pixel of the set of pixels being associated with a position (<NUM>;<NUM>) of a point on a surface (<NUM>) of the object based on a depth value of the depth image associated with that pixel:
generating, for each pixel of the set of pixels, a respective blend weight for that pixel based on a surface normal (<NUM>;<NUM>) of the depth image at the corresponding position (<NUM>;<NUM>) and the viewpoint of the color image;
determining a value of a delta quantization parameter (ΔQP) associated with the macroblock (<NUM>), wherein the value of the delta quantization parameter (ΔQP) depends on the values of the blend weights for all pixels in the set of pixels; and
performing a compression operation on the color image, a compression ratio of each of the plurality of macroblocks of the color image being based on the value of the delta quantization parameter (ΔQP) with which the macroblock (<NUM>) is associated.