Patent Description:
Digital photography and videography are widely used applications and are available in various formats and included in image recording devices such as mobile phones, digital still cameras, and video cameras. The traditional purpose of these image recording devices is to record images with the highest possible fidelity. A new trend, however, is to modify the recoded image with additional visual content that was not present in the original image. This technique of augmenting a recorded image with additional visual content, i.e. with another image, and subsequently displaying the modified image to a viewer, is generally referred to as AR (augmented reality).

A device that produces an AR image requires depth information to augment or overlay the original image with the additional visual content. The depth information provides the device with knowledge about the distance relationship between the observer and different parts of the original image, which is essential for rendering a realistic AR experience.

In a conventional AR device, a depth map imaging unit may thus be provided next to a main camera. This allows overlaying an original image obtained with the main camera with depth information at each point of the image, based on the information provided by the depth mapping unit.

Unfortunately, the performance of existing depth mapping units is not sufficient to achieve satisfactory results. The available depth mapping units provide insufficient resolution, depth range, depth accuracy, frame rate, compactness, ruggedness, cost, or power efficiency, and mostly fall short on multiple of these items at the same time. In particular, the registration of high resolution ("XY" plane, "Z" direction, "t" in terms of frames for a video) depth maps is a big challenge, which can only be achieved with considerable effort, i.e. significant cost, power consumption, and volume of the AR device.

Therefore, alternative ways for achieving satisfactory AR rendering are required.

'<NPL>, proposes using a high-accuracy time-of-flight (TOF) camera to capture a depth map of real-world in real time, in order to resolve incorrect occlusion problems in outdoor AR applications.

<CIT> discloses a range-gated depth camera assembly.

In view of the above-mentioned challenges and disadvantages of the conventional techniques, embodiments of the present invention aim to provide an improved device for generating an AR image (or AR video).

The crucial task to be performed when rendering an AR image is the placement of an image of a virtual object (e.g. a figurine to be added to or replacing the background of) an original image (still image or video frame). A key task is determining which parts of the scene of the original image that should be shaded (occluded) by the virtual object, and vice versa, i.e. which parts of the original image will overlap the virtual object. This task is generally referred to as determining occlusion.

An objective is to provide a device that achieves satisfactory AR image rendering, in particular that handles occlusion, but that does not require a high-performance depth mapping unit. Accordingly, a simpler way for realizing AR image generation is desired. The occlusion should be handled by the device efficiently, without requiring increased cost, power, and/or volume.

The objective is achieved by the embodiments of the invention provided in the enclosed independent claims. Advantageous implementations of these embodiments are defined in the dependent claims.

A first aspect of the invention provides a device for generating an AR image, the device being configured to: obtain a first image of a scene, obtain a second image of an object, obtain a predetermined depth, which relates to a distance from an observer at which the object will appear to be located within the scene in the AR image, capture a third image of only the parts of the scene in front of the predetermined depth, wherein the third image is captured with a range gated imager and has a field of view that is substantially the same as a field of view of the first image, generate occlusion information based on the third image, wherein the occlusion information identifies parts of the first image that are in front of the predetermined depth and parts of the first image that are not in front of the predetermined depth, and overlay the first image and the second image based on the occlusion information, in order to generate the AR image, wherein the occlusion information identifies the parts of the first image that are to be replaced by the second image when overlaying the first image and the second image, in order to generate the AR image.

The "first image", "second image" and "third image" are named like this without implying any order, but solely to distinguish the different images. That means, the device of the first aspect is configured to overlay the image of the scene and the image of the object based on the occlusion information, in order to generate the AR image.

The device does not need a depth mapping unit to provide an accurate measurement of the distance between observer and an object in the AR image at every point of the scene, and e.g. for every frame in a video stream. Thus, the device requires lower cost, power, and volume.

The full depth information is not necessary. Once the device of the first aspect obtains the predetermined depth - also referred to as "depth range" in this disclosure - at which the object will appear in the final AR image, it determines which parts of the first image are located either in front of or behind that predetermined depth. This is a much easier task than obtaining full depth information, and can be well achieved at high resolution, high frame rate and good power efficiency, e.g. by using known image sensing techniques.

Notably, there are different options for implementing the "second image" and the "object" of the second image, respectively, according to embodiments of the invention.

For instance, the "second image" may be a picture of the same resolution as the first image. In this case, for each pixel of the first image, the device may be configured to determine, whether the pixel needs to be replaced by the equivalent pixel of the second image or not in the AR image, using the occlusion information. In this case, the "second image" includes pixels associated with the "object", i.e. there is not a separate object picture. In this case, the device may be considered to perform a simple background swap application.

The "second image" may also be a picture of the same resolution as the first image, in which pixels are marked "opaque" or "transparent" (in particular, the pixels other than the pixels associated with the "object" are marked "transparent"). When overlaying the second image and the first image, the device may in this case be configured to first determine, for each pixel of the first image, if the pixel must be rendered in the AR image because it is in front. Further, the device may also be configured to determine, if the pixel needs to be rendered in the AR image because the corresponding pixel of the second image is labeled "transparent". This approach automatically provides means to position the object laterally (XY) with respect to the first image.

Alternatively, the device could only obtain pixels (and their location relative to each other and/or relative to the pixels of the first image) that make up an "object", i.e. there is a picture of the object. In this case, there is not really a "second image" in addition to the object picture, nor are there any "transparent" pixels. The "second image" in this case can be a lateral XY positioning reference frame defined relative to the first image.

In the following disclosure "second image" and "object" are used without further distinction. To sum up these can be one of the following:.

In an implementation, the device is configured to, in order to generate the AR image: position the object of the second image in the scene of the first image, wherein parts of the scene are occluded by the object and/or the object is occluded by parts of the scene, according to the occlusion information.

That means, in the final AR image, the object will appear to be located within the scene within the predetermined depth. The "depth range" in this disclosure is the predetermined depth, which relates to the distance of an observer to the object. That is, the depth range is just one number. The rendering of the AR image is achieved by respecting the occlusion relationship between the object of the second image and objects of/in the scene of the first image, as determined by the occlusion information.

Generally, the occlusion information can be or include an occlusion map. For instance, for each pixel of the first image, the occlusion map may indicate whether that pixel is behind the predetermined depth or in front of the predetermined depth.

In an implementation of the device, the occlusion information is a binary occlusion map.

In this way, the occlusion information can be provided at high resolution with little data. The binary occlusion map may, for instance, indicated for each pixel whether the pixel is in front of or behind the predetermined depth.

In an implementation of the device, the occlusion map has the same pixel resolution as the third image.

Thus, the device is able to generate high resolution AR images.

In an implementation of the device, the occlusion map has a field of view that is substantially the same as the field of view of the first image.

Accordingly, the device is able to generate a realistic AR image.

In an implementation, the device comprises an imaging device configured to capture the first image and the third image.

In accordance with the invention, the device comprises a range gate imager as a distance imaging unit configured to capture the third image.

The range gated imager may be part of the imaging device. For example, the range gated imager may be configured to: send a light pulse onto the scene, synchronize a shutter, which may e.g. be arranged in front of an image sensor, based on an expected return time of light reflected from the parts of the scene within the depth range, and detect the light that passes the shutter with the image sensor, in order to capture the third image.

With the range gated imager, the required depth information for generating the occlusion information can be efficiently obtained with good accuracy.

In an implementation, the device is further configured to display the generated AR image.

In an implementation of the device, the first image is a frame of a video, and the AR image is a frame of an AR video.

A second aspect of the invention provides a method of generating an augmented reality (AR) image, the method comprising: obtaining a first image of a scene, obtaining a second image of an object, obtaining a predetermined depth which relates to a distance from an observer at which the object will appear to be located within the scene in the AR image, capturing a third image of only the parts of the scene in front of the predetermined depth, wherein the third image is captured with a range gated imager and has a field of view that is substantially the same as a field of view of the first image, generating occlusion information based on the third image, wherein the occlusion information identifies parts of the first image that are in front of the predetermined depth and parts of the first image that are not in front of the predetermined depth, and overlaying the first image and the second image based on the occlusion information, in order to generate the AR image, wherein the occlusion information further identifies the parts of the first image that are to be replaced by the second image when overlaying the first image and the second image, in order to generate the AR image.

In an implementation, the method comprises, in order to generate the AR image: positioning the object of the second image in the scene of the first image, wherein parts of the scene are occluded by the object and/or the object is occluded by parts of the scene, according to the occlusion information.

In an implementation of the method, the occlusion information is a binary occlusion map.

In an implementation of the method, the occlusion map has the same pixel resolution as the third image.

In an implementation of the method, the occlusion map has a field of view that is substantially the same as the field of view of the first image.

In an implementation, the method comprises capturing the first image and the third image.

In an implementation, the method further comprises displaying the generated AR image.

In an implementation of the method, the first image is a frame of a video, and the AR image is a frame of an AR video.

The method of the second aspect and its implementations achieves the same advantages and effects as described above for the device of the first aspect and its respective implementations.

A third aspect of the invention provides a computer program product comprising a program code for carrying out, when implemented on an imaging device, the method according to the second aspect or any of its implementation forms.

The above described aspects and implementations are explained in the following description of embodiments with respect to the enclosed drawings:.

<FIG> shows a device <NUM> according to an embodiment of the invention. In particular, the device <NUM> is configured to generate an AR image <NUM>. Specifically, the device <NUM> overlays different images of one or more scenes and/or objects to generate the AR image <NUM>. The AR image <NUM> is a still image or may be a frame of an AR video.

The device <NUM> is configured to obtain a first image <NUM> of a scene 12a, and to obtain a second image <NUM> of an object 13a. The images <NUM> and <NUM> may have the same pixel resolution. The device <NUM> may be capable to capture the first image <NUM> and/or the second image <NUM> by means of e.g. an imaging unit, like one or more cameras.

The device <NUM> is further configured to obtain a predetermined depth range <NUM>, which is one depth value, and to capture a third image <NUM> of only the parts of the scene 12a in front of this obtained predetermined depth. The device <NUM> is configured to capture the third image <NUM> by means of a range gated imager.

The device <NUM> is further configured to generate occlusion information <NUM> based on the third image <NUM>. The occlusion information <NUM> may be an occlusion map having a pixel resolution of the third image <NUM>. The occlusion information <NUM> may be a binary occlusion map. The occlusion information <NUM> provides information regarding an occlusion relationship between the scene 12a of the first image <NUM> and the object 13a of the second image <NUM>. For instance, it may define parts of the scene 12a, which are occluded by the object 13a, and/or it may define parts of the scene 12a, which occlude the object 13a, when the object 13a of the second image <NUM> is arranged in the scene 12a of the first image <NUM> to form the AR image <NUM>.

The device <NUM> is further configured to overlay the first image <NUM> and the second image <NUM> based on the occlusion information <NUM>, to generate the AR image <NUM>. Thereby, according to the occlusion information <NUM>, parts of the first image <NUM> are replaced by the second image <NUM>. Further, according to the occlusion information <NUM>, parts of the second image <NUM>, particularly pixels of the object <NUM>, may be replaced by pixels of the first image <NUM> (as illustrated in <FIG>).

For generating the AR image <NUM>, it is necessary for the device <NUM> to know a priori information regarding the distance relative to the observer, at which the virtual object 13a is to be placed, before an occlusion determination can be carried out. The origin of this a priori placement distance information is dependent on the specific AR application, and is outside the scope of this disclosure. This distance information correlates to the depth range <NUM> that is obtained by the device <NUM>. The depth range <NUM> may, for example, be provided by to the device <NUM> by a content supplier of the first image <NUM> and/or second image <NUM>.

The depth range <NUM> corresponds to the virtual object 13a placement in the depth extent of the scene 12a. The device <NUM> may further be configured <NUM> to obtain the virtual object 13a placement in the lateral extent of the scene 12a, i.e. it may obtain the vertical/horizontal position, at which the object <NUM> of the second image <NUM> should appear in the scene 12a of the first image <NUM>. Once the virtual object placement in both the depth and lateral extent is obtained by the device <NUM>, the device <NUM> can determine which part of the scene <NUM> a of the first image <NUM> is either in front of, or behind the object 13a in the AR image <NUM>, at the location where the virtual object 13a of the second image <NUM> is to be inserted into the scene 12a of the first image <NUM>. In order to obtain a good AR experience, the occlusion should be determined by the device <NUM> at high resolution and a high video frame rate (in case that the AR image <NUM> is a frame of an AR video).

<FIG> shows a schematic of the device <NUM>, which builds on the device <NUM> shown in <FIG>. Same elements in <FIG> and <FIG> share the same reference signs and function likewise.

As shown in <FIG>, the device <NUM> comprises a distance imaging unit <NUM>, which is configured to image the real-world scene 12a according to the depth range <NUM>, which is the predetermined depth, in order to determine the occlusion information. The device <NUM> may further comprise an image rendering unit <NUM>, which is configured to generate the AR image <NUM> taking as an input: the first visual image <NUM> of the scene 12a, the image <NUM> of the graphical object 13a, and the occlusion information.

The distance imaging unit <NUM> is a range gated imager. That means, the device <NUM> uses active illumination gated imaging to determine the third image <NUM>, in order to determine which part of the scene 12a is in front of the predetermined depth, and which part of the scene 12a is behind the predetermined depth. This technique combines, or optionally may combine a pulsed light source with a global shutter image sensor. By synchronizing the timing of the light pulse and the shutter, the image sensor can be made to have increased sensitivity to objects that are within the predetermined depth from the distance imaging unit <NUM>. For instance, it can be made to have increased sensitivity for objects that are within <NUM> meters distance from the image sensor. This technique may be based on the amount of time that a pulse of light needs to travel from the light emitter towards the object and back to the camera after reflecting off of the object. This technique is well known in literature and is generally referred to as Time-Of-Flight (TOF). TOF applied to active illumination gated imaging is referred to as range gated imaging.

The range of higher sensitivity is typically determined primarily by the width of the light pulse and the trigger signal, and by the relative timing of light pulse and shutter. By comparing images recorded with and without active illumination (i.e. the first image <NUM> and the third image <NUM>), the increased sensitivity of the sensor for objects within a specific depth allows the device <NUM> to obtain the occlusion information <NUM>. The occlusion information <NUM> may identify parts of the image <NUM> that are within the predetermined depth and parts that are not.

Given the speed of light, the device <NUM> may be configured to control the light pulse, shutter and relative timing between them at nanosecond time scales for reasonable levels of depth accuracy to be obtained, typically at least on the order of centimeters. Notably, active illumination gated image systems with such timing capabilities, resolution and frame rate have been reported in literature.

<FIG> shows a flow-diagram of a method <NUM> according to an embodiment of the invention. The method <NUM> of <FIG> can be carried out by the device <NUM>, e.g. as shown in <FIG> or <FIG>.

The method <NUM> comprises: a step <NUM> of obtaining a first image <NUM> of a scene 12a; a step <NUM> of obtaining a second image <NUM> of an object 13a; a step <NUM> of obtaining <NUM> the predetermined depth <NUM>; and a step <NUM> of capturing <NUM> a third image <NUM> of only the parts of the scene 12a within the predetermined depth. There is no particular order in which these steps have to be carried out.

The method further comprises: a step <NUM> of generating occlusion information <NUM> based on the third image <NUM>; and a step <NUM> of overlaying the first image <NUM> and the second image <NUM> based on the occlusion information <NUM>, in order to generate the AR image <NUM>.

Claim 1:
A device (<NUM>) for generating an Augmented Reality, AR, image (<NUM>), the device (<NUM>) being configured to:
- obtain a first image (<NUM>) of a scene (12a),
- obtain a second image (<NUM>) of an object (13a),
- obtain a predetermined depth which relates to a distance from an observer at which the object (13a) will appear to be located within the scene (12a) in the AR image (<NUM>);
characterised by being configured to:
- capture a third image (<NUM>) of only the parts of the scene (12a) in front of the predetermined depth, wherein the third image (<NUM>) is captured with a range gated imager and has a field of view that is substantially the same as a field of view of the first image,
- generate occlusion information (<NUM>) by comparing the first image (<NUM>) and the third image (<NUM>), wherein the occlusion information (<NUM>) identifies parts of the first image (<NUM>) that are in front of the predetermined depth and parts of the first image (<NUM>) that are not in front of the predetermined depth, and
- overlay the first image (<NUM>) and the second image (<NUM>) based on the occlusion information (<NUM>), in order to generate the AR image (<NUM>), wherein the occlusion information (<NUM>) identifies the parts of the first image (<NUM>) that are to be replaced by the second image (<NUM>) when overlaying the first image (<NUM>) and the second image (<NUM>), in order to generate the AR image (<NUM>).