Patent Description:
Personal display devices make it possible to provide image content to a viewer in applications where the use of conventional display screens would be an encumbrance. Head-mounted display (HMD) devices (also referred to as near-eye display (NED) or near-to-eye (NTE) devices), such as display goggles, are being considered as a useful type of wearable personal display device usable in a variety of fields, with applications ranging from military, medical, dental, industrial, and game presentation, among others. For many of these applications it can be advantageous to have a three-dimensional (3D) or stereoscopic display. However, stereoscopic displays conventionally suffer from mismatched convergence and accommodation cues. This conflict hinders visual performance and causes visual fatigue.

Multifocal displays (MFDs) are one approach that has attempted to address the convergence-accommodation conflict. MFDs typically use rapid temporal (i.e. time multiplexed) and focal modulation of a series of <NUM>-dimensional (2D) images to render 3D scenes. This series of images is typically focused at parallel planes positioned at different, discrete distances from the viewer. The number of focal planes directly affects the viewers' eye accommodation and 3D perception quality of a displayed scene.

Usually, in conventional MFDs the series of <NUM>-dimensional (2D) images is provided from an image generator to an image display controller for displaying the images at different focal planes. In the absence of any transmission errors, where the images are not exposed to any kind of error sources between the image generator, i.e. the source of the images and the image display controller, those images can be displayed properly. If, however, the transmission channel is imperfect, the images may be corrupted or lost, and the image quality will suffer substantially.

Thus, there is a need for improved multifocal display devices and methods allowing a better handling of image sequences affected by transmission errors.

<CIT> describes a method for transmitting video images. In the method, video frames are divided into slices where the slice layer is the lowest possible level that allows resynchronization to the data stream in case of transmission errors. The method is intended to reduce the effect of errors in packet transmission on the quality of a video signal. The method comprises dividing the frame slices into bins temporarily, before forming packets from the slices. Each slice of one frame is put into a different bin, and different slices of each consecutive frames are put into a bin. In case of nine slices this means that there are nine bins B0-B8 each of them containing a differently positioned slice from maximum nine following frames T0-T8.

<CIT> describes a multifocal display for rendering a 3D scene as a series of 2D images. In one aspect, the multifocal display includes a display, an optical imaging system, a refractive focus actuator and a controller. The display renders the 2D images. The optical imaging system is image-side telecentric and creates an image of the display. The refractive focus actuator is positioned at the pupil of the optical imaging system. Thus, adjusting the refractive focus actuator alters a location of the image of the display but does not significantly alter a size of the image. The controller coordinates adjustment of the refractive focus actuator with rendering of the 2D images on the display. The waveform driving the focus actuator is preferably designed to reduce ringing and jitter effects.

Embodiments of the invention are defined by the features of the independent claims, and further advantageous implementations of the embodiments by the features of the dependent claims.

Generally, embodiments of the invention are based on the idea to apply an error concealment scheme to the images in a time-multiplexed MFD system. Once the error concealment scheme has been applied, any further conventional error concealment method can be applied. According to embodiments of the invention the error concealment is implemented by partitioning the image area for each focal plane and combining the image parts corresponding to all focal planes (depths) into one composite image frame to be transmitted. The partitioning may be arbitrary and based on several aspects such as the depth maps. In this way, each transmitted image contains image parts coming from (preferably) all different focal plane images. In other words, each focal plane image is distributed across several transmitted image frames.

As used herein, when referring to a first image and a second image which have the same size and shape, the statement that a portion of the first image and a portion of the second image are "corresponding portions" or that they "correspond to each other" means that the two image portions overlap each other when the two images are stacked atop each other. In other words, corresponding portions of different images are identical in size, shape and position within the respective image.

According to a first aspect the invention relates to an image reception device, e.g. a display controller of a multifocal display system, for receiving a set of composite images from the image transmission device according to the first aspect of the invention. The image reception device comprises features as set out in claim <NUM>.

As used herein, an available image portion of a primary image is an image portion that is contained in one of the composite images available to the image reception device. An unavailable (or missing) image portion is an image portion that is not contained in any of the available composite images. Any missing portion of a primary image can be filled with image content obtained by inter- or extrapolation. The inter- or extrapolation is based on available portions of one or more of the other N-<NUM> primary images. Additionally or alternatively, the inter- or extrapolation may be based on available portions of the primary image (in particular, portions located near the missing portion in the primary image). In particular, if the N primary images have the same size and shape, the inter- or extrapolation may be based on one or more available portions that have the same position as the missing portion, i.e. based on one or more available portions that correspond to the missing portion. Note that image portions of different primary images which are located at the same position (e.g., upper left corner) will generally show the same part of a captured scene and will differ from each other mainly in resolution or blurriness on account of the fact that they are associated with different focus distances (i.e. with different depths).

According to a second aspect the invention relates to a corresponding image reception method for receiving a set of composite images from an image transmission device, as set out in claim <NUM>.

The method according to the second aspect of the invention can be performed by the image reception device according to the first aspect of the invention. Further features of the method according to the second aspect of the invention result directly from the functionality of the image reception device according to the first aspect of the invention and its different implementation forms described above and below.

According to a third aspect the invention relates to a multifocal display system for displaying a set of N primary images, wherein the multifocal display system comprises an image transmission device and an image reception device according to the first aspect of the invention, wherein the multifocal display system is configured to display the set of N primary images reconstructed by the image reception device. The multifocal display system could be implemented in a near eye three-dimensional (3D) display device.

In a further possible implementation form of the third aspect, the multifocal display system is configured to display the reconstructed primary images one after the other. Thus a time-multiplexing scheme can be implemented.

In a further possible implementation form of the third aspect, the multifocal display system is configured to display each of the reconstructed primary images with an optical power that conforms to the respective focus distance thereof.

More specifically, according to the third aspect, the image transmission device is configured to act as an image source of a multifocal display system for transmitting a set of N composite images based on a set of N primary images to an image reception device, in particular a display controller of the multifocal display system with N greater than or equal <NUM>. The N primary images are 2D images of a same 3D video frame, and each of the N primary images has a different focus distance associated with it. The image transmission device comprises: processing circuitry configured to partition each of the N primary images into a plurality of image portions and to generate the N composite images by placing each image portion of each of the N primary images into one of the N composite images such that each of the N composite images comprises image portions from two or more of the N primary images; and a communication interface configured to transmit the N composite images to the image reception device.

In other words, the content of the N primary images is distributed over the N composite images. When no composite image is lost or corrupted, the set of composite images available to the image reception device will comprise the N composite images provided by the image transmission device (and thus the whole image content of the N primary images). The image reception device can then reconstruct the N primary images from the N available composite images exactly (i.e. with no loss in image quality). When one or more composite images are lost or corrupted (e.g., in the course of transmitting them to the image reception device), these lost or corrupted composite images will not be available for reconstructing the primary images. In other words, they are not included in the set of composite images available to the image reception device. In this case, i.e. when the set of composite images available to the image reception device comprises one or more but not all of the N composite images send by the image transmission device, the image reception device can reconstruct the N primary images from the set of available composite images in an approximate manner, e.g., by inter- or extrapolation.

Thus, embodiments of the invention provide an error concealment scheme, where images are modified in such a way that when the image sequence is sent, artifacts in the rendered image that are caused by transmission channel errors will be less noticeable. Advantageously, compared to conventional schemes a much smaller artifact can be observed when an error corrupts one of the images in the sequence.

In an embodiment, the image portions of the N primary images are placed into the composite images such that each composite image comprises image portions from all the N primary images. This minimizes the loss in image quality when one of the composite images is lost or corrupted.

In an embodiment, the N primary images are identical in size and shape. This facilitates transforming the N primary images into the N composite images and transforming the N composite images back into the N primary images.

In a further possible implementation form of the third aspect, the processing circuitry of the image transmission device is configured to place any two image portions of any one of the N primary images into different composite images if the two image portions are adjoining image portions. In other words, adjoining image portions of a primary image can be placed into different composite images. Two image portions are considered adjoining if they share a common boundary. Two image portions which share merely a point (e.g., a corner point) are not considered adjoining. Thus, if one the two image portions is lost or corrupted (e.g., due to loss or corruption of the composite frame which contains this image portion), the other image portion can be reconstructed by inter- or extrapolation of neighboring image portions contained in composite frames that have not been lost.

In a further possible implementation form of the third aspect, the communication interface of the image transmission device is configured to transmit the N composite images in a predefined sequence to the image reception device. Advantageously, this allows the image reception device to identify each composite image based on the position of that composite image in the received sequence of composite images, and no further information for distinguishing the composite images from each other needs to be provided to the image reception device. In an embodiment, the N primary images are images of a video frame, the video frame is an element of a sequence of video frames, and the predefined sequence is the same in every frame of the sequence of video frames.

In a further possible implementation form of the third aspect, the processing circuitry of the image transmission device is configured to place any two corresponding image portions of any two of the N primary images into different composite images. This facilitates reconstruction of the primary images by the image reception device when the set of composite images available to the image reception device is incomplete (i.e. when not all of the N composite images have been received by the image reception device).

In a further possible implementation form of the third aspect, the N primary images are identical in size and shape, wherein the processing circuitry is configured to partition the N primary images into image portions in the same manner. In other words, the geometrical layout of image portions can be the same in each of the N primary images. Each image portion of each of the N composite images can thus be filled with the corresponding image portion of one of the N primary images.

In a further possible implementation form of the third aspect, the processing circuitry of the image transmission device is configured to place each image portion of each of the primary images at the same position within the composite image as in the primary image. Thus a simple and efficient transformation from primary to composite images is provided.

In a further possible implementation form of the third aspect, the image portions of the N primary images have a rectangular or quadratic shape.

In a further possible implementation form of the third aspect, the image portions of the N primary images are pixels.

In a further possible implementation form of the third aspect, the N primary images are images of a video frame. It is understood that the video frame is part of a sequence of video frames. The sequence of video frames provides an animated picture, i.e. a moving picture.

According to a fourth aspect the invention relates to a computer program product comprising program code for performing the method according to the second aspect when executed on a computer.

In the following identical reference signs refer to identical or at least functionally equivalent features.

In the following description, reference is made to the accompanying figures, which form part of the disclosure, and which show, by way of illustration, specific aspects of embodiments of the invention or specific aspects in which embodiments of the invention may be used. It is understood that embodiments of the invention may be used in other aspects and comprise structural or logical changes not depicted in the figures. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the invention is defined by the appended claims.

<FIG> shows an example of a multifocal display system <NUM> according to an embodiment comprising an image transmission device <NUM> according to an embodiment and an image reception device <NUM> according to an embodiment. As will be described in more detail further below, the multifocal display system <NUM> is configured to display a set of N reconstructed primary images. The multifocal display system <NUM> could be implemented as part of a near eye three-dimensional (3D) display device. In an embodiment, the N primary images are images of a video frame. It is understood that the video frame is part of a sequence of video frames. The sequence of video frames provides an animated picture, i.e. a moving picture.

In an embodiment, the multifocal display system <NUM> is configured to display the reconstructed primary images one after the other. Thus, the multifocal display system <NUM> can implement a time-multiplexing image display scheme.

As illustrated in <FIG>, in an embodiment, the multifocal display system <NUM> is configured to display each of the reconstructed primary images with an optical power that conforms to the respective focus distance. By way of example, an image sequence I<NUM>, I<NUM>, I<NUM>, I<NUM>, which corresponds to a certain depth sequence represented by a staircase function (of optical power), can be sequentially displayed within a time period of <NUM>/<NUM>.

In an embodiment, the multifocal display system <NUM> according to an embodiment comprising the image transmission device <NUM> according to an embodiment and the image reception device <NUM> according to an embodiment can be implemented as illustrated in <FIG>. The multifocal display system <NUM> shown in <FIG> comprises the image transmission device <NUM> in the form of a personal computer <NUM>, the image reception device <NUM> in the form of a display controller <NUM> as well as two optical branches for generating and displaying images. In the embodiment shown in <FIG>, respective displays 105a, 105b comprise LED drivers 107a, 107b, RGB units 109a, 109b, collimators 111a, 111b, DMDs 115a, 115b, and DMD controllers 113a, 113b. Via optical elements 117a, 117b the respective images provided by the displays 105a, 105b are provided to respective focal modulator elements 119a, 119b. In the embodiment shown in <FIG>, the respective focal modulator elements 119a, 119b comprise a focus tunable lens 121a, 121b as well as a lens controller 123a, 123b. Via further optical elements <NUM>, 125b the respective images are provided from the focal modulator elements 119a, 119b to the right and left eye of a user. As will be appreciated, the displays 105a and 105b can be responsible for generating the images, whereas the focal modulator elements 119a and 119b are responsible for generating the optical power related to the output images provided by the displays 105a and 105b.

As will be described in more detail further below, the image transmission device <NUM>, i.e. the image source of the multifocal display system <NUM> is configured to transmit a set of N composite images based on a set of N primary images via a potentially error prone communication channel to the image reception device <NUM>, e.g. a display controller <NUM> of the multifocal display system <NUM> with N greater than or equal <NUM>. Each of the N primary images has a focus distance associated with it. The image reception device <NUM>, e.g. the display controller <NUM> of the multifocal display system <NUM> is configured to receive a set of composite images from the image transmission device <NUM>.

In order to provide a better understanding of the invention <FIG> illustrates a conventional time-multiplexed multifocal display system <NUM> with an image generator <NUM> and a display controller <NUM>. By way of example, a sequence of four focal-plane images I<NUM>, I<NUM>, I<NUM>, I<NUM> which corresponds to N=<NUM> depths is sent from the image generator <NUM> to the display controller <NUM>. Each of the four images I<NUM>, I<NUM>, I<NUM>, I<NUM> will normally comprise some out-of-focus (or blurred or diffuse) regions, which are regions with depths different from the depth of the image in question. Differently put, if a region in one of the images (e.g., in I<NUM>) is focused (i.e. the region is filled with the correct depth), the same geometric region in each of the other three images (e.g., in I<NUM>, I<NUM>, I<NUM>) will be out-of-focus (i.e. blurred or diffuse). Embodiments of the invention make advantageous use of this finding. In the exemplary scenario illustrated in <FIG>, image I<NUM> is lost due to an error in the transmission channel. This results in a severe artifact when displayed by the display controller <NUM> of the conventional multifocal display system <NUM>, since an image which corresponds to a certain depth is completely missing.

In order to overcome this limitation, as can be taken from <FIG>, the image transmission device <NUM> of the multifocal display system <NUM> comprises processing circuitry, in particular one or more processors 101a configured to partition each of the N primary images into a plurality of image portions and to generate the N composite images by placing each image portion of each of the N primary images into one of the N composite images such that each of the N composite images comprises image portions from two or more of the N primary images. In an embodiment, the plurality of image portions of the N primary images can have a rectangular or quadratic shape. In an embodiment, a least some of the plurality of image portions of the N primary images are provided by pixels. Moreover, the image transmission device <NUM> comprises a communication interface 101b configured to transmit the N composite images to the image reception device <NUM>.

Likewise, the image reception device <NUM> comprises a communication interface 103b configured to receive the set of composite images from the image transmission device <NUM>, wherein the composite images are based on the set of N primary images and wherein each of the composite images received by the image reception device <NUM> comprises image portions from two or more of the N primary images. Moreover, the image reception device <NUM> comprises processing circuitry 103a, e.g. one or more processors 103a configured to reconstruct the N primary images from the set of composite images by placing each image portion of each of the composite images into one of the N primary images.

<FIG> illustrates the same scenario as in <FIG> in the context of the multifocal display system according to an embodiment comprising the image transmission device <NUM>, e.g. the image generator <NUM> according to an embodiment and the image reception device <NUM>, e.g. the image display controller <NUM> according to an embodiment. In the exemplary scenario of <FIG> the N=<NUM> primary images I<NUM>, I<NUM>, I<NUM>, is have been processed by the processing circuitry 101a (referred to as image sequence arrangement processor in <FIG>) of the image transmission device <NUM> to generate a sequence of N composite images denoted by A, B, C, D. When the same error as in the scenario of <FIG> happens in the scenario of <FIG>, the processing circuitry 103a (referred to as image sequence inverse arrangement in <FIG>) can decode the remaining received images into a sequence of <NUM> reconstructed primary images. As will be appreciated, these reconstructed primary images will likely contain error due to a missing part of the images, but the perceived quality will be much better in comparison with the scenario illustrated in <FIG>.

Thus, as illustrated in <FIG>, the image transmission device <NUM> is configured to distribute the content of the N primary images over the N composite images. When no composite image is lost or corrupted during transmission, the set of composite images available to the image reception device <NUM> will comprise the N composite images provided by the image transmission device <NUM> (and thus the whole image content of the N primary images). The image reception device <NUM> can then reconstruct the N primary images from the N available composite images exactly (i.e. with no loss in image quality). When one or more composite images are lost or corrupted (e.g., in the course of transmitting them to the image reception device <NUM>), these lost or corrupted composite images will not be available for reconstructing the primary images. In this case, i.e. when the set of composite images available to the image reception device <NUM> comprises one or more but not all of the N composite images send by the image transmission device <NUM>, the image reception device <NUM> can reconstruct the N primary images from the set of available composite images in an approximate manner, e.g., by inter- or extrapolation.

Thus, in an embodiment, the processing circuitry 103a of the image reception device <NUM> is configured to reconstruct the N primary images in an approximate manner from the set of composite images provided by the image transmission device <NUM>, when the set of composite images includes one or more but less than N composite images, i.e. in case not all N composite images generated and transmitted by the image transmission device <NUM> have been received by the image reception device <NUM> or in case those composite images have been corrupted.

In an embodiment, the processing circuitry 103a of the image reception device <NUM> is configured to reconstruct the N primary images in an approximate manner by reconstructing a missing image portion of one of the N primary images based on one or more available image portions. As used herein, an available image portion of a primary image is an image portion that is contained in one of the composite images available to the image reception device <NUM>. An unavailable (or missing) image portion is an image portion that is not contained in any of the available composite images. Any missing portion of a primary image can be filled by the processing circuitry 103a of the image reception device <NUM> with image content obtained by inter- or extrapolation. The inter- or extrapolation may be based on available portions of the primary image (in particular, portions located near the missing portion in the primary image). Alternatively or additionally, the inter- or extrapolation may be based on available portions of one or more of the other N-<NUM> primary images. In particular, if the N primary images have the same size and shape, the inter- or extrapolation may be based on one or more available portions that have the same position as the missing portion, i.e. based on one or more available portions that correspond to the missing portion. Note that image portions of different primary images which are located at the same position (e.g., upper left corner) will generally show the same part of a captured scene and will differ from each other mainly in resolution or blurriness on account of the fact that they are associated with different focus distances (i.e. with different depths).

In an embodiment, the processing circuitry 101a of the image transmission device <NUM> is configured to place the image portions of the N primary images into the composite images such that each composite image comprises image portions from all the N primary images. In an embodiment, the N primary images are identical in size and shape.

In an embodiment, the processing circuitry 101a of the image transmission device <NUM> is configured to place any two image portions of any one of the N primary images into different composite images if the two image portions are adjoining image portions. In other words, adjoining image portions of a primary image can be placed into different composite images. Two image portions are considered adjoining if they share a common boundary. Two image portions which share merely a point (e.g., a corner point) are not considered adjoining. Thus, if one the two image portions is lost or corrupted (e.g., due to loss or corruption of the composite frame which contains this image portion), the other image portion can be reconstructed by the image reception device <NUM> by inter- or extrapolation of neighboring image portions contained in composite frames that have not been lost or are not corrupted.

In an embodiment, the communication interface 101b of the image transmission device <NUM> is configured to transmit the N composite images in a predefined sequence to the image reception device <NUM>. This allows the image reception device to identify each composite image based on the position of that composite image in the received sequence of composite images, and no further information for distinguishing the composite images from each other needs to be provided to the image reception device <NUM>. In an embodiment, the N primary images are images of a video frame, the video frame is an element of a sequence of video frames, and the predefined sequence is the same in every frame of the sequence of video frames.

In an embodiment, the processing circuitry 101a of the image transmission device <NUM> is configured to place any two corresponding image portions of any two of the N primary images into different composite images. This facilitates reconstruction of the primary images by the image reception device <NUM> when the set of composite images available to the image reception device <NUM> is incomplete (i.e. when not all of the N composite images have been received by the image reception device <NUM>).

In an embodiment, the N primary images are identical in size and shape, wherein the processing circuitry 101a of the image transmission device <NUM> is configured to partition the N primary images into image portions in the same manner. In other words, the geometrical layout of image portions can be the same in each of the N primary images. Each image portion of each of the N composite images can thus be filled with the corresponding image portion of one of the N primary images.

In an embodiment, the processing circuitry 101a of the image transmission device <NUM> is configured to place each image portion of each of the primary images at the same position within the composite image as in the primary image, which allows a simple and efficient transformation from primary to composite images.

<FIG> and <FIG> illustrate two exemplary primary images I<NUM>, I<NUM> as well as respective portions Ao, Bo, Co, Do and A<NUM>, B<NUM>, C<NUM>, D<NUM> of the composite images A, B, C, D generated by the image transmission device <NUM> and possibly received by the image reception device <NUM>. Given N=<NUM> primary images I<NUM>, I<NUM>, I<NUM>, is in a sequence, a sequence of composite images A, B, C, D is generated by distributing the primary images I<NUM>, I<NUM>, I<NUM>, is across the composite images A, B, C, D such that each of the primary images is spread across two or more (preferably, all) of the composite images A, B, C, D. For example, as shown in <FIG>, each of the composite images A, B, C, D comprises a first portion Ao, Bo, Co, Do, which is a collection of parts or blocks of the first primary image Io as indicated by the respective texture. The same applies to the second original image I<NUM> as shown in <FIG>. All parts of the composite images are then combined to form a sequence of composite images A, B, C, D as shown in <FIG> and <FIG>. In this way, when, for instance, the composite image B is lost or corrupted during the transmission, the sequence of primary images reconstructed by the image reception device <NUM> on the basis of the composite images A, C and D can limit the artifact caused by the loss or corruption of composite image B.

<FIG> is a flow diagram showing an example of an image transmission method for transmitting a set of N composite images based on a set of N primary images from the image transmission device <NUM> of the multifocal display system <NUM> to the image reception device <NUM> thereof according to an embodiment. The image transmission method <NUM> comprises the steps of: partitioning <NUM> each of the N primary images into image portions; generating <NUM> the N composite images by placing each image portion of each of the N primary images into one of the N composite images such that each of the N composite images comprises image portions from two or more of the N primary images; and transmitting <NUM> the N composite images to the image reception device <NUM>.

<FIG> is a flow diagram showing an example of a corresponding image reception method <NUM> for receiving the set of composite images from the image transmission device <NUM> at the image reception device <NUM> of the multifocal display system <NUM> according to an embodiment. The image reception method <NUM> comprises the steps of: receiving <NUM> the set of composite images from the image transmission device <NUM>, wherein the composite images are based on a set of N primary images, wherein N is greater or equal <NUM>, wherein each of the N primary images has a focus distance associated with it and wherein each of the composite images comprises image portions from two or more of the N primary images; and reconstructing <NUM> the N primary images from the set of composite images by placing each image portion of each of the composite images into one of the N primary images.

The person skilled in the art will understand that the "blocks" ("units") of the various figures (method and apparatus) represent or describe functionalities of embodiments of the invention (rather than necessarily individual "units" in hardware or software) and thus describe equally functions or features of apparatus embodiments as well as method embodiments (unit = step).

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus, and method may be implemented in other manners. For example, the described apparatus embodiment is merely exemplary.

Claim 1:
An image reception device (<NUM>) for receiving a set of composite images from an image transmission device (<NUM>), wherein the image reception device (<NUM>) comprises:
a communication interface (103b) configured to receive the set of composite images from the image transmission device (<NUM>), wherein the composite images are based on a set of N primary images, wherein N is greater or equal <NUM>, wherein the N primary images are 2D images of a same 3D video frame and each of the N primary images has a different focus distance associated with it, and wherein each of the composite images comprises image portions from two or more of the N primary images; and
processing circuitry (103a) configured to reconstruct the N primary images from the set of composite images by placing each image portion of each of the composite images into one of the N primary images,
wherein the processing circuitry (103a) is configured to reconstruct the N primary images in an approximate manner from the set of composite images when the set of composite images includes one or more but less than N composite images,
wherein the processing circuitry (<NUM>) is configured to reconstruct the N primary images in an approximate manner by reconstructing a missing image portion of one of the N primary images based on one or more available image portions of one or more of the other N-<NUM> primary images that have the same position as the missing portion in the 3D video frame but are associated with a different focus distance.