Optical see-through glass type display device and corresponding optical unit

An optical see-through glass type display device comprises: an optical unit having a front correcting surface and a rear correcting surface; an image projector projecting a virtual image; and an image sensor for capturing an ambient scene image. The optical unit is configured to guide light of the ambient scene image coming through the front correcting surface to the image sensor and to guide light of the virtual image so that the light of the virtual image outgoes through the rear correcting surface. A first optical compensation element is located between the optical unit and the image sensor and a second optical compensation element is located between the image projector and the optical unit. The device further comprises a processing module configured to analyze the ambient scene image captured by the image sensor.

TECHNICAL FIELD

The present invention generally relates to an optical see-through glass type display device.

BACKGROUND ART

An optical see-through glass type display device provides a viewer with a virtual image superimposed onto an ambient scene seen through the glass. The virtual image may be projected by a projector and guided into an eye of the viewer via an optical element on the glass. The optical see-through glass type display device may present a stereoscopic virtual image by displaying a left image on the left glass and a right image on the right glass of the device so that the viewer can experience a three-dimensional perception of depth.

In such an optical see-through glass type display device, it is preferable for the viewer to perceive the virtual image in spatial coherence with the ambient scene image even if the device comprises corrective surfaces providing a prescription glasses function in order to present the virtual image on the ambient scene image in proper position and size as well as in proper depth if the virtual image is a stereoscopic virtual image.

EyeTap Personal Imaging (ePI) Lab, a research laboratory from University of Toronto, developed the “EyeTap” technology which combines a display and camera in an eyewear for preserving correspondence between the real world and the virtual world through the use of a flat beam splitter plate. However, this technology is not directed to a situation in which the eyewear comprises prescription glasses.

SUMMARY OF THE INVENTION

According to an aspect of the present invention, an optical see-through glass type display device comprises an optical unit having a front correcting surface and a rear correcting surface; an image projector projecting a virtual image; and an image sensor for capturing an ambient scene image. The optical unit is configured to guide light of the ambient scene image coming through the front correcting surface to the image sensor and to guide light of the virtual image so that the light of the virtual image outgoes through the rear correcting surface. A first optical compensation element is located in a light path between the optical unit and the image sensor and a second optical compensation element is located in a light path between the image projector and the optical unit. The device further comprises a processing module configured to analyze the ambient scene image captured by the image sensor to prepare the virtual image to be provided to the image projector according to the analysis result of the ambient scene image.

DETAILED DESCRIPTION

In the following description, various aspects of an embodiment of the present invention will be described. For the purpose of explanation, specific configurations and details are set forth in order to provide a thorough understanding. However, it will also be apparent to one skilled in the art that the present invention may be implemented without the specific details present herein.

FIG. 1illustrates a see-through glass type display device according to an embodiment of the present invention.

As shown inFIG. 1, a see-through glass type display device100may include a glass plate unit110having a front glass plate112, intermediate glass plate114and rear glass plate116, an image sensor120, an image projector130. The image sensor120is optically connected to the front glass plate112via a first optical light guide element122and the image projector130is optically connected to the intermediate glass plate114via a second optical light guide element132. A first optical compensation element124is located in a light path between the front plate112and the image sensor120, behind an emission end of the front plate112. Also, a second optical compensation element134is located in a light path between the image projector130and the intermediate plate114, ahead of an incident end of the intermediate plate114. Details on the optical compensation elements124and134will be described hereinafter.

The device100may be an eye glasses type device, thus the device100also comprises a bridge (not shown) connecting two glass plate units each other and temple arms (not shown) that will extend respectively over the ears of a viewer to help hold the device100in place. InFIG. 1, only the half components of the device100for the right eye105of a viewer are illustrated for the simplicity of the illustration.

The front glass plate112has a front correcting surface112a(curvature not depicted inFIG. 1) to be located at ambient scene side for correcting near-sight (myopia) or far-sight (hyperopia) of the viewer. The front correcting surface112aof the glass plate112refracts a light150coming from ambient scene when the light comes into the glass plate112.

The glass plate unit110has a separating layer113between the front glass plate112and the intermediate glass plate114. The separating layer113can have a Fresnel structure, for example. The separating layer113has semi-reflective characteristics for reflecting a part of incoming light and for transmitting the other part of the incoming light.

The separating layer113will redirect a light beam coming from ambient scene through the front correcting surface112aof the front glass plate112so that the light beam is laterally propagated within the front glass plate112by total internal reflection (TIR) between the both surfaces of the plate112. The light beam propagated within the glass plate112will travel to the image sensor120via the first optical light guide element122, then be captured on the image sensor120. The separating layer113is also transmissive for a light coming from ambient scene to travel through the separating layer113toward the eye105of the viewer.

The image projector130is configured to project a virtual image. A light beam of the virtual image projected by the image projector130is guided via the second optical light guide element132and then comes into the intermediate glass plate114. The light beam160is laterally propagated within intermediate glass plate114by total internal reflection (TIR) between both surfaces of the plate114. Then, at least a part of the light beam is reflected and directed toward the eye105of the viewer by the separating layer113. As a result of this, the virtual image is presented to the viewer.

The rear glass plate116behind the intermediate glass plate114has a rear correcting surface116ato be located at viewer's eye side for correcting near-sight (myopia) or far-sight (hyperopia) of the viewer. The rear correcting surface refracts116aa light outgoing from the rear glass plate116.

It should be noted that curvatures of the front correcting surface112aof the front glass plate112and the rear correcting surface116aof the rear glass plate116can be predetermined depending on the viewer's vision characteristics, which may be determined in advance through an examination for visual acuity. Also, it should be noted that dimensions, angles and reflection/transmission level of semi-reflection elements of the separating layer113may be defined so that the above described light paths for the ambient scene image light and virtual image light are established in the glass plate unit110. Further, for example, a boundary surface between the front glass plate112and the intermediate glass plate114excluding the area of the separating layer113and a boundary surface between the intermediate glass plate114and the rear glass plate116may be formed with a transparent resin having lower refraction index than that of glass plates112,114and116so that the total internal reflections (TIRs) can be realized within the front glass plate112and the intermediate glass plate114.

The device100further comprises a processing module140connected to the image sensor120and the image projector130. The module140is configured to perform: 1) analysis of ambient scene image captured by the image sensor120for specifying a tracked feature in the ambient scene image, for example; and 2) virtual image overlay for providing a virtual image so that the virtual image is presented on the glass plate unit110at suitable position and in suitable size according to the position and size of the tracked feature in the captured ambient scene image.

FIG. 2is a block diagram of components of the see-through glass type display device according to an embodiment of the present invention. As shown inFIG. 2, the components of the device200comprise an image projector210, a processing module230and an image sensor250. The image projector210and the image sensor250are connected to the module230via wired or wireless connections, respectively.

The image projector210comprises a display215for projecting the virtual image, a controller220for controlling the display215and an optical element225for guiding light from the display215to the optical light guide element132(FIG. 1). An exemplary implementation of the display215can be made by an LCD (Liquid crystal display) and an LED (Light Emitting Diode) RGB light source module.

It should be noted that any other technologies can be employed for implementing the display215.

The module230comprises a receiver235and a memory240. These receiver235and memory240are configured to receive and store images or videos to be projected as the virtual image, which images or videos are stored in any type of external device (not shown) and received from the external device via a wired or wireless connection and also configured to receive and store the captured ambient scene image from the image sensor250.

The module230further comprises a processor245. The processor245performs analysis of ambient scene image stored in the memory240to specify the size and/or position of the tracked feature in the ambient scene image, for example. In an example, the tracked feature may be the viewer's hand outreached in front of the viewer's eyes. The processor245also performs virtual image overlay for providing a virtual image so that the virtual image forming beam is presented on the glass plate unit110at suitable position and in suitable size according to the position and size of the tracked feature in the captured ambient scene image.

In an example, the processor245may determine the size and position of a virtual image to be displayed in the area of the separating layer113in the glass plate unit110according to the position and size of the tracked viewer's hand in the captured ambient scene image so that the virtual image is superimposed on or adjacent to the tracked viewer's hand in the ambient scene image seen through the glass plate unit110and that the virtual image moves on with the viewer's hand. The virtual image can be an image, video, text information or the like received by the receiver235and stored in the memory240. The virtual image to be provided to the image projector210is prepared by the processor245according to the analysis result of the ambient scene image as described above. Data of the virtual image with adjusted size and position is output by the processor245to the image projector210, then the virtual image with adjusted size and position is projected by the image projector210.

As discussed above, the analysis of ambient scene image is performed by the processor245to specify the size and/or position of the tracked feature in the ambient scene image. The analysis of ambient scene image is needed for the calibration of virtual image to be displayed, therefore the analysis process on the processor245and imaging operation on the image sensor250may be deactivated or removed once the calibration is established. In case of deactivation those process and operation may be re-activated when size and/or position of a tracked feature in an ambient scene image should be specified again, for example when an ambient scene image and a feature to be tracked are changed. Removal of the image sensor and imaging operation is adequate for consumer usages not needing scene capture. In this case calibration may be performed in e.g. an optician shop with professional equipment. In this case the complexity and cost of the consumer glasses product may be decreased.

Referring toFIG. 1, virtual image light160projected by the projector130is propagated in the intermediate glass plate114and at least part of the virtual image light160is redirected toward the eye105of the viewer by the separating layer113. The redirected light reaches the retina in the eye105, then the viewer will perceive the virtual image superimposed on the ambient scene image seen through the glass plate unit110. The rear correcting surface116arefracts both ambient scene image light and virtual image light outgoing from the rear glass plate116.

FIG. 3is a schematic diagram illustrating a first optical path for a light coming from ambient scene and captured by the image sensor and a second optical path for a light projected from the image projector and directed to the eye of a viewer. Front glass plate310, first optical compensation element315and image sensor320shown inFIG. 3correspond to the elements112,124and120shown inFIG. 1, respectively. Also, image projector350, second optical compensation element355, rear glass plate360shown inFIG. 3correspond to the elements130,134and116shown inFIG. 1, respectively.

ReferringFIGS. 1 and 3, as discussed above, a light coming into the front glass plate112is refracted by the front correcting surface112aof the plate112. Therefore, the ambient scene image, coming into the front glass plate112, propagated within the plate112and captured by the image sensor120is deformed (magnified, demagnified or distorted) by the front correcting surface112aof the front glass plate112. If no compensation is applied to the captured ambient scene image, the ambient scene image is not recognized correctly by the processor245in the module230(FIG. 2), which would cause miss-positioning or miss-sizing of a virtual image to be output by the processor245that results in spatial incoherency between the ambient scene image seen through the glass unit110and the virtual image to be superimposed on the ambient scene image. Such compensation might be realized by any kinds of image processing to be performed by the processor245, but this solution needs a lot of computational costs on the processor245and would cause quality loss of the captured ambient scene image due to resampling or interpolation during the image processing.

A different approach, which does not need image processing for compensating the deformation of the captured ambient scene image, is provided by an embodiment according to the invention. In the embodiment, the first optical compensation element124is provided at the input end of the image sensor120. The optical compensation element124can be an optical lens or optical surface or the like. The optical compensation element124has an optical characteristic which compensate or cancel the deformation of the ambient scene image caused by the front correcting surface112aof the front glass plate112. The optical characteristic of the optical compensation element124may be selected when it is manufactured, based on the optical characteristic of the front correcting surface112aof the front glass plate112. Thanks to the optical compensation element124, the image sensor120can capture a compensated, “original” ambient scene image formed on the imaging area of the image sensor120.

Also, as discussed above, a light projected from the image projector130, propagated within the intermediate glass plate114and redirected toward the eye105of the viewer by the separating layer160is refracted by the rear correcting surface116aof the rear glass plate116. Therefore, the eye105of the viewer sees a virtual image deformed (magnified, demagnified or distorted) by the rear correcting surface116aof the rear glass plate116. If any compensation is not applied to the virtual image to be presented to the viewer, the viewer would see the virtual image superimposed on the ambient scene seen through the glass plate unit110at inappropriate position and/or in inappropriate size, which would cause spatial incoherency between the ambient scene image seen through the glass unit110and the virtual image to be superimposed on the ambient scene image. The virtual image might be deformed in advance by image processing in view of the deformation to be caused by the rear correcting surface116a,however this approach needs a lot of computational costs and also would cause quality loss of the virtual image due to resampling or interpolation during the image processing.

In this embodiment, a second optical compensation element134is provided at the output end of the image projector130. The optical compensation element134can be an optical lens or optical surface or the like. The optical compensation element134has an optical characteristic which compensate or cancel the deformation of the virtual image to be caused by the rear correcting surface116aof the rear glass plate116. The optical characteristic of the optical compensation element134may be selected when it is manufactured, based on the optical characteristic of the rear correcting surface116aof the rear glass plate116. According to the embodiment, an “original” virtual image projected by the image projector130is deformed (magnified, demagnified or distorted) by the optical compensation element134, then the “deformed” virtual image light is propagated within the intermediate glass plate114by total internal reflection (TIR), reflected and redirected toward the eye105of the viewer by the separating layer160and finally outgoes from the rear glass plate116through its rear correcting surface116a.The “deformed” virtual image is restored to the “original” virtual image by refraction to be caused when the “deformed” virtual image light outgoes from the rear correcting surface116a.Thus, the “original” virtual image is presented to the eye105of the viewer.

As described above, inFIG. 1, only the half components of the device100for the right eye of a viewer are illustrated for the simplicity of the illustration. However, it should be noted that the device100may comprise the same components for the left eye of a viewer as illustrated inFIG. 1in symmetrical manner, which will provide the virtual image to both eyes of the viewer. Alternatively, the device100may comprise only a single, simple glass plate and a temple connected to the glass plate, or only an empty frame without a glass plate and a temple connected to the frame, for the left eye of a viewer, which will provide the virtual image to only one eye of the viewer but it would be acceptable to a certain purpose of use.

The glass plate unit110has been discussed hereinabove in a context that it comprises three elements of the front glass plate112, intermediate glass plate114and rear glass plate116. However, it should be noted that the glass plate unit100can have any configurations as long as it performs the aforementioned functions of the glass plate unit, which functions may include at least, injection and extraction of the virtual image light, guiding the ambient scene image light and the virtual image light, and visual correction. In this sense, two adjacent plates112and114and/or two adjacent plates114and116may be consolidated each other, thus the glass plate unit110may have less plates than three plates.

Further, in the aforementioned examples, the ambient scene image light150and the virtual image light160are guided by the total internal reflection (TIR) within the plates112and114, respectively. It should be noted that the glass plate unit110may be configured to guide at least one of the lights150and160with different approach than the TIR approach. For example, the ambient scene image light150reflected on the separating layer113may be directly condensed on the image sensor120through the first optical compensation element124and/or the virtual image light160projected from the image projector130and passed through the second optical compensation element134may be directly applied to the separating layer113.

Yet further, in one embodiment of the invention, the glass plate unit110may comprise, instead of the separating layer113, an array of reflective mirror elements that are positioned in such way that a gap between at least two of the elements exists. In another embodiment of the invention, the glass plate unit110may comprise, instead of the separating layer113, an array of semi-reflective mirror elements that are also spaced out each other in such way that a gap between at least two of the elements exists. Such a gap enables an external light to go through it.