Patent Description:
Wearable systems can integrate various elements, such as miniaturized computers, input devices, sensors, image displays, wireless communication devices, and image and audio processors, into a device that can be worn by a user. Such systems can provide a mobile and lightweight solution to communicating, computing, and interacting with a user's environment. With the advance of technologies associated with wearable systems and miniaturized optical elements, it has become possible to consider wearable compact optical display systems that augment the user's experience of a real-world environment.

In one example, by placing an image display element or component close to the user's eye(s), an artificial or virtual computer-generated image can be displayed over the user's view of the real-world environment. One or more such image display elements can be incorporated into optical display systems and referred to generally as near-eye displays, head-mounted displays ("HMDs"), or heads-up displays ("HUDs"). Depending upon the size of the display element and the distance to the user's eye, the artificial image may fill or nearly fill the user's field of view. <CIT> discloses an interactive head-mounted eyepiece including an optical assembly through which the user views a surrounding environment and displayed content.

In a first aspect, a method includes generating a light pattern using a display panel and forming a virtual image from the light pattern utilizing one or more optical components. The virtual image is viewable from a viewing location. The method also includes receiving external light from a real-world environment incident on an optical sensor. The real-world environment is viewable from the viewing location. Further, the method includes obtaining an image of the real-world environment from the received external light, identifying a background feature in the image of the real-world environment over which the virtual image is overlaid, and extracting one or more visual characteristics of the background feature. Additionally, the method includes comparing the one or more visual characteristics to an upper threshold value and a lower threshold value and controlling the generation of the light pattern based on the comparison.

In a second aspect, a display system includes an image generator configured to generate a virtual image and a first beam splitter optically coupled to the image generator. The virtual image and a real-world view are viewable through the first beam splitter from a viewing location. The display system also includes a second beam splitter optically coupled to the first beam splitter and a camera optically coupled to the second beam splitter. The camera is configured to image the real-world view. Further, the display system includes a controller operatively coupled to the camera and the image generator. The controller is configured to enhance the contrast of the virtual image with respect to the real-world view based on the image of the real-world view.

In a third aspect, a display system includes a display panel configured to generate a light pattern and one or more optical components coupled to the display panel and configured to transmit the light pattern and external light from a real-world environment. The light pattern is viewable from a viewing location through the one or more optical components as a virtual image superimposed over the real-world environment. The display system also includes an optical sensor coupled to the one or more optical components and configured to receive the external light to obtain an image of the real-world environment. Further, the display system includes a processor coupled to the display panel and the optical sensor and configured to identify a background portion of the real-world environment over which the virtual image is superimposed, to extract pixel data corresponding to the background portion, to compare the pixel data to an upper threshold value and a lower threshold value, and to control the generation of the light pattern based the comparison.

The present disclosure generally relates to an optical display system that enables a user to observe the user's real-world surroundings or environment and to view a computer-generated virtual image. In some cases, the virtual image overlays a portion of the user's field of view of the real world.

In accordance with one example, the display system of the present disclosure includes a see-through wearable computer system, such as an HMD that displays a computer-generated virtual image that may be overlaid over a portion of the user's field of view of the real-world environment or surroundings. Thus, while the user of the HMD is going about his or her daily activities, such as walking, driving, exercising, etc., the user may be able to see a displayed image generated by the HMD at the same time that the user is looking out at his or her real-world surroundings.

The virtual image may include, for example, graphics, text, and/or video that provide content, such as data, alerts, or indications relating to the user's real-world environment. The content of the virtual image can relate to any number of contexts, including, but not limited to, the user's current environment, an activity in which the user is currently engaged, a biometric status of the user, and any audio, video, or textual communications that have been directed to the user. The virtual image may also be part of an interactive user interface and include menus, selection boxes, navigation icons, or other user interface features that enable the user to interact with the display system and other devices.

The content of the virtual image can be updated or modified dynamically in response to a change in the context, such as a change in the user's real-world field of view, a change in the user's current activity, a received communication, a preset alarm or reminder, an interaction with a user interface or menu, etc. Further, the appearance of the virtual image can be altered or modified in response to background features of the real-world field of view over which the virtual image is overlaid. More particularly, visual characteristics of the virtual image can be altered or modified to increase or decrease the contrast between the virtual image the background features.

Referring now to <FIG>, a display system <NUM> in accordance with an example embodiment enables a user <NUM> to observe a real-world environment and to view a computer-generated virtual image. In <FIG>, the user's view of the real-world environment is observed by receiving external light <NUM> from the real world. The illustrated display system <NUM> includes an image generator <NUM>, one or more optical components <NUM>, an optical sensor <NUM>, a processor <NUM>, data storage <NUM>, a power supply <NUM>, and other input/output ("I/O") components <NUM>. The various components <NUM>-<NUM> of the display system <NUM> of <FIG> are operatively coupled together by a connection <NUM>, which can represent any number of wired or wireless electrical connections and/or direct or indirect physical or optical couplings, for example.

Generally, the processor <NUM> controls the image generator <NUM> to generate a light pattern that is directed through the optical component(s) <NUM> to form the virtual image that is viewable by the user <NUM>. In addition, the processor <NUM> and the optical sensor <NUM> are configured to obtain an image or representation of the real-world environment and to identify a background feature in the real-world environment over which the virtual image is overlaid. The processor <NUM> is further configured to determine a visual characteristic of the background feature and to control the light pattern generated by the image generator <NUM> adjust the contrast between the virtual image and the background feature.

In one example, the light pattern is modified to increase the contrast between the virtual image and the background so that the virtual image is more distinguishable from the background. In this example, the contrast may be increased so that information displayed by the virtual image is more easily identified by a user.

In another example, the light pattern is modified to decrease the contrast between the virtual image and the background so that the background may be more visible through the virtual image. In this example, the contrast may be decreased to provide so that the user can more clearly see the real-world environment.

The contrast between the virtual image and the background feature can be adjusted by modifying one or more visual characteristics of the virtual image, for example, hue, saturation, brightness or intensity of the virtual image and/or a background brightness of the image generator <NUM>, size, location, font, etc..

For example, in response to the processor <NUM> determining that the virtual image is overlaid on a background feature that is relatively dark, the virtual image can be adjusted to include brighter colors to increase the contrast or to include darker colors to decrease the contrast. In another example, in response to the processor <NUM> determining that the virtual image is overlaid on a background feature that is relatively bright, the virtual image can be adjusted to include darker colors to increase the contrast or to include brighter colors to decrease the contrast.

In the present example, the data storage <NUM> can be any suitable device or computer readable medium that is capable of storing data and instructions that can be executed by the processor <NUM> to control the image generator <NUM>, to obtain the representation of the real-world environment, to identify a background feature in the real-world environment over which the virtual image is overlaid, to determine visual characteristics of the background feature, and to control other components of the display system <NUM>, for example. The power supply <NUM> provides electrical power to the various components of the display system <NUM> and can be any suitable rechargeable or non-rechargeable power supply. Further the I/O components <NUM> may include switches, dials, buttons, touch screens, etc. that allow the user <NUM> to interact with the display system <NUM>. The I/O components <NUM> may also include, for example, speakers, microphones, biometric sensors, environmental sensors, and transmitters and/or receivers for communicating with other devices, servers, networks, and the like.

<FIG> shows an isometric schematic view of an optical system <NUM> in accordance with an example embodiment. For purposes of illustration, the optical system <NUM> is described with reference to an XYZ coordinate system <NUM> and in relation to a viewing location <NUM>. The optical system <NUM> generally includes a first proximal portion <NUM> and a second distal portion <NUM>. In typical operation, the proximal portion <NUM> is disposed adjacent the viewing location <NUM> and defines a viewing axis <NUM> therethrough. An object <NUM>, such as an eye of a user or a camera or other optical sensor, can be positioned generally at the viewing location <NUM> to view a real-world environment and a computer-generated virtual image. The real-world environment and the virtual image can be viewable simultaneously. For example, the virtual image may overlay a portion of the user's view of the real-world environment.

In <FIG>, the distal portion <NUM> extends generally horizontally along the x-axis from the proximal portion <NUM> such that the distal portion is to the right of the proximal portion from the perspective of the viewing location <NUM>. However, other configurations are possible, for example, the distal portion <NUM> can be to the left of the proximal portion <NUM>, the optical system <NUM> can extend vertically with the distal portion located above or below the proximal portion, or the distal portion can extend in any other direction from the proximal portion.

In the illustrated optical system <NUM>, the proximal portion <NUM> includes a proximal beam splitter <NUM> that has faces generally parallel to XY, XZ, and YZ planes. In <FIG>, a viewing window <NUM> is coupled to a front side of the proximal beam splitter <NUM> and allows external light into the proximal beam splitter. The viewing axis <NUM> is defined through the proximal beam splitter <NUM> and the viewing window <NUM> and is directed substantially parallel to the z-axis.

Generally, in use, the viewing location <NUM> and the eye of the user <NUM> are positioned at a back side of the proximal beam splitter <NUM> so that the user can view the real world through the viewing window <NUM> and the proximal beam splitter along the viewing axis <NUM>. In the present example, the optical system <NUM> further includes an image former <NUM> optically coupled to the proximal beam splitter <NUM>. In one example, the image former <NUM> is configured to reflect light corresponding to the virtual image in the direction of the x-axis.

The proximal beam splitter <NUM> of <FIG> includes a proximal beam-splitting interface <NUM> that is configured to combine the external light entering the proximal beam splitter through the viewing window <NUM> with the light that represents the virtual image generated by the optical system <NUM>. In this manner, the real-world environment and the virtual image can be viewed along the viewing axis <NUM>. In one example, the proximal beam-splitting interface <NUM> is in a plane that forms about <NUM>-degree angles with the faces of the proximal beam splitter <NUM> that are in the XY-plane and the YZ-plane and is perpendicular to the faces in the XZ-plane. As a result, the proximal beam-splitting interface <NUM> intersects the viewing axis <NUM> at about <NUM> degrees. It is to be understood, however, that other angles and configurations are possible.

As seen in <FIG>, the distal portion <NUM> of the optical system <NUM> includes a distal beam splitter <NUM> that has faces generally parallel to XY, XZ, and YZ planes. The distal beam splitter <NUM> is, in turn, optically coupled to the proximal beam splitter <NUM> by a light pipe <NUM>, for example. The distal beam splitter <NUM> includes a distal beam-splitting interface <NUM> that is generally configured to transmit and reflect light to and from the proximal beam splitter <NUM> through the light pipe <NUM>. Such transmitted and reflected light can be utilized to generate the virtual image. In one example, the distal beam-splitting interface <NUM> is a plane that forms an angle with the faces of the distal beam splitter <NUM> that are in the XY-plane and the YZ-plane and is perpendicular to the faces in the XZ-plane. The distal beam-splitting interface <NUM> is arranged at a non-zero angle with respect to the proximal beam-splitting interface <NUM>. In one example, the distal beam-splitting interface <NUM> is generally orthogonal to the proximal beam-splitting interface <NUM>. It is to be understood, however, that the orientation of the distal beam-splitting interface <NUM> may be modified in other examples. For example, the distal beam-splitting interface <NUM> can be in a plane that is parallel to the proximal beam-splitting interface <NUM> or parallel to the viewing axis <NUM>.

In one embodiment, the proximal beam splitter <NUM>, the distal beam splitter <NUM>, and the light pipe <NUM> are made of glass. Alternatively, some or all of such optical components may be made partially or entirely of plastic, which can also function to reduce the weight of the optical system <NUM>. A suitable plastic material is Zeonex® E48R cyclo olefin optical grade polymer, which is available from Zeon Chemicals L. , Louisville, Kentucky. Another suitable plastic material is polymethyl methacrylate ("PMMA").

The distal portion <NUM> further includes a display panel <NUM> and a light source <NUM> optically coupled to the distal beam splitter <NUM>. In the present example, the display panel <NUM> is generally vertically oriented and coupled to a right side of the distal beam splitter <NUM> and the light source <NUM> is coupled to a back side of the distal beam splitter.

The display panel <NUM> is configured to generate a light pattern from which the virtual image is formed. The display panel <NUM> may be an emissive display such as an Organic Light Emitting Diode ("OLED") display. Alternatively, the display panel <NUM> may be a Liquid-Crystal on Silicon ("LCOS") or a micro-mirror display such as a Digital Light Projector ("DLP") that generates the light pattern by spatially modulating light from a light source, such as the light source <NUM>. The light source <NUM> may include, for example, one or more light-emitting diodes ("LEDs") and/or laser diodes. The light pattern generated by the display panel <NUM> can be monochromatic or may include multiple colors, such as red, green, and blue, to provide a color gamut for the virtual image.

In one example of the optical system <NUM> in use, the light source <NUM> emits light toward the distal beam-splitting interface <NUM>, which reflects the light toward the display panel <NUM>. The display panel <NUM> generates a light pattern by spatially modulating the incident light to provide spatially modulated light reflected toward the distal beam-splitting interface <NUM>. The distal beam-splitting interface <NUM> transmits the spatially modulated light through the light pipe <NUM> and toward the proximal beam splitter <NUM>. The proximal beam-splitting interface <NUM> transmits the spatially-modulated light so that it reaches the image former <NUM>. The image former <NUM> reflects the spatially-modulated light back toward the proximal beam-splitting interface <NUM>, which reflects the spatially-modulated light toward the viewing location <NUM> so that the virtual image is viewable along the viewing axis <NUM>.

As a general matter, the reflection and/or transmission of light by and/or through the beam splitters <NUM>, <NUM> or other optical components of the optical system <NUM> may refer to the reflection and/or transmission of substantially all of the light or of a portion of the light. Consequently, such terms and descriptions should be interpreted broadly in the present disclosure.

In some embodiments, the proximal and/or distal beam splitters <NUM>, <NUM> may be polarizing beam splitters, such that the beam splitters preferentially transmit p-polarized light and preferentially reflect s-polarized light, for example. Alternatively, the proximal and/or distal beam splitters <NUM>, <NUM> may be non-polarizing beam splitters that transmit a portion of the incident light and reflect a portion of the incident light independent (or largely independent) of polarization.

In one embodiment, the proximal beam splitter <NUM> and the distal beam splitter <NUM> are polarizing beam splitters that preferentially transmit p-polarized light and preferentially reflect s-polarized light. With this configuration, the external light that is viewable along the viewing axis <NUM> is generally p-polarized and the light that is viewable along the viewing axis as the virtual image is generally s-polarized. The light source <NUM> may provide s-polarized light that is partly reflected by the distal beam-splitting interface <NUM> toward the display panel <NUM>. The display panel <NUM> spatially modulates the incident s-polarized light and also changes its polarization. Thus, in this example, the display panel <NUM> converts the incident s-polarized light into a spatially-modulated light pattern of p-polarized light. At least a portion of the p-polarized light is transmitted through the distal beam-splitting interface <NUM>, through the light pipe <NUM>, and through the polarizing proximal beam-splitting interface <NUM> to the image former <NUM>.

In the present example, the image former <NUM> includes a reflector <NUM>, such as a concave mirror or Fresnel reflector, and a quarter-wave plate <NUM>. The p-polarized light passes through the quarter-wave plate <NUM> and is reflected by the reflector <NUM> back through the quarter-wave plate toward the proximal beam-splitting interface <NUM>. After the light pattern interacts with the image former <NUM> in this way, the polarization is changed from p-polarization to s-polarization and the s-polarized, spatially-modulated light is reflected by the proximal beam-splitting interface <NUM> toward the viewing location <NUM> so that the virtual image is viewable along the viewing axis <NUM>.

Referring back to <FIG>, the optical system <NUM> further includes an optical sensor <NUM> that is optically coupled to the distal beam splitter <NUM>. In <FIG>, the optical sensor <NUM> is generally vertically oriented and coupled to a front side of the distal beam splitter <NUM>. The optical sensor <NUM> can be a camera, such as a wafer-level camera, an infrared ("IR") camera, a CCD image sensor, a CMOS sensor, and the like, with an image sensing portion of the optical sensor directed towards or facing the distal beam splitter <NUM>. The optical sensor <NUM> is configured to image the external light entering through the viewing window <NUM> and viewable by the user along the viewing axis <NUM>. The optical sensor <NUM> may be configured to capture still images and/or video. The still images and/or video captured by the optical sensor <NUM> may substantially correspond to the view of the real world that the user sees when looking through the viewing window <NUM> and may be processed with the virtual image to determine where the virtual image is disposed with respect to the real-world environment.

In an example of the optical system <NUM> in use, external light from the real world enters through the viewing window <NUM> and is reflected by the proximal beam-splitting interface <NUM>, through the light pipe <NUM>, and toward the distal beam splitter <NUM>. The distal beam-splitting interface <NUM> reflects the incident external light to the optical sensor <NUM> to obtain an image of the real-world environment.

In another example, the optical sensor <NUM> may be disposed on the distal beam splitter <NUM> with an image sensing portion thereof directed away from the distal beam splitter. In this example, the optical sensor <NUM> may receive external light directly to image the real-world environment.

In yet another example, the optical sensor <NUM> may be disposed proximate the viewing location <NUM>, such as on the proximal beam splitter <NUM>. In the present example, an image sensing portion of the optical sensor <NUM> may be directed away from the viewing location <NUM> generally along the viewing axis <NUM> to receive external light directly to image the real-world environment. Alternatively, an image sensing portion of the optical sensor <NUM> may be directed in another direction and the optical system <NUM> configured to transmit external light to the image sensing portion to image the real-world environment.

Various modifications can be made to the optical system <NUM> of <FIG> without departing from the present disclosure. For example, the optical system <NUM> of <FIG> may be part of the display system <NUM> of <FIG>, so as to be coupled to the processor <NUM>, the data storage <NUM>, the power supply <NUM>, and/or the I/O components <NUM>. Such components <NUM>-<NUM> may be coupled to the display panel <NUM>, the light source <NUM>, and/or the optical sensor <NUM> in any known manner. In another example, the proximal and/or distal beam-splitting interfaces <NUM>, <NUM> may be curved to account for a curvature of the reflector <NUM> and/or a curvature of a lens (not shown) of the optical sensor <NUM>.

Referring now to <FIG>, an example flowchart <NUM> is illustrated that includes a process for obtaining and using real-world and virtual image contrast information. The process of <FIG> may be performed by utilizing various hardware and/or software components of the display system <NUM> of <FIG> and the optical system <NUM> of <FIG>, for example.

In <FIG>, the flowchart <NUM> begins at a block <NUM>, during which external light that represents the real-world environment is received, such as by the optical sensor <NUM> described above. Next control passes to a block <NUM> and the received external light is processed to obtain an image or other representation of the real world.

Following the block <NUM>, control passes to a block <NUM>, which generates a light pattern from which a virtual image can be formed. Referring to <FIG>, for example, the light pattern may be generated by the display panel <NUM> and/or the display panel in combination with the light source <NUM>. Next, control passes to a block <NUM> and the light pattern is formed into a virtual image viewable by a user, as described above, for example.

In the illustrated flowchart <NUM>, control then passes to a block <NUM> during which a calibration process may be performed. In one example, the calibration process includes calibrating the virtual image with the user's view of the real-world environment. The calibration process in accordance with one non-limiting example includes displaying one or more markers or indicia in the virtual image overlaid on the user's real-world view. The indicia may correspond to background features in the real-world view. In the present example, the user may be instructed to align the indicia with the background features in the real-world view as the image of the real world obtained by the display system is processed. The user may be instructed to provide an input, for example, through the I/O components <NUM>, when the indicia are aligned or may be instructed to align the indicia for a given time period, for example, about <NUM>-<NUM> seconds. In one example, the indicia may include portions of the image of the real-world view that can be aligned with the user's actual view of the real world. The processing of the image of the real world as the user aligns the indicia can be used to calibrate the virtual image with the user's real-world environment.

At a block <NUM>, such calibration may be used to accurately identify background features in the real-world environment over which the virtual image is overlaid. In one example, the block <NUM> identifies image pixels that correspond to background features that are directly overlaid by the virtual image. In another example, the block <NUM> also identifies image pixels that are adjacent to the pixels directly overlaid by the virtual image. The block <NUM> also processes the identified background features to determine visual characteristics thereof. Such visual characteristics may include pixel data relating to intensity, hue, and/or saturation of the background features. The visual characteristics of the background features may be compared to threshold values and/or to visual characteristics of the virtual image to analyze the contrast between the virtual image overlaid on the background features.

At a block <NUM>, the display system can be controlled in response to the contrast analysis. In one example, the block <NUM> may control the display system by modifying the light pattern to increase the contrast between the virtual image and the background features so that the virtual image is more distinguishable from the background. In another example, the block <NUM> may control the display system by modifying the light pattern to decrease the contrast between the virtual image and the background features so that the background may be more visible through the virtual image.

Various modifications can be made to the flowchart <NUM> of <FIG>, for example, additional or fewer process blocks can be utilized and arranged in any appropriate order or even executed concurrently. For example, the calibration process may be omitted, performed only once by a manufacturer of the display device, or performed multiple times in any order with respect to the other process blocks.

<FIG> illustrates another flowchart <NUM> of a process for analyzing and adjusting the contrast between a virtual image and a background feature in a real-world environment over which the virtual image is overlaid. The flowchart <NUM> begins at a block <NUM>, which extracts pixel data that corresponds to the background features, such as the background features identified in the block <NUM> of <FIG>. In one example, pixel data from the image of the background feature are extracted and converted into an image data mask. The mask can be a <NUM>-bit monochrome mask with pixel intensity values of <NUM> (dark) or <NUM> (bright) for each pixel. In other examples, the monochrome mask may be a <NUM>-bit, <NUM>-bit, <NUM>-bit, etc. data mask associated with real values between and including <NUM> and <NUM>. The values <NUM> and <NUM> may correspond to darkest and brightest values, respectively, associated with an optical sensor and/or image generator of an optical system. In other examples, the dark and bright values may be defined by any other scale, such as <NUM> to <NUM>, <NUM> to <NUM>, <NUM>% to <NUM>%, etc. In further examples, pixel data from the image of the background feature may be converted to a color data mask that includes color pixel data.

In the illustrated flowchart <NUM>, control then passes to a block <NUM> to correct for misalignment between the image of the real-world environment, a view of the real-world environment from the viewing location, and/or the virtual image. In one example, misalignment may be corrected utilizing data from the calibration process of the block <NUM> of <FIG>. Alternatively or in combination, the image data mask may be the blurred with an averaging filter, such as a Gaussian filter, to account for misalignment.

After the block <NUM>, control passes to a block <NUM> to compare pixel data values of the background features to an upper threshold pixel value. Generally, the block <NUM> is performed to identify bright background features that make it more difficult to identify virtual image details, such as text. In one example, pixel values of the data mask corresponding to the background features are compared to the upper threshold value on a per pixel basis. Although, in other examples, the data mask of the background features can be divided into pixel groups having pixel values that are averaged and compared to the upper threshold value.

Next, control passes to a block <NUM> to adjust the contrast of pixels of the virtual image that correspond to pixels of the background features that have values greater than the upper threshold value. The contrast of such virtual image pixels can be adjusted by enhancing the original virtual image using any suitable method, such as histogram equalization. In one example, an enhanced virtual image is obtained by rescaling and clipping pixel values of the original virtual image. For example, the following algorithm (<NUM>) may be used when intensity values are scaled between <NUM> and <NUM>: <MAT> In the above algorithm, I_enhanced(x,y) is the intensity of an enhanced virtual image pixel located at (x,y); I_original(x,y) is the intensity of a pixel located at (x,y) in the original virtual image; V_dark and V_bright are pixel values in the original virtual image for the darkest and brightest points, respectively. V_dark and V_bright can be determined, for example, by histogram equalization. Various modifications can be made to algorithm (<NUM>), such as to modify the algorithm when the intensity values are defined by other scales.

In another example, the contrast of the virtual image can be further adjusted by blending the original virtual image with an enhanced virtual image. One example of the blending process includes a per pixel linear interpolation between the original virtual image and an enhanced image using the data mask as a blending factor. In one example, the following blending algorithm (<NUM>) may be used when intensity values are scaled between <NUM> and <NUM>: <MAT> In the above algorithm, I_corrected(x,y) is the intensity of a blended pixel located at (x,y); M(x,y) is the pixel value of the data mask at (x,y); I_original(x,y) is the intensity of a pixel located at (x,y) in the original image; and I_enhanced(x,y) is the intensity of an enhanced virtual image pixel located at (x,y). Various modifications can be made to algorithm (<NUM>), such as when the intensity values are defined by other scales.

After the block <NUM>, control passes to a block <NUM> to compare pixel data values of the background features to a lower threshold pixel value. Generally, the block <NUM> is performed to identify dark background features that make it more difficult to identify virtual image details, such as text. In one example, pixel values of the data mask corresponding to the background features are compared to the lower threshold value on a per pixel basis. Although, in other examples, the data mask of the background features can be divided into pixel groups having pixel values that are averaged and compared to the lower threshold value.

Next, control passes to a block <NUM> to adjust the contrast of pixels of the virtual image that correspond to pixels of the background features that have values less than the lower threshold value. The contrast of such virtual image pixels can be adjusted by enhancing the original virtual image using any suitable method, such as histogram equalization. In one example, an enhanced virtual image is obtained by rescaling and clipping pixel values of the original virtual image. For example, the following algorithm (<NUM>) may be used: <MAT>.

The contrast of the virtual image over dark backgrounds can also be further adjusted by blending the original virtual image with the enhanced virtual image. The blending may be performed using blending algorithm (<NUM>), as discussed above: <MAT>.

Following the block <NUM>, control passes to a block <NUM> to compare the image of the real-world environment to background lighting of the virtual image. In one example, the block <NUM> compares a minimum intensity value of the real-world image, such as a minimum intensity value of a background feature overlaid by a virtual image, to an intensity value of the background lighting of the virtual image. Generally, the block <NUM> is performed to identify portions of the virtual image that may be too bright and cause an undesirable graying effect in portions of the user's real-world view. In particular, such graying effect may occur with dark background features. The comparison in the block <NUM> determines whether the background lighting for the virtual image is too bright based on a mapping of optical background lighting intensities in relation to real-world light intensities. Generally, such mapping is specific to a particular optical display system and identifies highest background lighting intensity values with respect to real-world light intensity values to avoid the graying effect. In the block <NUM>, a minimum intensity value of the real-world image is identified and the background lighting of the virtual image is set to an intensity value based on such mapping.

Thereafter, control passes to a block <NUM> and the contrast of the virtual image can be adjusted, for example, as described above. In one example, the background lighting intensity value is taken into account in determining the values of V_bright and V_dark.

Various modifications can be made to the flowchart <NUM> of <FIG>, for example, additional or fewer process blocks can be utilized and arranged in any appropriate order or even executed concurrently. For example, the flowchart <NUM> may include a single block that compares the image data to upper and lower thresholds instead of having separate blocks <NUM>, <NUM>.

Referring now to <FIG>, and <FIG>, the systems described above for <FIG> may be attached to a head-mounted support in a position such that the viewing axis is conveniently viewable by either a left or right eye of the wearer or user. In this way, a head-mounted display (HMD) may be provided through which the outside world is viewable. The HMD may also function as a wearable computing device. In <FIG>, an HMD <NUM> includes see-through display devices <NUM> and <NUM> for the wearer's right eye <NUM> and left eye <NUM>, respectively. The display devices <NUM>, <NUM> are attached to a head-mountable support <NUM>. In this example, the head-mountable support <NUM> is configured in the form of eyeglasses with lenses <NUM>, <NUM> positioned over the right eye <NUM> and the left eye <NUM>, respectively. The lenses <NUM>, <NUM> and are held in place by respective frames <NUM> and <NUM>. The head-mountable support <NUM> also includes a bridge piece <NUM> that is connected to the frames <NUM>, <NUM> and is configured to be supported by the bridge of the user's nose. In addition, the head-mountable support <NUM> includes side-pieces <NUM> and <NUM> connected to frames <NUM>, <NUM>, respectively, which may hook behind the user's ears.

The right-side display device <NUM> may be attached to the frame <NUM> by a mount <NUM> and the left-side display device <NUM> may be attached to the frame <NUM> by a mount <NUM>. The mounts <NUM>, <NUM> position the display devices <NUM>, <NUM> so that their respective viewing axes <NUM>, <NUM> are generally aligned with the user's right eye <NUM> and left eye <NUM>, respectively. Thus, as shown in <FIG>, the viewing axis <NUM> of the right-side display device <NUM> may extend to the user's right eye <NUM> through the lens <NUM> and the viewing axis <NUM> of the left-side display device <NUM> may extend to the user's left eye <NUM> through the lens <NUM>. To achieve this configuration, the mounts <NUM>, <NUM> can be fixed mounts or they can be adjustable by the user in order to properly and comfortably align the display devices <NUM>, <NUM>.

Although <FIG> illustrate the HMD <NUM> with the display devices <NUM>, <NUM> separate from the lenses <NUM>, <NUM> and the frames <NUM>, <NUM>, it should be understood that other configurations are possible. For example, some or all of the components of the display devices <NUM>, <NUM> can be integrated into the lenses <NUM>, <NUM> and/or the frames <NUM>, <NUM>. For example, beam splitters and light pipes may be integrated into the lenses <NUM>, <NUM> and/or display panels may be integrated into the frames <NUM>, <NUM>. In addition, other embodiments may include a display device for only one of the wearer's eyes. In other examples, the HMD <NUM> may be configured as goggles, a helmet, a head-band, or a hat. Further, instead of a head-mountable support <NUM>, the support mount can be on a user-mountable support that can be mounted on the user in other ways, such as on one or both of the user's shoulders or on a backpack being worn by the user.

As noted above, the HMD <NUM> may function as a wearable computing device. In this regard, the HMD may include a processor <NUM>, which can be located inside of or attached to part of the head-mountable support <NUM>. For example, the processor <NUM> can be located inside of the side-piece <NUM>, as shown in <FIG>. However, other configurations are possible.

In one embodiment, the processor <NUM> is configured to control display panels in the display devices <NUM>, <NUM> in order to control the virtual images that are generated and displayed to the user. Further, the processor <NUM> is configured to control optical sensors and to receive images or video captured by the optical sensors. The processor <NUM> may be communicatively coupled to the display devices <NUM>, <NUM> by wires inside of the head-mountable support <NUM>, for example. Alternatively, the processor <NUM> may communicate with the display devices <NUM>, <NUM> through external wires or through a wireless connection.

The HMD <NUM> may also include other components that are operatively coupled to the processor <NUM> to provide desired functionality. For example, the HMD <NUM> may include one or more touchpads, microphones, and sensors, which are exemplified in <FIG> by a touchpad <NUM>, a microphone <NUM>, and a sensor <NUM> on the side-piece <NUM>. It is to be understood, however, that these components can be located elsewhere in the HMD <NUM>. By appropriate touch interaction with the touchpad <NUM>, the user may control or provide input to the HMD <NUM>. The microphone <NUM> may be used to receive voice commands from the user and/or to record audio data from the user's surroundings. The sensor <NUM> may include an accelerometer and/or gyroscope configured to sense movement of the HMD <NUM>. The sensor <NUM> may also include a global positioning system receiver for determining the location of the HMD. Additionally, the sensor <NUM> may represent a camera or plurality of cameras that may be configured to observe various fields of view around the HMD <NUM>. The HMD <NUM> may also include a wired and/or wireless interface through which the processor <NUM> may exchange data with other computing systems or other devices. In addition to the foregoing, the HMD <NUM> could also include other types of sensors, user interface components, and/or communication interface components.

The processor <NUM> may control the content of the virtual images generated by the display systems <NUM>, <NUM> and in response to various inputs. Such inputs may come from the touchpad <NUM>, the microphone <NUM>, the sensor <NUM>, and/or a wired or wireless communication interfaces of HMD. The processor <NUM> may also control the appearance of the virtual images in response to background feature analysis, as described above. In this way, the processor <NUM> may control the appearance of the virtual images so that it is appropriate for the user's current surroundings and/or tasks in which the user is involved.

The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying drawings. In the drawings, similar symbols typically identify similar components, unless context dictates otherwise. The illustrative embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments can be utilized, and other changes can be made, without departing from the scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the drawings, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are explicitly contemplated herein.

Claim 1:
A method comprising:
generating a light pattern using a display panel;
forming a virtual image from the light pattern utilizing one or more optical components, wherein the virtual image is viewable from a viewing location;
receiving, by an optical sensor, external light from a real-world environment incident on the optical sensor, wherein the real-world environment is viewable from the viewing location;
obtaining an image of the real-world environment from the received external light;
identifying a background feature in the image of the real-world environment over which the virtual image is overlaid;
extracting, from the image of the real-world environment, pixel data corresponding to the background feature, the pixel data being indicative of one or more visual characteristics of the background feature, wherein the pixel data relate to an intensity of the background features;
comparing the pixel data indicative of the one or more visual characteristics to an upper threshold value and a lower threshold value;
converting the extracted pixel data into an image data mask M associated with real values between and including <NUM> and <NUM>;
controlling the generation of the light pattern based on the comparison, comprising increasing the contrast between the virtual image and the background feature if the pixel data is higher than the upper threshold or lower than the lower threshold, wherein increasing the contrast between the virtual image and the background feature includes enhancing the virtual image to obtain an enhanced virtual image using histogram equalization and blending the virtual image with the enhanced virtual image using the image data mask M as a blending factor, wherein a blended image having an intensity I_corrected is obtained according to <MAT> wherein I_enhanced(x,y) is the intensity of a pixel of the enhanced virtual image located at (x,y) and I_original(x,y) is the intensity of a pixel located at (x,y) in the virtual image.