Patent Description:
In some implementations an optical waveguide can be used to spatially translate a generated image from one position to another position in an optical system. For example, in a near-eye display (NED) device, an optical waveguide made of a substrate can spatially translate propagating light waves representing imagery generated by a light engine and convey them along an optical path toward one or both eyes of a user. Such technology may be incorporated into an NED device in the form of eyeglasses, goggles, a helmet, a visor, or some other type of head-mounted display (HMD) device or eyewear.

However, for typical NED devices, reproduction of an image having a wide field of view (FOV) can be difficult, as existing techniques for increasing FOV can require the use of waveguide substrates that have a high reflective index, which can be difficult to procure, and also significantly increases costs associated with the device. As such, while NED devices can provide a wide FOV by use of higher index substrates, there remain difficulties in generating a wide FOV using less expensive materials that are readily available.

<CIT> Al describes embodiments are disclosed for an optical waveguide display configured for use with a near-eye display (NED) device. In an embodiment the waveguide display includes a light-transmissive substrate and an optical coupling element configured to input light rays to the substrate or output light rays from the substrate. the optical coupling element configured to deflect a plurality of wavelengths of an incident light ray collinearly for propagation within the light-transmissive substrate through total internal reflection (TIR).

The invention is defined in the independent appended claims. Embodiments of the invention are described in the dependent claims.

The description generally relates to techniques for providing wide field of view images in an NED device. One example includes a device that includes an optical assembly configured to generate an image for propagation from a first point of the device to a second point of the device via a waveguide having a transmissive substrate, an in-coupler, and an out-coupler, the in-coupler comprising a deflection grating that provides a constant deflection angle for angles of incidence associated with the generated image and being configured to propagate the generated image through the transmissive substrate of the waveguide to the out-coupler, and the out-coupler comprising a metasurface grating that provides a non-constant deflection angle for angles of incidence associated with the generated image, the metasurface grating increasing a field of view (FOV) of the generated image.

Another example includes a method or technique that can be performed on a device. The method can include generating an image that has a compressed field of view (FOV) based on a wide FOV image, coupling the generated image into an in-coupler of a waveguide, the in-coupler having a first grating, propagating the generated image via a transmissive substrate of the waveguide, coupling the generated image into an out-coupler of the waveguide, the out-coupler having a second grating that provides FOV expansion to recreate the wide-FOV image, and propagating the recreated wide-FOV image for display.

Another example includes a device including a renderer configured to process an image having a first field of view (FOV) to generate an image having a second FOV, a light engine configured to produce light waves based on the generated image having the second FOV, and a waveguide in-coupler configured to couple the produced light waves to a waveguide out-coupler via a transmissive substrate of the waveguide, the waveguide out-coupler configured to expand the generated image having the second FOV to recreate the first FOV.

The above listed examples are intended to provide a quick reference to aid the reader and are not intended to define the scope of the concepts described herein.

The use of similar reference numbers in different instances in the description and the figures may indicate similar or identical items.

Certain NED devices, such as HMD devices, can include optical systems for spatially translating a generated image from one position to another position, for example from a light engine to an eye of a user. Such optical systems can include one or more transparent waveguides arranged so that they are located directly in front of each eye of the user when the NED device is worn by the user, to project light representing generated images into the eye of the user. In NED devices that utilize these waveguides, light can propagate through the waveguides over certain internal angles. Light propagating at some non-zero angle of incidence to a surface of the waveguide can travel within the waveguide via a transmissive substrate, bouncing back and forth between the surfaces via total internal reflection (TIR).

With such a configuration, images generated by the NED device can be overlaid on the user's view of the surrounding physical environment. One aspect of translating a generated image from one position to another via a waveguide involves receiving the light waves into the waveguide ("in-coupling") at a first location and outputting the light waves from the waveguide ("out-coupling") at a second location. Light waves can be in-coupled to and out-coupled from a waveguide via an optical element that may function as an optical input port or an optical output port for the light waves. For example, in some implementations, an optical coupling element configured to in-couple and/or out-couple light waves can comprise a diffractive optical element (DOE). Such a DOE may be a grating structure such as a surface relief grating (SRG). However, it is to be appreciated that other types of outcouplers can be used with NED devices, such as binary or multilevel relief gratings, volume holograms, resonant waveguide gratings, partially reflective mirrors, etc..

The angle of deflection of light waves resulting from deflection by a DOE, such as an SRG, can be a constant deflection angle over all angles of incidence (also referred to herein as a constant momentum change of angles of incidence). Therefore, for a given wavelength, an incoming light wave at incident angle θ to the normal would receive a constant angular change β, and would therefore exit the SRG at an angle θ + β. However, In the context of an optical waveguide for an NED device, this angular change caused by the optical coupling elements can limit the ability to provide a wide FOV for the NED device.

When using optical coupling elements such as SRGs, the FOV for any image produced by a light engine of the NED device can be limited based on the TIR critical angles of the waveguide, which directly correlates to the index of the transmissive substrate of the waveguide. In certain implementations, the transmissive substrate may be glass, and while increasing the refractive index of the glass or other such substrate may improve the TIR critical angles, allowing higher FOV, such high index substrates are both rare and expensive. As such, while a wide FOV image can be sent to the waveguide via an SRG coupler, a portion of the image may be undesirably cropped due to the limited TIR capabilities of the waveguide.

As an alternative to using optical coupling elements such as SRGs, which provides for a constant angular change, metasurfaces that utilize sub-wavelength sized structures, such as nanostructures (i.e., structures that are nanoscale elements), can be used to significantly increase the FOV. By specifying the design of the structures on the metasurface, a phase profile of the metasurface can be created that allows for tailoring of angle-dependent properties. Such nanostructures can be composed of a semiconductor, an oxide (e.g., a metal or non-metal oxide), a nitride (e.g., a metal or non-metal nitride), a sulfide (e.g., a metal or non-metal sulfide), a pure element, or a combination of two or more of these.

Certain nanostructures can be assembled to form a metasurface that can provide a compression/expansion of the FOV by way of a linear momentum change, or angular "kick. " This compression/expansion ratio term can be represented by a constant multiplicative term µ, such that for incident angles greater in magnitude, the angular "kick" would be greater. Upon interaction with the nanostructures, the outgoing angle can be µθ + β, which can result in an approximately linear relationship between incoming and outgoing angles. Thus, such metasurfaces can provide a tailored angular change, rather than the constant angular change provided by conventional optical coupling elements, such as SRGs. Such a metasurface may be referred to as an Engineered Angular Dispersion Metasurface (EADM).

Using EADMs in place of SRG couplers can allow for an optical system associated with the NED device to paint a large FOV image, and based on the nanostructures associated with the metasurfaces, the light waves associated with such a large FOV image can be given the necessary angular "kick" in order to ensure that a wide FOV image is effectively compressed as a result of the in-coupler EADM, ensuring that the compressed image is not cropped by the maximum TIR angles associated with the waveguide substrate. Then, at the end of the waveguide, an out-coupler EADM can be used to expand the FOV back to the original angular range produced by the light engine.

However, even with such metasurfaces, achieving a wide FOV image can be difficult because the compression and expansion of the image via the in-coupler and out-coupler EADMs can result in discrepancies between the input image and the resulting image shown to the user. More specifically, because of the challenges in design and repeatable manufacture of the nanostructures on the metasurfaces, production of completely symmetrical metasurfaces for the in-coupler and out-coupler can be difficult, and when identical symmetry is not achieved, the resulting image displayed to a user may not be a faithful reproduction of the input image.

Furthermore, while the use of EADMs for both in-coupling and out-coupling can produce a wide FOV image, the optical system may require a larger and more expensive light engine in order to generate such a wide FOV. Finally, while EADMs can be created to provide specific angular "kicks" based on the incident angle, the linearity of the angular "kick" can limit the capabilities of the NED device.

Accordingly, implementations disclosed herein are directed to a waveguide-based optical device that can increase the FOV without incurring potentially high costs associated with the NED device by use of, for example, high refractive index substrates, while also avoiding the difficulties posed by utilizing EADMs as both an in-coupler and an out-coupler. Specifically, an optical system can be provided that precompensates an image to provide, from the optical system, a compressed image for in-coupling to the waveguide. Upon generation of the compressed image, a standard DOE can be used in place of a specialized metasurface, such as an EADM, for coupling of the image to the waveguide, thereby reducing the need to utilize two separate EADMs. Additionally, the device can utilize metasurfaces that can be designed so as to provide a nonlinear angular "kick," which allows for additional benefits to be obtained over the use of linearly deflecting EADMs, such as the ability to provide foveation and distortion correction to the resulting image.

<FIG> depicts an example NED device <NUM> in which the methods discussed herein can be incorporated. The NED device <NUM> may provide VR and/or AR display modes for a user, i.e., the wearer of the device. To facilitate description, it is henceforth assumed that the NED device <NUM> is an HMD device designed for AR visualization.

As depicted in <FIG>, NED device <NUM> can include a chassis <NUM>, along with a protective visor <NUM>, left side arm <NUM>, and right side arm 106R, which all may be mounted to chassis <NUM>. Visor <NUM> may be a transparent material that can form a protective enclosure for various display elements coupled to the visor that are discussed below. Chassis <NUM> may provide a mounting structure for visor <NUM> and side arms 106A and 106B, and may be further connected to an adjustable headband <NUM> that enables wearing of NED device <NUM> on a user's head. An optical system <NUM> mounted to chassis <NUM> and enclosed within protective visor <NUM> can generate images for AR visualization, and is described in greater detail below.

In one implementation, optical system <NUM> may include a precompensation renderer <NUM>, which may perform rendering of an image that is to be displayed to a user of NED device <NUM>. Precompensation renderer <NUM> may render a precompensated image (processing details of which are described below) that can be provided to light engine <NUM>, which may generate light waves representing the image for displaying via NED device <NUM>. Precompensation renderer <NUM> may be either a hardware or software renderer, and may be configured to precompensate a wide FOV image into a compressed FOV image for coupling to a waveguide, which will be discussed in further detail below with respect to <FIG>.

Light engine <NUM> may be any sort of device capable of emitting light sources, such as one or more light emitting diodes or laser diodes. Optical system <NUM> may also include a display engine <NUM> for consolidating light waves generated by light engine <NUM> and directing the light waves as appropriate. In one implementation, display engine <NUM> may be a micromechanical system (MEMS)-based scanning system that can "paint" an image based on light waves produced by light engine <NUM>.

Protective visor <NUM> and/or chassis <NUM> may also house controller <NUM>, which may provide various components for providing functionality to NED device <NUM>. In one implementation, controller <NUM> may include various components such as a processing unit <NUM>, a memory <NUM> accessible to the processing unit <NUM> for storing processor readable instructions and data, and a communication module <NUM> communicatively coupled to the processing unit <NUM> which can act as a network interface for connecting the NED device to another computer system. Processing unit <NUM> may include one or more processors including a central processing unit (CPU) and/or a graphics processing unit (GPU). Memory <NUM> can be a computer-readable storage media that may store instructions for execution by processing unit <NUM>, to provide various functionality to NED device <NUM>. Finally, a power supply <NUM> can provide power for the components of controller <NUM> and the other components of NED device <NUM>, such as optical system <NUM> and additional components that may be included in NED device <NUM>, such as image capture devices (e.g. cameras), audio devices (e.g. microphones and speakers), and location/motion capture devices (e.g. accelerometers).

<FIG> depict, in accordance with certain implementations, right side and front orthogonal views, respectively, of aspects that may be part of optical system <NUM> and may be contained within protective visor <NUM> of NED device <NUM> for propagation of imagery toward a user's eye <NUM>. During operation of NED device <NUM>, the display components can be positioned relative to the user's left eye <NUM> and right eye 202R as shown. The display components can be mounted to an interior surface of the chassis <NUM>, which is shown in cross-section in <FIG>.

The display components can be designed to overlay three-dimensional images on the user's view of his real-world environment, e.g., by projecting light into the user's eyes. An image that is generated by precompensation renderer <NUM> can be provided to light engine <NUM>, and light waves from light engine <NUM> may be directed by way of display engine <NUM>, such that the light can be projected into the user's eyes. Furthermore, optical system <NUM> may be connected via a flexible circuit connector <NUM> to controller <NUM> for controlling the generation of light waves.

The display components can further include a waveguide carrier <NUM> to which optical system <NUM> can be mounted. Waveguide carrier <NUM> may include one or more waveguides <NUM> stacked on the user's side of the waveguide carrier <NUM>, for each of the left eye and right eye of the user. The waveguide carrier <NUM> may have a central nose bridge portion <NUM>, from which its left and right waveguide mounting surfaces can extend. One or more waveguides <NUM> can be stacked on each of the left and right waveguide mounting surfaces of the waveguide carrier <NUM>, for receiving light emitted from light engine <NUM>, and projecting such light into user's left eye <NUM> and right eye 202R of the user. Optical system <NUM> can be mounted to chassis <NUM> through a center tab <NUM> located at the top of waveguide carrier <NUM> over the central nose bridge portion <NUM>.

<FIG> depicts a waveguide 300R, which can be mounted on the waveguide carrier <NUM> to convey light to a particular eye of the user, in this example, the right eye of user. A similar waveguide can be designed for the left eye, for example, as a (horizontal) mirror image of the waveguide shown in <FIG>. The waveguide 300R can be transparent and, as can be seen from <FIG>, can be disposed directly in front of the right eye of the user during operation of NED device <NUM>, e.g., as one of the waveguides <NUM> in <FIG>. Waveguide 300R is therefore shown from the user's perspective during operation of NED device <NUM>.

Waveguide 300R can include a single optical input port 302R (also called an in-coupling element) located in the region of the waveguide 300R that may be closest to the user's nose bridge when NED device <NUM> is worn by the user. In certain embodiments the input port 302R can be a diffractive optical element such as a surface relief grating, or can be an EADM.

Waveguide 300R can further include a transmission channel 304R and an optical output port 306R (also called out-coupling element). As with input port 302R, in certain implementations, output port 306R can be a DOE, such as an SRG, or can be an EADM. An optical coupling (not shown) can provide light output from display engine <NUM> to input port 302R of waveguide <NUM> during operation.

Transmission channel 304R may be a transmissive substrate that can convey light from the input port 302R to output port 306R. In some implementations, transmission channel 304R may be a DOE, such as an SRG, or a reflective component such as a substrate with multiple internally reflective surfaces. The transmission channel 304R a may be configured to transmit light by use of TIR, such that light waves received at input port 302R can be transmitted to output port 306R. Light waves representing an image for the right eye may then be projected from output port 306R to the user's eye.

<FIG> depicts waveguide 300R when placed within an alternative NED device 100A. As depicted in <FIG>, waveguide 300R may have a mirror waveguide that provides similar functionality for the left eye, via waveguide <NUM>. Similar to waveguide 300R, waveguide <NUM> also includes an input port <NUM>, a transmission channel <NUM>, and an output port <NUM> for enabling projection of light waves to the user's eye.

<FIG> illustrates the propagation of light waves using in-coupling and out-coupling elements via a waveguide, such as waveguide <NUM>, whereby an image <NUM> that has been compressed by precompensation renderer <NUM> can be propagated to user's eye, such as user's eye <NUM>. As depicted, waveguide <NUM> may include a transmissive substrate <NUM>, which is manifest as an example transmission channel <NUM> with surface 506A and surface 506B that may be substantially parallel to each other and that may be internally reflective so as to provide TIR of light waves <NUM> propagating within transmissive substrate <NUM>. Light waves <NUM> associated with image <NUM> may be generated by light engine <NUM>, and sent to display engine <NUM>. Waveguide <NUM> may also include an in-coupling element <NUM>, which is manifest as an example input port <NUM>, and may be configured to input light waves <NUM> to transmissive substrate <NUM>, for example, by deflecting the light waves <NUM> at an angle suitable for TIR.

Waveguide <NUM> can also include an out-coupling element <NUM>, which is manifest as an example output port <NUM>, and may be configured to output light waves <NUM> from transmissive substrate <NUM>. As previously discussed, in-coupling element <NUM> and out-coupling element <NUM> may in some cases include DOEs such as an SRG, or may be an EADM, which may be formed as part of or proximate to a given surface (i.e., a surface parallel to the direction of propagation of the light waves within the substrate) of transmissive substrate <NUM>. For example, the embodiment illustrated in <FIG> depicted in-coupling element <NUM> and out-coupling element <NUM> formed on surface 506A of transmissive substrate <NUM>, or may be located immediately proximate to surface 506A, such as within one micrometer from the surface.

Transmissive substrate <NUM> can be made of any material or combination of materials with appropriate optical properties to facilitate light propagation by TIR. In some embodiments, transmissive substrate <NUM> can be made of optical-grade glass, for example, formed through an injection molding process. The glass used to form transmissive substrate <NUM> can, in some implementations, include silicon dioxide (SiO2). Alternatively, in other implementations, transmissive substrate <NUM> may be formed of a polymer resin.

<FIG> depicts the processing of an image by NED device <NUM> according to one implementation. An optical assembly <NUM>, such as a MEMS mirror-based display engine, may generate an image <NUM> that is to be transmitted via a waveguide <NUM> such that the image may be propagated to a user's eye <NUM>. In some implementations, waveguide <NUM> may be of a low index transmissive substrate, such as glass, that may be limited in its ability to guide certain light waves according to TIR.

Specifically, in one implementation, waveguide <NUM> may have a minimum TIR angle 610A and a maximum TIR angle 610B, which are the critical guide angles associated with the particular index of waveguide <NUM>. That is, minimum TIR angle 610A and maximum TIR angle 610B define the bounds of the field of view that can be propagated via TIR through waveguide <NUM>. Minimum TIR angle 610A may be approximately equal to <MAT>, where n is the refractive index of waveguide <NUM>, and maximum TIR angle 610B may be approximately <NUM> degrees for waveguide <NUM>. The FOV of a resulting image can therefore be based on the difference between the angles of incidence of the incoming light waves output from optical assembly <NUM> and the minimum and maximum TIR angles for waveguide <NUM>, and if incoming light waves exhibit a FOV that exceeds than the minimum and maximum TIR angles, certain portions of the resulting image can be cropped following out-coupling of the image from the waveguide.

While the use of EADM couplers may assist in compressing a wide FOV image for transmission through a lower FOV waveguide, as discussed earlier, the use of EADMs for both in-coupling and out-coupling introduces difficulties in faithfully reproducing an exact copy of the input image, due to difficulties in creating symmetrical metasurfaces for the EADMs. Furthermore, the optical assembly associated with a device that utilizes EADMs for both in-coupling and out-coupling must be larger and more expensive in order to house a display engine that can generate the large FOV for the initial image.

To address such difficulties, and because an image having a wide FOV may be outside of the critical TIR angles for waveguide <NUM>, precompensation by optical assembly <NUM> can be utilized. Specifically, the image generated by optical assembly <NUM> may be generated in a compressed fashion, and the FOV of the compressed image can be smaller than the wide FOV of the image that is ultimately output and displayed via NED device <NUM>. In one implementation, the painted image generated by optical assembly <NUM> may depict a base image that, upon generation, appears as if compression has been applied to the image. Therefore, as depicted in <FIG>, image <NUM> may be perceived as distorted if viewed directly from the output of optical assembly <NUM>. However, as a user of NED device <NUM> is only intended to see the final image projected on user's eye <NUM>, the distortion of image <NUM> will not be noticeable to the user.

Image <NUM>, in its compressed form, can then be provided to in-coupler <NUM>, which may be a standard grating in-coupler, such as an SRG, without the necessity of utilizing metasurfaces for the in-coupler because the image <NUM> is already within the critical TIR angles of waveguide <NUM>. Image <NUM> can then be transferred through waveguide <NUM> by TIR, and can be provided to out-coupler <NUM>. Out-coupler <NUM> may be an EADM that is capable of performing FOV expansion, such as by use of a linear momentum change. Furthermore, in some implementations, the EADM may utilize a non-linear momentum change, as will be discussed in further detail below. As a result of image <NUM> being expanded by out-coupler <NUM>, expanded image <NUM> can be created, which can be propagated to user's eye <NUM> in order to provide a wide FOV image that does not have any distortion.

As such, the generation of a compressed image <NUM> allows the use of a standard in-coupler for in-coupler <NUM>, reducing the requirement that the dual EADM in-coupler and out-coupler configurations be completely symmetric in their metasurface architecture. Furthermore, because optical assembly <NUM> does not need to produce a wide FOV and instead paints a compressed image that can later be expanded by out-coupler <NUM>, optical system <NUM> can be of a smaller size than what would necessary to create the larger FOV image, which may result in cost savings, a reduction in size, and an increase in reliability. For example, when optical assembly <NUM> is a MEMS-based scanning system, the mirror swing angle can be reduced as a result of the methods described above.

In another implementation, the generation of a compressed image <NUM> can be used to compensate for distortions, amplitude (brightness), or geometrical aberrations imparted by components of the device, such as optical assembly <NUM>, or by one of in-coupler <NUM> or out-coupler <NUM>, or both. For example, if it is known that particular in-couplers or out-couplers yield certain aberrations, optical assembly <NUM> may precompensate for such aberrations when generating compressed image <NUM>, in addition to the FOV compression being applied to the image.

<FIG> depict example angular momentum processing that can be performed by NED device <NUM> based on the type of in-couplers and out-couplers used in the connection with the waveguides of the device. As depicted in <FIG>, graph <NUM> shows for a given angular momentum associated with a light wave <NUM> the use of an EADM coupler vs. the use of a standard grating, such as an SRG. Specifically, graph <NUM> depicts an EADM linear dispersion line <NUM>, where a momentum change, or angular "kick," can be provided as a linear function of the angle of incidence. This is contrasted with a standard grating dispersion line <NUM>, which may define the dispersion associated with an SRG, where no additional angular "kick" is provided. The angular "kick" enables the ability to compress and expand light waves that are received by the couplers in accordance with the discussion provided above with regard to the compression and expansion of images through a waveguide.

As noted earlier, for a standard grating, an incoming light wave at incident angle θ to the normal would receive a constant angular change β, and would therefore exit an SRG at an angle θ + β, represented by standard grating dispersion line <NUM>. In contrast, an EADM can provide a compression/expansion ratio term which can be represented by a constant multiplicative term µ, such that for incident angles greater in magnitude, the angular "kick" would be greater. Upon interaction with the nanostructures, the outgoing angle would be µθ + β, which can result in an approximately linear relationship between incoming and outgoing angles. For example, portion <NUM> indicates that negative angles may be less deflected by use of a lesser term µ, while portion <NUM> indicates that positive angles can be more deflected by use of a higher term µ.

While a linear function can provide useful compression and expansion of images, another implementation can utilize a non-linearly deflecting EADM grating for additional benefits. For example, <FIG> depicts a graph <NUM>, where for a given light wave <NUM>, an EADM non-linear dispersion line <NUM> is shown, where the angular "kick" changes non-linearly over the angle of incidence, in contrast to standard grating dispersion line <NUM>. Using a non-linear function can involve the use of a non-constant multiplicative term µ, and leveraging a non-linear function f(θ), such that the compression/expansion ratio is not constant over the range of incident angles, resulting in an angular "kick" that varies over incident angle non-linearly. The outgoing angle can therefore be represented by f(θ) + β, where f(θ) is non-linear. As a result, portions <NUM> and <NUM> depict that near the edges of the FOV, the magnitude of the "kick" can be larger resulting in the periphery of the field of view being stretched, while portion <NUM> indicates that near the center of the FOV, there is less of an angular "kick.

The use of a non-linearly deflecting EADM can provide certain benefits over the linearly deflecting EADM discussed earlier. For example, non-linearly deflecting EADMs can be used to create a foveated image, and can also be used to compensate for distortion that may be introduced from a display engine, each of which will be described in further detail below. Non-linearly deflecting EADMs can, in certain implementations, be used either as an in-coupler or an out-coupler, or both. Furthermore, in certain implementations, the implementation of a non-linear function can be combined with the earlier discussed compression/expansion feature to be able to achieve larger FOVs without having to increase the refractive index of the waveguide substrate.

<FIG> depicts the processing of an image by NED device <NUM> according to one implementation using a non-linear function. An optical assembly <NUM> may generate an image <NUM> that is to be transmitted via a waveguide <NUM> such that the image may be propagated to a user's eye <NUM>. In some implementations, waveguide <NUM> may be of a low index substrate, such as glass, that may be limited in its ability to guide certain light waves according to TIR.

Similar to the discussion earlier regarding <FIG>, waveguide <NUM> may have a minimum TIR angle 910A and a maximum TIR angle 910B. As discussed, the FOV of a resulting image can therefore be based on the difference between the angles of incidence of the incoming light waves output from optical assembly <NUM> and the minimum and maximum TIR angles for waveguide <NUM>, and if incoming light waves have a higher angle of incidence than the minimum and maximum TIR angles, certain portions of the resulting image can be cropped following out-coupling of the image from the waveguide.

As such, to avoid cropping of the image and to ensure that a wide FOV can be produced, image <NUM> can be generated by optical assembly <NUM> to fit within the critical angles of waveguide <NUM>. Specifically, in certain implementations, optical assembly <NUM> may generate image <NUM> as an image having a FOV that is within the limits of the critical angles of waveguide <NUM> without any compression applied to the image, such as by precompensation processing discussed earlier. As such, as depicted in <FIG>, image <NUM> may be an uncompressed image that does not exhibit distortion, in contrast to image <NUM>.

Image <NUM>, in its uncompressed form, can then be provided to in-coupler <NUM>, which may be a standard grating in-coupler, such as an SRG that utilizes a constant angular "kick," without the necessity of utilizing metasurfaces for the in-coupler because the image <NUM> is already within the critical TIR angles of waveguide <NUM>. Image <NUM> can then be transferred through waveguide <NUM> by TIR, and can be provided to out-coupler <NUM>. Out-coupler <NUM> may be an EADM that utilizes a non-linear function to provide an angular "kick" for purposes of image expansion and to introduce foveation to image <NUM>, resulting in foveated image <NUM>, which can be propagated to user's eye <NUM>. Foveated image <NUM> may exhibit different resolutions at different portions of the image.

Image foveation can be beneficial because visual perception toward the edge of a user's FOV drops significantly in resolution. As such, by concentrating display resolution within the center portion of the FOV and reducing display resolution toward edges, a larger FOV and/or effective resolution can be achieved with a reduction in computational requirements. As depicted by foveated image <NUM>, a pincushion distortion optical function in out-coupler <NUM> can be used to achieve this foveation effect, where foveated image <NUM> displays portions of the image in the center in higher resolution (e.g., the "I" is displayed in normal size and resolution) while the edges of the image may appear stretched or elongated (e.g., the "F" and "E" may be displayed in such a manner as to appear stretched on the outer periphery of the image) or of lower resolution. Additionally, such a pincushion distortion optical function can be used to compensate for display parallax, which can tend to pack more pixels under the same angular range when a user looks at the waveguide under a large angle, which is known as the cosine effect.

While <FIG> depicts image <NUM> as being in an uncompressed state for ease of understanding, it is to be appreciated that image <NUM> may also be pre-warped by optical assembly <NUM>, such that upon stretching by out-coupler <NUM>, foveated image <NUM> would be correctly projected for propagation to user's eye <NUM>.

<FIG> depicts the processing of an image by NED device <NUM> according to another implementation using a non-linear function. An optical assembly <NUM> may generate an image <NUM> that is to be transmitted via a waveguide <NUM> such that the image may be propagated to a user's eye <NUM>, within minimum TIR angle 1010A and maximum TIR angle 1010B of waveguide <NUM>.

As depicted in <FIG>, image <NUM> may have image distortion introduced by optical assembly <NUM>. For example, as depicted, barrel distortion from a MEMS-based scanning mirror light injector can occur, as in some implementations, a mirror driven with a sinusoidal angular velocity can move fastest in the center of the FOV and slowest toward the edges of the FOV. Therefore, as pixels are raster scanned in the center of the image, the light source needs to modulate more quickly to achieve the same or higher resolution in the center of the FOV than towards the edges of the FOV. Therefore, the use of the non-linearly deflecting EADM as an in-coupler can be used to compensative for such image distortions.

As such, to correct such image distortions, image <NUM> can be provided to in-coupler <NUM>, which may be an EADM in-coupler that utilizes a non-linear function to compensate for barrel distortion from optical assembly <NUM>. Image <NUM> can then be transferred through waveguide <NUM> by TIR, and can be provided to out-coupler <NUM>. Out-coupler <NUM> may be a standard grating in-coupler, such as an SRG, which can produce corrected image <NUM> for propagation to user's eye <NUM>.

Furthermore, even in instances where image <NUM> does not exhibit distortion, processing by an EADM utilizing such a non-linear function can be beneficial as this may allow for reduction of the modulation frequency of the light source in the middle of the FOV. This can therefore result in a lowering of required computational resources, lowered computational latency, and can also allow for the light source to be located farther away from the driver.

<FIG> depicts the processing of an image by NED device <NUM> according to another implementation using a non-linear function, and illustrates a combination of concepts set forth in <FIG> and <FIG>. An optical assembly <NUM> may generate an image <NUM> that is to be transmitted via a waveguide <NUM> such that the image may be propagated to a user's eye <NUM>, within minimum TIR angle 1110A and maximum TIR angle 1110B of waveguide <NUM>.

As depicted in <FIG>, image <NUM> may have image distortion introduced by optical assembly <NUM>. For example, as depicted, barrel distortion from a MEMS-based scanning mirror light injector can occur, as set forth above regarding <FIG>. Therefore, the use of the non-linearly deflecting EADM as an in-coupler can be used to compensative for such image distortions. As such, to correct such image distortions, image <NUM> can be provided to in-coupler <NUM>, which may be an EADM in-coupler that utilizes a non-linear function to compensate for barrel distortion from optical assembly <NUM>. Image <NUM> can then be transferred through waveguide <NUM> by TIR, and can be provided to out-coupler <NUM>.

Out-coupler <NUM> may be an EADM that utilizes a non-linear function to provide an angular "kick" for purposes of image expansion and to introduce foveation to image <NUM>, resulting in foveated image <NUM>, which can be propagated to user's eye <NUM>. Foveated image <NUM> may exhibit different resolutions at different portions of the image, but as a result of the distortion correction provided by in-coupler <NUM>, does not exhibit the image distortion introduced by optical assembly <NUM>.

The following discussion presents an overview of precompensation functionality described above. <FIG> illustrates an example method <NUM>, consistent with the present concepts. Method <NUM> can be implemented by a single device, e.g., NED device <NUM>, or various steps can be distributed over one or more servers, client devices, etc. Moreover, method <NUM> can be performed by one or more components, such as precompensation renderer <NUM>, light engine <NUM>, display engine <NUM>, and controller <NUM> of <FIG>, and/or by other components.

At block <NUM>, precompensation renderer <NUM> may generate a compressed image, which in some implementations, may be based on a base image that has a wide FOV, and the compressed image can have a reduced FOV that is within the minimum and maximum TIR angles associated with a waveguide of NED device <NUM>. In certain implementations, the amount of compression applied to the base image can be dependent on the material of the waveguide, and compression can be controlled until the FOV is within the critical TIR angles associated with the waveguide material. For example, NED device <NUM> may need to depict a particular image to a user of the device, and the image may have a wide field of view, such as <NUM> degrees. However, the waveguide material may not be able to transmit the <NUM>-degree image, and therefore, the <NUM>-degree image could be compressed to have a <NUM>-degree field of view, which would fit within the critical TIR angles of the waveguide.

At block <NUM>, the generated image can be provided to light engine <NUM>, which may produce light waves associated with the generated image.

At block <NUM>, the generated image can be coupled into the in-coupler of the waveguide. In certain implementations, the in-coupler of the waveguide may be a standard grating, such as a surface relief grating, that provides a constant deflection angle over all angles of incidence associated with the generated image.

At block <NUM>, the generated image can be propagated via a transmissive substrate of the waveguide. For example, the generated image can be propagated by TIR, which can propagate light waves from the in-coupler to the out-coupler of the waveguide.

At block <NUM>, the generated image can be coupled into the out-coupler of the waveguide for field of view expansion. Specifically, the out-coupler may be a metasurface grating, such as an EADM, which can provide a non-constant momentum "kick" to the light waves, in order to expand the field of view associated with the generated image, and therefore recreate the wide field of view that was originally processed by precompensation renderer <NUM>.

Finally, at block <NUM>, the recreated wide field of view image can be propagated to enable displaying of the image to a user of NED device <NUM>.

The following discussion presents an overview of non-linear angular momentum processing described above. <FIG> illustrates an example method <NUM>, consistent with the present concepts. Method <NUM> can be implemented by a single device, e.g., NED device <NUM>, or various steps can be distributed over one or more servers, client devices, etc. Moreover, method <NUM> can be performed by one or more components, such as precompensation renderer <NUM>, light engine <NUM>, display engine <NUM>, and controller <NUM> of <FIG>, and/or by other components.

At block <NUM>, light engine <NUM> receive light waves that represent an image. For example, NED device <NUM> may need to depict a particular image to a user of the device. Light engine <NUM> may therefore produce light waves that depict the particular image, and the light waves can be propagated to display engine <NUM> for relay of the light waves.

At block <NUM>, display engine <NUM> may generate an image based on the light waves propagated from light engine <NUM>. In certain instances, display engine <NUM> may introduce distortion to the generated image. For example, display engine <NUM> may introduce barrel distortion to the image. However, it is to be appreciated that display engine <NUM> may be capable of producing an undistorted image.

At block <NUM>, the generated image may be modified by way of the in-coupler or the out-coupler of the waveguide. For example, in certain implementations, the in-coupler utilize a standard grating so as to provide a constant momentum change to the generated image, and the out-coupler may utilize a metasurface grating that can provide a non-linear angular "kick" to the light waves associated with the generated image. In another implementation, the in-coupler may utilize a metasurface grating to provide the non-linear angular "kick," while the out-coupler utilizes a standard grating. In another implementation, both the in-coupler and the out-coupler may utilize a metasurface grating so as to provide non-linear angular "kicks" at both points. As discussed above, the non-linear angular "kick" provided by either the in-coupler or the out-coupler, or both, can be utilized to correct barrel distortion associated with the generated image, or to introduce image foveation to the generated image, resulting in a modified image.

Finally, at block <NUM>, the modified image can be propagated to enable displaying of the image to a user of NED device <NUM>.

As noted above with respect to <FIG>, NED device <NUM> may include several components and/or devices, including an optical system <NUM>, and a controller <NUM>. As also noted, not all device implementations can be illustrated, and other device implementations should be apparent to the skilled artisan from the description above and below.

The term "device", "computer," "computing device," "client device," "server," and or "server device" as possibly used herein can mean any type of device that has some amount of hardware processing capability and/or hardware storage/memory capability. Processing capability can be provided by one or more hardware processors (e.g., hardware processing units/cores) that can execute computer-readable instructions to provide functionality. Computer-readable instructions and/or data can be stored on persistent storage or volatile memory. The term "system" as used herein can refer to a single device, multiple devices, etc. For example, a "precompensation system" can include one or more devices that perform precompensation, such as processing performed by NED device <NUM>, or via devices externally connected to NED device <NUM>.

Memory <NUM> can be storage resources that are internal or external to any respective devices with which it is associated. Memory <NUM> can include any one or more of volatile or non-volatile memory, hard drives, flash storage devices, and/or optical storage devices (e.g., CDs, DVDs, etc.), among others. As used herein, the term "computer-readable media" can include signals. In contrast, the term "computer-readable storage media" excludes signals. Computer-readable storage media includes "computer-readable storage devices. " Examples of computer-readable storage devices include volatile storage media, such as RAM, and non-volatile storage media, such as hard drives, optical discs, and flash memory, among others, which may constitute memory <NUM>.

In some cases, the devices are configured with a general-purpose hardware processor and storage resources. In other cases, a device can include a system on a chip (SOC) type design. In SOC design implementations, functionality provided by the device can be integrated on a single SOC or multiple coupled SOCs. One or more associated processors can be configured to coordinate with shared resources, such as memory, storage, etc., and/or one or more dedicated resources, such as hardware blocks configured to perform certain specific functionality. Thus, the term "processor," "hardware processor" or "hardware processing unit" as used herein can also refer to central processing units (CPUs), graphical processing units (GPUs), controllers, microcontrollers, processor cores, or other types of processing devices suitable for implementation both in conventional computing architectures as well as SOC designs.

In some configurations, any of the modules/code discussed herein can be implemented in software, hardware, and/or firmware. In any case, the modules/code can be provided during manufacture of the device or by an intermediary that prepares the device for sale to the end user. In other instances, the end user may install these modules/code later, such as by downloading executable code and installing the executable code on the corresponding device.

Also note that the components and/or devices described herein can function in a stand-alone or cooperative manner to implement the described techniques. For example, the methods described herein can be performed on a single computing device and/or distributed across multiple computing devices that communicate over one or more network(s). Without limitation, such one or more network(s) can include one or more local area networks (LANs), wide area networks (WANs), the Internet, and the like.

Various device examples are described above. Additional examples are described below. One example includes a device comprising an optical assembly configured to generate an image for propagation from a first point of the device to a second point of the device via a waveguide having a transmissive substrate, an in-coupler, and an out-coupler, the in-coupler comprising a deflection grating that provides a constant deflection angle for angles of incidence associated with the generated image and being configured to propagate the generated image through the transmissive substrate of the waveguide to the out-coupler, and the out-coupler comprising a metasurface grating that provides a non-constant deflection angle for angles of incidence associated with the generated image, the metasurface grating increasing a field of view (FOV) of the generated image.

Another example can include any of the above and/or below examples where the device further comprises a light engine configured to generate light waves associated with a wide-FOV image.

Another example can include any of the above and/or below examples where the wide-FOV image has a FOV that is larger than minimum and maximum TIR angles associated with the transmissive substrate of the waveguide.

Another example can include any of the above and/or below examples where the generated image is generated by compressing the wide-FOV image by the optical assembly such that the FOV of the generated image is within the minimum and maximum TIR angles associated with the transmissive substrate of the waveguide.

Another example can include any of the above and/or below examples where the out-coupler expands the generated image to produce an expanded image having a FOV equal to the wide-FOV image.

Another example can include any of the above and/or below examples where the minimum and maximum TIR angles are based on a refractive index of the transmissive substrate of the waveguide.

Another example can include any of the above and/or below examples where wherein the transmissive substrate is a glass or polymer.

Another example can include any of the above and/or below examples where the in-coupler is a surface relief grating.

Another example can include any of the above and/or below examples where the out-coupler is a metasurface grating comprising a plurality of nanoscale elements.

Another example can include any of the above and/or below examples where generated image is propagated through the waveguide by total internal reflection (TIR).

Another example can include any of the above and/or below examples where the optical assembly is a MEMS-based scanning system.

Another example includes a method comprising generating an image that has a compressed field of view (FOV) based on a wide-FOV image, coupling the generated image into an in-coupler of a waveguide, the in-coupler having a first grating, propagating the generated image via a transmissive substrate of the waveguide, coupling the generated image into an out-coupler of the waveguide, the out-coupler having a second grating that provides FOV expansion to recreate the wide-FOV image, and propagating the recreated wide-FOV image for display.

Another example can include any of the above and/or below examples where the method further comprising generating light waves representing the generated image.

Another example can include any of the above and/or below examples where the wide-FOV image has a FOV that exceeds minimum and maximum FOV angles of the transmissive substrate of the waveguide.

Another example can include any of the above and/or below examples where the first grating is a surface relief grating.

Another example can include any of the above and/or below examples where the second grating is a metasurface grating comprising a plurality of nanoscale elements.

Another example includes a device comprising a renderer configured to process an image having a first field of view (FOV) to generate an image having a second FOV, a light engine configured to produce light waves based on the generated image having the second FOV, and a waveguide in-coupler configured to couple the produced light waves to a waveguide out-coupler via a transmissive substrate of the waveguide, the waveguide out-coupler configured to expand the generated image having the second FOV to recreate the first FOV.

Another example can include any of the above and/or below examples where the first FOV is wider than minimum and maximum FOV angles of the transmissive substrate of the waveguide, and the second FOV is within the minimum and maximum FOV angles.

Another example can include any of the above and/or below examples where the waveguide out-coupler is a metasurface grating comprising a plurality of nanoscale elements.

Another example can include any of the above and/or below examples where the waveguide in-coupler is a surface relief grating.

Claim 1:
A device comprising:
a precompensation renderer (<NUM>) configured to precompensate an image with a wide field of view, FOV, into a compressed image having a compressed FOV for coupling to a waveguide (<NUM>);
a light engine (<NUM>) configured to produce light waves corresponding to the compressed FOV-image, an optical assembly (<NUM>); and
said waveguide(<NUM>) having a transmissive substrate, an in-coupler, and an out-coupler; wherein the optical assembly (<NUM>) is configured to introduce distortion to the image having the compressed FOV, thereby generating an image precompensating for aberrations imparted by components of the device during propagation via the waveguide (<NUM>);
the in-coupler (<NUM>) comprising a deflection grating that is configured to provide a constant deflection angle for angles of incidence associated with the image from the optical assembly and being configured to propagate the generated image through the transmissive substrate of the waveguide (<NUM>) to the out-coupler (<NUM>); and
the out-coupler (<NUM>) comprising a metasurface grating that provides a non-constant deflection angle for angles of incidence associated with the generated image, the metasurface grating increasing a FOV of the generated image.