Patent Description:
The present invention generally relates to systems and methods configured to facilitate interactive virtual or augmented reality environments for one or more users.

Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences, wherein digitally reproduced images or portions thereof are presented to a user in a manner where they seem to be, or may be perceived as, real. Document <CIT> is an example of an optical device that can be used in such applications. A virtual reality (VR) scenario typically involves presentation of digital or virtual image information without transparency to other actual real-world visual input, whereas an augmented reality (AR) scenario typically involves presentation of digital or virtual image information as an augmentation to visualization of the actual world around the end user.

For example, referring to <FIG>, an augmented reality scene <NUM> is depicted wherein a user of an AR technology sees a real-world park-like setting <NUM> featuring people, trees, buildings in the background, and a concrete platform <NUM>. In addition to these items, the end user of the AR technology also perceives that he "sees" a robot statue <NUM> standing upon the real-world platform <NUM>, and a cartoon-like avatar character <NUM> flying by which seems to be a personification of a bumble bee, even though these elements <NUM>, <NUM> do not exist in the real world. As it turns out, the human visual perception system is very complex, and producing a VR or AR technology that facilitates a comfortable, natural-feeling, rich presentation of virtual image elements amongst other virtual or real-world imagery elements is challenging.

VR and AR systems typically employ head-worn displays (or helmet-mounted displays, or smart glasses) that are at least loosely coupled to a user's head, and thus move when the end user's head moves. If the end user's head motions are detected by the display subsystem, the data being displayed can be updated to take the change in head pose (i.e., the orientation and/or location of user's head) into account.

As an example, if a user wearing a head-worn display views a virtual representation of a three-dimensional (3D) object on the display and walks around the area where the 3D object appears, that 3D object can be re-rendered for each viewpoint, giving the end user the perception that he or she is walking around an object that occupies real space. If the head-worn display is used to present multiple objects within a virtual space (for instance, a rich virtual world), measurements of head pose can be used to re-render the scene to match the end user's dynamically changing head location and orientation and provide an increased sense of immersion in the virtual space.

Head-worn displays that enable AR (i.e., the concurrent viewing of real and virtual elements) can have several different types of configurations. In one such configuration, often referred to as a "video see-through" display, a camera captures elements of a real scene, a computing system superimposes virtual elements onto the captured real scene, and a non-transparent display presents the composite image to the eyes. Another configuration is often referred to as an "optical see-through" display, in which the end user can see through transparent (or semi-transparent) elements in the display subsystem to view directly the light from real objects in the environment. The transparent element, often referred to as a "combiner," superimposes light from the display over the end user's view of the real world.

VR and AR systems typically employ a display subsystem having a projection subsystem and a display surface positioned in front of the end user's field of view and on which the projection subsystem sequentially projects image frames. In true three-dimensional systems, the depth of the display surface can be controlled at frame rates or sub-frame rates. The projection subsystem may include one or more optical fibers into which light from one or more light sources emit light of different colors in defined patterns, and a scanning device that scans the optical fiber(s) in a predetermined pattern to create the image frames that sequentially displayed to the end user.

In one embodiment, the display subsystem includes one or more planar optical waveguides that are generally parallel to the field of view of the user, and into which light from the optical fiber(s) is injected. One or more linear diffraction gratings are embedded within the waveguide(s) to change the angle of incident light propagating along the waveguide(s). By changing the angle of light beyond the threshold of total internal reflection (TIR), the light escapes from one or more lateral faces of the waveguide(s). The linear diffraction grating(s) have a low diffraction efficiency, so only a fraction of the light energy is directed out of the waveguide(s), each time the light encounters the linear diffraction grating(s). By outcoupling the light at multiple locations along the grating(s), the exit pupil of the display subsystem is effectively increased. The display subsystem may further comprise one or more collimation elements that collimate light coming from the optical fiber(s), and an optical input apparatus that optically couples the collimated light to, or from, an edge of the waveguide(s).

With reference to <FIG>, one embodiment of a display subsystem <NUM> comprises one or more light sources <NUM> that generates light, an optical fiber <NUM> that emits the light, and a collimation element <NUM> that collimates the light exiting the distal end of the optical fiber <NUM> into a light beam <NUM>. The display subsystem <NUM> further comprises a piezoelectric element <NUM> to or in which the optical fiber <NUM> is mounted as a fixed-free flexible cantilever, and drive electronics <NUM> electrically coupled to the piezoelectric element <NUM> to activate electrically stimulate the piezoelectric element <NUM>, thereby causing the distal end of the optical fiber <NUM> to vibrate in a pre-determined scan pattern that creates deflections <NUM> about a fulcrum <NUM>, thereby scanning the collimated light beam <NUM> in accordance with the scan pattern.

The display subsystem <NUM> comprises a waveguide apparatus <NUM> that includes a planar optical waveguide <NUM> that is generally parallel to the field-of-view of the end user, a diffractive optical element (DOE) <NUM> associated with the planar optical waveguides <NUM>, and in-coupling element (ICE) <NUM> (which take the form of a DOE) integrated within the end of the planar optical waveguide <NUM>. The ICE <NUM> in-couples and redirects the collimated light <NUM> from the collimation element <NUM> into the planar optical waveguide <NUM>. The collimated light beam <NUM> from the collimation element <NUM> propagates along the planar optical waveguide <NUM> and intersects with the DOE <NUM>, causing a portion of the light to exit the face of the waveguide apparatus <NUM> as light rays <NUM> towards the eyes of the end user that are focused at a viewing distance depending on the lensing factor of the planar optical waveguide <NUM>. Thus, the light source(s) <NUM> in conjunction with the drive electronics <NUM> generate image data encoded in the form of light that is spatially and/or temporally varying.

The location of each pixel visualized by the end user is highly dependent on the angle of the light rays <NUM> that exit the planar optical waveguide <NUM>. Thus, light rays <NUM> that exit the waveguide <NUM> at different angles will create pixels at different locations in the field of view of the end user. For example, if it is desired to locate a pixel at the top right of the field of view of the end user, a collimated light beam <NUM> may be input into the waveguide apparatus <NUM> at one angle, and if is desired to locate a pixel at the center of the field of view of the end user, the collimated light beam <NUM> may be input into the waveguide apparatus <NUM> at a second different angle. Thus, as the optical fiber <NUM> is being scanned in accordance with a scan pattern, the light beam <NUM> originating from the optical fiber <NUM> will be input into the waveguide apparatus <NUM> at different angles, thereby creating pixels at different locations in the field of view of the end user. Thus, the location of each pixel in the field of view of the end user is highly dependent on the angle of the light rays <NUM> exiting the planar optical waveguide <NUM>, and thus, the locations of these pixels are encoded within the image data generated by the display subsystem <NUM>.

Although the angle of the light beam <NUM> entering the waveguide apparatus <NUM>, and thus, the angle of the light beam <NUM> entering the planar optical waveguide <NUM> will differ from the angles of the light rays <NUM> exiting the planar optical waveguide <NUM>, the relationships between the angle of the light beam <NUM> entering the waveguide apparatus <NUM> and the angles of the light rays <NUM> exiting the planar optical waveguide <NUM> is well-known and predictable, and thus, the angles of the light rays <NUM> exiting the planar optical waveguide <NUM> can be easily predicted from the angle of the collimated light beam <NUM> entering the waveguide apparatus <NUM>.

It can be appreciated from the foregoing that the actual angles of the light beams <NUM> entering the waveguide apparatus <NUM> from the optical fiber <NUM>, and thus, the actual angles of the light rays <NUM> exiting the waveguide <NUM> towards the end user be identical or near identical or one-to-one in relationship to the designed angles of the exiting light rays <NUM>, such that the locations of the pixels visualized by the end user are properly encoded in the image data generated by the display subsystem <NUM>. However, due to manufacturing tolerances between different scanners, as well, as changing environmental conditions, such as variations in temperature that may change the consistency of bonding materials used to integrate the display subsystem <NUM> together, the actual angles of the exiting light rays <NUM>, without compensation, will vary from the designed angles of the exiting light rays <NUM>, thereby creating pixels that are in the incorrect locations within the field of view of the end user, resulting in image distortion.

There, thus, is a need to ensure that the actual angles of light rays exiting the waveguide of a display subsystem in a virtual reality or augmented reality environment are as close to identical to the designed angles encoded within the image data generated by the display subsystem.

Embodiments of the present invention are directed to devices, systems and methods for facilitating virtual reality and/or augmented reality interaction for one or more users.

In accordance with a first aspect of the present invention, a display subsystem for a virtual image generation system used by an end user is provided. The display subsystem comprises a waveguide apparatus. In one embodiment, the waveguide apparatus is configured for being positioned in front of the eyes of the end user. In another embodiment, the waveguide apparatus has a partially transparent display surface configured for being positioned in the field of view between the eyes of the end user and an ambient environment. In still another embodiment, the display subsystem comprises a frame structure configured for being worn by the end user. In this case, the frame structure carries the waveguide apparatus.

The display subsystem further comprises an imaging element configured for emitting light, and a collimation element configured for collimating the light from the imaging element into a light beam. In one embodiment, the imaging element comprises at least one light source configured for generating the light, an optical fiber configured for emitting the light, and a mechanical drive assembly to which the optical fiber is mounted. The mechanical drive assembly is configured for displacing the optical fiber in accordance with a scan pattern. In one embodiment, the mechanical drive assembly comprises a piezoelectric element to which the optical fiber is mounted, and drive electronics configured for conveying electrical signals to the piezoelectric element, thereby causing the optical fiber to vibrate in accordance with the scan pattern.

The display subsystem further comprises an in-coupling element (ICE) configured for directing the light beam from the collimation element down the waveguide apparatus, such that light rays exit the waveguide apparatus to display a pixel of an image frame to the end user. The pixel has a location encoded with angles of the exiting light rays. In one embodiment, the waveguide apparatus comprises a planar optical waveguide (e.g., one formed of a single pan of optically transparent material), in which case, the ICE is configured for optically coupling the collimated light beam from the imaging element into the planar optical waveguide as an in-coupled light beam. The waveguide apparatus may further comprise an orthogonal pupil expansion (OPE) element associated with the planar optical waveguide for splitting the in-coupled light beam into a plurality of orthogonal light beams, and an exit pupil expansion (EPE) element associated with the planar optical waveguide for splitting the plurality of orthogonal light beams into the light rays that exit the planar optical waveguide.

The display subsystem further comprises a sensing assembly configured for sensing at least one parameter indicative of at least one of the exiting light ray angles. In one embodiment, the sensed parameter(s) are indicative of the exiting light ray angle(s) projected in at least one plane (e.g., two orthogonal planes that are orthogonal to each other) that is orthogonal to a plane coincident with the exterior surface of the waveguide apparatus.

In another embodiment, the sensed parameter(s) comprises an intensity of at least one light ray representative of the plurality of exiting light rays. The representative light ray(s) may be different from the plurality of exiting light rays. In this case, the representative light ray(s) may exit the waveguide apparatus at a different location from the plurality of exiting light rays outside of the field of view of the end user. Alternatively, the plurality of exiting light rays may comprise the representative light ray(s).

In any event, the sensing assembly may comprise at least one angle sensor, each of which includes a photo-intensity sensor and an angle selective layer mounted between the waveguide apparatus and the photo-intensity sensor. In one embodiment, the angle sensor(s) comprise a pair of orthogonal sensors respectively configured for sensing first and second orthogonal intensity components of the representative light ray(s). The pair of orthogonal sensors may respectively comprise first and second cylindrical lenses configured for respectively passing the first and second orthogonal intensity components of the representative light ray(s). Or, the pair of orthogonal sensors respectively may comprise first and second diffractive optical elements configured for respectively passing the first and second orthogonal intensity components of the representative light ray(s). Or, the pair of orthogonal sensors may respectively comprise first and second polarization elements configured for respectively polarizing each of the representative light ray(s) into orthogonally polarized light rays. Or, the angle selective layers may be strained in orthogonal orientations.

In still another embodiment, the sensed parameter(s) may comprise an absolute intensity of the at least one representative light ray, such that the first and second orthogonal intensity components can be normalized. In this case, the sensing assembly may comprise another photo-intensity sensor configured for measuring the absolute intensity of the at least one representative light ray. In yet another embodiment, the sensed parameter(s) is indicative of relative angles of the plurality of exiting light rays. For example, the sensed parameter(s) may comprise a quadrant in which the collimated light beam is projected in a plane. In this case, the sensing assembly may comprise a plurality of sensors spaced apart in the quadrants of a reference plane or a quadrant position sensing detector (PSD).

In accordance with a second aspect of the present inventions, a virtual image generation system comprises the afore-described display subsystem, and a control subsystem configured for generating image data defining a location of the pixel, and controlling an angle of the light beam relative to the ICE based on the defined location of the pixel and the sensed parameter(s). The virtual image generation system may further comprise memory storing a three-dimensional scene, in which case, the control subsystem may be configured for rendering a plurality of synthetic image frames of the three-dimensional scene, and the display subsystem may be configured for sequentially displaying the plurality of image frames to the end user. The control subsystem may comprise a graphics processing unit (GPU).

In accordance with a third aspect of the present inventions, a virtual image generation system for use by an end user is provided. The virtual image generation system comprises a projection subsystem configured for generating a collimated light beam. In one embodiment, the projection subsystem comprises at least one light source configured for generating light, an optical fiber configured for emitting the light, a collimation element configured for collimating the light from the optical fiber into the collimated light beam, and a mechanical drive assembly to which the optical fiber is mounted. The mechanical drive assembly is configured for displacing the optical fiber in accordance with a scan pattern. In one embodiment, the mechanical drive assembly may comprise a piezoelectric element to which the optical fiber is mounted, and drive electronics configured for conveying electrical signals to the piezoelectric element, thereby causing the optical fiber to vibrate in accordance with the scan pattern.

The virtual image generation system further comprises a display configured emitting light rays in response to the collimated light beam to display a pixel of an image frame to the end user. The pixel has a location encoded with angles of the emitted light rays. In one embodiment, the display is configured for displaying the pixel of the image frame selectively at one of a plurality of different focal points to the end user. The display may be configured for being positioned in front of the eyes of the end user. The display may have a partially transparent display surface configured for being positioned in the field of view between the eyes of the end user and an ambient environment. In another embodiment, the virtual image generation system further comprises a frame structure configured for being worn by the end user, the frame structure carrying the display.

The virtual image generation system further comprises a sensing assembly configured for sensing at least one parameter indicative of the emitted light ray angle(s). In one embodiment, the parameter(s) sensed by the sensing assembly is indicative of the exiting light ray angle(s) projected in at least one plane (e.g., two orthogonal planes that are orthogonal to each other) that is orthogonal to a plane coincides with the exterior surface of the waveguide apparatus.

In another embodiment, the sensed parameter(s) are indicative of the emitted light ray angle(s) projected in at least one plane (e.g., two orthogonal planes that are orthogonal to each other) that is orthogonal to a plane coincident with the exterior surface of the waveguide apparatus.

In still another embodiment, the sensed parameter(s) comprises an intensity of at least one light ray representative of the plurality of emitted light rays. The representative light ray(s) may be different from the plurality of emitted light rays. In this case, the representative light ray(s) may exit the waveguide apparatus at a different location from the plurality of emitted light rays outside of the field of view of the end user. Alternatively, the plurality of emitted light rays may comprise the representative light ray(s).

In yet another embodiment, the sensed parameter(s) may comprise an absolute intensity of the at least one representative light ray, such that the first and second orthogonal intensity components can be normalized. In this case, the sensing assembly may comprise another photo-intensity sensor configured for measuring the absolute intensity of the at least one representative light ray. In yet another embodiment, the sensed parameter(s) is indicative of relative angles of the plurality of emitted light rays. For example, the sensed parameter(s) may comprise a quadrant in which the collimated light beam is projected in a plane. In this case, the sensing assembly may comprise a plurality of sensors spaced apart in the quadrants of a reference plane or a quadrant position sensing detector (PSD).

The virtual image generation system further comprises a control subsystem configured for generating image data defining a location of the pixel, and controlling an angle of the light beam relative to the display based on the defined location of the pixel and the sensed parameter(s). The virtual image generation system may further comprise memory storing a three-dimensional scene, in which case, the control subsystem may be configured for rendering a plurality of synthetic image frames of the three-dimensional scene, and the display may be configured for sequentially displaying the plurality of image frames to the end user. The control subsystem may comprise a graphics processing unit (GPU).

Additional and other objects, features, and advantages of the invention are described in the detail description, figures and claims.

The drawings illustrate the design and utility of preferred embodiments of the present invention, in which similar elements are referred to by common reference numerals. In order to better appreciate how the above-recited and other advantages and objects of the present inventions are obtained, a more particular description of the present inventions briefly described above will be rendered by reference to specific embodiments thereof, which are illustrated in the accompanying drawings. Understanding that these drawings depict only typical embodiments of the invention and are not therefore to be considered limiting of its scope, the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:.

The description that follows relates to display subsystems and methods to be used in virtual reality and/or augmented reality systems. However, it is to be understood that the while the invention lends itself well to applications in virtual or augmented reality systems, the invention, in its broadest aspects, may not be so limited.

Referring to <FIG>, one embodiment of a virtual image generation system <NUM> constructed in accordance with present inventions will now be described. The virtual image generation system <NUM> may be operated as an augmented reality subsystem, providing images of virtual objects intermixed with physical objects in a field of view of an end user <NUM>. There are two fundamental approaches when operating the virtual image generation system <NUM>. A first approach employs one or more imagers (e.g., cameras) to capture images of the ambient environment. The virtual image generation system <NUM> inter-mixes the virtual images into the data representing the images of the ambient environment. A second approach employs one or more at least partially transparent surfaces through which the ambient environment can be seen and onto which the virtual image generation system <NUM> produces images of virtual objects.

The virtual image generation system <NUM>, and the various techniques taught herein, may be employed in applications other than augmented reality and virtual reality subsystems. For example, various techniques may be applied to any projection or display subsystem, or may be applied to pico projectors where movement may be made by an end user's hand rather than the head. Thus, while often described herein in terms of an augmented reality subsystem or virtual reality subsystem, the teachings should not be limited to such subsystems of such uses.

At least for augmented reality applications, it may be desirable to spatially position various virtual objects relative to respective physical objects in a field of view of the end user <NUM>. Virtual objects, also referred to herein as virtual tags or tag or call outs, may take any of a large variety of forms, basically any variety of data, information, concept, or logical construct capable of being represented as an image. Non-limiting examples of virtual objects may include: a virtual text object, a virtual numeric object, a virtual alphanumeric object, a virtual tag object, a virtual field object, a virtual chart object, a virtual map object, a virtual instrumentation object, or a virtual visual representation of a physical object.

The virtual image generation system <NUM> comprises a frame structure <NUM> worn by an end user <NUM>, a display subsystem <NUM> carried by the frame structure <NUM>, such that the display subsystem <NUM> is positioned in front of the eyes <NUM> of the end user <NUM>, and a speaker <NUM> carried by the frame structure <NUM>, such that the speaker <NUM> is positioned adjacent the ear canal of the end user <NUM> (optionally, another speaker (not shown) is positioned adjacent the other ear canal of the end user <NUM> to provide for stereo/shapeable sound control). The display subsystem <NUM> is designed to present the eyes <NUM> of the end user <NUM> with photo-based radiation patterns that can be comfortably perceived as augmentations to physical reality, with high-levels of image quality and three-dimensional perception, as well as being capable of presenting two-dimensional content. The display subsystem <NUM> presents a sequence of frames at high frequency that provides the perception of a single coherent scene.

In the illustrated embodiment, the display subsystem <NUM> employs "optical see-through" display through which the user can directly view light from real objects via transparent (or semi-transparent) elements. The transparent element, often referred to as a "combiner," superimposes light from the display over the user's view of the real world. To this end, the display subsystem <NUM> comprises a projection subsystem <NUM> and a partially transparent display screen <NUM> on which the projection subsystem <NUM> projects images. The display screen <NUM> is positioned in the end user's <NUM> field of view between the eyes <NUM> of the end user <NUM> and an ambient environment, such that direct light from the ambient environment is transmitted through the display screen <NUM> to the eyes <NUM> of the end user <NUM>.

In the illustrated embodiment, the projection assembly <NUM> provides a scanned light to the partially transparent display screen <NUM>, thereby combining with the direct light from the ambient environment, and being transmitted from the display screen <NUM> to the eyes <NUM> of the user <NUM>. In the illustrated embodiment, the projection subsystem <NUM> takes the form of an optical fiber scan-based projection device, and the display screen <NUM> takes the form of a waveguide-based display into which the scanned light from the projection subsystem <NUM> is injected to produce, e.g., images at a single optical viewing distance closer than infinity (e.g., arm's length), images at multiple, discrete optical viewing distances or focal planes, and/or image layers stacked at multiple viewing distances or focal planes to represent volumetric 3D objects. These layers in the light field may be stacked closely enough together to appear continuous to the human visual subsystem (i.e., one layer is within the cone of confusion of an adjacent layer). Additionally or alternatively, picture elements may be blended across two or more layers to increase perceived continuity of transition between layers in the light field, even if those layers are more sparsely stacked (i.e., one layer is outside the cone of confusion of an adjacent layer). The display subsystem <NUM> may be monocular or binocular.

The virtual image generation system <NUM> further comprises one or more sensors (not shown) mounted to the frame structure <NUM> for detecting the position and movement of the head <NUM> of the end user <NUM> and/or the eye position and inter-ocular distance of the end user <NUM>. Such sensor(s) may include image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radio devices, and/or gyros).

The virtual image generation system <NUM> further comprises a user orientation detection module <NUM>. The user orientation module <NUM> detects the instantaneous position of the head <NUM> of the end user <NUM> and may predict the position of the head <NUM> of the end user <NUM> based on position data received from the sensor(s). Detecting the instantaneous position of the head <NUM> of the end user <NUM> facilitates determination of the specific actual object that the end user <NUM> is looking at, thereby providing an indication of the specific textual message to be generated for that actual object and further providing an indication of the textual region in which the textual message is to be streamed. The user orientation module <NUM> also tracks the eyes <NUM> of the end user <NUM> based on the tracking data received from the sensor(s).

The virtual image generation system <NUM> further comprises a control subsystem that may take any of a large variety of forms. The control subsystem includes a number of controllers, for instance one or more microcontrollers, microprocessors or central processing units (CPUs), digital signal processors, graphics processing units (GPUs), other integrated circuit controllers, such as application specific integrated circuits (ASICs), programmable gate arrays (PGAs), for instance field PGAs (FPGAs), and/or programmable logic controllers (PLUs).

The control subsystem of virtual image generation system <NUM> comprises a central processing unit (CPU) <NUM>, a graphics processing unit (GPU) <NUM>, one or more frame buffers <NUM>, and three-dimensional data base <NUM> for storing three-dimensional scene data. The CPU <NUM> controls overall operation, while the GPU <NUM> renders frames (i.e., translating a three-dimensional scene into a two-dimensional image) from the three-dimensional data stored in the three-dimensional data base <NUM> and stores these frames in the frame buffer(s) <NUM>. While not illustrated, one or more additional integrated circuits may control the reading into and/or reading out of frames from the frame buffer(s) <NUM> and operation of the projection assembly <NUM> of the display subsystem <NUM>.

More significant to the present inventions, the virtual image generation system <NUM> further comprises a light ray angle sensing assembly <NUM> that directly or indirectly senses the angle of one or more light rays exiting the display screen <NUM> towards the eyes <NUM> of the end user <NUM>. As will be described in further detail below, the desired location of each pixel of the image frame within the field of view of the end user <NUM> is highly correlated to the angles of the light rays exiting the display screen <NUM>, and thus, the sensed angles of the exiting light rays may be used to calibrate the display subsystem <NUM> in order to ensure that the actual angles of exiting light rays are as close to identical to the designed angles of the exiting light rays encoded within the image data generated by the display subsystem <NUM>.

The various processing components of the virtual image generation system <NUM> may be physically contained in a distributed subsystem. For example, as illustrated in <FIG>, the virtual image generation system <NUM> comprises a local processing and data module <NUM> operatively coupled, such as by a wired lead or wireless connectivity <NUM>, to the display subsystem <NUM> and sensors. The local processing and data module <NUM> may be mounted in a variety of configurations, such as fixedly attached to the frame structure <NUM> (<FIG>), fixedly attached to a helmet or hat <NUM> (<FIG>), embedded in headphones, removably attached to the torso <NUM> of the end user <NUM> (<FIG>), or removably attached to the hip <NUM> of the end user <NUM> in a belt-coupling style configuration (<FIG>). The virtual image generation system <NUM> further comprises a remote processing module <NUM> and remote data repository <NUM> operatively coupled, such as by a wired lead or wireless connectivity <NUM>, <NUM>, to the local processing and data module <NUM>, such that these remote modules <NUM>, <NUM> are operatively coupled to each other and available as resources to the local processing and data module <NUM>.

The local processing and data module <NUM> may comprise a power-efficient processor or controller, as well as digital memory, such as flash memory, both of which may be utilized to assist in the processing, caching, and storage of data captured from the sensors and/or acquired and/or processed using the remote processing module <NUM> and/or remote data repository <NUM>, possibly for passage to the display subsystem <NUM> after such processing or retrieval. The remote processing module <NUM> may comprise one or more relatively powerful processors or controllers configured to analyze and process data and/or image information. The remote data repository <NUM> may comprise a relatively large-scale digital data storage facility, which may be available through the internet or other networking configuration in a "cloud" resource configuration. In one embodiment, all data is stored and all computation is performed in the local processing and data module <NUM>, allowing fully autonomous use from any remote modules.

The couplings <NUM>, <NUM>, <NUM> between the various components described above may include one or more wired interfaces or ports for providing wires or optical communications, or one or more wireless interfaces or ports, such as via RF, microwave, and IR for providing wireless communications. In some implementations, all communications may be wired, while in other implementations all communications may be wireless. In still further implementations, the choice of wired and wireless communications may be different from that illustrated in <FIG>. Thus, the particular choice of wired or wireless communications should not be considered limiting.

In the illustrated embodiment, the user orientation module <NUM> is contained in the local processing and data module <NUM>, while CPU <NUM> and GPU <NUM> are contained in the remote processing module <NUM>, although in alternative embodiments, the CPU <NUM>, GPU <NUM>, or portions thereof may be contained in the local processing and data module <NUM>. The 3D database <NUM> can be associated with the remote data repository <NUM>.

Referring now to <FIG>, the projection assembly <NUM> includes one or more light sources <NUM> that produces the light (e.g., emits light of different colors in defined patterns). The light source(s) <NUM> may take any of a large variety of forms, for instance, a set of RGB lasers (e.g., laser diodes capable of outputting red, green, and blue light) operable to respectively produce red, green, and blue coherent collimated light according to defined pixel patterns specified in respective frames of pixel information or data. Laser light provides high color saturation and is highly energy efficient.

The projection assembly <NUM> further comprises a scanning device <NUM> that scans the light in a predetermined scan pattern in response to control signals. The scanning device <NUM> comprises one or more optical fibers <NUM> (e.g., single mode optical fiber), each of which has a proximal end 154a into which light is received from the light source(s) <NUM> and a distal end 154b from which light is provided to the display screen <NUM>. The scanning device <NUM> further comprises a mechanical drive assembly <NUM> to which the optical fiber(s) <NUM> is mounted. The drive assembly <NUM> is configured for displacing the distal end 154b of each optical fiber <NUM> about a fulcrum <NUM> in accordance with a scan pattern.

To this end, the drive assembly <NUM> comprises a piezoelectric element <NUM> to which the optical fiber(s) <NUM> is mounted, and drive electronics <NUM> configured for conveying electrical signals to the piezoelectric element <NUM>, thereby causing the distal end 154b of the optical fiber <NUM> to vibrate in accordance with the scan pattern. Thus, operation of the light source(s) <NUM> and drive electronics <NUM> are coordinated in a manner that generates image data that is encoded in the form of light that is spatially and/or temporally varying. Descriptions of optical fiber scanning techniques are provided in <CIT>.

The projection assembly <NUM> further comprises an optical coupling assembly <NUM> that couples the light from the scanning device <NUM> into the display screen <NUM>. The optical coupling assembly <NUM> comprises a collimation element <NUM> that collimates the light emitted by the scanning device <NUM> into a collimated light beam <NUM>. Although the collimation element <NUM> is illustrated in <FIG> as being physically separated from the optical fiber(s) <NUM>, a collimation element may be physically mounted to the distal end 154b of each optical fiber <NUM> in a "micro-lens" arrangement, as described in <CIT>, entitled "Microlens Collimator for Scanning Optical Fiber in Virtual/Augmented Reality System". The optical coupling subsystem <NUM> further comprises an in-coupling element (ICE) <NUM>, for instance, one or more reflective surfaces, diffraction gratings, mirrors, dichroic mirrors, or prisms to optically couple light into the end of the display screen <NUM>.

The display screen <NUM> takes the form of a waveguide apparatus <NUM> that includes a planar optical waveguide <NUM> and one or more diffractive optical elements (DOEs) <NUM> associated with the planar optical waveguide <NUM>. In alternative embodiments, the waveguide apparatus <NUM> may comprise multiple planar optical waveguides <NUM> and DOEs <NUM> respectively associated with the planar optical waveguides <NUM>. As best illustrated in <FIG>, the planar optical waveguide <NUM> has a first end 176a and a second end 176b, the second end 176b opposed to the first end 176a along a length <NUM> of the planar optical waveguide <NUM>. The planar optical waveguide <NUM> has a first face 180a and a second face 180b, at least the first and the second faces 180a, 180b (collectively <NUM>) forming an at least partially internally reflective optical path (illustrated by solid line arrow 182a and broken line arrow 182b, collectively <NUM>) along at least a portion of the length <NUM> of the planar optical waveguide <NUM>. The planar optical waveguide <NUM> may take a variety of forms that provide for substantially total internal reflection (TIR) for light striking the faces <NUM> at less than a defined critical angle.

The DOE(s) <NUM> (illustrated in <FIG> by dash-dot double lines) may take a large variety of forms which interrupt the TIR optical path <NUM>, providing a plurality of optical paths (illustrated by solid line arrows 184a and broken line arrows 184b, collectively <NUM>) between an interior <NUM> and an exterior <NUM> of the planar optical waveguide <NUM> extending along at least a portion of the length <NUM> of the planar optical waveguide <NUM>. In the illustrated embodiment, the DOE(s) <NUM> comprise one or more diffraction gratings, each of which can be characterized as an optical component with a periodic structure on the order of the light wavelength that splits and diffracts light into several beams travelling in different directions. The diffraction gratings can be composed of, e.g., surface nano-ridges, nano-patterns, slits, etc. that may be photolithographically printed on a substrate. The DOE(s) <NUM> may allow positioning of apparent objects and focus plane for apparent objects. Such may be achieved on a frame-by-frame, subframe-by-subframe, or even pixel-by-pixel basis.

As illustrated in <FIG>, the light propagates along the planar optical waveguide <NUM> with at least some reflections or "bounces" resulting from the TIR propagation. It is noted that some implementations may employ one or more reflectors in the internal optical path, for instance thin-films, dielectric coatings, metalized coatings, etc., which may facilitate reflection. Light propagates along the length <NUM> of the planar optical waveguide <NUM>, and intersects with the DOE(s) <NUM> at various positions along the length <NUM>. The DOE(s) <NUM> may be incorporated within the planar optical waveguide <NUM> or abutting or adjacent one or more of the faces <NUM> of the planar optical waveguide <NUM>. The DOE(s) <NUM> accomplishes at least two functions. The DOE(s) <NUM> shifts an angle of the light, causing a portion of the light to escape TIR, and emerge from the interior <NUM> to the exterior the face <NUM> of the planar optical waveguide <NUM>. The DOE(s) <NUM> focuses the out-coupled light at a viewing distance. Thus, someone looking through the face <NUM> of the planar optical waveguides <NUM> can see digital imagery at one or more viewing distances.

A collimated light beam <NUM> entering the waveguide <NUM> at one of two different angles will follow one of the two TIR optical paths 182a, 182b, resulting in light rays <NUM> exiting the planar optical waveguide <NUM> along one of the two sets of external optical paths. That is, a collimated light beam 200a that enters the waveguide <NUM> at an angle represented by the TIR optical path 182a will result in the light rays 202a exiting the planar optical waveguide <NUM> along the set of external optical paths, and a collimated light beam 200b that enters the waveguide <NUM> at an angle represented by the TIR optical path 182b will result in the light rays 202b exiting the planar optical waveguide <NUM> along the set of external optical paths. As shown in <FIG>, the light ray angle sensing assembly <NUM> is located between the ICE <NUM> and the DOE(s) <NUM> for directly or indirectly sensing the angle of the light rays <NUM> exiting the waveguide apparatus <NUM>, although the sensing assembly <NUM> may be located anywhere along the optical path of the collimated light beam <NUM>. Further details discussing the sensing assembly <NUM> will be described in further detail below.

In can be appreciated from the foregoing, the display subsystem <NUM> generates a series of synthetic image frames of pixel information that present an image of one or more virtual objects to the user. For example, referring to <FIG>, a synthetic image frame <NUM> is schematically illustrated with cells 252a-<NUM> divided into horizontal rows or lines 254a-254n. Each cell <NUM> of the frame <NUM> may specify values for each of a plurality of colors for the respective pixel to which the cell <NUM> corresponds and/or intensities. For instance, the frame <NUM> may specify one or more values for red 256a, one or more values for green 256b, and one or more values for blue 256c for each pixel. The values <NUM> may be specified as binary representations for each of the colors, for instance, a respective <NUM>-bit number for each color. Each cell <NUM> of the frame <NUM> may additionally include a value 256d that specifies an amplitude.

The frame <NUM> may include one or more fields, collectively <NUM>. The frame <NUM> may consist of a single field. Alternatively, the frame <NUM> may comprise two, or even more fields 258a-258b. The pixel information for a complete first field 258a of the frame <NUM> may be specified before the pixel information for the complete second field 258b, for example, occurring before the pixel information for the second field 258b in an array, an ordered list, or other data structure (e.g., record, linked list). A third or even a fourth field may follow the second field 258b, assuming a presentation subsystem is configured to handle more than two fields 258a-258b.

Referring now to <FIG>, the frame <NUM> is generated using a raster scan pattern <NUM>. In the raster scan pattern <NUM>, pixels <NUM> (only one called out) are sequentially presented. The raster scan pattern <NUM> typically presents pixels <NUM> from left to right (indicated by arrows 262a, 262b, then from top to bottom (indicated by arrow <NUM>). Thus, the presentation may start at the upper right corner and traverse left across a first line 266a until the end of the line is reached. The raster scan pattern <NUM> typically then starts from the left in a next line down. The presentation may be temporarily blacked out or blanked when returning from the end of one line to the start of the next line. This process repeats line-by-line until the bottom line 266n is completed, for example, at the bottom right most pixel <NUM>. With the frame <NUM> being complete, a new frame is started, again returning to the right of the top most line of the next frame. Again, the presentation may be blanked while returning from the bottom left to the top right to present the next frame.

Many implementations of raster scanning employ what is termed as an interlaced scan pattern. In interlaced raster scan patterns, lines from the first and the second fields 258a, 258b are interlaced. For example, when presenting lines of the first field 258a, the pixel information for the first field 258a may be used for the odd numbered lines only, while the pixel information for the second field 258b may be used for the even numbered lines only. Thus, all of the lines of the first field 258a of the frame <NUM> (<FIG>) are typically presented before the lines of the second field 258b. The first field 258a may be presented using the pixel information of the first field 258a to sequentially present line <NUM>, line <NUM>, line <NUM>, etc. Then the second field 258b of the frame <NUM> (<FIG>) may be presented following the first field 258a, by using the pixel information of the second field 258b to sequentially present line <NUM>, line <NUM>, line <NUM>, etc..

Referring to <FIG>, a spiral scan pattern <NUM> may be used instead of the raster scan pattern <NUM> to generate the frame <NUM>. The spiral scan pattern <NUM> may consist of a single spiral scan line <NUM>, which may include one or more complete angular cycles (e.g., <NUM> degrees) which may be denominated as coils or loops. As with the raster scan pattern <NUM> illustrated in <FIG>, the pixel information in the spiral scan pattern <NUM> is used to specify the color and/or intensity of each sequential pixel, as the angle increments. An amplitude or radial value <NUM> specifies a radial dimension from a starting point <NUM> of the spiral scan line <NUM>.

Referring to <FIG>, a Lissajous scan pattern <NUM> may alternatively be used to generate the frame <NUM>. The Lissajous scan pattern <NUM> may consist of a single Lissajous scan line <NUM>, which may include one or more complete angular cycles (e.g., <NUM> degrees), which may be denominated as coils or loops. Alternatively, the Lissajous scan pattern <NUM> may include two or more Lissajous scan lines <NUM>, each phase shifted with respect to one another to nest the Lissajous scan lines <NUM>. The pixel information is used to specify the color and/or intensity of each sequential pixel, as the angle increments. An amplitude or radial value specifies a radial dimension <NUM> from a starting point <NUM> of the Lissajous scan line <NUM>.

Referring to <FIG>, a multi-field spiral scan pattern <NUM> may alternatively be used to generate the frame <NUM>. The multi-field spiral scan pattern <NUM> includes two or more distinct spiral scan lines, collectively <NUM>, and in specifically, four spiral scan lines 292a-160d. The pixel information for each spiral scan line <NUM> may be specified by a respective field of a frame. Advantageously, multiple spiral scan lines <NUM> may be nested simply by shifting a phase between each successive ones of the spiral scan lines <NUM>. The phase difference between spiral scan lines <NUM> should be a function of the total number of spiral scan lines <NUM> that will be employed. For example, four spiral scan lines 292a-292d may be separated by a <NUM>-degree phase shift. An exemplary embodiment may operate at a <NUM> refresh rate with <NUM> distinct spiral scan lines (i.e., subspirals). Similar to the embodiment of <FIG>, one or more amplitude or radial values specify a radial dimension <NUM> from a starting point <NUM> of the spiral scan lines <NUM>.

Further details describing display subsystems are provided in <CIT>, entitled "Display Subsystem and Method," and <CIT>, entitled "Planar optical waveguide Apparatus With Diffraction Element(s) and Subsystem Employing Same".

Referring now to <FIG>, one specific embodiment of the display screen <NUM> and associated light ray angle sensing assembly <NUM> will be described. The planar optical waveguide <NUM> of the waveguide apparatus <NUM> takes the form of an optically transparent planar substrate. As shown in <FIG>, the substrate <NUM> is a single unitary substrate or plane of an optically transparent material, such as, e.g., glass, fused silica, acrylic, or polycarbonate, although in alternative embodiments, the substrate <NUM> may be composed of separate distinct panes of optically transparent material that are bonded together in the same plane or in different planes. The ICE <NUM> is embedded in the face 180b of the substrate <NUM> for receiving the collimated light beam <NUM> from the projection assembly <NUM> into the substrate <NUM> via the face 180b, although in alternative embodiments, the ICE <NUM> may be embedded in the other face 180a or even the edge of the substrate <NUM> for coupling the collimated light beam <NUM> into the substrate <NUM> as an in-coupled light beam.

The DOE(s) <NUM> are associated with the substrate <NUM> (e.g., incorporated within the substrate <NUM> or abutting or adjacent one or more of the faces 180a, 180b of the substrate <NUM>) for two-dimensionally expanding the effective exit pupil of the collimated light beam <NUM> optically coupled into the substrate <NUM>. To this end, the DOE(s) <NUM> comprises one or more orthogonal pupil expansion (OPE) elements 174a (only one shown in <FIG>) adjacent the face 180b of the substrate <NUM> for splitting the in-coupled light beam <NUM> into orthogonal light beams <NUM>, and an exit pupil expansion (EPE) element 174b associated with the substrate <NUM> for splitting each orthogonal light beam <NUM> into the out-coupled light rays <NUM> that exit the face 180b of the substrate <NUM> towards the eye(s) <NUM> of the end user <NUM>. In the alternative embodiment where the substrate <NUM> is composed of distinct panes, the OPE element(s) <NUM> and EPE element 174b may be incorporated into different panes of the substrate <NUM>.

The OPE element 174a relays light along a first axis (horizontal or x-axis in <FIG>), and expands the effective exit pupil of light along a second axis (vertical or y-axis in <FIG>). In particular, the ICE <NUM> optically in-couples the collimated light beam <NUM> for propagation within the substrate <NUM> via TIR along an internally reflective optical path 204a (in this case, along the vertical or y-axis), and in doing so, repeatedly intersects the OPE element 174a. In the illustrated embodiment, the OPE element 174a has a relatively low diffraction efficiency (e.g., less than <NUM>%), and comprises a series of diagonal diffractive elements (forty-five degrees relative to the x-axis), such that, at each point of intersection with the OPE element 174a, a portion (e.g., greater than <NUM>%) of the in-coupled light beam <NUM> continues to propagate within the substrate <NUM> via TIR along the internally reflective optical path 204a, and the remaining portion (e.g., less than <NUM>%) of the in-coupled light beam <NUM> is diffracted as an orthogonal light beam <NUM> that propagates within the substrate <NUM> via TIR along an internally reflective optical path 204b (in this case, along the horizontal or x-axis) toward the EPE element 174b. It should be appreciated that although the optical paths 204b are described as being perpendicular or orthogonal to the optical path 204a, the optical paths 204b may alternatively be obliquely oriented with respect to the optical path 204a. Thus, by dividing the in-coupled light beam <NUM> into multiple orthogonal beams <NUM> that propagate along parallel internally reflective optical paths 204b, the exit pupil of the collimated light beam <NUM> in-coupled into the waveguide apparatus <NUM> is expanded vertically along the y-axis by the OPE element 174a.

The EPE element 174b, in turn, further expands the light's effective exit pupil along the first axis (horizontal x-axis in <FIG>). In particular, the EPE element 174b, like the OPE element 174a, has a relatively low diffraction efficiency (e.g., less than <NUM>%), such that, at each point of intersection with the EPE element 174b, a portion (e.g., greater than <NUM>%) of each orthogonal light beam <NUM> continues to propagate along the respective internally reflective optical path 204b, and the remaining portion of each orthogonal light beam <NUM> is diffracted as an out-coupled light ray <NUM> that exits the face 180b of the substrate <NUM> (along the z-axis), as illustrated in <FIG> and <FIG>. That is, every time a light beam hits the EPE element 174b, a portion of it will be diffracted toward the face 180b of the substrate <NUM>, while the remaining portion will continue to propagate along the respective internally reflective optical path 204b.

Thus, by dividing each orthogonal light beam <NUM> into multiple out-coupled light rays <NUM>, the exit pupil of the in-coupled light beam <NUM> is further expanded horizontally along the x-axis by the EPE element 174b, resulting in a two-dimensional array of out-coupled light rays <NUM> that resemble a larger version of the original in-coupled light beam <NUM>. It should be noted that although the OPE element 174a and EPE element 174b are illustrated in <FIG> as non-overlapping in the x-y plane, the OPE element 174a and EPE element 174b may fully overlap each other in the x-y plane, as illustrated in <FIG>, or may partially overlap each other in the x-y plane, as illustrated in <FIG>. In these cases, the OPE element 174a and EPE element 174b will need to be respectively disposed on opposite faces 180a, 180b of the substrate <NUM>.

In addition to the function of out-coupling the light beamlets from the face 180b of the substrate <NUM>, the EPE element 174b serves to focus the output set of light beamlets at along a given focal plane, such that a portion of an image or virtual object is seen by end user <NUM> at a viewing distance matching that focal plane. For example, if the EPE element 174b has only a linear diffraction pattern, the out-coupled light rays <NUM> exiting the face 180b of the substrate <NUM> toward the eye(s) <NUM> of the end user <NUM> will be substantially parallel, as shown in <FIG>, which would be interpreted by the brain of the end user <NUM> as light from a viewing distance (focal plane) at optical infinity. However, if the EPE element 174b has both a linear diffraction pattern component and a radially symmetric diffraction pattern component, the out-coupled light rays <NUM> exiting the face 180b of the substrate <NUM> will be rendered more divergent from the perspective of the eye(s) <NUM> of the end user <NUM> (i.e., a convex curvature will be imparted on the light wavefront), and require the eye(s) <NUM> to accommodate to a closer distance to bring the resulting image into focus on the retina and would be interpreted by the brain of the end user <NUM> as light from a viewing distance (e.g., four meters) closer to the eye(s) <NUM> than optical infinity, as shown in <FIG>. The out-coupled light rays <NUM> exiting the face 180b of the substrate <NUM> can be rendered even more divergent from the perspective of the eye(s) <NUM> of the end user <NUM> (i.e., a more convex curvature will be imparted on the light wavefront), and require the eye(s) <NUM> to accommodate to an even closer distance to bring the resulting image into focus on the retina and would be interpreted by the brain of the end user <NUM> as light from a viewing distance (e.g., <NUM> meters) closer to the eye(s) <NUM>, as shown in <FIG>.

Although the waveguide apparatus <NUM> has been described herein as having only one focal plane, it should be appreciated that multiple planar optical waveguides <NUM> with associated OPEs <NUM> and EPEs <NUM> can be used to simultaneously or concurrently generate images at multiple focal planes, as discussed in <CIT> and<CIT>.

As briefly discussed above, the display subsystem <NUM> comprises a sensing assembly <NUM> configured for sensing at least one parameter indicative of the angle of at least one of the light rays <NUM> exiting the waveguide apparatus <NUM>. In the illustrated embodiment, the sensing assembly <NUM> senses the parameter(s) indicative of the angle of the light ray(s) <NUM> relative to one or more reference planes. For example, these reference planes may comprise the x-z plane, y-z plane, and x-y plane, as described in further detail below. Notably, these reference planes may be flat, but because the exterior surface of the waveguide apparatus <NUM> may alternatively be curved to conform to the head <NUM> of the user <NUM>, these reference planes may be curved as well.

As also briefly discussed above, the angles of exiting light ray(s) are highly correlated to the positions of the pixels within the image frame. For example, as illustrated in <FIG>, a collimated light beam <NUM> from the projection subsystem <NUM> enters the waveguide apparatus <NUM> via the ICE <NUM> and propagates within the planar optical waveguide <NUM>. The exit pupil of the propagating light beam <NUM> is expanded along the x-axis and y-axis by the DOE(s) <NUM>, e.g., as described above with respect to <FIG>, a light ray <NUM> that exits the face 180b of the planar optical waveguide <NUM>. It should be appreciated that although only one light ray <NUM> in correspondence with the collimated light beam <NUM> input into the waveguide apparatus <NUM> is shown for purposes of brevity, there will typically be many light rays <NUM> that exit the waveguide apparatus <NUM> in correspondence with a single collimated light beam <NUM>, with all angles of all of the exiting light rays <NUM> being related to the scan angle of the collimated light beam <NUM>.

The collimated light beam <NUM> is scanned by the projection subsystem <NUM> to produce an input cone of light 210a, with each beam-angle in this cone corresponding to a pixel <NUM> in the field of view (FOV) <NUM> of the user. As shown in <FIG>, if the collimated light beam <NUM> has one particular beam-angle, a corresponding pixel 212a is generated in the bottom-left region of the FOV <NUM>, whereas if the collimated light beam <NUM> has another particular beam-angle, a corresponding pixel 212b is generated in the top-right region of the FOV <NUM>. The waveguide apparatus <NUM> presents an x-y image plane to the user by transmitting the input light cone 210a to the emission face 180b of the planar optical waveguide <NUM> an output cone of light 210b.

The CPU <NUM> (shown in <FIG>) generates image data, which in addition to defining the colors and intensities of the pixels, defines the locations of the pixels, and thus controls the angles of the light beams <NUM> generated by the projection subsystem <NUM> relative to the display screen <NUM> based on the designed angles of the exiting light rays <NUM> corresponding to the defined locations of the pixels, as well as the actual angles of the exiting light rays <NUM> sensed by the sensing assembly <NUM>, thereby ensuring that the actual angles of exiting light rays <NUM> are as close to identical to the designed angles of the exiting light rays <NUM> as possible.

For example, referring to <FIG>, the orientation of an exiting light ray <NUM> from an origin in three-dimensional space may be defined by two angles, one on the x-z plane 216a and another on the y-z plane 216b, which closely correspond to the respective x- and y-coordinates of the pixel <NUM> in the x-y plane of the FOV <NUM>. The CPU <NUM> may determine the actual angles of the exiting light ray <NUM> in the x-z and y-z planes 216a, 216b based on parameters sensed by the sensing assembly <NUM>, compute a deviation between the actual angles of the exiting light ray <NUM> and the as-designed angles of the exiting light ray <NUM> for the corresponding pixel <NUM>, and modifies the operation of the projection subsystem <NUM> to compensate for the discrepancy between the actual angles of the exiting light ray <NUM> and the as-designed angles of the exiting light ray <NUM>.

For example, the CPU <NUM> may instruct the projection subsystem <NUM> to adjust the scan position of the collimated light beam <NUM>. In the illustrated embodiment, the scan position of the collimated light beam <NUM> may be adjusted by modifying the actuation/drive profile (e.g., the drive voltages, drive signals, drive patterns, etc. provided by the drive electronics <NUM> to the piezoelectric element <NUM>) of the scanning device <NUM> (see <FIG>), so that the mechanical response of the scanning device <NUM> is more in agreement with the desired mechanical response for the desired pixel positions. As another example, the CPU <NUM> may modify the image data (e.g., by modifying the pixel modulation/pixel synchronization) to compensate for the known mismatch between the mechanical scan response and the desired scan response of the scanning device <NUM>. In this case, the "incorrect angles" of the exiting light ray <NUM> are measured but not corrected. As still another example, a combination of modifying the scan position of the collimated light beam <NUM> and modifying the image data may be employed by the CPU <NUM>.

The scan position of the collimated light beam <NUM> and/or the image data can be modified to compensate for the mismatch between the actual angles of the exiting light ray <NUM> and the desired angles of the exiting light ray <NUM> by employing a software/hardware controller (similar to, e.g., a proportional-integral-derivative (PID) that monitors the angle measurements in real-time, and effects the adjustment to the projection subsystem <NUM> as quickly as possible to minimize delay due to processing and causality. Alternatively, since the display subsystem <NUM> is a repetitive system, where an identical target scan pattern is used to generate each image frame, angle measurements acquired for a previous image frame can be computed and stored, and then corrections can be applied to a subsequent image frame. In the case where there is a high image frame rate, a delay on the order of a few milliseconds may be incurred.

The parameters detected by the sensing assembly <NUM> comprises an intensity of at least one of the light rays <NUM> relative to the x-z and y-z planes 216a, 216b. In the embodiment illustrated in <FIG>, the sensing assembly <NUM> measures the intensity of at least one light ray <NUM> (only one described herein) representative of the light rays <NUM> exiting or emitted from the waveguide apparatus <NUM>. In this embodiment, the representative light ray <NUM> is different from the exiting light rays <NUM>, and will exit the waveguide apparatus <NUM> at a different location from the exiting light rays <NUM>, preferably outside of the FOV <NUM> of the end user <NUM>. To this end, the waveguide apparatus <NUM> further comprises an additional DOE <NUM> for out-coupling light from the waveguide apparatus <NUM> as the representative light ray <NUM> to the sensing assembly <NUM>, as illustrated in <FIG>.

As best shown in <FIG>, the sensing assembly <NUM> comprises a pair of orthogonal angle sensors 220a and 220b (collectively, <NUM>) configured for sensing the orthogonal components of the representative light ray <NUM> exiting the waveguide apparatus <NUM>, and a normalization sensor <NUM> configured for sensing the absolute intensity of the representative light ray <NUM>, such that the readings from the orthogonal angle sensors <NUM> can be normalized to the intensity of the light in the planar optical waveguide <NUM>. That is, when an image is generated, the pixel intensities are modulated corresponding to the color of different pixels. Thus, the pixel intensity modulation measured by the photo-intensity sensor <NUM> can be taken into account when interpreting the measurements of the angle sensors <NUM>.

In the illustrated embodiment, the angle sensors <NUM> and normalization sensor <NUM> are mounted to the planar optical waveguide <NUM> of the waveguide apparatus <NUM> in close association with the DOE <NUM>, such that the light ray <NUM> passing through the DOE <NUM> is incident on the sensors <NUM>, <NUM>. The sensors <NUM>, <NUM> are preferably located outside the FOV <NUM> of the end user <NUM>, such that they do not interfere with the image experienced by the end user <NUM>.

Referring further to <FIG>, the pair of angle sensors 220a, 220b respectively comprise a pair of photo-intensity sensors 224a, 224b (collectively, <NUM>), a pair of angle selective dielectric layers 226a, 226b (collectively, <NUM>), and a pair of cylindrical lenses (e.g., GRIN lenses) 228a, 228b (collectively, <NUM>). The GRIN lenses <NUM> are mounted directly to the outward facing surface of the DOE <NUM>, the dielectric layers <NUM> are respectively mounted directly to the outward facing surface of the GRIN lenses <NUM>, and the photo-intensity sensors <NUM> are respectively mounted directly to the outward facing surface of the dielectric layers <NUM>.

Significantly, the directional properties of each dielectric layer <NUM> transmit light energy as a known function of the angle at which the light energy is incident on the plane of the dielectric layer <NUM> (which is located in the x-y plane). For example, as can be seen from an exemplary dielectric layer transmission-angle relationship illustrated in <FIG>, the closer the angle of the representative light ray <NUM> is to the normal of the plane of the dielectric layer <NUM> (<NUM> degrees), the greater the energy of the representative light ray <NUM> is transmitted to the photo-intensity sensor <NUM>. Furthermore, each photo-intensity sensor <NUM> will generate a voltage that is a known function of the intensity of light energy incident at an angle normal to the plane of the respective dielectric layer <NUM>. For example, as can be from an exemplary photodetector intensity-voltage relationship illustrated in <FIG>, the higher the intensity of the light energy incident on the dielectric layer <NUM>, the greater the voltage generated by the dielectric layer <NUM>. As a result, the angle at which the representative light ray <NUM> is incident on the plane of the photo-intensity sensor <NUM> can be determined from these relationship curves, as will be described in further detail below. It should be noted that the relationship curves illustrated in <FIG> may be generated analytically, or may be generated by measuring or calibrating the relevant values per unit, thereby resulting in more accurate and calibrated relationship curves.

It should also be noted that, due to "cosine falloff," where the projection of the sensor aperture to the plane normal to the incident light decreases in area the higher the angle of incidence of the incident light, as well as the opto-physical characteristics of the light sensor, the photo-intensity sensors <NUM>, themselves will have some degree of angular dependency, which can be utilized as a primary means of sensing the angle of the representative light ray <NUM>, in which case, the angle sensors <NUM> may not include dielectric layers <NUM>, or can be utilized as a secondary or auxiliary means of sensing the angle of the representative light ray <NUM> in addition to the use of the dielectric layers <NUM> in the angle sensors <NUM>. In either case, a photodetector intensity-angle relationship (not shown), which correlates the voltage sensed by the respective photo-intensity sensor <NUM> to a range of light incident angles, may be generated. This photodetector intensity-angle relationship, by itself, can be used to determine the angle of the representative light ray <NUM>, or may be used to confirm the angle of the representative light ray <NUM> determined from the dielectric layer transmission-angle relationship (<FIG>) and the photodetector intensity-voltage relationship (<FIG>).

Notably, because the dielectric layers <NUM> are isotropic in nature in that they will equally transmit the energy from the representative light ray <NUM> at the same incidence angle but different radial directions, the sensing assembly <NUM> breaks the circular symmetry of the dielectric layers <NUM>, thereby allowing the orientation of the representative light ray <NUM> to be projected into the x-z and y-z planes 216a, 216b. To this end, the cylindrical lenses 228a, 228b are configured for respectively passing the first and second orthogonal components of the representative light ray <NUM> (corresponding to the x-z and y-z planes 216a, 216b) to the respective dielectric layers 226a, 226b.

Thus, one lens 228a separates the energy of the representative light ray 203a into a horizontal (x-) component, and the other lens 228b separates the energy of the representative light ray 203b into a vertical (y-) component. Thus, one photo-intensity sensor 224a will only receive the horizontal energy component of the representative light ray 203a exiting the DOE <NUM>, and the other photo-intensity sensor 224b will only receive the vertical energy component of the representative light ray 203b via the DOE <NUM>. The angle of the representative light ray <NUM> projected onto the x-z and y-z planes 216a, 216b can then be determined from these horizontal and vertical components of the representative light ray <NUM>, as will be discussed below.

Notably, although each angle sensor <NUM> is described as detecting one representative light ray <NUM> for purposes of brevity and clarity in illustration, each of the angle sensors <NUM>, in reality, detects many light rays, and therefore, the voltage outputs of angle sensors <NUM> will be representative of the composite of the horizontal component 203a or vertical component 203b of the representative light ray <NUM>. However, the DOE <NUM> preferably has a linear diffraction pattern, such that the angles of the representative light ray <NUM> exiting the DOE <NUM> are uniform given a specific angle of the in-coupled collimated beam <NUM>.

In an alternative embodiment illustrated in <FIG>, rather than using cylindrical lenses <NUM> in association with the already existing DOE <NUM>, the DOE <NUM>, itself, may be segmented into a portion that passes one orthogonal component to the angle sensor 220a and another portion that passes the other orthogonal component to the angle sensor 220b in the same manner that as the cylindrical lenses <NUM> described above. In still another alternative embodiment illustrated in <FIG>, the angle sensors <NUM> respectively comprise a pair of polarization elements 230a, 230b (collectively, <NUM>) configured for polarizing the representative light ray <NUM> into orthogonally polarized light rays, which are then passed to the respective photo-intensity sensors 224a, 224b via the dielectric layers 226a, 226b.

In yet another alternative embodiment, instead of cylindrical lenses or any of the other devices described above, the dielectric layers 226a, 226b, themselves, can be strained in orthogonal orientations, such that the dielectric layers <NUM> are no longer isotropic, but rather are anisotropic, and tend to pass the representative light ray <NUM> in one orthogonal direction more than in the other orthogonal direction, and vice versa. Although the anisotropic dielectric layers <NUM> do not perfectly transmit the representative light ray <NUM> in orthogonal manners, the angle of the representative light ray <NUM>, when projected onto the x-z and y-z planes 216a, 216b, can still be determined in view of the known orthogonal transmission coefficient of the dielectric layers <NUM> (i.e., the ratio of light energy transmitted by each dielectric layer <NUM> in one orthogonal direction relative to the other orthogonal direction).

Although each of the angle sensors <NUM> is described as being closely associated with the DOE <NUM>, it should be appreciated that one or both of the angle sensors <NUM> can be mounted in the waveguide apparatus <NUM> at any interface that takes the form of a grating or material with a different refractive index than the material of which the planar optical waveguides <NUM> are composed, thereby allowing the light rays to escape and enter the sensors <NUM>. However, the angle of the representative light ray <NUM> exiting the DOE <NUM> will closely match the nominal angle of the light rays <NUM> exiting the EPE element 174b for any given scan angle of the in-coupled collimated beam <NUM>.

Notwithstanding the foregoing, the angle sensors <NUM> may alternatively be closely associated with the EPE element 174b. In this case, the light rays <NUM> exiting the waveguide apparatus <NUM> will comprise the representative light ray sensed by the angle sensors <NUM>. In this case, the light rays <NUM> sensed by the angle sensors <NUM> may be divergent, and thus somewhat non-uniform, given any particular angle of the in-coupled collimated beam <NUM>, since the EPE element 174b may have a radially symmetric diffraction pattern that creates a focal plane that is not at optical-infinity. In this case, the size of the angle sensors <NUM> will preferably be relatively small, and as such, the angle variance between the light rays impinging on an angle sensor <NUM> will be insignificant.

By the same token, if the angle sensors <NUM> are closely associated with the EPE element 174b, it is desired that the spacing between respective angle sensors <NUM> be relatively small, such that the horizontal and vertical light components of the exiting light ray <NUM> detected by the angle sensors 220a, 220b will essentially serve as the components of a nominal light ray exiting the waveguide apparatus <NUM>. In the end, the function of the angles of the light rays 202a, 202b projected onto the x-z and y-z planes 216a, 216b will essentially be close to the average of the angles of all the light rays <NUM> impinging on the angle sensors <NUM>, which due to the small variance between the angles, is representative of a nominal angle of the light rays <NUM> exiting the waveguide apparatus <NUM>. In an alternative embodiment, the angle sensors <NUM> may be placed on the waveguide <NUM> that produces images at optical infinity (i.e., no lensing) or a special area on the waveguide apparatus <NUM> without lensing, whereby the angles of the light rays are parallel to each other.

Notably, the angle sensors <NUM>, as disclosed, are only capable of sensing information from which an absolute angle of the exiting light rays <NUM> in the x-z plane 216a or y-z plane 216b can be obtained (e.g., an absolute angle of <NUM>° may be +<NUM>° or -<NUM>°). Thus, the sensing assembly <NUM> senses another parameter indicative of the relative angle of the exiting light rays <NUM> in the x-z plane 216a or y-z plane 216b. In the illustrated embodiment, this sensed parameter comprises a quadrant in which the collimated light beam <NUM> is projected in a plane, so that the sign of the angles of the light rays <NUM> projected onto the x-z and y-z planes 216a, 216b can be determined.

For example, as illustrated in <FIG>, one exiting light ray <NUM> forms an angle relative to the x-y plane 216c. That is, when projected onto the x-y plane 216c, the exiting light ray <NUM> may form an angle in the x-y plane 216c, as illustrated in <FIG>. As there shown, the angle of the exiting light ray <NUM> projected within second quadrant of the x-y plane 216c, and therefore, it can be determined that the angle that the exiting light ray <NUM> makes in the x-z plane 216a has a negative sign, and the angle that the exiting light ray <NUM> makes in the y-z plane 216b should have a positive sign.

The sensing assembly <NUM> is configured for indirectly detecting the quadrant in which the exiting light ray <NUM> is projected by detecting a quadrant that the collimated light beam <NUM> that enters the ICE <NUM> from the collimation element <NUM> is pointed (shown in <FIG>). In particular, referring back to <FIG> and <FIG>, the sensing assembly <NUM> comprises one or more quadrant sensors <NUM> mounted to ICE <NUM>. As illustrated in <FIG>, four quadrant sensors <NUM> are spaced apart in quadrants of a reference plane <NUM>, such that activation of one of the sensors <NUM> by the collimated light beam <NUM> will indicate the quadrant at which the light beam <NUM> is currently pointed. Alternatively, as illustrated in <FIG>, the sensing assembly <NUM> may comprise a quadrant position sensing detector (PSD) <NUM> centered at the intersection of the quadrants. In any event, because the angular position of the collimated light beam <NUM> is correlateable to the angle of the light rays <NUM> exiting the waveguide apparatus <NUM>, the quadrant in which the light rays <NUM> are projected into the x-y plane 216c can be derived from the quadrant of the reference plane <NUM> at which the collimated light beam <NUM> is pointed.

In an alternative embodiment, the quadrant in which the light rays <NUM> are projected into the x-y plane 216c can simply be inferred from the quadrant of the current scan position in the scan pattern when the intensity of the exiting light rays <NUM> is sensed by the sensing assembly <NUM>.

Although sensors that detect the angle of the exiting light rays <NUM> have been described as being closely associated with the planar optical waveguides <NUM>, one or more angle detecting sensors can be incorporated into any portion of the display subsystem <NUM> where a light ray or beam correlatable to the angles of the light rays exiting the waveguide apparatus <NUM> can be detected. For example, a PSD <NUM> that detects both an angle of collimated light beam <NUM> projected on the x-z and y-z planes 216a, 216b, as well as the quadrant of the x-y plane 216c in which the collimated light beam <NUM> is projected, can be mounted to the ICE <NUM>, as illustrated in <FIG>.

The PSD <NUM> directly senses the angle of the collimated beam <NUM>, rather than the light rays <NUM> exiting the waveguide apparatus <NUM>. However, because the angle of the collimated beam <NUM> is highly correlatable to the angles of the light rays <NUM> exiting the waveguide apparatus <NUM>, the PSD <NUM> indirectly senses the angles of the light rays <NUM> exiting the waveguide apparatus <NUM> by virtue of directly sensing the angle of the collimated beam <NUM>.

One method of determining the angles of a light ray <NUM> projected onto the x-z plane 216a and y-z plane 216b will now be described. Assume that the photo-intensity sensor <NUM> measures a voltage of <NUM> mV. In accordance with the exemplary photodetector intensity-voltage relationship illustrated in <FIG>, the absolute intensity of the representative light ray <NUM> can then be determined to be <NUM> nits.

Assume that the angle sensor 220a measures a voltage of <NUM> mV. In accordance with the exemplary photodetector intensity-voltage relationship illustrated in <FIG>, the intensity of the representative light ray <NUM> transmitted by the angle selective dielectric layer 226a to the photo-intensity sensor 224a can then be determined to be <NUM> nits. Thus, based on the known intensity of the light ray of <NUM> nits, it can be determined that the dielectric layer 226a transmits <NUM>/<NUM> = <NUM>% of the light energy to the photo-intensity sensor 224a. In accordance with the exemplary dielectric layer transmission-angle relationship illustrated in <FIG>, the absolute angle of the representative light ray <NUM> projected in the x-z plane 216a can then be determined to be <NUM> degrees.

Similarly, assume the angle sensor 220b measures a voltage of <NUM> mV. In accordance with the exemplary photodetector intensity-voltage relationship illustrated in <FIG>, the intensity of the representative light ray <NUM> transmitted by the angle selective dielectric layer 226b to the photo-intensity sensor 224b can then be determined to be <NUM> nits. Thus, based on the known intensity of the light ray of <NUM> nits, it can be determined that the dielectric layer 226b transmits <NUM>/<NUM> = <NUM>% of the light energy to the photo-intensity sensor 224b. In accordance with the exemplary dielectric layer transmission-angle relationship illustrated in <FIG>, the absolute angle of the representative light ray <NUM> projected in the y-z plane 216a can then be determined to be <NUM> degrees.

If the sensors <NUM> or PSD <NUM> detect that the angle of the representative light ray <NUM> projected on the x-y plane 216c is in the third quadrant, or it is otherwise known that the angle of the representative light ray <NUM> projected on the x-y plane 216c is in the third quadrant derived from information of the known scan angle of the collimated beam <NUM>, it can be determined that the angles of the representative light ray <NUM> respectively projected into the x-z plane 216a and y-z plane 216b should both be negative, and thus, be -<NUM> degrees and -<NUM> degrees.

Although particular embodiments of the present inventions have been shown and described, it will be understood that it is not intended to limit the present inventions to the preferred embodiments, and it will be obvious to those skilled in the art that various changes and modifications may be made.

Claim 1:
A virtual image generation system (<NUM>) for use by an end user, comprising:
a waveguide apparatus (<NUM>);
a projection subsystem (<NUM>) configured for generating a collimated light beam;
a display (<NUM>) configured for emitting light rays in response to the collimated light beam to display a pixel of an image frame to the end user, the pixel having a location encoded with angles of the emitted light rays;
a sensing assembly (<NUM>) configured for sensing at least one parameter indicative of at least one of the emitted light ray angles; and
a control subsystem (<NUM>, <NUM>, <NUM>, <NUM>) configured for generating image data defining a location of the pixel, and controlling an angle of the light beam relative to the display based on the defined location of the pixel and the at least one sensed parameter, wherein the at least one sensed parameter comprises an intensity of at least one light ray representative of the plurality of exiting light rays,
wherein the sensing assembly (<NUM>) comprises at least one angle sensor (220a, 220b), each of which includes a photo-intensity sensor (224a, 224b) and an angle selective layer (226a, 226b) mounted between the waveguide apparatus (<NUM>) and the photo-intensity sensor (224a, 224b).