Patent Description:
Modern computing and display technologies have facilitated the development of systems for so called "virtual reality" or "augmented reality" experiences, wherein digitally reproduced images or portions thereof are presented to a user in a manner wherein they seem to be, or may be perceived as, real. A virtual reality, or "VR," scenario typically involves presentation of digital or virtual image information without transparency to other actual real-world visual input; an augmented reality, or "AR," scenario typically involves presentation of digital or virtual image information as an augmentation to visualization of the actual world around the user. <CIT> and <CIT> disclose various optical systems and operating methods, with an eyepiece through which a user can see a world object and to which a virtual object can be projected, and a dimmer which can reduce an intensity of light associated with the world object in a portion of a field of view of the optical system.

Despite the progress made in these display technologies, there is a need in the art for improved methods, systems, and devices related to augmented reality systems, particularly, display systems.

The present disclosure relates generally to techniques for improving optical systems in varying ambient light conditions. More particularly, embodiments of the present disclosure provide systems and methods for operating an augmented reality (AR) device comprising a dimming element. Although the present invention is described in reference to an AR device, the disclosure is applicable to a variety of applications in computer vision and image display systems.

Numerous benefits are achieved by way of the present disclosure over conventional techniques. For example, augmented reality (AR) devices described herein may be used in varying light levels, from dark indoors to bright outdoors, by globally dimming and/or selectively dimming the ambient light reaching the user's eyes. Embodiments of the present invention allow for AR and virtual reality (VR) capabilities in a single device by using the pixelated dimmer to attenuate the world light by greater than <NUM>%. Embodiments of the present invention also mitigate vergence accommodation conflict using a variable focal element with discrete or continuous variable depth plane switching technologies. Embodiments of the present invention improve the battery life of the AR device by optimizing the projector brightness based on the amount of detected ambient light. Other benefits of the present disclosure will be readily apparent to those skilled in the art.

An ongoing technical challenge with optical see through (OST) augmented reality (AR) devices is the variation in the opacity and/or visibility of the virtual content under varying ambient light conditions. The problem worsens in extreme lighting conditions such as a completely dark room or outside in full bright sunlight. Embodiments of the present invention solve these and other problems by dimming the world light at different spatial locations within the field of view of the AR device. The portion of the field of view to which dimming is applied and the amount of dimming that is applied are each determined based on various information detected by the AR device. This information may include detected ambient light, detected gaze information, and/or the detected brightness of the virtual content being projected. The functionality of the AR device is further improved by detecting a direction associated with the ambient light by, for example, detecting a plurality of spatially-resolved light values. This allows the AR device to improve its battery life by only dimming the portions of the field of view in which dimming is needed and/or increasing the projector brightness in certain portions of the field of view. Accordingly, embodiments of the present invention enable usage of the AR device in a much wider variety of conditions than traditionally possible.

<FIG> illustrates an AR scene <NUM> as viewed through a wearable AR device, according to some embodiments of the present invention. AR scene <NUM> is depicted wherein a user of an AR technology sees a real-world park-like setting <NUM> featuring various real-world objects <NUM> such as people, trees, buildings in the background, and a real-world concrete platform <NUM>. In addition to these items, the user of the AR technology also perceives that they "see" various virtual objects <NUM> such as a robot statue <NUM>-<NUM> standing upon the real-world concrete platform <NUM>, and a cartoon-like avatar character <NUM>-<NUM> flying by, which seems to be a personification of a bumble bee, even though these elements (character <NUM>-<NUM> and statue <NUM>-<NUM>) do not exist in the real world. Due to the extreme complexity of the human visual perception and nervous system, it is challenging to produce a virtual reality (VR) or AR technology that facilitates a comfortable, natural-feeling, rich presentation of virtual image elements amongst other virtual or real-world imagery elements.

<FIG> illustrates one or more general features of an AR device <NUM> according to the present invention. In some embodiments, an AR device <NUM> may include an eyepiece <NUM> and a dynamic dimmer <NUM> configured to be transparent or semi-transparent when AR device <NUM> is in an inactive mode or an off mode such that a user may view one or more world objects <NUM> when looking through eyepiece <NUM> and dynamic dimmer <NUM>. As illustrated, eyepiece <NUM> and dynamic dimmer <NUM> may be arranged in a side-by-side configuration and may form a system field of view that a user sees when looking through eyepiece <NUM> and dynamic dimmer <NUM>. In some embodiments, the system field of view is defined as the entire two-dimensional region occupied by one or both of eyepiece <NUM> and dynamic dimmer <NUM>. Although <FIG> illustrates a single eyepiece <NUM> and a single dynamic dimmer <NUM> (for illustrative reasons), AR device <NUM> may include two eyepieces and two dynamic dimmers, one for each eye of a user.

During operation, dynamic dimmer <NUM> may be adjusted to reduce an intensity of a world light <NUM> associated with world objects <NUM> impinging on dynamic dimmer <NUM>, thereby producing a dimmed area <NUM> within the system field of view. Dimmed area <NUM> may be a portion or subset of the system field of view, and may be partially or completely dimmed. Dynamic dimmer <NUM> may be adjusted according to a plurality of spatially-resolved dimming values for dimmed area <NUM>. Furthermore, during operation of AR device <NUM>, projector <NUM> may project a virtual image light <NUM> (i.e., light associated with virtual content) onto eyepiece <NUM> which may be observed by the user along with world light <NUM>.

Projecting virtual image light <NUM> onto eyepiece <NUM> may cause a light field (i.e., an angular representation of virtual content) to be projected onto the user's retina in a manner such that the user perceives the corresponding virtual content as being positioned at some location within the user's environment. For example, virtual image light <NUM> outcoupled by eyepiece <NUM> may cause the user to perceive character <NUM>-<NUM> as being positioned at a first virtual depth plane <NUM>-<NUM> and statue <NUM>-<NUM> as being positioned at a second virtual depth plane <NUM>-<NUM>. The user perceives the virtual content along with world light <NUM> corresponding to one or more world objects <NUM>, such as platform <NUM>.

In some embodiments, AR device <NUM> may include an ambient light sensor <NUM> configured to detect world light <NUM>. Ambient light sensor <NUM> may be positioned such that world light <NUM> detected by ambient light sensor <NUM> is similar to and/or representative of world light <NUM> that impinges on dynamic dimmer <NUM> and/or eyepiece <NUM>. In some embodiments, ambient light sensor <NUM> may be configured to detect a plurality of spatially-resolved light values corresponding to different pixels of dynamic dimmer <NUM>. In these embodiments, ambient light sensor <NUM> may, for example, correspond to an imaging sensor (e.g., CMOS, CCD, etc.) or a plurality of photodiodes (e.g., in an array or another spatially-distributed arrangement). In some embodiments, or in the same embodiments, ambient light sensor <NUM> may be configured to detect a global light value corresponding to an average light intensity or a single light intensity of world light <NUM>. In these embodiments, ambient light sensor <NUM> may, for example, correspond to a set of one or more photodiodes. Other possibilities are contemplated.

<FIG> illustrates an example of AR device <NUM> in which dimmed area <NUM> is determined based on detected light information corresponding to world light <NUM>. Specifically, ambient light sensor <NUM> may detect world light <NUM> associated with the sun and may further detect a direction and/or a portion of the system field of view at which world light <NUM> associated with the sun passes through AR device <NUM>. In response, dynamic dimmer <NUM> may be adjusted to set dimmed area <NUM> to cover a portion of the system field of view corresponding to the detected world light. As illustrated, dynamic dimmer <NUM> may be adjusted so as to reduce the intensity of world light <NUM> at the center of dimmed area <NUM> at a greater amount than the extremities of dimmed area <NUM>.

<FIG> illustrates an example of AR device <NUM> in which dimmed area <NUM> is determined based on virtual image light <NUM>. Specifically, dimmed area <NUM> may be determined based on the virtual content perceived by the user resulting from the user observing virtual image light <NUM>. In some embodiments, AR device <NUM> may detect image information that includes a location of virtual image light <NUM> (e.g., a location within dynamic dimmer <NUM> through which the user perceives the virtual content) and/or a brightness of virtual image light <NUM> (e.g., a brightness of the perceived virtual content and/or the light generated at projector <NUM>), among other possibilities. As illustrated, dynamic dimmer <NUM> may be adjusted to set dimmed area <NUM> to cover a portion of the system field of view corresponding to virtual image light <NUM> or, alternatively, in some embodiments dimmed area <NUM> may cover a portion of the system field of view that is not aligned with virtual image light <NUM>. In some embodiments, the dimming values of dimmed area <NUM> may be determined based on world light <NUM> detected by ambient light sensor <NUM> and/or the brightness of virtual image light <NUM>.

<FIG> illustrates an example of AR device <NUM> in which dimmed area <NUM> is determined based on gaze information corresponding to an eye of a user. In some embodiments, the gaze information includes a gaze vector <NUM> of the user and/or a pixel location of dynamic dimmer <NUM> at which gaze vector <NUM> intersects with dynamic dimmer <NUM>. As illustrated, dynamic dimmer <NUM> may be adjusted to set dimmed area <NUM> to cover a portion of the system field of view corresponding to an intersection point (or intersection region) between gaze vector <NUM> and dynamic dimmer <NUM> or, alternatively, in some embodiments dimmed area <NUM> may cover a portion of the system field of view that does not correspond to the intersection point (or intersection region) between gaze vector <NUM> and dynamic dimmer <NUM>. In some embodiments, the dimming values of dimmed area <NUM> may be determined based on world light <NUM> detected by ambient light sensor <NUM> and/or the brightness of virtual image light <NUM>. In some embodiments, gaze information may be detected by an eye tracker <NUM> mounted to AR device <NUM>.

<FIG> illustrates a schematic view of a wearable AR device <NUM> according to the present invention. AR device <NUM> may include a left eyepiece 302A and a left dynamic dimmer 303A arranged in a side-by-side configuration and a right eyepiece 302B and a right dynamic dimmer 303B also arranged in a side-by-side configuration. In some embodiments, AR device <NUM> includes one or more sensors including, but not limited to: a left front-facing world camera 306A attached directly to or near left eyepiece 302A, a right front-facing world camera 306B attached directly to or near right eyepiece 302B, a left side-facing world camera 306C attached directly to or near left eyepiece 302A, a right side-facing world camera 306D attached directly to or near right eyepiece 302B, a left eye tracker 340A positioned so as to observe a left eye of a user, a right eye tracker 340B positioned so as to observe a right eye of a user, and an ambient light sensor <NUM>. In some embodiments, AR device <NUM> includes one or more image projection devices such as a left projector 314A optically linked to left eyepiece 302A and a right projector 314B optically linked to right eyepiece 302B.

Some or all of the components of AR device <NUM> may be head mounted such that projected images may be viewed by a user. In one particular implementation, all of the components of AR device <NUM> shown in <FIG> are mounted onto a single device (e.g., a single headset) wearable by a user. In another implementation, a processing module <NUM> is physically separate from and communicatively coupled to the other components of AR device <NUM> by one or more wired and/or wireless connections. For example, processing module <NUM> may be mounted in a variety of configurations, such as fixedly attached to a frame, fixedly attached to a helmet or hat worn by a user, embedded in headphones, or otherwise removably attached to a user (e.g., in a backpack-style configuration, in a belt-coupling style configuration, etc.).

Processing module <NUM> may include a processor <NUM> and an associated digital memory <NUM>, such as non-volatile memory (e.g., flash memory), both of which may be utilized to assist in the processing, caching, and storage of data. The data may include data captured from sensors (which may be, e.g., operatively coupled to AR device <NUM>) or otherwise attached to a user, such as cameras <NUM>, ambient light sensor <NUM>, eye trackers <NUM>, microphones, inertial measurement units, accelerometers, compasses, GPS units, radio devices, and/or gyros. For example, processing module <NUM> may receive image(s) <NUM> from cameras <NUM>. Specifically, processing module <NUM> may receive left front image(s) 320A from left front-facing world camera 306A, right front image(s) 320B from right front-facing world camera 306B, left side image(s) 320C from left side-facing world camera 306C, and right side image(s) 320D from right side-facing world camera 306D. In some embodiments, image(s) <NUM> may include a single image, a pair of images, a video comprising a stream of images, a video comprising a stream of paired images, and the like. Image(s) <NUM> may be periodically generated and sent to processing module <NUM> while AR device <NUM> is powered on, or may be generated in response to an instruction sent by processing module <NUM> to one or more of the cameras. As another example, processing module <NUM> may receive light information from ambient light sensor <NUM>. In some embodiments, some or all of the functionality of ambient light sensor <NUM> may be provided by way of one or more of world cameras 306A-306D. As another example, processing module <NUM> may receive gaze information from one or both of eye trackers <NUM>. As another example, processing module <NUM> may receive image information (e.g., image brightness values) from one or both of projectors <NUM>.

Eyepieces 302A and 302B may comprise transparent or semi-transparent waveguides configured to direct light from projectors 314A and 314B, respectively. Specifically, processing module <NUM> may cause left projector 314A to output a left virtual image light 322A onto left eyepiece 302A (causing a corresponding light field associated with left virtual image light 322A to be projected onto the user's retina), and may cause right projector 314B to output a right virtual image light 322B onto right eyepiece 302B (causing a corresponding light field associated with right virtual image light 322B to be projected onto the user's retina). In some embodiments, each of eyepieces <NUM> may comprise a plurality of waveguides corresponding to different colors and/or different depth planes. In some embodiments, dynamic dimmers <NUM> may be coupled to and/or integrated with eyepieces <NUM>. For example, one of dynamic dimmers <NUM> may be incorporated into a multi-layer eyepiece and may form one or more layers that make up one of eyepieces <NUM>.

Cameras 306A and 306B may be positioned to capture images that substantially overlap with the field of view of a user's left and right eyes, respectively. Accordingly, placement of cameras <NUM> may be near a user's eyes but not so near as to obscure the user's field of view. Alternatively or additionally, cameras 306A and 306B may be positioned so as to align with the incoupling locations of virtual image light 322A and 322B, respectively. Cameras 306C and 306D may be positioned to capture images to the side of a user, e.g., in a user's peripheral vision or outside the user's peripheral vision. Image(s) 320C and 320D captured using cameras 306C and 306D need not necessarily overlap with image(s) 320A and 320B captured using cameras 306A and 306B.

One or more components of AR device <NUM> may be similar to one or more components described in reference to <FIG>. For example, in the some embodiments the functionality of eyepieces <NUM>, dynamic dimmers <NUM>, projectors <NUM>, ambient light sensor <NUM>, and eye trackers <NUM> may be similar to eyepiece <NUM>, dynamic dimmer <NUM>, projector <NUM>, ambient light sensor <NUM>, and eye tracker <NUM>, respectively. In some embodiments, the functionality of processing module <NUM> may be implemented by two or more sets of electronic hardware components that are housed separately but communicatively coupled. For example, the functionality of processing module <NUM> may be carried out by electronic hardware components housed within a headset in conjunction with electronic hardware components housed within a computing device physically tethered to the headset, one or more electronic devices within the environment of the headset (e.g., smart phones, computers, peripheral devices, smart appliances, etc.), one or more remotely-located computing devices (e.g., servers, cloud computing devices, etc.), or a combination thereof. One example of such a configuration is described in further detail below in reference to <FIG>.

<FIG> illustrates a method <NUM> for operating an optical system (e.g., AR device <NUM> or <NUM>). Steps of method <NUM> may be performed in a different order than that shown in <FIG>, and not all of the steps need be performed. For example, in some embodiments, one or more of steps <NUM>, <NUM>, and <NUM> may be omitted during performance of method <NUM>. One or more steps of method <NUM> may be performed by a processor (e.g., processor <NUM>) or by some other component within the optical system.

At step <NUM>, light (e.g., world light <NUM>) associated with a world object (e.g., world objects <NUM>) is received at the optical system. The world object may be any number of real-world objects, such as a tree, a person, a house, a building, the sun, etc., that is viewed by a user of the optical system. In some embodiments, the light associated with the world object is first received by a dynamic dimmer (e.g., dynamic dimmer <NUM> or <NUM>) or by an external cosmetic lens of the optical system. In some embodiments, the light associated with the world object is considered to be received at the optical system when the light reaches one or more components of the optical system (e.g., when the light reaches the dynamic dimmer).

At step <NUM>, virtual image light (e.g., virtual image light <NUM> or <NUM>) is projected onto an eyepiece (e.g., eyepiece <NUM> or <NUM>). The virtual image light may be projected onto the eyepiece by a projector (e.g., projector <NUM> or <NUM>) of the optical system. The virtual image light may correspond to a single image, a pair of images, a video comprising a stream of images, a video comprising a stream of paired images, and the like. In some embodiments, the virtual image light is considered to be projected onto the eyepiece when any light associated with the virtual image light reaches the eyepiece. In some embodiments, projecting the virtual image light onto the eyepiece causes a light field (i.e., an angular representation of virtual content) to be projected onto the user's retina in a manner such that the user perceives the corresponding virtual content as being positioned at some location within the user's environment.

During steps <NUM>, <NUM>, and <NUM>, information may be detected by the optical system using, for example, one or more sensors of the optical system. At step <NUM>, light information corresponding to the light associated with the world object is detected. The light information may be detected using a light sensor (e.g., ambient light sensor <NUM> or <NUM>) mounted to the optical system. In some embodiments, the light information includes a plurality of spatially-resolved light values. Each of the plurality of spatially-resolved light values may correspond to a two-dimensional position within the system field of view. For example, each of the light values may be associated with a pixel of the dynamic dimmer. In other embodiments, or in the same embodiments, the light information may include a global light value. The global light value may be associated with the entire system field of view (e.g., an average light value of light impinging on all pixels of the dynamic dimmer).

At step <NUM>, gaze information corresponding to an eye of a user of the optical system is detected. The gaze information may be detected using an eye tracker (e.g., eye tracker <NUM> or <NUM>) mounted to the optical system. In some embodiments, the gaze information includes a gaze vector (e.g., gaze vector <NUM>) of the eye of the user. In some embodiments, the gaze information includes one or more of a pupil position of the eye of the user, a center of rotation of the eye of the user, a pupil size of the eye of the user, a pupil diameter of the eye of the user, and cone and rod locations of the eye of the user. The gaze vector may be determined based on one or more components of the gaze information, such as the pupil position, the center of rotation of the eye, the pupil size, the pupil diameter, and/or the cone and rod locations. When the gaze vector is determined based on the cone and rod locations, it may further be determined based on the light information (e.g., the global light value) so as to determine an origin of the gaze vector within a retinal layer of the eye containing the cone and rod locations. In some embodiments, the gaze information includes a pixel or group of pixels of the dynamic dimmer at which the gaze vector intersects with the dynamic dimmer.

At step <NUM>, image information corresponding to the virtual image light (e.g., virtual image light <NUM> or <NUM>) projected by the projector onto the eyepiece is detected. The image information may be detected by the projector, by a processor (e.g., processor <NUM>), or by a separate light sensor. In some embodiments, the image information includes one or more locations within the dynamic dimmer through which the user perceives the virtual content when the user observes the virtual image light. In some embodiments, the image information includes a plurality of spatially-resolved image brightness values (e.g., brightness of the perceived virtual content). For example, each of the image brightness values may be associated with a pixel of the eyepiece or of the dynamic dimmer. In one particular implementation, when the processor sends instructions to the projector to project the virtual image light onto the eyepiece, the processor may determine, based on the instructions, the spatially-resolved image brightness values. In another particular implementation, when the projector receives the instructions from the processor to project the virtual image light onto the eyepiece, the projector sends the spatially-resolved image brightness values to the processor. In another particular implementation, a light sensor positioned on or near the eyepiece detects and sends the spatially-resolved image brightness values to the processor. In other embodiments, or in the same embodiments, the image information includes a global image brightness value. The global image brightness value may be associated with the entire system field of view (e.g., an average image brightness value of all of the virtual image light).

At step <NUM>, a portion of the system field of view to be at least partially dimmed is determined based on the detected information. The detected information may include the light information detected during step <NUM>, the gaze information detected during step <NUM>, and/or the image information detected during step <NUM>. In some embodiments, the portion of the system field of view is equal to the entire system field of view. In various embodiments, the portion of the system field of view may be equal to <NUM>%, <NUM>%, <NUM>%, <NUM>%, <NUM>%, or <NUM>%, etc., of the system field of view. In some embodiments, the different types of information may be weighted differently in determining the portion to be at least partially dimmed. For example, gaze information, when available, may be weighted more heavily in determining the portion to be at least partially dimmed than light information and image information. In one particular implementation, each type of information may independently be used to determine a different portion of the system field of view to be at least partially dimmed, and subsequently the different portions may be combined into a single portion using an AND or an OR operation.

In some embodiments, the information used to determine a portion of the system field of view to be at least partially dimmed includes information associated with one or more objects that are presented within the virtual content. For example, the virtual content may include text, navigational indicators (e.g., arrows), and/or other content. The portion of the field of view in which such content is to be presented, and/or the field of view proximal to the content, can be dimmed such that the user can more easily read perceive and understand the content, and distinguish the content from world object(s). The dimmer can selectively dim one or more pixels and/or zone(s) of pixels, or enhance viewing of the content. In one example, a section of the lower portion of the field of view can be selectively and dynamically dimmed to make is easier for the user to see directional (e.g., navigation) arrows, text messages, and so forth. Such dimming may be performed while the content is being displayed in response to a determination that such content is to be displayed, and the dimming may be removed when the content is no longer displayed. In some instances, the dimming may be performed to mitigate artifacts caused by the pixel structure that enables dimming over the entire field of view.

At step <NUM>, a plurality of spatially-resolved dimming values for the portion of the system field of view are determined based on the detected information. In some embodiments, the dimming values are determined using a formulaic approach based on a desired opacity or visibility of the virtual content. In one particular implementation, the visibility of the virtual content may be calculated using the following equation: <MAT> where Vis the visibility, Imax is the brightness of the virtual image light as indicated by the image information, Iback is related to a light value associated with the world object as indicated by the light information (which may be modified by the determined dimming value), and C is a desired contrast (e.g., <NUM>: <NUM>). For example, the visibility equation may be used at each pixel location of the dimmer to calculate a dimming value for the particular pixel location using the brightness of the virtual image light at the particular pixel location and the light value associated with the world object at the particular pixel location. In some embodiments, Iback may be defined using the following equation: <MAT> where Tv is the percentage of light that is allowed to pass through one or more pixels of the dimmer, and Iworld is the brightness of ambient light from the world as indicated by the light information. In some examples, Tv may be representative of or related to a dimming value.

At step <NUM>, the dimmer is adjusted to reduce an intensity of the light associated with the object in the portion of the system field of view. For example, the dimmer may be adjusted such that the intensity of the light associated with the object impinging on each pixel location of the dimmer is reduced according to the dimming value determined for that particular pixel location. As used in the present disclosure, adjusting the dimmer may include initializing the dimmer, activating the dimmer, powering on the dimmer, modifying or changing a previously initialized, activated, and/or powered on dimmer, and the like. In some embodiments, the processor may send data to the dimmer indicating both the portion of the system field of view and the plurality of spatially-resolved dimming values.

At step <NUM>, the projector is adjusted to adjust a brightness associated with the virtual image light. For example, in some embodiments it may be difficult to achieve a desired opacity or visibility of the virtual content without increasing or decreasing the brightness of the virtual object. In such embodiments, the brightness of the virtual image light may be adjusted before, after, simultaneously, or concurrently with adjusting the dimmer.

<FIG> illustrates an AR device <NUM> with an eyepiece <NUM> and a pixelated dimming element <NUM> consisting of a spatial grid of dimming areas (i.e., pixels) that can have various levels of dimming. Each of the dimming areas may have an associated size (i.e., width) and an associated spacing (i.e., pitch). As illustrated, the spatial grid of dimming areas may include one or more dark pixels <NUM> providing complete dimming of incident light and one or more clear pixels <NUM> providing complete transmission of incident light. Adjacent pixels within pixelated dimming element <NUM> may be bordering (e.g., when the pitch is equal to the size) or may be separated by gaps (e.g., when the pitch is greater than the size). In various embodiments, pixelated dimming element <NUM> may employ liquid crystal technology such as dye doped or guest host liquid crystals, twisted nematic (TN) or vertically aligned (VA) liquid crystals, or ferroelectric liquid crystals. In some embodiments, pixelated dimming element <NUM> may comprise an electrochromic device, among other possibilities. In some implementations, pixelated dimming element <NUM> may employ electrically controlled birefringence ("ECB") technology, such as an ECB cell.

<FIG> illustrates a technique for determining the gaze vector based on the pupil position of the eye of the user. In some instances, the pupil position relative to the AR device is detected using the eye tracker and the gaze vector is subsequently defined as the vector orthogonal to the surface of the eye at the pupil position. The gaze vector may alternatively or additionally be defined as the vector intersecting the center of rotation of the eye and the pupil position. The center of rotation may be estimated using data gathered by the eye tracker. The gaze vector may alternatively or additionally be defined as the vector intersecting the geometric center of the eye and the pupil position. The geometric center of the eye may be estimated using data gathered by the eye tracker. Other possibilities are contemplated.

One of several inherent problems with using the pupil position to determine the gaze vector is illustrated in <FIG>. In the upper diagram, a first distance D<NUM> between the pupil position and the eyepiece is shown when the eye is looking generally toward the center of the eyepiece. In the lower diagram, a second distance D<NUM> between the pupil position and the eyepiece is shown when the eye is looking generally toward the top of the eyepiece. Here, the first distance D<NUM> is less than the second distance D<NUM>, causing render registration problems due to the varying vergence distance as the eye of the user moves.

<FIG> illustrates a technique for determining the gaze vector based on the center of rotation of the eye of a user. This technique is described in depth in <CIT> titled "EYE CENTER OF ROTATION DETERMINATION, DEPTH PLANE SELECTION, AND RENDER CAMERA POSITIONING IN DISPLAY SYSTEMS". The center of rotation may be estimated using data gathered by the eye tracker, and the gaze vector may subsequently be defined as the vector formed by connecting the center of rotation and the pupil position. One of several benefits of using the center of rotation for determining the gaze vector is that the distance between the center of rotation and the eyepiece may be the same irrespective of the direction the eye is looking. In the upper diagram of <FIG>, a third distance D<NUM> between the center of rotation and the eyepiece is shown when the eye is looking generally toward the center of the eyepiece. In the lower diagram, a fourth distance D<NUM> between the center of rotation and the eyepiece is shown when the eye is looking generally toward the top of the eyepiece. Here, the third distance D<NUM> is the same as the fourth distance D<NUM> thereby improving the render registration.

<FIG> illustrates a technique for determining the gaze vector based on detected light information and the cone and rod locations within the eye. Because cones are more sensitive to light in high light conditions and rods are more sensitive to light in low light conditions, as the detected ambient light decreases (e.g., the global light value), the origin of the gaze vector may be adjusted from a center position of the retinal layer corresponding to a high density of cones outward to one or more points along an annulus corresponding to a high density of rods. Accordingly, in high light conditions the determined gaze vector may be a single gaze vector formed by connecting a center position of the retinal layer to the pupil position, and in low light conditions the determined gaze vector(s) may be a single or a plurality of gaze vectors formed by connecting one or more points along an annulus surrounding the center position of the retinal layer to the pupil position. Alternatively or additionally, the plurality of gaze vectors may be described/represented as a cone of gaze vectors or a "gaze cone" comprising an infinite number of possible gaze vectors.

Cone and rod locations may be estimated using information gathered by the eye tracker or, in some embodiments, the center position of the retinal layer corresponding to a high density of cones may be defined by continuing the gaze vector determined using the pupil position through the eye toward the back of the eye such that the gaze vector determined using the pupil position is co-linear with the gaze vector determined using cone and rod locations in high light conditions. In some embodiments, the AR device is configured such that the gaze vector is determined using cone and rod locations in low light conditions (e.g., "low light mode") and is determined using the center of rotation of the eye in high light conditions. In such embodiments, a light threshold may be established that a detected light value may be evaluated against, causing the gaze vector to be determined using cone and rod locations when the detected light value is below the light threshold and causing the gaze vector to be determined using the center of rotation of the eye when the detected light value is above the light threshold.

In some embodiments in which the dimmed area is significantly large and/or the dimming values are significantly high, the detected ambient light using the light sensor of the AR device may not be indicative of the actual amount of light reaching the eye. In such embodiments, the size of the pupil may be used as a proxy for the amount of light reaching the eye. For example, the AR device may switch to a "low light mode" (causing the gaze vector to be determined using cone and rod locations) when the pupil size exceeds a pupil size threshold. For example, in some implementations the pupil size threshold may be set to be <NUM>% above an average pupil size of a user in high light conditions (e.g., pupil size may correspond to area, diameter, circumference, etc. of the pupil). In another particular embodiment, the pupil size threshold may be predetermined based on average known pupil sizes in low light and high light conditions. Other possibilities are contemplated.

<FIG> illustrates a determined gaze vector in high light conditions in which the pupil is contracted. In some embodiments, the pupil size may be used to estimate the ambient light (e.g., the global light value) or, alternatively or additionally, the origin(s) of the gaze vector(s) may be determined directly using the pupil size without estimation or detection of the ambient light. For example, different pupil diameters may be related to different cone and rod locations within the retinal layer at which the origin(s) of the gaze vector(s) may be defined.

<FIG> illustrates determined gaze vectors in low light conditions in which the pupil is dilated. Similar to the scenario in high light conditions, in low light conditions the pupil size may be used to estimate the ambient light (e.g., the global light value) or, alternatively or additionally, the origin(s) of the gaze vector(s) may be determined directly using the pupil size.

<FIG> illustrates three techniques for determining the gaze vector in high light conditions and the corresponding dimmed areas determined using each of the three techniques. In the first technique, the gaze vector is determined using the pupil position, i.e., "GAZE VECTOR(A)", resulting in a gaze vector that extends orthogonally from the surface of the pupil toward the dimmed area A (or in some embodiments, the area that is not dimmed). In the second technique, the gaze vector is determined using cone and rod locations within the eye, i.e., "GAZE VECTOR(B)", resulting in a gaze vector that extends from a center position of the retinal layer through the pupil position toward the dimmed area B (or in some embodiments, the area that is not dimmed). The second technique may be further facilitated by one or more of: the pupil position (for providing a second point for defining the gaze vector), the detected ambient light (for determining the origin(s) of the gaze vector(s) along the retinal layer), and the pupil size/diameter (for estimating the ambient light and/or for directly determining the origin(s) of the gaze vector(s) along the retinal layer). In the third technique, the gaze vector is determined using the center of rotation of the eye, i.e., "GAZE VECTOR(C)", resulting in a gaze vector that extends from the center of rotation of the eye through the pupil position toward the dimmed area C (or in some embodiments, the area that is not dimmed). In the example of <FIG>, dimmed area A is the same as dimmed area B.

<FIG> illustrates the same techniques illustrated in <FIG> but in low light conditions. The determined gaze vectors and the corresponding dimmed areas are the same using the first and third techniques (using pupil position and center of rotation, respectively) but have been modified using the second technique (using cone and rod locations). In the second technique, the gaze vectors are determined using cone and rod locations within the eye, i.e., "GAZE VECTOR(B')", resulting in a set of gaze vectors that extend from various points along an annulus surrounding the center position of the retinal layer through the pupil position toward the dimmed area B' (or in some embodiments, the area that is not dimmed). In the example shown in <FIG>, each of dimmed areas A, B', and C differ from each other.

<FIG> illustrates a dimmer having been adjusted to produce dimmed area A determined using a gaze vector calculated using the pupil position.

<FIG> illustrates a dimmer having been adjusted to produce dimmed area B determined using a gaze vector calculated using cone and rod locations in high light conditions.

<FIG> illustrates a dimmer having been adjusted to produce dimmed area B' determined using a gaze vector calculated using cone and rod locations in low light conditions. In alternative embodiments, dimmed area B' may include only portions of the annular region shown in <FIG> and not the region in its entirety.

<FIG> illustrates an example in which dimmed area B' further includes the center portion within the annular region.

<FIG> illustrates a dimmer having been adjusted to produce dimmed area C determined using a gaze vector calculated using the center of rotation of the eye.

<FIG> and <FIG> illustrate an approach for determining a portion of the system field of view to be dimmed based on image information. For example, one or more steps shown in <FIG> and <FIG> may correspond to steps <NUM> and/or <NUM>. In some embodiments, the AR device may project light onto the eyepiece in such a way that virtual content is perceived by the user at various points in space beyond the eyepiece and the dynamic dimmer, such as points <NUM>. Points <NUM> may, for example, correspond to locations in three-dimensional space including locations at which pixels of virtual content (e.g., one or more virtual objects) are to be perceived by the user when presented through the eyepiece, locations at which dark virtual content (e.g., a virtual "shadow" cast by or otherwise associated with virtual content presented through the eyepiece) is to be perceived by the user, locations physically occupied by one or more real-world objects or persons located in the user's environment (e.g., a virtual black "top hat" anchored to the head of someone in the user's environment), and the like. In some implementations, points <NUM> may be randomly sampled from the virtual content or, in some embodiments, points <NUM> may be selected based on key features of the virtual content, such as edges, corners, centers of surfaces, among other possibilities. In some embodiments, points <NUM> may be sampled from the outer perimeter of the virtual content (as viewed from a reference point). In other embodiments, or in the same embodiments, an image brightness of the virtual content is also determined at each of points <NUM>, which may be used to determine a level of dimming (i.e., dimming value) at points <NUM> to achieve a desired visibility V of the virtual content. The number of points <NUM> used may vary based on a speed-accuracy tradeoff.

To dim in alignment with the perceived virtual content, vectors <NUM> may be defined as intersecting each of points <NUM> and the pupil position (i.e., a reference point). Intersection points <NUM> may then be defined at each location where vectors <NUM> intersect with the dynamic dimmer. As shown in reference to <FIG>, dimmed portions <NUM> may be determined based on intersection points <NUM>. In some implementations, one or more ray- or cone-casting techniques may be employed to define vectors <NUM> and identify or otherwise determine intersection points <NUM>. In some embodiments, each of dimmed portions <NUM> may be set to an area encompassing each of intersection points <NUM>, or to particular pixels of the dynamic dimmer encompassing intersection points <NUM>. In some embodiments, the size of dimmed portions <NUM> may be a function of the number of sampled points <NUM> and/or the density of points <NUM>. For example, in some instances the size of dimmed portions <NUM> may be inversely proportional to the number of points <NUM>. In embodiments in which points <NUM> are sampled from the outer perimeter of the virtual content, dimmed portions <NUM> may be formed by connecting neighboring intersection points <NUM> and dimming the enclosed area. In some examples, the size and/or shading of dimmed portions <NUM> may be a function of determined distances from the reference point to intersection points <NUM>, determined distances from intersection points <NUM> to points <NUM>, or a combination thereof. In the example of <FIG> and <FIG>, the pupil position (e.g., center of the pupil), which is the location from which vectors <NUM> are defined (i.e., a reference point), may change over time as eye movement occurs. As such, the locations of intersection points <NUM> and dimmed portions <NUM> may also change over time as eye movement occurs.

<FIG> and <FIG> illustrate an approach for determining a portion of the system field of view to be dimmed based on image information similar to shown in reference to <FIG> and <FIG> but with a different reference point. Points <NUM> may represent different points in space where the virtual content is perceived by the user. Vectors <NUM> may be defined as intersecting each of points <NUM> and the center of rotation of the eye (i.e., a reference point). Intersection points <NUM> may then be defined at each location where vectors <NUM> intersect with the dynamic dimmer. As shown in reference to <FIG>, dimmed portions <NUM> may be determined based on intersection points <NUM>. In some embodiments, each of dimmed portions <NUM> may be set to an area encompassing each of intersection points <NUM> or to particular pixels of the dynamic dimmer encompassing intersection points <NUM>. In some examples, the size and/or shading of dimmed portions <NUM> may be a function of determined distances from the reference point to intersection points <NUM>, determined distances from intersection points <NUM> to points <NUM>, or a combination thereof. The position of the center of rotation of the eye, which is the location from which vectors <NUM> are defined (i.e., a reference point) in the example of <FIG> and <FIG>, may be more stable over time as eye movement occurs than that of the pupil position, which is the reference point in the example of <FIG> and <FIG>. It follows that, in the example of <FIG> and <FIG>, the locations of intersection points <NUM> and dimmed portions <NUM> may remain static or change relatively little over time as eye movement occurs. Although the pupil position and the center of rotation of the eye are described above in reference to <FIG>, <FIG>, <FIG>, and <FIG> as examples of reference points that may be utilized in determining a portion of the system field of view to be dimmed, it is to be understood that examples of such reference points may also include any of a variety of other locations along the optical axis of the eye. Systems and techniques for identifying the optical axis of the eye and the locations of particular anatomical regions of the eye that lie along the optical axis, such as the center of the pupil and the center of rotation of the eye, are described in further detail in U. Publication <CIT> titled "EYE CENTER OF ROTATION DETERMINATION, DEPTH PLANE SELECTION, AND RENDER CAMERA POSITIONING IN DISPLAY SYSTEMS,".

<FIG> illustrates examples of improving the solidity of displayed virtual content using any of the techniques described herein, such as adjusting the dimmer and/or adjusting the projector based on light information, gaze information, and/or image information. In reference to both the left and right side field of views, virtual content <NUM> which is displayed alongside world objects <NUM> appears washed out except for portions <NUM> of virtual content <NUM> where the virtual content appears more solid than the remaining portions of virtual content <NUM>. As shown in the illustrated examples, the solidity of the virtual content is improved only at the portions of the system field of view where the user is looking.

<FIG> illustrates an example of improving the solidity of a displayed virtual object <NUM> by dimming a portion of the system field of view corresponding to the virtual object. As illustrated, the opacity and visibility of a portion <NUM> of virtual object <NUM> in the region with dimming is relatively greater than that of a portion <NUM> of virtual object <NUM> in the region without dimming. By dimming the light associated with world objects <NUM> at portion <NUM>, the virtual content can be more clearly perceived by the user.

<FIG> illustrates a plot showing the relationship between virtual image light brightness (x-axis) and ambient light levels for maintaining a visibility equal to <NUM> (i.e., V = <NUM>). The solid slanted lines are fixed visibility level lines (for V = <NUM>) for different ambient light level conditions. For example, for a projector brightness of <NUM> nits being used in an indoor area of about <NUM> nits, a dimming level close to <NUM>% may be employed to keep the visibility close to <NUM>. Referring once again to the visibility equation described above in reference to <FIG>, in some examples, the x and y axes of the plot illustrated in <FIG> may correspond to Imax and Tv, respectively, while the solid slanted lines are fixed visibility level lines (for V = <NUM>) for different Iworld values.

<FIG> and <FIG> illustrate diagrams showing the effect of a small occlusion on a world scene. <FIG> illustrates a simple case in which an eye of the user is looking at infinity. The eye includes a retina <NUM>, a pupil <NUM>, and a lens <NUM>. Light from different angles are focused to different positions on retina <NUM>. <FIG> shows an occlusor <NUM> placed in front of the eye at a distance d away from pupil <NUM>. A gradient disk at the retina may be constructed using simple ray geometry. Ignoring diffraction, the relative transmission at the center of the gradient disk is t<NUM> = <NUM>- (h/p)<NUM>, where h is the diameter of the occlusor and p is the diameter of the pupil. Put another way, t<NUM> = <NUM>- Aocclusor/Apupil, where Aocclusor is the area of the occlusor and Apupil is the area of the pupil.

<FIG> illustrates a plot showing the effect of varying the occlusor diameter on the transmission of the dimming element as a function of angular extent (in degrees). As illustrated, a smaller occlusor diameter (e.g., <NUM>) has very little effect on the transmission but is much more stable over angular extent than a larger occlusor diameter (e.g., <NUM>) which has a higher effect on the transmission which varies significantly over angular extent.

<FIG> illustrates an example of dimming using a single occlusor in which d = <NUM>, p = <NUM> mm, and h = <NUM>. The dimmed area shows a point spread function (PSF) <NUM> of a single pixel. Using the dimming shown, the pixel size requirement for the particular dimming element used can be estimated as a <NUM> pixel.

<FIG> illustrates an example of an architecture of an OST head-mounted display (HMD) consisting of a diffractive waveguide eyepiece <NUM> that delivers the virtual content to the user's eyes. The diffractive waveguide eyepiece <NUM> may include one or more diffractive optical elements (DOEs), such as an in-coupling grating (ICG), an orthogonal pupil expander (OPE), and/or an exit pupil expander (EPE). The world light also passes through the same element to reach the user's eyes. As shown, a dynamic dimmer <NUM> allows management of the world light level to keep the virtual content at a certain opacity level. In some embodiments, the dimmer <NUM> may correspond to a pixelated dimming element that is functionally similar or equivalent to pixelated dimming element <NUM> as described above in reference to <FIG>. In other embodiments, the dimmer <NUM> may correspond to a global (non-pixelated) dimming element. As shown in <FIG>, in some implementations, the dimmer <NUM> may be shaped and curved independent of the eyepiece so as to improve the aesthetics and/or the functionality of the OST-HMD.

<FIG> illustrates an additional example of an architecture of an OST-HMD consisting of a micro-display (e.g., LCOS, MEMS, or fiber scanner display type) that delivers light with a relay optics system into an in-coupling grating of a diffractive waveguide structure. The waveguide structure may include an outcoupling grating (e.g., EPE) that magnifies the input image plane and delivers to the user's eyebox. As shown, various elements may be positioned between the user's eye and the world objects. An eyepiece <NUM> may be a diffractive waveguide combiner that delivers the virtual light to the user's eye and also allows the world light to transmit through. A variable focal element <NUM> may consist of a depth plane variation/switching element between the eye and the eyepiece to act on the virtual display. In some embodiments, variable focus element <NUM> is a back lens assembly (BLA) <NUM>. The BLA also invariably acts on the world light and therefore a front lens assembly (FLA) <NUM> is added to cancel the impact on the world display.

A dynamic dimming element <NUM> in this embodiment is mounted on the outside of the integrated stack. This allows switching from a transparent display for an AR mode to an opaque display completely blocking out the world light for a VR mode. The dimming element <NUM> may correspond to a global dimming element or a pixelated dimming element. An external lens <NUM> is positioned separate from the optical stack so as to provide a protective and/or supportive structure for the OST-HMD. External lens <NUM> may also provide an amount of dimming to the entire system field of view.

<FIG> illustrates an additional example of an architecture of an OST-HMD in which a flat dynamic dimmer <NUM> is positioned along the inside of a curved external cosmetic lens <NUM>. The dimmer <NUM> may correspond to a global dimming element or a pixelated dimming element. In some embodiments, external cosmetic lens <NUM> may provide an amount of dimming to the entire system field of view which may be accounted for when determining the spatially-resolved dimming values of the dynamic dimmer. The OST-HMD may also include an eyepiece <NUM>, an adaptive BLA <NUM>, and an adaptive FLA <NUM>, as described herein.

<FIG> illustrates a schematic view of an AR device <NUM> according to the present invention. AR device <NUM> generally includes a local module <NUM> and a remote module <NUM>. Partitioning of components of AR device <NUM> between local module <NUM> and remote module may allow separation of bulky and/or high power-consuming components from those positioned close to the user's head when AR device <NUM> is in use, thereby increasing user comfort as well as device performance. Local module <NUM> may be head mounted and may include various mechanical and electronic modules to facilitate control of pixelated dimmers <NUM> and spatial light modulators <NUM>. Control of spatial light modulators <NUM> may cause virtual content to be projected onto eyepieces <NUM> which, in conjunction with world light modified by dimmers <NUM>, are viewed by the user of AR device <NUM>. One or more sensors <NUM> of local module <NUM> may detect information from the world and/or the user and send the detected information to a sensor headset processor <NUM> which may send a data stream to a display headset processor <NUM> of local module <NUM> and raw or processed images to perception processing unit <NUM> of remote module <NUM>.

In some embodiments, one or more components of local module <NUM> may be similar to one or more components described with reference to <FIG>. For example, in such embodiments, the functionality of eyepieces <NUM> and dimmers <NUM> may be similar to that of eyepieces <NUM> and dimmers <NUM>, respectively. In some examples, the one or more sensors <NUM> may include one or more world cameras, ambient light sensors, and/or eye trackers similar to one or more of world cameras <NUM>, ambient light sensor <NUM>, and/or eye trackers <NUM>, respectively. In some embodiments, the functionality of spatial light modulators <NUM> may be similar to that of one or more components included in projectors <NUM>, and the functionality of one or both of the sensor headset processor <NUM> and the display headset processor <NUM> may be similar to that of one or more components included in processing module <NUM>.

In some embodiments, display headset processor <NUM> may receive virtual content data and pixelated dimmer data from a graphics processing unit (GPU) <NUM> of remote module <NUM> and may perform various correction and warping techniques prior to controlling pixelated dimmers <NUM> and spatial light modulators <NUM>. Dimmer data generated by display headset processor <NUM> may pass through one or more drivers which may modify or generate voltages for controlling dimmers <NUM>. In some embodiments, display headset processor <NUM> may receive a depth image and a headset pose from sensor headset processor <NUM> which can be used to improve the accuracy of the dimming and the projected virtual content.

Remote module <NUM> may be electrically coupled to local module <NUM> through one or more wired or wireless connections, and may be fixedly attached to the user or carried by the user, among other possibilities. Remote module <NUM> may include a perception processing unit <NUM> for performing/generating an environment lighting map, a headset pose, and eye sensing. Perception processing unit <NUM> may send data to a CPU <NUM> which may be configured to perform/generate passable world geometry and app scene geometry. CPU <NUM> may send data to GPU <NUM> which may be configured to perform a check for a minimum world luminance throughput, a dimming pixel alignment, a late frame time warp, and a render pipeline, among other operations. In some embodiments, CPU <NUM> may be integrated with GPU <NUM> such that a single processing unit may perform one or more of the functions described in reference to each. In some embodiments, the functionality of one or more of the components included in remote module <NUM> may be similar to that of one or more components included in processing module <NUM>.

<FIG> illustrates a method <NUM> for sharpening out-of-focus pixelated dimming. Method <NUM> may be used in addition to method <NUM> to improve the performance of the dimmer. For example, one or more steps of method <NUM> may be performed prior to step <NUM> and/or subsequent to step <NUM>. Steps of method <NUM> may be performed in a different order than that shown in <FIG>, and not all of the steps need be performed. For example, in some embodiments, one or more of steps <NUM>, <NUM>, and <NUM> may be omitted during performance of method <NUM>. One or more steps of method <NUM> may be performed by a processor (e.g., processor <NUM>) or by some other component within the AR device.

At step <NUM>, a pixelated mask is generated. When method <NUM> is used in conjunction with method <NUM>, the pixelated mask may be generated based on the portion of the system field of view to be dimmed determined in step <NUM> and/or the dimming values determined in step <NUM> (i.e., dimmed area <NUM>). In some embodiments, the pixelated mask may be generated using one or both of the techniques described in reference to <FIG> and <FIG>.

At step <NUM>, the dimmer is adjusted in accordance with the pixelated mask. For example, each pixel of the dimmer may be set to a particular dimming value as indicated by the pixelated mask. For examples in which method <NUM> is used in conjunction with method <NUM>, one or more operations of step <NUM> may at least in part correspond to one or more operations of step <NUM>.

At step <NUM>, the user looks through the dimmer and the observable dimming (to the user) is equivalent to the pixelated mask convolved with the PSF of a single pixel. Accordingly, step <NUM> is inherently performed by the optics of the eye when the user is wearing the AR device, rather than being performed directly by a component of the AR device. On the other hand, steps <NUM> and <NUM> may, for example, be performed by one or more components of the AR device.

<FIG> illustrates a method <NUM> for sharpening out-of-focus pixelated dimming in which step <NUM> comprises step <NUM> at which a desired pixelated mask is generated and step <NUM> at which the pixelated mask is set to the desired pixelated mask. The observable dimming to the user (shown adjacent to step <NUM>) is significantly blurred in comparison to the pixelated mask (shown adjacent to step <NUM>) due to the smearing caused by the PSF of a single pixel. In some embodiments, one or more steps of method <NUM> may correspond to one or more steps of method <NUM>.

<FIG> illustrates a method <NUM> for sharpening out-of-focus pixelated dimming in which step <NUM> includes a deconvolution technique. At step <NUM>, the diameter of the pupil of the eye of the user is detected using a sensor directed at the eye (e.g., a camera such as eye tracker <NUM>). At step <NUM>, the accommodative state of the eye lens is detected using the same or a different sensor as used in step <NUM>.

At step <NUM>, the PSF of a single pixel is estimated based on the pupil diameter, the accommodative state of the eye lens, the size/shape of a pixel, and the distance from a pixel to the pupil. Where the shape of a pixel can be approximated using a circle, the size/shape of a pixel may be represented as diameter h, the pupil diameter as diameter p, and the distance from a pixel to the pupil as distance d (using the nomenclature established in <FIG> and <FIG>). In some embodiments, the distance from a pixel to the pupil may be different for different pixels of the dimmer such that the estimated PSF may be pixel dependent. In some embodiments, the distance from the centermost pixel to the pupil is used as an approximation for the remaining pixels.

At step <NUM>, a desired pixelated mask at the depth of the virtual content is generated. At step <NUM>, the desired pixelated mask is deconvolved using the estimated PSF of a single pixel. In some embodiments, the deconvolution is performed in the spatial frequency domain by dividing the Fourier Transform of the desired pixelated mask at the depth of the virtual content by the Fourier Transform of the estimated PSF of a single pixel and performing an inverse-Fourier Transform. Alternatively or additionally, the deconvolution may be performed dividing the Fourier Transform of the estimated PSF of a single pixel by the Fourier Transform of the desired pixelated mask at the depth of the virtual content and performing an inverse-Fourier Transform. The pixelated mask used subsequently in step <NUM> is set to the result of the deconvolution.

As a result of performing method <NUM>, the observable dimming to the user (shown adjacent to step <NUM>) is significantly less blurred in comparison to the technique in method <NUM> despite the dissimilarity between the pixelated mask (shown adjacent to step <NUM>) and the desired pixelated mask (shown adjacent to step <NUM>). In some embodiments, one or more steps of method <NUM> may correspond to one or more steps of methods <NUM> and <NUM>. In some embodiments, one or more steps of method <NUM> may be omitted or modified.

<FIG> illustrates a simplified computer system <NUM> according to some embodiments described herein. Computer system <NUM> as illustrated in <FIG> may be incorporated into devices such as AR device <NUM> or <NUM> as described herein. <FIG> provides a schematic illustration of one example of computer system <NUM> that can perform some or all of the steps of the methods provided by various embodiments. It should be noted that <FIG> is meant only to provide a generalized illustration of various components, any or all of which may be utilized as appropriate. <FIG>, therefore, broadly illustrates how individual system elements may be implemented in a relatively separated or relatively more integrated manner.

Computer system <NUM> is shown comprising hardware elements that can be electrically coupled via a bus <NUM>, or may otherwise be in communication, as appropriate. The hardware elements may include one or more processors <NUM>, including without limitation one or more general-purpose processors and/or one or more special-purpose processors such as digital signal processing chips, graphics acceleration processors, and/or the like; one or more input devices <NUM>, which can include without limitation a mouse, a keyboard, a camera, and/or the like; and one or more output devices <NUM>, which can include without limitation a display device, a printer, and/or the like.

Computer system <NUM> may further include and/or be in communication with one or more non-transitory storage devices <NUM>, which can comprise, without limitation, local and/or network accessible storage, and/or can include, without limitation, a disk drive, a drive array, an optical storage device, a solid-state storage device, such as a random access memory ("RAM"), and/or a read-only memory ("ROM"), which can be programmable, flash-updateable, and/or the like. Such storage devices may be configured to implement any appropriate data stores, including without limitation, various file systems, database structures, and/or the like.

Computer system <NUM> might also include a communications subsystem <NUM>, which can include without limitation a modem, a network card (wireless or wired), an infrared communication device, a wireless communication device, and/or a chipset such as a Bluetooth™ device, an <NUM> device, a WiFi device, a WiMax device, cellular communication facilities, etc., and/or the like. The communications subsystem <NUM> may include one or more input and/or output communication interfaces to permit data to be exchanged with a network such as the network described below to name one example, other computer systems, television, and/or any other devices described herein. Depending on the desired functionality and/or other implementation concerns, a portable electronic device or similar device may communicate image and/or other information via the communications subsystem <NUM>. In other embodiments, a portable electronic device, e.g. the first electronic device, may be incorporated into computer system <NUM>, e.g., an electronic device as an input device <NUM>. In some embodiments, computer system <NUM> will further comprise a working memory <NUM>, which can include a RAM or ROM device, as described above.

Computer system <NUM> also can include software elements, shown as being currently located within the working memory <NUM>, including an operating system <NUM>, device drivers, executable libraries, and/or other code, such as one or more application programs <NUM>, which may comprise computer programs provided by various embodiments, and/or may be designed to implement methods, and/or configure systems, provided by other embodiments, as described herein. Merely by way of example, one or more procedures described with respect to the methods discussed above, might be implemented as code and/or instructions executable by a computer and/or a processor within a computer; in an aspect, then, such code and/or instructions can be used to configure and/or adapt a general purpose computer or other device to perform one or more operations in accordance with the described methods.

A set of these instructions and/or code may be stored on a non-transitory computer-readable storage medium, such as the storage device(s) <NUM> described above. In some cases, the storage medium might be incorporated within a computer system, such as computer system <NUM>. In other embodiments, the storage medium might be separate from a computer system e.g., a removable medium, such as a compact disc, and/or provided in an installation package, such that the storage medium can be used to program, configure, and/or adapt a general purpose computer with the instructions/code stored thereon. These instructions might take the form of executable code, which is executable by computer system <NUM> and/or might take the form of source and/or installable code, which, upon compilation and/or installation on computer system <NUM> e.g., using any of a variety of generally available compilers, installation programs, compression/decompression utilities, etc., then takes the form of executable code.

For example, customized hardware might also be used, and/or particular elements might be implemented in hardware, software including portable software, such as applets, etc., or both.

As mentioned above, in one aspect, some embodiments may employ a computer system such as computer system <NUM> to perform methods in accordance with various embodiments of the technology. According to a set of embodiments, some or all of the procedures of such methods are performed by computer system <NUM> in response to processor <NUM> executing one or more sequences of one or more instructions, which might be incorporated into the operating system <NUM> and/or other code, such as an application program <NUM>, contained in the working memory <NUM>. Such instructions may be read into the working memory <NUM> from another computer-readable medium, such as one or more of the storage device(s) <NUM>. Merely by way of example, execution of the sequences of instructions contained in the working memory <NUM> might cause the processor(s) <NUM> to perform one or more procedures of the methods described herein. Additionally or alternatively, portions of the methods described herein may be executed through specialized hardware.

The terms "machine-readable medium" and "computer-readable medium," as used herein, refer to any medium that participates in providing data that causes a machine to operate in a specific fashion. In embodiments implemented using computer system <NUM>, various computer-readable media might be involved in providing instructions/code to processor(s) <NUM> for execution and/or might be used to store and/or carry such instructions/code. In many implementations, a computer-readable medium is a physical and/or tangible storage medium. Such a medium may take the form of a non-volatile media or volatile media. Non-volatile media include, for example, optical and/or magnetic disks, such as the storage device(s) <NUM>. Volatile media include, without limitation, dynamic memory, such as the working memory <NUM>.

Common forms of physical and/or tangible computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, punchcards, papertape, any other physical medium with patterns of holes, a RAM, a PROM, EPROM, a FLASH-EPROM, any other memory chip or cartridge, or any other medium from which a computer can read instructions and/or code.

Various forms of computer-readable media may be involved in carrying one or more sequences of one or more instructions to the processor(s) <NUM> for execution. Merely by way of example, the instructions may initially be carried on a magnetic disk and/or optical disc of a remote computer. A remote computer might load the instructions into its dynamic memory and send the instructions as signals over a transmission medium to be received and/or executed by computer system <NUM>.

Subject to the limitations defined by the claimed invention, various configurations may omit, substitute, or add various procedures or components as appropriate.

Specific details are given in the description to provide a thorough understanding of exemplary configurations including implementations. However, subject to the limitations defined by the claimed invention, configurations may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary detail in order to avoid obscuring the configurations. This description provides example configurations only, and does not limit the scope, applicability, or configurations of the claims. Rather, the preceding description of the configurations will provide those skilled in the art with an enabling description for implementing described techniques. Various changes may be made in the function and arrangement of elements without departing from the scope of the invention, which is defined by the claims.

Also, configurations may be described as a process which is depicted as a schematic flowchart or block diagram. Although each may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. A process may have additional steps not included in the figure. Furthermore, examples of the methods may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored in a non-transitory computer-readable medium such as a storage medium. Processors may perform the described tasks.

Having described several example configurations, various modifications, alternative constructions, and equivalents may be used without departing from the scope of the invention, which is defined by the claims. For example, the above elements may be components of a larger system, wherein other rules may take precedence over or otherwise modify the application of the technology. Also, a number of steps may be undertaken before, during, or after the above elements are considered. Accordingly, the above description does not bind the scope of the claims.

Also, the words "comprise", "comprising", "contains", "containing", "include", "including", and "includes", when used in this specification and in the following claims, are intended to specify the presence of stated features, integers, components, or steps, but they do not preclude the presence or addition of one or more other features, integers, components, steps, acts, or groups.

Claim 1:
A method (<NUM>) of operating an optical system (<NUM>), the method (<NUM>) comprising:
receiving (<NUM>) data from an eye tracker (<NUM>) of the optical system (<NUM>), wherein the optical system (<NUM>) includes a frame configured to be worn about a head of a user of the optical system (<NUM>) and a dimming component (<NUM>) carried by the frame and configured to be positioned between an eye of the user and an environment of the user;
determining (<NUM>), based on the data received from the eye tracker (<NUM>), a location within the eye of the user at which a center of rotation of the eye of the user is positioned;
identifying one or more points (<NUM>) in three-dimensional space located within the environment of the user;
for each of the one or more identified points (<NUM>) within the environment of the user:
identifying (<NUM>) a set of one or more pixels (<NUM>) of the dimming component (<NUM>) based at least in part on the determined location of the center of rotation of the eye of the user and the respective point in three-dimensional space located within the environment of the user; and
controlling (<NUM>) the dimming component (<NUM>) to dim the identified set of one or more pixels (<NUM>).