Patent ID: 12244977

DETAILED DESCRIPTION

A virtual reality (VR) or an augmented reality (AR) device often includes a left projector coupled to a left beam path and a right projector coupled to a right beam path. The left projector is configured to generate a left image, and the left image is then propagated over the left beam path into a left eye of a user. The right projector is configured to generate a right image, and the right image is then propagated over the right beam path into a right eye of a user.

The structure of such a VR/AR device may change from regular use, temperature change, and/or shock. When the structure of the VR/AR device changes, the images may lose boresight and no longer be registered correctly to the VR/AR device. Also, the left and right eye images may lose relative boresight to each other. This problem can become severe in the VR/AR devices that resemble eyeglasses, because such VR/AR devices are not as rigid as traditional VR/AR devices.

Some of the existing VR/AR devices are configured to project calibration images and use the calibration images to determine whether the displays are properly positioned. Because such calibration images are visible to users, the user experience is often interfered with by calibration/correction operations.

The principles described herein solve the above-described problem by using a monitor light source (such as a laser diode or an array of laser diodes) to generate a monitor light beam to monitor the position and orientation of a projector. The projector has a reflective spatial light modulator, such as (but not limited to) liquid crystal on silicon (LCOS), digital micromirror device (DMD), grating light valve (GLV). The monitor light beam is directed into a monitor camera to obtain direct feedback, which can then be used to correct boresight changes in a VR/AR device. In some embodiments, a single camera is configured to combine the images from left and right eyes projectors and reduce the error in that measurement. In some embodiments, the monitor light source may be one or more edge emitter diodes or one or more vertical-external-cavity surface-emitting-laser (VECSEL) diodes with a very narrow wavelength band. In some embodiments, the monitor light source is an array of edge emitter diodes or VECSEL diodes configured to project an array of dots.

It is advantageous to use a laser light beam with a very narrow wavelength band as the monitor light beam because such a laser wavelength band may be chosen to be invisible to the user, or different than the illumination light beam (which is a visible light beam), such that the monitor light beam may be filtered from the illumination light beam. Further, the laser light power can be set to be greater than the illumination light itself, which overcomes signal-to-noise ratio issues at the monitor camera. Additionally, such a narrow wavelength band can be used with a very compact monitor camera based on a phase lens.

In some embodiments, the monitor light beam is modulated by the reflective spatial light modulator of the projector before being propagated over the monitor beam path to generate the monitor image. In such embodiments, the monitor camera can include (but is not limited to) a lensless camera or an angular sensitive pixel detector, a position sensing detector, and/or a quadrant diode detector. Alternatively, the monitor light beam is not modulated by the reflective spatial light modulator of the projector before being propagated over the monitor beam path to generate the monitor image, and the monitor camera can include (but is not limited to) a quadrant diode detector, a camera, and/or a lensless camera.

In some embodiments, the monitor light beam is directed into a different path in the beam path than the projector, which further improves the signal to noise at the monitor camera. Further, the dedicated beam path for the monitor signal that is separate from the projector signal provides laser safety to the end-user because there is no pathway that directs it toward the eyes of the user. The monitor camera enables measurement of the pose of left and right images or changes of both images together and monitors the correction applied. The projector can use multiple different projector designs, including (but not limited to) a multi-element lens system similar to the phone camera, birdbath that uses an additional curved mirror, etc. The integration of the monitor light beam and the illumination light beam can be from a same side (in parallel or forming an angle) or from different sides combined with a beam combiner, such as a dichroic beam combiner.

FIG.1Aillustrates an example architecture of a projection system100A that implements the principles described herein. The projection system100A includes an illumination light source110, a monitor light source120, and a projector140. The illumination light source110is configured to emit an illumination light beam112A at the projector140, and the monitor light source120is configured to emit a monitor light beam122A at the projector140. In some embodiments, the projector140includes a reflective spatial light modulator142configured to modulate the light beams112A and122A to generate an output144A of the projector140. The output144A of the projector140is a projected combined light beam144A that contains a projected illumination light beam144I and a projected monitor light beam144M.

Thereafter, the projected illumination light beam144I is directed to an illumination beam path152toward an eye160of a user, causing the eye160of the user to see a display image corresponding to the illumination light beam112A; the projected monitor light beam144M is directed to the monitor camera170, causing the monitor camera170to capture a monitor image corresponding to the monitor light beam122A. Since the illumination light beam112A and the monitor light beam122A are both projected through the projector140, the monitor image captured by the monitor camera170can be used to determine an orientation or a position of the monitor image. The monitor camera170can be (but is not limited to) a lensless camera, an angular sensitive pixel detector, and/or a quadrant diode detector as a position sensing detector.

In some embodiments, the illumination light source110is configured to emit light beams in a first wavelength band, e.g., visible Red-Green-Blue (RGB) light beam, including a red light beam, a green light beam, a blue light beam, or a combination thereof. The monitor light source120is configured to emit light beams in a second wavelength band, e.g., invisible light, such that the image generated by the monitor light source is only detectable to the monitor camera170, but invisible to the human eyes.

In some embodiments, the illumination beam path152is configured to propagate light in the first wavelength band, and the monitor beam path154is configured to propagate light in the second wavelength band. As such, the projected combined light beam144A is split into the illumination beam path152and the monitor beam path154. In some embodiments, a filter is disposed before the illumination beam path152to filter out the light beam in the second wavelength, such that only the light beam in the first wavelength band is propagated over the illumination beam path152. Alternatively, or in addition, a filter is disposed before the monitor beam path154to filter out the light beam in the first wavelength, such that only the monitor light beam in the second wavelength band is propagated over the monitor beam path154.

Since the monitor light beam112A is directed into a different path than the illumination light beam, it further improves the signal-to-noise ratio at the monitor camera170. In some embodiments, a power of the monitor light beam is greater than a power of the illumination light beam, such that the signal-to-noise ratio is further improved to allow identification of the monitor image at the camera170.

In some embodiments, the illumination light source110is configured to emit an illumination light beam in a first direction, and the monitor light source120is configured to emit a monitor light beam in a second direction that intersects the first direction. In some embodiments, the illumination light beam and the monitor light beam intersect at a first location in the projector, and output at two separate locations of the projector, namely a second location and a third location. The output beams are then propagated in different directions. The projected illumination light beam is propagated in a first direction toward an eye of a user, and the projected monitor light beam is propagated in a second direction toward a monitor camera.

Alternatively, in some embodiments, a beam combiner can be used to combine the illumination light and the monitor light into a combined light beam directed to the projector140.

FIG.1Billustrates an example architecture of a projection system100B that includes a beam combiner130configured to combine an illumination light112B and a monitor light beam122B into a combined light beam132directed to the projector140. The projector140is configured to project the combined light beam132into a projected combined light beam144B. Similar to the projected combined light beam144A inFIG.1A, the projected combined light beam144B is split and propagated over two different beam paths152,154.

The projection system100A or100B illustrated inFIG.1A or1Bcan be implemented in a portable projector and/or a head-mounted device, such as a VR/AR device, allowing the portable projector and/or the head-mounted device to self-monitor and/or adjust its boresight alignment. Notably, when implemented in a head-mounted device, two such projection systems are likely implemented, one for the left eye, and the other for the right eye.

Note, even though, as illustrated inFIGS.1A and1B, the monitor light beam122A or122B appears to be modulated by the reflective spatial light modulator142, it is not necessary that the monitor light beam122A or122B must be modulated by the reflective spatial light modulator142. In some embodiments, the monitor light beam122A is not modulated by the reflective spatial light modulator142before being propagated over the monitor beam path154. In such an embodiment, the monitor camera can be (but is not limited to) a quadrant diode detector, a camerae, or a lensless camera.

FIGS.2A and2Billustrate a front view and a top view of an example head-mounted device200that implements a left projection system200L and a right projection system200R, each of which corresponds to the projection system100A or100B ofFIG.1A or1B. As illustrated, the left projection system200L includes an illumination light source210L (corresponding to the illumination light source110ofFIG.1A or1B), and a monitor light source220L (corresponding to the monitor light source120ofFIG.1A or1B). In some embodiments, the left projection system200L also includes a beam combiner230L (corresponding to the beam combiner130ofFIG.1B).

Referring toFIG.2B, the illumination light source210L is configured to emit an illumination light beam212L, and the monitor light source220L is configured to emit a monitor light beam222L. The beam combiner230L is configured to combine the illumination light beam212L and the monitor light beam222L into a combined light beam232L, which is then projected by a projector240L into a projected combined light beam242L.

Referring toFIG.2A, the head-mounted device200also includes an illumination beam path252L (corresponding to the illumination beam path152ofFIG.1A or1B) and a monitor beam path254L (corresponding to the monitor beam path154ofFIG.1A or1B). A first portion of the projected combined light beam is propagated over the illumination beam path252L toward an eye of a user (not shown), and a second portion of the projected combined light beam is propagated over the monitor beam path254L toward the camera270(which corresponds to the monitor camera170ofFIG.1A or1B). The first portion of the projected combined light beam contains at least a portion of the illumination light beam projected by the projector240L, causing the eye of the user to see a display image corresponding to the illumination light beam. The second portion of the projected combined light beam contains at least a portion of the first monitor light beam projected by the projector240L.

Referring toFIG.2Bagain, in some embodiments, the two monitor beam paths254L and256L include a beam combiner260configured to combine the two monitor beams into a combined monitor beam262. The combined monitor beam262is then propagated into the monitor camera270.

The monitor camera270is configured to receive the second portion of the projected combined light beam and capture a monitor image corresponding to the monitor light beam projected by the projector240L. The monitor image is then analyzed to determine an orientation or a position of the monitor image. In response to determining that the monitor image is not properly oriented or positioned, an orientation or a position of the projector240L is adjusted; alternatively, or in addition, an orientation or a position of the illumination image is adjusted. For example, in some embodiments, the image data associated with the illumination image is transformed to cause the illumination image to be rotated for a particular angle based on the orientation of the monitor image. As another example, in some embodiments, the image data associated with the illumination image may be transformed to cause the illumination image to be moved, enlarged, and/or reduced.

In some embodiments, the illumination light source210L is configured to emit light beams in a first wavelength band, e.g., visible light, and the monitor light source220L is configured to emit light beams in a second wavelength band, e.g., invisible light, such that the image generated by the monitor light source is only visible to the monitor camera270, but invisible to the user. In some embodiments, a power of the monitor light beam is greater than a power of the illumination light beam, such that the monitor image captured by the monitor camera270has a sufficient signal-to-noise ratio to allow identification of the monitor image.

In some embodiments, the illumination beam path252L is configured to propagate light in the first wavelength band, and the monitor beam path254L is configured to propagate light in the second wavelength band. As such, the projected combined light beam is split into the illumination beam path252L and the monitor beam path254L. In some embodiments, a filter is disposed before the illumination beam path252L to filter out the light beam in the second wavelength, such that only the light beam in the first wavelength band is propagated over the illumination beam path252L. Alternatively, or in addition, a filter is disposed before the monitor beam path254L to filter out the light beam in the first wavelength, such that only the monitor light beam in the second wavelength band is propagated over the monitor beam path254L.

In some embodiments, the monitor image includes a predetermined set of dots or lines.FIGS.3A and3Billustrate examples of monitor images that are captured by the monitor camera270. As illustrated inFIG.3A, a monitor image300A includes a grid of four dots302A,304A,306A,308A, which can be achieved by filtering out light beams in the first wavelength band, and/or using a monitor beam path254L configured to propagate light beams in the second wavelength band. In some embodiments, the monitor image300A is compared with a boresight310A of the monitor camera270to determine whether the monitor image330A is properly oriented or positioned.

As illustrated inFIG.3B, a monitor image300B includes a grid of four dots302B,304B,306B,308B overlaid with a display image320B, which may be caused by without filtering out the light beam in the first wavelength band, or using a beam path configured to propagate light beams in both the first wavelength band and the second wavelength band. The monitor image300B can also be compared with a boresight310B of the monitor camera270to determine whether the monitor image300B is properly oriented or positioned. In some embodiments, a power of the monitor light beam222L,222R is greater (or significantly greater) than a power of the illumination light beam212L,212R, such that the signal-to-noise ratio is further improved to allow the identification of the monitor image.

Referring back toFIGS.2A-2B, the head-mounted device200further includes a second illumination light source210R, a second monitor light source220R, a second beam combiner230R, a second illumination beam path252R, and a second monitor beam path254R, each of which is similar to the respective first illumination light source210L, a first monitor light source220L, a first beam combiner230L, a first illumination beam path252L, and a first monitor beam path254L. The first set of components210L,220L,230L, and240L, and the second set of components210R,220R,230R, and240R are symmetrically disposed on the left and right sides of the head-mounted device200. The first set of components210L,220L,230L, and240L are configured to project a first image at a first eye of the user, and the second set of components210R,220R,230R, and240R are configured to project a second image at a second eye of the user.

In some embodiments, the monitor camera270is configured to receive a portion of a first projected combined light beam from the first monitor beam path254L and/or a portion of a second projected combined light beam from the second monitor beam path254R. The monitor camera270is configured to capture a first monitor image based on the first light beam received from the first monitor beam path254L, and/or capture a second monitor image based on the second light beam received from a second monitor beam path254R. In some embodiments, the first monitor image or the second monitor image is individually analyzed to determine whether each of the first monitor image or the second monitor is properly oriented or positioned. In some embodiments, the monitor camera270is configured to capture the first monitor image and the second monitor image overlaid with each other. In some embodiments, the first monitor image is compared with the second monitor image to determine whether relative boresight to two eyes is aligned.

In some embodiments, a separate monitor camera is implemented for each projection system. For example, in some embodiments, a head-mounted device includes a first monitor camera configured to capture a first monitor image from a first projector, and a second monitor camera configured to capture a second monitor image from a second projector. The captured first monitor image and second monitor image can then be compared with a respective boresight of the first camera and the second camera, or compared with each other to determine a relative boresight to each other.

FIGS.4A and4Billustrate an example of images400A,400B captured by the monitor camera270in which a first monitor image (received from the first monitor beam path254L) and a second monitor image (received from the second monitor beam path254R) are overlaid with each other. As illustrated inFIG.4A, image400A captured by the monitor camera270includes a first monitor image having a first grid of four dots402A,404A,406A,408A; and a second monitor image having a second grid of four dots412A,414A,416A,418A. The first monitor image and the second monitor image are overlaid with each other. As shown inFIG.4A, the first grid of four dots402A,404A,406A,408A are not aligned with the second grid of four dots412A,414A,416A,418A, indicating that the relative boresight to two eyes is not aligned. In some embodiments, in response to determining that the relative boresight to two eyes is not aligned, the head-mounted device200is configured to adjust an orientation or a position of at least one of the first projector or a second projector (or the first illumination image and/or the second illumination image) to cause the relative boresight to be aligned.

FIG.4Billustrates an example of an image400B captured by the monitor camera270includes a first display image410B, a first monitor image having a first grid of four dots402B,404B,406B,408B, second display image420B, and a second monitor image having a second grid of four dots412B,414B,416B,418B. The first display image410B, the first monitor image, the second display image420B, and the second monitor image are overlaid with each other. In some embodiments, the overlaid image400B is further processed to extract the first monitor image and/or the second monitor image. The extracted first monitor image and/or the second monitor image are then analyzed to determine whether monitor image and/or the second monitor image is properly oriented or positioned.

FIG.5illustrates an example architecture of a head-mounted device500(which corresponds to the head-mounted device200ofFIG.2A or2B). The head-mounted device500includes a left projection system510and a right projection system520, each of which corresponds to a projection system100A or100B ofFIG.1A or2B. The left projection system510includes an illumination light source512, a monitor light source514, and a projector516. The right projection system520includes an illumination light source522, a monitor light source524, and a projector526.

The head-mounted device500further includes one or more monitor cameras530configured to receive and capture a monitor image generated by the monitor light source514and/or524. In some embodiments, a single monitor camera is used to capture monitor images from both the left projection system510and the right projection system520. In some embodiments, a left camera is configured to capture a monitor image from the left projection system510, and a right camera is configured to capture a monitor image from the right projection system520.

The head-mounted device500is also a computer system including one or more processors540, one or more memories550, and one or more hardware storage devices560. Firmware and/or other applications570are stored in the one or more hardware storage devices560and can be loaded into the one or more memories550. The applications570includes at least a monitoring application572configured to cause the monitor light source514,524to emit a monitor light beam, cause the monitor camera(s)530to capture one or more monitor images, and analyze the captured one or more monitor images to determine whether the monitor images are properly positioned. In some embodiments, the monitoring application572is configured to compare the captured monitor image with a boresight of the camera to determine whether the projector516,526is aligned with the respective boresight. In some embodiments, the monitoring application572is configured to compare a monitor image generated by the left projection system510and a monitor image generated by the right projection system520to determine whether the relative boresight between the left projector516and the right projector526is aligned to each other. In response to determining that the monitor image is not properly oriented or positioned, the monitoring application572causes a position and/or an orientation of the projector516or526to be adjusted; alternatively or in addition, the monitoring application572causes a position and/or an orientation of the illumination image to be adjusted. For example, in some embodiments, the image data associated with the illumination image is transformed to cause the illumination image to be rotated for a particular angle based on the orientation of the monitor image. As another example, in some embodiments, the image data associated with the illumination image may be transformed to cause the illumination image to be moved, enlarged, and/or reduced.

In some embodiments, the monitoring application572is configured to check on the projectors516,526at a predetermined frequency, such as (but not limited to) every month, every hour, every 30 minutes, every 10 minutes, etc. In some embodiments, the monitoring application572is configured to check on the projectors516,526each time the head-mounted device500is powered on and/or turned off. In some embodiments, the monitoring application572is configured to check on the projectors516,526based on a user setup or input. For example, in some embodiments, a hardware or software button is implemented on the head-mounted device500, and in response to pressing the button, the monitoring application572is configured to check on the projectors516,526. As another example, in some embodiments, the user can input the times and/or the frequency that the projectors516,526are checked on and adjusted.

FIG.6illustrates an example of a glass-formed head-mounted device600, which corresponds to the head-mounted device200ofFIG.2A or2B, or500ofFIG.5. As illustrated inFIG.6, the glass-formed head-mounted device600includes a left projection system620(which corresponds to the left projection system510ofFIG.5), a right projection system620(which corresponds to the right projection system520ofFIG.5), and a monitor camera630(which corresponds to the camera530ofFIG.5). In embodiments, each of the projection systems610,620is embedded in temples of glass-formed head-mounted device600, and the monitor camera630is embedded in a nose bridge of the head-mounted device600.

The following discussion now refers to a number of methods and method acts that may be performed. Although the method acts may be discussed in a certain order or illustrated in a flow chart as occurring in a particular order, no particular ordering is required unless specifically stated, or required because an act is dependent on another act being completed prior to the act being performed.

FIG.7illustrates a flowchart of an example method700implemented at a projection system (e.g., a projection system100A,100B) for monitoring or adjusting positions or orientations of a projector installed thereon. The method700includes emitting an illumination light beam from an illumination light source (act710), and emitting a monitor light beam from a monitor light source (act720). The method also includes projecting, by the projector, the illumination light beam, and the monitor light beam into a projected combined light beam (act740).

In some embodiments, the illumination light beam and the monitor light beam are directed at different directions that intersect each other, and the method700further includes combining the illumination light beam and the monitor light beam into a combined light beam directed at the projector (act730), and the projector is configured to project the combined light beam into the projected combined light beam.

The method700further includes propagating a first portion of the projected light beam over a first beam path toward an eye of a user (act750), and propagating a second portion of the projected light beam over a second beam path toward a monitor camera (act760). The first portion of the projected combined light beam contains at least a portion of the illumination light beam projected by the projector, causing the eye of the user to see a display image corresponding to the illumination light beam. The second portion of the projected combined light beam contains at least a portion of the monitor light beam projected by the projector.

The method700further includes capturing a monitor image, by the monitor camera, corresponding to the monitor light beam (act770), analyzing the monitor image to determine an orientation or a position of the monitor image (act780), and determining whether the monitor image is properly oriented or positioned (act790) In response to determining that the monitor image is not oriented or positioned properly, an orientation or a position of the projector is adjusted (act792); alternatively or in addition, an orientation or a position of the illumination image is adjusted (act792). In some embodiments, in response to determining that the monitor image is oriented and/or positioned properly, the projection system repeats the acts710-792again, which may be based on a user input, at a predetermined time, and/or at a predetermined frequency.

FIG.8illustrates a flowchart of an example method800implemented at a head-mounted device (e.g., an VR/AR device) for monitoring and/or adjusting relative boresight to the left and right eyes projectors. The method800includes capturing a first monitor image from a first projector (act810) and capturing a second monitor image from a second projector (act820). Each of the first projector and second projector configured to project an image for a first eye or a right eye of a user. Each of act810or820includes acts710through770of method700.

Capturing the first monitor image from the first projector (act810) is performed by a first projection system, including a first illumination light source, a first monitor light source, a first beam path, a second beam path, and a camera. Act810includes emitting a first illumination light beam from a first illumination light source (act710), emitting a first monitor light beam from a first monitor light source (act720), and projecting (by a first projector) the illumination light beam and the monitor light beam into a first projected light beam (act740). In some embodiments, the first illumination light source and the first monitor light source are configured to emit light in different directions that interest each other, and the act810further includes combining the first illumination light beam and the first monitor light beam into a first combined light beam directed at the first projector (act730). Act810also includes propagating a first portion of the first projected combined light beam over a first beam path toward a first eye of a user (act750), and propagating a second portion of the projected light beam over a second beam path toward a camera (act760), and capturing a monitor image by the camera (act770).

Similarly, capturing the second monitor image from the second projector (act820) (including acts710-770) is performed by a second projection system, including a second illumination light source, a second monitor light source, a third beam path, a fourth beam path, and a camera.

In some embodiments, the first projection system and the second projection system share a same camera. In some embodiments, the first projection system includes a first camera, and the second projection system includes a second camera. In some embodiments, the first monitor image and the second monitor image are captured as overlaid with each other. In some embodiments, the first monitor image and the second monitor image are captured separately.

The first monitor image and the second monitor image are then compared with each other to determine whether relative boresight to the first projector and the second projector is aligned with each other (act830). In response to determining that the relative boresight is not aligned, an orientation or a position of at least one of the first or second projector is adjusted (act840). In some embodiments, in response to determining that the relative boresight is aligned, the projection system repeats the acts810-830again, which may be based on a user input, at a predetermined time, and/or at a predetermined frequency.

In some embodiments, the illumination beam and monitor beam are not combined or in parallel. The illumination beam and monitor beam intersect inside the projector, and come out at two separate locations. Such embodiments make it easy to send illumination beam to the eye of the user and monitor beam to the monitor camera.

FIG.9illustrates a flowchart of an example method900for implementing the above-described embodiments. The method900includes emitting an illumination light beam from an illumination light source (act910) and emitting a monitor light beam from a monitor light source (act920). The method900further includes projecting the illumination light beam into a projected illumination light beam (act930) and projecting the monitor light beam into a projected monitor light beam (act940). In embodiments, the illumination beam and monitor beam intersect inside the projector, and come out at two separate locations as projected illumination beam and projected monitor beam.

The projected illumination beam is propagated over a first beam path toward an eye of a user (act950), and the projected monitor light beam is propagated over a second beam path toward a camera (act960). A monitor image is captured by the camera (act970) and analyzed to determine an orientation or a position thereof (act980). It is then determined whether the monitor image is oriented and/or positioned properly (act990). Similar to method700ofFIG.7, in response to determining that the monitor image is not oriented and/or positioned properly, an orientation and/or a position of the projector is adjusted (act992). Alternatively, or in addition, an orientation and/or a position of the illumination image is adjusted (act992). Also similar to method700ofFIG.7, acts910-970can be used in act810or820ofFIG.8to align a relative boresight of a head-mounted device.

Finally, because the principles described herein may be performed in the context of a computer system, some introductory discussion of a computer system will be described with respect toFIG.10.

Computer systems are now increasingly taking a wide variety of forms. Computer systems may, for example, be hand-held devices, appliances, laptop computers, desktop computers, mainframes, distributed computer systems, data centers, or even devices that have not conventionally been considered a computer system, such as wearables (e.g., glasses). In this description and in the claims, the term “computer system” is defined broadly as including any device or system (or a combination thereof) that includes at least one physical and tangible processor, and a physical and tangible memory capable of having thereon computer-executable instructions that may be executed by a processor. For example, a projector or a head-mounted device200,500,600is a computer system. The memory may take any form and may depend on the nature and form of the computer system. A computer system may be distributed over a network environment and may include multiple constituent computer systems.

As illustrated inFIG.10, in its most basic configuration, a computer system1000typically includes at least one hardware processing unit1002and memory1004. The processing unit1002may include a general-purpose processor and may also include a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or any other specialized circuit. The memory1004may be physical system memory, which may be volatile, non-volatile, or some combination of the two. The term “memory” may also be used herein to refer to non-volatile mass storage such as physical storage media. If the computer system is distributed, the processing, memory and/or storage capability may be distributed as well.

The computer system1000also has thereon multiple structures often referred to as an “executable component”. For instance, memory1004of the computer system1000is illustrated as including executable component1006. The term “executable component” is the name for a structure that is well understood to one of ordinary skill in the art in the field of computing as being a structure that can be software, hardware, or a combination thereof. For instance, when implemented in software, one of ordinary skill in the art would understand that the structure of an executable component may include software objects, routines, methods, and so forth, that may be executed on the computer system, whether such an executable component exists in the heap of a computer system, or whether the executable component exists on computer-readable storage media.

In such a case, one of ordinary skill in the art will recognize that the structure of the executable component exists on a computer-readable medium such that, when interpreted by one or more processors of a computer system (e.g., by a processor thread), the computer system is caused to perform a function. Such a structure may be computer-readable directly by the processors (as is the case if the executable component were binary). Alternatively, the structure may be structured to be interpretable and/or compiled (whether in a single stage or in multiple stages) so as to generate such binary that is directly interpretable by the processors. Such an understanding of example structures of an executable component is well within the understanding of one of ordinary skill in the art of computing when using the term “executable component.”

The term “executable component” is also well understood by one of ordinary skill as including structures, such as hardcoded or hard-wired logic gates, which are implemented exclusively or near-exclusively in hardware, such as within a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or any other specialized circuit. Accordingly, the term “executable component” is a term for a structure that is well understood by those of ordinary skill in the art of computing, whether implemented in software, hardware, or a combination. In this description, the terms “component”, “agent”, “manager”, “service”, “engine”, “module”, “virtual machine” or the like may also be used. As used in this description and in the case, these terms (whether expressed with or without a modifying clause) are also intended to be synonymous with the term “executable component”, and thus also have a structure that is well understood by those of ordinary skill in the art of computing.

In the description above, embodiments are described with reference to acts that are performed by one or more computer systems. If such acts are implemented in software, one or more processors (of the associated computer system that performs the act) direct the operation of the computer system in response to having executed computer-executable instructions that constitute an executable component. For example, such computer-executable instructions may be embodied in one or more computer-readable media that form a computer program product. An example of such an operation involves the manipulation of data. If such acts are implemented exclusively or near-exclusively in hardware, such as within an FPGA or an ASIC, the computer-executable instructions may be hardcoded or hard-wired logic gates. The computer-executable instructions (and the manipulated data) may be stored in the memory1004of the computer system1000. Computer system1000may also contain communication channels1008that allow the computer system1000to communicate with other computer systems over, for example, network1010.

While not all computer systems require a user interface, in some embodiments, the computer system1000includes a user interface system1012for use in interfacing with a user. The user interface system1012may include output mechanisms1012A as well as input mechanisms1012B. The principles described herein are not limited to the precise output mechanisms1012A or input mechanisms1012B; as such will depend on the nature of the device. However, output mechanisms1012A might include, for instance, speakers, displays, tactile output, holograms, and so forth. Examples of input mechanisms1012B might include, for instance, microphones, touchscreens, holograms, cameras, keyboards, mouse or other pointer input, sensors of any type, and so forth.

Embodiments described herein may comprise or utilize a special purpose or general-purpose computer system, including computer hardware, such as, for example, one or more processors and system memory, as discussed in greater detail below. Embodiments described herein also include physical and other computer-readable media for carrying or storing computer-executable instructions and/or data structures. Such computer-readable media can be any available media that can be accessed by a general-purpose or special-purpose computer system. Computer-readable media that store computer-executable instructions are physical storage media. Computer-readable media that carry computer-executable instructions are transmission media. Thus, by way of example, and not limitation, embodiments of the invention can comprise at least two distinctly different kinds of computer-readable media: storage media and transmission media.

Computer-readable storage media includes RAM, ROM, EEPROM, CD-ROM, or other optical disk storage, magnetic disk storage, or other magnetic storage devices, or any other physical and tangible storage medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general-purpose or special-purpose computer system.

A “network” is defined as one or more data links that enable the transport of electronic data between computer systems and/or modules and/or other electronic devices. When information is transferred or provided over a network or another communications connection (either hard-wired, wireless, or a combination of hard-wired or wireless) to a computer system, the computer system properly views the connection as a transmission medium. Transmissions media can include a network and/or data links that can be used to carry desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general-purpose or special-purpose computer system. Combinations of the above should also be included within the scope of computer-readable media.

Further, upon reaching various computer system components, program code means in the form of computer-executable instructions or data structures can be transferred automatically from transmission media to storage media (or vice versa). For example, computer-executable instructions or data structures received over a network or data link can be buffered in RAM within a network interface module (e.g., a “NIC”), and then eventually transferred to computer system RAM and/or to less volatile storage media at a computer system. Thus, it should be understood that storage media can be included in computer system components that also (or even primarily) utilize transmission media.

Computer-executable instructions comprise, for example, instructions and data which, when executed at a processor, cause a general-purpose computer system, special-purpose computer system, or special purpose processing device to perform a certain function or group of functions. Alternatively, or in addition, the computer-executable instructions may configure the computer system to perform a certain function or group of functions. The computer-executable instructions may be, for example, binaries or even instructions that undergo some translation (such as compilation) before direct execution by the processors, such as intermediate format instructions such as assembly language, or even source code.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the described features or acts described above. Rather, the described features and acts are disclosed as example forms of implementing the claims.

Those skilled in the art will appreciate that the invention may be practiced in network computing environments with many types of computer system configurations, including, personal computers, desktop computers, laptop computers, message processors, hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAs, pagers, routers, switches, data centers, wearables (such as glasses) and the like. The invention may also be practiced in distributed system environments where local and remote computer systems, which are linked (either by hard-wired data links, wireless data links, or by a combination of hard-wired and wireless data links) through a network, both perform tasks. In a distributed system environment, program modules may be located in both local and remote memory storage devices.

Those skilled in the art will also appreciate that the invention may be practiced in a cloud computing environment. Cloud computing environments may be distributed, although this is not required. When distributed, cloud computing environments may be distributed internationally within an organization and/or have components possessed across multiple organizations. In this description and the following claims, “cloud computing” is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services). The definition of “cloud computing” is not limited to any of the other numerous advantages that can be obtained from such a model when properly deployed.

The computer systems of the remaining figures (such as the head-mounted device500ofFIG.5) include various components or functional blocks that may implement the various embodiments disclosed herein, as have been explained. The various components or functional blocks may be implemented on a local computer system or may be implemented on a distributed computer system that includes elements resident in the cloud or that implement aspect of cloud computing. The various components or functional blocks may be implemented as software, hardware, or a combination of software and hardware. The computer systems of the remaining figures may include more or less than the components illustrated in the figures, and some of the components may be combined as circumstances warrant. Although not necessarily illustrated, the various components of the computer systems may access and/or utilize a processor and memory, such as processing unit1002and memory1004, as needed to perform their various functions.

For the processes and methods disclosed herein, the operations performed in the processes and methods may be implemented in differing order. Furthermore, the outlined operations are only provided as examples, and some of the operations may be optional, combined into fewer steps and operations, supplemented with further operations, or expanded into additional operations without detracting from the essence of the disclosed embodiments.

The present invention may be embodied in other specific forms without departing from its spirit or characteristics. The described embodiments are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is, therefore, indicated by the appended claims rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be embraced within their scope.