Patent Description:
The wearable device may include augmented and/or virtual reality glasses. A camera may be coupled to the glasses. In order to view the actual location of a real-world object on the augmented and/or virtual reality glasses, the camera needs to be calibrated. Calibrating a camera may include determining intrinsic and/or extrinsic parameters of the camera. The intrinsic parameters represent a projective transformation from the <NUM>-D camera's coordinates into the <NUM>-D image coordinates. The intrinsic parameters may include the focal length (fx, fy), the principal point and the distortion coefficient(s). On the other hand, the extrinsic parameters represent a transformation from the world coordinate system to the coordinate system of the camera. The extrinsic parameters include a rotation matrix and a translation vector. The extrinsic parameters may help to determine the position of the camera center and the camera's heading in world coordinates.

Accordingly, the camera calibration may estimate the parameters of a lens and image sensor of a camera. The determined parameters may be used to correct for lens distortion, measure the size of an object in world units, or determine the location of the camera in a scene in a <NUM>-D scene reconstruction.

As it turns out, the human visual perception system is very complex, and producing a VR, AR, or MR technology that facilitates a comfortable, natural-feeling, rich presentation of virtual image elements amongst other virtual or real-world imagery elements is challenging. Systems and methods disclosed herein address various challenges related to VR, AR, and MR technology.

A publication by <NPL> discloses a system enabling vergence adjustment without the need for new camera calibrations.

A publication by <NPL> discloses methods to measure and correct vertical disparity in graphics combining real and graphical depictions.

Embodiments relate generally to image display systems and methods for display system calibration. Embodiments provide a camera design (e.g., an eyeball camera) that mimics a human eye in geometry, optical performance and/or motion. The eyeball camera adopts the same cornea and pupil geometry from the human eye and has the iris and pupil configured with multiple texture, color, or diameter options. The resolution of the eyeball camera is designed to match the acuity of typical <NUM>/<NUM> human vision and focus is adjusted in some implementations from <NUM> to <NUM> diopters. A pair of eyeball cameras can be mounted independently on two hexapods to simulate the human eye gaze and vergence. The perceived virtual and real world can thus be calibrated and evaluated based on eye conditions like pupil location and gaze using the eyeball cameras. The eyeball camera serves as a bridge to combine data from spatial computing like eye tracking, 3D geometry of the digital world, display color accuracy/uniformity, and display optical quality (sharpness, contrast, etc.).

Numerous benefits are achieved by way of the present disclosure over conventional techniques. For example, embodiments provide methods and systems that calibrate a real-world image capture device and/or an AR/VR display system using one or more cameras (e.g., eyeball cameras) that mimic the human eye. The properties of the eyeball camera such as one or more of a cornea position, a cornea geometry, an eyeball position, a pupil size, a pupil position, a gaze distance, or a gaze orientation are known and can be controlled. Thus, the calibration can be performed using a set of known and controllable parameters. For example, it is possible to fix the gaze on infinity and calibrate the device using the data. Thus, embodiments provide for a more accurate calibration of an AR/VR display system, therefore resulting in a more accurate and seamless AR/VR experience.

Neither this summary nor the following detailed description purports to define or limit the scope of the inventive subject matter.

Embodiments are directed to image display systems and methods for display system calibration. Spatial computing enables overlay of digital world content on real world content in a spatially interacting way through combining digital light-fields, sensing, and computing. The digital content presented by spatial computing techniques preferably works in tandem with real-world surroundings, and more importantly the human eye-brain system, which is the ultimate judge for the system's success. As a result, to develop such a spatial computing system, it would be essential to have a proxy for the human eye-brain system to calibrate and verify the performance of the spatial computing system.

Embodiments provide a camera design that mimics human physiology (e.g., the human eye) at least in one or more of the following aspects: geometry, optical performance, and/or motion. Specifically, embodiments provide a camera (e.g., an eyeball camera) that not only adopts the same cornea and pupil geometry from the human eye, but also has an iris and pupil that can be configured with multiple texture, color, or diameter options. Furthermore, the resolution of the eyeball camera is designed to match the acuity of typical <NUM>/<NUM> human vision and focus can be adjusted by a piezo motor or other suitable mechanical system, for example, from <NUM> to <NUM> diopters.

According to various embodiments, a pair of eyeball cameras are mounted independently on two hexapods to simulate the human eye gaze and vergence. With the help of eyeball cameras, both perceived virtual and real world content can be calibrated and evaluated in deterministic and quantifiable eye conditions, for example, pupil location and gaze. According to various embodiments, the eyeball camera serves as a bridge to combine data from spatial computing like eye tracking, 3D geometry of the digital world, display color accuracy/uniformity, and display optical quality (sharpness, contrast, etc.) for a holistic view, which helps to effectively blend the virtual and real worlds together seamlessly.

<FIG> illustrates an exemplary system <NUM> for calibrating a virtual image display system <NUM>. The system <NUM> may include the virtual image display system <NUM>, an image capture device <NUM> and a data processing module <NUM>. The virtual image display system <NUM> may include a display device <NUM> configured to display a virtual image, a real world image capture device <NUM> that is configured to capture and display images on the display device <NUM>, and various mechanical and electronic modules and systems to support the functioning of the virtual image display system <NUM>. The display device <NUM> may be coupled to a frame <NUM>, which is wearable by a display system user, wearer, or viewer and that is configured to position the display device <NUM> in front of the eyes of the wearer during use. The display device <NUM> may be a light field display. In some embodiments, a speaker may be coupled to the frame <NUM> and positioned adjacent the ear canal of the user (in some embodiments, another speaker, not shown, is positioned adjacent the other ear canal of the user to provide for stereo/shapeable sound control). The real world image capture device <NUM> may be coupled to the display device <NUM> via a wired lead or wireless connectivity. In some embodiments, the real world image capture device <NUM> may be provided on the frame <NUM>. In other embodiments, the real world image capture device <NUM> may be provided separately from the frame <NUM>, and configured to be worn (e.g., on the head as a head camera, on the body as a belt, a wristlet, or a watch) by the user of the virtual image display system <NUM>.

According to various embodiments, the image capture device <NUM> may be positioned in front of the display device <NUM>. The system may also include a data processing module <NUM> that is operatively coupled, such as by a wired lead or wireless connectivity, to the virtual image display system <NUM> and the image capture device <NUM>. The data processing module <NUM> may include various input/output devices and may receive data from external modules.

The data processing module <NUM> may include one or more processors configured to analyze and process data and/or image information such as an image or video information captured by the real world image capture device <NUM>. The image or video data may be stored locally in the data processing module <NUM> and/or remotely in a remote data repository. In some embodiments, the remote data repository may comprise a digital data storage facility, which may be available through the internet or other networking configuration in a "cloud" resource configuration. In some embodiments, all data is stored and all computations are performed in the data processing module <NUM>, allowing fully autonomous use from a remote module, whereas in other embodiments, data storage and/or computations are distributed. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

The display device <NUM> may include a first display <NUM> (e.g., a left display element) and a second display <NUM> (e.g., a right display element). Similarly, the image capture device <NUM> may include a first image capture element <NUM> (e.g., a left image capture element) and a second image capture element <NUM> (e.g., a right image capture element). The image capture device <NUM> may be positioned in front of the display device <NUM> such that the first image capture element <NUM> is positioned in front of the first display <NUM> and the second image capture element <NUM> is positioned in front of the second display <NUM>.

According to various embodiments, the image capture device <NUM> may simulate a human eye. That is, the image capture device <NUM> may include properties similar to a human eye. For example, each of the first image capture element <NUM> and the second image capture element <NUM> may include a cornea position, a cornea geometry, an eyeball position, a pupil size, a pupil position, a gaze distance, a gaze orientation, and an iris color. These properties may be configurable and controlled for each one of the first image capture element <NUM> and the second image capture element <NUM> independently from each other. For example, the first image capture element <NUM> and the second image capture element <NUM> may include an eyeball camera.

<FIG> illustrate the first image capture element <NUM> according to some embodiments of the present disclosure. The second image capture element <NUM> is similar (or identical) to the first image capture element <NUM>, and the description of the first image capture element <NUM> is applicable to the second image capture element <NUM>. The first image capture element <NUM> and/or the second image capture element <NUM> may simulate a human eye, and include an artificial cornea <NUM>, an artificial pupil <NUM>, and an artificial iris <NUM> mounted at an end of a lens mount <NUM> that is coupled to a camera body <NUM>. The artificial cornea <NUM>, the artificial pupil <NUM>, and the artificial iris <NUM> may be referred as an artificial eyeball <NUM>. The camera body <NUM> may include a light proof box housing camera elements such as a camera shutter and an image sensor.

Various properties of the artificial cornea <NUM>, the artificial pupil <NUM>, and the artificial iris <NUM> may be configured according to various embodiments. For example, the position and/or the geometry of the artificial cornea <NUM>, the position and size of the artificial pupil <NUM>, and the position and/or the color of the artificial iris <NUM> may be configured, customized, or otherwise adjusted according to various embodiments. Similarly, the position of the artificial eyeball <NUM> at an end of the lens mount <NUM> may also be configured. Embodiments allow for determining, and adjusting, the vergence (including convergence - the rotating of the eyes toward each other - that occurs when looking at an object closer by, and divergence - the rotating of the eyes away from each other - that occurs when looking at an object far away), the gaze distance, and/or the gaze orientation of the first image capture element <NUM> and/or the second image capture element <NUM>. For example, embodiments allow for fixing the gaze distance at infinity for a desired (predetermined) amount of time. Since a human eye is not capable of holding a steady gaze at infinity, the first image capture element <NUM> and/or the second image capture element <NUM> may mimic, and build upon the capabilities of the human eye. Thus, embodiments allow for a more accurate calibration of the virtual image display system <NUM> (namely the real world image capture device <NUM> of the virtual image display system <NUM>).

According to various embodiments, the properties of the first image capture element <NUM> can be configured and adjusted independently from the second image capture element <NUM>. Similarly, the first image capture element <NUM> can be controlled to move independently from the second image capture element <NUM>. This may be achieved by placing each of the first image capture element <NUM> and the second image capture element <NUM> on a separate, individual hexapod, as illustrated in <FIG>.

<FIG> is a simplified schematic diagram illustrating a camera rig <NUM> according to some embodiments of the present disclosure. The camera rig <NUM> may include a first controllable mount <NUM> and a second controllable mount <NUM>. The controllable mounts <NUM>, <NUM> may be controlled to move independently from each other, or simultaneously with each other. The controllable mounts <NUM>, <NUM> may be controlled to mimic human eye muscles as well as a human head.

As illustrated in <FIG>, the first image capture element <NUM> may be mounted on the first controllable mount <NUM>, and the second image capture element <NUM> may be mounted on the second controllable mount <NUM>. The camera rig <NUM> and the image capture elements <NUM>, <NUM> may work together to mimic (e.g., simulate) a pair of human eyes moving in their sockets.

According to various embodiments, the controllable mounts <NUM>, <NUM> may include one or more actuators <NUM> (e.g., linear actuators, prismatic joints, or the like) that are coupled to a bottom platform <NUM> and/or a top platform <NUM>. In some embodiments, the controllable mounts <NUM>, <NUM> may be in the form of a parallel manipulator including six linear actuators.

The controllable mounts <NUM>, <NUM> may be controlled via a computing device <NUM> that is configured to communicate with the controllable mounts <NUM>, <NUM> via a wired or wireless connection. In some embodiments, the computing device <NUM> may include the data processing module <NUM>, or may be coupled to the data processing module <NUM> to work in tandem with the data processing module <NUM>. According to various embodiments, the computing device <NUM> may send commands (e.g., in the form of signals) to the controllable mounts <NUM>, <NUM> and receive feedback (e.g., in the form of signals) from the controllable mounts <NUM>, <NUM>.

According to various embodiments, the camera rig <NUM> may be used during production of a virtual image display system to fine tune the calibration of the virtual image display system. Yet in some embodiments, the camera rig <NUM> may be sized and dimensioned to be provided at a retail location where the users may bring their virtual image display systems <NUM> for calibration or to fine tune the virtual image display system according to the user's particular eye properties (e.g., geometry, color, sight).

Embodiments allow for calibrating a virtual image display system (e.g., a real world image capture device of an AR/VR system) using an image capture device (including one or more image capture elements). In some embodiments, the image capture device may be controlled or configured to mimic a human eye.

<FIG> shows an exemplary block diagram illustrating a method for calibrating a virtual image display system according to some embodiments of the present disclosure. An image <NUM> of a scene, an environment or an object in real world <NUM> may be captured using a real-world image capture device (e.g., world camera) <NUM>. The captured image <NUM> may be sent to a rendering engine <NUM> that then processes the received image and transmits the processed image to a projector <NUM> that will then display on a display device (e.g., the eyepiece of the virtual image display system) <NUM>. The displayed image may correspond to a group of pixels <NUM> illuminated on the display device <NUM>. However, the displayed image may not exactly match (e.g., align) with the real world <NUM> that the image is supposed to represent. For example, a contour of an object in the image may not align with the contour of the object in real world. To correct this mismatch, the real-world image capture device <NUM> may need to be calibrated.

For example, the image captured using the real-world image capture device <NUM> may be processed (e.g., refined) using data from an eye tracking camera <NUM> and one or more eye tracking light emitters (e.g., light emitting diodes (LEDs)) <NUM>. The data from the eye tracking camera <NUM> and the one or more eye tracking light emitters <NUM> may include additional information about where the user is looking to render the image at a specific depth or range. For example, vergence of the eyes may be determined using the eye tracking camera <NUM> and one or more eye tracking light emitters <NUM>, and calibration/adjustment for the virtual image display system may be determined using a vergence/accommodation model.

According to various embodiments, the eye tracking camera <NUM> detects a cornea position of the user and determines the eye aperture based on the cornea position, and calculates the gaze of the user. That is, the eye tracking camera <NUM> estimates the eye position of the user. Therefore, it would be desirable to know the details of the eye geometry, position and gaze of the user to more accurately calibrate the virtual image display system.

Embodiments may use an image capture device <NUM> simulating a human eye (or a pair of human eyes) for calibrating the virtual image display system. The image capture device <NUM> may be used in connection with the eye tracking camera <NUM>. According to various embodiments, the image capture device <NUM> may have the physical properties of a human eye. For example, as described in connection with <FIG>, the image capture device <NUM> may have a geometry similar to the human eye geometry including an artificial cornea, an artificial pupil, and an artificial iris. Thus, the eye tracking camera <NUM> and the one or more eye tracking light emitters <NUM>, when used in connection with the image capture device <NUM>, provide more accurate data for calibrating the virtual image display system.

Since the properties of the image capture device <NUM> are known, the eye tracking camera <NUM> and the one or more eye tracking light emitters <NUM> use the data for the properties (e.g., cornea position, a cornea geometry, an eyeball position, a pupil size, a pupil position, a gaze distance, and a gaze orientation) instead of estimating or determining these properties. A correction for the virtual image display system may be determined based on one or more of these properties of the image capture device <NUM> in order to more accurately align the image <NUM> received from the real-world image capture device <NUM> with an image <NUM> of the same scene, environment or object in real world <NUM> captured using the image capture device <NUM>.

The rendering engine <NUM> may receive the image <NUM> from the image capture device <NUM> and compare the image <NUM> to the image <NUM> received from the real-world image capture device <NUM>. If the image <NUM> and the image <NUM> are determined not to overlap to a predetermined degree or target, the rendering engine <NUM> may determine one or more corrections to be applied to the real-world image capture device <NUM> to more accurately capture an image of the scene, environment, or object in real world <NUM>. The virtual image display system (and more particularly, the real-world image capture device <NUM> and/or the display device <NUM>) may be calibrated until the image <NUM> captured by the real-world image capture device <NUM> aligns with the image <NUM> captured with the image capture device <NUM> to within a predetermined threshold.

<FIG> shows an exemplary block diagram illustrating various calibrations that may be performed in connection with a virtual image display system. As described in connection with <FIG> as well, the virtual image display system <NUM> may have a sensing component <NUM> (e.g., the real world image capture device <NUM>) and a display component <NUM> (e.g., the display device <NUM>). The sensing component <NUM> may be calibrated using the eye tracking camera calibration <NUM> and eye tracking LED calibration <NUM> to verify and improve the eye tracking <NUM>. The display component <NUM> can be calibrated using geometry calibration <NUM> to verify and improve the geometry <NUM> of the displayed objects, and color calibration <NUM> to verify and improve the color <NUM> of the displayed objects. Overall, the display quality <NUM> of the virtual image display system may be verified and improved.

The image capture device simulating the human eye described in connection with the various embodiments may be used to combine and concurrently perform all above-described calibrations to improve eye tracking, geometry, color, and display quality for the virtual image display system.

<FIG> shows an exemplary block diagram illustrating a detailed view of the eye tracking verification. The eye tracking camera <NUM> and the eye tracking LEDs <NUM> can estimate properties <NUM> of the image capture device <NUM> such as a cornea position, a cornea geometry, an eyeball position, a pupil size, a pupil position, a gaze distance, and a gaze orientation. The estimated properties <NUM> may be provided to a computing device <NUM> (e.g., data processing module <NUM>) along with the actual values <NUM> for these properties. The computing device may compare the estimated properties <NUM> with the actual values <NUM> for these properties of the image capture device <NUM> and determine one or more corrections <NUM> for the virtual image display system based on gaze direction, depth pane switch, rendering center, convergence plane, etc..

<FIG> is a simplified flowchart <NUM> illustrating a method for determining and applying a correction to the virtual image display system according to some embodiments of the present disclosure. The virtual image display system may include at least a display device and a world camera. According to various embodiments, an image capture element simulating a human eye (or a pair of human eyes) may be coupled to the virtual image display system.

At block <NUM>, a computing device including a processor receives a first image and a second image from a virtual image display system. The first image may be captured using a world camera of the virtual image display system and the second image may be captured using an image capture device (e.g., an eye ball camera). The image capture device has one or more properties similar to a human eye including, but not limited to, a cornea position, a cornea geometry, an eyeball position, a pupil size, a pupil position, a gaze distance, and/or a gaze orientation. These properties of the image capture device may have predetermined values, and may be configurable.

At block <NUM>, the computing device may determine visual properties of the first image and the second image. The computing device may determine a first set of visual properties associated with the first image and a second set of visual properties associated with the second image. The first set of visual properties and the second set of visual properties may include special positioning of an object (e.g., a location, coordinates of an object in the image, or a distance between two objects or points in the image), color of an object (e.g., hue, saturation, or contrast of an object), geometry attributes of an object (collinearity, curvature, length, width, breadth, added marks, missing marks, numerosity, shape, size, spatial grouping, and spatial orientation of elements of the object).

At block <NUM>, the computing device may identify a discrepancy between the first set of visual properties of the first image and the second set of visual properties of the second image.

At block <NUM>, the computing device may determine a correction to apply to the virtual image display system to at least partially correct for the discrepancy using the one or more known properties of the image capture device.

At block <NUM>, the computing device may apply the correction to the virtual image display system based on the one or more properties of the image capture device. The correction may calibrate the display device or the world camera of the virtual image display system.

It should be appreciated that the specific steps illustrated in <FIG> provide a particular method for determining and applying a correction to the virtual image display system according to some embodiments of the present disclosure. Moreover, the individual steps illustrated in <FIG> may include multiple sub-steps that may be performed in various sequences as appropriate to an individual step.

<FIG> is a simplified flowchart <NUM> illustrating a method for determining and applying a correction to the virtual image display system to adjust display color according to some embodiments of the present disclosure. The virtual image display system may include at least a display device and a world camera. According to various embodiments, an image capture element simulating a human eye (or a pair of human eyes) may be coupled to the virtual image display system.

At block <NUM>, a computing device including a processor receives a first image and a second image from a virtual image display system. The first image may be captured using a world camera of the virtual image display system, and the second image may be captured using an image capture device (e.g., an eye ball camera). The image capture device has one or more properties similar to a human eye including, but not limited to, a cornea position, a cornea geometry, an eyeball position, a pupil size, a pupil position, a gaze distance, and/or a gaze orientation. These properties of the image capture device may have predetermined values, and may be configurable.

At block <NUM>, the computing device may determine visual properties of the first image and the second image. The computing device may determine a spatial positioning of a point in the first image, and a spatial positioning of a corresponding point in the second image.

At block <NUM>, the computing device may identify a discrepancy between the spatial positioning of the point in the first image and the spatial positioning of the corresponding point in the second image.

At block <NUM>, the computing device may determine a correction to apply to the virtual image display system to align the point in the first image with the corresponding point in the second image.

At block <NUM>, the computing device may apply the correction to the virtual image display system based on the one or more properties of the image capture device. The correction may calibrate the display device or the world camera of the virtual image display system, and result in the point of the first image aligning more accurately with the corresponding point in the second image.

It should be appreciated that the specific steps illustrated in <FIG> provide a particular method for determining and applying a correction to the virtual image display system to adjust display color according to some embodiments of the present disclosure. Moreover, the individual steps illustrated in <FIG> may include multiple sub-steps that may be performed in various sequences as appropriate to an individual step.

<FIG> is a simplified flowchart <NUM> illustrating a method for determining and applying a correction to the virtual image display system to adjust geometry attributes according to some embodiments of the present disclosure. The virtual image display system may include at least a display device and a world camera. According to various embodiments, an image capture element simulating a human eye (or a pair of human eyes) may be coupled to the virtual image display system.

At block <NUM>, the computing device may determine visual properties of the first image and the second image. The computing device may determine a color (e.g., a hue, a saturation, or a contrast) of an area (e.g., a point, an object, or a pixel) in the first image, and a color of a corresponding area in the second image.

At block <NUM>, the computing device may identify a discrepancy between the color of the area in the first image and the color of the corresponding area in the second image.

At block <NUM>, the computing device may determine a correction to apply to the virtual image display system to match the color of the area in the first image with the color of the corresponding area in the second image.

At block <NUM>, the computing device may apply the correction to the virtual image display system based on the one or more properties of the image capture device. The correction may calibrate the display device or the world camera of the virtual image display system, and result in the color of the area in the first image matching more accurately with the color of the corresponding area in the second image. The correction may adjust one or more of the hue, saturation and contrast of the image displayed on the display device.

It should be appreciated that the specific steps illustrated in <FIG> provide a particular method for determining and applying a correction to the virtual image display system to adjust geometry attributes according to some embodiments of the present disclosure. Moreover, the individual steps illustrated in <FIG> may include multiple sub-steps that may be performed in various sequences as appropriate to an individual step.

<FIG> is a simplified flowchart <NUM> illustrating a method for determining and applying a correction to the virtual image display system to adjust spatial positioning according to some embodiments of the present disclosure. The virtual image display system may include at least a display device and a world camera. According to various embodiments, an image capture element simulating a human eye (or a pair of human eyes) may be coupled to the virtual image display system.

At block <NUM>, the computing device may determine visual properties of the first image and the second image. The computing device may determine a geometry attribute (e.g., form, collinearity, curvature, length, width, breadth, added marks, missing marks, numerosity, shape, size, spatial grouping, and/or spatial orientation) of an object in the first image, and a geometry attribute of a corresponding object in the second image.

At block <NUM>, the computing device may identify a discrepancy between the geometry attribute of the object in the first image and the geometry attribute of the corresponding object in the second image.

At block <NUM>, the computing device may determine a correction to apply to the virtual image display system to match the geometry attribute of the object in the first image with the geometry attribute of the corresponding object in the second image.

At block <NUM>, the computing device may apply the correction to the virtual image display system based on the one or more properties of the image capture device. The correction may calibrate the display device or the world camera of the virtual image display system, and result in the geometry attribute of the object in the first image matching more accurately with the geometry attribute of the corresponding area in the second image. The correction may adjust one or more of the collinearity, curvature, length, width, breadth, added marks, missing marks, numerosity, shape, size, spatial grouping, and/or spatial orientation of an object displayed on the display device.

It should be appreciated that the specific steps illustrated in <FIG> provide a particular method for determining and applying a correction to the virtual image display system to adjust spatial positioning according to some embodiments of the present disclosure. Moreover, the individual steps illustrated in <FIG> may include multiple sub-steps that may be performed in various sequences as appropriate to an individual step.

Embodiments provide a number of advantages over prior systems. Embodiments allow calibrating the real-world image capture device and/or an AR/VR display system using one or more cameras (e.g., eyeball cameras) that mimic the human eye. The properties of the eyeball camera such as one or more of a cornea position, a cornea geometry, an eyeball position, a pupil size, a pupil position, a gaze distance, or a gaze orientation are known and can be controlled. Thus, the calibration can be performed using a set of known and controllable parameters. For example, it is possible to fix the gaze on infinity and calibrate the device using the data. Thus, embodiments provide for a more accurate calibration of an AR/VR display system therefore resulting in a more accurate and seamless AR/VR experience.

Each of the processes, methods, and algorithms described herein and/or depicted in the attached figures may be embodied in, and fully or partially automated by, code modules executed by one or more physical computing systems, hardware computer processors, application-specific circuitry, and/or electronic hardware configured to execute specific and particular computer instructions. For example, computing systems can include general purpose computers (e.g., servers) programmed with specific computer instructions or special purpose computers, special purpose circuitry, and so forth. A code module may be compiled and linked into an executable program, installed in a dynamic link library, or may be written in an interpreted programming language. In some implementations, particular operations and methods may be performed by circuitry that is specific to a given function.

Further, certain implementations of the functionality of the present disclosure are sufficiently mathematically, computationally, or technically complex that application-specific hardware or one or more physical computing devices (utilizing appropriate specialized executable instructions) may be necessary to perform the functionality, for example, due to the volume or complexity of the calculations involved or to provide results substantially in real-time. For example, a video may include many frames, with each frame having millions of pixels, and specifically programmed computer hardware is necessary to process the video data to provide a desired image processing task or application in a commercially reasonable amount of time.

Code modules or any type of data may be stored on any type of non-transitory computer-readable medium, such as physical computer storage including hard drives, solid state memory, random access memory (RAM), read only memory (ROM), optical disc, volatile or non-volatile storage, combinations of the same and/or the like. The methods and modules (or data) may also be transmitted as generated data signals (e.g., as part of a carrier wave or other analog or digital propagated signal) on a variety of computer-readable transmission mediums, including wireless-based and wired/cable-based mediums, and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The results of the disclosed processes or process steps may be stored, persistently or otherwise, in any type of non-transitory, tangible computer storage or may be communicated via a computer-readable transmission medium.

Any processes, blocks, states, steps, or functionalities in flow diagrams described herein and/or depicted in the attached figures should be understood as potentially representing code modules, segments, or portions of code which include one or more executable instructions for implementing specific functions (e.g., logical or arithmetical) or steps in the process. In some embodiments, additional or different computing systems or code modules may perform some or all of the functionalities described herein. Moreover, the separation of various system components in the implementations described herein is for illustrative purposes and should not be understood as requiring such separation in all implementations. It should be understood that the described program components, methods, and systems can generally be integrated together in a single computer product or packaged into multiple computer products. Many implementation variations are possible.

The processes, methods, and systems may be implemented in a network (or distributed) computing environment. Network environments include enterprise-wide computer networks, intranets, local area networks (LAN), wide area networks (WAN), personal area networks (PAN), cloud computing networks, crowd-sourced computing networks, the Internet, and the World Wide Web. The network may be a wired or a wireless network or any other type of communication network.

Certain features that are described in this specification in the context of separate implementations also can be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation also can be implemented in multiple implementations separately or in any suitable subcombination. No single feature or group of features is necessary or indispensable to each and every embodiment.

Conditional language used herein, such as, among others, "can," "could," "might," "may," "e.g.," and the like, unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain embodiments include, while other embodiments do not include, certain features, elements and/or steps. Thus, such conditional language is not generally intended to imply that features, elements and/or steps are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without author input or prompting, whether these features, elements and/or steps are included or are to be performed in any particular embodiment. In addition, the articles "a," "an," and "the" as used in this application and the appended claims are to be construed to mean "one or more" or "at least one" unless specified otherwise.

As an example, "at least one of: A, B, or C" is intended to cover: A, B, C, A and B, A and C, B and C, and A, B, and C. Conjunctive language such as the phrase "at least one of X, Y and Z," unless specifically stated otherwise, is otherwise understood with the context as used in general to convey that an item, term, etc. may be at least one of X, Y or Z. Thus, such conjunctive language is not generally intended to imply that certain embodiments require at least one of X, at least one of Y and at least one of Z to each be present.

Claim 1:
A system (<NUM>) for applying a correction to a virtual image display system (<NUM>), the system (<NUM>) comprising:
a display device (<NUM>) configured to display a virtual image;
an image capture device (<NUM>) positioned in front of the display device (<NUM>), wherein the image capture device (<NUM>) has one or more artificial eyeballs (<NUM>), each comprised of an artificial cornea (<NUM>), an artificial pupil (<NUM>), or an artificial iris (<NUM>) comprising one or more sensing properties; and
a processor coupled to the display device (<NUM>) and the image capture device (<NUM>) to receive image data from the image capture device (<NUM>), wherein the processor is programmed to:
receive a first image of a scene from a real-world image capture device (<NUM>);
determine a first set of visual properties associated with the first image;
receive a second image of the scene captured with the image capture device (<NUM>);
determine a second set of visual properties associated with the second image, wherein the first set of visual properties and the second set of visual properties include at least one of a spatial positioning, a color, or a geometry attribute;
identify a discrepancy between the first set of visual properties and the second set of visual properties;
determine a correction to apply to the virtual image display system (<NUM>) to at least partially correct for the discrepancy using the one or more properties of the image capture device (<NUM>); and
apply the correction to the virtual image display system (<NUM>) based on the one or more sensing properties of the image capture device (<NUM>).