Patent Description:
Externally facing cameras have been used in a variety of sensing applications for virtual reality (VR) and augmented reality (AR) devices. The associated imaging sensors have the ability to capture scene information, which may then be processed and provided back to the user on a display screen. For VR devices, external cameras may provide a video passthrough mode that enables the user to see and interact with the world around them as if the user were viewing their environment directly. For AR devices, imaging sensors may provide the user with improved night vision using low light cameras or with the ability to see the temperature of the scene around them via thermal imaging sensors. These imaging capabilities are particularly useful to first responders and others operating in a variety of different, low-light environments.

As an example, <FIG> depicts an environment <NUM>, wherein a user <NUM> is viewing the environment through computing system <NUM>, which includes a head-mounted display (HMD). Computing system <NUM> includes an imaging device and a near-eye display. Other features of an example head-mounted display are discussed herein and with regard to <FIG>. User <NUM> and/or computing system <NUM> may move relative to the environment (as indicated by arrows). Movement relative to the environment may include 6DoF movement of computing system <NUM> relative to a fixed point in the environment, (e.g., rotation of computing system <NUM>), movement of objects in the environment relative to computing system <NUM>, etc..

User <NUM> views a portion of environment <NUM> via computing system <NUM>, yielding a field of view (FOV) <NUM> that may be augmented via the near-eye display. For example, image data of environment <NUM> taken by the imaging device may be displayed on the near-eye display. As such, user <NUM> may view real-world objects, such as trees <NUM>, <NUM>, and <NUM> (solid lines) and/or their video representations 110a, 111a, and 112a (dotted lines). Additional virtual objects <NUM> and <NUM> may also be displayed based on the acquired image data of environment <NUM>.

In some examples, such as when computing system <NUM> is configured to include a see-through display, both real-world objects and their video representations may be viewed simultaneously. If the real-world objects and their video representations are precisely overlaid, user <NUM> may be able to experience an enhanced version of environment <NUM>. For example, user <NUM> may be able to make out object outlines in low light, be able to view both an object and its thermal profile, etc. However, capturing and displaying objects on a head-mounted display in a low-light environment, such as environment <NUM> presents additional challenges. The computing device must operate in such a way that sufficient image data is obtained without adding significant noise or blurring.

Standard photodetector-based focal plane arrays are somewhat limited in that when ambient lighting level is reduced, the sensor image is dominated by dark current (e.g., current from reverse-biased photodetectors) and read noise (e.g., the noise from the readout circuitry). Typical digital cameras employ auto-exposure and auto-gain algorithms to capture and process images based on environmental lighting conditions, resulting in an increased signal-to-noise ratio in the captured images. However, in low-light imagery situations, exposure is likely to already be maximized. Increasing gain thus functions merely to increase noise.

Generally, there are two basic methods for reducing noise in these low light situations. One is to use hardware binning by combining the outputs of multiple pixels into a single analog readout (e.g., analog binning). The downside of binning is a reduction in the resolution of the captured imagery. Another approach is to increase the integration time, generally performed by decreasing the frame rate of the imaging device. With an increased integration time the amount of received signal is increased while maintaining the same noise characteristics. However, this may result in additional image blur to the system. Additionally or alternatively, images may be post-processed using multi-frame temporal averaging and appropriate motion compensation. In this way, image noise may be reduced by the square root of the number of averaged frames. However, this approach may also increase latency and reduce the effective frame rate, both of which may be undesirable in high-motion applications.

Herein, systems and methods are presented for automatically selecting the mode of operation for an imaging device of a head-mounted display system. During low-light conditions, frame rate and binning modes for the imaging device are determined based on an amount of motion of the head-mounted display system. For example, an on-board Inertial Measurement Unit may be used to determine motion of the device. During low signal, high motion conditions, pixel binning is implemented. During low signal, low motion conditions, frame rates are reduced. During high motion conditions where signal-to-noise ratio remains below an acceptable threshold, frame rates are reduced in addition to pixel binning. Multiple frame averaging, spatial filtering, & image signal processing (ISP) may also be implemented based on IMU signals.

<FIG> is an example implementation of the computing system <NUM> according to an example implementation of the present disclosure. In this example, as in <FIG>, the computing system <NUM> is integrated into and/or implemented as an HMD device <NUM>. In one example implementation, the computing system <NUM> may include one or more optical sensors, such as the depicted optical sensor <NUM> which may be a camera positioned centrally on an upper portion of HMD device <NUM> as shown in <FIG>. Optical sensor <NUM> may be an RGB camera and/or low-light camera such as an IR camera configured for night vision. In some examples, optical sensor <NUM> may include a thermal camera, such as those configured to receive and detect IR light in a range of <NUM> to <NUM>,<NUM>. It will be appreciated however that cameras included in the computing system <NUM> may be sensitive to various ranges of electromagnetic radiation as preferred by designers of the computing system <NUM>, such as UV light, visible light, near infrared light, or other suitable frequencies. Optical sensor <NUM> may additionally or alternatively include a dedicated ambient light sensor, such as a digital light sensor, configured to indicate a total amount of visible light present within an environment.

Optical sensor <NUM> may include a sensor array of individually addressable pixels. In some implementations, the pixels may be complementary metal-oxide semiconductor (CMOS) elements, but other suitable architectures are envisaged as well. For example, pixels may include light sensing elements having compositions such as SiGe, InGaAs, InGaN, GaAsPh, and/or other III-V compounds. In some examples, the shuttered optical sensors may include PIN diodes made from alloy semiconductors such as InGaAs. As non-limiting examples, optical sensor <NUM> may include one or more of an array of silicon sensors having >5x5 um pixels, SiGe detectors having an extended detection range of up to <NUM>, InGaAs focal plane arrays (FPA) exhibiting longer wavelength detection, and/or cooled detectors for infrared (e. g short-wave infrared, medium infrared, long-wave infrared) detection. Each pixel is responsive to light over a broad wavelength band. For silicon-based (e.g. CMOS) pixels, the wavelength response may range from <NUM> to <NUM>, for example.

In some examples, an optical shutter arranged over sensor array <NUM>, so as to optically cover the sensor array. Such an optical shutter may be configured as a rolling shutter, wherein readouts of different portions of the image frames are performed at different times, such as on a sequential, line-by-line basis. Additionally or alternatively such an optical shutter may be configured as a global shutter, wherein accumulated charge is stored in a light-shielded region on a per-pixel or per-group-of pixels basis.

In some implementations, the pixels of the sensor array may be differential pixels. Each differential pixel may include different collection terminals that are energized according to two different clock signals. In one example, to measure modulated active illumination, the two clock signals may be substantially complementary (e.g., the two clock signals have <NUM>% duty cycles that are <NUM> degrees out of phase). In other examples, the two different clock signals may have a different relationship, such as for measuring ambient illumination or non-modulated active illumination. While differential pixels provide the advantages described herein, it will be appreciated that other types of sensor array, including non-differential sensor arrays, may be used.

In some examples, a depth detection system (not shown) may also be included in the computing system <NUM> and integrated into the HMD device <NUM>. The depth detection system may also include components such as a pair of stereo cameras and/or a pair of stereo low-level light cameras. Other depth detection systems may include a single camera and a light projector, a pair of cameras and a light projector, and/or a laser light source and a camera. However, active stereo methods of depth detection may additionally process light projected by a projector that may be received at right and left cameras. A structured light method of depth detection may also be integrated into the computing system <NUM>, in which case a projector and one camera to receive reflected projected light may be utilized. If a time-of-flight method of depth detection is preferred, the HMD device <NUM> may include a laser light source and corresponding sensor such as an IR laser in addition to a camera to receive reflected laser light. In another configuration, an inertial measurement unit (IMU) <NUM> and a single camera may be used to detect depth.

The example computing system <NUM> includes a processor <NUM> and associated storage, which in <FIG> includes volatile memory <NUM> and non-volatile memory <NUM>. The processor <NUM> is configured to execute instructions stored in the storage, using volatile memory <NUM> while executing instructions belonging to various programs and non-volatile memory <NUM> for storage of the programs. Other sensors that may be included in the computing system <NUM> as embodied in the HMD device <NUM> may be inward-facing cameras <NUM> to identify the position and orientation of each of a user's eyes and subsequently generate eye-tracking data. Also, a microphone <NUM> may receive natural language (NL) input from a user of the HMD device <NUM>.

IMU <NUM> may be implemented in the HMD device <NUM> as described above, which may include accelerometers, gyroscopes, and/or a compass that can be used to detect, for example, a <NUM> degree of freedom (3DOF, e.g., orientation) of the HMD device. A <NUM> degree of freedom (6DOF) position and orientation of the HMD device <NUM>, may also be detected. Processor <NUM> may further refine the 6DOF output of IMU <NUM> using visual tracking systems that search for movement of identified visual features in a series of images captured by optical sensor <NUM> and/or other cameras to generate an estimate of the relative movement of the HMD device <NUM> based upon the movement of these visual features within successive image frames captured by optical sensor <NUM> over time. It will be appreciated that components such as the microphone <NUM> and/or one or more optical sensors <NUM> may be integrated with the HMD device <NUM> or provided separately therefrom. It will be further appreciated that other types of sensors not displayed in <FIG> may be included in the computing system <NUM>.

A display <NUM> may be integrated with the HMD device <NUM>, or optionally provided separately. Speakers <NUM> may also be included in the HMD device <NUM>, or also provided separately. It will be appreciated that electronic and computing components may be connected via a bus. Furthermore, <FIG> depicts various computing system components that may correspond to the components of <FIG>, and the descriptions of those components in <FIG> may therefore apply to such corresponding components in <FIG>.

As shown in <FIG>, the processor <NUM>, volatile and non-volatile memories <NUM>, <NUM>, inward-facing cameras <NUM>, optical sensor <NUM>, microphone <NUM>, IMU <NUM>, and speakers <NUM> may be incorporated within a housing of HMD device <NUM> as shown. HMD device <NUM> may include a mounting frame <NUM> that at least partially encircles the head of a user, and the display <NUM> may include a pair of right and left near-eye displays 236A and 236B. The near-eye displays 236A and 236B may be positioned behind a visor <NUM> through which a user may observe the physical surroundings in an augmented reality (AR) system. It will be appreciated that the near eye displays 236A and 236B and visor <NUM> may be at least partially transparent, enabling the user to see through these components to view the real environment, at least when content is not opaquely displayed on the near-eye displays 236A and 236B.

While described predominantly in terms of augmented reality systems with see-through displays, the systems and methods described herein are equally applicable to virtual reality systems and other mixed reality systems that with more opaque display units. Similarly, while described predominantly with reference to head-mounted display units that include a single, centered optical sensor or camera, the systems and methods described are equally applicable head-mounted display units that include offset cameras and/or multiple cameras, including depth cameras in scenarios where processing speed and power limit the amount of depth information can be processed to enable near-real time image reprojection. Further, the described methods may be applied to movable cameras that are not coupled to a head-mounted display, such as those attached to drones, vehicles, aircraft, watercraft, etc. Additionally, one or more additional computing devices, such as cloud computing devices may be communicatively coupled to the head-mounted display system in order to process image data and/or otherwise provide processing bandwidth to execute the described method and processes.

Both read noise and dark current are independent of signal level, and thus may significantly impact the signal-to-noise ratio of captured images at low light levels. As scene brightness increases, the analog gain of the sensor is reduced, resulting in higher read noise and greater full well capacity. As such, it may be desirable to increase an amount of signal received at an imaging device prior to issuing data, in order to increase the signal captured by a factor greater than this inherent noise factor.

Method <NUM> is a method for operating a head-mounted display system including an imaging device. More specifically, method <NUM> is targeted to adjusting operating modes of the imaging device responsive to low light conditions and based on movement of the head-mounted display in order to increase signal-to-noise ratio. In some aspects, this is accomplished by selectively adjusting the integration time and/or binning mode of the imaging device based on operating conditions. A binning mode, as applied here may include any suitable means, analog and/or digital, of combining the output of two or more individual pixels into a single superpixel.

At <NUM>, method <NUM> includes, responsive to an indication that an ambient light condition in an environment is below a lighting threshold, determining an amount of motion of the head-mounted display relative to the environment based on one or more signals received from an inertial measurement unit included in the head-mounted display system. In some examples, the IMU may be continuously or periodically measuring and/or determining motion of the head-mounted display relative to the environment. In such examples, determining an amount of motion may include sampling the output of the IMU responsive and comparing the signal to the indication of an ambient light condition decreasing below a threshold.

In some examples, the ambient light condition may be determined based on a signal received by an optical sensor, such as optical sensor <NUM> depicted in <FIG>. Such an optical sensor may be included in an imaging device, such as a 2D and/or 3D camera, and/or may be a standalone digital light sensor. The ambient light sensor may provide and/or generate a signal that correlates with the level of signal received by the imaging device. The criteria for establishing a lighting threshold may be pre-determined, and/or may be based on operating conditions, user preferences, etc..

The signals received from the inertial measurement unit may include acceleration, velocity, rotational velocity, trajectory, and/or other signals that indicate an amount of motion of the head-mounted display, and thus the included imaging device, relative to objects within an environment. In some examples, the amount of motion of the head-mounted display may be determined at least in part based on visual tracking of the HMD using one or more external cameras and a computing device configured to search for movement of identified visual features correlating to the HMD.

At <NUM>, method <NUM> includes automatically selecting an exposure time, frame rate, and a pixel-binning mode for the imaging device based on the determined amount of motion. As described, one or more thresholds may be set to which the amount of motion may be compared. In some examples, a plurality of thresholds may be implemented over a continuum. In some examples, such thresholds may be user-adjustable. In other words, the user may determine an amount of motion above which the frame rate or binning mode may be adjusted, as some individuals may be more sensitive to motion blur than others. In some examples, the motion thresholds may be determined based on application specific criteria. For example, applications for generating a landscape image may have different thresholds than applications for traversing outdoor terrain at night, which may have different thresholds than for rapidly moving through a building. For example, for night terrain traversal, operating in a binned mode at the lowest possible frame rate may be preferable because the user is likely to be moving slowly, and thus the captured imagery is less susceptible to motion blur. For applications used in a fast paced, time-critical environment, modes and frame rates that reduce integration time and corresponding motion blur and latency may be preferred. In some examples, the selection of an exposure time, frame rate, and/or an operating mode may be iterative, wherein results of a mode selection are assessed and then adjusted based on those assessments.

At <NUM>, method <NUM> includes increasing the integration time and reducing a frame rate of the imaging device responsive to the amount of motion being below a motion threshold. For example, in environments with low IMU movement and low signal, a reduced frame rate may be applied, as motion blur is lessened as a compounding factor as compared to high IMU movement conditions.

Reducing the frame rate, and thus inversely increasing the maximum potential exposure time effectively increases the integration time for the sensor, resulting in additional signal processed per image frame. However, the integration time need not be set at the highest possible duration for a given frame rate. Rather, the integration time may be adjusted based on operating conditions within the parameters set by the frame rate. It should be noted, however, that very short integration times can result in significantly reduced camera sensitivity. This may result in applying increased gain which may introduce additional noise to the signal.

Multiple thresholds (e.g., ambient light thresholds) may be applied having multiple resulting frame rates. The applied sensor frame rates may be integer ratios of the display presentation rate. For example, an HMD having a display operating at <NUM> may operate the imaging sensor at <NUM>, <NUM> (e.g., <NUM> capture frame to <NUM> display frames), <NUM> (<NUM>:<NUM>), <NUM> (<NUM>:<NUM>), etc. Further, the reduced frame rates allow for a longer integration time. For instance, a <NUM> frame rate allows for a maximum integration time of <NUM>, a <NUM> frame rate allows for a <NUM> integration time, a <NUM> frame rate allows for a <NUM> integration time, and a <NUM> frame rate allows for a <NUM> integration time.

As an example, <FIG> shows a pair of example image frames. Image <NUM> was acquired at <NUM> operating speed. Image <NUM> was acquired at <NUM> operating speed. Objects in the images were not moving relative to the imaging device during image capture. The improvement in SNR is thus attributable to the increase in integration time stemming from the decrease in frame rate.

Returning to <FIG>, at <NUM>, method <NUM> includes operating the imaging device in a binned-pixel mode responsive to the amount of motion being above the motion threshold. In each of the described scenarios, the amount and type of filtering may be deliberately applied. For instance, under normal operations, e.g., where a person is walking or maneuvering through a scene), the imaging device could operate in the standard <NUM> mode without pixel binning. As the light level drops, but the motion remains modest, the capture frequency may be reduced to increase the SNR without fear of motion blue. In scenarios where there is a high amount of motion, but also low signal level, pixel binning is applied to increase the SNR.

In some examples, the binned-pixel mode may include pixel binning in an analog domain, as only a single event is incurred for multiple pixels. However, in some examples, pixel binning may additionally or alternatively be performed in a digital domain. Analog binning may include combining the electric charge from adjacent CMOS or CCD sensor pixels into one super-pixel, to reduce noise by increasing the signal-to-noise ratio.

As an example, at <NUM>, <FIG> schematically shows a pixel array <NUM> operating at full resolution (e.g., unbinned mode). Pixel array <NUM> is depicted as an 8x6 block of <NUM> pixels, coupled to Analog-to-digital converter (ADC) block <NUM>. In this example, each of the <NUM> pixels within array <NUM> are activated and send an individual signal to ADC block <NUM>. At <NUM>, pixel array <NUM> is shown operating in a binned mode. Pixel array <NUM> is divided into <NUM>-pixel clusters <NUM>. However, in some example, the clusters could be larger (e.g., 3x3), or smaller, and do not need to be isometric (e.g., 1x3, 2x1). In some examples, multiple levels of binning may be achieved, so that multiple thresholds may evoke binning of increasing number of pixels.

For analog binning, the ADC readout circuitry gathers collective charge from <NUM> neighboring pixels. As such, the read noise for the <NUM>-pixel cluster is reduced <NUM>-fold. For digital binning, all <NUM> pixels are read out to ADC <NUM>, then combined and averaged. As such, the read-noise amount is carried from all read-out pixels, and SNR scales as the square root of the number of read-out pixels.

In some examples, operating in a binned mode additionally or alternatively includes applying digital filtering and/or increasing the strength of an applied digital spatial filter. As one example, a bilateral filter or other filter that takes the mean of two or more pixels within a pixel neighborhood may be applied. In some examples, one or more of the strength, size, and weight of the filter kernel may be dynamically controlled based on signals output from the IMU. Such an approach may be used to increase SNR without impacting image resolution.

Returning to <FIG>, at <NUM>, method <NUM> includes, while operating the imaging device in the binned-pixel mode, receiving an indication that a signal-to-noise ratio of the imaging device is below a threshold. If the reflectivity and content of the environment is known SNR may be directly measured. If not, the signal-to-noise ratio of the imaging device may be determined based on grey level counts for one or more image frames. A gray level count may be obtained for each pixel of a frame, generating a histogram, or a description of the measured intensity values. A higher range of gray tones correlates with a higher SNR (assuming that we correctly perform dark current subtraction). For example, if the grey level count distribution for a scene is below a threshold, the SNR may be presumed to be above a threshold. In some examples, if there is a very low grey level count in the scene, it can be assumed that the overall signal level is low (e.g., ambient light condition below a threshold). In some examples, average pixel intensity (e.g., signal) and standard deviation (e.g., noise) may be computed for each pixel over time, and an average taken across one or more frames of the pixel array. Some motion filtering may be applied to account for pixels or regions of pixels with expected motion. In some examples, the ambient light condition, amount of motion, and signal-to-noise ratio of the imaging device are determined on a frame-by-frame basis. In some examples, the ambient light condition, amount of motion, and signal-to-noise ratio of the imaging device may be determined based on a rolling average of two or more image frames.

At <NUM>, method <NUM> includes, reducing a frame rate of the imaging device in response to the indication that the signal-to-noise ratio of the imaging device is below the threshold. In such a scenario, including high IMU movement, low ambient lighting, and low SNR, both pixel binning and an increased integration time (through a reduced frame rate) are applied.

As an example, <FIG>. shows a plot <NUM> indicating signal-to-noise ratios across increasing ambient light levels for various imaging device operating modes. Line <NUM> (dotted line) represents an imaging device operating in an unbinned mode at a <NUM> frame rate. Line <NUM> (dashed line) represents the imaging device operating in an analog binned mode at <NUM>. An imaging device operating in an unbinned mode at <NUM> would demonstrate a similar response to line <NUM>. Line <NUM> (solid line) represents the imaging device operating in an analog binned mode at <NUM>). Lines <NUM> and <NUM> assume that the binned mode is an analog binned mode, accounting for the significant increase in SNR over line <NUM>.

By reducing the frame rate (e.g., increasing the integration time) or operating in binned mode, the SNR of the image may be substantially increased. Although the combination of a lower frame rate (longer integration time) and pixel binning shows a dramatic increase in SNR, it may not be used in all scenarios due to the increase motion blur and latency that is occurred with the longer integration time.

In one example, referring to <FIG>, if user <NUM> was attempting to pan environment <NUM> with the highest possible SNR, but the objects (trees <NUM>, <NUM>, <NUM>) were at a far distance away, binned mode would not be preferred, due to the loss of spatial resolution. However, as the motion of computing system <NUM> is relatively low, the integration time increased and frame rate may be reduced without causing significant motion blur. For higher motion applications, operating at a full frame rate, either binned or unbinned, may be preferred. For low light navigation applications, a binned mode with a reduced frame rate may be preferred. Such modes may be entered via user and/or application preferences, and/or may be automatically selected as part of the automatic exposure and gain control algorithms of the camera system.

Optionally, at <NUM>, method <NUM> includes operating the imaging device at full resolution and full frame rate responsive to determining that an ambient light condition is above the lighting threshold. As described at <NUM>, the ambient light condition may be determined on a frame-by-frame basis or based on a rolling average of two or more images.

At <NUM>, method <NUM> includes capturing imagery from the environment using the automatically selected frame rate and a pixel-binning mode for the imaging device. Continuing at <NUM>, method <NUM> includes displaying the captured imagery at the head-mounted display system. Prior to displaying the captured imagery, in some examples, one or more Image Signal Processing algorithms, such as motion compensation, spatial filtering, and multi-frame temporal averaging, may be applied based on one or more of the ambient light condition, the amount of motion of the head-mounted display, and the signal-to-noise ration of the captured images.

Implementing the systems and methods described herein allow for adjustment from a high frame rate to a low frame rate and an unbinned operating mode to a binned operating mode without requiring user intervention. Rather operating conditions can be measured, and the operating mode of the imaging device adjusted dynamically based on how the device is being used. This enables an improvement in SNR of images captured by the imaging device while allowing the user to attain an immersive viewing experience appropriate for the conditions at hand.

Computing system <NUM> may be considered an embodiment of computing systems <NUM> and <NUM>.

Computing system <NUM> includes a logic machine <NUM> and a storage machine <NUM> Computing system <NUM> may optionally include a display subsystem <NUM>, input subsystem <NUM>, communication subsystem <NUM>, and/or other components not shown in <FIG>.

In one example, a method for operating a head-mounted display system including an imaging device comprises responsive to an indication that an ambient light condition in an environment is below a lighting threshold, determining an amount of motion of the head-mounted display relative to the environment based on one or more signals received from an inertial measurement unit included in the head-mounted display system; automatically selecting an exposure time, frame rate and a pixel-binning mode for the imaging device based on the determined amount of motion; capturing imagery from the environment using the automatically selected exposure time, frame rate and pixel-binning mode for the imaging device; and displaying the captured imagery at the head-mounted display system. In such an example, or any other example, automatically selecting the exposure time, frame rate, and the pixel-binning mode for the imaging device based on the determined amount of motion additionally includes: increasing an integration time and reducing a frame rate of the imaging device responsive to the amount of motion being below a motion threshold; and operating the imaging device in a binned-pixel mode responsive to the amount of motion being above the motion threshold. In any of the preceding examples, or any other example, the method additionally comprises, while operating the imaging device in the binned-pixel mode, receiving an indication that a signal-to-noise ratio of the imaging device is below a threshold; and reducing a frame rate of the imaging device in response to the indication that the signal-to-noise ratio of the imaging device is below the threshold. In any of the preceding examples, or any other example, the method additionally comprises, operating the imaging device at full resolution and full frame rate responsive to determining that an ambient light condition is above the lighting threshold.

In another example, a method for operating a head-mounted display system including an imaging device, comprises determining that an ambient light condition is below a lighting threshold; responsive to determining that the ambient light condition is below the lighting threshold, determining an amount of motion of the head-mounted display system; increasing an integration time and reducing a frame rate of the imaging device responsive to the amount of motion being below a motion threshold; operating the imaging device in a binned-pixel mode responsive to the amount of motion being above the motion threshold; capturing imagery of the environment using a selected frame rate and operating mode; and streaming the captured imagery to the head-mounted display system. In such an example, or any other example, the method additionally comprises, while operating the imaging device in the binned-pixel mode, receiving an indication that a signal-to-noise ratio of the imaging device is below a threshold; and reducing a frame rate of the imaging device in response to the indication that the signal-to-noise ratio of the imaging device is below the threshold. In any of the preceding examples, or any other example, the signal-to-noise ratio of the imaging device is additionally or alternatively determined based on grey level counts for one or more image frames. In any of the preceding examples, or any other example, the ambient light condition, amount of motion, and signal-to-noise ratio of the imaging device are additionally or alternatively determined on a frame-by-frame basis. In any of the preceding examples, or any other example, the ambient light condition, amount of motion, and signal-to-noise ratio of the imaging device are additionally or alternatively determined based on a rolling average of two or more image frames. In any of the preceding examples, or any other example, the binned-pixel mode additionally or alternatively includes pixel binning in an analog domain. In any of the preceding examples, or any other example, the method additionally or alternatively comprises operating the imaging device at full resolution and full frame rate responsive to determining that an ambient light condition is above the lighting threshold. In any of the preceding examples, or any other example, the amount of motion is additionally or alternatively determined based on one or more signals received from an IMU included in the head-mounted display system. In any of the preceding examples, or any other example, a strength of applied spatial filtering is additionally or alternatively adjusted based on the one or more signals received from the IMU.

In yet another example, a head-mounted display system, comprises an imaging device; a near-eye display; and a controller configured to: determine that an ambient light condition is below a lighting threshold; responsive to determining that the ambient light condition is below the lighting threshold, determine an amount of motion of the head-mounted display system; increase an integration time and reduce a frame rate of the imaging device responsive to the amount of motion being below a motion threshold; operate the imaging device in a binned-pixel mode responsive to the amount of motion being above the motion threshold; capture imagery of the environment using a selected frame rate and operating mode; and stream the captured imagery to the near-eye display. In such an example, or any other example, the controller is additionally configured to, while operating the imaging device in the binned-pixel mode, receive an indication that a signal-to-noise ratio of the imaging device is below a threshold; and reduce a frame rate of the imaging device in response to the indication that the signal-to-noise ratio of the imaging device is below the threshold. In any of the preceding examples, or any other example, the signal-to-noise ratio of the imaging device is additionally or alternatively determined based on grey level counts for one or more image frames. In any of the preceding examples, or any other example, the ambient light condition, amount of motion, and signal-to-noise ratio of the imaging device are additionally or alternatively determined on a frame-by-frame basis. In any of the preceding examples, or any other example, the binned-pixel mode additionally or alternatively includes pixel binning in an analog domain. In any of the preceding examples, or any other example the controller is additionally or alternatively configured to: operate the imaging device at full resolution and full frame rate responsive to determining that an ambient light condition is above the lighting threshold. In any of the preceding examples, or any other example, the head-mounted display system additionally or alternatively comprises an inertial measurement unit, and the controller is additionally or alternatively configured to: determine the amount of motion based on one or more signals received from the inertial measurement unit.

Claim 1:
A method (<NUM>) for operating a head-mounted display system (<NUM>) including an imaging device (<NUM>), comprising:
responsive to an indication that an ambient light condition in an environment (<NUM>) is below a lighting threshold, determining (<NUM>) an amount of motion of the head-mounted display (<NUM>) relative to the environment (<NUM>) based on one or more signals received from an inertial measurement unit (<NUM>) included in the head-mounted display system (<NUM>);
automatically selecting (<NUM>) an exposure time, frame rate and a pixel-binning mode for the imaging device (<NUM>) based on the determined amount of motion;
capturing (<NUM>) imagery from the environment (<NUM>) using the automatically selected exposure time, frame rate and pixel-binning mode for the imaging device (<NUM>); and
displaying (<NUM>) the captured imagery at the head-mounted display system (<NUM>); wherein automatically selecting the exposure time, frame rate, and the pixel-binning mode for the imaging device based on the determined amount of motion includes:
increasing an integration time and reducing a frame rate of the imaging device responsive to the amount of motion being below a motion threshold; and
operating the imaging device in a binned-pixel mode responsive to the amount of motion being above the motion threshold;
while operating the imaging device in the binned-pixel mode, receiving an indication that a signal-to-noise ratio of the imaging device is below a threshold; and
reducing a frame rate of the imaging device in response to the indication that the signal-to-noise ratio of the imaging device is below the threshold.