Electronic Devices With Illumination Systems

A head-mounted device may include displays for presenting images to a user. The head-mounted device may have a hand tracker for gathering hand tracking input from a user. The hand tracker may include a steerable illumination system such as a steerable infrared illuminator. Infrared light may be steered by selectively activating light-emitting diodes in an array and/or using positioners to move an infrared light source and/or lens in the illuminator. An infrared image sensor may gather images of the hands. Based on the monitored movement of a hand, the steerable illumination system may steer the infrared light to ensure that the hand remains illuminated even during hand movements.

FIELD

This relates generally to illumination systems, and, more particularly, to electronic devices with illumination systems.

BACKGROUND

Electronic devices have components such as image sensors. It can be challenging to operate image sensors in certain ambient lighting conditions such as low ambient lighting conditions.

SUMMARY

Electronic devices may be provided with sensors and adjustable illumination systems that provide illumination for the sensors.

A head-mounted device may include displays for presenting images to a user. The head-mounted device may have a hand tracker for gathering hand input from a user. The hand tracker may include a steerable illumination system such as a steerable infrared illuminator. Using the steerable infrared illuminator, infrared illumination may be provided in a relatively narrow beam while illuminating a user's hand. This may help the head-mounted device conserve power while providing satisfactorily illumination.

Infrared light may be steered by selectively activating light-emitting diodes in an array of diodes in an infrared light source and/or by using positioners to move the infrared light source and/or lens in the illuminator. During operation of a device, an infrared image sensor may gather images of a user's hand. Based on the monitored movement of the hand, the steerable illumination system may steer the infrared light so that the infrared light tracks the moving hand. This helps to ensure that the moving hand will remain illuminated during hand motions and will not move out of the area illuminated by the illuminator.

If desired, adjustable illumination systems may be used in laptop computers, tablet computers, cellular telephones, wristwatches, and other devices.

DETAILED DESCRIPTION

Electronic devices may have sensors for gathering user input and making other measurements. Some electronic devices may use image sensors. As an example, an electronic device with video conferencing capabilities may use an image sensor to capture images of a user during a video call. As another example, a gaze tracking system in an electronic device may use an image sensor to capture images of infrared light glints on a user's eyes and thereby track the direction of the user's gaze. Some electronic devices may have hand tracking systems. In hand tracking systems, image sensors may be used to capture images of a user's hands. By processing these images, the movement of the user's hands and gestures made by the user's hands and/or fingers can be identified and used as a form of user input.

In some ambient lighting conditions, there may be sufficient ambient light available to operate image sensors. For example, in bright ambient lighting conditions, images of a user's hands may be captured satisfactorily. In dim ambient lighting conditions, however, it may be desirable for an electronic device to emit supplemental illumination to ensure that the user's hands remain visible to the image sensors of the device. Supplemental illumination may be provided at visible and/or infrared wavelengths. As an example, when tracking a user's hands in dim lighting, supplemental infrared illumination may be emitted. The presence of the supplemental infrared illumination may help infrared image sensors capture satisfactory hand images. At the same time, the use of infrared illumination, which is invisible to the naked eye, may help avoid disturbing people who are nearby.

To help ensure that excessive power is not consumed during the production of supplemental illumination, electronic devices may be provided with adjustable illumination systems. The angular spread and intensity of the emitted illumination may be adjusted in real time to focus on target objects of interest. For example, when tracking a user's hand, an adjustable illumination system may produce a beam of infrared light that is focused on the user's hand. As the user's hand is moved, the infrared light beam may be steered to follow the hand. With this approach, it is not necessary to illuminate extraneous background objects in a scene. The use of adjustable illumination systems may therefore help conserve power.

FIG.1is a schematic diagram of an illustrative electronic device. Device10ofFIG.1may be a head-mounted device (e.g., goggles, glasses, a helmet, and/or other head-mounted device), a laptop computer, a portable electronic device such as a cellular telephone, tablet computer, or wristwatch device, or other electronic equipment.

As shown inFIG.1, device10may have a controller such as control12. Control12may have control circuitry such as microprocessors, microcontrollers, digital signal processors, application-specific integrated circuits, storage such as volatile and non-volatile memory, and other storage and processing circuitry. Control12may be used in processing data gathered by the components in device10and may be used in controlling device operations.

Device10may have one or more infrared image sensors14(sometimes referred to as infrared cameras). For example, device10may use one or more image sensors14for gaze tracking, hand tracking, facial recognition, face and/or body tracking, and/or other operations involving the capturing and processing of infrared images. Adjustable infrared illumination (e.g., supplemental infrared illumination) may be provided by one or more infrared light sources such as infrared light source16. Source16may include one or more light-emitting elements (e.g., light-emitting diodes, lasers, etc.).

Device10may also include additional components18(e.g., additional sensors such as visible light image sensors (sometimes referred to as visible light cameras), displays and other output devices that emit light (e.g. head-mounted device displays that present images to eye boxes for viewing by a user), touch sensors, force sensors, proximity sensors, temperature sensors, accelerometers, microphones, and other sensors, haptic output devices, speakers, actuators (sometimes referred to as positioners or motors), a battery for supplying device10with power, and other components.

Components for device10such as image sensors14, light sources16, and other components18may be mounted at any suitable location in device10. Illustrative mounting arrangements for device components are shown inFIGS.2,3, and4.

In the example ofFIG.2, device10is a head-mounted device having a head-mounted housing20and head straps22. Housing20may have a front F and a rear R. The components of device10ofFIG.1may be mounted in head-mounted housing20. During operation, rear R of housing20may rest against a user's face. Optical assemblies containing displays and lenses may present images in a pair of eye boxes at rear R for viewing by a user. An outwardly facing display may optionally be mounted in region24at front F. Components for device10such as image sensors14, light sources16, and other components18may be mounted on outwardly facing surfaces of housing20(e.g., at one or more illustrative locations26at front F and/or in other portions of housing20).

In the example ofFIG.3, device10is a laptop computer having housing portions20-1and20-2that rotate with respect to each other about hinge28. A display may be mounted on portion20-1and a keyboard may be mounted on portion20-2. Components for the laptop computer such as image sensors14, light sources16, and other components18may be mounted on one or more outwardly facing surfaces of housing20(e.g., at one or more illustrative locations26on housing20-1or elsewhere in housing20).

FIG.4is a perspective view of device10in an illustrative configuration in which device10is a portable electronic device such as a tablet computer, cellular telephone, or wristwatch device. As shown inFIG.4, device10may have a housing such as housing20. A display such as a touch-screen display may be mounted on face30of device10. Components for device10such as image sensors14, light sources16, and other components18may be mounted on one or more outwardly facing surfaces of housing20(e.g., at one or more illustrative locations26on face30).

FIG.5is a diagram of an illustrative electronic device that is providing illumination to illuminate a target. As shown inFIG.5, electronic device10may include adjustable illumination system32and sensor34. Adjustable illumination system32, which may sometimes be referred to as a steerable light source, steerable lighting system, adjustable lighting system, adjustable illuminator, or steerable illuminator, may have one or more light-emitting components such as infrared light source16ofFIG.1that emit light36(sometimes referred to as illumination). Light36may illuminate a target object such as target38. Light36may be visible and/or infrared light. In an illustrative configuration, which is sometimes described herein as an example, light36is infrared light (infrared illumination) and system32may be referred to as an adjustable infrared illuminator or steerable infrared illuminator.

Target38may be an animate or inanimate object. As an example, target38may include one or more body parts of a user (e.g., a user's hand, fingers, eye, head, full body, etc.) or inanimate objects (e.g., a wand). Illustrative configurations in which target38is a hand of a user and light36is infrared hand tracker illumination may sometimes be described as an example.

When system32and sensor34are used together as combined system31in a head-mounted device (or other device) for hand tracking, this combined system31may sometimes be referred to as a hand tracker or hand tracking system. During hand tracking operations with the hand tracker, hand motions, finger gestures and other hand input may be used as a form of user input for controlling the operation of device10. For example, hand gestures may be used to move virtual objects being presented to the user with the display resources of device10, may be used to make menu selections in displayed content, etc. Virtual objects (which may sometimes be referred to as computer-generated content) may be presented in isolation or may be merged with real-world content (e.g., real-time real-world content captured using forward-facing cameras in housing20). If desired, virtual hands may be presented to a user in positions that correspond to the real-world positions of the user's hands as sensed with the hand tracker.

During operation, while target38is illuminated by light36and by any ambient light that may or may not be present, a sensor such as sensor34may be used to gather data on target38. As an example, sensor34may include one or more infrared image sensors such as infrared image sensors (cameras)14ofFIG.1and/or visible light image sensors (see, e.g., components18). In an arrangement in which sensor34is an infrared image sensor in a hand tracker, sensor34may gather infrared hand images for hand tracking (e.g., infrared hand images illuminated by infrared light36, sometimes referred to as hand tracking image or infrared hand tracking images). The fields of view of the image sensors may at least partly overlap so that three-dimensional images may be captured. This allows device10to track the position of target38in three dimensions. For example, sensor34may track movement of a user's hand as the hand is moved up/down, left/right, and/or forward/backward with respect to sensor34. In a head-mounted device, for example, one or both of the user's hands may be within the field-of-view of sensor34and may be illuminated by light36(e.g., light36may be produced as supplemental illumination whenever ambient lighting conditions are dim and/or may be produced during other ambient lighting conditions).

Using sensor34, images (e.g., three-dimensional images from stereoscopic infrared image sensors) may be captured that reveal the location (e.g., position in three dimensions) of target38. By comparing the position of target38between successive image frames or other suitable time period, the speed and direction of motion of target38may be determined by sensor34. Based on this monitored hand motion information, sensor34can predict the future position of target38. This allows a beam of light36from system32to be steered or otherwise adjusted to ensure that target38can be tracked even as target38is moved.

In general, adjustments to the light36that is produced by system32may include adjustments to the direction in which light36is emitted and adjustments to the angular coverage of light36(e.g., light beam direction and/or light beam size). Light36may also be split into multiple beams, if desired. In some situations, adjustments may be made to the intensity of light36(e.g., output light intensity may be increased whenever appropriate to ensure that a desired minimum signal-to-noise ratio is obtained when capturing images with sensor34and may be otherwise decreased to conserve power). Conditions that may impact the signal-to-noise ratio include ambient light level, target distance, and target reflectivity (as examples).

Consider, as an example, a scenario in which target38is being moved in direction40to position38′. Using sensor34, the present location of target38can be captured and motion information indicating the speed and direction of movement of object38can be determined. Sensor34can then predict the location (in three dimensions) of position38′. Using this information from sensor34, system32can steer light36towards position38′. As shown inFIG.5, for example, the direction in which light36is emitted can be changed to new direction36′ by making angular adjustment42to the direction of the emitted illumination from system32.

Initially, when hand tracking (or other target tracking) operations are commenced, device10may not know the location of target38. Hand tracking may also sometimes be interrupted momentarily (e.g., when the user's hands are blocked by an obstruction). As shown inFIG.6, in situations such as these, system32may be adjusted to provide flood illumination (e.g., light36may completely cover area44, which may be, for example, the entire field of view of sensor34). After the location of target38is reacquired by sensor34while light36is emitted over all of area44in this way, system32may be adjusted to narrow the angular spread of light36. System32may, as an example, be used to produce a beam of light36that covers reduced-size area44′ ofFIG.7. Area44′ may be adjusted to overlap target38(with sufficient buffer around the edge of target38to account for processing latency), so system34can continue to track target38despite covering a smaller area than area44. The size of area44′ may be 70% or less, 30% or less, or 15% or less of the total size of area44, thereby helping to conserve power.

In the event that target38moves (e.g., in direction46ofFIG.7), sensor34can detect this movement and, based on the monitored target motion (e.g., monitored hand motion), system32may be adjusted so that area44′ is moved to illuminate target38in its new location.

If desired, area44′ may be divided into multiple subareas. As shown inFIG.9, for example, area44′ may cover two targets38when targets38are located adjacent to each other. In the event that targets38are separated from each other (e.g., if the user's hands are moved apart), system32can be adjusted to produce two separate beams of light36for two respective separate areas. As shown inFIG.10, for example, first area44′-1may be provided with a first beam of light36to illuminate a first of targets38and second area44′-2may be provided with a second beam of light36to illuminate a second of targets38. System32may, if desired, use multiple light-emitting devices such as multiple infrared light-emitting diodes and multiple corresponding light steering systems to create multiple discrete (unconnected and nonoverlapping) illumination areas that can independently track each of a user's two hands.

FIG.11is a diagram of an illustrative adjustable illumination system for illuminating a target38with light36. As shown inFIG.11, system32may include a light source16having one or more individual light-emitting devices70. Devices70may be individually adjustable light-emitting diodes or lasers (as example). During operation, light source16is turned on to produce light36. Light36may be emitted from system32through lens54. Lens54may include one or more lens elements that collimate and/or otherwise shape the emitted light36from source16and thereby produce one or more desired beams of light. In the example ofFIG.11, light is being emitted in direction60and, as illustrated by dashed lines58is characterized by an angular spread A. In proximity of target38this creates an illumination area44′ that overlaps target38and thereby illuminates target38. This ensures that target38will be visible to sensor34(FIG.5), even in low ambient lighting conditions.

To steer and shape the beam of light36emitted from system32, system32may adjust which of devices70are active and may use one or more positioners such as positioners50and52to adjust the positions of light source16and lens54. If desired, a given system32may have multiple light sources16, multiple positioners50, multiple corresponding lenses54, and multiple positioners52. Configurations in which system32has a single light source16, a single positioner50, a single lens54(which may be a single-element or multi-element lens), and a single positioner52may be described as an example.

Positioners50and52may be electromechanical actuators based on piezoelectric elements, motors, solenoids, and/or other adjustable actuators. Positioners50and52may, in general, make position adjustments over six degrees of freedom (e.g., each positioner may be capable of translating its corresponding component along one or more of three linear dimensions X, Y, and Z and may be capable of tilting its corresponding component about each of these three linear dimensions). Arrangements in which positioners50and52are only capable of making a subset of these position adjustments (e.g., a subset of the lateral motions along X, Y, and Z and a subset of the angular rotations about X, Y, and Z) may also be used.

Light source16and lens54may be mounted on gimbal mounts or other mounts that allow them to rotate and/or translate under control of positioners50and52, respectively.FIG.12shows how component62(which may be light source16and/or lens54) may be rotated (tilted) in direction64about an axis to new (tilted) position62′ by positioner50or positioner52. This type of operation may be used on source16and/or lens54to change (steer) the direction in which light36is emitted from system32.FIG.13shows how component62may be translated in direction66to new position62′ using positioner50or positioner52. This type of translation may, as an example, be used to move source16and lens54closer to each other or farther apart from each other (e.g., to adjust the angular spread of the output light). If desired, an accelerometer or other sensor in device10can detect when device10is in free-fall (zero gravity conditions) and can therefore quickly move source16and lens54apart to avoid undesired and potentially damaging contact between source16and lens54when device10contacts the ground. In theFIG.14example, component62is being shifted (translated) in direction68by positioner50or52to new position62′. This type of movement may steer the output beam direction.

FIG.15shows how a subset of one or more devices (light-emitting elements)70may be turned on while remaining devices70are turned off. By selective activation of desired devices70, the size, direction, and shape of the output beam may be adjusted. There may be N devices70in source16and devices70may be arranged in a one-dimensional or two-dimensional array. The value of N may be at least 1, at least 2, at least 4, at least 16, at least 25, at least 36, fewer than 100, fewer than 40, and/or other suitable value. In the example ofFIG.15, only device70-1is initially turned on while devices70-2,70-3, and70-4are turned off. As illustrated by arrow72, the pattern of activated devices70may be adjusted when it is desired to adjust the pattern of illumination produced by system32. In theFIG.15arrangement, device70-1is deactivated and devices70-3and70-4remain inactive while device70-2is turned on. Because device70-2is now producing output light rather than device70-1, the position from which light is emitted from devices70towards lens54is altered and the resulting direction of output light from system32is therefore altered. Beam size may be altered by increasing and decreasing the number of devices70that are active. For example, light36will cover a wider area when all of devices70are turned on than when only a single of devices70is turned on. Techniques in which light beam shaping (steering) using selective activation of devices70in system32may be used in combination with use of positioner50and/or positioner52to move source16and/or lens54or may be used separately (e.g., positioners50and/or52may be omitted).

Although sometimes described in connection with producing supplemental infrared illumination for hand tracking in a head-mounted device, adjustable illumination system32may be used in other contexts, as described in connection withFIGS.3and4. For example, in a laptop computer configuration (FIG.3) or cellular telephone, tablet, or wristwatch configuration (FIG.4), light36may be produced to help illuminate a user's face (or entire body) during a video conference. In this scenario, light36may include infrared and/or visible light. In any type of device10, device10may use a display to present infrared images for user viewing that have been captured with one or more infrared image sensors. In these arrangements, light36may be used to illuminate some or all of the area covered by the infrared image sensors (e.g., device10may be used for infrared photography or to support operation of device10in a night-vision mode). In some arrangements, system32may provide adjustable visible illumination while sensor34is collecting still and/or video images (e.g., to ensure a target is visible, to achieve desired artistic effects, etc.). Steerable light beams produced by system32may also be used in a gaze tracking system for device10(e.g., a head-mounted device or other device10). In particular, system32may steer a narrow beam of infrared light to various glint locations on a user's eye surface while sensor34detects and analyzes these glint locations to determine the direction in which the user's eyes are pointed (sometimes referred to as the user's point of gaze or gaze direction). One or more adjustable illumination systems32and sensors34may be used in each device10.

To help protect the privacy of users, any personal user information that is gathered by device10may be handled using best practices. These best practices including meeting or exceeding any privacy regulations that are applicable. Opt-in and opt-out options and/or other options may be provided that allow users to control usage of their personal data.