Patent ID: 12198276

DETAILED DESCRIPTION

Electronic devices such as head-mounted devices and other devices may be used for virtual reality and mixed reality (augmented reality) systems. These devices may include portable consumer electronics (e.g., portable electronic devices such as cellular telephones, tablet computers, head-mounted device such as googles, glasses, helmets, hats, etc. and/or other wearable equipment), head-up displays in cockpits, vehicles, and other systems, and display-based equipment (projectors, televisions, etc.). Device configurations in which virtual reality and/or mixed reality content is provided to a user with a head-mounted display device are described herein as an example. This is, however, merely illustrative. Any suitable equipment may be used in providing a user with virtual reality and/or mixed reality content.

A head-mounted device such as a pair of mixed reality glasses that is worn on the head of a user may have a camera such as an outwardly facing camera. During operation, the camera can capture a moving image of the real-world environment surrounding a user so that control circuitry in the electronic device can display the real-world content for the user. The user may also be provided a user with computer-generated content (sometimes referred to as virtual content).

The head-mounted device may operate in a virtual reality mode in which virtual reality (computer-generated) content is displayed for a user. In this mode of operation, real-world content captured by the camera is not displayed. The head-mounted device may also operate in a mixed reality mode. In the mixed reality mode, virtual content (sometimes referred to as non-real-world content) is overlaid on the real-world content captured by the camera. The virtual content may, for example, be text, graphics, moving images, and/or other content that is displayed over portions of the real-world content that is displayed.

To ensure that content is displayed satisfactorily, display and camera settings may be adjusted dynamically when transitioning between virtual reality and mixed reality modes. A camera can captured images at a camera frame rate (sometimes referred to as an image capture frame rate) and a display can display content (e.g., a moving image) at a display frame rate. When transitioning between virtual and mixed reality modes, the camera and display frame rates can be adjusted dynamically and operations such as camera reconfiguration operations may be performed at times that minimize disruptions to system operation.

A schematic diagram of an illustrative electronic device of the type that may adjust camera and display operation when transitioning between virtual reality and mixed reality modes is shown inFIG.1. As shown inFIG.1, electronic device10(e.g., a head-mounted device) may have control circuitry50. Control circuitry50may include storage and processing circuitry for controlling the operation of head-mounted display10. Circuitry50may include storage such as hard disk drive storage, nonvolatile memory (e.g., electrically-programmable-read-only memory configured to form a solid state drive), volatile memory (e.g., static or dynamic random-access-memory), etc. Processing circuitry in control circuitry50may be based on one or more microprocessors, microcontrollers, digital signal processors, baseband processors, power management units, audio chips, graphics processing units, application specific integrated circuits, and other integrated circuits. Software code may be stored on storage in circuitry50and run on processing circuitry in circuitry50to implement operations for electronic device10(e.g., controlling a camera during image operations, controlling sensors and other components during other data gathering operations, controlling displays and other components, performing image rendering operations and other operations involved in generating computer-generated content to be displayed for a user in virtual reality and/or mixed reality operating modes, etc.).

Electronic device10may include input-output circuitry52. Input-output circuitry52may be used to allow virtual-reality content and other data to be received by device10from external equipment (e.g., a tethered computer, a portable device such as a handheld device or laptop computer, or other electrical equipment) and to allow a user to provide device10with user input. Input-output circuitry52may also be used to gather information on the environment in which electronic device10is operating. Output components in circuitry52may allow electronic device10to provide a user with output and may be used to communicate with external electrical equipment.

As shown inFIG.1, input-output circuitry52may include one or more displays such as display26. Display26may be used to display images for a user of device10. Display26has a pixel array to generate images that are presented to a user (e.g., through a lens). Display26may be an organic light-emitting diode display, a display having a pixel array formed from crystalline semiconductor light-emitting diode dies, a liquid crystal display, a liquid-crystal-on-silicon display, an electrophoretic display, a microelectromechanical systems display, and/or other suitable display.

Input-output circuitry52may have one or more cameras such as camera100. Camera100may be an outwardly facing camera that captures real-world content (a moving image of the environment surrounding the user).

Sensors and other components70in input-output circuitry may include sensors such as ambient light sensors that measure ambient light intensity and/or ambient light color, force sensors, temperature sensors, touch sensors, capacitive proximity sensors, light-based proximity sensors, other proximity sensors, strain gauges, gas sensors, pressure sensors, moisture sensors, magnetic sensors, position and motion sensors (e.g., compasses, gyroscopes, accelerometers, and/or other devices for monitoring the location, orientation, and movement of device10), microphones, and other input-output devices such as buttons, keyboard keys, haptic output devices, speakers, etc.

Input-output circuitry52may include communications circuitry74that allows electronic device10(e.g., control circuitry50) to communicate with external equipment (e.g., remote controls, joysticks and other input controllers, portable electronic devices, computers, displays, etc.) and that allows signals to be conveyed between components (circuitry) at different locations in device10. Communications circuitry74may include wired and/or wireless communications circuitry (e.g., antennas and radio-frequency transceiver circuitry operating in cellular telephone bands, wireless local area network bands, etc.).

The components of electronic device10may be supported by a head-mountable support structure such as illustrative support structure16ofFIG.2. Support structure16, which may sometimes be referred to as a housing, may be configured to form a frame of a pair of glasses (e.g., left and right temples and other frame members), may be configured to form a helmet, may be configured to form a pair of goggles, or may have other head-mountable configurations.

One or more cameras such as camera100may capture real-world content (e.g., images of external objects in the user's environment such as an image of external object30ofFIG.2). Display system102may include display26and lens104for displaying images in direction92for viewing by a user such as viewer90.

During operation of device10, it may sometimes be desirable to operate camera100and display26in a beam chasing mode. As shown inFIG.3, camera100may have an array of image sensor pixels (e.g., rows and columns of sensor pixels106in a digital image sensor integrated circuit). Images may be captured by scanning through the rows of image sensor pixels106in direction108. Images may be captured in frames using this arrangement. The rate at which frames is captured is sometimes referred to at the frame rate. As shown inFIG.4, display26may have an array of display pixels (e.g., rows and columns of display pixels112in a pixel array for display26). During operation of display26, control signals for rows of display pixels112may be asserted in sequence (e.g., so that images are displayed by scanning through the rows of display pixels112in direction110). Images may be displayed in frames. The rate at which frames are refreshed on display26is sometimes referred to as the display frame rate. In beam chasing mode, it is not necessary to wait until all rows of image sensor pixels106have been scanned before displaying captured image sensor data on corresponding rows of display pixels112, thereby lowering display latency.

Display quality can also be enhanced by operating display26in a low persistence mode. In the low persistence mode, output light from pixels112is displayed for a fraction (e.g., 25%, at least 20%, less than 30%, etc.) of the total frame duration for display26to reduce motion blur effects. As shown in the example ofFIG.5, output light pulse duration TW is a fraction of frame duration TP. In the example ofFIG.5, the frame rate of display26is 120 Hz. In the example ofFIG.6, the frame rate of display26is 96 Hz, so frame duration TP is lengthened relative to frame duration TP ofFIG.5and output light pulse duration TW is lengthened relative to output light pulse duration TW ofFIG.5.

In certain lighting environments such as florescent lighting environments, images captured with certain frame rates may flicker. For example, in certain countries, florescent lighting is driven with 50 Hz alternating-current signals. Images captured with a camera frame rate of 120 Hz while a scene is illuminated with this florescent lighting tend to exhibit flicker. The flicker can be reduced or eliminated by operating the camera at 96 Hz (e.g., so that each 10.4 ms period of the 96 Hz frame rate is occupied by a 0.4 ms buffer time and a 10 ms camera exposure time that reduces flicker). Other camera frame rates less than 100 Hz may also be used. These camera frame rates may be used when capturing real-world content while operating in mixed reality mode. In the mixed reality mode, display26can be operated with a corresponding display frame rate (refresh rate) of 96 Hz to implement beam chasing.

When it is desired to operate in virtual reality mode, camera100can be used at a low camera frame rate (e.g., 30 Hz). The 96 Hz frame rate that is used to capture real-world content to display on display26is not needed during mixed reality mode. Rather, camera100can use the 30 Hz camera frame rate to detect external object location and motion. Detected external objects in this mode may be rendered in a virtual world, but because camera100is being used to detect object location and motion and not to capture high quality images for displaying to a user, it is not necessary to use the 96 Hz frame rate. Computer generated content (e.g., virtual reality content) can be displayed at a relatively high display frame rate (e.g., 120 Hz or other integral multiple of the 30 Hz camera frame rate) to minimize motion blur.

There are therefore two different sets of camera and display settings for device10. In virtual reality mode, display126can be operated with a first display frame rate (e.g., 120 Hz to minimize motion blur) and camera100can be operated with a first camera frame rate (e.g., 30 Hz). In this configuration, the 120 Hz display frame rate is an integral multiple (4) of the 30 Hz camera frame rate, so beam chasing operations may be performed). In the mixed reality mode, camera100can be operated at a second camera frame rate (e.g., 96 Hz with a 10 ms exposure time and a 0.4 ms buffer time) to help reduce flickering from florescent lighting and display26can be operated at a second display frame rate (e.g., a 96 Hz rate that is equal to the second camera frame rate to allow beam chasing operations to be performed).

During operation of device10, device10may transition between virtual reality mode and mixed reality mode (and vice versa). As an example, a user may wish to toggle between these modes and may press a button or otherwise supply input that directs device10to switch modes. As another example, camera100or other sensors in device10may detect that a person is walking into the field of view of camera100and may automatically switch between virtual reality mode (in which the user cannot view the surrounding environment) and mixed reality mode (in which the person walking into the field of view of camera100is visible).

As described in connection withFIGS.5and6, abrupt transitions between different display frame rates may result in undesired visibly noticeable display luminance variations. Accordingly, when transitioning between virtual reality mode and mixed reality mode (e.g., from virtual reality mode to mixed reality mode or vice versa), a gradual display frame rate transition may be made. As an example, the frame period associated with the frame rate may be varied by less than 0.1 ms per frame (e.g., the frame period increment rate or decrement rate may be less than 0.1 ms per frame or other suitable value).

The use of gradual transitions in frame rate when switching between virtual reality mode and mixed reality mode is shown inFIG.7. Curve200corresponds to the display frame rate of display26. During the time period before time t1, device10is operating in virtual reality mode. Accordingly, the display frame rate is 120 Hz. At time t1, device10transitions between virtual reality (VR) mode and mixed reality (MR) mode. There is a gradual change in frame rate between time t1(when the frame rate begins to decrease) and time t3(when the frame rate for display26has been reduced to 96 Hz from its initial value of 120 Hz). Similarly, when exiting mixed reality mode, there is a gradual change in frame rate between time t4(when the frame rate begins to increase) and time t6(when the frame rate for display26has been increased to 120 Hz).

Camera reconfiguration operations may also be performed in a way that helps minimize visual disruptions for a user. A camera reconfiguration operation takes place when control circuitry50adjusts camera26(e.g., to provide camera26with a new frame rate and/or additional settings such as a frame rate adjustment rate and time period). During the camera reconfiguration operation, a camera that has been directed to change its frame rate settings reconfigures itself for operation at the new settings. There is typically a 100 ms delay during camera reconfiguration. During this time, camera100does not capture images.

The transition time associated with adjusting frame rate between virtual reality mode and mixed reality mode (the time period between times t1and t3in the example ofFIG.7) may take 200 ms (as an example). To avoid visual disruptions that might otherwise arise from waiting to reconfigure camera26only after this transition time is complete, camera reconfiguration may be performed immediately upon exiting virtual reality mode (e.g., at time t1ofFIG.7). For example, camera100can be reconfigured between times t1and t2ofFIG.7. This avoids situations in which a person or other external object entering the field of view of camera suddenly pops into view on display26.

Following camera reconfiguration at time t2, display frame rate transitioning may not be complete. To avoid latency issues that might otherwise arise from lack of synchronization between the camera frame rate and the display frame rate, control circuitry50can direct camera26to adjust its frame rate to synchronize the camera and display frame rates during the transition period. Curve202ofFIG.7corresponds to the camera frame rate for camera100. At times before time t1, the camera frame rate is set to 30 Hz for virtual reality mode operation. During the frame rate transition period (e.g., between times t1and t3or at least during times between time t2and t3as shown in the example ofFIG.7), camera frame rate202is synchronized to display frame rate200to avoid latency issues.

Between time t3and t4in the example ofFIG.7, device10operates normally in mixed reality mode. Upon exiting mixed reality mode at time t4, camera100can be reconfigured (times t4to t5) and the camera frame rate reduced to 30 Hz (at time t5) to conserve power. No camera content is being displayed in the virtual reality mode, so it is not necessary to synchronize the camera and display frame rates during the display frame rate transition taking place between times t4to t6.

The foregoing is merely illustrative and various modifications can be made to the described embodiments. The foregoing embodiments may be implemented individually or in any combination.