REMOTE DEVICE INPUT ENTRY

Various implementations disclosed herein include devices, systems, and methods that provide an improved remote device user interface for enabling accurate text input into a user device. For example, a user device (e.g., an HMD) may be enabled to receive touchscreen input via input controls displayed by a remote device (e.g., a mobile phone, a tablet computer, etc.) using a display mode enhanced for pass-through viewing. For example, an enhanced display mode may enable a larger than normal display to enhance display mode attributes associated with input entry with respect to the user device. In some implementations display mode attributes may be enabled based on context such as distance, lens distortion, lighting, minimum character resolution, etc. Likewise, some implementations may augment input controls (on an HMD) such as 3 dimensional bubbles, auto-correct controls, auto-fill controls, spatialized audio controls, virtual touch/function bar controls, etc.

TECHNICAL FIELD

The present disclosure generally relates to systems, methods, and devices such as head-mounted devices (HMDs) that receive touchscreen input via input controls displayed by remote devices.

BACKGROUND

To enable user input to devices such as HMDs, it may be desirable to enable a user to provide input via a separate device, such as a touch screen of a separate mobile device. However, existing systems may not adequately enable such input, for example, because such systems may not provide adequate views of the user interfaces provided by such separate devices, e.g., pass-through video of such user interfaces may lack resolution, clarity, or other display capabilities sufficient for users to see or use the displayed user interface elements of those separate devices.

SUMMARY

Various implementations disclosed herein include devices, systems, and methods that provide improved remote device input entry for devices such as HMDs. For example, a user device (e.g., an HMD) may be enabled to receive touchscreen input via input controls displayed by a remote device (e.g., a mobile phone, a tablet computer, etc.), where the remote device uses a display mode enhanced for pass-through viewing on the user device. An enhanced display mode may utilize the remote device's standard input features (e.g., standard touch-screen keyboard) enhanced with selected display mode attributes (e.g., enhanced element sizing, contrast, brightness, etc.) to facilitate improved visibility or easier use by a user viewing that user interface via the user's device, e.g., via pass-through video on an HMD. For example, a mobile device's standard touch-based keyboard may be altered to have attributes enhanced for viewing and use by a user viewing that user interface via an HMD's pass-through video. In some implementations, display mode attributes may be selected based on context such as distance, lens distortion, lighting, minimum character resolution, etc.

Some implementations may augment the view of the remote device's input controls by providing associated user interface content, e.g., three dimensional (3D) bubbles, auto-correct controls, auto-fill controls, spatialized audio controls, virtual touch/function bar controls, etc., located proximate or otherwise presented in a way indicative of an association with the viewed remote device user interface. For example, an HMD may display a view that includes both (a) a depiction of a mobile device including a keyboard displayed on the mobile device's touch screen and (b) a virtual function key bar located just above that displayed remote device such that the HMD user can interact with the mobile device-displayed keyboard or the HMD-displayed virtual function key bar, or both to provide input that is received and used by the HMD.

In some implementations, a first electronic device has a first display and a processor (e.g., one or more processors) that executes instructions stored in a non-transitory computer-readable medium to perform a method. The method performs one or more steps or processes. In some implementations, the first electronic device displays a user interface within a view of a three-dimensional (3D) environment via the first display. The view of the 3D environment may include or be based upon pass-through video of a physical environment. The first electronic device determines an opportunity for a second electronic device to receive input for the user interface via one or more input controls displayed on a second display of the second electronic device. In accordance with determining the opportunity, the first electronic device transmits a notification to the second electronic device. The notification indicates (explicitly or implicitly) a first display mode for displaying the one or more input controls. The second electronic device displays the one or more input controls on the second display based on the first display mode and the one or more input controls displayed on the second display are visible in the view of the 3D environment displayed via the first display, e.g., in the view provided that includes or is based upon pass-through video of the physical environment. The second electronic device receives the input via the one or more input controls and data corresponding to the input is provided to and received at the first electronic device, which may respond accordingly.

DESCRIPTION

FIG.1illustrates a block diagram view of an example physical environment100including an HMD105and a mobile device110. The mobile device110may include any type of mobile device including, inter alia, a smart phone, a tablet computer, etc. The HMD105is configured to provide pass-through video views of the physical environment100, including video that includes image frames having depictions of mobile device110, input controls, and content displayed thereon. In this example, physical environment100also includes a desk120and a monitor122.

The HMD105is configured to receive touchscreen input via input controls displayed by the mobile device110by enabling a display mode enhanced for pass-through viewing. As non-limiting examples, a keyboard, a keypad, a color-picker, an emoji-picker, a slider bar, a timeline, or a signature block input may be displayed using a larger than normal display mode to improve visibility of the input controls while wearing the HMD105. Display mode attributes may be enabled based on context such as, inter alia, device distance, lens distortion, lighting attributes, a minimum character resolution, etc. The input controls displayed on the HMD105may be augmented with the following exemplary and non-limiting augmentations: 3D information bubbles, auto-correct/auto-fill augmentations, spatialized audio augmentations, a virtual touch/function bar augmentation, etc. A display of the mobile device may be enabled/triggered based on context such as, inter alia, an open text entry window, a user gazing at the mobile device110, a distance of the mobile device110from the user105or the HMD105, etc.

The HMD105and/or mobile device110may include one or more cameras, microphones, depth sensors, or other sensors that can be used to capture information about and evaluate the physical environments100and the objects therein, as well as information about a user102. The HMD105and mobile device110may use information about the physical environment100or user102that obtained from the sensors to provide visual and audio content.

In some implementations, the HMD105is configured to present views that it generates to the user102, including views that may be based on the physical environment100. According to some implementations, the HMD105generates and presents views of an extended reality (XR) environment. In some implementations, the HMD105may be replaced with a handheld electronic device (e.g., a smartphone or a tablet). The HMD105may include one or more displays provided to display content, e.g., one display for each eye. The HMD105may enclose the field-of-view of the user102. In some implementations, the functionalities of HMD105are provided by more than one device. In some implementations, the HMD105communicates with a separate controller or server to manage and coordinate an experience for the user. Such a controller or server may be local or remote relative to the physical environment100.

A device, such as HMD105, may have a visual acuity that is less than a user's acuity. Thus, for example, while the user102may be able to see with 20/20 acuity, the device may have a 20/35 acuity in its central region at the peak focal distance. Thus, a user's view of pass-through video may be less clear or otherwise different than the view the user102would see viewing the surrounding physical environment without the HMD105. This difference can be significant when viewing the physical environment100via the device, e.g., as pass-through video on HMD105, especially with respect to input controls (e.g., a keyboard, a keypad, etc.) for which clarity improves readability, understandability, or other aspects associated with entering accurate input data. For example, user102may find it difficult to enter text using a default text-entry virtual keyboard displayed on mobile device110in pass-through video provided by HMD105. For example, when holding the mobile device110at a typical distance/position for reading, e.g., at the distance the user102would use when not wearing HMD105, the default input control sizes (e.g., a size of alpha numeric keys of a keyboard) may be illegible. This might encourage the user to hold the mobile device110closer than usual to their eyes, which may be uncomfortable or undesirable.

In some implementations, the mobile device110provides a display mode for modifying the input controls without necessarily requiring the user102to hold the mobile device110closer than normal. The mobile device110may do so, for example, by changing the appearance of its input controls, e.g., increasing a keyboard size (e.g., to cover more of or the entire display area of the mobile device110, etc.), changing input control color, changing input control contrast, bolding the input controls, adjusting transparency of the input controls, enabling accessibility features, etc. The altered input controls may be configured to provide more accurate viewing when viewed via pass-through video on HMD105. In some implementations, the altered input controls are displayed initially, e.g., as soon as the input controls are to be first displayed on the mobile device110, and thus are not initially displayed on the mobile device110in their “original” unaltered form. In some implementations, the input controls on the mobile device110are altered (e.g., from its original to its altered form) and thus the original form of the input controls may be transitioned to the altered form. Such a transition may occur based on determining that the content is being viewed via an HMD's pass-through video or other criteria.

FIG.2illustrates a view for modifying input control attributes on the mobile device110to improve a pass-through video view provided by the HMD105(ofFIG.1). In this example, a view200ais a view of pass-through video of physical environment100provided by HMD105. The view200aincludes a depiction210of the mobile device110, a depiction212aof input controls being displayed on the mobile device110, and a depiction220of the desk120.FIG.2illustrates the depiction212aof the input controls (e.g., a keyboard) as being relatively small and thus being too small to read or use given the acuity provided by the HMD105. In contrast, the view200bis a view of pass-through video of physical environment100provided by HMD105where the input controls being displayed are using an enhanced display mode by device110. The view200bincludes a depiction210of the mobile device110, a depiction212bof the input controls being displayed on the mobile device110, and a depiction220of the desk120.FIG.2shows the depiction212bof the input controls as being relatively larger and thus enabling accurate input entry.

In some implementations, an enhanced display mode is triggered manually, e.g., based on user input. In some implementations, such an enhanced display mode is triggered automatically based on one or more criteria. For example, an enhanced display mode may be automatically triggered based on detecting that the user102is looking at a depiction of the mobile device110in a view provided by the HMD105. The HMD105, in this example, may track the user's gaze using eye-facing cameras or other sensors to determine when a user's gaze direction is directed towards such a depiction. Gaze data may be interpreted to distinguish non-intentional gazes (e.g., non-intentional saccades, etc.) from gazes made intentionally at the mobile device110or the content thereon. In this example, the HMD105may determine that the user is looking at the mobile device110or its content and send a message or request to the mobile device110to initiate or otherwise cause the mobile device110to switch its content display to the enhanced display mode. In some implementations, the HMD105may determine that the device110is within a field of vision of HMD105and send a message or request to the mobile device110to initiate or otherwise cause the mobile device110to switch its content display to the enhanced display mode.

In some implementations, an enhanced display mode for an input element (e.g., a virtual keyboard on a mobile device) is triggered based on determining that the HMD105is in a state associated with receiving input of a corresponding type (e.g., a text dialog box is currently open, active, etc. in the view provided by the HMD105). In one example, when the HMD105detects that a new text-receiving element has been added to the HMD's view (e.g., the user has opened a text-entry window, etc. on the HMD's user interface), the HMD105determines whether there are any remote devices available to facilitate the corresponding input type and, if so, initiates a notification to such devices regarding receiving input via an input element provided by such a remote device110. The notification may indicate that an enhanced display mode should be used to present the input element, e.g., that a virtual keyboard should be displayed with enhanced size, enhanced brightness, or other enhanced characteristics. The remote device110may automatically initiate the display of the input element according to the enhanced display mode or provide an option for the user to initiate the input element (in the enhanced display mode).

In another example, an enhanced display mode is automatically triggered additionally, or alternatively, based upon one or both of the mobile device110and HMD105being within a threshold distance of one another. Sensors on either device may determine such distances, e.g., using depth sensor measurements, interpreting image data, detecting wireless signal strength, or using other sensor data. For example, if the HMD105determines that the mobile device110is within the threshold it may send a message to the mobile device110notifying the mobile device110of this circumstance.

Similarly, the enhanced display mode may automatically be triggered additionally, or alternatively, based on determining that a user102is both wearing the HMD105and interacting with the mobile device110at the same time, e.g., by providing touch input on a touch surface of the mobile device110. For example, the HMD105may use images or other sensor data to determine that a hand of a user of the HMD is interacting with another device, identify that device, and send a communication to that device (e.g., mobile device110) of this circumstance.

The optimized display mode may automatically be triggered additionally, or alternatively, based on determining that the respective orientations of the mobile device110and HMD105are facing one another or are otherwise oriented in a way that is indicative of the HMD user being able to see or use the input controls of the mobile device110. Such orientations may be determined based on sensor data of one or both devices, e.g., image data from cameras, motion tracking data from accelerometers, gyroscopes, etc., depth data from depth sensors, etc. Such orientations may be determined based on information provided using ultrawide band radios. In some implementations, inter-device communication are used to confirm orientation. For example, the HMD105may determine that there is a device oriented toward it a few feet in front of the HMD105. The HMD may ping one or more nearby devices with which it has an association (e.g., same user, same account, devices of known contacts, etc.) and ask those one or more nearby devices to confirm which of those devices (if any) is the one that is oriented towards it or whether the user of the HMD105is looking at any of those devices. The mobile device110may confirm its orientation or that the user of the HMD105is looking at it. In one example, the mobile device110has an infrared camera that periodically (e.g., every 18 seconds) captures data to determine whether a user is looking at it for power saving purposes (e.g., to go into sleep mode if the user is not looking at it) and this sensing capability is additionally or alternatively used to determine if and when a user of the HMD105is looking at the mobile device110.

The triggering of an enhanced display mode may be based on determining that there are one or more persons who are in the physical environment100other than the user of the HMD105. For example, the enhanced display mode may only be triggered if both devices (e.g., HMD105and tablet mobile device110) are associated with the same person, account, or organization, and that the user is the only user in the physical environment, within a threshold distance (e.g., 5 feet of the mobile device110), etc. In another example, the enhanced display mode may additionally or alternatively be triggered based on determining that the user102of HMD105is authorized to view the content being displayed by mobile device110. In another example, the enhanced display mode (e.g., larger input controls) may not be triggered to ensure privacy in certain situations, e.g., when there are other people nearby that may see the input entry.

It may be desirable to only trigger the enhanced display mode in circumstances based on determining a type of environment or particular location. For example, the enhanced display mode may only be triggered if the physical environment100is a private (e.g., non-public) environment such as the user's home, office, etc. In another example, the enhanced display mode (e.g., larger input controls) may not be triggered based on determining that the physical environment100is a public elevator, a plane, a bus, a train, a hotel lobby, a coffee shop, etc.

In some implementations, the HMD105and tablet device110communicate with one another to share information that facilitates the switching of the mobile device110into an enhanced display mode for input in certain circumstances. Such communications may be facilitated based on the devices being part of the same ecosystem/brand of devices and/or being associated with a same user account. In some implementations, the inter-device communications share acuity-related information. For example, the HMD105may notify the mobile device110its acuity is 20:35 or request that the tablet device switch to an enhanced display mode in certain circumstances. The HMD105may share information with the mobile device110indicating that a possible circumstance for an enhanced display mode is occurring and that the mobile device110should switch to the comfortable viewing mode.

In some implementations, the mobile device110is configured to determine whether to switch to an enhanced display mode for input based on the device that is providing a pass-through view of its content and the characteristics (e.g., acuity) of that pass-through device. Thus, for example, the HMD105may send its acuity to the mobile device110and then the mobile device110may switch to the enhanced display mode whenever the mobile device110further detects appropriate criteria are satisfied, e.g., when one of the devices determines the devices are within a threshold distance, the HMD105determines that the user102is looking at the mobile device110, either device determines that the devices are orientated towards one another, either device determines that the view of the input controls would be unlikely to be accurately viewed in the current circumstances, etc.

FIG.3is an example physical environment300in which a watch device310and an HMD305may operate. The HMD305is configured to provide pass-through video views of the physical environment300, including video that includes depictions of watch device310and content displayed thereon. In this example, physical environment300also includes a desk320and a monitor322.

The HMD305and/or watch device310may include one or more cameras, microphones, depth sensors, or other sensors that can be used to capture information about and evaluate the physical environments300and the objects therein, as well as information about the user302. The HMD305and watch device310may use information about the physical environment300or user302that is obtained from the sensors to provide visual and audio content.

In some implementations, the HMD305is configured to present views that it generates to the user302, including views that may be based on the physical environment300. According to some implementations, the HMD305generates and presents views of an extended reality (XR) environment. In some implementations, the HMD305is instead a handheld electronic device (e.g., a smartphone or a tablet). The HMD305may include one or more displays provided to display content, e.g., a display for each eye. The HMD305may enclose the field-of-view of the user302. In some implementations, the functionalities of HMD305are provided by more than one device. In some implementations, the HMD305communicates with a separate controller or server to manage and coordinate an experience for the user. Such a controller or server may be local or remote relative to the physical environment300.

In some implementations, the watch device310provides an enhanced display mode by changing the appearance of its contents, e.g., increasing a keyboard size (e.g., to cover an entire display area of the watch device310, etc.). The altered content provides more enhanced/customized viewing when viewed via pass-through video on HMD305.

FIG.4illustrates a graphical view for modifying input control attributes on the watch device310(ofFIG.3) to improve a pass-through video view provided by the HMD305. In this example, a view400ais a view of pass-through video of physical environment300provided by HMD305. The view400aincludes a depiction410of the watch device310, a depiction412aof input controls being displayed on the watch device310.FIG.4illustrates the depiction412aof the input controls (e.g., a keyboard) as being relatively small and thus being too small to read given the acuity provided by the HMD305. In contrast, the view400bis a view of pass-through video of physical environment300provided by HMD305where the input controls are being displayed using an enhanced display mode by watch device310. The view400bincludes a depiction410of the watch device310and a depiction412bof the input controls being displayed on the watch device310.FIG.4shows the depiction412bof the input controls as being relatively larger and thus enabling accurate input entry.

FIG.5illustrates an HMD view505aof the mobile device510with modified (e.g., increased in size) input controls512(of a mobile device510of a user502) and generated augmentations525and527provided by HMD505. The generated augmentations525comprise text entry field augmentations. The generated augmentations527comprise auto-fill augmentations. For example, user502initiates a typing sequence (presented to user502in text entry field augmentations (e.g., generated augmentations525) that include the letters “aut”. In response, generated augmentations527(e.g., comprising auto-fill term suggestions including the terms “auto”, “automobile”, and “automatic”) are presented to user502(for user selection) via a generated augmentation field. Alternatively, generated augmentations may include, inter alai, 3D information bubbles, auto-correct/auto-fill augmentations, spatialized audio augmentations, a virtual touch/function bar augmentation, etc.

FIG.6is a flowchart representation of an exemplary method600for modifying input controls from an HMD point of view. In some implementations, the method600is performed by a first electronic device (e.g., HMD105ofFIG.1) having a processor and a first display. In some implementations, the method600is performed by processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method600is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory). Each of the blocks in the method600may be enabled and executed in any order.

At block610, the method600displays a user interface within a view of a three-dimensional (3D) environment (e.g., an XR environment) via the first display. The 3D environment is generated based at least in part on pass-through video of a physical environment as described with respect toFIGS.2-5.

At block620, the method600determines an opportunity for a second electronic device (e.g., a mobile device) to receive input for the user interface via one or more input controls displayed on a second display of the second electronic device. Such an opportunity may be determined based on content displayed by the first device, e.g., whether there are input elements being displayed that require input of a type corresponding to an input element available via the second electronic device. Such an opportunity may be determined based on the proximity, orientations, or other attributes of the second electronic device or the environment.

At block630, in accordance with determining the opportunity (of block620), the method600transmits a notification to the second electronic device. The notification indicates (explicitly or implicitly) a first display mode (e.g., a pass-through mode, an enhanced mode, an enlarged input interface mode, etc.) for displaying the one or more input controls.

At block640, data corresponding to input received at the second electronic device is received. In response, the second electronic device displays the one or more input controls on the second display based on the first display mode and receives the input via the one or more input controls. The one or more input controls displayed on the second display are visible in the view of the 3D environment displayed via the first display as described with respect toFIGS.2-5.

In some implementations, the first display mode (e.g., a pass-through enhanced display mode) differs from a second display mode (e.g., normal/non-pass-through display mode) available on the second device. For example, the first display mode may correspond to viewing the second display via pass-through video and the second display mode may correspond to viewing the second display directly.

In some implementations, a position of the second electronic device is determined within in the 3D environment and associated augmented content is provided. The augmented content is associated with the one or more input controls in the view of the 3D environment based on the position of the second electronic device. Alternatively, a predetermined layout of the one or more input controls may be detected with respect to the second electronic device in the 3D environment and augmented content may be provided based on the predetermined layout as described with respect toFIG.5.

The augmented content may include a 3D representation corresponding to one of the input controls. For example, the augmented content may include a popup bubble of a specified character being enabled. As a first alternative the augmented content may include a 3D representation of auto-correct content, auto-fill content, or a virtual key touch bar. As a second alternative, the augmented content may include spatialized audio corresponding to input events on the second display.

In some implementations, determining the opportunity (of block620) may include determining that the user interface comprises content associated with an input modality such as, inter alia, a keyboard input modality. Alternatively, determining the opportunity may include determining a gaze corresponding to an element of the user interface or the second electronic device. The element of the user interface may be associated with an input modality. Likewise determining the opportunity may include determining a distance of the second electronic device relative to the first electronic device.

In some implementations, the first display mode may include a size attribute (e.g., large) for the one or more input controls. The size attribute may include a minimum size attribute based on a characteristic of the first electronic device. The characteristic may include a quality pass-through video or a lens distortion of the HMD. The first display mode may be determined based on a distance of the second electronic device relative to the first electronic device. Likewise, the first display mode may provide a minimum character size or minimum character resolution. Alternatively, the first display mode may be generated based on a lighting of the physical environment.

In some implementations, the second electronic device automatically displays the one or more input controls in response to the notification (transmitted at block630). Alternatively, the second electronic device may display a selectable element (e.g., a user may select to use mobile device as input) based on the notification. In response, the one or more input controls may be displayed based on selection of the element.

FIG.7is a flowchart representation of an exemplary method700for modifying input controls from a mobile device point of view. In some implementations, the method700is performed by a first electronic device (e.g., mobile device110ofFIG.1) having a processor and a first display. In some implementations, the method700is performed by processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method700is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory). Each of the blocks in the method700may be enabled and executed in any order.

At block710, the method700receives (from a second electronic device such as an HMD, a video streaming device, etc.) a notification to display an input interface. At block720, in accordance with a first determination that the second electronic device is a first type of device (and based on the notification), the input interface is displayed (via the first display device) with respect to a first visual characteristic (e.g., a size characteristic, a contrast characteristic, a color characteristic, a layout characteristic, etc.). At block730, in accordance with a second determination that the second electronic device is a second type of device differing from the first type of device (and based on the notification), a command is transmitted to the second electronic device. The command instructs the second electronic device to display (via a second display device of the second electronic device) a view of the input interface having a second visual characteristic (e.g., size, contrast, color, layout, etc.) differing from the first visual characteristic. In block740, input (e.g., text) is received via the input interface. In block750, data, corresponding to the input is transmitted to the second electronic device.

In some implementations, a first display mode (e.g., normal/non-pass-through display mode) available on the first electronic device differs from a second display mode (e.g., a pass-through enhanced display mode) available on the second electronic device. For example, the first display mode may correspond to viewing the second display directly and the second display mode may correspond to viewing the first display via pass-through video.

In some implementations, the second display mode may include a size attribute (e.g., large) for the one or more input controls. The size attribute may include a minimum size attribute based on a characteristic of the first electronic device. The second display mode may be determined based on a distance of the first electronic device relative to the second electronic device. Likewise, the second display mode may provide a minimum character size or minimum character resolution. Alternatively, the second display mode may be generated based on a lighting of the physical environment.

In some implementations, the first electronic device automatically displays the one or more input controls in response to the notification (of block710).

FIG.8is a block diagram of an example of a device800(e.g., HMD105and/or mobile device110ofFIG.1) in accordance with some implementations. While certain specific features are illustrated, those skilled in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, as a non-limiting example, in some implementations the device800includes one or more processing units802(e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, and/or the like), one or more input/output (I/O) devices and sensors806, one or more communication interfaces808(e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, and/or the like type interface), one or more programming (e.g., I/O) interfaces810, one or more AR/VR displays812, one or more interior and/or exterior facing image sensor systems814, a memory820, and one or more communication buses804for interconnecting these and various other components.

In some implementations, the one or more communication buses804include circuitry that interconnects and controls communications between system components. In some implementations, the one or more I/O devices and sensors806include at least one of an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, an ambient light sensor (ALS), one or more physiological sensors (e.g., blood pressure monitor, heart rate monitor, blood oxygen sensor, blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptics engine, one or more depth sensors (e.g., a structured light, a time-of-flight, or the like), and/or the like.

In some implementations, the one or more displays812are configured to present the experience to the user. In some implementations, the one or more displays812correspond to holographic, digital light processing (DLP), liquid-crystal display (LCD), liquid-crystal on silicon (LCoS), organic light-emitting field-effect transitory (OLET), organic light-emitting diode (OLED), surface-conduction electron-emitter display (SED), field-emission display (FED), quantum-dot light-emitting diode (QD-LED), micro-electromechanical system (MEMS), and/or the like display types. In some implementations, the one or more displays812correspond to diffractive, reflective, polarized, holographic, etc. waveguide displays. For example, the device800includes a single display. In another example, the device800includes a display for each eye of the user.

In some implementations, the one or more image sensor systems814are configured to obtain image data that corresponds to at least a portion of the physical environment100. For example, the one or more image sensor systems814include one or more RGB cameras (e.g., with a complimentary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), monochrome cameras, IR cameras, event-based cameras, and/or the like. In various implementations, the one or more image sensor systems814further include illumination sources that emit light, such as a flash. In various implementations, the one or more image sensor systems814further include an on-camera image signal processor (ISP) configured to execute a plurality of processing operations on the image data including at least a portion of the processes and techniques described herein.

The memory820includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some implementations, the memory820includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory820optionally includes one or more storage devices remotely located from the one or more processing units802. The memory820includes a non-transitory computer readable storage medium. In some implementations, the memory820or the non-transitory computer readable storage medium of the memory820stores the following programs, modules and data structures, or a subset thereof including an optional operating system830and one or more instruction set(s)840.

The operating system830includes procedures for handling various basic system services and for performing hardware dependent tasks. In some implementations, the instruction set(s)840are configured to manage and coordinate one or more experiences for one or more users (e.g., a single experience for one or more users, or multiple experiences for respective groups of one or more users). The instruction set(s)840include a display mode instruction set842configured with instructions executable by a processor to enable a display mode enhanced for pass-through viewing of input controls provided by another device as described herein. In some implementations, multiple devices (e.g., a mobile device and an HMD) are each configured with display mode adjustment instructions sets and communicate with one another to provide input controls (e.g., on the mobile device) and pass-through views of a physical environment including the displayed input controls (e.g., on the HMD) as described herein.

Although these elements are shown as residing on a single device (e.g., HMD105), it should be understood that in other implementations, any combination of the elements may be located in separate computing devices. Moreover,FIG.8is intended more as functional description of the various features which are present in a particular implementation as opposed to a structural schematic of the implementations described herein. As recognized by those of ordinary skill in the art, items shown separately could be combined and some items could be separated. For example, some functional modules (e.g., instruction set(s)840) shown separately inFIG.8could be implemented in a single module and the various functions of single functional blocks (e.g., instruction sets) could be implemented by one or more functional blocks in various implementations. The actual number of modules and the division of particular functions and how features are allocated among them will vary from one implementation to another and, in some implementations, depends in part on the particular combination of hardware, software, and/or firmware chosen for a particular implementation.

Numerous specific details are provided herein to afford those skilled in the art a thorough understanding of the claimed subject matter. However, the claimed subject matter may be practiced without these details. In other instances, methods, apparatuses, or systems, that would be known by one of ordinary skill, have not been described in detail so as not to obscure claimed subject matter.

The term “data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations, of the foregoing. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment can realize various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures. Unless specifically stated otherwise, it is appreciated that throughout this specification discussions utilizing the terms such as “processing,” “computing,” “calculating,” “determining,” and “identifying” or the like refer to actions or processes of a computing device, such as one or more computers or a similar electronic computing device or devices, that manipulate or transform data represented as physical electronic or magnetic quantities within memories, registers, or other information storage devices, transmission devices, or display devices of the computing platform.