Method And Device For Navigating Windows In 3D

In one implementation, a method for navigating windows in 3D. The method includes: displaying a first content pane with a first appearance at a first z-depth within an extended reality (XR) environment, wherein the first content pane includes first content and an input field; detecting a user input directed to the input field; and, in response to detecting the user input directed to the input field: moving the content first pane to a second z-depth within the XR environment, wherein the second z-depth is different from the first z-depth; modifying the first content pane by changing the first content pane from the first appearance to a second appearance; and displaying a second content pane with the first appearance at the first z-depth within the XR environment.

TECHNICAL FIELD

The present disclosure generally relates to navigating windows and, in particular, to systems, methods, and methods for navigating windows in 3D.

BACKGROUND

Current web browsers may use a tab arrangement for open web pages and also provide a means for viewing browsing history. This organization structure makes it difficult to concurrently view past and present web pages and/or searches.

SUMMARY

Various implementations disclosed herein include devices, systems, and methods for navigating windows in 3D. According to some implementations, the method is performed at a computing system including non-transitory memory and one or more processors, wherein the computing system is communicatively coupled to a display device and one or more input devices. The method includes: displaying a first content pane with a first appearance at a first z-depth within an extended reality (XR) environment, wherein the first content pane includes first content and an input field: detecting a user input directed to the input field: and, in response to detecting the user input: moving the first content pane to a second z-depth within the XR environment, wherein the second z-depth is different from the first z-depth; modifying the first content pane by changing the first content pane from the first appearance to a second appearance: and displaying a second content pane with the first appearance at the first z-depth within the XR environment.

In accordance with some implementations, a computing system includes one or more processors, non-transitory memory, an interface for communicating with a display device and one or more input devices, and one or more programs: the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors and the one or more programs include instructions for performing or causing performance of the operations of any of the methods described herein. In accordance with some implementations, a non-transitory computer readable storage medium has stored therein instructions which when executed by one or more processors of a computing system with an interface for communicating with a display device and one or more input devices, cause the computing system to perform or cause performance of the operations of any of the methods described herein. In accordance with some implementations, a computing system includes one or more processors, non-transitory memory, an interface for communicating with a display device and one or more input devices, and means for performing or causing performance of the operations of any of the methods described herein.

DESCRIPTION

People may sense or interact with a physical environment or world without using an electronic device. Physical features, such as a physical object or surface, may be included within a physical environment. For instance, a physical environment may correspond to a physical city having physical buildings, roads, and vehicles. People may directly sense or interact with a physical environment through various means, such as smell, sight, taste, hearing, and touch. This can be in contrast to an extended reality (XR) environment that may refer to a partially or wholly simulated environment that people may sense or interact with using an electronic device. The XR environment may include virtual reality (VR) content, mixed reality (MR) content, augmented reality (AR) content, or the like. Using an XR system, a portion of a person's physical motions, or representations thereof, may be tracked and, in response, properties of virtual objects in the XR environment may be changed in a way that complies with at least one law of nature. For example, the XR system may detect a user's head movement and adjust auditory and graphical content presented to the user in a way that simulates how sounds and views would change in a physical environment. In other examples, the XR system may detect movement of an electronic device (e.g., a laptop, tablet, mobile phone, or the like) presenting the XR environment. Accordingly, the XR system may adjust auditory and graphical content presented to the user in a way that simulates how sounds and views would change in a physical environment. In some instances, other inputs, such as a representation of physical motion (e.g., a voice command), may cause the XR system to adjust properties of graphical content.

Numerous types of electronic systems may allow a user to sense or interact with an XR environment. A non-exhaustive list of examples includes lenses having integrated display capability to be placed on a user's eyes (e.g., contact lenses), heads-up displays (HUDs), projection-based systems, head mountable systems, windows or windshields having integrated display technology, headphones/earphones, input systems with or without haptic feedback (e.g., handheld or wearable controllers), smartphones, tablets, desktop/laptop computers, and speaker arrays. Head mountable systems may include an opaque display and one or more speakers. Other head mountable systems may be configured to receive an opaque external display, such as that of a smartphone. Head mountable systems may capture images/video of the physical environment using one or more image sensors or capture audio of the physical environment using one or more microphones. Instead of an opaque display, some head mountable systems may include a transparent or translucent display. Transparent or translucent displays may direct light representative of images to a user's eyes through a medium, such as a hologram medium, optical waveguide, an optical combiner, optical reflector, other similar technologies, or combinations thereof. Various display technologies, such as liquid crystal on silicon, LEDs, μLEDs, OLEDs, laser scanning light source, digital light projection, or combinations thereof, may be used. In some examples, the transparent or translucent display may be selectively controlled to become opaque. Projection-based systems may utilize retinal projection technology that projects images onto a user's retina or may project virtual content into the physical environment, such as onto a physical surface or as a hologram.

FIG.1is a block diagram of an example operating architecture100in accordance with some implementations. While pertinent features are shown, those of ordinary skill in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity and so as not to obscure more pertinent aspects of the example implementations disclosed herein. To that end, as a non-limiting example, the operating architecture100includes an optional controller110and an electronic device120(e.g., a tablet, mobile phone, laptop, near-eye system, wearable computing device, or the like).

In some implementations, the controller110is configured to manage and coordinate an XR experience (sometimes also referred to herein as a “XR environment” or a “virtual environment” or a “graphical environment”) for a user150and optionally other users. In some implementations, the controller110includes a suitable combination of software, firmware, and/or hardware. The controller110is described in greater detail below with respect toFIG.2. In some implementations, the controller110is a computing device that is local or remote relative to the physical environment105. For example, the controller110is a local server located within the physical environment105. In another example, the controller110is a remote server located outside of the physical environment105(e.g., a cloud server, central server, etc.). In some implementations, the controller110is communicatively coupled with the electronic device120via one or more wired or wireless communication channels144(e.g., BLUETOOTH, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In some implementations, the functions of the controller110are provided by the electronic device120. As such, in some implementations, the components of the controller110are integrated into the electronic device120.

In some implementations, the electronic device120is configured to present audio and/or video (A/V) content to the user150. In some implementations, the electronic device120is configured to present a user interface (UI) and/or an XR environment128to the user150. In some implementations, the electronic device120includes a suitable combination of software, firmware, and/or hardware. The electronic device120is described in greater detail below with respect toFIG.3.

According to some implementations, the electronic device120presents an XR experience to the user150while the user150is physically present within a physical environment105that includes a table107within the field-of-view (FOV)111of the electronic device120. As such, in some implementations, the user150holds the electronic device120in his/her hand(s). In some implementations, while presenting the XR experience, the electronic device120is configured to present XR content (sometimes also referred to herein as “graphical content” or “virtual content”), including an XR cylinder109, and to enable video pass-through of the physical environment105(e.g., including the table107) on a display122. For example, the XR environment128, including the XR cylinder109, is volumetric or three-dimensional (3D).

In one example, the XR cylinder109corresponds to display-locked content such that the XR cylinder109remains displayed at the same location on the display122as the FOV111changes due to translational and/or rotational movement of the electronic device120. As another example, the XR cylinder109corresponds to world-locked content such that the XR cylinder109remains displayed at its origin location as the FOV111changes due to translational and/or rotational movement of the electronic device120. As such, in this example, if the FOV111does not include the origin location, the XR environment128will not include the XR cylinder109. For example, the electronic device120corresponds to a near-eye system, mobile phone, tablet, laptop, wearable computing device, or the like.

In some implementations, the display122corresponds to an additive display that enables optical see-through of the physical environment105including the table107. For example, the display122corresponds to a transparent lens, and the electronic device120corresponds to a pair of glasses worn by the user150. As such, in some implementations, the electronic device120presents a user interface by projecting the XR content (e.g., the XR cylinder109) onto the additive display, which is, in turn, overlaid on the physical environment105from the perspective of the user150. In some implementations, the electronic device120presents the user interface by displaying the XR content (e.g., the XR cylinder109) on the additive display, which is, in turn, overlaid on the physical environment105from the perspective of the user150.

In some implementations, the user150wears the electronic device120such as a near-eye system. As such, the electronic device120includes one or more displays provided to display the XR content (e.g., a single display or one for each eye). For example, the electronic device120encloses the FOV of the user150. In such implementations, the electronic device120presents the XR environment128by displaying data corresponding to the XR environment128on the one or more displays or by projecting data corresponding to the XR environment128onto the retinas of the user150.

In some implementations, the electronic device120includes an integrated display (e.g., a built-in display) that displays the XR environment128. In some implementations, the electronic device120includes a head-mountable enclosure. In various implementations, the head-mountable enclosure includes an attachment region to which another device with a display can be attached. For example, in some implementations, the electronic device120can be attached to the head-mountable enclosure. In various implementations, the head-mountable enclosure is shaped to form a receptacle for receiving another device that includes a display (e.g., the electronic device120). For example, in some implementations, the electronic device120slides/snaps into or otherwise attaches to the head-mountable enclosure. In some implementations, the display of the device attached to the head-mountable enclosure presents (e.g., displays) the XR environment128. In some implementations, the electronic device120is replaced with an XR chamber, enclosure, or room configured to present XR content in which the user150does not wear the electronic device120.

In some implementations, the controller110and/or the electronic device120cause an XR representation of the user150to move within the XR environment128based on movement information (e.g., body pose data, eye tracking data, hand/limb/finger/extremity tracking data, etc.) from the electronic device120and/or optional remote input devices within the physical environment105. In some implementations, the optional remote input devices correspond to fixed or movable sensory equipment within the physical environment105(e.g., image sensors, depth sensors, infrared (IR) sensors, event cameras, microphones, etc.). In some implementations, each of the remote input devices is configured to collect/capture input data and provide the input data to the controller110and/or the electronic device120while the user150is physically within the physical environment105. In some implementations, the remote input devices include microphones, and the input data includes audio data associated with the user150(e.g., speech samples). In some implementations, the remote input devices include image sensors (e.g., cameras), and the input data includes images of the user150. In some implementations, the input data characterizes body poses of the user150at different times. In some implementations, the input data characterizes head poses of the user150at different times. In some implementations, the input data characterizes hand tracking information associated with the hands of the user150at different times. In some implementations, the input data characterizes the velocity and/or acceleration of body parts of the user150such as his/her hands. In some implementations, the input data indicates joint positions and/or joint orientations of the user150. In some implementations, the remote input devices include feedback devices such as speakers, lights, or the like.

The memory220includes high-speed random-access memory, such as dynamic random-access memory (DRAM), static random-access memory (SRAM), double-data-rate random-access memory (DDR RAM), or other random-access solid-state memory devices. In some implementations, the memory220includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory220optionally includes one or more storage devices remotely located from the one or more processing units202. The memory220comprises a non-transitory computer readable storage medium. In some implementations, the memory220or the non-transitory computer readable storage medium of the memory220stores the following programs, modules and data structures, or a subset thereof described below with respect toFIG.2.

The operating system230includes procedures for handling various basic system services and for performing hardware dependent tasks.

In some implementations, a data obtainer242is configured to obtain data (e.g., captured image frames of the physical environment105, presentation data, input data, user interaction data, camera pose tracking information, eye tracking information, head/body pose tracking information, hand/limb/finger/extremity tracking information, sensor data, location data, etc.) from at least one of the I/O devices206of the controller110, the I/O devices and sensors306of the electronic device120, and the optional remote input devices. To that end, in various implementations, the data obtainer242includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, a mapper and locator engine244is configured to map the physical environment105and to track the position/location of at least the electronic device120or the user150with respect to the physical environment105. To that end, in various implementations, the mapper and locator engine244includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, a data transmitter246is configured to transmit data (e.g., presentation data such as rendered image frames associated with the XR environment, location data, etc.) to at least the electronic device120and optionally one or more other devices. To that end, in various implementations, the data transmitter246includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, a privacy architecture408is configured to ingest input data and filter user information and/or identifying information within the input data based on one or more privacy filters. The privacy architecture408is described in more detail below with reference toFIG.4A. To that end, in various implementations, the privacy architecture408includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, an eye tracking engine412is configured to determine an eye tracking vector413(e.g., with a gaze direction) based on the input data and update the eye tracking vector413over time as shown inFIGS.4A and4B. For example, the gaze direction indicates a point (e.g., associated with x, y, and z coordinates relative to the physical environment105or the world at-large), a physical object, or a region of interest (ROI) in the physical environment105at which the user150is currently looking. As another example, the gaze direction indicates a point (e.g., associated with x, y, and z coordinates relative to the XR environment128), an XR object, or a region of interest (ROI) in the XR environment128at which the user150is currently looking. The eye tracking engine412is described in more detail below with reference toFIG.4A. To that end, in various implementations, the eye tracking engine412includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, a body/head pose tracking engine414is configured to determine a pose characterization vector415based on the input data and update the pose characterization vector415over time. For example, as shown inFIG.4B, the pose characterization vector415includes a head pose descriptor492A (e.g., upward, downward, neutral, etc.), translational values for the head pose492B, rotational values for the head pose492C, a body pose descriptor494A (e.g., standing, sitting, prone, etc.), translational values for body sections/extremities/limbs/joints494B, rotational values for the body sections/extremities/limbs/joints494C, and/or the like. The body/head pose tracking engine414is described in more detail below with reference toFIG.4A. To that end, in various implementations, the body/head pose tracking engine414includes instructions and/or logic therefor, and heuristics and metadata therefor. In some implementations, the eye tracking engine412and the body/head pose tracking engine414may be located on the electronic device120in addition to or in place of the controller110.

In some implementations, a content selector422is configured to select XR content (sometimes also referred to herein as “graphical content” or “virtual content”) from a content library425based on one or more user requests and/or inputs (e.g., a voice command, a selection from a user interface (UI) menu of XR content items, and/or the like). The content selector422is described in more detail below with reference toFIG.4A. To that end, in various implementations, the content selector422includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the content library425includes a plurality of content items such as audio/visual (A/V) content, virtual agents (VAs), and/or XR content, objects, items, scenery, etc. As one example, the XR content includes 3D reconstructions of user captured videos, movies, TV episodes, and/or other XR content. In some implementations, the content library425is pre-populated or manually authored by the user150. In some implementations, the content library425is located local relative to the controller110. In some implementations, the content library425is located remote from the controller110(e.g., at a remote server, a cloud server, or the like).

In some implementations, a content manager430is configured to manage and update the layout, setup, structure, and/or the like for the XR environment128including one or more of VA(s), XR content, one or more user interface (UI) elements associated with the XR content, and/or the like. The content manager430is described in more detail below with reference toFIG.4A. To that end, in various implementations, the content manager430) includes instructions and/or logic therefor, and heuristics and metadata therefor. In some implementations, the content manager430) includes a frame buffer434, a content updater436, and a feedback engine438. In some implementations, the frame buffer434includes XR content, a rendered image frame, and/or the like for one or more past instances and/or frames.

In some implementations, the content updater436is configured to modify the XR environment128over time based on translational or rotational movement, user commands, user inputs, and/or the like. To that end, in various implementations, the content updater436includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the feedback engine438is configured to generate sensory feedback (e.g., visual feedback such as text or lighting changes, audio feedback, haptic feedback, etc.) associated with the XR environment128. To that end, in various implementations, the feedback engine438includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, a rendering engine450) is configured to render an XR environment128(sometimes also referred to herein as a “graphical environment” or “virtual environment”) or image frame associated therewith as well as the VA(s), XR content, one or more UI elements associated with the XR content, and/or the like. To that end, in various implementations, the rendering engine450includes instructions and/or logic therefor, and heuristics and metadata therefor. In some implementations, the rendering engine450) includes a pose determiner452, a renderer454, an optional image processing architecture462, and an optional compositor464. One of ordinary skill in the art will appreciate that the optional image processing architecture462and the optional compositor464may be present for video pass-through configuration but may be removed for fully VR or optical see-through configurations.

In some implementations, the pose determiner452is configured to determine a current camera pose of the electronic device120and/or the user150relative to the A/V content and/or XR content. The pose determiner452is described in more detail below with reference toFIG.4A. To that end, in various implementations, the pose determiner452includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the renderer454is configured to render the A/V content and/or the XR content according to the current camera pose relative thereto. The renderer454is described in more detail below with reference toFIG.4A. To that end, in various implementations, the renderer454includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the image processing architecture462is configured to obtain (e.g., receive, retrieve, or capture) an image stream including one or more images of the physical environment105from the current camera pose of the electronic device120and/or the user150. In some implementations, the image processing architecture462is also configured to perform one or more image processing operations on the image stream such as warping, color correction, gamma correction, sharpening, noise reduction, white balance, and/or the like. The image processing architecture462is described in more detail below with reference toFIG.4A. To that end, in various implementations, the image processing architecture462includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the compositor464is configured to composite the rendered A/V content and/or XR content with the processed image stream of the physical environment105from the image processing architecture462to produce rendered image frames of the XR environment128for display. The compositor464is described in more detail below with reference toFIG.4A. To that end, in various implementations, the compositor464includes instructions and/or logic therefor, and heuristics and metadata therefor.

Although the data obtainer242, the mapper and locator engine244, the data transmitter246, the privacy architecture408, the eye tracking engine412, the body/head pose tracking engine414, the content selector422, the content manager430, and the rendering engine450are shown as residing on a single device (e.g., the controller110), it should be understood that in other implementations, any combination of the data obtainer242, the mapper and locator engine244, the data transmitter246, the privacy architecture408, the eye tracking engine412, the body/head pose tracking engine414, the content selector422, the content manager430, and the rendering engine450may be located in separate computing devices.

In some implementations, the functions and/or components of the controller110are combined with or provided by the electronic device120shown below inFIG.3. Moreover,FIG.2is intended more as a functional description of the various features which be present in a particular implementation as opposed to a structural schematic of the implementations described herein. As recognized by those of ordinary skill in the art, items shown separately could be combined and some items could be separated. For example, some functional modules shown separately inFIG.2could be implemented in a single module and the various functions of single functional blocks could be implemented by one or more functional blocks in various implementations. The actual number of modules and the division of particular functions and how features are allocated among them will vary from one implementation to another and, in some implementations, depends in part on the particular combination of hardware, software, and/or firmware chosen for a particular implementation.

In some implementations, the one or more communication buses304include circuitry that interconnects and controls communications between system components. In some implementations, the one or more I/O devices and sensors306include at least one of an inertial measurement unit (IMU), an accelerometer, a gyroscope, a magnetometer, a thermometer, one or more physiological sensors (e.g., blood pressure monitor, heart rate monitor, blood oximetry monitor, blood glucose monitor, etc.), one or more microphones, one or more speakers, a haptics engine, a heating and/or cooling unit, a skin shear engine, one or more depth sensors (e.g., structured light, time-of-flight, LiDAR, or the like), a localization and mapping engine, an eye tracking engine, a body/head pose tracking engine, a hand/limb/finger/extremity tracking engine, a camera pose tracking engine, or the like.

In some implementations, the one or more displays312are configured to present the XR environment to the user. In some implementations, the one or more displays312are also configured to present flat video content to the user (e.g., a 2-dimensional or “flat” AVI, FLV, WMV, MOV, MP4, or the like file associated with a TV episode or a movie, or live video pass-through of the physical environment105). In some implementations, the one or more displays312correspond to touchscreen displays. In some implementations, the one or more displays312correspond to holographic, digital light processing (DLP), liquid-crystal display (LCD), liquid-crystal on silicon (LCoS), organic light-emitting field-effect transitory (OLET), organic light-emitting diode (OLED), surface-conduction electron-emitter display (SED), field-emission display (FED), quantum-dot light-emitting diode (QD-LED), micro-electro-mechanical system (MEMS), and/or the like display types. In some implementations, the one or more displays312correspond to diffractive, reflective, polarized, holographic, etc. waveguide displays. For example, the electronic device120includes a single display. In another example, the electronic device120includes a display for each eye of the user. In some implementations, the one or more displays312are capable of presenting AR and VR content. In some implementations, the one or more displays312are capable of presenting AR or VR content.

In some implementations, the image capture device370correspond to one or more RGB cameras (e.g., with a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), IR image sensors, event-based cameras, and/or the like. In some implementations, the image capture device370includes a lens assembly, a photodiode, and a front-end architecture. In some implementations, the image capture device370includes exterior-facing and/or interior-facing image sensors.

The memory320includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some implementations, the memory320includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory320optionally includes one or more storage devices remotely located from the one or more processing units302. The memory320comprises a non-transitory computer readable storage medium. In some implementations, the memory320or the non-transitory computer readable storage medium of the memory320stores the following programs, modules and data structures, or a subset thereof including an optional operating system330and a presentation engine340.

The operating system330includes procedures for handling various basic system services and for performing hardware dependent tasks. In some implementations, the presentation engine340is configured to present media items and/or XR content to the user via the one or more displays312. To that end, in various implementations, the presentation engine340includes a data obtainer342, a presenter470, an interaction handler420, and a data transmitter350.

In some implementations, the data obtainer342is configured to obtain data (e.g., presentation data such as rendered image frames associated with the user interface or the XR environment, input data, user interaction data, head tracking information, camera pose tracking information, eye tracking information, hand/limb/finger/extremity tracking information, sensor data, location data, etc.) from at least one of the I/O devices and sensors306of the electronic device120, the controller110, and the remote input devices. To that end, in various implementations, the data obtainer342includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the interaction handler420is configured to detect user interactions with the presented A/V content and/or XR content (e.g., gestural inputs detected via hand tracking, eye gaze inputs detected via eye tracking, voice commands, etc.). To that end, in various implementations, the interaction handler420includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the presenter470is configured to present and update A/V content and/or XR content (e.g., the rendered image frames associated with the user interface or the XR environment128including the VA(s), the XR content, one or more UI elements associated with the XR content, and/or the like) via the one or more displays312. To that end, in various implementations, the presenter470includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the data transmitter350is configured to transmit data (e.g., presentation data, location data, user interaction data, head tracking information, camera pose tracking information, eye tracking information, hand/limb/finger/extremity tracking information, etc.) to at least the controller110. To that end, in various implementations, the data transmitter350includes instructions and/or logic therefor, and heuristics and metadata therefor.

Although the data obtainer342, the interaction handler420, the presenter470, and the data transmitter350are shown as residing on a single device (e.g., the electronic device120), it should be understood that in other implementations, any combination of the data obtainer342, the interaction handler420, the presenter470, and the data transmitter350may be located in separate computing devices.

FIG.4Ais a block diagram of an example content delivery architecture400in accordance with some implementations. While pertinent features are shown, those of ordinary skill in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity and so as not to obscure more pertinent aspects of the example implementations disclosed herein. To that end, as a non-limiting example, the content delivery architecture400is included in a computing system such as the controller110shown inFIGS.1and2: the electronic device120shown inFIGS.1and3; and/or a suitable combination thereof.

As shown inFIG.4A, one or more local sensors402of the controller110, the electronic device120, and/or a combination thereof obtain local sensor data403associated with the physical environment105. For example, the local sensor data403includes images or a stream thereof of the physical environment105, simultaneous location and mapping (SLAM) information for the physical environment105and the location of the electronic device120or the user150relative to the physical environment105, ambient lighting information for the physical environment105, ambient audio information for the physical environment105, acoustic information for the physical environment105, dimensional information for the physical environment105, semantic labels for objects within the physical environment105, and/or the like. In some implementations, the local sensor data403includes un-processed or post-processed information.

Similarly, as shown inFIG.4A, one or more remote sensors404associated with the optional remote input devices within the physical environment105obtain remote sensor data405associated with the physical environment105. For example, the remote sensor data405includes images or a stream thereof of the physical environment105, SLAM information for the physical environment105and the location of the electronic device120or the user150relative to the physical environment105, ambient lighting information for the physical environment105, ambient audio information for the physical environment105, acoustic information for the physical environment105, dimensional information for the physical environment105, semantic labels for objects within the physical environment105, and/or the like. In some implementations, the remote sensor data405includes un-processed or post-processed information.

According to some implementations, the privacy architecture408ingests the local sensor data403and the remote sensor data405. In some implementations, the privacy architecture408includes one or more privacy filters associated with user information and/or identifying information. In some implementations, the privacy architecture408includes an opt-in feature where the electronic device120informs the user150as to what user information and/or identifying information is being monitored and how the user information and/or the identifying information will be used. In some implementations, the privacy architecture408selectively prevents and/or limits content delivery architecture400or portions thereof from obtaining and/or transmitting the user information. To this end, the privacy architecture408receives user preferences and/or selections from the user150in response to prompting the user150for the same. In some implementations, the privacy architecture408prevents the content delivery architecture400from obtaining and/or transmitting the user information unless and until the privacy architecture408obtains informed consent from the user150. In some implementations, the privacy architecture408anonymizes (e.g., scrambles, obscures, encrypts, and/or the like) certain types of user information. For example, the privacy architecture408receives user inputs designating which types of user information the privacy architecture408anonymizes. As another example, the privacy architecture408anonymizes certain types of user information likely to include sensitive and/or identifying information, independent of user designation (e.g., automatically).

According to some implementations, the eye tracking engine412obtains the local sensor data403and the remote sensor data405after it has been subjected to the privacy architecture408. In some implementations, the eye tracking engine412determines an eye tracking vector413based on the input data and updates the eye tracking vector413over time.

FIG.4Bshows an example data structure for the eye tracking vector413in accordance with some implementations. As shown inFIG.4B, the eye tracking vector413may correspond to an N-tuple characterization vector or characterization tensor that includes a timestamp481(e.g., the most recent time the eye tracking vector413was updated), one or more angular values482for a current gaze direction (e.g., roll, pitch, and yaw values), one or more translational values484for the current gaze direction (e.g., x, y, and z values relative to the physical environment105, the world, and/or the like), and/or miscellaneous information486. One of ordinary skill in the art will appreciate that the data structure for the eye tracking vector413inFIG.4Bis merely an example that may include different information portions in various other implementations and be structured in myriad ways in various other implementations.

For example, the gaze direction indicates a point (e.g., associated with x, y, and z coordinates relative to the physical environment105or the world at-large), a physical object, or a region of interest (ROI) in the physical environment105at which the user150is currently looking. As another example, the gaze direction indicates a point (e.g., associated with x, y, and z coordinates relative to the XR environment128), an XR object, or a region of interest (ROI) in the XR environment128at which the user150is currently looking.

According to some implementations, the body/head pose tracking engine414obtains the local sensor data403and the remote sensor data405after it has been subjected to the privacy architecture408. In some implementations, the body/head pose tracking engine414determines a pose characterization vector415based on the input data and updates the pose characterization vector415over time.

FIG.4Bshows an example data structure for the pose characterization vector415in accordance with some implementations. As shown inFIG.4B, the pose characterization vector415may correspond to an N-tuple characterization vector or characterization tensor that includes a timestamp491(e.g., the most recent time the pose characterization vector415was updated), a head pose descriptor492A (e.g., upward, downward, neutral, etc.), translational values for the head pose492B, rotational values for the head pose492C, a body pose descriptor494A (e.g., standing, sitting, prone, etc.), translational values for body sections/extremities/limbs/joints494B, rotational values for the body sections/extremities/limbs/joints494C, and/or miscellaneous information496. In some implementations, the pose characterization vector413also includes information associated with hand/extremity tracking. One of ordinary skill in the art will appreciate that the data structure for the pose characterization vector415inFIG.4Bis merely an example that may include different information portions in various other implementations and be structured in myriad ways in various other implementations.

According to some implementations, the interaction handler420obtains (e.g., receives, retrieves, or detects) one or more user inputs421provided by the user150that are associated with selecting A/V content, one or more VAs, and/or XR content for presentation. For example, the one or more user inputs421correspond to a gestural input selecting XR content from a UI menu detected via hand/extremity tracking, an eye gaze input selecting XR content from the UI menu detected via eye tracking, a voice command selecting XR content from the UI menu detected via a microphone, and/or the like. In some implementations, the content selector422selects XR content427from the content library425based on one or more user inputs421(e.g., a voice command, a selection from a menu of XR content items, and/or the like).

In various implementations, the content manager430manages and updates the layout, setup, structure, and/or the like for the XR environment128including one or more of VAs, XR content, one or more UI elements associated with the XR content, and/or the like. To that end, the content manager430includes the frame buffer434, the content updater436, and the feedback engine438.

In some implementations, the frame buffer434includes XR content, a rendered image frame, and/or the like for one or more past instances and/or frames. In some implementations, the content updater436modifies the XR environment128over time based on the eye tracking vector413, the pose characterization vector415, user inputs421associated with modifying and/or manipulating the XR content or VA(s), translational or rotational movement of objects within the physical environment105, translational or rotational movement of the electronic device120(or the user150), and/or the like. In some implementations, the feedback engine438generates sensory feedback (e.g., visual feedback such as text or lighting changes, audio feedback, haptic feedback, etc.) associated with the XR environment128.

According to some implementations, the pose determiner452determines a current camera pose of the electronic device120and/or the user150relative to the XR environment128and/or the physical environment105based at least in part on the pose characterization vector415. In some implementations, the renderer454renders the VA(s), the XR content427, one or more UI elements associated with the XR content, and/or the like according to the current camera pose relative thereto.

According to some implementations, the optional image processing architecture462obtains an image stream from an image capture device370including one or more images of the physical environment105from the current camera pose of the electronic device120and/or the user150. In some implementations, the image processing architecture462also performs one or more image processing operations on the image stream such as warping, color correction, gamma correction, sharpening, noise reduction, white balance, and/or the like. In some implementations, the optional compositor464composites the rendered XR content with the processed image stream of the physical environment105from the image processing architecture462to produce rendered image frames of the XR environment128. In various implementations, the presenter470presents the rendered image frames of the XR environment128to the user150via the one or more displays312. One of ordinary skill in the art will appreciate that the optional image processing architecture462and the optional compositor464may not be applicable for fully virtual environments (or optical see-through scenarios).

FIGS.5A-5Hillustrate a sequence of instances510-580for a content navigation scenario in accordance with some implementations. While certain specific features are illustrated, those skilled in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, as a non-limiting example, the sequence of instances510-580are rendered and presented by a computing system such as the controller110shown inFIGS.1and2: the electronic device120shown inFIGS.1and3: and/or a suitable combination thereof.

As shown inFIGS.5A-5H, the content navigation scenario includes a physical environment105and an XR environment128displayed on the display122of the electronic device120(e.g., associated with the user150). The electronic device120presents the XR environment128to the user150while the user150is physically present within the physical environment105that includes a door115, which is currently within the FOV111of an exterior-facing image sensor of the electronic device120. As such, in some implementations, the user150holds the electronic device120in his/her hand(s) similar to the operating environment100inFIG.1.

In other words, in some implementations, the electronic device120is configured to present XR content and to enable optical see-through or video pass-through of at least a portion of the physical environment105on the display122(e.g., the door115). For example, the electronic device120corresponds to a mobile phone, tablet, laptop, near-eye system, wearable computing device, or the like.

As shown inFIG.5A, during the instance510(e.g., associated with time T1) of the content navigation scenario, the electronic device120presents an XR environment128including a virtual agent (VA)506and a first content pane514A with a first appearance and a first z-depth value within the XR environment128. In some implementations, the first appearance corresponds to a first translucency value, a first blur radius value, and/or the like. InFIG.5A, the first content pane514A includes an input field516(e.g., a search bar) and first content. In some implementations, the first content pane514A is volumetric or three-dimensional (3D). For example, the first content pane514A corresponds to a web browser window, an application window; and/or the like. Continuing with this example, the first content corresponds to text, image(s), video(s), audio, and/or the like. One of ordinary skill in the art will appreciate that the first content pane514A is an example that may be modified in various other implementations.

As shown inFIG.5A, the XR environment128also includes a visualization512of the gaze direction of the user150relative to the XR environment128. One of ordinary skill in the art will appreciate that the visualization512may be modified or may not be displayed in various implementations. As shown inFIG.5A, during the instance510, the visualization512of the gaze direction of the user150is directed to the input field516.

As shown inFIG.5B, during the instance520) (e.g., associated with time T2) of the content navigation scenario, the electronic device120presents a notification522overlaid on the XR environment128in response to detecting the gaze direction of the user150directed to the input field516inFIG.5Afor at least a predetermined amount of time (e.g., X seconds). As shown inFIG.5B, the notification522indicates that: “Input field516is selected. Please provide a search string.” One of ordinary skill in the art will appreciate that the notification522is an example that may be modified or may not be displayed in various other implementations. One of ordinary skill in the art will appreciate that the input field516may be selected via other input modalities, such as a touch input on the display122, a speech input, a hand tracking input, and/or the like in various other implementations. As shown inFIG.5B, during the instance520) (e.g., associated with time T2) of the content navigation scenario, the electronic device120detects a speech input524from the user150that corresponds to an input search string.

As shown inFIG.5C, during the instance530) (e.g., associated with time T3) of the content navigation scenario, the electronic device120presents a notification532overlaid on the XR environment128in response to detecting the speech input524inFIG.5B. As shown inFIG.5C, the notification532indicates that the electronic device120is performing a search operation based on the input search string: “Searching based on search string provided via the speech input524.” One of ordinary skill in the art will appreciate that the notification532is an example that may be modified or may not be displayed in various other implementations.

As shown inFIG.5D, during the instance540(e.g., associated with time T4) of the content navigation scenario, the electronic device120presents a second content second content plane542A (sometimes also referred to herein as a “search pane”) with the first appearance and the first z-depth value within the XR environment128in response to performing the search operation based on the input search string provided via the speech input524inFIG.5B. As shown inFIG.5D, the second content plane542A is overlaid on a first content pane514B (e.g., a modified version of the first content pane514A inFIG.5A) with a second appearance and a second z-depth value within the XR environment128. In some implementations, the second z-depth value is different from (e.g., greater than) the first z-depth value. In some implementations, the second appearance corresponds to a second translucency value, a second blur radius value, and/or the like, which are greater than the first translucency value, a first blur radius value, and/or the like associated with the first appearance.

InFIG.5D, the second content plane542A includes the input field516and a plurality of search results544A,544B, and544N associated with the input search string provided via the speech input524inFIG.5B. For example, the search results544A,544B, and544N include media content, hyperlinks, and/or the like. In some implementations, the second content plane542A is volumetric or 3D. One of ordinary skill in the art will appreciate that the second content plane542A is an example that may be modified in various other implementations.

As shown inFIG.5D, during the instance540, the visualization512of the gaze direction of the user150is directed to the search result544B within the second content plane542A. One of ordinary skill in the art will appreciate that the visualization512may be modified or may not be displayed in various implementations.

In some implementations, in response to detecting selection of one of the search results544A,544B, and544N within the second content plane542A with a first input type (e.g., a speech input, touch input, hand tracking input, eye tracking input, or the like), the electronic device120ceases to display the second content plane542A and presents content associated with the selected search result within the first content pane514A with the first appearance and the first z-depth value within the XR environment128. In some implementations, in response to detecting selection of one of the search results544A,544B, and544N within the second content plane542A with a second input type (e.g., a pinch and pull gesture, another gesture, or the like), the electronic device120may open an associated web page or display associated content in a new tab within a web browser application. In some implementations, in response to detecting selection of one of the search results544A,544B, and544N within the second content plane542A with a second input type (e.g., a pinch and pull gesture, another gesture, or the like), the electronic device120may open an associated web page or display associated content in a pane associated with a new stack or an existing stack of content panes. One of ordinary skill in the art will appreciate that the stacks of content panes may be configured similar to Provisional patent application No. 62/210,415, filed on Jun. 14, 2021 (Attorney Docket No. 27753-50477PR1), which is incorporated by reference herein in its entirety.

As shown inFIG.5E, during the instance550) (e.g., associated with time T5) of the content navigation scenario, the electronic device120presents a preview pane552A associated with the search result544B inFIG.5Dwith the first appearance and the first z-depth value within the XR environment128in response to detecting the gaze direction of the user150directed to the search result544B within the second content plane542A inFIG.5Dfor at least the predetermined amount of time (e.g., X seconds). As shown inFIG.5E, the preview pane552A is overlaid on a second content pane542B (e.g., a modified version of the second content plane542A) with the second appearance and the second z-depth value. Furthermore, inFIG.5E, the second content pane542B is overlaid on the first content pane514B with the second appearance and a third z-depth value. In some implementations, the third z-depth value is different from (e.g., greater than) the second z-depth value. For example, the preview pane552A includes text, image(s), video(s), audio, and/or the like associated with the search result544B inFIG.5D. One of ordinary skill in the art will appreciate that the preview pane552A is an example that may be modified or may not be displayed in various other implementations.

In some implementations, the second z-depth value associated with the second content pane542B inFIG.5Eand the second z-depth value associated with the first content pane514B inFIG.5Dcorrespond to a same z-depth value. In some implementations, the second z-depth value associated with the second content pane542B inFIG.5Eand the second z-depth value associated with the first content pane514B inFIG.5Dcorrespond to different z-depth values. In some implementations, the second z-depth value associated with the second content pane542B inFIG.5Eand the second z-depth value associated with the first content pane514B inFIG.5Dcorrespond to similar z-depth values within a predefined or deterministic offset of one another. In some implementations, the second z-depth value is greater when two panes are displayed within the XR environment128as inFIG.5Dthan when more than two panes are displayed within the XR environment128as inFIG.5E.

In some implementations, the second appearance associated with the second content pane542B inFIG.5Eand the second appearance associated with the first content pane514B inFIG.5Ecorrespond to a same appearance. In some implementations, the second appearance associated with the second content pane542B inFIG.5Eand the second appearance associated with the first content pane514B inFIG.5Ecorrespond to different appearances. As one example, the second appearance associated with the second content pane542B inFIG.5Emay correspond to a second blur radius, a second color, a second texture, and/or the like that is different from the first appearance of the second content plane542A inFIG.5D. Continuing with this example, the second appearance associated with the first content pane514B inFIG.5Emay correspond to a third blur radius, a third color, a third texture, and/or the like that is different from the first appearance of the first content pane514A inFIG.5A. In some implementations, the second appearance associated with the second content pane542B inFIG.5Eand the second appearance associated with the first content pane514B inFIG.5Ecorrespond to similar appearances within a predefined or deterministic tolerance of one another.

As shown inFIG.5F, during the instance560) (e.g., associated with time T6) of the content navigation scenario, the electronic device120presents the preview pane552A associated with the search result544B inFIG.5Dwith the first appearance and the first z-depth value within the XR environment128and a recommendation pane562A with the first appearance and the first z-depth value within the XR environment128in response to a selection of the input field516, such as by detecting the gaze direction of the user150directed to the search result544B within the second content plane542A inFIG.5Dfor at least the predetermined amount of time (e.g., X seconds). As shown inFIG.5F, the recommendation pane562A includes: the input field516: a plurality of content recommendations564A,564B, and564N based on the search result544B: and a plurality of search recommendations566A,566B, and566N based on the search result544B. For example, the plurality of content recommendations564A,564B, and564N include media content, hyperlinks, and/or the like. For example, the plurality of search recommendations566A,566B, and566N include media content, hyperlinks, and/or the like. One of ordinary skill in the art will appreciate that the recommendation pane562A is an example that may be modified or may not be displayed in various other implementations.

As shown inFIG.5F, the preview pane552A and the recommendation pane562A are overlaid on the second content pane542B with the second appearance and the second z-depth value. Furthermore, inFIG.5F, the second content pane542B is overlaid on the first content pane514B with the second appearance and the third z-depth value. In some implementations, the third z-depth value is different from (e.g., greater than) the second z-depth value.

In some implementation, in response to detecting selection of the second content pane542B inFIG.5E, the electronic device120ceases to display the preview pane552A and displays both the second content plane542A and the first content pane514B closer to the user150(e.g., with the first z-depth value and the second z-depth value, respectively). In some implementation, in response to detecting selection of the first content pane514B inFIG.5E, the electronic device120ceases to display the preview pane552A and the second content pane542B and displays the first content pane514A closer to the user150(e.g., with the first z-depth value). However, continuing with this example, in response to detecting a subsequent selection of the input field516within the first content pane514A, the electronic device120displays the second content plane542A overlaid on the first content pane514B, wherein the second content plane542A includes the same search results544A,544B, and544N as inFIG.5D.

As shown inFIG.5G, during the instance570(e.g., associated with time T7) of the content navigation scenario, the electronic device120presents a recommendation pane572A overlaid on the XR environment128in response to detecting the gaze direction of the user150directed to the input field516inFIG.5Afor at least a predetermined amount of time (e.g., X seconds). As shown inFIG.5G, the recommendation pane572A includes: the input field516: a plurality of content recommendations574A,574B, and574N based on user preferences, search history, current context, etc.; and a plurality of search recommendations576A,576B, and576N based on user preferences, search history, current context, etc. For example, the plurality of content recommendations574A,574B, and574N include media content, hyperlinks, and/or the like. For example, the plurality of search recommendations576A,576B, and576N include media content, hyperlinks, and/or the like. One of ordinary skill in the art will appreciate that the recommendation pane572A is an example that may be modified or may not be displayed in various other implementations.

As shown inFIG.5G, during the instance570(e.g., associated with time T7) of the content navigation scenario, the electronic device120detects, via the body/head pose tracking engine414, a hand tracking input with a left hand151of the user150directed to the first content pane514B. InFIG.5G, the electronic device120presents a representation575of the left hand151of the user150within the XR environment128. One of ordinary skill in the art will appreciate that the hand tracking input with the left hand151of the user150is merely an example user input and that the electronic device120may detect various other input modalities such as speech inputs, touch inputs, eye tracking inputs, and/or the like.

As shown inFIG.5H, during the instance580) (e.g., associated with time T8) of the content navigation scenario, the electronic device120presents the first content pane514A with the first appearance and the first z-depth value within the XR environment128in response to detecting the hand tracking input with a left hand151directed to the first content pane514B inFIG.5G. As shown inFIG.5H, the first content pane514A is overlaid on a recommendation pane572B (e.g., a modified version of the recommendation pane572A) with a second appearance and a second z-depth value within the XR environment128. InFIG.5H, the first content pane514A includes the input field516and the first content. In some implementations, in response to detecting the hand tracking input with a left hand151directed to the first content pane514B inFIG.5G, the electronic device120presents the first content pane514A with the first appearance and the first z-depth value within the XR environment128and ceases to display the recommendation pane572A/B.

FIG.6is a flowchart representation of a method600of navigating windows in 3D in accordance with some implementations. In various implementations, the method600is performed at a computing system including non-transitory memory and one or more processors, wherein the computing system is communicatively coupled to a display device and one or more input devices (e.g., the electronic device120shown inFIGS.1and3; the controller110inFIGS.1and2; or a suitable combination thereof). In some implementations, the method600is performed by processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method600is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory). In some implementations, the computing system corresponds to one of a tablet, a laptop, a mobile phone, a near-eye system, a wearable computing device, or the like.

As discussed above, current web browsers may use a tab arrangement for open web pages and also provide a means for viewing browsing history. This organization structure makes it difficult to concurrently view past and present web pages and/or searches. While in an extended reality (XR) environment, a computing system content panes in z-depth where the panes are selectable/accessible via hand tracking inputs, eye tracking inputs, speech inputs, and/or the like. As such, in various implementations described herein, the selection of n input field (e.g., a search bar) within a first content pane causes the first content pane, including first content (e.g., a web page or other media), to be pushed backward in z-depth and a second content pane to be displayed at the former z-depth of the first content pane.

As represented by block610, the method600includes displaying a first content pane with a first appearance at a first z-depth within an extended reality (XR) environment, wherein the first content pane includes first content and an input field. InFIG.5A, for example, the electronic device120presents an XR environment128including a virtual agent (VA)506and a first content pane514A with a first appearance and a first z-depth value within the XR environment128. In some implementations, the first appearance corresponds to a first translucency value, a first blur radius value, and/or the like. InFIG.5A, the first content pane514A includes a input field516and first content. In some implementations, the first z-depth corresponds to a distance between the relative location for the first content pane within the XR environment in real-world coordinates and a location in real-world coordinates for one of the computing system, a viewpoint of a user associated with the computing system, the user associated with the computing system, a portion of the user associated with the computing system (e.g., a body part of the user, a viewpoint of the user, a midpoint between their eyes, the tip of the user's nose, a centroid associated with the user's head, a centroid associated with the user's face, etc.), and/or the like.

In some implementations, as represented by block612, the first content pane corresponds to one of a web browser window, an application window, or an operating system window; and the first content corresponds to one of text, one or more images, one or more videos, or audio data. In some implementations, the first content pane is volumetric or three-dimensional (3D).

In some implementations, the first content pane is overlaid on the physical environment while displayed within the XR environment. As shown inFIG.5A, for example, the first content pane514A is overlaid on a video pass-through or optical see-through version of the physical environment105.

In some implementations, the display device corresponds to a transparent lens assembly, and wherein the XR environment is projected onto the transparent lens assembly. In some implementations, the display device corresponds to a near-eye system, and wherein presenting the XR environment includes compositing the XR environment with one or more images of a physical environment captured by an exterior-facing image sensor. In some implementations, the XR environment corresponds to AR content overlaid on the physical environment. In one example, the XR environment is associated with an optical see-through configuration. In another example, the XR environment is associated with a video pass-through configuration. In some implementations, the XR environment corresponds a VR environment with VR content.

As represented by block620, the method600includes detecting a user input directed to the input field. InFIG.5A, for example, the electronic device120detects a gaze direction of the user150(e.g., associated with the visualization512) directed to the input field516for at least a predetermined amount of time (e.g., X seconds). In some implementations, as represented by block622, the user input corresponds to one of a hand tracking input, an eye tracking input, a touch input, or a speech input.

As represented by block630, in response to detecting the user input directed to the input field, the method600includes: moving the first content pane to a second z-depth within the XR environment, wherein the second z-depth is different from the first z-depth; modifying the first content pane by changing the first content pane from the first appearance to a second appearance: and displaying a second content pane with the first appearance at the first z-depth within the XR environment. InFIG.5D, for example, the electronic device120presents a second content plane542A with the first appearance and the first z-depth value within the XR environment128in response to performing the search operation based on the input search string provided via the speech input524inFIG.5B. As shown inFIG.5D, the second content plane542A is overlaid on a first content pane514B (e.g., a modified version of the first content pane514A inFIG.5A) with a second appearance and a second z-depth value within the XR environment128. In some implementations, the second z-depth value is different from the first z-depth value. In some implementations, the second appearance corresponds to a second translucency value, a second blur radius value, and/or the like, which are greater than the first translucency value, a first blur radius value, and/or the like associated with the first appearance. In some implementations, the second content pane at least partially overlaps the first content pane. InFIG.5D, for example, the second content plane542A is overlaid on and partially overlaps the first content pane514B. In some implementations, the second z-depth corresponds to a distance between the relative location for the first content pane within the XR environment in real-world coordinates when displayed with the second appearance and a location in real-world coordinates for one of the computing system, a viewpoint of a user associated with the computing system, the user associated with the computing system, a portion of the user associated with the computing system (e.g., a body part of the user, a midpoint between their eyes, the tip of the user's nose, a centroid associated with the user's head, a centroid associated with the user's face, etc.), and/or the like.

In some implementations, as represented by block632, the first appearance is different from the second appearance. In some implementations, as represented by block634A, the second appearance is associated with a higher translucency value than the first appearance. In some implementations, as represented by block634B, the second appearance is associated with a higher blur radius value than the first appearance. In some implementations, modifying the first content pane includes changing from the first appearance to a second appearance by blurring at least a portion of the first content pane. As one example, inFIG.6D, the second content plane542A is overlaid on a first content pane514B (e.g., a modified version of the first content pane514A inFIG.5A) with the second appearance such as a higher translucency value of a higher blur radius value than the first appearance.

In some implementations, the second content pane includes at least one of: one or more previous search queries, one or more search recommendations, or one or more content recommendations based on at least one of one or more user preferences, a user search history, or current context. As one example, inFIG.5G, the electronic device120presents a recommendation pane572A overlaid on the XR environment128in response to detecting the gaze direction of the user150directed to the input field516inFIG.5Afor at least a predetermined amount of time (e.g., X seconds). As shown inFIG.5G, the recommendation pane572A includes: the input field516: a plurality of content recommendations574A,574B, and574N based on user preferences, search history, current context, etc.: and a plurality of search recommendations576A,576B, and576N based on user preferences, search history, current context, etc. For example, the plurality of content recommendations574A,574B, and574N include media content, hyperlinks, and/or the like. For example, the plurality of search recommendations576A,576B, and576N include media content, hyperlinks, and/or the like.

In some implementations, the user input includes a search string provided via a virtual keyboard or a speech input. In some implementations, the second content pane includes at least one of: one or more search results, one or more search recommendations, or one or more content recommendations based on the search string. As one example, inFIG.5D, the electronic device120presents a second content plane542A with the first appearance and the first z-depth value within the XR environment128in response to performing the search operation based on the input search string provided via the speech input524inFIG.5B. InFIG.5D, the second content plane542A includes the input field516and a plurality of search results544A,544B, and544N associated with the input search string provided via the speech input524inFIG.5B. For example, the search results544A,544B, and544N include media content, hyperlinks, and/or the like.

In some implementations, the method600further includes: detecting a subsequent user input associated with selecting a respective search result among the one or more search results: and in response to detecting the subsequent user input: displaying a preview pane with the first appearance at the first z-depth within the XR environment, wherein the preview pane is associated with the respective search result; moving the first content pane to a third z-depth within the XR environment, wherein the third z-depth is different from the second z-depth: moving the second content pane to a fourth z-depth within the XR environment: and modifying the second content pane by changing the second content pane from the first appearance to the second appearance. In some implementations, the fourth z-depth value corresponds to a z-depth value that is different from the first z-depth value and less than the third z-depth value. In some implementations, the fourth z-depth value corresponds to the second z-depth value. For example, the subsequent user input corresponds to a gaze input directed to the respective search result for at least a predetermined amount of time such as X seconds. As another example, the subsequent user input corresponds to one of a touch input, speech input, hand tracking input, eye tracking input, gestural input, and/or the like. In some implementations, the preview pane at least partially overlaps the second content pane. In some implementations, the second content pane closes at least temporarily while the preview pane is presented.

As one example, inFIG.5E, the electronic device120presents a preview pane552A associated with the search result544B inFIG.5Dwith the first appearance and the first z-depth value within the XR environment128in response to detecting the gaze direction of the user150directed to the search result544B within the second content plane542A inFIG.5Dfor at least the predetermined amount of time (e.g., X seconds). As shown inFIG.5E, the preview pane552A is overlaid on a second content pane542B (e.g., a modified version of the second content plane542A) with the second appearance and the second z-depth value. Furthermore, inFIG.5E, the second content pane542B is overlaid on the first content pane514B with the second appearance and a third z-depth value. In some implementations, the third z-depth value is different from the second z-depth value. For example, the preview pane552A includes text, image(s), video(s), audio, and/or the like associated with the search result544B inFIG.5D.

As another example, inFIG.5F, the electronic device120presents the preview pane552A associated with the search result544B inFIG.5Dwith the first appearance and the first z-depth value within the XR environment128and a recommendation pane562A with the first appearance and the first z-depth value within the XR environment128in response to detecting the gaze direction of the user150directed to the search result544B within the second content plane542A inFIG.5Dfor at least the predetermined amount of time (e.g., X seconds). As shown inFIG.5F, the recommendation pane562A includes: the input field516: a plurality of content recommendations564A,564B, and564N based on the search result544B: and a plurality of search recommendations566A,566B, and566N based on the search result544B. For example, the plurality of content recommendations564A,564B, and564N include media content, hyperlinks, and/or the like. For example, the plurality of search recommendations566A,566B, and566N include media content, hyperlinks, and/or the like.

In some implementations, the method600further includes: detecting a subsequent user input associated with selecting the first content pane: and in response to detecting the subsequent user input: moving the first content pane to the first z-depth within the XR environment: moving the second content pane to the second z-depth within the XR environment: modifying the second content pane by changing the second content pane from the first appearance to the second appearance: and modifying the first content pane by changing the first content pane from the second appearance to the first appearance. In some implementations, the subsequent user input corresponds to one of a hand tracking input, an eye tracking input, a touch input, a gestural input, or a speech input. As one example, inFIG.5H, the electronic device120presents the first content pane514A with the first appearance and the first z-depth value within the XR environment128in response to detecting the hand tracking input with a left hand151directed to the first content pane514B inFIG.5G. As shown inFIG.5H, the first content pane514A is overlaid on a recommendation pane572B (e.g., a modified version of the recommendation pane572A) with a second appearance and a second z-depth value within the XR environment128. InFIG.5H, the first content pane514A includes the input field516and the first content. Alternatively, in some implementations, the method600further includes: detecting a subsequent user input associated with selecting the first content pane: and in response to detecting the subsequent user input: ceasing to display the second content pane: and displaying the first content pane with the first z-depth within the XR environment.

In some implementations, the method600further includes: detecting a subsequent user input associated with manipulating the second content pane: and in response to detecting the subsequent user input, manipulating the second content pane based on the subsequent user input by at least one of: translating the second content pane, rotating the second content pane, scaling the second content pane, or modifying an appearance parameter of the second content pane. For example, the appearance parameter corresponds to one of a color, contrast, texture, brightness, etc. As one example, the user150may interact with the second content plane542A inFIG.5Dwith touch inputs, speech inputs, hand tracking inputs, eye tracking inputs, and/or the like in order to translate, rotate, scale, or otherwise modify the second content plane542A inFIG.5D. As another example, the user150may interact with the first content pane514A shown inFIG.5Awith touch inputs, speech inputs, hand tracking inputs, eye tracking inputs, gestural inputs, and/or the like in order to translate, rotate, scale, or otherwise modify the first content pane514A shown inFIG.5A. As yet another example, the user150may interact with the preview pane552A shown inFIG.5Ewith touch inputs, speech inputs, hand tracking inputs, eye tracking inputs, gestural inputs, and/or the like in order to translate, rotate, scale, or otherwise modify the preview pane552A shown inFIG.5E.

It will also be understood that, although the terms “first”, “second”, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first media item could be termed a second media item, and, similarly, a second media item could be termed a first media item, which changing the meaning of the description, so long as the occurrences of the “first media item” are renamed consistently and the occurrences of the “second media item” are renamed consistently. The first media item and the second media item are both media items, but they are not the same media item.