Patent Description:
The present description relates generally to multi-user environments in computing platforms.

A user of an electronic device commonly uses applications running on that electronic device to view and/or interact with data. Often, users exchange data, such as by sending files over email, that each user can locally view and manipulate with the application running on their own electronic device. Each user can then re-share updated data with the other user if desired. In order to improve the efficiency of sharing data, some applications provide for cooperative viewing and/or manipulation of data by multiple users of a common application that is installed and running locally on multiple devices. However, in scenarios in which one of the users does not have the application installed or running on their device, the sharing of application data is typically limited or unavailable.

<CIT> discloses a method and system for sharing a user interface of a first device with a second device and enabling a user of the second device to interact with the user interface via gestures received by the second device. The first device (e.g., a smartphone) can host an application and generate a graphical user interface, which it transmits to the second device (e.g., a tablet computer) for display by the second device. The second device can receive input from a user, such as a touch input via a touchscreen of the second device, and transmit a representation of the input to the first device for providing input to the application hosted by the first device.

However, for purpose of explanation, several implementations of the subject technology are set forth in the following figures.

A physical environment refers to a physical world that people can sense and/or interact with without aid of electronic devices. The physical environment may include physical features such as a physical surface or a physical object. For example, the physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly sense and/or interact with the physical environment such as through sight, touch, hearing, taste, and smell. In contrast, an extended reality (XR) environment refers to a wholly or partially simulated environment that people sense and/or interact with via an electronic device. For example, the XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and/or the like. With an XR system, a subset of a person's physical motions, or representations thereof, are tracked, and, in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that comports with at least one law of physics. As one example, the XR system may detect head movement and, in response, adjust graphical content and an acoustic field presented to the person in a manner similar to how such views and sounds would change in a physical environment. As another example, the XR system may detect movement of the electronic device presenting the XR environment (e.g., a mobile phone, a tablet, a laptop, or the like) and, in response, adjust graphical content and an acoustic field presented to the person in a manner similar to how such views and sounds would change in a physical environment. In some situations (e.g., for accessibility reasons), the XR system may adjust characteristic(s) of graphical content in the XR environment in response to representations of physical motions (e.g., vocal commands).

There are many different types of electronic systems that enable a person to sense and/or interact with various XR environments. Examples include head mountable systems, projection-based systems, heads-up displays (HUDs), vehicle windshields having integrated display capability, windows having integrated display capability, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones/earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop/laptop computers. A head mountable system may have one or more speaker(s) and an integrated opaque display. Alternatively, a head mountable system may be configured to accept an external opaque display (e.g., a smartphone). The head mountable system may incorporate one or more imaging sensors to capture images or video of the physical environment, and/or one or more microphones to capture audio of the physical environment. Rather than an opaque display, a head mountable system may have a transparent or translucent display. The transparent or translucent display may have a medium through which light representative of images is directed to a person's eyes. The display may utilize digital light projection, OLEDs, LEDs, uLEDs, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium may be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In some implementations, the transparent or translucent display may be configured to become opaque selectively. Projection-based systems may employ retinal projection technology that projects graphical images onto a person's retina. Projection systems also may be configured to project virtual objects into the physical environment, for example, as a hologram or on a physical surface.

Implementations of the subject technology described herein provide for sharing of applications between electronic devices, particularly when one of the electronic devices does not have access to the application (e.g., because the application or an updated version of the application is not installed and/or is not running at that device). In some scenarios, all of the application data that is used for displaying a user interface (UI) of the application at the device running the application data can be continuously transmitted to another electronic device, so that the full functionality of the application is provided at the other electronic device. However, transmitting the full application data in this way can consume large amounts of bandwidth and processing power that may be unavailable or impractical in various scenarios.

In other scenarios, a flat image of the UI can be transmitted from the application running the device to the application that does not have the application so that the user of the other device can view the UI exactly as displayed at the device running the application (e.g., in a screen sharing mode of a video conferencing application). However, providing only flat images to the other device can prevent the other device from displaying or modifying the UI in accordance with a preferred format, another preference, or a position or orientation of a user of the second device. This can be particularly problematic when the UI of application being shared for display in a three-dimensional environment, such as a XR environment.

Aspects of the subject technology facilitate collaborative use of an application by first and second users of first and second devices when an application is only installed at one of the devices. In one or more implementations, an application-agnostic framework (e.g., a system level framework on both devices) allows the device running the application to cast a version of an application user interface (UI) that is displayed at the first device running the application, to the second device on which the application is not installed.

The version of the application can be a non-interactive version of the UI that is displayed at the first device, but may include sufficient state information to allow the second device to render the UI using one or more user or device preferences stored at the second device and/or using features of the environment of the second device. In other implementations, the state information can include sufficient information to allow interaction with the version of the UI that is displayed at the second device, such interacting including moving, resizing, rotating, or recoloring the UI independently of the UI displayed at the first device. In other implementations, the state information can include sufficient information to allow interactions (e.g., user inputs to the application) that are captured by the second device and transmitted back to the first device as inputs to the application running on the first device (e.g., and recast back to the second device).

In one or more implementations, the application casts a version of an application user interface that is displayed at the first device running the application to the second device on which the application is not installed, together with anchoring information that allows the first and second devices to display their respective versions of the UI at coordinated locations in respective three-dimensional (e.g., XR) environments of each device. The anchoring information can be provided to ensure that the UI is displayed by both devices as a common location relative to a shared origin in the environment of each device. This can be useful when multiple UI elements and/or other (e.g., shared) applications are displayed concurrently, to allow transfer of content between elements or applications by one of the devices, and/or to allow element and/or application interactions to be displayed correctly at both devices. For example, this can be useful to adjust the size or orientation of the version of the UI at the second device to account for a position of the user of the second device relative to the displayed version of the UI.

In one or more implementations, the version of the application that is displayed by the second device can be moved to a new location in the second physical environment of the second device, with or without affecting the location of the UI displayed by the first device. In one or more implementations, sufficient state information for the UI can be provided from the first device to the second device to allow partial or full user interaction with the version of the user interface that is displayed by the second device to control the application running at the first device.

<FIG> illustrates an example system architecture <NUM> including various electronic devices that may implement the subject system in accordance with one or more implementations. Not all of the depicted components may be used in all implementations, however, and one or more implementations may include additional or different components than those shown in the figure. Variations in the arrangement and type of the components may be made without departing from the scope of the claims as set forth herein Additional components, different components, or fewer components may be provided.

The system architecture <NUM> includes an electronic device <NUM>, an electronic device <NUM>, an electronic device <NUM>, and a server <NUM>. For explanatory purposes, the system architecture <NUM> is illustrated in <FIG> as including the electronic device <NUM>, the electronic device <NUM>, the electronic device <NUM>, and the server <NUM>; however, the system architecture <NUM> may include any number of electronic devices and any number of servers or a data center including multiple servers.

The electronic device <NUM> may be smart phone, a tablet device, or a wearable device such as a head mountable portable system, that includes a display system capable of presenting a visualization of an extended reality environment to a user <NUM>. The electronic device <NUM> may be powered with a battery and/or any other power supply. In an example, the display system of the electronic device <NUM> provides a stereoscopic presentation of the extended reality environment, enabling a three-dimensional visual display of a rendering of a particular scene, to the user. In one or more implementations, instead of, or in addition to, utilizing the electronic device <NUM> to access an extended reality environment, the user may use a handheld electronic device <NUM>, such as a tablet, watch, mobile device, and the like.

The electronic device <NUM> may include one or more cameras such as camera(s) <NUM> (e.g., visible light cameras, infrared cameras, etc.) Further, the electronic device <NUM> may include various sensors <NUM> including, but not limited to, cameras, image sensors, touch sensors, microphones, inertial measurement units (IMU), heart rate sensors, temperature sensors, Lidar sensors, radar sensors, sonar sensors, GPS sensors, Wi-Fi sensors, near-field communications sensors, etc.) Moreover, the electronic device <NUM> may include hardware elements that can receive user input such as hardware buttons or switches. User input detected by such sensors and/or hardware elements correspond to various input modalities for initiating recording within a given extended reality environment. For example, such input modalities may include, but not limited to, facial tracking, eye tracking (e.g., gaze direction), hand tracking, gesture tracking, biometric readings (e.g., heart rate, pulse, pupil dilation, breath, temperature, electroencephalogram, olfactory), recognizing speech or audio (e.g., particular hotwords), and activating buttons or switches, etc. The electronic device <NUM> may also detect and/or classify physical objects in the physical environment of the electronic device <NUM>.

The electronic device <NUM> may be communicatively coupled to a base device such as the electronic device <NUM> and/or the electronic device <NUM>. Such a base device may, in general, include more computing resources and/or available power in comparison with the electronic device <NUM>. In an example, the electronic device <NUM> may operate in various modes. For instance, the electronic device <NUM> can operate in a standalone mode independent of any base device. When the electronic device <NUM> operates in the standalone mode, the number of input modalities may be constrained by power limitations of the electronic device <NUM> such as available battery power of the device. In response to power limitations, the electronic device <NUM> may deactivate certain sensors within the device itself to preserve battery power.

The electronic device <NUM> may also operate in a wireless tethered mode (e.g., connected via a wireless connection with a base device), working in conjunction with a given base device. The electronic device <NUM> may also work in a connected mode where the electronic device <NUM> is physically connected to a base device (e.g., via a cable or some other physical connector) and may utilize power resources provided by the base device (e.g., where the base device is charging the electronic device <NUM> while physically connected).

When the electronic device <NUM> operates in the wireless tethered mode or the connected mode, a least a portion of processing user inputs and/or rendering the extended reality environment may be offloaded to the base device thereby reducing processing burdens on the electronic device <NUM>. For instance, in an implementation, the electronic device <NUM> works in conjunction with the electronic device <NUM> or the electronic device <NUM> to generate an extended reality environment including physical and/or virtual objects that enables different forms of interaction (e.g., visual, auditory, and/or physical or tactile interaction) between the user and the extended reality environment in a real-time manner. In an example, the electronic device <NUM> provides a rendering of a scene corresponding to the extended reality environment that can be perceived by the user and interacted with in a real-time manner. Additionally, as part of presenting the rendered scene, the electronic device <NUM> may provide sound, and/or haptic or tactile feedback to the user. The content of a given rendered scene may be dependent on available processing capability, network availability and capacity, available battery power, and current system workload.

The electronic device <NUM> may also detect events that have occurred within the scene of the extended reality environment. Examples of such events include detecting a presence of a particular person, entity, or object in the scene. Detected physical objects may be classified by electronic device <NUM>, electronic device <NUM>, and/or electronic device <NUM> and the location, position, size, dimensions, shape, and/or other characteristics of the physical objects can be used to provide physical anchor objects to an XR application generating virtual content, such as a UI of an application, for display within the XR environment.

It is further appreciated that the electronic device <NUM> and/or the electronic device <NUM> can also generate such extended reality environments either working in conjunction with the electronic device <NUM> or independently of the electronic device <NUM>.

The network <NUM> may communicatively (directly or indirectly) couple, for example, the electronic device <NUM>, the electronic device <NUM> and/or the electronic device <NUM> with the server <NUM> and/or one or more electronic devices of one or more other users. In one or more implementations, the network <NUM> may be an interconnected network of devices that may include, or may be communicatively coupled to, the Internet.

The electronic device <NUM> may include a touchscreen and may be, for example, a smartphone that includes a touchscreen, a portable computing device such as a laptop computer that includes a touchscreen, a peripheral device that includes a touchscreen (e.g., a digital camera, headphones), a tablet device that includes a touchscreen, a wearable device that includes a touchscreen such as a watch, a band, and the like, any other appropriate device that includes, for example, a touchscreen, or any electronic device with a touchpad. In one or more implementations, the electronic device <NUM> may not include a touchscreen but may support touchscreen-like gestures, such as in an extended reality environment. In one or more implementations, the electronic device <NUM> may include a touchpad. In <FIG>, by way of example, the electronic device <NUM> is depicted as a mobile smartphone device with a touchscreen. In one or more implementations, the electronic device <NUM>, the handheld electronic device <NUM>, and/or the electronic device <NUM> may be, and/or may include all or part of, the electronic system discussed below with respect to <FIG>. In one or more implementations, the electronic device <NUM> may be another device such as an Internet Protocol (IP) camera, a tablet, or a peripheral device such as an electronic stylus, etc..

The electronic device <NUM> may be, for example, desktop computer, a portable computing device such as a laptop computer, a smartphone, a peripheral device (e.g., a digital camera, headphones), a tablet device, a wearable device such as a watch, a band, and the like. In <FIG>, by way of example, the electronic device <NUM> is depicted as a desktop computer. The electronic device <NUM> may be, and/or may include all or part of, the electronic system discussed below with respect to <FIG>.

The server <NUM> may form all or part of a network of computers or a group of servers <NUM>, such as in a cloud computing or data center implementation. For example, the server <NUM> stores data and software, and includes specific hardware (e.g., processors, graphics processors and other specialized or custom processors) for rendering and generating content such as graphics, images, video, audio and multi-media files for extended reality environments. In an implementation, the server <NUM> may function as a cloud storage server that stores any of the aforementioned extended reality content generated by the above-discussed devices and/or the server <NUM>.

<FIG> illustrates an example architecture that may be implemented by the electronic device <NUM> and another electronic device <NUM> (e.g., the handheld electronic device <NUM>, the electronic device <NUM>, the electronic device <NUM>, or another electronic device <NUM>) in accordance with one or more implementations of the subject technology. For explanatory purposes, portions of the architecture of <FIG> is described as being implemented by the electronic device <NUM> of <FIG>, such as by a processor and/or memory of the electronic device; however, appropriate portions of the architecture may be implemented by any other electronic device, including the electronic device <NUM>, electronic device <NUM>, and/or server <NUM>. Not all of the depicted components may be used in all implementations, however, and one or more implementations may include additional or different components than those shown in the figure. Variations in the arrangement and type of the components may be made without departing from the scope of the claims as set forth herein. Additional components, different components, or fewer components may be provided.

Various portions of the architecture of <FIG> can be implemented in software or hardware, including by one or more processors and a memory device containing instructions, which when executed by the processor cause the processor to perform the operations described herein. In the example of <FIG>, an application such as application <NUM> provides application data to a rendering engine <NUM> for rendering of a UI of the application. The application data may include application-generated content (e.g., windows, buttons, tools, etc.) and/or user-generated content (e.g., text, images, etc.), and information for rendering the content in the UI. Rendering engine <NUM> renders the UI for display by a display such as display <NUM> of the electronic device <NUM>.

As shown in <FIG>, another electronic device <NUM>, in communication with electronic device <NUM>, does not have the application <NUM> installed or available. In the example of <FIG>, electronic device <NUM> casts the application (e.g., the UI of the application) to the other electronic device <NUM> by providing remote UI information to the other electronic device <NUM> (e.g., to a rendering engine <NUM> of the other electronic device <NUM>). In the example of <FIG>, rendering engine <NUM> is shown as providing the remote UI information to the other electronic device <NUM>. However, this is merely illustrative, and the remote UI information may be provided by application <NUM> and/or other system processes running at electronic device <NUM>, including system processes that may be implemented before or after the rendering engine <NUM> in the pipeline for providing a UI for application <NUM> to display <NUM>.

The remote UI information that is provided from the electronic device <NUM> to electronic device <NUM> may include one or more images, one or more video streams, and/or one or more other assets associated with one or more elements of the user interface, one or more layer trees that describe the layout and/or appearance of the one or elements of the user interface, and metadata for the user interface. In one or more implementations, the metadata can include the one or more layer trees. For example, when the rendering engine <NUM> generates rendered display frames for display of the UI by display <NUM>, the rendering engine <NUM> may also generate one or more display frames of portions of the UI (e.g., elements of the UI such as text fields, buttons, tools, windows, dynamic content, images, embedded videos, etc.). The display frames of the portions of the UI can be provided as images for static elements, and/or can form corresponding video streams that can be provided to the other electronic device <NUM> for rendering of a version of the UI at by rendering engine <NUM>. In one or more implementations, the electronic device <NUM> may combine one or more of the elements of the UI into a combined video stream. For example, the electronic device <NUM> may determine that two or more elements of the UI are coplanar and partially overlapping, and generate a video stream representing the current view of the two or more coplanar and partially overlapping elements.

Rendering engine <NUM>, application <NUM>, and/or other processes at electronic device <NUM> may also generate, based on the application data, one or more layer trees that describe (e.g., in hierarchical form) how the one or more images, video streams, and/or state information associated with elements of the UI can be combined to form a version of the UI for display <NUM>. Rendering engine <NUM>, application <NUM>, and/or other processes at electronic device <NUM> may also generate metadata that includes timing information for synchronizing the layer trees with the images, video streams, and/or state information (e.g., primitives) of the various portions of the UI, for generation of the version of the UI at the other electronic device <NUM>. In one or more implementations, when the UI displayed at electronic device <NUM> changes (e.g., due to user input at the electronic device <NUM> and/or due to application activity of application <NUM>), one or more new layer trees, or one or more delta (difference) layer trees may be sent to the other electronic device <NUM> to update the version of the UI at the other device.

At the other electronic device <NUM>, rendering engine <NUM> renders the version of the user interface using the one or more layer trees and the one or more video streams, by applying a preference of the second device to at least one of the one or more layer trees, and synchronizing the one or more video streams and the one or more layer trees using the metadata received from the first device. The other electronic device <NUM> then displays the rendered version of the user interface using display <NUM> (e.g., with the one or more video streams and/or one or more other assets displayed according to the instructions in the one or more layer trees). In some use cases, a layer tree may arrive at the electronic device <NUM> before a corresponding asset for a layer of the layer tree. In such a use case, the electronic device <NUM> may display a placeholder at the location of the asset until the asset arrives from the electronic device <NUM>.

In one or more implementations, a layer tree may provide a hierarchy of layers, a tree or graph structure (e.g., a view tree, and/or a scene graph), and/or any other declarative form of describing a UI. For example, a layer tree may include and/or may be associated with a hierarchy of layers that describes each layer of the UI for display. For example, the UI may include content in conjunction with a backdrop, which may utilize one or more blur layers and/or other filter layers. Thus, the tree may include a node and/or subtree that contains one or more attributes describing the blur layer, such as depth, size, placement, and the like. In one or more implementations, rendering engine <NUM> may parse a layer tree to manage the rendering of portions of the UI, such as portions corresponding to individual video streams for the UI. In one or more implementations, the electronic device <NUM> (e.g., the rendering engine <NUM> or a system process at the electronic device <NUM>) may serialize the layer tree and/or one or more assets such as images, video streams, or the like, for transmission to the electronic device <NUM>.

In some use cases, the electronic device <NUM> may determine that the version of the UI that is transmitted for rendering at the electronic device <NUM> can no longer be transmitted as a combination asset(s), layer tree(s), and/or metadata. For example, the electronic device <NUM> may determine that the bandwidth and/or quality of the connection between the electronic device <NUM> and the electronic device <NUM> is insufficient, that computing resources (e.g., power, memory, and/or processing resources) of the receiving device are insufficient, and/or that compatibility issue between the operating systems of the electronic device <NUM> and the electronic device <NUM> may lead to and/or may have led to an invalid state of the UI, or a portion thereof, at the electronic device <NUM>. In a use case in which the electronic device <NUM> determines that the version of the UI that is transmitted for rendering at the electronic device <NUM> can no longer be transmitted as a combination asset(s), layer tree(s), and/or metadata (e.g., due to a reduced capability of the other electronic device <NUM>), the electronic device may switch to a fallback mode in which the UI (e.g., the entire UI) is encoded into a video stream and transmitted to the electronic device <NUM> (e.g., without metadata for reconstruction and/or rendering of the UI at the electronic device <NUM>). In one or more implementations, the electronic device <NUM> may monitor the connection and/or the computing resources of the electronic device <NUM>, and switch back to transmitting the version of the UI as a combination asset(s), layer tree(s), and/or metadata when the connection and/or computing resources are sufficient to avoid an invalid state at the electronic device <NUM>.

In one or more implementations, layers of a layer tree may be associated with depth information that can be translated to a z-coordinate (and/or z-plane) in a three-dimensional coordinate system. In the example of <FIG>, application <NUM> is casted for, for example, display of a UI of the application <NUM> in a two-dimensional scene such as on a display of smartphone, a tablet device, or a computer monitor, or a television.

<FIG> illustrates another implementation, in which additional information is provided for display of the UI of the application <NUM> in a three-dimensional (e.g., XR) scene. In the example of <FIG>, sensors <NUM> provide environment information (e.g., depth information from one or more depth sensors) to an operating system (OS) service, such as OS service <NUM>. In one or more implementations, OS service <NUM> may be a service that is provided by the operating system of the electronic device <NUM>, and that performs operations for generating an XR environment. Camera(s) <NUM> may also provide images of a physical environment to OS service <NUM>. OS service <NUM> may generate three-dimensional scene information, such as three-dimensional map, of some or all of the physical environment of electronic device <NUM> using the environment information (e.g., the depth information and/or the images) from sensors <NUM> and camera(s) <NUM>.

As illustrated in <FIG>, application <NUM> may request an anchor, such as a physical object anchor, from the OS service <NUM> in an anchor request. Application <NUM> may be a gaming application, a media player application, a content-editor application, a training application, a simulator application, or generally any application that provides a UI for display at a location that depends on the physical environment, such as by anchoring the UI to a physical object anchor.

A physical object anchor can be a general physical object such as a horizontal planar surface (e.g., a surface of a floor or a tabletop), a vertical planar surface (e.g., a surface of a wall), or a specific physical object such a table, a wall, a television stand, a couch, a refrigerator, a desk, a chair, etc. Application <NUM> may include code that, when executed by one or more processors of electronic device <NUM>, generates application data, for display of a UI of the application on, near, attached to, or otherwise associated with the physical object anchor.

Once the application data has been generated, the application data can be provided to the OS service <NUM> and/or the rendering engine <NUM>, as illustrated in <FIG>. Environment information such as a depth map of the physical environment, and/or object information for detected objects in the physical environment, can also be provided to rendering engine <NUM>. Rendering engine <NUM> can then render the application data from application <NUM> for display by display <NUM> of electronic device <NUM>. The UI of application <NUM> is rendered for display at the appropriate location on the display <NUM>, to appear in association with the physical anchor object or other anchor provided by OS service <NUM>. Display <NUM> may be, for example, an opaque display, and camera(s) <NUM> may be configured to provide a pass-through video feed to the opaque display. The UI may be rendered for display at a location on the display corresponding to the displayed location of the physical anchor object in the pass-through video. Display <NUM> may be, as another example, a transparent or translucent display. The UI may be rendered for display at a location on the display corresponding to a direct view, through the transparent or translucent display, of the physical anchor object.

As shown, electronic device <NUM> can also include a compositing engine <NUM> that composites video images of the physical environment, based on images from camera(s) <NUM>, for display together with the rendered UI from rendering engine <NUM>. For example, compositing engine <NUM> may be provided in an electronic device <NUM> that includes an opaque display, to provide pass-through video to the display. In an electronic device <NUM> that is implemented with a transparent or translucent display that allows the user to directly view the physical environment, compositing engine <NUM> may be omitted or unused in some circumstances, or may be incorporated in rendering engine <NUM>. Although the example of <FIG> illustrates a rendering engine <NUM> that is separate from OS service <NUM>, it should be appreciated that OS service <NUM> and rendering engine <NUM> may form a common service and/or that rendering operations for rendering content for display can be performed by the OS service <NUM>). Although the example of <FIG> illustrates a rendering engine <NUM> that is separate from application <NUM>, it should be appreciated that, in some implementations, application <NUM> may render content for display by display <NUM> without using a separate rendering engine.

Electronic device <NUM> may allow application <NUM> to request and obtain anchor information from OS service <NUM> (e.g., via an application programming interface, or API) as illustrated in <FIG>, which can facilitate efficient development, implementation, and/or runtime execution of application <NUM> (e.g., since each application <NUM> does not have to do its own object detection, scene mapping, etc.). As shown in <FIG>, when casting an application <NUM> for three-dimensional display by another electronic device such as the other electronic device <NUM>, OS service <NUM> (e.g., or application <NUM> or rendering engine <NUM>) can provide anchoring information (e.g., remote anchor information) for the UI of application <NUM> to the other electronic device <NUM> (e.g., to an operating system (OS) service such as OS service <NUM> and/or to a rendering engine <NUM> at the other electronic device). In one or more implementations, OS service <NUM> may be a service that is provided by the operating system of the electronic device <NUM>, and that performs operations for generating an XR environment. In one or more implementations, the anchoring information may be serialized, along with one or more assets (e.g., images, video streams, etc.), one or more layer trees, and/or other metadata (e.g., including timing information) for transmission from the electronic device <NUM> to the electronic device <NUM>. Although the example of <FIG> illustrates a rendering engine <NUM> that is separate from OS service <NUM>, it should be appreciated that OS service <NUM> and rendering engine <NUM> may form a common service and/or that rendering operations for rendering content for display can be performed by the OS service <NUM>).

As shown in <FIG>, even though application <NUM> is not available at the other electronic device <NUM>, OS service <NUM> (e.g., and/or rendering engine <NUM>) can generate scene information using the remote UI information and the remote anchor information (e.g., in combination with environment information obtained by sensors <NUM> and/or camera (s) <NUM> of the other electronic device <NUM>). As shown, the other electronic device <NUM> may also include a compositing engine <NUM> that composites video (e.g., from camera(s) <NUM> at the other electronic device <NUM>) for display by display <NUM>.

The anchoring information (e.g., remote anchor information) provided from electronic device <NUM> to the other electronic device <NUM> may include information indicating a location at which the other electronic device <NUM> should render the UI corresponding to application <NUM> in the environment (e.g., a physical environment, mixed reality environment, or a virtual environment) of the other electronic device <NUM>. In one example, the anchoring information may include a transform that causes the anchor location for the UI of the other electronic device <NUM> to be similarly positioned relative to an origin in the environment of the other electronic device <NUM> as the UI is positioned by electronic device <NUM> relative to an origin in the environment of the electronic device <NUM>. In operational scenarios in which the electronic device <NUM> and the other electronic device <NUM> are co-located (e.g., in a common or overlapping physical environment), the origin of the electronic device <NUM> and the origin for the other electronic device <NUM> may be at the same location. In operational scenarios in which the electronic device <NUM> and the other electronic device <NUM> are not co-located (e.g., the electronic device <NUM> and the other device are in remote physical environments), the origin of the electronic device <NUM> and the origin for the other electronic device <NUM> may be at different locations.

In one or more implementations, electronic device <NUM> displays a user interface of an application, such as application <NUM>, running on the device at an anchor location in a physical environment of the electronic device. <FIG> illustrates an example in which a user interface <NUM> (e.g., of application <NUM>) is displayed by electronic device <NUM> to appear at a location <NUM> in an environment such as physical environment <NUM> of the electronic device <NUM>. In the example of <FIG>, UI <NUM> includes multiple windows <NUM>, each of which may include one or more elements <NUM>. Elements <NUM> may include text entry fields, buttons, selectable tools, scrollbars, menus, drop-down menus, links, plugins, image viewers, media players, sliders, or the like. In the example of <FIG>, UI <NUM> of another application is also displayed. In one or more implementations, the application corresponding to UI <NUM> may be a shared application that is running on electronic device <NUM> and one or more other electronic devices such as the other electronic device <NUM> of <FIG> and <FIG>.

In the example of <FIG>, UI <NUM> and UI <NUM> are both displayed in the viewable area <NUM> of the display of the electronic device <NUM> to appear, in a three-dimensional environment of electronic device <NUM>, as if they are on a physical wall <NUM> in the physical environment <NUM>. In this example, a physical table <NUM> is also present in the physical environment <NUM>. Display of the UI <NUM> to appear as though on the physical wall <NUM> can be achieved, in part, by defining an anchor location for UI <NUM> at location <NUM> on the physical wall. The anchor location can be defined by detection of the physical wall, and/or relative to an origin <NUM> for electronic device <NUM> in the physical environment <NUM>. For example, electronic device <NUM> may generate and/or store a transform between the origin <NUM> and the anchor location at location <NUM>. In this way, if electronic device <NUM> is moved within the physical environment <NUM>, the display UI <NUM> remains at the anchored location on physical wall <NUM>. In one or more implementations, when electronic device <NUM> and one or more other electronic devices, such as the other electronic device <NUM> of <FIG> and <FIG>, are communicatively coupled, the electronic device <NUM> may share origin information with the other electronic device(s).

In one or more implementations, the user of electronic device <NUM> may desire to share the UI <NUM> of application <NUM> with another user of another device (e.g., another device that is located in the same physical environment <NUM> or an a remote, separate physical environment). In one or more implementations, electronic device <NUM> may determine that the other device (e.g., the other electronic device <NUM> of <FIG> and <FIG>) that is in communication with the device (e.g., and with which the user has indicated a desire to share the UI <NUM>) does not have the application installed. Responsive to determining that the other device does not have the application installed, electronic device <NUM> may provide, to the other device on which the application is not installed, information associated with the user interface <NUM> of the application <NUM>. As discussed herein, the information associated with the user interface <NUM> may include visual display information (e.g., remote UI information as described in connection with <FIG> and/or <NUM>) and anchoring information (e.g., remote anchor information as described in connection with <FIG>) for the user interface <NUM>. For example, the anchoring information may define the anchor location at location <NUM> relative to an origin <NUM> in the physical environment <NUM> of the electronic device <NUM>.

<FIG> illustrates an example of a physical environment <NUM> of the other electronic device <NUM>. In this example, the physical environment <NUM> is separate and remote from the physical environment <NUM> of electronic device <NUM>. In the example of <FIG>, the other electronic device <NUM> has received and is displaying a version <NUM> of the UI <NUM> that is displayed by electronic device <NUM> running the application <NUM>. In this example, the version <NUM> of the UI <NUM> includes versions <NUM> of the multiple windows <NUM> of UI <NUM> of <FIG>, and versions <NUM> of the elements <NUM> of UI <NUM> of <FIG>. The versions <NUM> of the elements <NUM> may be generated based on state information for the element, an image of the element, and/or a video stream of the element as provided by electronic device <NUM>. The arrangement of elements reconstructed from state information, the image(s) corresponding to the elements, and/or video streams corresponding to the elements, to form the versions <NUM> of the windows <NUM> and the version <NUM> of the UI <NUM> overall, may be determined using one or more layer trees and corresponding metadata provided from electronic device <NUM>. For example, the OS service <NUM>, the rendering engine <NUM> and/or the compositing engine <NUM> may parse the one or more layer trees to manage the rendering various elements <NUM> of the UI.

As shown in <FIG>, some of the versions <NUM> of the elements <NUM> appear differently in the version <NUM> of UI <NUM> than the corresponding element <NUM> appear in the UI <NUM> displayed by the electronic device <NUM>. In this example, two of the versions <NUM> of the elements <NUM> are larger in size than the corresponding elements <NUM> of UI <NUM>, and one of the versions <NUM> of the elements <NUM> is the same size (but a different color) as the corresponding UI element <NUM> of UI <NUM>. These differences in parts (but not all) of the UI <NUM> and the version <NUM> can be due to the application of one or more preferences of the other electronic device <NUM> (e.g., user preferences) being applied to one or more layer trees included in the visual display information received from electronic device <NUM>. Other differences between the elements <NUM> and windows <NUM> and the corresponding versions <NUM> and <NUM> can include differences in color, font, font size, theme, orientation with respect to the user, or the like.

In one or more implementations, the other electronic device <NUM>, on which the application <NUM> is not installed, receives, from electronic device <NUM> running application <NUM>, information associated with user interface <NUM> of the application <NUM>, such as while the user interface <NUM> is displayed by the electronic device <NUM> at a first anchor location (e.g., location <NUM>) in a first environment (e.g., physical environment <NUM>) of the electronic device <NUM>. As illustrated in <FIG>, the electronic device <NUM> (e.g., a second electronic device), may render a version <NUM> of the user interface <NUM> using the visual display information from electronic device <NUM>, and may display the version <NUM> of the user interface <NUM> anchored to a second anchor location (e.g., at location <NUM>) that is defined, using the anchoring information, relative to a second origin <NUM> in a second environment (e.g., physical environment <NUM>) of the second device.

As illustrated in <FIG>, the version <NUM> of UI <NUM> that is displayed by the other electronic device <NUM> may appear at a different position within the viewable area <NUM> corresponding to the display of the other electronic device <NUM> (e.g., due to the current position, orientation, etc. of the other electronic device <NUM> and/or the user of the other electronic device <NUM>) than the position of UI <NUM> in the viewable area <NUM> of electronic device <NUM>. However, the version <NUM> of UI <NUM> may be displayed, using the received anchoring information, at the same relative location with respect to the origin <NUM> as the relative location <NUM> of the UI <NUM> with respect to the origin <NUM> (see <FIG>). In the example of <FIG>, the physical environment <NUM> does not include a physical wall <NUM> and includes a different physical table <NUM> at a different physical location.

As shown in the example of <FIG>, because the physical wall <NUM> of physical environment <NUM> does not exist in the physical environment <NUM>, the version <NUM> of UI <NUM> displayed by the other electronic device <NUM> may be displayed to appear as a floating UI.

In the example of <FIG> and <FIG>, a first environment (e.g., physical environment <NUM>) of a first device (e.g., electronic device <NUM>) is remote from a second environment (e.g., physical environment <NUM>) of a second device (e.g., the other electronic device <NUM>), a first origin (e.g., origin <NUM>) is local to the first environment and a second origin (e.g., origin <NUM>) is local to the second environment. In this example, anchoring information that is provided from the first device to the second device may include, for example, a transform that causes a second anchor location of the version <NUM> to be similarly positioned relative to the second origin as the first anchor location for UI <NUM> is positioned relative to the first origin. However, it should also be appreciated that, in some scenarios, the first environment of the first device is the same as the second environment of the second device (e.g., the same physical environment), and the first origin and the second origin are a common origin at a single location.

As discussed herein, the version <NUM> of UI <NUM> that is displayed by the other electronic device <NUM> on which the application <NUM> is not installed may be a non-interactive version of the UI. However, in one or more other implementations, the user of the other electronic device <NUM> may be provided with interactivity with the version <NUM>, such as the ability to move the version <NUM> to a new location, or to resize or rotate the version <NUM>. This interactivity at the other electronic device <NUM> may be independent of the display of the UI <NUM> at electronic device <NUM>, and/or information associated with the interactions can be transmitted to the electronic device <NUM> to cause corresponding movements, resizes, rotations, etc. of the UI <NUM> displayed by the electronic device <NUM>.

For example, in one or more implementations, the other electronic device <NUM> may receive a user input to the version <NUM> of the user interface that is displayed at the other electronic device <NUM>. Responsive to the user input, the other electronic device <NUM> may de-anchor the version <NUM> of the user interface displayed at the other electronic device <NUM> from the corresponding anchor location (e.g., at location <NUM>), and move (and/or resize and/or rotate) the version of the user interface displayed at the second device to a new anchor location in the physical environment <NUM>. In one or more implementations, moving the version <NUM> of the user interface displayed at the other electronic device <NUM> is independent of the display of the user interface <NUM> at the electronic device <NUM>. In one or more other implementations, moving the version <NUM> of the user interface displayed at the other electronic device <NUM> causes a corresponding movement of the user interface <NUM> displayed at the electronic device <NUM> (e.g., using information describing the movement and/or the user input provided from the other electronic device <NUM> to the electronic device <NUM>).

As shown in the example of <FIG>, when the other device <NUM> has the same application installed as the application running on the electronic device <NUM>, and the application is a shared application, both electronic device <NUM> and the other electronic device <NUM> can display the same UI <NUM> of the shared application. This is because, in contrast with the application <NUM> which is only installed on one of the devices, the same application running on both devices can interpret the same application data in the same way to generate local UIs for the shared application at both devices.

Although the example of <FIG> shows UI <NUM> anchored to a physical wall <NUM>, this is merely illustrative, and the UI <NUM> may be initially displayed at, and/or moved by a user of electronic device <NUM> to, other locations. For example, <FIG> illustrates an example in which the UI <NUM> displayed on (e.g., anchored to) the physical table <NUM> in the physical environment <NUM> of electronic device <NUM>. As illustrated in <FIG>, because the physical table <NUM> does not exist in the physical environment <NUM> of the other electronic device <NUM>, the version <NUM> of the UI <NUM> that is displayed by the other electronic device <NUM> may be displayed to appear as a floating UI at the same relative location with respect to the origin <NUM> as the UI <NUM> is displayed with respect to the origin <NUM>.

In the example of <FIG> and <FIG>, a first anchor location of the UI <NUM> corresponds to a physical anchor object (e.g., physical table <NUM>) in a physical environment (e.g., physical environment <NUM>) of the electronic device <NUM>, and a second anchor location for the version <NUM> corresponds to a virtual anchor in a second environment (e.g., physical environment <NUM>) of the other electronic device <NUM>. As illustrated in <FIG>, in some scenarios, when the physical anchor object (e.g., physical table <NUM> in this example) is not available in the physical environment <NUM> of the other electronic device, a virtual anchor object such as virtual table <NUM> may be generated for anchoring of the version <NUM> displayed by the other electronic device <NUM>. In the example of <FIG>, a virtual anchor object has been rendered for display at the virtual anchor for version <NUM>. In this example, the virtual anchor object has a form (e.g., the form of a virtual table <NUM>) that corresponds to a form of the physical anchor object.

As discussed above in connection with <FIG> and <FIG>, in one or more implementations, the user of the other electronic device <NUM> may be provided with interactivity with the version <NUM>, such as the ability to move the version <NUM> to a new location, or to resize or rotate the version <NUM>. This interactivity at the other electronic device <NUM> may be independent of the display of the UI <NUM> at electronic device <NUM>, and/or information associated with the interactions can be transmitted to the electronic device <NUM> to cause corresponding movements, resizes, rotations, etc. of the UI <NUM> displayed by the electronic device <NUM>.

For example, in one illustrative use case, the UI <NUM> of <FIG> may be or include a representation of a chessboard, of a chess application running at the electronic device <NUM>, that is anchored to the physical table <NUM>. In one or more implementations, the anchoring information provided by the electronic device <NUM> to the electronic device <NUM> may be used by the electronic device <NUM> to orient the chessboard in the same orientation as the chessboard that is displayed by the electronic device <NUM>. In this example, a user of the electronic device <NUM> may walk or otherwise move around the displayed version <NUM> of the chessboard to position their self opposite the user of the electronic device <NUM>, even when the user of the electronic device <NUM> and the user of the electronic device <NUM> are in remote locations.

In one or more other implementations, the orientation of the version <NUM> of the chessboard that is displayed by the electronic device <NUM> may be oriented differently from the orientation of the chessboard that is displayed by the electronic device <NUM>. For example, the version <NUM> of the chessboard that is displayed by the electronic device <NUM> may be positioned similarly relative to the origin <NUM> as the positioning of the UI <NUM> relative to the origin <NUM> (e.g., using the anchoring information received from the electronic device <NUM>), but rotated (e.g., based on a preference at the electronic device <NUM>) such that the opposite side of the chessboard faces the user of the electronic device <NUM> than the side of the chessboard that faces the user of the electronic device <NUM>.

In the example use case of a chessboard UI, the other electronic device <NUM> may receive a user input to a version <NUM> of the chessboard that is displayed at the other electronic device <NUM>. For example, the user input may be a gesture input corresponding to lifting the version <NUM> of the chessboard that is displayed by the electronic device <NUM> from the virtual anchor location and placing the version <NUM> of the chessboard that is displayed by the electronic device <NUM> on the physical table <NUM>. Responsive to this user input, the other electronic device <NUM> may de-anchor the version <NUM> of the chessboard displayed at the other electronic device <NUM> from the corresponding anchor location (e.g., at location <NUM>), and move the version of the chessboard displayed at the electronic device <NUM> to a new anchor location associated with the physical table <NUM>. As another example, the user input may be a gesture input corresponding to rotating the version <NUM> of the chessboard that is displayed by the electronic device <NUM> (e.g., so that a desired side of the chessboard faces the user of the electronic device <NUM>). Responsive to this user input, the other electronic device <NUM> may rotate the version <NUM> of the chessboard displayed at the other electronic device <NUM> according to the rotation gesture. In one or more implementations, moving and/or rotating the version <NUM> of the chessboard displayed at the other electronic device <NUM> is independent of the display of the chessboard displayed at the electronic device <NUM>. In one or more other implementations, moving and/or rotating the version <NUM> of the chessboard displayed at the other electronic device <NUM> causes a corresponding movement and/or rotation of the chessboard displayed at the electronic device <NUM> (e.g., using information describing the movement and/or the user input provided from the other electronic device <NUM> to the electronic device <NUM>).

In one or more implementations, some of the user input at the electronic device <NUM> may cause a change in the version <NUM> of the chessboard that is displayed by the electronic device <NUM> without affecting the display of the chessboard displayed by the electronic device <NUM>, and other user inputs to the version <NUM> of the chessboard that is displayed by the electronic device <NUM> may cause a change to the display of the chessboard at the electronic device <NUM>. For example, the user of the electronic device <NUM> may be provided with the ability to rotate the chessboard without affecting the rotation of the chessboard displayed by the electronic device <NUM>, and to move a chess piece on the version <NUM> of the chessboard that is displayed by the electronic device <NUM> causing a corresponding motion of the same chess piece on the chessboard displayed by the electronic device <NUM>.

For example, the electronic device <NUM> may determine (e.g., based on the metadata provided with the version of the UI <NUM> transmitted from the electronic device <NUM>) that a gesture input for rotating the chessboard in not to be transmitted back to the electronic device <NUM>, and that gesture inputs corresponding to moving chess pieces on the chessboard are to be transmitted back to the electronic device <NUM>. In various implementations, the gesture input information corresponding to the chess piece movement may be applied locally to the version <NUM> of the chessboard that is displayed by the electronic device <NUM> by the electronic device <NUM>, and transmitted to the electronic device <NUM> for application to the chessboard displayed by the electronic device <NUM> (e.g., and to the underlying application for game management), or the gesture input information may be transmitted to the electronic device <NUM> for affecting the chessboard displayed by the electronic device <NUM> (e.g., and for updating the underlying application for game management), and then the version <NUM> of the chessboard that is displayed by the electronic device <NUM> may be updated based on updated UI information and/or anchoring information generated by the electronic device <NUM> (e.g., and transmitted from the electronic device <NUM> to the electronic device <NUM>) responsive to the electronic device <NUM> receiving the gesture input that was provided to the electronic device <NUM>.

<FIG> illustrates a flow diagram of an example process for receiving a casted application according to aspects of the subject technology. The blocks of process <NUM> are described herein as occurring in serial, or linearly. However, multiple blocks of process <NUM> may occur in parallel. In addition, the blocks of process <NUM> need not be performed in the order shown and/or one or more blocks of process <NUM> need not be performed and/or can be replaced by other operations.

In the example of <FIG>, at block <NUM>, a version of a user interface for an application is received, from a first device running the application, at a second device on which the application is not installed. In one or more implementations, the version of the user interface includes one or more video streams associated with one or more elements of the user interface. In one or more implementations, the version of the user interface also includes one or more layer trees associated with the one or elements of the user interface, and metadata for the user interface. For example, the first device and the second device may be in communication via a secure wireless connection. In various implementations, the first device and the second device (and/or one or more additional devices) may be participant devices is a video conferencing session or a co-presence session.

In one or more implementations, the one or more elements of the user interface comprise a data editing field and a button. The one or more video streams may include a first video stream corresponding to the data editing field and a second video stream corresponding to the button.

At block <NUM>, the version of the user interface is rendered with the second device, using the one or more layer trees and the one or more video streams. Further details of the rendering of the version of the user interface described hereinafter in connection with, for example, <FIG>.

At block <NUM>, the rendered version of the user interface is displayed with the second device. In one or more implementations, a user input to the version of the user interface that is displayed with the second device may be received, such as by the second device. The version of the user interface displayed at the second device may be modified according to the user input, with or without causing a modification of the user interface displayed at the first device.

For example, modifying the version of the user interface displayed at the second device may include moving, resizing, rotating, or recoloring the version of the user interface displayed at the second device independently of the user interface that is displayed at the first device (e.g., without changing the location, size, rotation, or color of the user interface that is displayed at the first device).

As another example, the second device may transmit, to the first device, information associated with the user input for causing a corresponding modification of the user interface displayed at the first device. The information associated with the user input may include a location, a motion, a direction, a depth, or other information describing the user input, relative to one or more elements of the version of the user input displayed at the second device.

<FIG> illustrates a flow diagram of an example process for rendering a version of a user interface received from another device according to aspects of the subject technology. The blocks of process <NUM> are described herein as occurring in serial, or linearly. However, multiple blocks of process <NUM> may occur in parallel. In addition, the blocks of process <NUM> need not be performed in the order shown and/or one or more blocks of process <NUM> need not be performed and/or can be replaced by other operations.

In the example of <FIG>, at block <NUM>, a preference of the second device is applied to at least one of the one or more layer trees. For example, the preference stored at the second device may include a text size, a font, a color, or a theme that is stored at the second device for display of user interfaces of other applications that are installed on the device. Because the version of the user interface provided by the first device includes the one or more layer trees in addition to the one or more video streams, the second device preferences can be applied to render the version of the user interface differently at the second device than the UI displayed at the first device.

At block <NUM>, the one or more video streams and the one or more layer trees are synchronized using the metadata received from the first device. Synchronizing the one or more video streams and the one or more layer trees using the metadata received from the first device may include synchronizing the one or more video streams and the one or more layer trees in time, using the metadata received from the first device. For example, the metadata may include time information that indicates which frames of each of the one or more video streams is to be included at any given time in the version of the user interface described by the one or more layer trees (which themselves may be changing over time in accordance with changes in the user interface displayed at the first device).

In one or more implementations, at least one of the one or more elements in the user interface displayed at the first device appears differently from the at least one of the one or more elements in the version of the user interface displayed at the second device, due to the application of the preference to the at least one of the one or more layer trees. For example, applying the preference of the second device to at least one of the one or more layer trees comprises modifying a size, a shape, or a color indicated by a portion of one of the one or more layer trees according to the preference, the portion corresponding to one or more of the elements of the user interface. For example, in one exemplary operational scenario, a button or other interaction tool displayed at the second device using a video stream of the button from the first device may be enlarged at the second device (e.g., relative to other elements of the user interface) based on a text size preference of the second device. As another example operational scenario, the button may have a substantially orange color as displayed by the first device and may be modified to have a substantially blue color as displayed by the second device according to a theme or a color preference of the second device.

<FIG> illustrates a flow diagram of an example process for casting an application according to aspects of the subject technology. The blocks of process <NUM> are described herein as occurring in serial, or linearly. However, multiple blocks of process <NUM> may occur in parallel. In addition, the blocks of process <NUM> need not be performed in the order shown and/or one or more blocks of process <NUM> need not be performed and/or can be replaced by other operations.

In the example of <FIG>, at block <NUM>, a device displays a user interface of an application running on the device. The user interface may include one or more elements such as elements <NUM> of the user interface <NUM> of <FIG>.

At block <NUM>, the device determines that another device that is in communication with the device does not have the application installed. Determining that the other device does not have the application installed may include sending a query from the first device to the other device or to a server associated with the device and the other device, and receiving, responsive to the query from the other device or the server, an indication that the application is not available or not installed at the other device. Determining that the other device does not have the application installed may be performed when the application is launched at the device, responsive to a request for application sharing from a user of the application running on the device, or prior to launch of the application (e.g., during a handshake operation and/or during establishment of a communication session between the device and the other device). Determining that the other device does not have the application installed may be performed by the application and/or by one or more background or system processes running on the device and in communication with the application.

At block <NUM>, the device provides, to the other device on which the application is not installed, a version of a user interface that is displayed at the device for the application. The version of the user interface that is provided from the device to the other device includes one or more video streams associated with one or more elements of the user interface, one or more layer trees associated with the one or elements of the user interface, and metadata for the user interface. For example, the one or more video streams include a plurality of video streams each corresponding to one of a plurality of elements of the user interface. The metadata may include time information for synchronization of the one or more video streams and the one or more layer trees.

In one or more implementations, the device receives a user input to the user interface displayed at the device. As examples, the user input may include text input to a text entry field of the UI, a click of a displayed button of the UI, or a resizing of an element (e.g., a sub-window) of the UI). The device may modify the user interface displayed at the device according to the user input (e.g., to display the typed text in the text entry field, to perform the action corresponding to the button, or to resize the element of the user interface). The device may modify at least one of the one or more video streams and the one or more layer trees according to the user input so that the other device can render corresponding modifications to the version of the user interface displayed at the other device. For example, modifying at least one of the one or more video streams and the one or more layer trees according to the user input may include modifying one of the plurality of video streams (e.g., a video stream corresponding to the text entry field or a video stream corresponding to the button), independently of another one of the plurality of video streams.

As described herein, the version of the user interface displayed at the other device may be a non-interactive version of the user interface. However, in some implementations, some interactivity with the version of the user interface may be provided by the other device. For example, in one or more implementations, the device may receive, from the other device, an indication of a user input to the version of the user interface displayed at the other device, and modify the user interface displayed at the device according to the user input to the version of the user interface displayed at the other device.

In one or more implementations, the process <NUM> may also include determining, by the device, a reduced capability (e.g., a reduced bandwidth and/or a reduced computing capability) of other device, ceasing providing the version of the user interface that includes the one or more video streams, the one or more layer trees, and the metadata to the other device, and providing, from the device to the other device, a single video stream that represents the entire user interface that is displayed at the device for the application.

In one or more implementations, the user interface displayed by the device may be displayed in a three-dimensional display such as a mixed reality or virtual reality environment. To allow the other device to display the version of the UI similarly to the UI displayed by the device, anchoring information may also be provided from the device to the other device. For example, displaying the user interface of the application running on the device may include displaying the user interface anchored to a physical anchor in a physical environment of the device, and anchor information for the version of the user interface can be provided to the other device. The anchor information can include, for example, a transform between an origin in the physical environment of the first device and the UI displayed by the first device.

Providing anchoring information to the other device can be helpful, for example, if the user of the device is verbally describing the location of the user interface in their environment or the location of content of an element of the user interface relative to other elements of the user interface, and the user of the other device (e.g., at a remote location) desires to view the UI or element thereof that is being described.

Further features of implementations in which application casting is provided for three-dimensional display are described in connection with <FIG>.

<FIG> illustrates a flow diagram of an example process for receiving a casted application for three-dimensional display according to aspects of the subject technology. The blocks of process <NUM> are described herein as occurring in serial, or linearly. However, multiple blocks of process <NUM> may occur in parallel. In addition, the blocks of process <NUM> need not be performed in the order shown and/or one or more blocks of process <NUM> need not be performed and/or can be replaced by other operations.

In the example of <FIG>, at block <NUM>, a second device on which an application is not installed receives, from a first device running the application, information associated with a user interface of the application displayed by the first device at a first anchor location in a first environment of the first device. For example, the information associated with the user interface may include visual display information and anchoring information for the user interface. The anchoring information may define the first anchor location relative to a first origin in the first environment of the first device.

At block <NUM>, the second device renders a version of the user interface using the visual display information (e.g., remote UI information). Rendering the version of the user interface may include synchronizing one or more video streams in the visual display information with one or more layer trees in the visual display information using metadata included in the visual display information (e.g., by performing one or more of the operations described above in connection with <FIG>).

At block <NUM>, the second device displays the version of the user interface anchored to a second anchor location that is defined, using the anchoring information, relative to a second origin in a second environment of the second device. For example, the anchoring information may include a transform between the first origin and the first anchor location for the user interface. Displaying the version of the user interface anchored to the second anchor location may include determining the second anchor location by applying the same transform relative to the second origin.

In one or more implementations, the first anchor location corresponds to a physical anchor object in a physical environment of the first device, and the second anchor location corresponds to a virtual anchor in the second environment of the second device (e.g., as described above in connection with <FIG>). In one or more implementations, the second device may render and/or display a virtual anchor object at the virtual anchor location (e.g., as described above in connection with <FIG>). For example, the virtual anchor object may have a form that corresponds to a form of the physical anchor object.

In one or more implementations, the first environment of the first device is the same as the second environment of the second device (e.g., the same physical environment), and the first origin and the second origin are a common origin at a single location. In other implementations, the first environment of the first device is remote from the second environment of the second device, the first origin is local to the first environment and the second origin is local to the second environment, and the anchoring information includes a transform that causes the second anchor location to be similarly positioned relative to the second origin as the first anchor location is positioned relative to the first origin.

As described herein, the version of the user interface displayed at the other device may be a non-interactive version of the user interface. However, in some implementations, some interactivity with the version of the user interface may be provided by the other device. For example, in one or more implementations, the second device may receive a user input to the version of the user interface displayed at the second device (e.g., a user input such as a gesture corresponding to grabbing and moving the user interface). Responsive to the user input, the second device may de-anchor the version of the user interface displayed at the second device from the second anchor location, and move the version of the user interface displayed at the second device to a new anchor location in the second environment. In one or more implementations, moving the version of the user interface displayed at the second device is independent of display of the user interface at the first device. In one or more other implementations, moving the version of the user interface displayed at the second device causes a corresponding movement of the user interface displayed at the first device. For example, an indication of the user input may be transmitted from the second device to the first device so that the first device can perform a corresponding movement of the user interface as displayed at the first device.

<FIG> illustrates a flow diagram of an example process for casting an application for three-dimensional display according to aspects of the subject technology. The blocks of process <NUM> are described herein as occurring in serial, or linearly. However, multiple blocks of process <NUM> may occur in parallel. In addition, the blocks of process <NUM> need not be performed in the order shown and/or one or more blocks of process <NUM> need not be performed and/or can be replaced by other operations.

In the example of <FIG>, at block <NUM>, a user interface of an application running on a device is displayed at an anchor location in an environment of the device.

At block <NUM>, the device may determine that another device that is in communication with the device does not have the application installed. Determining that the other device does not have the application installed may include sending a query from the first device to the other device or to a server associated with the device and the other device, and receiving, responsive to the query from the other device or the server, an indication that the application is not available or not installed at the other device. Determining that the other device does not have the application installed may be performed when the application is launched at the device, responsive to a request for application sharing from a user of the application running on the device, or prior to launch of the application (e.g., during a handshake operation and/or during establishment of a communication session between the device and the other device). Determining that the other device does not have the application installed may be performed by the application and/or by one or more background processes running on the device and in communication with the application.

At block <NUM>, the device may provide, to the other device on which the application is not installed, information associated with the user interface of the application. The information associated with the user interface may include visual display information and anchoring information for the user interface. The anchoring information may define the anchor location relative to an origin in the environment of the device. For example, the anchoring information may include a transform between the origin and the anchor location for the user interface. The visual display information (e.g., remote UI information) may include multiple video streams, each corresponding to an element of the user interface. The visual display information may also include metadata that includes time information for rendering of a version of the user interface at the other device using the plurality of video streams.

In one or more implementations, the anchor location in the environment of the device corresponds to a physical anchor object in a physical environment of the device, and the physical anchor object is unavailable in another physical environment of the other device (e.g., as described above in connection with <FIG>). In one or more implementations, the device may identify the physical anchor object responsive to a request from the application for the physical anchor object (e.g., a request for a particular physical object such as a table or a wall, or for a more general physical object such as a vertical plane or a horizontal plane).

As described above, aspects of the subject technology may include the collection and transfer of data from an application to other users' computing devices. The present disclosure contemplates that in some instances, this collected data may include personal information data that uniquely identifies or can be used to identify a specific person. Such personal information data can include demographic data, location-based data, online identifiers, telephone numbers, email addresses, home addresses, data or records relating to a user's health or level of fitness (e.g., vital signs measurements, medication information, exercise information), date of birth, or any other personal information.

The present disclosure recognizes that the use of such personal information data, in the present technology, can be used to the benefit of users. For example, the personal information data can be used in a collaborative setting with multiple users. Further, other uses for personal information data that benefit the user are also contemplated by the present disclosure. For instance, health and fitness data may be used, in accordance with the user's preferences to provide insights into their general wellness, or may be used as positive feedback to individuals using technology to pursue wellness goals.

In addition, policies and practices should be adapted for the particular types of personal information data being collected and/or accessed and adapted to applicable laws and standards, including jurisdiction-specific considerations which may serve to impose ahigher standard.

Despite the foregoing, the present disclosure also contemplates implementations in which users selectively block the use of, or access to, personal information data. That is, the present disclosure contemplates that hardware and/or software elements can be provided to prevent or block access to such personal information data. For example, in the case of sharing information from a particular application, the present technology can be configured to allow users to select to "opt in" or "opt out" of participation in the collection of personal information data during registration for services or anytime thereafter. In addition to providing "opt in" and "opt out" options, the present disclosure contemplates providing notifications relating to the access or use of personal information. For instance, a user may be notified upon downloading an app that their personal information data will be accessed and then reminded again just before personal information data is accessed by the app.

<FIG> illustrates an example computing device with which aspects of the subject technology may be implemented in accordance with one or more implementations. The computing device <NUM> can be, and/or can be a part of, any computing device or server for generating the features and processes described above, including but not limited to a laptop computer, a smartphone, a tablet device, a wearable device such as a goggles or glasses, and the like. The computing device <NUM> may include various types of computer readable media and interfaces for various other types of computer readable media. The computing device <NUM> includes a permanent storage device <NUM>, a system memory <NUM> (and/or buffer), an input device interface <NUM>, an output device interface <NUM>, a bus <NUM>, a ROM <NUM>, one or more processing unit(s) <NUM>, one or more network interface(s) <NUM>, and/or subsets and variations thereof.

The bus <NUM> collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of the computing device <NUM>.

The ROM <NUM> stores static data and instructions that are needed by the one or more processing unit(s) <NUM> and other modules of the computing device <NUM>. The permanent storage device <NUM> may be a non-volatile memory unit that stores instructions and data even when the computing device <NUM> is off.

The input device interface <NUM> enables a user to communicate information and select commands to the computing device <NUM>. The output device interface <NUM> may enable, for example, the display of images generated by computing device <NUM>.

Finally, as shown in <FIG>, the bus <NUM> also couples the computing device <NUM> to one or more networks and/or to one or more network nodes through the one or more network interface(s) <NUM>. In this manner, the computing device <NUM> can be a part of a network of computers (such as a LAN, a wide area network ("WAN"), or an Intranet, or a network of networks, such as the Internet. Any or all components of the computing device <NUM> can be used in conjunction with the subject disclosure.

It is understood that any specific order or hierarchy of blocks in the processes disclosed is an illustration of example approaches. Based upon design preferences, it is understood that the specific order or hierarchy of blocks in the processes may be rearranged, or that all illustrated blocks be performed. Any of the blocks may be performed simultaneously. In one or more implementations, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components (e.g., computer program products) and systems can generally be integrated together in a single software product or packaged into multiple software products.

As used herein, the phrase "at least one of' preceding a series of items, with the term "and" or "or" to separate any of the items, modifies the list as a whole, rather than each member of the list (i.e., each item).

The word "exemplary" is used herein to mean "serving as an example, instance, or illustration". Any embodiment described herein as "exemplary" or as an "example" is not necessarily to be construed as preferred or advantageous over other implementations. Furthermore, to the extent that the term "include", "have", or the like is used in the description or the claims, such term is intended to be inclusive in a manner similar to the term "comprise" as "comprise" is interpreted when employed as a transitional word in a claim.

Claim 1:
A method, comprising:
receiving (<NUM>), from a first device (<NUM>) running an application (<NUM>), at a second device (<NUM>) on which the application is not installed, a version of a user interface that is displayed at the first device for the application,
wherein the version of the user interface comprises:
one or more video streams associated with one or more elements of the user interface,
one or more layer trees associated with the one or more elements of the user interface, and
metadata for the user interface;
rendering (<NUM>) the version of the user interface with the second device using the one or more layer trees and the one or more video streams, by:
applying (<NUM>) a preference of the second device to at least one of the one or more layer trees, and
synchronizing (<NUM>) the one or more video streams and the one or more layer trees using the metadata received from the first device; and
displaying (<NUM>) the rendered version of the user interface with the second device.