Patent Description:
Augmented reality technology aims to bridge a gap between virtual environments and a physical environment by providing an enhanced physical environment that is augmented with electronic information. As a result, the electronic information appears to be part of the physical environment as perceived by a user while the user views the physical environment via the augmented reality technology. However, the electronic information does not persist in the physical environment when not being viewed by the user via the augmented reality technology.

<CIT> describes a system comprising: a display provided with zones; a token; a proximity sensor for detecting the proximity of a token to one zone; a processor to register in a memory an association upon the detection of the proximity of said token to said zone, the association being logged as a set of data comprising the data of which a representation is made on the zone in the proximity of which the token has been brought; the processor being programmed to update a log file with said logged association; the processor performing an operation in accordance with the content of the log file. <CIT> describes an information reading system performing near field communication with an information storage medium. The information reading system includes an antenna portion and an information acquisition section. In the antenna portion, a plurality of antenna coils are each arranged across at least one of the others and each have a reading area. The information acquisition section performs near field communication with the information storage medium in each reading area, thereby reading information from the information storage medium in proximity to the reading area. <CIT> describes a system for interacting with one or more services in a service delivery platform, which comprises one or more examined objects able to hold information indicative for the action and/or the media and one or more examining objects able to retrieve information from the examined objects. The system determines the action based on a combination of pieces of the information. <CIT> describes a system for developing a query, which includes a display device for displaying elements, such as images, textures, color palettes, ad textual elements, which are selectable as query elements for forming a query. A physical space is provided on the display in which a physical object can be positioned. At least one physical object is positioned in the physical space, which is recognized by the system as having an identifier stored in memory to which a query can be linked. A sensor is provided for detecting a physical manipulation of the physical object in the physical space which represents absorbing a query element into a query. A query generator generates a query based on the absorbed query element. The query is thus associated in memory with the identifier for the recognized object. In this way, several physical objects can be manipulated on the display device, each one corresponding to a respective, single query.

A physical environment refers to a physical world that people can sense and/or interact with without aid of electronic devices. The physical environment may include physical features such as a physical surface or a physical object. For example, the physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly sense and/or interact with the physical environment such as through sight, touch, hearing, taste, and smell. In contrast, an extended reality (XR) environment refers to a wholly or partially simulated environment that people sense and/or interact with via an electronic device. For example, the XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and/or the like. With an XR system, a subset of a person's physical motions, or representations thereof, are tracked, and, in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that comports with at least one law of physics. As one example, the XR system may detect head movement and, in response, adjust graphical content and an acoustic field presented to the person in a manner similar to how such views and sounds would change in a physical environment. As another example, the XR system may detect movement of the electronic device presenting the XR environment (e.g., a mobile phone, a tablet, a laptop, or the like) and, in response, adjust graphical content and an acoustic field presented to the person in a manner similar to how such views and sounds would change in a physical environment. In some situations (e.g., for accessibility reasons), the XR system may adjust characteristic(s) of graphical content in the XR environment in response to representations of physical motions (e.g., vocal commands).

There are many different types of electronic systems that enable a person to sense and/or interact with various XR environments. Examples include head mountable systems, projection-based systems, heads-up displays (HUDs), vehicle windshields having integrated display capability, windows having integrated display capability, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones/earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop/laptop computers. A head mountable system may have one or more speaker(s) and an integrated opaque display. Alternatively, a head mountable system may be configured to accept an external opaque display (e.g., a smartphone). The head mountable system may incorporate one or more imaging sensors to capture images or video of the physical environment, and/or one or more microphones to capture audio of the physical environment. Rather than an opaque display, a head mountable system may have a transparent or translucent display. The transparent or translucent display may have a medium through which light representative of images is directed to a person's eyes. The display may utilize digital light projection, OLEDs, LEDs, uLEDs, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium may be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In some implementations, the transparent or translucent display may be configured to become opaque selectively. Projection-based systems may employ retinal projection technology that projects graphical images onto a person's retina. Projection systems also may be configured to project virtual objects into the physical environment, for example, as a hologram or on a physical surface.

Implementations of the subject technology described herein provide for extended reality systems to utilize one or more physical companion devices to which content generated and/or stored within an XR environment can be associated. The systems and methods disclosed herein associate system generated and/or stored content with the physical companion device such that the association persists even when the XR components of the system are idle or powered off. In this way, a user of an XR system can be provided with a portable companion device (sometimes referred to herein as a "stone" or a "go stone") that can be carried with the user to transport content from the XR environment within the physical world. The associated content can then be accessed at a later time by the same XR system when the companion device is detected by the XR system, and/or by other devices and/or XR systems when the companion device is detected by those devices and/or systems. In this way, the user of an XR system can be provided with the ability to carry content, such as a stack of digital photographs, the state of an active document or application, and/or digital currency (as examples) within the physical world for later access by various devices or systems.

In various implementations, the companion device can be an inactive device with a unique identifier that can be detected by a camera of another device, a passive device having a stored unique identifier that can be transmitted using wireless power from another device, or an active device having processing circuitry and/or memory for handling storage and/or transmission of identifiers, content, etc..

<FIG> illustrates an example system architecture <NUM> including various electronic devices that may implement the subject system in accordance with one or more implementations. Not all of the depicted components may be used in all implementations, however, and one or more implementations may include additional or different components than those shown in the figure.

The system architecture <NUM> includes an electronic device <NUM>, a handheld electronic device <NUM>, an electronic device <NUM>, an electronic device <NUM>, a companion device <NUM>, and a server <NUM>. For explanatory purposes, the system architecture <NUM> is illustrated in <FIG> as including the electronic device <NUM>, the handheld electronic device <NUM>, the electronic device <NUM>, the electronic device <NUM>, the companion device <NUM>, and the server <NUM>; however, the system architecture <NUM> may include any number of electronic devices, companion devices, and any number of servers or a data center including multiple servers.

The electronic device <NUM> may be a smartphone, a tablet, or a head mountable portable system (e.g., a head mountable display device that can be worn by a user), that includes a display system capable of presenting a visualization of an extended reality environment to the user. The electronic device <NUM> may be powered with a battery and/or another power supply. In an example, the display system of the electronic device <NUM> provides a stereoscopic presentation of the extended reality environment, enabling a three-dimensional visual display of a rendering of a particular scene, to the user. In one or more implementations, instead of, or in addition to, utilizing the electronic device <NUM> to access an extended reality environment, the user may use a handheld electronic device <NUM>, such as a tablet, watch, mobile device, and the like.

The electronic device <NUM> may include one or more cameras such as camera(s) <NUM> (e.g., visible light cameras, infrared cameras, etc.) Further, the electronic device <NUM> may include various sensors <NUM> including, but not limited to, cameras, image sensors, touch sensors, microphones, inertial measurement units (IMU), heart rate sensors, temperature sensors, Lidar sensors, radar sensors, sonar sensors, GPS sensors, Wi-Fi sensors, near-field communications sensors, radio frequency sensors, etc.) Moreover, the electronic device <NUM> may include hardware elements that can receive user input such as hardware buttons or switches. User input detected by such sensors and/or hardware elements correspond to various input modalities for initiating generating supplemental virtual content within a given extended reality environment. For example, such input modalities may include, but are not limited to, facial tracking, eye tracking (e.g., gaze direction), hand tracking, gesture tracking, biometric readings (e.g., heart rate, pulse, pupil dilation, breath, temperature, electroencephalogram, olfactory), recognizing speech or audio (e.g., particular hotwords), and activating buttons or switches, etc. The electronic device <NUM> may also detect a presence of a person or object and/or an occurrence of an event in a scene to initiate providing supplemental virtual content within the extended reality environment.

The electronic device <NUM> may be communicatively coupled to a base device such as the electronic device <NUM> and/or the electronic device <NUM>. Such a base device may, in general, include more computing resources and/or available power in comparison with the electronic device <NUM>. In an example, the electronic device <NUM> may operate in various modes. For instance, the electronic device <NUM> can operate in a standalone mode independent of any base device. When the electronic device <NUM> operates in the standalone mode, the number of input modalities may be constrained by power and/or processing limitations of the electronic device <NUM> such as available battery power of the device. In response to power limitations, the electronic device <NUM> may deactivate certain sensors within the device itself to preserve battery power and/or to free processing resources.

The electronic device <NUM> may also operate in a wireless tethered mode (e.g., connected via a wireless connection with a base device), working in conjunction with a given base device. The electronic device <NUM> may also work in a connected mode where the electronic device <NUM> is physically connected to a base device (e.g., via a cable or some other physical connector) and may utilize power resources provided by the base device (e.g., where the base device is charging the electronic device <NUM> while physically connected).

When the electronic device <NUM> operates in the wireless tethered mode or the connected mode, a least a portion of processing user inputs and/or rendering the extended reality environment may be offloaded to the base device thereby reducing processing burdens on the electronic device <NUM>. For instance, in an implementation, the electronic device <NUM> works in conjunction with the electronic device <NUM> or the electronic device <NUM> to generate an extended reality environment including physical and/or virtual objects that enables different forms of interaction (e.g., visual, auditory, and/or physical or tactile interaction) between the user and the generated extended reality environment in a real-time manner. In an example, the electronic device <NUM> provides a rendering of a scene corresponding to the extended reality environment that can be perceived by the user and interacted with in a real-time manner. Additionally, as part of presenting the rendered scene, the electronic device <NUM> may provide sound, and/or haptic or tactile feedback to the user. The content of a given rendered scene may be dependent on available processing capability, network availability and capacity, available battery power, and current system workload.

In an implementation, the electronic device <NUM> detects companion device <NUM> in proximity to the electronic device <NUM> (e.g., within range of one or more sensors <NUM> and/or within a field of view of one or more cameras of the electronic device <NUM> such as camera(s) <NUM>). The electronic device <NUM> may detect a unique identifier of companion device <NUM> (e.g., using camera(s) <NUM>, sensors <NUM>, and/or communications circuitry internal to the electronic device <NUM>). The electronic device <NUM> may determine (e.g., via communication with the companion device <NUM>, the server <NUM>, and/or memory at the electronic device <NUM>) whether any content stored at the companion device <NUM>, the server <NUM>, the electronic device <NUM>, and/or any other device is associated with the companion device <NUM>.

For example, if companion device <NUM> is a passive device, electronic device <NUM> may obtain an identifier of the companion device <NUM> and obtain content, previously associated with the companion device <NUM>, that is stored at the electronic device <NUM>, server <NUM>, and/or another device using that identifier. In other examples, a key (e.g., a decryption key) for accessing the content may be obtained from the companion device <NUM> and/or may be obtained and/or derived based on the identifier of the companion device <NUM>. In another example, the electronic device <NUM> may obtain content that is stored at the companion device <NUM>. In other examples, the electronic device <NUM> can assign and/or transfer content that is not yet associated with the companion device to the companion device <NUM>.

The network <NUM> may communicatively (directly or indirectly) couple, for example, the electronic device <NUM>, the electronic device <NUM>, the companion device <NUM>, and/or the electronic device <NUM> with each other device and/or the server <NUM>. In one or more implementations, the network <NUM> may be an interconnected network of devices that may include, or may be communicatively coupled to, the Internet.

The electronic device <NUM> may include a touchscreen and may be, for example, a smartphone that includes a touchscreen, a portable computing device such as a laptop computer that includes a touchscreen, a companion device that includes a touchscreen (e.g., a digital camera, headphones), a tablet device that includes a touchscreen, a wearable device that includes a touchscreen such as a watch, a band, and the like, any other appropriate device that includes, for example, a touchscreen, or any electronic device with a touchpad. In one or more implementations, the electronic device <NUM> may not include a touchscreen but may support touchscreen-like gestures, such as in an extended reality environment. In one or more implementations, the electronic device <NUM> may include a touchpad. In <FIG>, by way of example, the electronic device <NUM> is depicted as a mobile smartphone device with a touchscreen. In one or more implementations, the electronic device <NUM>, the handheld electronic device <NUM>, and/or the electronic device <NUM> may be, and/or may include all or part of, the electronic device discussed below with respect to the electronic system discussed below with respect to <FIG>. In one or more implementations, the electronic device <NUM> may be another device such as an Internet Protocol (IP) camera, a tablet, or a companion device such as an electronic stylus, etc..

The electronic device <NUM> may be, for example, desktop computer, a portable computing device such as a laptop computer, a smartphone, a companion device (e.g., a digital camera, headphones), a tablet device, a wearable device such as a watch, a band, and the like. In <FIG>, by way of example, the electronic device <NUM> is depicted as a desktop computer. The electronic device <NUM> may be, and/or may include all or part of, the electronic system discussed below with respect to <FIG>.

The server <NUM> may form all or part of a network of computers or a group of servers <NUM>, such as in a cloud computing or data center implementation. For example, the server <NUM> stores data and software, and includes specific hardware (e.g., processors, graphics processors and other specialized or custom processors) for rendering and generating content such as graphics, images, video, audio and multi-media files for extended reality environments. In an implementation, the server <NUM> may function as a cloud storage server that stores any of the aforementioned extended reality content generated by the abovediscussed devices and/or the server <NUM>.

<FIG> illustrates a block diagram of various components that may be included in electronic device <NUM> and companion device <NUM> in accordance with aspects of the disclosure. As shown in <FIG>, electronic device <NUM> may include one or more cameras such as camera(s) <NUM> that capture images of the physical environment around the electronic device, one or more sensors <NUM> that obtain environment information associated with the physical environment around the electronic device <NUM>. Sensors <NUM> may include depth sensors (e.g., time-of-flight sensors, infrared sensors, radar, sonar, lidar, etc.), one or more microphones, and/or other types of sensors for sensing the physical environment. Electronic device <NUM> also includes communications circuitry <NUM> for communication with companion device <NUM>, electronic device <NUM>, electronic device <NUM>, servers <NUM>, and/or other devices and/or systems in some implementations. Communications circuitry <NUM> may include radio frequency (RF) communications circuitry for detecting radio frequency identification (RFID) tags, Bluetooth Low Energy (BLE) communications circuitry, other near-field communications (NFC) circuitry, WiFi communications circuitry, cellular communications circuitry, and/or other wired and/or wireless communications circuitry.

As shown, electronic device <NUM> includes processing circuitry <NUM> (e.g., one or more processors and/or integrated circuits) and memory <NUM>. Memory <NUM> may store instructions or code for execution by processing circuitry <NUM>. Memory <NUM> may store (e.g., temporarily or permanently) content generated by and/or otherwise obtained by electronic device <NUM>. In some operational scenarios, memory <NUM> may store identifiers of one or more companion devices such as companion device <NUM> of <FIG> in association with portions of the content stored in memory <NUM>.

Processing circuitry <NUM> may generate display content for display by display <NUM>. The display content may include images of portions of the physical environment and/or computer-generated content such as virtual reality content, augmented reality content, mixed reality content, etc. The display content may include one or more indicators of the content stored in memory <NUM>. For example, when a user opens a digital photograph stored in memory <NUM> with electronic device <NUM>, display <NUM> may display a copy of the image, a thumbnail of the image, etc. As described herein display <NUM> of electronic device <NUM> may be an opaque display the provides a pass-through video view of the physical environment around electronic device <NUM> to display <NUM> (e.g., using images from camera(s) <NUM>) or may include a transparent or translucent display that allows the user of electronic device <NUM> to directly view portions of the physical environment through the display. Display <NUM> may overlay computer-generated (CG) content over the user's view of portions of the physical environment via or through the display.

In order to allow electronic device <NUM> and/or another device to associate content with companion device <NUM>, companion device <NUM> includes an identifier <NUM> that is unique for that companion device and that can be accessed by electronic device <NUM> and/or another device. As one example, the identifier <NUM> may be a code that is printed or attached to a physical body of the companion device. The code, such as an alphanumeric code, an imagebased code, a quick response (QR) code, or generally any code, may be obtained in, and/or derived from, an image captured by camera(s) <NUM> of electronic device <NUM> and/or another device to identify that companion device <NUM>. In one or more implementations, the code may be detectable and/or obtainable using a specific image sensor, such as an infrared (IR) sensor.

In some implementations, companion device <NUM> is an inactive device without any internal circuitry that includes an inactive identifier <NUM>. In this example, identifier <NUM> may be identifiable with a light-based sensor such as a camera, an optical sensor, an IR sensor, or another sensor capable identifying identifier <NUM> using light that is reflected by or emitted by identifier <NUM> at or near the surface of the companion device. However, in other examples, companion device <NUM> may be a passive device with communications circuitry <NUM> that is activated by another device such as by communications circuitry <NUM> of electronic device <NUM>, or an active device having its own processing circuitry <NUM>, memory <NUM>, a battery <NUM>, and/or a display <NUM> (e.g., an e-ink display or other display) housed in the body of the companion device. In implementations in which companion device <NUM> includes a display <NUM>, display <NUM> may be an electronic ink (e-ink) display so that companion device <NUM> can operate with little power. In implementations in which companion device <NUM> includes processing circuitry <NUM> and/or communications circuitry <NUM>, the identifier <NUM> may be stored in memory <NUM> rather than being visible on the exterior of the device body. Information stored in memory <NUM> may be encrypted and/or password protected.

<FIG> illustrates an example of a physical environment in which an electronic device <NUM> and multiple companion devices <NUM> are provided. In the example of <FIG>, a user <NUM> wears an electronic device <NUM> in a physical environment <NUM>. The physical environment <NUM>, in the example of <FIG>, includes a physical object <NUM>, a portion of which can be viewed by user <NUM> via display <NUM> (e.g., based on images from one or more cameras such as camera(s) <NUM> that are provided to an opaque implementation of display <NUM> or directly through a transparent or translucent implementation of display <NUM>) and portions of which can be viewed directly by the user without the use of any technology (if not otherwise blocked from view). In the example of <FIG>, computer-generated content is being displayed by display <NUM> (e.g., overlaid on or in front of portions of physical environment <NUM>). In this example, an application window <NUM> (e.g., a file manager application window, a browser window, a social medial application window, a content editor application window, or any other application user interface) representing an application is displayed by display <NUM>. In this example, the user has also created a stack of photos <NUM> (e.g., by pulling digital images, from a photo library that is accessible by electronic device <NUM>, into a virtual stack).

As indicated in <FIG>, each companion device <NUM> may be a small (e.g., coin sized) physical object that is provided for use with an XR system that includes an XR device such as electronic device <NUM> that allows a wearer or user <NUM> to interact with virtual representations of content such as stored data (e.g., photos, videos, audio files, text files) and/or applications.

As shown in the example of <FIG>, the application window <NUM> and the stack of photos <NUM> are each provided with a location indicator <NUM> displayed by the display <NUM>, the location indicator having an approximate size and shape of one of companion devices <NUM>. In one example, the location indicator <NUM> is provided with a callout icon <NUM> visually linking the location indicator with the stack of photos <NUM> (e.g., digital images). However, <FIG> also shows another example in which a location indicator <NUM> is provided without a callout icon, the proximity to the application window <NUM> indicating the association between the location indicator <NUM> and the application window <NUM>. The location indicators <NUM> indicate a location at which a companion device <NUM> can be placed to associate that companion device with the corresponding content displayed on the display <NUM>.

<FIG> illustrates a scenario in which one of the companion devices <NUM> has been physically moved (e.g., by user <NUM>) to the physical location in physical environment <NUM> that corresponds to the location over which location indicator <NUM> for the stack of photos <NUM> is displayed by display <NUM>. In this scenario, when electronic device <NUM> detects (e.g., using camera(s) <NUM> and/or sensors <NUM>) that companion device <NUM> is at the location corresponding to location indicator <NUM>, electronic device <NUM> associates that companion device <NUM> with the stack of photos <NUM>.

When a companion device <NUM> is associated with particular content (e.g., the stack of photos <NUM> in this example), the companion device <NUM> can store that content (e.g., in memory <NUM>), the companion device <NUM> can store a link or a key to the content stored on another device (e.g., in the cloud at server <NUM>), or (e.g., if the companion device <NUM> is an inactive or a passive device), the association can be stored at the electronic device <NUM> and/or in the cloud (e.g., by storing the unique identifier of the stone in connection with the stored content).

The association generated by electronic device <NUM> between companion device <NUM> and the stack of photos <NUM> can be persistently stored at the electronic device <NUM>, in the cloud (e.g., at one or more of servers <NUM>), and/or at the companion device <NUM>. In this way, even when the companion device <NUM> is no longer in the field of view of camera(s) <NUM> and/or within the proximity of sensors <NUM> and/or communications circuitry <NUM> of electronic device <NUM>, the stack of photos <NUM> can be transported with the companion device <NUM>.

In the example of <FIG> and <FIG>, a device such as electronic device <NUM> that is configured to display computer-generated content overlaid on a view of a physical environment <NUM> is provided in an extended reality system that also includes a companion device that includes a portable body and a unique identifier that is obtainable by the device. As illustrated, the device may display an indicator of stored content (e.g., the stack of photos <NUM> or the state of the application corresponding to application window <NUM>) overlaid on the view of the physical environment. The device may detect the companion device <NUM> in the physical environment <NUM>, and responsive to a user request (e.g., moving the companion device to a location indicator <NUM> in this example) corresponding to the displayed indicator, associate the unique identifier of the companion device with information associated with the stored content.

<FIG> illustrates a scenario in which the companion device <NUM> that has been associated with the stack of photos <NUM> has been moved out of the portion of the physical environment <NUM> that is visible through or via display <NUM>. In this example, the stack of photos <NUM> is no longer displayed on display <NUM>, but is still persistently stored at companion device <NUM>.

<FIG> illustrates how, at a later time, when the companion device <NUM> is again detected by the electronic device <NUM> at any location in the field of view corresponding to the area of display <NUM>, an indicator of the set of photos <NUM> (or the set of photos <NUM> itself) can be displayed at that location. For example, a camera or another sensor of electronic device <NUM> can obtain an identifier of the companion device <NUM> when (or before) the companion device <NUM> enters the field of view corresponding to the area of display <NUM>, obtain the stack of photos, a list of the photos, or thumbnails of the stack of photos (e.g., from local storage at the electronic device <NUM>, from the companion device <NUM>, or from remote storage at a server such as server <NUM>) based on the identity of the companion device <NUM>, and display the indicator of the stack of photos <NUM> and/or the stack of photos <NUM> itself at or near the location at which the companion device <NUM> enters the field of view corresponding to the area of display <NUM>.

Further, and as discussed in more detail hereinafter in connection with <FIG>, if the companion device <NUM> is later brought into proximity with another device (e.g., the user's laptop or an electronic device <NUM> or other device registered to the user or to another user), the companion device <NUM> can provide access to the set of photos <NUM> (e.g., and/or any other content that has been associated with that companion device <NUM>) by the other device.

It should be appreciated that the example described in connection with <FIG> in which the content associated with companion device <NUM> is a stack of photos, this example is merely illustrative and the companion device <NUM> can be associated with any other content in the XR environment. As another example, if the user of the electronic device <NUM> is operating an application such as a social media application associated with application window <NUM>, the electronic device <NUM> can display a location indicator <NUM> associated with that application, at which that companion device <NUM> or another companion device <NUM> can be placed to associate that companion device <NUM> with that social media application (e.g., with the open user interface for that application, with the current state of the application, or any other aspect of the application).

Other examples of content from electronic device <NUM> that can be associated with a companion device <NUM> include, but are not limited to information for tethering apps to the companion device <NUM>, information for associating a digital assistant action to the companion device <NUM>, and/or information for associating a function to the companion device <NUM>. In one example, the electronic device <NUM> may associate a specific function for a specific application or website to a companion device <NUM>. For example, a companion device <NUM> may be associated with an order function or a purchase function for an electronic commerce application. In this way, a user of electronic device <NUM> can tap a physical object in their physical environment (i.e., the companion device <NUM> having been associated with that function) to place an order or make a purchase. Thus, in some implementations, the companion device <NUM> can provide a tactile object with which the user can interact physically while in an XR environment, and for which various different functions can be assigned. This can provide the user with a more intuitive way of interacting with the XR environment than systems in which tactile stimulus is unavailable when interacting with CG content in the XR environment (e.g., in VR, MR, or AR systems in which only gesture or voice control is provided) or in which a controller must be constantly held by the user to control the XR environment).

In yet another example, a group of companion devices <NUM> can be used to physically sort and/or arrange digital content, such as search results obtained using an electronic device <NUM>. In this example, when a search (e.g., a local storage search, a remote storage search, or a network search such as an internet search) is performed by electronic device <NUM> while multiple companion devices <NUM> are in proximity to the electronic device <NUM>, each companion device <NUM> can be associated with one of the multiple companion devices <NUM> to allow the user to physically move and/or select the companion devices <NUM> to sort and/or select search results.

<FIG> illustrate various implementations of a companion device <NUM>, in accordance with aspects of the disclosure. In the example of <FIG>, companion device <NUM> is implemented as a passive device having a physical body <NUM> (e.g., body formed from plastic, rubber, metal, glass, and/or a combination of these and/or other materials) having an identifier <NUM> on an external surface of the physical body <NUM>. In this example, the identifier <NUM> is implemented as a quick response (QR) code on an outer surface of physical body <NUM> that can be recognized by, for example, a camera of another device such as a camera of an electronic device <NUM> or a camera of another electronic device. The QR code can be printed, etched, or attached to an outer surface of the physical body <NUM> (as examples).

<FIG> illustrates another example implementation of companion device <NUM> in which communications circuitry <NUM> is disposed within the physical body <NUM>. For example, the physical body <NUM> may form a housing for companion device <NUM>, the housing having an internal cavity in which the communications circuitry <NUM> is disposed. In this example, the communications circuitry <NUM> may be passive communications circuitry such as an NFC tag or an RFID tag that transmits an identifier of the companion device <NUM> using inductive power from another device, or can be active (e.g., locally powered) communications circuitry <NUM> that can be activated to transmit the identifier.

<FIG> illustrates another example of a companion device <NUM> in an implementation in which the companion device is an active device having additional circuitry, such as processing circuitry <NUM> and/or memory <NUM> provided within the physical body <NUM> along with communications circuitry <NUM>. In this example, memory <NUM> may be used to store an identifier of the companion device <NUM>, an identifier of content stored at another device or server and that is associated with the companion device <NUM>, cryptographic information for access to content stored at another device or server and that is associated with the companion device <NUM>, and/or to store content at the companion device <NUM>. The cryptographic information may be, for example, a key and/or information from which a key may be derived and/or obtained, and/or the cryptographic information may be authorization information, such as corresponding to a software license, that authorizes a proximate device to access to one or more applications, features, and/or functions that are provided by the software license.

As shown in <FIG>, companion device <NUM> can also be implemented with a display <NUM> that can be operated (e.g., by processing circuitry <NUM> within physical body <NUM> and/or by processing circuitry of another device such as electronic device <NUM>) to display an indicator <NUM> of content that is associated with that companion device <NUM> (e.g., and stored at that companion device <NUM>, at electronic device <NUM>, and/or at a remote server such as server <NUM>). In the example of <FIG>, the display <NUM> displays the text "My Photos" to indicate that digital photos of the user have been associated with the companion device <NUM>. In other example, display <NUM> may display a thumbnail of one or more photos, or may display other indicators of other content that has been associated with the companion device.

Providing companion device <NUM> with a display <NUM>, as in the example of <FIG>, may facilitate the use of multiple companion devices <NUM>. For example, a user may be able to carry several companion devices <NUM> in their pocket which they can later easily sort for access to photos, applications, in-progress editing projects (e.g., word processing documents or presentations), currency, and/or other content that has been associated with the companion devices <NUM>. Display <NUM> can, in some implementations, provide a limited functionality user interface for companion device <NUM> (e.g., to allow the user to delete the content or the association with the content (e.g., by pressing and holding the display or a button provided on physical body <NUM>), or to input a password to access the associated content. In any of the various implementations discussed herein, a companion device <NUM> may be provided without any user input components such as buttons, touch interfaces, etc. or any physical connectors (e.g., such that the content associated with the companion device is only accessible by another device and without physically coupling or connecting to the other device). In some implementations, including implementations in which companion device <NUM> includes processing circuitry and memory, companion device <NUM> may have limited functionality (e.g., functionality for storing associated content or storing information for accessing the associated content, without functionality for user interaction, processing data, executing applications, or the like).

In the various examples described herein, physical body <NUM> of companion device <NUM> may have a size that small enough for the companion device <NUM> to be carried in, for example, the palm of a user's hand, or for several of the companion devices <NUM> to be carried in the user's hand or the user's pocket. For example, the physical body may have a maximum dimension of less than two inches, less than one inch, or less than half of an inch. The physical body can have a rounded shape (e.g., a spheroid shape or an ellipsoid shape) as in the examples of <FIG> or can have another shape such as a rectilinear shape. The physical body <NUM> can have a size and a shape that is smaller than the size of a smart phone or a smart watch, and that mimics the size and/or shape of a coin (e.g., a quarter, a nickel, a dime, etc.).

In the examples discussed in connection with <FIG>, content that is accessible by electronic device <NUM> is associated with a companion device <NUM> by placing the companion device <NUM> at a location in physical space that corresponds to the apparent displayed location of a location indicator <NUM>. However, it should be appreciated that content can be associated with a companion device <NUM> in other ways.

<FIG> illustrates an example in which a companion device <NUM> that is not associated with any content is visible within the field of view corresponding to the area of display <NUM>, and in which the stack of photos <NUM> and application window <NUM> have associated location indicators <NUM> (which, it should be appreciated, are also optional).

<FIG> illustrates one additional example in which the stack of photos <NUM> is associated with a companion device <NUM> by placing the companion device <NUM> at a location in the physical environment <NUM> that corresponds to the displayed representation of the stack of photos <NUM> on display <NUM>. In this example, a visual indicator <NUM> (e.g., dashed box surrounding the stack of photos or any other visual highlight or indicator) is displayed by display <NUM> to indicate to the user that the stack of photos <NUM> is being associated with the companion device <NUM>. In some implementations, the user may be presented with an option to approve or decline the association.

<FIG> illustrates yet another example in which the representation of the content itself (e.g., the displayed stack of photos <NUM> in this example) is moved to or toward the location of the companion device <NUM>, as indicated by arrow <NUM> to associate the content with the companion device. For example, the user <NUM> may provide a gesture input to electronic device <NUM> by virtually grabbing or pinching the displayed stack of photos and dragging the displayed stack to the location of companion device <NUM> or throwing or swiping the stack of photos to the companion device <NUM>.

As discussed herein, providing a companion device <NUM> that can be associated with content from an extended reality system can facilitate transfer of the content to another device and/or access to the content by another device.

<FIG> illustrates an example of another electronic device <NUM> implemented as a laptop computer that can interact with companion device <NUM> to access content associated with the companion device <NUM>. As indicated in <FIG>, electronic device <NUM> may obtain an identifier of companion device <NUM> (e.g., using a camera <NUM> to capture an image of a code on an exterior surface of a physical body <NUM> of the companion device <NUM> as in the example of <FIG>, or using NFC communications with communications circuitry of companion device <NUM> to obtain the identifier from communications circuitry of the companion device). The electronic device <NUM> may then obtain associated content of the companion device itself or from another source (e.g., from electronic device <NUM> or from server <NUM>) using the identifier of the companion device <NUM>. In some operational scenarios, electronic device <NUM> may also obtain a key for the content associated with companion device <NUM> directly from companion device <NUM> or from another source using the identity of the companion device <NUM>. In another example, electronic device <NUM> may obtain the content associated with companion device <NUM> directly from the companion device <NUM> itself (e.g., if the associated content is stored in memory <NUM> of the companion device).

In the example of <FIG>, electronic device <NUM> has obtained, using companion device <NUM>, the stack of photos <NUM> and the content for displaying the application window <NUM> as they were previously displayed by electronic device <NUM>, and displayed the same content on display <NUM>. As shown in <FIG>, electronic device <NUM> may also display content <NUM> that has been generated and/or obtained by electronic device <NUM> independently of companion device <NUM>. In some implementations, content <NUM> can also later be associated with companion device <NUM> (e.g., for later user with electronic device <NUM> or for transfer to another user or another device).

<FIG> illustrates a flow diagram of an example process <NUM> for associating content with companion device in accordance with implementations of the subject technology. For explanatory purposes, the process <NUM> is primarily described herein with reference to the electronic device <NUM> and companion device <NUM> of <FIG>, <FIG>, and <FIG>. However, the process <NUM> is not limited to the electronic device <NUM> and companion device <NUM> of <FIG>, <FIG>, and <FIG>, and one or more blocks (or operations) of the process <NUM> may be performed by one or more other components of other suitable devices, including the electronic device <NUM>, the electronic device <NUM>, and/or the servers <NUM>. Further for explanatory purposes, some of the blocks of the process <NUM> are described herein as occurring in serial, or linearly. shown and/or one or more blocks of the process <NUM> need not be performed and/or can be replaced by other operations.

As illustrated in <FIG>, at block <NUM>, a device such as electronic device <NUM> displays an indicator of content (e.g., the stack of photos <NUM>, the application window <NUM>, other computer-generated content, etc.) overlaid on a view of a physical environment. The content may, in various examples, be stored at the device or at a remote device such as another device or a server such as server <NUM>.

At block <NUM>, the device obtains an identifier such as an identifier <NUM> of a companion device such as companion device <NUM> in the physical environment. The device may obtain the identifier by capturing an image of the portable body of the companion device (e.g., using camera(s) <NUM>) or by receiving a wireless communication from the companion device (e.g., from communications circuitry <NUM> to communications circuitry <NUM> of <FIG>).

At block <NUM>, responsive to a user request, the device may associate the identifier of the companion device with information associated with the content. The device may also determine (e.g., using camera(s) <NUM> and/or sensors <NUM>) a location of the companion device <NUM> in the physical environment. Displaying the indicator of the content overlaid on the view of the physical environment at block <NUM> may include displaying the indicator of the content overlaid on another location in the physical environment, and the user request may include a change, by a user, of the location of the companion device or the other location of the indicator (e.g., by a placement of the companion device at a location associated with the displayed indicator of the stored content, or a movement of the displayed indicator of the stored content to or toward a location of the companion device, as described above in connection with, for example, <FIG>, and <FIG>).

Associating the identifier of the companion device <NUM> with the information associated with the content may include storing the identifier of the companion device and the information associated with the content at a remote server, such as server <NUM>, that is separate the companion device and the device, storing the content at the companion device, and/or storing the identifier of the companion device at the electronic device <NUM> in connection with the content. In one example, associating the companion device with the content may include storing the identifier of the companion device at the device in association with the stored content. In some examples, the content can be transmitted to the companion device, and the companion device may store the stored content at the companion device. In some examples, the device may provide a key, and/or information from which a key can be derived, for the stored content to the companion device. The key may be a decryption key, authorization key, or other key that facilitates access and/or authorization to encrypted content, and/or unencrypted content, stored at the companion device, the device, or a remote server such as server <NUM>. The content may be persistently associated with the identifier of the companion device, even after the companion device is removed from a field of view of the device.

In implementations in which the companion device includes a display such as an electronic ink display mounted to the portable body of the companion device, the companion device may also include a processor such as a processor of processing circuitry <NUM> of <FIG>, that operates the electronic ink display. After the content has been associated with the companion device, the processing circuitry may operate the electronic ink display to display another indicator of the stored content associated with the unique identifier (e.g., as shown in <FIG>).

As described above in connection with, for example, <FIG> and <FIG>, the device may detect the companion device at a new location in the physical environment, and display the indicator of the stored content overlaid on the view of the physical environment at the new location (e.g., responsive to detecting the companion device and based on the detected new location of the companion device using sensors and/or cameras of the device).

As described above in connection with, for example, <FIG>, the companion device may provide access to the content by an additional device, such as electronic device <NUM>, separate from the companion device and the device, when the companion device is in proximity to the additional device.

As described above, one aspect of the present technology is the gathering and use of data available from specific and legitimate sources for associating content to a portable companion device. The present disclosure contemplates that in some instances, this gathered data may include personal information data that uniquely identifies or can be used to identify a specific person. Such personal information data can include video data, three-dimensional geometry data, demographic data, location-based data, online identifiers, telephone numbers, email addresses, home addresses, biometric data or records relating to a user's health or level of fitness (e.g., vital signs measurements, medication information, exercise information), date of birth, or any other personal information.

The present disclosure recognizes that the use of such personal information data, in the present technology, can be used to the benefit of users. For example, the personal information data can be used for associating content such as CG content to a portable physical device.

Despite the foregoing, the present disclosure also contemplates embodiments in which users selectively block the use of, or access to, personal information data. That is, the present disclosure contemplates that hardware and/or software elements can be provided to prevent or block access to such personal information data. For example, in the case of associating displayed content to a physical companion device, the present technology can be configured to allow users to select to "opt in" or "opt out" of participation in the collection and/or sharing of personal information data during registration for services or anytime thereafter. In addition to providing "opt in" and "opt out" options, the present disclosure contemplates providing notifications relating to the access or use of personal information. For instance, a user may be notified upon downloading an app that their personal information data will be accessed and then reminded again just before personal information data is accessed by the app.

De-identification may be facilitated, when appropriate, by removing identifiers, controlling the amount or specificity of data stored (e.g., collecting location data at city level rather than at an address level or at a scale that is insufficient for facial recognition), controlling how data is stored (e.g., aggregating data across users), and/or other methods such as differential privacy.

<FIG> illustrates an electronic system <NUM> with which one or more implementations of the subject technology may be implemented. The electronic system <NUM> can be, and/or can be a part of, the electronic device <NUM>, the handheld electronic device <NUM>, the electronic device <NUM>, the electronic device <NUM>, and/or the server <NUM> as shown in <FIG>. The electronic system <NUM> may include various types of computer readable media and interfaces for various other types of computer readable media. The electronic system <NUM> includes a bus <NUM>, one or more processing unit(s) <NUM>, a system memory <NUM> (and/or buffer), a ROM <NUM>, a permanent storage device <NUM>, an input device interface <NUM>, an output device interface <NUM>, and one or more network interfaces <NUM>, or subsets and variations thereof.

In one or more implementations, the processes of the subject disclosure are stored in the system memory <NUM>, the permanent storage device <NUM>, and/or the ROM <NUM> (which are each implemented as a non-transitory computer-readable medium).

Finally, as shown in <FIG>, the bus <NUM> also couples the electronic system <NUM> to one or more networks and/or to one or more network nodes, such as the electronic device <NUM> shown in <FIG>, through the one or more network interface(s) <NUM>. In this manner, the electronic system <NUM> can be a part of a network of computers (such as a LAN, a wide area network ("WAN"), or an Intranet, or a network of networks, such as the Internet. Any or all components of the electronic system <NUM> can be used in conjunction with the subject disclosure.

These functions described above can be implemented in computer software, firmware or hardware. The techniques can be implemented using one or more computer program products. Programmable processors and computers can be included in or packaged as mobile devices. The processes and logic flows can be performed by one or more programmable processors and by one or more programmable logic circuitry. General and special purpose computing devices and storage devices can be interconnected through communication networks.

Some implementations include electronic components, such as microprocessors, storage and memory that store computer program instructions in a machine-readable or computer-readable medium (also referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, read-only compact discs (CD-ROM), recordable compact discs (CD-R), rewritable compact discs (CD-RW), read-only digital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a variety of recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.), magnetic and/or solid state hard drives, read-only and recordable Blu-Ray® discs, ultra density optical discs, any other optical or magnetic media, and floppy disks. The computer-readable media can store a computer program that is executable by at least one processing unit and includes sets of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as is produced by a compiler, and files including higher-level code that are executed by a computer, an electronic component, or a microprocessor using an interpreter.

While the above discussion primarily refers to microprocessor or multi-core processors that execute software, some implementations are performed by one or more integrated circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some implementations, such integrated circuits execute instructions that are stored on the circuit itself.

To provide for interaction with a user, implementations of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; e.g., feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; e.g., by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

A client and server are generally remote from each other and may interact through a communication network.

Claim 1:
A system (<NUM>), comprising:
a device (<NUM>) of an extended reality, XR, system, the device (<NUM>) being configured to display computer-generated content overlaid on a view of a physical environment (<NUM>); and
a companion device (<NUM>), comprising:
a portable body; and
a unique identifier that is obtainable by the device, wherein the companion device (<NUM>) is a physical device associated with content generated within an XR environment created by the XR system, wherein the association persists when components of the XR system are idle or powered off, wherein the companion device (<NUM>) saves cryptographic information to access content stored at another device or server (<NUM>) that is associated with the companion device (<NUM>), and/or stores encrypted content at the companion device (<NUM>),
wherein the device (<NUM>) is configured to:
display an indicator of stored content overlaid on the view of the physical environment (<NUM>);
detect the companion device (<NUM>) in the physical environment (<NUM>); and
responsive to a user request corresponding to the displayed indicator:
associate the unique identifier of the companion device (<NUM>) with information associated with the stored content.