Patent Description:
As another example, augmented reality (AR) merges a direct "real world" view of a physical environment, in which, the user is located with computer-generated output to enhance what is provided to the user and thus "augments" this real-world view. This may include overlaying computer-generated animations as part of this direct view, adding audio to real-world sounds, and so forth.

In both examples, the AR/VR devices may also implement natural user interfaces as a way to support user interaction with AR/VR environments, such as to body movements, hand gestures, head turns, and audible commands and initiate actions based on this user interaction. When interacting with the AR/VR environments, these natural interfaces mimic how the user would interact with the real-world, thus adding to the "immersion" a part of the user experience.

However, in some instances of AR/VR experiences, these natural interfaces can be less efficient, less intuitive, and cumbersome and are limited by the immersion promoted in AR/VR environments. To illustrate, consider an AR/VR environment that interfaces with a service provider system to allow a user to search for a particular product and/or make an in-experience purchase. In some instances, the service provider system requests the user enter a text-based search string, a user identification (ID), a password, and so forth to interact with these services. Conventional AR/VR devices, however, are typically ill configured to support this type of interaction. This may cause the user to first exit the AR/VR environment in order to interface with text entry fields, use the natural interfaces to enter the text-based input, and then return to into the AR/VR environment as part of modal interaction.

This not only disrupts the user's AR/VR experience, but can also cause frustration insofar as the natural interfaces do not allow the user to efficiently enter the text within a context of the AR/VR experience, i.e., is inefficient and/or does not support non-modal inputs. For example, the user may be required in conventional AR/VR devices to verbally spell out the text input one letter at a time using an audible natural interface. As another example, the user may be forced to navigate through a menu system and/or select one letter at a time through hand gestures. Since these interfaces can be less efficient relative to keyboards and/or touch screens, the user may quickly become frustrated and abandon the search or in-experience purchase. Further, this is also computationally inefficient and effects operation of the devices due to the inefficiencies in the user interaction. Thus, while natural interfaces provide an intuitive and instinctive way to interact with augmented and/or virtual worlds, the natural interfaces still suffer from limitations that may cause users to forgo these experiences. Further, <CIT> discloses systems, methods, and media for displaying interactive augmented reality presentations. One of the disclosed systems comprises: a plurality of head mounted displays, a first head mounted display comprising a transparent display; and at least one processor, wherein the at least one processor is programmed to: determine that a first physical location of a plurality of physical locations in a physical environment of the head mounted display is located closest to the head mounted display; receive first content comprising a first three dimensional model; receive second content comprising a second three dimensional model; present, using the transparent display, a first view of the first three dimensional model at a first time; and present, using the transparent display, a first view of the second three dimensional model at a second time subsequent to the first time based one or more instructions received from a server.

Augmented or virtual reality (AR/VR) companion device techniques are described. These techniques are configured to overcome the challenges and inefficiencies of conventional techniques. In one example, companion device pairing to an augmented reality or virtual reality (AR/VR) device techniques and systems are described. These techniques overcome the limitations of conventional techniques by displaying pairing data obtained from a network address associated with a service provider system, and communicatively coupling a companion device with an AR/VR device in response to verifying the pairing data. The companion device receives companion data that can be used to generate a companion user interface that supports user interaction via the companion device to initiate an action as part of an AR/VR environment output by the AR/VR device. In response to receiving user input via the companion user interface to initiate an action, the companion device communicates the action to the AR/VR device to initiate the action.

In a second example, an AR/VR device determines a portion of a markup file that corresponds to an AR/VR scene of a plurality of AR/VR scenes in an AR/VR environment. The AR/VR device communicates the portion of the markup file to the companion device to cause the companion device to configure a companion user interface associated with initiating an action as part of the AR/VR scene. The AR/VR device receives input data from the companion device to initiate the action. The AR/VR device initiates the action for the AR/VR scene based on the input data. In some implementations, the AR/VR device detects a change from the AR/VR scene to another AR/VR scene, and communicates another portion of the markup file to the companion device that corresponds to the other AR/VR scene.

In a third example, a digital image is captured of pairing data displayed by a display device of a companion device using a camera of an AR/VR device. The AR/VR device verifies the pairing data, and pairs with the companion device as communicatively coupled such that the AR/VR device is configured to initiate an action as part of an AR/VR environment based on input data received from the companion device.

In a fourth example, a computing device executes an AR/VR environment, and communicates AR/VR data via a network to an AR/VR device to cause rendering of the AR/VR environment by the AR/VR device. The computing device can alternately or additionally communicate companion data via the network to a companion device that causes the companion device to generate a companion user interface at a display device of the companion device, where the companion user interface is user selectable to initiate an action within the AR/VR environment. In Implementations, the computing device receives protected data from the companion device generated via user interaction with the companion user interface, and executes the action as part of the AR/VR environment without exposing the protected data to the AR/VR device.

In a fifth example, an AR/VR hub receives a request via a network from a companion device for companion data that corresponds to an AR/VR application configured for execution by an AR/VR device. The AR/VR hub locates the companion data from a plurality of companion data stored via the AR/VR hub, and communicates the companion data to the companion device. The AR/VR hub can alternately or additionally synchronize execution of the companion data by the companion device with execution of the AR/VR application by the AR/VR device.

In a sixth example, an AR/VR application and markup files are generated that specify linkages between controls at a companion device and actions for respective scenes of the AR/VR application executed at an AR/VR device. A computing device receives inputs that generate an AR/VR application having a plurality of scenes and corresponding actions as part of user interaction with respective scenes of the plurality of scenes. The computing device can alternately or additionally receive input that generates a markup file that specifies the linkages between controls at a companion device and the actions of the respective scenes of the AR/VR application. Implementations store the markup file as associated with the AR/VR application.

This Summary introduces a selection of concepts in a simplified form that are further described below in the Detailed Description. As such, this Summary is not intended to identify essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

Entities represented in the figures may be indicative of one or more entities and thus reference may be made interchangeably to single or plural forms of the entities in the discussion.

Augmented or virtual reality (AR/VR) devices are configured to immerse a user in an augmented or virtual reality experience by providing a variety of sensory inputs. For example, the AR/VR devices can render visual sensory inputs as 3D imagery that moves with the user, overlay computer-generated images on real-world images, output audio sensory inputs that simulate sounds in an AR/VR environment, deliver tactile feedback that simulates physical interactions in the environment, and so forth. The AR/VR devices can also receive inputs via natural user interfaces that mimic how the user would react to events in the real-world via gestures and motions that do not involve touch. While the natural user interfaces can enhance immersion of the user into the AR/VR experience, the natural user interfaces may encounter limitations regarding some types of inputs, such as text entry. Thus, conventional natural user interfaces may be challenged when trying to interact with service provider systems that involve significant amounts of text entry, such as online purchasing, online authentication, use of a search engine, social networking, and so forth.

To address these challenges, augmented or virtual reality (AR/VR) companion device techniques and systems are described that leverage a companion device in addition to an AR/VR device to increase user interaction within an AR/VR environment. The companion device may thus overcome the challenges of conventional techniques that are limited to an AR/VR device, alone. As part of this, the techniques described herein support pairing of the companion device with the AR/VR device. The techniques also leverage use of a companion user interface by a companion device and communication of portions of a markup file to the companion device that correspond to respective scenes of an AR/VR environment rendered by an AR/VR device. An AR/VR hub system is also described that supports generation of AR/VR application and markup files (e.g., as part of a software development kit) by a developer, synchronization of the companion user interface of the companion device with an AR/VR environment of the AR/VR device, and use of separate communication channels to protect data from exposure between the devices. As a result, the companion device techniques and AR/VR hub system described herein may increase user interaction with increased computational efficiency, e.g., through leveraging hardware and software "strengths" of the individual devices in supporting user interaction with the AR/VR environment, together.

In the pairing example, an AR/VR device pairs with a companion device to provide an alternative input mechanism to the user as part of interacting with an AR/VR environment output by the AR/VR device. Once paired, a user input received via a keyboard at the paired companion device, for instance, may be used to initiate a corresponding action at the AR/VR device, e.g., to enter text. Techniques used to pair an AR/VR device with a companion device encounter a variety of challenges in order to establish and synchronize communications between the devices. In one such challenge, pairing the devices is configured to avoid connecting to the wrong companion device and prevent rogue devices from accessing functionality shared between the paired devices. Conventional techniques to do so, however, require significant amounts of manual data entry by both devices, which, as stated above may be frustrating and inefficient, especially for entry by the AR/VR device.

Another challenge in pairing the devices is to ensure that an input entered at the companion device invokes a correct corresponding action at the AR/VR device. In other words, to ensure that the input at the companion device can be processed by a companion-side application that communicates with an AR/VR-side application and that initiates a desired action by the AR/VR device. Thus, pairing is also used to safeguard against incompatible applications, e.g., ensuring the application executing at the companion device is compatible with the application executing at the AR/VR device. Conventional techniques used to ensure compatibility, however, impact the development of each application, which is further exacerbated by conventional techniques that involve dedicated applications for each type of platform, i.e., operating system and/or device type.

Accordingly, techniques and systems are described that pair an AR/VR device with a companion device to leverage the capabilities at the companion device as a way to supplement functionality at the AR/VR device. Once paired, a user may then interact with a companion user interface to enter an input via a keyboard or touchscreen at the companion device, which is communicated to the AR/VR device to invoke a corresponding action.

In order to pair the device, the AR/VR device displays a network address of a service provider system, via which, pairing data is available. In one example, a generic application of the companion device, such as a browser, is used to access this network address rather than a dedicated companion application. In this way, the pairing data is accessible across a variety of platforms without special coding as opposed to conventional techniques that involve dedicated applications.

The pairing data is then received at the companion device, e.g., as an authentication code, a Quick Response (QR) code, and so forth. The pairing data is displayed on a display device of the companion device, and a digital image is captured using a front-facing camera by the AR/VR device. The AR/VR device may then verify the pairing data, and if valid, pair the companion device as communicatively coupled with the AR/VR device to interact with the AR/VR environment. The communicative coupling may be formed directly between the devices and/or indirectly through use of a service provider system via a network. In this way, the AR/VR device is paired with the companion device with minimal to no manual entry of text by a user with the AR/VR device, further discussion of which may be found in relation to <FIG>.

In another example, techniques and systems described herein also support rendering of the companion user interface by the companion device that is synchronized with rendering of an AR/VR environment of an AR/VR application by the AR/VR device. The AR/VR device, for instance, may execute an AR/VR application that includes multiple AR/VR scenes, where each AR/VR scene has respective actions. AR/VR scenes may include different views within a <NUM>-degree AR/VR environment as well as different <NUM>-degree environments and even to support movement within the environment. The companion user interface supports user interaction to initiate actions as part of the scene being rendered by the AR/VR device, and thus expands functionality of the AR/VR device. The companion user interface, for instance, may include controls that support user interaction to initiate respective actions, such as buttons, 3D models, digital images, spoken utterances, and so forth.

In order to generate the companion user interface, the AR/VR application is associated with a markup file. The markup file in this example is configured using platform independent languages (e.g., Hypertext Markup Language (HTML), Extensible Markup Language (XML)) and thus is usable across a wide range of platforms and device types. For example, this permits use of a general-purpose application (e.g., a browser) to process the markup file and thus may avoid use of platform-specific variations. A browser operating in an iOS® mobile operating system (OS), for instance, may process the same markup file as a browser operating in an Android® mobile OS.

In one instance, the markup file is included as an integral part of the AR/VR application. Therefore, a user downloading the AR/VR application to an AR/VR device automatically downloads the markup file. The bundling of an AR/VR application with a markup file also ensures a compatibility between the AR/VR application and the controls rendered as part of the companion user interface at the companion device, e.g., the controls rendered at the companion device correspond to a correct action at the AR/VR application. In another example, an AR/VR hub system at the service provider system manages the distribution of the markup file, or portions of the markup file, to the companion device, e.g., as part of an "app store.

During execution of the AR/VR application, the AR/VR device may output a variety of AR/VR scenes as part of an AR/VR environment as described above. As the AR/VR application transitions from one AR/VR scene to another AR/VR scene within an AR/VR environment, an AR/VR scene manager module at the AR/VR device analyzes the markup file to determine which portion of the markup file corresponds to a current scene being rendered by the AR/VR device. The AR/VR scene manager module then communicates the portion of the markup file as companion data to a paired companion device to generate the companion user interface.

The AR/VR scene manager module, for example, can determine a first portion of the markup file for a first AR/VR scene and communicate the first portion to the companion device. In changing AR/VR scenes, the AR/VR scene manager module can determine a second portion of the markup file for another AR/VR scene and communicate the second portion to the companion device. Once received, the companion device processes the companion data, e.g. the first portion of the markup file or the second portion of the markup file, to generate the companion user interface. The companion user interface, for instance, may include controls as part of the companion user interface that are linked to initiate an action in a current scene of the AR/VR environment.

In response to receiving user input via a control, for instance, the companion device sends input data describing the corresponding action to the AR/VR device, which is then executed by the AR/VR device as part of the AR/VR environment rendered by the AR/VR application. As one example, the companion device can render a rotation button that corresponds to a current scene in an AR/VR environment. When a user interacts with the rotation button, the companion device transmits a rotation action notification to the AR/VR device, and the AR/VR application executes the corresponding action. In this way, the companion user interface may be rendered as platform independent efficiently by computational and network resources of the companion device by communicating corresponding portions of the markup file. Further discussion of this example may be found in relation to <FIG> and <FIG>.

In a further example, an AR/VR hub system is described to manage creation of the AR/VR application and markup file as well as synchronization of interaction of the companion user interface with the AR/VR environment of the AR/VR application executed by the AR/VR device. The AR/VR hub system, for instance, may include an AR/VR application developer system that includes a software development kit (SDK) usable to code the AR/VR application as well as the companion data as the markup file. As part of this, linkages are specified between control of the markup file and actions of the AR/VR application and also linkages between scenes of the AR/VR environment and portions of the markup file to be communicated to the companion device. In this way, the markup file may be jointly created along with the AR/VR application, thereby improving user and computational efficiency and may do so in a platform independent format.

Conventional application development techniques, on the other hand, generate multiple variations of dedicated companion applications to support different operating systems. This adds complexity to the development cycle since adding an action to the AR/VR application corresponds to multiple changes in the variations of the dedicated companion application, e.g. a first change in the first variation, a second change in the second variation, and so forth. The conventional techniques are also susceptible to introduction of errors if a change in the first variation does not align with a change in the second variation.

Conversely, with the use of markup file via a platform independent format as supported by the techniques described herein, a developer may add a new action to a scene of the AR/VR application and simply modify a single markup file to support the new action through interaction with the SDK. When distributed, the single markup file synchronizes the control rendered at different types of companion devices to the new action, thus not only synchronizing the AR/VR application with the companion device, but synchronizing multiple platforms as well. Further discussion of this example may be found in relation to <FIG>.

In yet another example, the AR/VR hub system is used to manage the companion data used to generate the companion user interface by the companion device. For instance, a companion device may establish a connection to the AR/VR hub system and exchange communications with the system to facilitate the distribution of the markup files to the companion device to generate the companion user interface. The markup files in this example may originate from the AR/VR device and/or may be maintained by the AR/VR hub system, itself. This may also include communication of input data generated by user through interaction with the companion user interface to initiate the action by the AR/VR device. Distributing the AR/VR application and corresponding markup files via a hub ensures compatibility and synchronization between the AR/VR application at an AR/VR device and controls rendered at a companion device. In this way, the AR/VR hub system may also be used to synchronize the companion user interface of the companion device with the AR/VR application executed by the AR/VR device. Further discussion of this example may be found in relation to <FIG>.

In yet a further example, separate communication channels are supported between the AR/VR device and the companion device to protect against exposure of potentially sensitive data. Conventional communication techniques, one the other hand, may expose the potentially sensitive data, even when entered using separate input mechanisms that cause replication of the sensitive data and communication of the data through one of the devices. A conventional companion device, for instance, may be configured to communicate with a website as part of an AR/VR environment by communicating "through" an AR/VR device, or vice versa. Accordingly, in either scenario sensitive data may be exposed between the devices. A first input mechanism, for example, may reside at the AR/VR device that requests payment information or user credentials. A second input mechanism can similarly reside at the companion device to request the same sensitive data. The input mechanisms at multiple devices causes a duplication of the sensitive data, thus increasing the risk of exposure.

Conversely, through the use of separate secure communication channels to the service provider system as described in this example, sensitive data resident at the companion device can be utilized by the AR/VR device without duplicating the data, thus decreasing the risk of exposure. An AR/VR device, for instance, may establish a first communication channel (e.g., an AR/VR device communication channel) with a service provider system that is employed as part of an AR/VR environment (e.g., a webpage), while a companion device establishes a second communication channel (e.g. a companion device communication channel) with the service provider system. In this way, each of the devices may communicate with the service provider system without exposing potentially sensitive data to each other.

For example, a companion device may communicate protected data via a companion device communication channel to initiate a variety of actions at the service provider system without exposing this information to the AR/VR device. The protected data, for instance, may include a user credentials (e.g., user name and password) that are used to login to a web service implemented by the service provider system. Once verified, the service provider system may then permit access by the AR/VR device, thereby resulting in increased user efficiency in entering these credentials. In this way, the credentials remain secure and the AR/VR device may then join the AR/VR environment in an intuitive and efficient manner. In another example, the protected information includes payment information (e.g., from an electronic "wallet" maintained at the companion device to complete a purchase with the service provider system without sharing that information with the AR/VR device nor without requiring the AR/VR device to enter this information, itself, which may be complicated using a natural user interface.

The AR/VR hub system may also include an AR/VR environment manager module that synchronizes actions and communications between the companion device and the AR/VR device. The AR/VR device, for instance, can transmit companion data over the first communication channel to the AR/VR hub system. The companion device receives the companion data over the second communication channel, and processes the companion data to render a control in a companion user interface associated with inputting sensitive data at a companion user interface, such as a purchase action button, a payment selection drop-down menu, a biometric entry field, a password text field, and so on.

When a user interacts with the control, the companion device performs corresponding functionality that accesses and/or transmits protected data over the second communication channel to the service provider system. The AR/VR environment manager module receives the protected data, and completes a corresponding action, such as completing a purchase at the service provider system using the protected data. Further, the AR/VR environment manager AR/VR hub system can perform the corresponding action without exposing the protected data to the AR/VR device.

An AR/VR application resident at an AR/VR device, for instance, can forward companion data over the first communication channel that corresponds to rendering a purchase action control for an item displayed in an AR/VR environment. The companion device receives the companion data over the second communication channel, and processes the companion data to render the purchase action control within a companion user interface. Upon selection of the purchase action control, the companion device can then forward payment information to an AR/VR environment manager module that completes the purchase of the item without exposing the payment information to the AR/VR device. This technique can also be applied to protected data resident at the AR/VR device. The AR/VR device, for example, can forward protected data over the first communication channel to the AR/VR environment manager module for consumption and without exposing the protected data to the companion device as described above.

In other examples, the AR/VR environment manager module forwards protected data received from a companion device over to the AR/VR device for use, such as for auto-populating fields displayed in an AR/VR environment at the AR/VR device. This mechanism thus reduces the exposure of the protected data relative to conventional techniques insofar as it reduces human interaction with the protected data, thus reducing how the protected data is exposed. Further discussion of these examples may be found in relation to <FIG> and <FIG>.

In the following discussion, an example environment is first described that may employ the techniques described herein. Example procedures and systems are also described and shown as blocks which may be performed in the example environment as well as other environments. Consequently, performance of the example procedures is not limited to the example environment and systems and the example environment and systems are not limited to performance of the example procedures.

<FIG> is an illustration of a digital medium environment <NUM> in an example implementation that is operable to employ augmented reality (AR) digital content companion device techniques described herein. The illustrated environment <NUM> includes an AR/VR device <NUM>, a companion device <NUM>, and a service provider system <NUM> that are communicatively coupled, one to another, via a network <NUM>.

Computing devices that implement the AR/VR device <NUM>, companion device <NUM>, and service provider system <NUM> may be configured in a variety of ways. A computing device, for instance, may be configured as a desktop computer, a laptop computer, a mobile device (e.g., assuming a handheld configuration such as a tablet or mobile phone as illustrated for the companion device <NUM>), configured to be worn (e.g., as a head as shown for the AR/VR device <NUM>) and so forth. Thus, a computing device may range from full resource devices with substantial memory and processor resources (e.g., personal computers, game consoles) to a low-resource device with limited memory and/or processing resources (e.g., mobile devices). Additionally, although a single computing device is shown and described in some instances, a computing device may be representative of a plurality of different devices, such as multiple servers utilized by a business to perform operations "over the cloud" for the service provider system <NUM> as described in <FIG>.

The AR/VR device <NUM> is illustrated as including an augmented or virtual reality (AR/VR) application <NUM> that is executable to support user interaction with an AR or VR environment. In augmented reality, digital content is rendered along with a user's view (direct or indirect) of a physical environment, in which the user is disposed. Thus, the digital content rendered by the AR/VR application in an AR/VR user interface (e.g., visual, audio, tactile) "augments" the user's view of the physical environment. In virtual reality, an entirety of a user's view is rendered by digital content as a virtual reality environment. Thus, the virtual reality environment replaces a user's view of a physical environment, in which, the user is disposed. In the following, "AR/VR" refers to either augmented reality or virtual reality. The AR/VR device may be configured in a variety of ways to support rendering of an AR/VR environment, including devices used to provide an immersive experience such as goggles glasses, handheld devices such as a mobile phone or tablet, and so on.

Through the use of sensors (e.g., accelerometers, processing of digital images captured by a digital camera), the AR/VR device <NUM> may determine "where a user is looking" and render a corresponding scene of an AR/VR environment. This is done in <FIG> through execution of an AR/VR application <NUM> by the AR/VR device <NUM>. The AR/VR application <NUM> also includes a markup file <NUM> that is usable by a communication module <NUM> of the companion device <NUM> to generate a companion user interface <NUM>. The companion user interface <NUM>, for instance, may be rendered to include controls <NUM> that are configured to initiate corresponding actions of an AR/VR environment rendered by the AR/VR device <NUM>. In this way, the companion user interface <NUM> may expand functionality available via the rendering of the AR/VR device.

The following sections describe a variety of techniques that support use of the companion device <NUM> and companion user interface <NUM> in operation with the AR/VR device <NUM>. Pairing techniques are described in relation to <FIG>. Communication of portions of the markup file <NUM> to the companion device <NUM> that relate to a current scene being rendered by the AR/VR device <NUM> are described in relation to <FIG> and <FIG>. Creation of the AR/VR application <NUM> and markup file <NUM> through interaction with a software development kit received from an AR/VR hub system <NUM> is described in relation to <FIG> and <FIG>. Dissemination and synchronization of the markup file <NUM> by the AR/VR hub system <NUM> is described in relation to <FIG> and <FIG>. Use of separate communication channels to protect data is described in relation to <FIG> and <FIG>.

In general, functionality, features, and concepts described in relation to the examples above and below may be employed in the context of the example procedures described in this section. Further, functionality, features, and concepts described in relation to different figures and examples in this document may be interchanged among one another and are not limited to implementation in the context of a particular figure or procedure. Moreover, blocks associated with different representative procedures and corresponding figures herein may be applied together and/or combined in different ways. Thus, individual functionality, features, and concepts described in relation to different example environments, devices, components, figures, and procedures herein may be used in any suitable combinations and are not limited to the particular combinations represented by the enumerated examples in this description.

<FIG> depicts an example <NUM> showing operation of the AR/VR device <NUM> of <FIG> as pairing with the companion device <NUM> over the network <NUM>. <FIG> illustrates an example <NUM> that, in some implementations, can be considered a continuation of the example <NUM>, where the AR/VR device <NUM> establishes a communication channel with the companion device <NUM> to exchange companion data. <FIG> depicts an example <NUM> showing operation of the companion device <NUM> receiving input at a user interface and forwarding the input to the AR/VR device to invoke actions at the AR/VR device. <FIG> depicts a procedure <NUM> in an example implementation in which an AR/VR device pairs with a companion device through the use of a service provider. <FIG> depicts a procedure <NUM> in an example implementation in which a companion device generates a companion user interface that can be used to initiate actions in an AR/VR environment at an AR/VR device. The companion device receives input through the companion user interface, and communicates the input to the AR/VR device to initiate a corresponding action for the AR/VR environment.

The following discussion describes techniques that may be implemented utilizing the previously described systems and devices. Aspects of the example <NUM> as shown stepwise by the modules of <FIG> may be implemented in hardware, firmware, software, or a combination thereof. In portions of the following discussion, reference is made interchangeably to <FIG>. Further, <FIG> depicts the example <NUM> of pairing an AR/VR device with a companion device through use of first, second, and third stages <NUM>, <NUM>, and <NUM>.

To begin this example, at the first stage <NUM>, the AR/VR device <NUM> displays a network address of a service provider system, via which, pairing data is available (block <NUM>). To demonstrate, consider an example in which the user has downloaded an AR/VR application <NUM> to the AR/VR device <NUM>, such as an AR/VR application that simulates navigation through an online marketplace. AR/VR scenes of the corresponding AR/VR environment output by the AR/VR device can include various types of actions associated with the online marketplace, such as item searching, item selection, item comparison, bid submissions, purchase transactions, item rotation, user preference configuring, auction updates, category browsing, help support, and so on. These actions can pertain to a particular AR/VR scene and/or multiple AR/VR scenes. For example, a searching action may be accessible on multiple AR/VR scenes, an item rotation action may only be accessible on an item view AR/VR scene, an auction update action may only be accessible on a main AR/VR scene, etc. Thus, each AR/VR scene can be associated with a unique combination of actions relative to other scenes.

In implementations, the user initiates pairing of the AR/VR device <NUM> with a companion device <NUM> to access input mechanisms directed to the action at the companion device that provide easier access to the action relative to the natural interfaces of the AR/VR device <NUM>, such as a keyboard at the companion device <NUM> to enter a text-based search string, access functionality specific to the companion device <NUM> such as payment information, and so forth. To pair the AR/VR device <NUM> with the companion device <NUM>, the user manually invokes the pairing process in one example, such as through audible input to the AR/VR device <NUM>, through an input gesture at the AR/VR device <NUM>, navigating through a menu of the AR/VR application <NUM>, and so on. Alternately or additionally, the AR/VR device <NUM> may detect the presence of a companion device <NUM> and initiate the pairing process itself, i.e., automatically and without user intervention.

In initiating the paring process, the AR/VR device <NUM> displays the network address <NUM> that is associated with the service provider system <NUM> of <FIG>. In the illustrated example <NUM>, the AR/VR device <NUM> displays the network address <NUM> using a same display device that provides visual experiences to the user, e.g. the same display device used to provide visual experiences associated with an AR/VR application <NUM>. The network address <NUM> can be displayed for a predetermined amount of time, until a timeout occurs, or until a successful pairing to a companion device <NUM> occurs. In another example, the AR/VR device <NUM> displays the network address <NUM> until user input associated with terminating the paring process is received. Here, the AR/VR device <NUM> displays the network address <NUM> concurrently with an activation code (e.g., "AR12").

At the second stage <NUM>, the user manually enters network address <NUM> into a communication module <NUM> (e.g., a browser) of the companion device <NUM> (block <NUM>), such as through the use of a keyboard <NUM>. The network address <NUM> can be input at the companion device <NUM> via other input mechanisms as well, such as touch screen input, audible input, and so forth. In alternate or additional implementations, the AR/VR device <NUM> automatically communicates the network address <NUM> to the companion device <NUM>, such as by broadcasting and/or transmitting the network address <NUM> using a wireless signal to the companion device <NUM> as network address <NUM>. This can occur simultaneously with, and/or in lieu of, displaying the network address <NUM> at the AR/VR device <NUM>.

Once the broadcast is detected and the network address is received, the companion device <NUM> can automatically enter the network address as navigation input to the communication module <NUM>, e.g., the browser. Whether manually or automatically entered, the browser of the companion device <NUM> navigates to, and/or establishes a connection with, the service provider system <NUM> via the network <NUM> based on the network address <NUM>. Here, the browser represents a general-purpose network navigation application, rather than a dedicated AR/VR companion application synchronized to the AR/VR application executing on the AR/VR device, but in alternate implementations, a dedicated AR/VR companion application can be utilized.

Moving to the second stage <NUM>, the companion device <NUM> transmits authentication data to the network address and/or the service provider system <NUM> over the network <NUM> of <FIG>. The browser of the companion device <NUM>, for example, can render an input field that requests an activation code <NUM> based on instructions retrieved from the network address <NUM>. In turn, the user can manually enter the activation code <NUM> into the text field (e.g., "AR12") based on the information displayed at the AR/VR device <NUM>. Alternately or additionally, the AR/VR device <NUM> wirelessly transmits the activation code <NUM> to the companion device <NUM> over a local connection, such as over a Bluetooth connection, and the companion device <NUM> automatically populates the text field of the browser with the activation code <NUM>. In response to receiving the activation code <NUM> in the text field, the browser transmits the activation code <NUM> to the service provider system <NUM>.

The service provider system <NUM> analyzes the activation code <NUM> to authenticate the companion device <NUM>. For instance, an AR/VR hub system <NUM> that resides at the service provider system <NUM> can include a pairing module <NUM>, and use the pairing module <NUM> to authenticate the activation code <NUM>. The AR/VR hub system <NUM>, for example, can maintain a list of valid activation codes, distributed activation codes, software revision compatibility associated with activation code, and so forth. The pairing module <NUM> can then compare the received activation code with the maintained list to determine whether the activation code is valid or invalid. If the pairing module <NUM> determines that the activation code <NUM> is invalid, the pairing process terminates. However, in response to authenticating the companion device <NUM>, the pairing module <NUM> determines to transmit pairing data <NUM> back to the companion device.

Moving to the third stage <NUM>, the companion device <NUM> obtains pairing data <NUM> from the network address (block <NUM>). As one example, the pairing module <NUM> generates a QR code as the pairing data <NUM> used between the companion device <NUM> and the AR/VR device <NUM>. In other examples, the pairing module <NUM> can generate a digital certificate, a second authentication code, a verification code, a private/public key pair, a security question, a software token, and so forth, as the pairing data. In turn, the service provider system <NUM> transmits the pairing data <NUM> to the companion device <NUM> over the network <NUM>.

<FIG> continues the example <NUM> of <FIG> also through use of first, second, and third stages <NUM>, <NUM>, <NUM>. Aspects of the example <NUM> as shown stepwise by the modules of <FIG> may be implemented in hardware, firmware, software, or a combination thereof. At the first stage <NUM>, the companion device <NUM> displays the pairing data <NUM> obtained from the network address on a display device <NUM> of the companion device <NUM> (block <NUM>). For example, the browser can receive the network address <NUM> from the service provider system <NUM> as described with reference to <FIG>, and render the pairing data <NUM> with instructions to the user, such as directions to input the pairing data into the AR/VR device <NUM>. In this example, the browser renders the pairing data <NUM> on a display device <NUM> as a QR code, but other types of pairing data can be utilized, such as a second activation code, a password, a key, text, digital images, and so forth.

In implementations, the AR/VR device <NUM> captures a digital image of the pairing data <NUM> using a digital camera <NUM> (block <NUM>), where the digital camera <NUM> corresponds to a front-facing camera of the AR/VR device <NUM>. In alternate or additional implementations, the user can enter the pairing data using the natural user interfaces of the AR/VR device <NUM>, such as audibly spelling a passcode. In response to capturing the digital image, the AR/VR device <NUM> verifies the pairing data (block <NUM>). For example, the QR code can embed a key in the image. The AR/VR device <NUM> then analyzes the captured image of the QR code, and extracts the key which is then utilized to unlock companion device pairing features. The AR/VR device <NUM> alternately or additionally establishes a connection to the service provider system <NUM> over the network <NUM> of <FIG> using information embedded in the QR code. Thus, in this implementation the AR/VR device <NUM> forwards the pairing data to the service provider system <NUM> for verification.

Moving to the second stage <NUM>, implementations pair the AR/VR device <NUM> as communicatively coupled with the companion device <NUM> in response to verifying the pairing data (block <NUM>). For example, the AR/VR device <NUM> and the companion device <NUM> can establish an AR/VR communication channel <NUM> over the network <NUM>, such as through the service provider system <NUM>, through a local wireless connection, through a local wired connection, and so on. The AR/VR device <NUM> and companion device <NUM> can also employ cryptography to generate a secured communication channel between the devices, where encrypting and decrypting information (i.e., encryption keys) is embedded in the pairing data.

Some implementations prompt the user for input to establish the AR/VR communication channel <NUM>. The AR/VR device, for instance, can display a menu that prompts the user to confirm an identity and/or type of the computing device, such as a menu that prompts the user to confirm whether the companion device <NUM> is an iPhone® <NUM>, and iPhone® <NUM>, a Samsung Galaxy® s10, and so forth. This can include the user entering input that selects a particular one of multiple devices and/or input that confirms a detected device type. While described in terms of a computing device type, alternate or additional implementations prompt the user to confirm other types of information, such as user identity, location identity, application identifiers, application revisions, and so forth. Thus, in establishing the AR/VR communication <NUM>, various implementations prompt the user for additional input as part of a validation process, and/or provide additional information used, to establish the AR/VR communication channel <NUM>.

At the third stage <NUM>, the companion device <NUM> receives, in response to the pairing, companion data <NUM> from the AR/VR device (block <NUM>). Here, the AR/VR device <NUM> communicates the companion data <NUM> to the companion device <NUM> over the AR/VR communication channel <NUM>. The companion data <NUM> generally represents data that describes actions, controls and/or digital content associated with an AR/VR scene that is currently active at the AR/VR device, such as markup language that includes linkages to the actions. The markup language, for instance, may specify the actions that are used by the companion device <NUM> to select controls <NUM> for output in the companion user interface <NUM>. In another instance, the markup language identifies the controls <NUM>. Referring to the example of the online marketplace AR/VR application <NUM>, the companion data can correspond to instructions that cause the companion device <NUM> to render a rotation action control in the companion user interface <NUM> for an item AR/VR scene at the AR/VR device <NUM>, an auction compare control for a main AR/VR scene at the AR/VR device, and so forth.

In implementations, the AR/VR application <NUM> of <FIG> includes an AR/VR scene manager module <NUM> that identifies which AR/VR scene of the multiple scenes included in the AR/VR application <NUM> is currently active, e.g. the item scene, the main scene. In turn, the AR/VR scene manager module <NUM> selects the companion data <NUM> that corresponds to the currently active scene at the AR/VR device. The AR/VR scene manager module <NUM>, for instance, can parse through a markup file <NUM> that is included as part of the AR/VR application <NUM> to identify a portion of the markup file <NUM> that corresponds to the currently active AR/VR scene as further described in relation to <FIG> and <FIG>. As another example, the AR/VR scene manager module <NUM> selects a particular markup file of companion data <NUM> from multiple markup files included as part of the AR/VR application <NUM>, where the particular markup file includes the companion data <NUM> corresponding to the currently active scene. The companion data <NUM>, as further described herein, includes instructions on rendering controls that are usable to initiate actions at the companion device <NUM>, where the actions are performed by the AR/VR device <NUM>.

In response to receiving the companion data <NUM>, the companion device <NUM> generates a companion user interface <NUM> that supports user interaction via the companion device <NUM> to initiate an action as part of the AR/VR environment output associated with the AR/VR device <NUM> via the AR/VR application <NUM> (block <NUM>). A browser, for example, processes the companion data <NUM> and displays the companion user interface <NUM> within a user interface of the browser. This can include rendering multiple action controls in the companion user interface <NUM>, such as a search action text field, a category selection drop-down menu, an item selection radio button, a virtual keyboard, and so on. In implementations, the companion user interface <NUM> can receive and interpret particular input gestures as an action that corresponds to the currently active scene at the AR/VR device <NUM>, such as a pinch input gesture being interpreted as a zoom-in or zoom-out action, a finger swipe as a select and move item action, etc. In rendering the companion user interface <NUM>, the browser can select a particular control, from a plurality of controls <NUM> supported by the companion device <NUM>, for inclusion as part of the companion user interface based on the companion data.

<FIG> continues the examples <NUM>, <NUM> of <FIG> and <FIG> also through use of first, second, and third stages <NUM>, <NUM>, <NUM>. Aspects of the example <NUM> as shown stepwise by the modules of <FIG> may be implemented in hardware, firmware, software, or a combination thereof. At the first stage <NUM>, the companion device <NUM> displays the companion user interface <NUM> on a display device <NUM> of the companion device <NUM>. In implementations, a browser of the companion device <NUM> processes the companion data <NUM> to render the companion user interface <NUM>, but alternate markup language processing applications can be utilized, such as a general-purpose companion application dedicated to receiving and processing companion data from a plurality of AR/VR applications. Here, the companion user interface <NUM> corresponds to an AR/VR scene currently being displayed at the AR/VR device <NUM>, and includes an item <NUM> for purchase at an online marketplace.

The companion user interface also includes a control <NUM> that corresponds to a rotate action that visually rotates the item <NUM> at the AR/VR device. For example, the control <NUM> displayed at the companion device includes a left-pointing arrow and a right-pointing arrow. A user can select the left-pointing arrow at the companion device, such as via a touchscreen interface, and cause the visual display of item <NUM> at the AR/VR device <NUM> to rotate clockwise. Thus, the companion device <NUM> receives a user input to initiate an action at the AR/VR device via the companion user interface <NUM> (block <NUM>). While described as user input associated with a touchscreen interface, it is to be appreciated that any suitable type of user input can be received by the companion device to initiate an action, examples of which are provided herein.

Moving to the second stage <NUM>, the companion device <NUM> communicates input data <NUM> for receipt by the AR/VR device <NUM> to initiate the action as part of the AR/VR environment (block <NUM>). In the example <NUM>, the companion device <NUM> transmits the input data <NUM> over the AR/VR communication channel <NUM> as further described herein. The input data <NUM> can be formatted in any suitable manner and can include any combination of data, such as a message that includes an action identifier, a scene identifier, a companion device identifier, a security key, a command, and so on.

The AR/VR device <NUM> receives the input data <NUM> (block <NUM>), and initiates the action for the AR/VR environment output by the AR/VR device <NUM> based on the input data <NUM> (block <NUM>). To illustrate, the AR/VR device <NUM> can extract an action identifier from the input data <NUM> to identify which action to take. Alternately or additionally, the AR/VR device <NUM> can extract a scene identifier from the input data to validate that the currently active scene corresponds to the input data. In response to validating the input data <NUM> and/or identifying a corresponding action, the AR/VR device <NUM> invokes the corresponding action in the AR/VR application <NUM>. With reference to example <NUM>, the AR/VR device <NUM> visually rotates item <NUM> in the AR/VR environment output at the AR/VR device <NUM> based upon the user interaction with the control <NUM> at the companion device <NUM>.

At the third stage <NUM>, the AR/VR application <NUM> has transitioned to a different scene that is associated with different actions. For instance, based upon the user providing input via the natural interfaces of the AR/VR device <NUM>, the AR/VR device <NUM> can navigate from the currently active scene that corresponds to an item, to the different scene, such as a scene that corresponds to an auction search. Alternately or additionally, the user can provide input at the companion device <NUM> that corresponds to transitioning to the different scene.

In turn, the AR/VR scene manager module <NUM> identifies companion data <NUM> that generally describes controls and/or actions associated with the different scene. This can include the AR/VR scene manager module <NUM> parsing through a markup file <NUM> to identify the markup language instructions, or identifying a new markup file, associated with the different scene. The AR/VR device <NUM> then communicates the companion data <NUM> to the companion device <NUM> over the AR/VR communication channel <NUM>. The companion device then processes the companion data <NUM>, such as by way of a browser, to render action controls corresponding to the different scene at the companion device <NUM> as part of a modified companion user interface <NUM>. The modified companion user interface <NUM> generally represent a companion user interface that renders controls as specified by companion data <NUM> and/or controls associated with actions of the different scene. Additional discussion of communication of portions relating to particular scenes rendered by the AR/VR device <NUM> is described in the following section.

<FIG> depicts an example <NUM> showing operation of the AR/VR device <NUM> of <FIG> communicating companion data as portions to the companion device <NUM>. The example <NUM> also shows operation of the companion device <NUM> communicating input data to the AR/VR device based upon user interaction with a companion user interface. <FIG> depicts a procedure <NUM> in an example implementation in which an AR/VR device <NUM> determines portions of a markup file to transmit to a companion device <NUM> based on an AR/VR scene at the AR/VR device <NUM>. The procedure <NUM> also depicts transmitting input data from the companion device to the AR/VR device <NUM> effective to initiate actions at the AR/VR device <NUM>.

The following discussion describes techniques that may be implemented utilizing the previously described systems and devices. Aspects of the example <NUM> as shown stepwise by the modules of <FIG> may be implemented in hardware, firmware, software, or a combination thereof. In portions of the following discussion, reference is made interchangeably to <FIG> and <FIG>. Further, <FIG> depicts the example <NUM> of transmitting companion data to a companion device based on an AR/VR scene at an AR/VR device, and initiating actions at the AR/VR device by a companion device, through use of first, second, and third stages <NUM>, <NUM>, and <NUM>.

In the illustrated example <NUM>, the AR/VR device <NUM> includes the AR/VR application <NUM> and the markup file <NUM>. The AR/VR device <NUM> provides an AR/VR experience to a user through the execution of the AR/VR application <NUM>, which provides a plurality of AR/VR scenes. As further described herein, each AR/VR scene has respective actions that are described via the markup file <NUM>. The example <NUM> also includes the companion device <NUM> of <FIG>, where the companion device <NUM> and the AR/VR device <NUM> exchange data with one another through the AR/VR communication channel <NUM> of <FIG>.

At the first stage <NUM>, the AR/VR device <NUM> determines a portion <NUM> of a markup file <NUM> that corresponds to an AR/VR scene of the plurality of AR/VR scenes (block <NUM>). The AR/VR application <NUM>, for instance, can include the AR/VR scene manager module <NUM> of <FIG>. The AR/VR scene manager module <NUM>, when executed, generally synchronizes the distribution of companion data with AR/VR scenes active on the AR/VR device <NUM>. In other words, the AR/VR scene manager module <NUM> identifies which AR/VR scene is currently active at the AR/VR device <NUM>, determines portions of the markup file <NUM> that pertain to the AR/VR scene, and facilitates the distribution of those portions to a paired companion device <NUM>. Alternately or additionally, the AR/VR scene manager module <NUM> receives input data from the companion device <NUM> and initiates a corresponding action within the AR/VR scene.

Determining the AR/VR scene being currently delivered via the AR/VR device can occur at any point in time during execution of the AR/VR application <NUM>, such as at start-up of the AR/VR application <NUM>, during a termination of the AR/VR application <NUM>, or any other arbitrary point in time between. In one example, the AR/VR scene manager module <NUM> employs object-oriented techniques, where scene objects implement each respective AR/VR scene, and can be queried to provide status information about the scene (e.g., ACTIVE, INACTIVE). In another example, the AR/VR scene manager module <NUM> accesses library application programming interfaces (APIs) that provide scene state information. It is to be appreciated, however, that other techniques can be employed, such as managed lists, state variables, event triggers, and so forth.

The AR/VR scene manager module <NUM> analyzes the markup file <NUM> to identify markup instructions that correspond to the AR/VR scene that is currently active on the AR/VR device <NUM>. In one example, the markup file <NUM> includes scene identifiers that partition markup language within the markup file based on scene identifiers. The markup file <NUM>, for instance, can assign a first scene identifier to a first portion of markup language statements, a second scene identifier to a second portion of markup language statements, and so forth. The AR/VR scene manager module <NUM> can obtain the scene identifier of the AR/VR scene that is currently active, and parse through the markup file <NUM> to locate the corresponding portion of markup language statements.

In another example, the markup file <NUM> represents a plurality of markup files, each of which corresponds to a respective AR/VR scene. Each markup file <NUM> can include a scene identifier such that the AR/VR scene manager module <NUM> parses through the plurality of markup files to locate the markup file that includes the corresponding scene identifier. Alternately or additionally, each markup file <NUM> can employ a naming convention that indicates the corresponding scene identifier, and the AR/VR scene manager module <NUM> selects the markup file based upon matching the scene identifier to the naming convention. As yet another example, and referring to object-oriented implementations, a scene object can include references to a corresponding portion of markup file, and the AR/VR scene manager module <NUM> queries the scene object to determine the portion of markup file via the references.

The AR/VR device then communicates the portion <NUM> of the markup file <NUM> to the companion device <NUM>. This causes the companion device <NUM> to configure a companion user interface <NUM> to initiate an action as part of the AR/VR scene (block <NUM>). In the example <NUM>, the AR/VR device includes the portion <NUM> of the markup file <NUM> in the companion data <NUM> of <FIG>, and transmits the companion data <NUM>, with the portion <NUM> of the markup file <NUM>, to the companion device <NUM> over the AR/VR communication channel <NUM>. This can include transmitting the companion data <NUM> indirectly through a service provider system <NUM>, transmitting the companion data <NUM> directly over a local wireless connection, and so on. The companion device then processes the companion data <NUM> through the use of the communication module <NUM>, e.g. a browser.

The companion device <NUM> receives the companion data and processes the portion of markup file to render the companion user interface <NUM> of <FIG>, such as through a user interface of a browser. In rendering the companion user interface, the companion device can select a control to include in the companion user interface based on the companion data. The portion of markup file in the companion data, for instance, can specify an action and a corresponding control to assign to the action, such as a text field, a control button, a pull-down menu, a checkbox, a slider, a date field, and so forth. The companion device then selects the specified control to include on the companion user interface based upon the companion data.

Moving to the second stage <NUM>, a user interacts with the companion user interface <NUM> through use of a control at the companion device to provide user input directed towards initiating an action at the AR/VR device <NUM>. This can include any combination of user input, such as receiving selection of the control via a touch screen, entering text-based user input via a keyboard, entering a touch gesture at the touch screen, entering biometric data via a sensor, entering selection via the keyboard, and so forth, at the companion device. The companion device <NUM> then forwards input data <NUM> to the AR/VR device to initiate the action based on the user interaction with the companion user interface <NUM>. In one example, the companion device <NUM> encrypts the input data <NUM>, such as data encrypted through the use of a private/public key pair.

The input data <NUM> can specify any type of action supportable by the AR/VR device <NUM>, such as a rotation action, a pan-left action, a pan-right action, a move action, a select action, a search action, a configuration action, an assignment action, a bid action, a cancel action, a purchase action, a change-scene action, and so forth. The input data <NUM> can also include any combination of data, such as an action identifier, a revision number, a companion device identifier, a timestamp, location data, and so on. In one example, the input data <NUM> specifies parameter(s) for a corresponding action. The companion user interface <NUM>, for example can render multiple controls, such as a rotation arrow control that is coupled to a drop-down menu control. A user can enter a rotation degree via the drop-down menu control and then actuate the rotation arrow control. The corresponding input data <NUM> can include a rotate action identifier based on the actuation of the rotation arrow control, and a rotation degree parameter, e.g. <NUM>° left, <NUM>° right, <NUM>° upward, <NUM>° downward, and so forth, based on the drop-down menu control.

The AR/VR device <NUM> receives the input data <NUM> from the companion device <NUM> to initiate the action, where the input data <NUM> is generated based on user interaction with the companion user interface <NUM> at the companion device <NUM> (block <NUM>). In the example <NUM>, the AR/VR device <NUM> receives the input data <NUM> over the AR/VR communication channel <NUM>. In turn, the AR/VR device <NUM> initiates the action for the AR/VR scene (block <NUM>). In one example, the AR/VR scene manager module <NUM> at the AR/VR device <NUM> processes the input data <NUM> to determine a corresponding action to initiate, which can include identifying parameters used with the action. Referring to the rotation example, the AR/VR scene manager module <NUM> can analyze the input data <NUM> to determine that a clockwise rotation action of <NUM>° has been initiated via the user input at the companion device <NUM>. The AR/VR scene manager module <NUM> then initiates visually rotating the item in the corresponding item AR/VR scene.

To initiate the action, the AR/VR scene manager module <NUM> may implement various types of validation, such as validating the input data <NUM> through the use of a checksum, validating the input data based upon a public/private key, validating the requested action based on an action identifier included in the input data, validating the companion device based upon a companion device identifier in the input data, etc. If the various validations fail, the AR/VR scene manager module <NUM> refrains from initiating the action. Conversely, if the various validations succeed, the AR/VR device <NUM> proceeds with initiating the action.

The third stage <NUM> corresponds to a point in time when the AR/VR device <NUM> detects a change from the AR/VR scene to another AR/VR scene of the plurality of AR/VR scenes (block <NUM>). The AR/VR scene manager module <NUM>, for example, can receive a notification of the change from a scene object. In another example, the change in AR/VR scene corresponds to a user entering input at the AR/VR device <NUM> that causes navigation to a new AR/VR scene, such as input that switches from an item view AR/VR scene to a main AR/VR scene of an online marketplace AR/VR application.

The AR/VR scene manager module <NUM> then correlates the input to a change in AR/VR scenes. In response to detecting the change from the AR/VR scene to another AR/VR scene, the AR/VR scene manager module <NUM> analyzes the markup file <NUM> and determines another portion <NUM> of the markup file <NUM> that corresponds to the new AR/VR scene, examples of which are provided herein. The AR/VR device <NUM> then communicates the other portion <NUM> of the markup file <NUM>, e.g. the portion that corresponds to the new AR/VR scene, to the companion device <NUM>. This causes the companion device to configure the companion user interface <NUM>, such as a configuration associated with initiating another action as part of the other AR/VR scene (block <NUM>).

<FIG> is an illustration of a digital medium environment <NUM> that is operable to employ AR/VR digital content companion device techniques described herein. <FIG> depicts a procedure <NUM> in an example implementation in which a computing device generates markup language statements that are linked to AR/VR scenes as further described herein.

The following discussion describes techniques that may be implemented utilizing the previously described systems and devices. Aspects of the environment <NUM> may be implemented in hardware, firmware, software, or a combination thereof. In portions of the following discussion, reference will be made to <FIG> and <FIG>.

In the environment <NUM>, the service provider system <NUM> of <FIG> includes the AR/VR hub system <NUM>, where the AR/VR hub system <NUM> maintains an AR/VR application (app) developer system <NUM>. The AR/VR app developer system <NUM> generally represents any suitable type of system that can be utilized to develop an AR/VR application <NUM> and/or companion data <NUM>, such as markup files <NUM>. The AR/VR app developer system <NUM> can include any combination of software, hardware, and/or firmware. In one example, the AR/VR app developer system <NUM> includes a software development kit (SDK) <NUM> that generally represents a variety of software development tools and/or libraries.

The SDK <NUM> can include tools directed towards developing applications for particular platforms, such as tools to develop applications directed towards an iOS® platform, an Android® platform, a Windows® platform, a MAC® platform, and so forth. Thus, the SDK can include libraries particular to these platforms. The SDK can alternately or additionally include tools directed towards developing platform-independent functionality, such as Java script files, Hypertext Markup Language (HTML) files, Extensible Markup Language (XML) files, Python scripting files, and other types of markup files and/or platform-independent language files. The SDK <NUM> can include, for example editors, debuggers, linkers, compilers, drivers, libraries, and so forth, to generate platform-dependent and platform-independent code. Accordingly, the SDK <NUM> generally represents a collection of software development tools, such as Microsoft® Visual Studio, Xamarin, Microsoft® Azure, IntelliJ, and so on. Alternately or additionally, the SDK <NUM> can represent different software development tools that are independent from one another, e.g. a C++ editor from a first vendor, an HTML editor from a second vendor, etc..

The SDK <NUM> includes a coding module <NUM> that generally represents a coding module that is usable to generate platform-specific code, such as the AR/VR application <NUM>. The coding module <NUM> can include a code editor and a variety of compilers and linkers, where each compiler/linker pair generates executable code for a respective platform. Alternately or additionally, the coding module can include script editors and/or scripting engines. In one example, a C# coding module includes a coding editor that receives user input, and translates the user input into C# statements. The coding module <NUM> can also selectively employ a particular compiler/linker pair out of the variety of complier/linker pairs such that compiling a source file with a first compiler/linker pair generates executable processor instructions for a first platform, compiling the source file with a second compiler/linker pair generates executable processor instructions for a second platform, etc. The coding module <NUM>, however, can represent other types of coding modules that include scripting editors, coding editors, compilers, linkers, and so forth, such as C++, Java, Python, Lua, Objective C, JavaScript, HTML5, Papyrus, Cg, and so on. It is to be appreciated that, for discussion purposes, the description with respect to the generation of executable processor instructions has been simplified, and that the process can include additional or alternate actions, such as linking in platform-specific libraries, modifying platform-specific APIs, and so forth.

Similar to coding module <NUM>, companion data generation module <NUM> generally represents a coding module that includes a coding editor configured to receive user input, and translate the user input into platform-independent statements, such as the markup file <NUM>. In one example, the companion data generation module <NUM> includes an HTML editor, but alternate coding editors can be utilized for XML and so on. In the environment <NUM>, the companion data generation module <NUM> is illustrated as being included in the SDK <NUM>, but in alternate implementations, the companion data generation module <NUM> can be independent from the SDK.

The environment <NUM> also includes a developer computing device <NUM> that includes a communication module <NUM>. In the environment <NUM>, the communication module <NUM> provides the developer computing device <NUM> with connectivity to a network, such as network <NUM> of <FIG>. Thus, the communication module <NUM> generally represents any combination of software, hardware, and/or firmware utilized to provides connectivity, e.g. protocol stacks, transmitters, receivers, etc..

Through this connectivity, the developer computing device <NUM> accesses the AR/VR app developer system <NUM> to develop the AR/VR application <NUM> and the markup file <NUM>. In one example, the developer computing device <NUM> accesses a portion or all of the functionality provided by the AR/VR app developer system <NUM> over the network <NUM> to develop AR/VR applications <NUM> and/or markup files <NUM>. In another example, the developer computing device <NUM> downloads a portion or all the AR/VR app developer system <NUM> from the service provider system <NUM>, and accesses the corresponding functionality local to the developer computing device <NUM>. In yet another example, the developer computing device <NUM> can obtain a portion or all of the AR/VR app developer system <NUM> via static means, such as a CD, a flash drive, etc..

The AR/VR app developer system <NUM> receive inputs as part of creating an augmented or virtual reality (AR/VR) application <NUM> executable by an augmented reality (AR) device, where the AR/VR application <NUM> can have a plurality of scenes and corresponding actions as part of user interaction with respective scenes of the plurality of scenes (block <NUM>). A code developer, for example, can develop a plurality of AR/VR scenes that are included in the AR/VR application <NUM> by interacting with the coding module <NUM>.

The code developer can alternately or additionally add actions to the AR/VR scenes that can be initiated through user interaction with the AR/VR application <NUM>, such as by natural interfaces and/or a companion device. The actions can be scene-specific and/or apply to multiple AR/VR scenes. An item view AR/VR scene, for example, can include a rotation action as further described herein. The code developer can also provide access to the rotation action via the input to the AR/VR app developer system <NUM>, such as by entering input that generates an action API that exposes the rotation action.

The developer can also generate an AR/VR scene manager module <NUM> via input to the AR/VR app developer system <NUM>, where the AR/VR scene manager module oversees the various AR/VR scenes included in the AR/VR application. This can include generating an AR/VR scene manager module <NUM> that determines which AR/VR scene is a currently active scene, and identifies portions of markup file <NUM> that correspond to the AR/VR scene that is currently active as described in the previous section.

The AR/VR app development system <NUM> can also generate a markup file <NUM> that specifies linkages between controls usable as part of user interaction with a companion device <NUM> and the corresponding actions within the respective scenes of the AR/VR application <NUM> as executed by the AR/VR device <NUM> (block <NUM>). A code developer, for example, can develop the markup file by interacting with the companion data generation module <NUM>.

As one example, the code developer can enter input to the companion data generation module <NUM> that generates markup language that links an action API to an action identifier, specifies a control to display at a companion user interface <NUM> that links the control to the action identifier, links a first control at the companion user interface to a second control, and so forth. The markup language can alternately or additionally include scene identifiers, such as scene identifiers that indicate portions of markup language that correspond to a particular scene. Thus, the code developer can enter input to the companion data generation module <NUM> to generate markup language that indicates the corresponding actions to a particular AR/VR scene, controls that correspond to the actions, types of controls to render at a companion user interface, APIs that correspond to the actions, and so forth.

The code developer can also enter input to the companion data generation module <NUM> that stores the markup language in a markup file, such as markup file <NUM>. In at least one example, the markup file <NUM> is configured using a markup language that is executable by a general-purpose application, such as a browser of a companion device <NUM>.

To promote synchronization between a markup file <NUM> and a corresponding AR/VR application <NUM>, the AR/VR app developer system <NUM> can be used to store the markup file <NUM> as associated with the AR/VR application <NUM> (block <NUM>). As further described herein, the markup file can be configured to support communication, to a companion device <NUM>, of respective linkages between the controls <NUM> as executed by the companion device <NUM>, and corresponding actions for the respective scenes output by the AR/VR device <NUM>. In one example, the markup file <NUM> and the corresponding AR/VR application <NUM> are stored on the service provider system <NUM>, such as in a storage device <NUM>, such that a user downloading the AR/VR application <NUM> from the service provider system <NUM> also downloads the corresponding markup file <NUM> without additional user interactions, an example of which is further described in the following section.

<FIG> is an illustration of a digital medium environment <NUM> that is operable to employ AR/VR digital content companion device techniques described herein. <FIG> depicts a procedure <NUM> in an example implementation in which a hub synchronizes the delivery of companion data to a companion device based on an AR/VR scene at an AR/VR device as further described herein.

The environment <NUM> includes the service provider system <NUM>, the AR/VR device <NUM>, and the companion device <NUM> of <FIG>. The service provider system <NUM> has established a first communication channel with the AR/VR device <NUM>, and a second communication channel with the companion device <NUM>. In implementations, the first and second communication channels facilitate the pairing of the AR/VR device <NUM> and the companion device <NUM>. As such, the companion device <NUM> can be used to initiate actions at the AR/VR device <NUM>. The AR/VR device <NUM>, for example, includes the AR/VR application <NUM> of <FIG>, and the companion device <NUM> renders the companion user interface <NUM> through the use of the communication module <NUM> of <FIG>. A user can interact with controls rendered at the companion user interface <NUM> to initiate actions at the AR/VR device <NUM>.

The service provider system <NUM> also includes the AR/VR hub system <NUM> of <FIG>, where the AR/VR hub system <NUM> manages the distribution of companion data <NUM> to a companion device <NUM>. The AR/VR hub system <NUM> includes a request manager module <NUM> that can receive a request via the network <NUM> from the companion device <NUM> for companion data <NUM> that corresponds to an augmented or virtual reality (AR/VR) application <NUM> configured for execution by the AR/VR device <NUM>. In the environment <NUM>, the companion device <NUM> sends a request corresponding to companion data for the AR/VR application <NUM>. The companion device <NUM> can send a request for companion data <NUM> at any point in time, such as at an initial pairing with the AR/VR device <NUM>, in response to receiving notification of the AR/VR device <NUM> invoking the AR/VR application <NUM>, in response to receiving notification of a scene change, and so forth.

In one example, the request includes a scene identifier. For example, the AR/VR device <NUM> can notify the companion device <NUM> of a scene change and include the scene identifier in the notification. In turn, the notification triggers the companion device <NUM> to send the request, where the companion device <NUM> adds the scene identifier to the request. Alternately or additionally, the companion device <NUM> sends a request that includes an application identifier. For instance, the companion device <NUM> can query the AR/VR device <NUM> for the application identifier. In response to receiving the application identifier, the companion device <NUM> can include this information in the request for companion data <NUM>. In yet another example, the companion device <NUM> sends a request for companion data <NUM>, where the request indicates an AR/VR device <NUM> that the companion device is paired with, e.g. the AR/VR device <NUM>. Thus, the companion device <NUM> can direct the request manager module <NUM> to a particular AR/VR application <NUM> and/or a particular AR/VR scene. Alternately or additionally, the companion device <NUM> provides the request manager module <NUM> with device information which can be used by the AR/VR hub system <NUM> to derive the AR/VR application <NUM> and/or AR/VR scene information as further described herein.

In response to receiving the request, the request manager module <NUM> verifies that an originator of the request is to receive the companion data (block <NUM>). A request verification module <NUM>, for example, analyzes the request to validate that the companion device <NUM> has authority to receive the companion data <NUM> of <FIG> that corresponds to a scene of the AR/VR application <NUM>. This can include validating the request via a public/private key pair, validating a companion device ID included in the request as being on an allowed device list, validating the companion device is paired with the AR/VR device, requesting validation from the AR/VR device <NUM>, requesting authentication information from the companion device <NUM>, and so forth.

In response to verifying the originator of the request, the AR/VR hub system <NUM> locates, from storage, the companion data <NUM> from a plurality of companion data (block <NUM>) that corresponds to the request. The AR/VR hub system <NUM> of the environment <NUM> includes a companion markup location module <NUM> that accesses a storage device <NUM> to locate the companion data, e.g., companion data <NUM>. In one example, the companion markup location module <NUM> searches a storage device <NUM> for the AR/VR application <NUM>, and extracts the companion data <NUM> and/or the markup file <NUM> from the storage device <NUM>. The companion markup location module <NUM> can base the search on any type of information, such as the scene identifier or the application identifier included in the request from the companion device <NUM>.

Alternately or additionally, the AR/VR hub system <NUM> maintains a record of paired devices (e.g., a record that the AR/VR device <NUM> is paired to the companion device <NUM>) and/or a record of what applications are being executed (e.g., the AR/VR application <NUM>). The records can be maintained and updated based on information included in communications received and transmitted through the AR/VR hub system <NUM>, such as communications from and to the AR/VR device <NUM> and the companion device <NUM>. In turn, the companion markup location module <NUM> accesses these records based on the companion device ID included in the request, and locates the companion data <NUM>, such as by identifying the active application on the AR/VR device <NUM> paired to the companion device <NUM>.

In another example, the AR/VR device <NUM> and the companion device <NUM> can register with the AR/VR hub system <NUM> as part of the pairing process, and update the AR/VR hub system <NUM> with information associated with the pairing, such as what AR/VR application <NUM> is executing, what AR/VR scene is currently active, when an AR/VR scene changes, and so forth. The information can include any combination of companion device identifiers, AR/VR device identifiers, AR/VR application identifiers, AR/VR scene identifiers, revision information, time stamp information, etc..

The AR/VR hub system <NUM> then forms a communication for transmission via the network to the companion device that includes the companion data (block <NUM>). This can include determining which portions of a markup file <NUM> to include in the companion data <NUM>. The AR/VR hub system <NUM> in environment <NUM>, for example, includes an AR/VR environment manager module <NUM> that synchronizes the distribution of companion data <NUM> to the companion device <NUM> and an AR/VR scene that is active at the AR/VR device <NUM>. To help manage the synchronization, an AR/VR environment manager module <NUM> can include a portion or all of the functionality of the AR/VR scene manager module <NUM> of <FIG> to determine portions of a markup file <NUM> associated with actions of the currently active AR/VR scene. The AR/VR environment manager module <NUM> can identify and forward these portions of the markup file <NUM> to the data transmission module <NUM> to include in the communication.

In some implementations, the AR/VR environment manager module <NUM> delivers AR/VR data to the AR/VR device that causes the AR/VR device to provide an AR/VR environment with a plurality of scenes. The AR/VR environment manager module <NUM>, for example, can deliver cloud-based services to the AR/VR device as part of the AR/VR data used to provide the AR/VR environment. Any suitable type of cloud-based model and/or deployment mechanism across a network can be used in implement he AR/VR environment manager module, such as a Software as a Service (SaaS) model, a Platform as a Service (PaaS) model, an Infrastructure as a Service (IaaS) model, and so forth. Accordingly, the AR/VR environment manager module <NUM> can deploy and synchronize an AR/VR environment at an AR/VR device with an companion device using any one or combination of these models.

The AR/VR hub system <NUM> synchronizes execution of the companion data <NUM> by the companion device <NUM> with execution of the AR/VR application <NUM> by the AR/VR device <NUM> (block <NUM>), such as by the AR/VR environment manager module <NUM>. This can include the synchronization of AR/VR scene transitions and companion data as described herein.

In implementations, the AR/VR environment manager module <NUM> verifies the validity of transmitting the communication with the companion data to the companion device <NUM>. For example, the AR/VR environment manager module <NUM> can employ multiprocessor synchronization techniques that prevent the execution of AR/VR application <NUM> at the AR/VR device <NUM> from becoming out of sync with the companion user interface <NUM> displayed at the companion device <NUM>, such as handshake-based messaging, multi-platform semaphores, etc. The AR/VR environment manager module <NUM> can alternately or additionally include a state machine that maintains scene state information. In some implementations, the AR/VR environment manager module provides cloud-base services utilized by the AR/VR device and the companion device that implement an AR/VR environment. The AR/VR hub system <NUM>, for example, can direct the AR/VR application to wait to transition to a new scene until a confirmation from the companion device has been received. Upon receiving the confirmation from the companion device, the AR/VR hub system <NUM> can then direct the AR/VR application <NUM> to the transition to the new scene as described above. Communication between the companion device <NUM> and the AR/VR device <NUM> may be implemented in a variety of ways, examples of which are described in the following section.

<FIG> is an illustration of a digital medium environment <NUM> that is operable to employ AR/VR digital content companion device techniques described herein. <FIG> depicts a procedure <NUM> in an example implementation in which computing devices synchronize the use of protected data between paired devices in an AR/VR environment.

The environment <NUM> includes the service provider system <NUM>, the AR/VR device <NUM>, and the companion device <NUM> of <FIG>. The service provider system <NUM> includes the AR/VR environment manager module <NUM> of <FIG>. The AR/VR environment manager module <NUM> facilitates synchronization between the AR/VR device <NUM> and the companion device <NUM>. For instance, in the environment <NUM>, the service provider system <NUM> has established an AR/VR device communication channel <NUM> with the AR/VR device <NUM> over the network <NUM> of <FIG>, and a companion device communication channel <NUM> with the companion device <NUM> over the network <NUM>. The AR/VR environment manager module <NUM> can designate what information is transmitted over these communication channels and when, such as communications used to pair the devices, companion data synchronized to AR/VR scenes, and protected data.

As described herein, the companion device <NUM> can be used to initiate actions at the AR/VR device <NUM>. The AR/VR device <NUM>, for example, includes the AR/VR application <NUM> of <FIG>, and the companion device <NUM> renders the companion user interface <NUM> through the use of the communication module <NUM> of <FIG>. A user can interact with controls rendered at the companion user interface <NUM> to initiate actions at the AR/VR device <NUM>. This can include the transfer of protected data from the companion device <NUM> to the service provider system <NUM> and/or the AR/VR device <NUM>.

The companion device <NUM>, for instance, can include companion device functionality <NUM> that generally represents any suitable application or action associated with protected data. In one example, the companion device functionality implements an electronic wallet that stores and manages payment information <NUM>, such as credit card information, bank account information, mobile payment information, gift card information, and so forth. Alternately or additionally, the companion device functionality includes a credential management system that stores and manages user credentials <NUM>, such as user identifications (IDs), passwords, digital certificates, and so forth.

In some instances, both the companion device <NUM> and the AR/VR device <NUM> may both access a third party, represented by the service provider system <NUM>, to access functionality as part of an AR/VR environment, e.g., a website. Consider an example in which the AR/VR application <NUM> executes an AR/VR environment (block <NUM>), such as an item view AR/VR scene associated with an online marketplace of the service provider system <NUM>. In implementations, the AR/VR environment manager module <NUM> communicates AR/VR data via the network <NUM> to the AR/VR device <NUM> effective to cause rendering of the AR/VR environment by the AR/VR device <NUM> for viewing by a user (block <NUM>). In the illustrated example, the AR/VR data used to execute the AR/VR environment can be transmitted over the AR/VR device communication channel <NUM>.

To maintain synchronization between the AR/VR device <NUM> and the companion device <NUM>, the AR/VR environment manager module <NUM> communicates companion data <NUM> via a network <NUM> to the companion device <NUM>, where the companion data <NUM> is configured to cause the companion device <NUM> to generate the companion user interface <NUM> (block <NUM>). As described herein, the companion user interface <NUM> renders controls on a display device of the companion device <NUM> that are user selectable to initiate an action within the AR/VR environment.

In implementations, the companion user interface <NUM> displays controls associated with purchasing the item displayed in the item view AR/VR scene at the AR/VR device <NUM>. The controls, for instance, can correspond to a "Buy Now" action. Alternately or additionally, the companion user interface displays controls associated with logging into the online marketplace with a particular user profile.

In response to receiving the user interaction with the "Buy Now" control, the companion device <NUM> accesses the companion device functionality <NUM> to acquire protected data <NUM>, e.g. the payment information and/or the user credentials. To acquire the protected data, the companion device <NUM> can render additional controls in the companion user interface <NUM> that allow the user to navigate and select particular data, such as a drop-down menu to select a credit card, a checkbox to select a user profile, etc. The companion device <NUM> extracts the protected data <NUM> via the companion device functionality, and then transmits the protected data over the companion device communication channel <NUM> to the service provider system <NUM>. The companion device <NUM> can transmit the protected data with input data that designates an action, e.g. a "Buy Now" action, or transmit the protected data in a separate communication than the input data.

In implementations, the service provider system <NUM> may also forward protected data <NUM> to the AR/VR device <NUM> over the AR/VR device communication channel, where the protected data <NUM> corresponds to the protected data extracted from the companion device <NUM>. Similar to the companion device <NUM>, the service provider system <NUM> can also transmit the protected data with input data that designates an action, or transmit the protected data in a separate communication.

The service provider system <NUM> receives the protected data <NUM> from the companion device, via the service provider system and/or the AR/VR environment manager module, over the AR/VR device communication channel <NUM> (block <NUM>). As described herein, the protected data corresponds to protected data generated via user interaction with the companion user interface <NUM> via a user interface of the companion device <NUM>. In turn, the service provider system <NUM> executes a corresponding action as part of the AR/VR environment without exposing the protected data to the AR/VR device <NUM> (block <NUM>). In the "Buy Now" example, the service provider system <NUM> can apply the payment information and/or the user credentials without exposure, as payment information and/or the user credentials as part of the online marketplace.

Thus, the AR/VR environment manager module <NUM> manages the protected data <NUM> without forwarding the protected data <NUM> to the AR/VR device <NUM>. The AR/VR environment manager module <NUM>, for example, can be communicatively coupled to the online marketplace. In receiving the protected data <NUM>, the AR/VR environment manager module <NUM> distributes the protected data to the online marketplace without exposing the sensitive information to the AR/VR device. Since the AR/VR environment manager module <NUM> facilitates synchronization between the AR/VR device <NUM> and the companion device <NUM>, the AR/VR environment manager module <NUM> can forward notifications to each device upon the completion, or failure, of the transaction. Such functionality is not possible in conventional techniques that involve communication of the data from the companion device <NUM> "through" the AR/VR device <NUM> as an intermediary.

<FIG> illustrates an example system generally at <NUM> that includes an example computing device <NUM> that is representative of one or more computing systems and/or devices that may implement the various techniques described herein. This is illustrated through inclusion of the AR/VR hub system <NUM>. The computing device <NUM> may be, for example, a server of a service provider, a device associated with a client (e.g., a companion device), an on-chip system, and/or any other suitable computing device or computing system.

The example computing device <NUM> as illustrated includes a processing system <NUM>, one or more computer-readable media <NUM>, and one or more I/O interface <NUM> that are communicatively coupled, one to another. Although not shown, the computing device <NUM> may further include a system bus or other data and command transfer system that couples the various components, one to another. A system bus can include any one or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and/or a processor or local bus that utilizes any of a variety of bus architectures. A variety of other examples are also contemplated, such as control and data lines.

The processing system <NUM> is representative of functionality to perform one or more operations using hardware. Accordingly, the processing system <NUM> is illustrated as including hardware element <NUM> that may be configured as processors, functional blocks, and so forth. This may include implementation in hardware as an application specific integrated circuit or other logic device formed using one or more semiconductors. The hardware elements <NUM> are not limited by the materials from which they are formed or the processing mechanisms employed therein. For example, processors may be comprised of semiconductor(s) and/or transistors (e.g., electronic integrated circuits (ICs)). In such a context, processor-executable instructions may be electronically-executable instructions.

The computer-readable storage media <NUM> is illustrated as including memory/storage <NUM>. The memory/storage <NUM> represents memory/storage capacity associated with one or more computer-readable media. The memory/storage component <NUM> may include volatile media (such as random-access memory (RAM)) and/or nonvolatile media (such as read only memory (ROM), Flash memory, optical disks, magnetic disks, and so forth). The memory/storage component <NUM> may include fixed media (e.g., RAM, ROM, a fixed hard drive, and so on) as well as removable media (e.g., Flash memory, a removable hard drive, an optical disc, and so forth). The computer-readable media <NUM> may be configured in a variety of other ways as further described below.

Input/output interface(s) <NUM> are representative of functionality to allow a user to enter commands and information to computing device <NUM>, and also allow information to be presented to the user and/or other components or devices using various input/output devices. Examples of input devices include a keyboard, a cursor control device (e.g., a mouse), a microphone, a scanner, touch functionality (e.g., capacitive or other sensors that are configured to detect physical touch), a camera (e.g., which may employ visible or non-visible wavelengths such as infrared frequencies to recognize movement as gestures that do not involve touch), and so forth. Examples of output devices include a display device (e.g., a monitor or projector), speakers, a printer, a network card, tactile-response device, and so forth. Thus, the computing device <NUM> may be configured in a variety of ways as further described below to support user interaction.

Various techniques may be described herein in the general context of software, hardware elements, or program modules. Generally, such modules include routines, programs, objects, elements, components, data structures, and so forth that perform particular tasks or implement particular abstract data types. The terms "module," "functionality," and "component" as used herein generally represent software, firmware, hardware, or a combination thereof. The features of the techniques described herein are platform-independent, meaning that the techniques may be implemented on a variety of commercial computing platforms having a variety of processors.

"Computer-readable storage media" may refer to media and/or devices that enable persistent and/or non-transitory storage of information in contrast to mere signal transmission, carrier waves, or signals per se. Thus, computer-readable storage media refers to non-signal bearing media. The computer-readable storage media includes hardware such as volatile and non-volatile, removable and non-removable media and/or storage devices implemented in a method or technology suitable for storage of information such as computer readable instructions, data structures, program modules, logic elements/circuits, or other data. Examples of computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, hard disks, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or other storage device, tangible media, or article of manufacture suitable to store the desired information and which may be accessed by a computer.

"Computer-readable signal media" may refer to a signal-bearing medium that is configured to transmit instructions to the hardware of the computing device <NUM>, such as via a network. Signal media typically may embody computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as carrier waves, data signals, or other transport mechanism. Signal media also include any information delivery media. By way of example, and not limitation, communication media include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media.

As previously described, hardware elements <NUM> and computer-readable media <NUM> are representative of modules, programmable device logic and/or fixed device logic implemented in a hardware form that may be employed in some embodiments to implement at least some aspects of the techniques described herein, such as to perform one or more instructions. Hardware may include components of an integrated circuit or on-chip system, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a complex programmable logic device (CPLD), and other implementations in silicon or other hardware. In this context, hardware may operate as a processing device that performs program tasks defined by instructions and/or logic embodied by the hardware as well as a hardware utilized to store instructions for execution, e.g., the computer-readable storage media described previously.

Combinations of the foregoing may also be employed to implement various techniques described herein. Accordingly, software, hardware, or executable modules may be implemented as one or more instructions and/or logic embodied on some form of computer-readable storage media and/or by one or more hardware elements <NUM>. The computing device <NUM> may be configured to implement particular instructions and/or functions corresponding to the software and/or hardware modules. Accordingly, implementation of a module that is executable by the computing device <NUM> as software may be achieved at least partially in hardware, e.g., through use of computer-readable storage media and/or hardware elements <NUM> of the processing system <NUM>. The instructions and/or functions may be executable/operable by one or more articles of manufacture (for example, one or more computing devices <NUM> and/or processing systems <NUM>) to implement techniques, modules, and examples described herein.

The techniques described herein may be supported by various configurations of the computing device <NUM> and are not limited to the specific examples of the techniques described herein. This functionality may also be implemented all or in part through use of a distributed system, such as over a "cloud" <NUM> via a platform <NUM> as described below.

The platform <NUM> may abstract resources and functions to connect the computing device <NUM> with other computing devices. The platform <NUM> may also serve to abstract scaling of resources to provide a corresponding level of scale to encountered demand for the resources <NUM> that are implemented via the platform <NUM>. Accordingly, in an interconnected device embodiment, implementation of functionality described herein may be distributed throughout the system <NUM>. For example, the functionality may be implemented in part on the computing device <NUM> as well as via the platform <NUM> that abstracts the functionality of the cloud <NUM>.

Claim 1:
A method as implemented by a companion device (<NUM>), the method comprising:
displaying, by a display device of the companion device (<NUM>), pairing data obtained from a network address associated with a service provider system (<NUM>);
pairing the companion device (<NUM>) as communicatively coupled with an augmented or virtual reality, AR/VR, device (<NUM>) responsive to verification of the pairing data by the AR/VR device (<NUM>);
receiving, by the companion device (<NUM>), companion data in response to the pairing, wherein the companion data indicates an action within an AR/VR environment output by the AR/VR device (<NUM>);
generating, by the companion device (<NUM>), a companion user interface based on the companion data to support user interaction via the companion device (<NUM>) to initiate the action as part of the AR/VR environment output by the AR/VR device (<NUM>);
selecting a control for inclusion as part of the companion user interface by the companion device (<NUM>) from a plurality of controls based on the companion data;
receiving, by the companion device (<NUM>), a user input to initiate the action via the companion user interface, the user input being provided by a user interacting with the companion user interface through use of the control; and
communicating, by the companion device (<NUM>), input data for receipt by the AR/VR device (<NUM>) to initiate the action as part of the AR/VR environment, wherein the action is to be initiated as part of a scene being rendered by the AR/VR device (<NUM>).