CLOUD VIRTUAL REALITY ECOSYSTEM

A method for virtual reality cloud computing includes receiving, from a client device, a request by a user of the client device to execute a application, the application being provided by a third-party cloud service comprising a plurality of hosts. The method further includes selecting, from the plurality of hosts, a particular host to service the request, and receiving, from the particular host, connection parameters that are required for establishing a streaming connection to the particular host. The method further includes, in response to the request, providing the connection parameters to a streaming client executing on the client device, and instructing the streaming client to establish the streaming connection to the particular host and to execute the application.

TECHNICAL FIELD

The present disclosure generally relates to virtual reality, and more particularly to a cloud virtual reality ecosystem.

BACKGROUND

Virtual reality (VR) is a three-dimensional (3D) computer-generated environment with which a user can interact. A user's presence in the virtual environment is typically simulated by means of a headset that generates sights and sounds. Example applications for VR include video gaming, medical training, retail shopping, and interior designing. Typically, VR computer code is rendered locally on a VR headset. However, this approach requires locally installed powerful graphical hardware and computing resources.

Virtual reality (VR) may work with cloud computing platforms. However, there are cost, latency, and other resource constraints that pose serious challenges to any virtual reality (VR) provider considering hosting every instance of a cloud VR application running on the local hardware (e.g., headset) of the VR provider. Moreover, there are significant challenges related to hosting cloud VR applications (e.g., private hosting, data protection, and revenue models), such challenges can delay or even prevent the availability of cloud VR to users.

Data protection requirements compel VR providers to host cloud VR on their own hardware. It is a common for a VR provider to assume they must host everything for cloud VR to protect user data and secure the revenue stream of the VR provider. The VR provider may reason that they cannot protect data if they are not hosting everything.

As such, there is a need for offloading demanding computational tasks from headsets and other client devices, while protecting user data and the VR provider's revenue stream.

BRIEF SUMMARY

According to some embodiments, a method for virtual reality cloud computing includes receiving, from a client device, a request by a user of the client device to execute a application, the application being provided by a third-party cloud service having one or more hosts. The method further includes selecting, from the hosts, a particular host to service the request, and receiving, from the particular host, connection parameters that are required for establishing a streaming connection to the particular host. The method further includes, in response to the request, providing the connection parameters to a streaming client executing on the client device, and instructing the streaming client to establish the streaming connection to the particular host and to execute the application.

According to some embodiments, a non-transitory computer-readable medium stores a program for virtual reality cloud computing, which when executed by a computer, configures the computer to receive, from a client device, a first request by a user of the client device to execute an application, the application being provided by a third-party cloud service comprising one or more hosts. The executed program further configures the computer to select, from the hosts, a particular host to service the first request, and receive, from the particular host, connection parameters that are required for establishing a streaming connection to the particular host. The executed program further configures the computer to, in response to the first request, provide the connection parameters to a streaming client executing on the client device, and instruct the streaming client to establish the streaming connection to the particular host and to execute the application.

According to some embodiments, a system for virtual reality cloud computing includes a processor and a non-transitory computer readable medium storing a set of instructions, which when executed by the processor, configure the processor to receive, from a client device, a first request by a user of the client device to execute an application, the application being provided by a third-party cloud service comprising one or more hosts. The executed instructions further configure the processor to select, from the hosts, a particular host to service the first request, and to receive, from the particular host, connection parameters that are required for establishing a streaming connection to the particular host. The executed instructions further configure the processor to, in response to the first request, provide the connection parameters to a streaming client executing on the client device, and to instruct the streaming client to establish the streaming connection to the particular host and to execute the application. The executed instructions further configure the processor to receive, from the particular host, a second request for a transaction of a payment in exchange for a product, and to request, from the streaming client, an authorization to conduct the transaction. The executed instructions further configure the processor to receive, as a response from the streaming client, the authorization to conduct the transaction, and in response to the second request, receive the payment from the user. The executed instructions further configure the processor to, in further response to the second request, instruct the particular host to provide the product to the user.

DETAILED DESCRIPTION

The term “metaverse” as used herein refers, according to some embodiments, to a collective shared space in virtual reality, typically comprised of one or more interconnected digital environments, assets, and experiences. A metaverse may be a multifaceted, immersive platform that facilitates real-time interaction, collaboration, and engagement among its users. The metaverse may encompasses a vast array of digital spaces, ranging from simulated worlds and games to social hubs and commerce platforms. Within the metaverse, users and other entities (e.g., businesses, agencies, etc.) may create, own, and exchange digital assets, explore, socialize, conduct business, and participate in various activities within a seamless and dynamic virtual environment.

The term “virtual reality” as used herein refers, according to some embodiments, to a computer-generated simulation of an immersive, two-dimensional or (more typically) three-dimensional environment that can be explored and interacted with by individuals through sensory stimuli, typically employing headsets or other specialized devices. Virtual reality (abbreviated as “VR”) may provide a sensory-rich experience that simulates and/or replicates the real world or an imagined setting, enabling users to engage in activities, manipulate objects, and perceive a simulated environment as if it were real. VR systems often encompass visual, auditory, and/or haptic feedback to enhance the sense of presence and immersion, transporting users into a digitally generated world where they can interact, learn, or experience various scenarios in a highly immersive and interactive manner. As used herein, the term “virtual reality” is understood to include the Internet and “augmented reality.”

The term “cloud computing” as used herein refers, according to some embodiments, to a technology paradigm that enables access to a shared pool of configurable computing resources, such as networks, servers, storage, applications, and services, provided by a cloud services provider over a wide-area network such as the Internet. The cloud computing model provides on-demand availability and convenient access to these resources without requiring direct management of infrastructure. Users can swiftly deploy and scale applications, utilizing the cloud provider's vast and flexible infrastructure as needed. Cloud computing encompasses various service models like Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS), allowing for cost-effective, scalable, and flexible computing capabilities while abstracting the complexities of underlying hardware and enabling ubiquitous, on-demand network access to a wide range of computing resources.

The term “container” as used herein refers, according to some embodiments, to a lightweight, standalone, and executable package of software that includes everything needed to run a piece of code, including code, libraries, dependencies, and settings. Containers may be isolated from each other and the host system, often making use of firewalls and/or encryption, and possibly sharing only a kernel, allowing multiple containers to run on a single host or server without conflicts. Containers enable efficient and portable deployment of applications across different cloud environments, without requiring changes to the underlying infrastructure, and facilitates scalable and reliable distribution of services and applications. A container may also be equivalently referred to as a “sandbox.”

Embodiments of the present disclosure address the above identified problems using cloud infrastructures to offload demanding computational tasks to remote servers before streaming VR content to a user's client device (e.g., a headset).

Customers may be equivalently referred to herein as “users.” Servers may also be equivalently referred to herein as “hosts.” Applications may be equivalently referred to herein as “apps.”

Some embodiments provide, as a technical solution to the technical problems described above, a framework that makes it possible for developers to have computationally expensive portions of VR experiences (e.g., rendering) hosted by third-party cloud providers, and to communicate with the light-duty services of a first-party VR provider in a way that protects user data with a similar level of protection to data on a head-mounted display (HMD), while also protecting revenue streams just as well as would a fully hosted solution. Some embodiments restrict access to sensitive data and activities to a connection directly between the headset and the services of the first-party VR provider.

FIG.1illustrates a network architecture100used to implement a cloud VR ecosystem, according to some embodiments. Architecture100may include servers130and a database152, communicatively coupled with multiple client devices110via a network150. Client devices110may include any one of a laptop computer, a desktop computer, or a mobile device such as a smart phone, a palm device, video player, or a tablet device. The database152may store backup files from, for example, matrices, videos, and processing data.

Network150can include, for example, any one or more of a local area network (LAN), a wide area network (WAN), the Internet, and the like. Further, network150can include, but is not limited to, any one or more of the following network topologies, including a bus network, a star network, a ring network, a mesh network, a star-bus network, tree or hierarchical network, and the like.

FIG.2is a block diagram illustrating details of a system200having at least one client device110and at least one server130used in a network architecture as disclosed herein (e.g., architecture100), according to some embodiments. Client device110and server130are communicatively coupled over network150via respective communications modules218-1and218-2(hereinafter, collectively referred to as “communications modules218”). Communications modules218are configured to interface with network150to send and receive information, such as requests, uploads, messages, and commands to other devices on the network150. Communications modules218can be, for example, modems or Ethernet cards, and may include radio hardware and software for wireless communications (e.g., via electromagnetic radiation, such as radiofrequency -RF-, near field communications -NFC-, Wi-Fi, and Bluetooth radio technology). Client device110may be coupled with an input device214and with an output device216. A user may interact with client device110via the input device214and the output device216. Input device214may include a mouse, a keyboard, a pointer, a touchscreen, a microphone, a joystick, a virtual joystick, a touch-screen display that a user may use to interact with client device110, or the like. In some embodiments, input device214may include cameras, microphones, and sensors, such as touch sensors, acoustic sensors, inertial motion units -IMUs- and other sensors configured to provide input data to a VR/AR headset. Output device216may be a screen display, a touchscreen, a speaker, and the like.

Client device110may also include a processor212-1, configured to execute instructions stored in a memory220-1, and to cause client device110to perform at least some operations in methods consistent with the present disclosure. Memory220-1may further include an application222, configured to run in client device110and couple with input device214and output device216. The application222may be downloaded by the user from server130and may be hosted by server130. The application222includes specific instructions which, when executed by processor212-1, cause operations to be performed according to methods described herein. In some embodiments, the application222runs on an operating system (OS) installed in client device110. In some embodiments, application222may run out of a web browser. In some embodiments, the processor is configured to control a graphical user interface (GUI) for the user of one of client devices110accessing the server130.

A database252may store data and files associated with the server130from the application222. In some embodiments, client device110is a mobile phone used to collect a video or picture and upload to server130using a video or image collection application222, to store in the database252.

Server130includes a memory220-2, a processor212-2, and communications module218-2. Hereinafter, processors212-1and212-2, and memories220-1and220-2, will be collectively referred to, respectively, as “processors212” and “memories220.” Processors212are configured to execute instructions stored in memories220. In some embodiments, memory220-2includes an application engine232. The application engine232may be configured to perform operations and methods according to aspects of embodiments. The application engine232may share or provide features and resources with the client device, including multiple tools associated with data, image, or video collection, capture, or applications that use data, images, or video retrieved with application engine232(e.g., application222). The user may access application engine232through application222, installed in a memory220-1of client device110. Accordingly, application222may be installed by server130and perform scripts and other routines provided by server130through any one of multiple tools. Execution of application222may be controlled by processor212-1.

The system200may include multiple servers130. For example, one such server may be a VR platform server of a first-party VR service provider, and another such server may be a cloud host of a third-party cloud service.

Shifting Costs to Certified VR Cloud Service Providers

Some embodiments disclosed herein enable a first-party VR provider to define the requirements a third-party cloud VR provider must comply with in order to participate in an ecosystem of the first-party VR provider. Doing so may facilitate customers having a good user experience when using their cloud VR streams.

Some embodiments make use of a certification process by the first-party VR provider, that may require third-party cloud VR providers to meet performance and concurrency requirements, along with security protocols that may mitigate the risks associated with having code that would otherwise be executed locally on a user's headset instead running remotely in a data center.

In some embodiments, the first-party VR provider may define requirements to run code in containers with strict firewall and file system limitations to ensure that user data is protected from unauthorized use. Additional security and certification requirements (e.g., those for data centers that provide Health Insurance Portability and Accountability Act (HIPAA) compliant services) may be defined to ensure the necessary security measures are in place to meet the first-party VR provider's obligations (e.g., legal, regulatory) to protect user data.

FIG.3illustrates a system300executing a process301for publishing an application to a cloud VR ecosystem, according to some embodiments. The system300is similar to the embodiment of the system200discussed above with respect toFIG.2, and like reference numerals have been used to refer to the same or similar components. A detailed description of these components will be omitted, and the following discussion focuses on the differences between these embodiments. Any of the various features discussed with any one of the embodiments discussed herein may also apply to and be used with any other embodiments.

In the system300, data may be sent and received between a client device310(e.g., client device110), a first-party VR service provider320(e.g., one or more servers such as server130), a third-party cloud service330(e.g., one or more servers such as server130), and an application developer340. Various operations of the process301may be performed by different components of system300, as described below.

At350, the application developer340creates an application (e.g., a VR application) that is designed to run in a container. The application may include or communicate with a client-side component (e.g., a streaming client), such as the example of application222running within a portion of memory220-1on client device310. The application may also include a server-side component (e.g., a cloud server), such as application engine232running within a portion of memory220-2of a host of the cloud service330.

At355, the application developer340tests the application locally, and at360, tests the application in a hosted service of their choosing, at their own expense. Local testing may occur prior to hosted testing, or in parallel. The hosted service may be third-party cloud service330, for example, operating multiple server hosts.

At370, the third-party cloud service330receives the application from the application developer340and hosts the application.

At375, upon completion of local and hosted testing, the application developer340submits the application to the first-party VR service provider320. The submission process may include demonstrating compliance with certification, security, legal, and/or privacy requirements of the first-party VR service provider320.

As an example, one of the requirements may be that the application, when executing on a host operated by the cloud service330, be executed in a container on that host, the container being configured according to specifications provided by the VR service provider320. These specifications may include firewall software that limits incoming and/or outgoing connections from the container to a pre-defined list of Internet Protocol (IP) addresses. These specifications may also include requiring end-to-end encryption of all data between the client device310and the host.

If the first-party VR service provider320verifies that the application meets the requirements, the application is (at380) approved and added to the application store or library of the VR service. From the store (at385) the application is now visible and accessible via shareable links. The application may be provided from the store to the client310upon request.

Protecting Sensitive Data

In some embodiments, the protocols used for communicating between the first-party VR provider's services, the user's equipment, and the third-party cloud VR service may be defined so that sensitive data and actions are performed entirely on the headset, through the use of system overlays and interfaces that know how to interact with the streamed VR experience. For example, logic on a server may indicate the user has chosen to make an in-app purchase (IAP). There may be a bespoke interface within the application, but the request to make the purchase may be relayed to the user's device, which may then present to the user a dialog box to get approval from the user for the purchase. The user may cancel the purchase, and the server may be notified. If the customer approves the purchase, the request may be sent from the device directly to the first-party VR provider's servers along with a token that may then be sent directly to the game server along with the results of the transaction. This model of performing sensitive activities on the headset, outside the reach of the third-party cloud VR provider, may be applied to any sensitive information not required for rendering.

In some embodiments, a third-party cloud VR server may need to have access to various forms of personally identifiable information (PII), including but not limited to motion tracker data and face/eye tracking data. This PII is information that VR applications may already have access to. Moving the logic that relies on PII to a cloud VR server may be safer than having headsets process it locally because the server instances may be firewalled and monitored. With a third-party cloud VR client running on a device controlled by a first-party VR provider, the risks of a third-party developer violating the first-party VR provider's terms of use may become much smaller because all communications may require explicit firewall permissions and may be auditable.

FIG.4illustrates a system400executing a process401for protecting sensitive data in a cloud VR ecosystem, according to some embodiments. The system400is similar to the embodiments of the system200and the system300discussed above with respect toFIG.2andFIG.3, respectively, and like reference numerals have been used to refer to the same or similar components. A detailed description of these components will be omitted, and the following discussion focuses on the differences between these embodiments. Any of the various features discussed with any one of the embodiments discussed herein may also apply to and be used with any other embodiments.

In the system400, data may be sent and received between a client device410(e.g., client device110), a first-party VR platform service420(e.g., one or more servers such as server130), and a third-party cloud service430(e.g., one or more servers such as server130). Various operations of the process401may be performed by different components of system400, as described below.

At450, the customer launches an application on the client device410. The application connects to the VR platform service420upon launch.

At455, the VR platform service420manages the deployment and activation of applications, and handles all requests for sensitive data, including but not limited to in-app purchases (IAP). The VR platform420notifies the cloud service430that the application has been launched on the client device410, and may provide minimal but sufficient metadata associated with the application and/or the customer, such as an IP address of the client device410, a user identifier, and the like, for the cloud server430to establish a connection with the application and facilitate the user experience.

At460, the cloud service430runs the actual application, streaming data to the client device410and relaying purchase requests and other metadata to VR platform services420. In some embodiments, the actual application is executed within a container or sandbox. The cloud service430has no ability to directly bill customers or trigger IAP/sensitive data requests outside a well-defined process that ensures that the user is informed and in control.

At465, the client device410executes a local streaming client, that is compatible with the application, and which is designed to handle data coming from both the VR platform service420and the connected cloud service430. The VR platform service420may offer different client streaming applications to meet the varying and evolving needs of cloud streaming applications offered by different third-party cloud service providers and different generations or types of cloud streaming applications offered by any given cloud service provider.

At470, the VR platform service420handles all requests for sensitive data, coming from the cloud service430, and provides the requests to the client device410.

At475, the client device410handles sensitive requests only by the request of the VR platform service420. In some embodiments, the client device410uses secure user input dialogs that cannot be emulated by the cloud service430. In such a manner, users are able to clearly understand when they are making a purchase or otherwise disclosing sensitive information, and the cloud service cannot conduct a transaction without clear feedback to the user to inform them.

Maintaining a Curated, High Quality Customer Experience

In some embodiments, with cloud VR, a first-party VR provider may provide a quality of service (QOS) that meets customer expectations, by building the streaming client application and ensuring the application will connect only to approved experiences on certified hosts. The process for launching a streaming application on the device and connecting it to a VR host may provide this functionality.

In some embodiments, launching a cloud VR application from a client device may include some or all of the following operations:

1. The client may indicate they want to launch a cloud VR application by sending a request to the first-party VR provider's services, for example, through the first-party VR provider's VR shell or other entry point. The client may be software running on a customer headset that communicates with the first-party VR provider's services.

2. The first-party VR provider service may verify the client's request is valid by verifying the application is approved for use and by finding the appropriate host to launch the streaming application. These hosts may be operated by certified third-party cloud providers, paid for by the publishers, and registered with the first-party VR provider's service.

3. Once selected, the host may be notified of the request and either launch the application or deny the request. The request may include information where to download or access the application package.

4. The first-party VR provider may notify the client of the result. If the request is fulfilled, the client may receive the host connection data and the first-party VR provider session token.

5. The client may then use the provided connection data to launch the correct streaming client application and connect to the third-party host. There may be multiple versions of the client application as the system evolves because requiring developers to maintain functionality with the latest version may be impractical.

The solution of some embodiments may ensure that clients can only connect to trusted and authorized applications, and that the system will support legacy protocols to ensure applications do not have to be continually updated as the service matures. The result may be a secure environment that maintains the integrity of the first-party VR provider's business model because the first-party VR provider may be certain that all accessible applications have sufficient content curation and quality of service (QOS).

Securing the Revenue Stream

Some embodiments provide payment system integration. A compromised server or malicious application could bill users without their knowledge. The solution to this may be to require applications to use the first-party VR provider's streaming client(s) on the headset, ensuring that the application streams only to trusted services running on certified providers. One of the core responsibilities of the client may be to interface with the first-party VR provider's services for any activities that require extra protection. The hosted VR application (e.g., third-party-hosted app running developer code) may relay such operations through the first-party VR provider's services, which may validate them and forward them to the client as necessary.

An example of payment system integration according to some embodiments is now described.

1. When a user in a cloud VR application decides to purchase an IAP, the user may activate a “Buy” button (or equivalent) in whatever interface is available to trigger the purchase.

2. A message may be sent from the cloud VR host to the first-party VR provider's services, which then forwards the message to the headset with information (e.g., metadata) about the purchase request after performing any necessary filtering (e.g., parental controls or account issues).

3. The user may be presented with a system dialog box whereby they may approve or decline the purchase.

4. The response may be sent to the first-party VR provider's services, which may process the purchase as needed and notify the cloud VR host about the completed purchase (or the declined purchase).

In the above scenario, at no point, either through deliberate act of the developer or by way of a compromised server, may the user be at risk of an unauthorized purchase because the dialog box may be a system service communicating directly with the first-party VR provider's servers.

Although developers may put their own payment system in an application to bypass the first-party VR provider's payment system, the first-party VR provider may not approve the application for the first-party VR provider's ecosystem if it contains such a workaround.

Additionally, to evade the payment system integration, the third-party developers would have to circumvent the firewalls on the cloud servers. The first-party VR provider may disable applications discovered to be using alternative payment systems by simply blocking users and not sending the users to the applications' hosts.

FIGS.5A and5Billustrate a system500executing a process501for launching a third-party cloud VR application on a VR platform service, according to some embodiments. The system500is similar to the embodiments of the system200, the system300, and the system400discussed above with respect toFIG.2,FIG.3, andFIG.4, respectively, and like reference numerals have been used to refer to the same or similar components. A detailed description of these components will be omitted, and the following discussion focuses on the differences between these embodiments. Any of the various features discussed with any one of the embodiments discussed herein may also apply to and be used with any other embodiments.

In the system500, data may be sent and received between a client device510(e.g., client device110), a first-party VR platform service520(e.g., one or more servers such as server130), and a third-party cloud service530(e.g., one or more servers such as server130). Various operations of the process501may be performed by different components of system500, as described below.

The third-party cloud VR application may include or communicate with a client-side component (e.g., a streaming client), such as the example of application222running within a portion of memory220-1on client device510. The third-party cloud VR application may also include a server-side component (e.g., a cloud server), such as application engine232running within a portion of memory220-2of a host of the cloud service530.

At532, the user uses a client device510(e.g., a headset, a mobile phone, a computer, and the like) to access the VR platform service520.

At534, the user determines that they desire to launch a cloud app associated with the cloud service530. The customer may come across this cloud app in any number of ways, including but not limited to discovering the cloud app on the VR platform's app store, rediscovering the cloud app on their client device510, or discovering the cloud app via a link from a chat, a webpage, social media, or other sources.

At536, the customer launches a streaming client app associated with the cloud app. In some embodiments, there may be a limited number of client apps managed by the platform, with new ones added as necessary to support new features. The streaming client sends a request537to the VR platform service520to launch the cloud app, and at538waits for connection information in response.

At540, the VR platform service520validates the request537from the client device510against various constraints, including billing, parental controls, entitlement status, etc. Other factors used to validate the request537may include, but are not limited to, age requirements, whether a customer is restricted or banned, whether the customer has a subscription, or other policies that may control whether the user or the user's account is permitted or capable of launching the application.

If the VR platform service520determines that the client's request537is invalid, then at541the request537may be declined, and the process501may return to532, which was described above.

If the VR platform service520determines that the client's request537is valid, then the request is approved at542. The customer may be notified at543, with a status update sent to the client device510.

Upon approval, the VR platform service520identifies at544a cloud provider to service the request537. In this example, the identified cloud provider is cloud service530. In some embodiments the VR platform service520may scan a registry of cloud providers and/or available hosts and selects a host that is best for the customer. In other embodiments, the identified cloud service530may select the host. The VR platform service520sends a connection request545to the selected host.

In some embodiments, the host selection may be based on various factors, including but not limited to region (e.g., to minimize latency), restrictions by the developer (e.g., the cloud app should not be hosted or should not be available in a certain country or region), or region-specific mods (e.g., censorship of certain images or video according to local law, rule, or custom).

If the VR platform service520determines, at546, that it has failed to identify a suitable cloud service530or host to service the request537, then the client may be notified at547that the service is unavailable, in response to the initial request537. The process501may return to532, which was described above.

The request545is received by the cloud service530(for example, by a handler that waits for such requests and processes the requests as they are received). The cloud service530determines at548if it will fulfill or deny the request545. The cloud service530may deny the request545for various reasons, such as the identified/selected host not having sufficient storage space, computing capacity, or other technological constraints. If the request545cannot be fulfilled, then a response549may be sent to the VR platform service520. The process501may return to544, which was described above, to select a different host and/or cloud service530.

If the cloud service530is able to fulfill the request545, then the request545is accepted at550. In some embodiments the cloud service530determines at551if the required assets (also referred to as a package) for the cloud app are locally available on the selected host. If the assets are already locally available, then the cloud service530may fulfill the request545immediately, and the process501may proceed to565, which is described below.

If the selected host does not have the assets available locally, the cloud service530may send a request550to the VR platform service520for authorization to acquire the missing assets.

The VR platform service520, at552, validates the request550. In some embodiments, the request550is based on metadata associated with the request550. If the metadata does not meet specified criteria, then the VR platform service520may at554deny the request550. The process501may return to544and/or548, which were described above.

If the metadata meets the criteria, or the VR platform service520validates the request550in another manner, then at556the VR platform service520grants permission to the client device510and the cloud service530to download any required assets (e.g., from a secure and previously defined location, a content distribution network, etc.). The client device510may be notified with asset download instructions557, and the cloud service530may be notified with asset download instructions558.

The client device510receives, at560, the instructions557, and determines whether there are any client assets that need to be downloaded prior to application launch. If so, the client device510acquires those assets. The location of those assets may be specified in the instructions557. The process501may proceed to575, which is described below.

The cloud service530receives the instructions558, and at562, retrieves the required assets. The location of those assets may be specified in the instructions558.

The cloud service530activates, at565, a container for the cloud app, and waits at567for a connection from the client device510. The cloud service530further sends metadata568to the VR platform service520, that includes the connection parameters that the client device510will need to connect to the selected host. In some embodiments, the metadata568may also include an authorization token that the client device510may need to authenticate the user to the cloud service530.

The VR platform service520receives, at570, the metadata568, and uses the metadata568to generate instructions572that includes the connection parameters, which are provided to the client device510. Instructions572may also include the authorization token, if present. In some embodiments, the VR platform service520also begins to track the host status, including cloud app state, as well as billing and payment data.

The client device510, at575, receives the instructions572that includes the connection parameters. Based on the instructions572, the client device510launches the corresponding streaming client.

The client device510, at576, activates the streaming client and attempts to connect to the cloud service530, using the connection parameters provided in instructions572. If the client device510determines at577that the connection has failed, then the process501may return to532, which was described above. If the connection succeeds, then the client device notifies the VR platform service520. If the connection status changes, additional notifications may be provided to the VR platform service520.

The cloud service530accepts, at578, the incoming connection from the client device510. The cloud service530also notifies the VR platform service520of the successful connection. If the connection status changes, additional notifications may be provided to the VR platform service520.

The VR platform service520receives, at579, notifications from the client device510and the cloud service530, regarding the connection status and uses that information to keep track of the connection state. The VR platform service520may use the connection information for platform features, including but not limited to presence tracking, fulfilling transactions and payment, and enabling social features such as joining friends, stream broadcast, etc.

In some embodiments, the connection between the client device510and the cloud service530is a bidirectional streaming connection580that is used for all data required for the streaming experience, including but not limited to visual data, input data, metadata, geometry, textures, audio, and text. On the client side, the client device510(at581) processes the stream, handles input, and performs other logic depending on the streaming client type. On the host side, the cloud service530(at582) performs hosted application logic, streaming data to the client as needed, and also may support additional connections for multiplayer experiences and content streaming to other authorized clients.

In some embodiments, during the user interaction with the cloud app, actions by the user may result at583in a request584for sensitive data from the cloud service530. As an example, the cloud app may be a game, and the sensitive data may be account information for an in-app purchase (IAP) event. The request584is sent to the VR platform service520, not directly to the client device510.

The VR platform service520receives, at585, the request584and validates the request. If the request584is valid, then the VR platform service520provides the client device510with the request584. If the request584is invalid, then the request584is denied. Various factors may prevent the request584or render it invalid, including but not limited to parental controls, billing configuration, and other constraints on the user or the user's account.

The client device510receives, at586, the valid request584from the VR platform service520. The client device510displays a secure dialog to the user, which includes information about the request584(e.g., the details of the IAP event that the user triggered as described above with respect to583). If the user does not recognize the request584(i.e., it is a fraudulent request) or they change their mind about the IAP event, then the user may decline. Regardless of whether the user approves or denies the request584, the client device510sends a corresponding notification587to the VR platform service520to notify of the user's decision.

The VR platform service520receives, at588, the notification587indicating that the user has approved or denied the request584. The VR platform service520sends a corresponding notification589to the cloud service530to notify of the user's decision.

The cloud service530receives, at590, the notification589regarding the user's decision. If the decision was approved, then the cloud service530responds according to application-specific behavior and logic, accordingly. For example, if the cloud app is a video game and the request584was for an IAP event pertaining to an in-game item, then the cloud service530provides the in-game item to the user.

If (at592) the bidirectional streaming connection580is terminated at the client device510for any reason (including but not limited to application failure, device power down, connection failure, user activity, etc.), the process501may return to532, which was described above.

If (at594) the bidirectional streaming connection580is terminated at the cloud service530for any reason (including but not limited to application error, platform notification, budget limits, etc.), the process501may proceed to596, where the VR platform service520(which is tracking the connection state, as described above with respect to579) updates the server status and resolves any billing associated with payments or purchases.

FIG.6illustrates a system600executing a process601for a customer to join a friend's activity in a cloud VR ecosystem, according to some embodiments. The system600is similar to the embodiments of the system200, system300, system400, and system500discussed above with respect toFIG.2,FIG.3,FIG.4, andFIG.5, respectively, and like reference numerals have been used to refer to the same or similar components. A detailed description of these components will be omitted, and the following discussion focuses on the differences between these embodiments. Any of the various features discussed with any one of the embodiments discussed herein may also apply to and be used with any other embodiments.

In system600, data may be sent and received between a client device610(e.g., client device110), a first-party VR platform service620(e.g., one or more servers such as server130), a third-party cloud service630(e.g., one or more servers such as server130), and an application developer640. Various operations of the process601may be performed by different components of system600, as described below.

At650, on the client device610, the customer sees a friend engaging in an activity within the VR environment. The activity may be using an application (e.g., a VR application), playing a game, conducting a live stream, or other activity. The customer may indicate their intention to join their friend and/or participate in that activity, by using an interface provided within the VR environment. For example, the customer may be provided an option to “join session,” “watch stream,” etc.

At660, the VR platform service620negotiates with the cloud service630, in response to the customer's indication, for access for the customer as a new user. This access may be in addition to existing rules for access. In some embodiments, the VR platform service620provides only the minimum information necessary for the customer to join the activity (e.g., a user identifier, and an IP address). In some embodiments, the cloud service630remains unaware of the social connection between the user and the friend, as the social network functionality is entirely handled by the VR platform service620.

At670, the cloud service630instructs the existing cloud instance to enable a connection for the new user, which will allow them to join their friend's session, watch their stream, or otherwise participate in their activity.

At680, the VR platform service620is notified of the connection approval by the cloud service630.

At690, the customer joins their friend's activity on the client device610.

Personal Information Concerns

Personal information may include, but is not limited to, motion controller data, individual participant data (IPD), and physical height. These are just some examples of data that is subject to data tracking on users. Access to this information may be necessary for a VR application to operate correctly. A first-party VR provider may rely on developer agreements and policies, as well as legal requirements, to prevent such information from being misused.

For example, in some embodiments, a first-party VR provider's privacy policy may list all the sources of data the first-party VR provider processes and how the first-party VR provider protects it, including how third parties may also collect information from a user directly through the experiences the third parties provide. The privacy policy may not state whether the processing of user data by an application occurs on a headset or in the cloud. Developers are obligated to protect this data to various degrees as required by law and their privacy policies regardless of whether their applications are running on the client device or a server. Shifting more of, if not most of, the processing of user data by an application from a device onto the cloud does not change these obligations.

In some embodiments, by hosting VR applications in a certified VR service provider, the first-party VR provider may increase the security of user data because it may be possible to limit network access to well-defined protocols and services, services that may become richer as the metaverse takes form and standards emerge for common activities. The end result may be a situation where user data may be at least as safe, if not safer, than when it is exposed to developer code on a headset. Hosting VR applications in the cloud may shift the risk to a different physical location where it could potentially benefit from increased oversight.

Paying for the Hosting Costs

In some embodiments, a first-party VR provider may design into the system a business model that keeps costs down and adapts to changes in costs in the marketplace. For example, the first-party VR provider may facilitate the connection and payment to hosts, simplifying the task of working with providers and delivering a system that “just works” for developers.

In some embodiments, the first-party developer may work with certified cloud VR providers to ensure there is a sufficient pool of services to meet the needs of all developers who may rely on these services for their applications. Some examples may allow for a bidding system where third-party cloud VR providers may adjust rates depending on load, availability, or other reasons, and the first-party VR provider may direct customers to the provider with the best rates. The first-party VR provider may capture a portion of that revenue.

Third-party developers may put cloud VR hosting fees into their budget just like any other project costs. If, for example, a VR gaming title has no budget for cloud services, then they would not be able to activate any VR streams. It could be that purchasing a title with 10 total hours of play time over for each user on average would allocate the necessary portion of the sales to cover 10 hours of cloud service fees. In this case, the first-party VR provider may in some embodiments provide a means for users to pay for the cloud service directly through an in-app purchase mechanism, where the fees are used to pay for the cloud service. This may ensure unmaintained apps that users have purchased are still playable in the event the publisher is no longer available. A more complete system may provide support for minimum hours per user per title, with a user-friendly process for a user to get more server time if they go outside the budgeted amount.

According to a system of some embodiments, instead of users paying a flat fee for access, each developer may be responsible for their own server usage, selected from an open marketplace of certified providers competing with each other. The end result may be a system where the first-party VR provider does not shoulder the entire risk of building and operating a cloud VR ecosystem on their own. The first-party VR provider's role may be that of an intermediary, ensuring the system is operational and providing the underlying services that developers and customers can rely on. Building the framework may keep the first-party VR provider out of the business of maintaining the data centers while at the same time providing the first-party VR provider with opportunities to earn part of the revenue stream that flows through it.

FIG.7is a flowchart illustrating a process700performed by a first-party VR platform service (e.g., servers130, first-party VR service provider320, first-party VR platform service420, first-party VR platform service520, first-party VR platform service620, and the like) for virtual reality cloud computing, according to some embodiments. In some embodiments, one or more operations in process700may be performed by a processor circuit (e.g., processors212, etc.) executing instructions stored in a memory circuit (e.g., memories220, etc.) of a system (e.g., system200, system300, system400, system500, system600, and the like) as disclosed herein. Moreover, in some embodiments, a process consistent with this disclosure may include at least operations in process700performed in a different order, simultaneously, quasi-simultaneously, or overlapping in time.

At710, the process700receives, from a client device, a request by a user of the client device to execute an application, the application being provided by a third-party cloud service having one or more hosts.

At720, the process700selects, from the hosts, a particular host to service the request.

At730, the process700receives, from the particular host, connection parameters that are required for establishing a streaming connection to the particular host.

At740, the process700provides, in response to the request, the connection parameters to a streaming client executing on the client device.

At750, the process700instructs the streaming client to establish the streaming connection to the particular host and to execute the application.

In some embodiments, the particular host requests to acquire software assets required to execute the application. Based on a determination that the software assets meet certification criteria, the process700authorizes the particular host to retrieve the software assets. Based on a further determination that the streaming client requires additional software assets to execute the application, the process700further instructs the streaming client to retrieve the additional software assets.

In some embodiments, the process700further provides to the streaming client, a shared virtual environment and metadata associated with the user, the metadata including a list of friends of the user who are currently also within the shared virtual environment. In some such embodiments, the process700receives from a particular friend of the user within the shared virtual environment, a request to participate in an activity being performed by the user through the streaming client. The process700further provides, to the particular host, metadata associated with the particular friend, provides the connection parameters to a second streaming client executing on a second client device being used by the particular friend, and instructs the second streaming client to establish the streaming connection to the particular host and to execute the application.

FIG.8is a flowchart illustrating a process800performed by a first-party VR platform service (e.g., servers130, first-party VR service provider320, first-party VR platform service420, first-party VR platform service520, first-party VR platform service620, and the like) for processing in-application transactions, according to some embodiments. In some embodiments, one or more operations in process800may be performed by a processor circuit (e.g., processors212, etc.) executing instructions stored in a memory circuit (e.g., memories220, etc.) of a system (e.g., system200, system300, system400, system500, system600, and the like) as disclosed herein. Moreover, in some embodiments, a process consistent with this disclosure may include at least operations in process800performed in a different order, simultaneously, quasi-simultaneously, or overlapping in time.

At810, the process800receives, from a particular host of a third-party cloud service, a request for a transaction, the transaction being a payment by a user in exchange for a product. In some embodiments, the transaction is an in-application purchase, and the product is at least one of additional content and services within the application.

At820, the process800requests, from a streaming client used by the user on a client device, an authorization to conduct the transaction.

At830, the process800receives, as a response from the streaming client, the authorization to conduct the transaction.

At840, the process800receives, in response to the request, a payment from the user.

At850, further in response to the request, the process800instructs the particular host to provide the product to the user through the streaming client on the client device.

Alternatively, if at830, the process800does not receive authorization to conduct the transaction, the process800denies the request from the particular host.

Many of the above-described features and applications may be implemented as software processes that are specified as a set of instructions recorded on a computer-readable storage medium (alternatively referred to as computer-readable media, machine-readable media, or machine-readable storage media). When these instructions are executed by one or more processing unit(s) (e.g., one or more processors, cores of processors, or other processing units), they cause the processing unit(s) to perform the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, RAM, ROM, read-only compact discs (CD-ROM), recordable compact discs (CD-R), rewritable compact discs (CD-RW), read-only digital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a variety of recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.), magnetic and/or solid state hard drives, ultra-density optical discs, any other optical or magnetic media, and floppy disks. In one or more embodiments, the computer-readable media does not include carrier waves and electronic signals passing wirelessly or over wired connections, or any other ephemeral signals. For example, the computer-readable media may be entirely restricted to tangible, physical objects that store information in a form that is readable by a computer. In one or more embodiments, the computer-readable media is non-transitory computer-readable media, computer-readable storage media, or non-transitory computer-readable storage media.

While the above discussion primarily refers to microprocessor or multi-core processors that execute software, one or more embodiments are performed by one or more integrated circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In one or more embodiments, such integrated circuits execute instructions that are stored on the circuit itself.

It is understood that any specific order or hierarchy of blocks in the processes disclosed is an illustration of example approaches. Based upon implementation preferences, it is understood that the specific order or hierarchy of blocks in the processes may be rearranged, or that not all illustrated blocks be performed. Any of the blocks may be performed simultaneously.

The subject technology is illustrated, for example, according to various aspects described above. The present disclosure is provided to enable any person skilled in the art to practice the various aspects described herein. The disclosure provides various examples of the subject technology, and the subject technology is not limited to these examples. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects.

The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments. In one aspect, various alternative configurations and operations described herein may be considered to be at least equivalent.

In one aspect, unless otherwise stated, all measurements, values, ratings, positions, magnitudes, sizes, and other specifications that are set forth in this specification, including in the claims that follow, are approximate, not exact. In one aspect, they are intended to have a reasonable range that is consistent with the functions to which they relate and with what is customary in the art to which they pertain. It is understood that some or all steps, operations, or processes may be performed automatically, without the intervention of a user. Method claims may be provided to present elements of the various steps, operations, or processes in a sample order, and are not meant to be limited to the specific order or hierarchy presented.

All structural and functional equivalents to the elements of the various configurations described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are expressly incorporated herein by reference and intended to be encompassed by the subject technology. Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether such disclosure is explicitly recited in the above description. No claim element is to be construed under the provisions of 35 U.S.C. § 112, sixth paragraph, unless the element is expressly recited using the phrase “means for” or, in the case of a method claim, the element is recited using the phrase “step for.”

The Title, Background, and Brief Description of the Drawings of the disclosure are hereby incorporated into the disclosure and are provided as illustrative examples of the disclosure, not as restrictive descriptions. It is submitted with the understanding that they will not be used to limit the scope or meaning of the claims. In addition, in the Detailed Description, it can be seen that the description provides illustrative examples, and the various features are grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the included subject matter requires more features than are expressly recited in any claim. Rather, as the claims reflect, inventive subject matter lies in less than all features of a single disclosed configuration or operation. The claims are hereby incorporated into the Detailed Description, with each claim standing on its own to represent separately patentable subject matter.

Embodiments consistent with the present disclosure may be combined with any combination of features or aspects of embodiments described herein.