Distributed sample-based game profiling with game metadata and metrics and gaming API platform supporting third-party content

This application is directed to real-time game profiling and analysis. While a game server executes multiple game instances for a gaming title in parallel, a profiling server generates a game profiling instruction to collect data samples associated with a subset of the game instances and sends the game profiling instruction to the game server. In response to the game profiling instruction, the game server identifies the subset of the plurality of game instances, and collects data samples associated with each game instance in the identified subset. Each collected data sample includes at least game state data and system performance data associated with synchronous gaming sessions corresponding to the respective game instance. The game server sends the collected data samples associated with each game instance in the identified subset to the profiling server, which provides the plurality of data samples to an analytics server for further analysis.

TECHNICAL FIELD

This application relates generally to computer technology, including but not limited to methods and systems for creating a gaming environment to enable real-time game profiling and analysis.

BACKGROUND

Internet-connected electronic devices can support a variety of cloud-based media and entertainment applications. These applications include media streaming applications in which a server streams content to user devices, gaming applications in which a user interacts from a user device with a game that executes on a server, and a variety of social media and communication applications that allow large numbers of users to interact concurrently with each other and with cloud-hosted content and applications via their Internet-connected devices. Among cloud-based applications, cloud gaming presents some unique challenges due to: the widely varying hardware demands of gaming titles; the diverse topologies in which cloud-based games can be played (e.g., by a single player, by multiple players in a single location, or by multiple players in multiple locations); the need to transmit reliably and without latency player inputs to a gaming server that executes a gaming session and gaming session outputs from the gaming server to the players' devices/displays; widely varying player expectations as to speed and responsiveness of gameplay; and the desire in some situations to provide near-real time gaming content to spectators. Other challenges of cloud based gaming relate to providing a consistent gameplay experience for players regardless of where they are located (e.g., close or far from the server), how they connect to the gaming service (e.g., via a fast or slow Internet connection), and what type of device(s) they use to play the game (e.g., a generic personal device or a dedicated game controller) and view gameplay outputs (e.g., a personal device or a media device connected to a media streaming device).

Specifically, there is a need for a cloud gaming system that supports multiple gaming sessions for multiple gaming titles, where the games can execute concurrently with acceptable latency and responsiveness, including for multiple players who are playing the same gaming title from the same or different locations, with a wide variety of input and output devices and network connections. In addition, there is a need for a cloud gaming system that, upon receiving a player input (e.g., a gaming input entered on an end use gaming device/controller) in a gaming session, processes the user input promptly and outputs high-definition images reflecting the outcome of the player input action for all of the game players simultaneously and with acceptable latency. There is also a need to track statuses of different gaming titles, instances and sessions at a micro-level and understand causes of their performance improvement or degradation at a macro-level. As such, it would be beneficial to provide a cloud gaming system hosting online interactive gameplay for multiple users from different locations with efficient game profiling and analysis mechanisms without compromising game processing and outputting or gaming experience at the cloud gaming system.

SUMMARY

Implementations described in this specification are directed to providing a gaming application programming interface (API) and cloud gaming platform to enable efficient game profiling and performance analysis of online interactive gameplay of a selected gaming title hosted by one or more game servers. Specifically, game profiling and analysis is performed in real time while one or more game instances of the gaming title are executed on the game servers to enable gameplay in multiple synchronous gaming sessions for multiple users from different locations. Computation, storage and communication resources of the cloud gaming platform are allocated among game processing, game profiling and performance analysis to bring efficient and concurrent gameplay, game profiling and performance analysis. For example, processing capabilities of the game servers are substantially reserved for game processing (e.g., gaming content rendering) by allocating tasks of game profiling control and performance analysis to servers distinct from the game servers, e.g., a dedicated game profiling server and an offline analytics server. In some situations, a game profiling and analysis application is hosted and executed by the game profiling server and/or analytics server, independently of execution of a game application managing gameplay on the game servers. By these means, gameplay can be conducted efficiently with the processing capabilities conserved on the game servers, and more importantly, can benefit from performance feedback provided by real-time game profiling and performance analysis.

In one aspect of this application, a method is implemented at a distributed real-time game profiling system. The method includes executing by a game server a plurality of game instances for a gaming title in parallel. Each of the plurality of game instances includes one or more synchronous gaming sessions, and each of the one or more gaming sessions is executed at a game application of a client device. The method further includes while the game server is executing the plurality of game instances, generating by a profiling server a game profiling instruction to collect a plurality of data samples for each of a subset of the plurality of game instances. The profiling server is coupled to the game server over one or more wireless communication networks. The method further includes sending the game profiling instruction to the game server by the profiling server. The method further includes in response to the game profiling instruction, identifying by a local orchestrator module of the game server the subset of the plurality of game instances according to the game profiling instruction and collecting by a collection module of the game server the plurality of data samples associated with each game instance in the identified subset. Each of the plurality of data samples includes at least game state data and system performance data associated with the one or more synchronous gaming sessions corresponding to the respective game instance. The method further includes sending by the game server the plurality of data samples associated with each game instance in the identified subset to the profiling server, and providing the plurality of data samples to an analytics server, thereby allowing the analytics server to analyze the plurality of data samples to improve gaming performance and/or user experience associated with the gaming title.

In some implementations, the game state data includes game context data associated with a gaming session, e.g., a timestamp corresponding to a specific instant of time, a game level, a weapon level, a costume type and a number of objects in a game scene. In some implementations, the game state data includes one of more of game settings including a sliding window frame rate, and a frame resolution of image frames rendered for the corresponding gaming session. In some implementations, for each game instance, the system performance data includes performance data of a CPU, a GPU, a memory, an operating system, one or more input/output ports and/or a data bus, which are assigned to execute each of the one or more gaming sessions of the respective game instance. In some implementations, the system performance data are stored in at least one of CPU performance counters, GPU performance counters, a stack trace, and architectural counters.

In some implementations, the method further includes visualizing by the analytics server a subset of the game state data and system performance data in the plurality of data samples. Alternatively, the method further includes at the analytics server, comparing two of the plurality of data samples corresponding to a first time and a second time subsequent to the first time, identifying an anomalous profile event based on a comparison result, and generating an alert message indicating occurrence of the anomalous profile event. Alternatively, in some implementations, the game server includes a plurality of processors, and the method further includes one or more of: conducting hotspot analysis of the plurality of processors based on the plurality of data samples, profiling the plurality of processors based on the plurality of data samples, and profiling the plurality of game instances and the one or more synchronous gaming sessions corresponding to each game instance.

In another aspect of the invention, a distributed real-time game profiling system includes one or more processors, and memory having instructions stored thereon, which when executed by the one or more processors cause the distributed real-time game profiling system to perform any of the methods described above.

In another aspect of the invention, a non-transitory computer-readable medium has instructions stored thereon, which when executed by one or more processors cause processors of a distributed real-time game profiling system to perform any of the methods described above.

Other embodiments and advantages may be apparent to those skilled in the art in light of the descriptions and drawings in this specification.

DESCRIPTION OF IMPLEMENTATIONS

In various implementations described in this application, a gaming environment includes a plurality of media devices (e.g., a casting device) each of which is optionally coupled to an output device, a plurality of client devices communicatively coupled to the media devices, and a server system coupled to the media devices and the client devices. The server system executes a plurality of game instances for a gaming title in parallel, and each of the plurality of game instances includes one or more synchronous gaming sessions. Each of the one or more gaming sessions is executed at a game application of a client device to enable gameplay via the client device and optionally via a corresponding media device and/or output device. Optionally, the server system is configured to manage a virtual game domain in which the plurality of media devices and the plurality of client devices are linked to each other via a user account of the game application.

Gameplay activities in the gaming environment is monitored and analyzed by a distributed real-time profiling system including at least a profiling server in addition to one or more game servers. While the one or more game servers are executing the plurality of game instances, the profiling server instructs the game servers to collect a plurality of data samples for a subset of the game instances. In response, the game servers identify the subset of game instances and collects the plurality of data samples each of which includes at least game state data and system performance data associated with the one or more synchronous gaming sessions corresponding to each of the subset of game instances. Optionally, the plurality of data samples is provided to the profiling server or an analytics server to allow the profiling or analytics server to analyze the plurality of data samples to improve gaming performance and/or user experience associated with the gaming title. By these means, the game instances are monitored and analyzed in real time by the profiling server and/or the analytics server that are distinct from the game servers, allowing gameplay to occupy a substantial portion of resources of the game servers while still benefiting from performance improvement enabled by real-time analysis results.

Online Gaming Environment

FIG.1is an example online interactive gaming environment100in accordance with some implementations. The online interactive gaming environment100includes one or more client devices (e.g., client devices102and104). Each of the client devices102executes one or more game applications. An online gaming session can be initiated and run on a specific game application to allow a user of the client device102to play an online interactive game (also called “gaming title”) hosted by a server system114. In some implementations, the client device102(e.g., a host client) is configured to invite one or more other client devices102to join a game scene of the online interactive game. Gaming sessions of these client devices102are synchronized to display the same game scene of the online interactive game, optionally with distinct perspectives corresponding to their respective users. These synchronous gaming sessions are collectively called a game instance. That said, in some implementations, the server system114hosts a plurality of gaming titles each of which includes a plurality of game instances, and each game instance includes a plurality of synchronous gaming sessions executed at a game application of a client device102.

The server system114hosts an online interactive game platform to support the client devices102to play the one or more game applications including the specific game application. Specifically, the server system114includes a plurality of user accounts associated with the client devices102, and authenticates the users of the client devices102in association with each of the one or more game applications. The server system114renders and refreshes a game scene of a gaming title for the client devices102that join corresponding gaming sessions associated with the scene in a substantially synchronous manner. In some implementations, the server system114may assess the capabilities of the client devices102and/or a quality of the communicative connection between the server system114and each of the client devices102, and adaptively generates synchronous data streams for the gaming sessions associated with the client devices102. By these means, the server system114is configured to facilitate synchronous gaming sessions of an online interactive game on two or more client devices102simultaneously and with substantially low latencies.

In some implementations, a client device102(e.g., client device102A or102B) is coupled to a media device106. The media device106is further coupled to one or more output devices108that can output visual and/or audio content (e.g., a television, a display monitor, a sound system, speakers, etc.), and thereby configured to output content to the output device(s)108. The client device102is configured to control gameplay, and the media device106is configured to receive gaming content from the server system114via communication networks112and casts the received gaming content onto the output device108. Specifically, the client device102can be communicatively coupled to the media device106directly (e.g., via a wire, Bluetooth or other wireless communication links), via a local network110(e.g., a Wi-Fi network), or via one or more communication networks112. In some implementations, the client device102and the media device106are local to each other (e.g., in the same room, in the same house, etc.). In some implementations, the media device106is a casting device (e.g., Chromecast by Google Inc.) or a device that otherwise includes casting functionality. Alternatively, in some implementations, a client device102(e.g., client device102C) has a display screen integrated therein for displaying media or gaming content associated with a respective online gaming session directly without casting the content onto an output device108via a media device106.

In some implementations, the online interactive gaming environment100further includes an administrator/developer client device104configured to provide a graphic user interface (GUI) in a game profiling and analysis application. The GUI is displayed on the client device104, and an administrator or developer can enter user inputs on the GUI to request specific game profiling or analysis to monitor game and system performance of one or more game instances while a corresponding gaming title is being executed in the online interactive gaming environment100. The client device104is optionally connected with the communication networks112directly via a cellular communication network, or indirectly via a local area network110′.

In some implementations, the server system114includes one or more game servers118and a media streaming server120. The server system114is configured to provide two or more media streams130and132concurrently for an online interactive gaming session running on a first client device102A. The two or more media streams include a low latency stream130and a normal latency stream132that are provided to the first client device102A and a reviewer device via one or more communication network112, respectively. While a user of the first client device102A plays the gaming session on the first client device102A, the gaming session is recorded and broadcast to one or more spectators via the normal latency stream132, i.e., the spectators can review the gaming session on the reviewer client device. The low latency stream130corresponds to gameplay of the online interactive gaming session, and has a faster response rate and lower transmission latency than the normal latency stream132that corresponds to an associated review session. Optionally, the reviewer device is located in the same venue as the first client device102A. Optionally, the reviewer device (e.g., the client device102C) is remote from the first client device102A. In some implementations, the reviewer device is another client device like the first client device102A, and optionally has an integrated or external display. In some implementations, the reviewer device is a media device106(e.g., a device having casting functionality) coupled to one or more output devices108that can output visual and/or audio content. In some implementations, both the low latency stream130and the normal latency stream132are concurrently displayed on a screen (e.g., on a split screen, using an embedded window).

In some implementations, the game server118provides the normal latency stream132to the review device while providing the low latency stream130to the first client device102A. In some implementations, the game server118provides the normal latency stream132to a media streaming server120while providing the low latency stream130to the first client device102A. The media streaming server120provides the normal latency stream132to the review device. The media streaming server120optionally hosts a media streaming site (e.g., YouTube) to provide media content to subscribers. In some implementations, the media content (e.g., the normal latency stream132) may be provided to a subset of the subscribers based on a membership level.

Each client device102or104is capable of data communication and information sharing with a central server or cloud-computing system (e.g., the server system114), and/or other devices (e.g., another client device102or104, a media device106and an output device108) that are network-connected. Data communication may be carried out using any of a variety of custom or standard wireless protocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.11a, WirelessHART, MiWi, etc.) and/or any of a variety of custom or standard wired protocols (e.g., Ethernet, HomePlug, etc.), or any other suitable communication protocol, including communication protocols not yet developed as of the filing date of this document. In some embodiments, the online interactive gaming environment100includes a conventional network device (e.g., a router associated with a local area network110) via which a set of client devices102and their corresponding media and output devices (if any) are communicatively coupled to each other on a local area network110. The local area network is communicatively coupled to other part of the communication networks112(e.g., wide-area networks and the Internet). In some embodiments, each of the client devices102optionally communicates with one or more other client devices102, a respective media device106, or a respective output device108using one or more radio communication networks (e.g., ZigBee, Z-Wave, Insteon, Bluetooth, Wi-Fi, and/or other radio communication networks). In an example, the client device102A or102C is optionally connected with the communication networks112directly via a cellular communication network, or indirectly via a local area network110′.

In some implementations, the client devices102are remote from each other, i.e., they are not located in the same room or even structure. A gaming title may be started on a client device102by launching a game application (e.g., game application732,FIG.7) for execution at each client device102. In some implementations, for each client device102, the game application establishes an online gaming session116with the server system114independently. The online gaming sessions116of two or more client devices102(e.g.,102A and102B) are related to each other (e.g., because they are played in the same game domain/instance of the gaming title), and therefore, share a game scene in the game application. The related online gaming sessions116are synchronized with each other, and each online gaming session116optionally shows the same game scene with a unique player perspective corresponding to the respective client device102. A user of each client device102can therefore play the game on the respective client device and influence the output from the online gaming sessions116on the other client device(s)102of the same gaming instance. For example, in some implementations, server-side game applications628are configured to maintain consistent game states among and between users who are concurrently playing the same game instance. In some implementations, the server system114and server-side game applications628maintain individual sessions for each user playing a game that might result in a rendered view of a scene that is particular to that user's individual game play state and view, but each of those sessions are consistent with other sessions of users who are playing the same game instance and interacting with and viewing the same scene.

Alternatively, in some other implementations, after the game application of a first client device102A establishes an online gaming session116A, one or more second client devices102B are invited to join the online gaming session116A by an invitation message, and for example, a message with the link (e.g., a URL address) to join the online gaming session116A is sent to each of the second client devices102B. An appropriate controller configuration is provided to each second client device102B that is invited to join the online gaming session116A. When the second clients102B join an online gaming session116A, the server system114creates a separate gaming session116B for each individual second client device102B and associates the gaming sessions116A and116B with a game instance. In the game instance, each separate gaming session116B of the respective second client device102B is synchronized with and shares the same scene with the gaming session116A of the first client device102A, but can have a unique player perspective corresponding to the respective second client device102B. After each second client device102B has joined the online gaming session116A (more accurately, started its related online gaming session116B), a user can play the game on the respective second client device102B and influence the output of the online gaming sessions116A running on the other client device(s)102in the same game instance.

The client device102is a device that includes, and can run, one or more distinct user applications (e.g., game applications628inFIG.6A) including the game application. In some implementations, the client device102is a smartphone, a tablet device, a laptop computer, a desktop computer, a multimedia device or a remote control. In some implementations, the client device102includes a dedicated game controller device (e.g., game controller102B). In some implementations, the client device102includes one or more user applications that are configured to operate in conjunction with the media device106. In some implementations, the applications include a media device application for pairing the client device102with the media device106and configuring the media device106. The applications also include one or more applications that can cast associated content to the media device106. In some implementations, an application casts data and/or content to the media device106by sending the data/content directly to the media device106(e.g., via the local network) and/or by directing the media device106to a remote location (e.g., a URL or other link to a location at the server system114or a gaming content host) from which the media device106can stream or otherwise receive data/content. The media device106receives data/content from the application and/or the remote location and outputs visual and/or audio content corresponding to the received data/content to the output device108. Thus, an online gaming session116is established between the game application running on the client device102, the remote server system114, and the media device106.

In some implementations, as part of the process of linking related online gaming sessions116in a game instance, the server system114assesses the capabilities of each corresponding client device102and/or a quality of the communicative connection between the server system114and the client device102. In some implementations, the server system114measures network latency between the client device102and the server system114. If the measured latency is above a threshold and a lower-latency connection is available, the server system114can suggest that the client device102change to the lower latency connection, or invite a user of the client device102to change the client device102to the lower latency connection. For example, if the client device102is on a cellular wireless connection, and a local network is available, the server system114can suggest that the client device102should connect through the available local network. In some implementations, the latency threshold requirements differ between games. For example, some games (e.g., action games) are best experienced on lower latency connections, and some other games (e.g., online board games or card games) are not as demanding with respect to latency. The server system114may make connection recommendations in view of these different requirements associated with different types of games.

As part of a client device102starting or joining the gaming session116, the server system114communicates with the client device102to set up a controller on the client device102. In some implementations, this includes the server system114assessing whether the client device102has the needed resources and communication capability for the controller. Depending on available resources at the client device102, connection quality, and requirements for the game, the controller may be implemented differently at the client device102. In some implementations, a gaming title can be played with a webpage-based controller interface. For example, a controller interface for the game may be embedded in a webpage, and the webpage is rendered in a web browser on the client device102. Alternatively, in some implementations, a standardized controller is implemented in a predefined application not specific to the game or directly associated with the game (e.g., a casting device application, such as Chromecast or Google Cast by Google Inc., or other media device application), or in the operating system of the client device102. For example, the device operating system or a predefined application on the client device102may have a controller sub-module. The controller sub-module includes one or more standardized controller configurations, templates, or the like. Each of the standardized controller configurations configures the controller sub-module to utilize input devices and/or sensors on the client device102in some way to implement a virtual controller. The standardized controller configuration is used may vary with the game and/or with the type of client device.

Further, in some implementations, a game has a specific controller configuration that may be implemented on the controller sub-module. Such a configuration may be stored at the server system114and transmitted to the client devices102, as part of the process of the client devices102joining or starting the online gaming session116. In some implementations, a specific controller configuration can be an entirely custom controller or a mix of standard controller and a custom controller. Additionally, in some implementations, a game requires a specific application associated with the game. For example, a game may require a controller application associated specifically with the game. In some implementations, the client device102may be directed to download the specific application or the predefined application as part of starting or joining the session116. For example, if the client device102does not already have the predefined application (with the controller sub-module) or the specific application associated with game, and such an application is required for play, the server system114instructs the client device102to prompt its user that a download is needed and to ask the user for permission to proceed.

The server system114further includes a database160. The database160stores at least user information associated with user accounts of each of one or more game applications (e.g., game applications732,FIG.7) that are hosted on the server system114. Examples of the user information include, but are not limited to, user account information (e.g., identification and passwords), membership type, preference, and activity history. In some implementations, the database160stores session data (also called game state data) associated with the online gaming sessions that are played on the client devices102. Examples of the session data for each online gaming session116include, but are not limited to, a frame rate, a rendering specification, a normal latency requirement, information of GPU allocation, information of encoder allocation, identifications of related sessions, shader pipeline identification, and latest status information. The session data is accessed more frequently than the user information. In some implementations, memory storing the user information and the session data is made of a flash memory and random-access memory (RAM), respectively. In some implementations, a subset of the user information and session data is stored in each game server118corresponding a respective gaming title.

After the client devices102log onto their respective user accounts and start/join their online gaming sessions116, the game servers118set up the gaming sessions116, and manage each specific gaming session116for a respective client device102by obtaining gaming contents from content servers (e.g., content servers450inFIG.4), sending the gaming contents to the game applications executed on the client devices102or designated media devices106, identifying user requests or actions, rendering gameplay outputs for the client devices102or media devices106in response to the user requests or actions, and storing game state data during the respective gaming session116. In some implementations, the content servers provide third party gaming content, e.g., sports games, racing games, role playing games (RPG) and first person shooter (FPS) games. Different instances of these games may have widely varying cloud hardware requirements and network (e.g., to ensure an optimal user gaming experience—consistent in some instances with different subscription performance tiers) based on different associated latency requirements and expectations, output video resolution, and gaming server computational workload and video encoding/streaming resources, and network bandwidth.

In some implementations, the game servers118dynamically allocate cloud gaming hardware resources (e.g., GPUs406and encoder408inFIG.4) and monitor and utilize network bandwidth available to individual end users to provide optimal cloud gaming experience. In some implementations, the game servers118provide multiple performance tiers, including a tier that supports high performance, real time gaming sessions with high definition video/media streams and another tier that supports review media streams that has correspond with little or no latency to one or more actual gaming streams.

FIG.2illustrates a network-level view of an extensible devices and services platform200with which the online interactive gaming environment100ofFIG.1is integrated in accordance with some implementations. The extensible devices and services platform200includes remote servers or cloud computing system114(i.e., a server system114inFIG.1). Each client device102or media device106may communicate with the remote servers or cloud computing system114. For example, a connection to the one or more networks112may be established either directly (e.g., using 3G/4G connectivity to a wireless carrier), or through a network interface110(e.g., via a router, switch, gateway, hub, or an intelligent, dedicated whole-home control node), or through any combination thereof.

In some implementations, the devices and services platform200communicates with and collects data from the client devices102of the online gaming environment100. In addition, in some implementations, the devices and services platform200communicates with and collects data from a plurality of online gaming environments across the world. For example, the server system114collects game data202from the client devices of one or more online gaming environments, where the client devices may routinely transmit game data or may transmit game data in specific instances (e.g., when a device queries the game data302). Example collected game data202includes, without limitation, game context, game settings and image frames data of different online gaming sessions of different gaming titles, and performance data of hardware and software systems that execute the corresponding gaming titles.

In some implementations, the server system114provides one or more services204to the online gaming environments. Example services204include, without limitation, software updates, customer support, game data collection/logging, remote access, remote or distributed control, and/or use suggestions (e.g., based on the collected game data202) to improve performance, reduce utility cost, increase safety, etc. In some implementations, data associated with the services204is stored at the server system114, and the server system114retrieves and transmits the data at appropriate times (e.g., at regular intervals, upon receiving a request from a user, etc.).

In some implementations, the extensible devices and the services platform200includes a processing engine206, which may be concentrated at a single server or distributed among several different computing entities without limitation. In some implementations, the processing engine206includes engines configured to receive data from the client devices of online gaming environments (e.g., via the Internet and/or a network interface), to index the data, to analyze the data and/or to generate statistics based on the analysis or as part of the analysis. In some implementations, the analyzed data is stored as game analysis data208.

Results of the analysis or statistics may thereafter be transmitted back to the client devices102that has provided game data202used to derive the results, to other devices, to a server providing a webpage to a user of the device, or to other non-gameplay device entities (e.g., game developers, researchers, policymakers). In some implementations, use statistics, use statistics relative to use of other devices, use patterns, and/or statistics summarizing game data202are generated by the processing engine206and transmitted. The results or statistics may be provided via the one or more networks112. In this manner, the processing engine206may be configured and programmed to derive a variety of useful information from the game data202. A single server may include one or more processing engines.

The game analysis data208may be used at different granularities for a variety of useful purposes, ranging from explicit programmed control of the client devices102on a per-session, per-instance, or per-gaming title basis, to the generation of inferential abstractions that may assist on a session, instance or gaming title basis (for example, an inference may be drawn that the game player has seen frame drops and register overflow when a L block appears in Tetris and so more computational and register resources has to be allocated to a corresponding session when L blocks appear), to the generation of statistics and associated inferential abstractions that may be used for other purposes. For example, the processing engine206may generate statistics about frame drops across a population of client devices and send the statistics to device users, game providers/developers, system administrators, or other entities (e.g., entities that have requested the statistics and/or entities that have provided monetary compensation for the statistics).

In some implementations, to encourage innovation and research and to increase products and services available to users, the devices and services platform200exposes a range of application programming interfaces (APIs)210to third parties, such as gaming content providers214, third party game platforms216, academic institutions218(e.g., university researchers), related businesses220(e.g., providing device warranties or service to related equipment, targeting advertisements based on game data), game administrators224, and other third parties. The APIs210are coupled to and permit third-party systems to communicate with the server system114, including the services204, the processing engine206, the game data202, and the game analysis data208. In some implementations, the APIs210allow applications executed by the third parties to initiate specific data processing tasks that are executed by the server system114, as well as to receive dynamic updates to the game data202and the game analysis data208.

For example, third parties (e.g., a third party game platform) may develop programs and/or applications, such as web applications or mobile applications, that integrate with the server system114to provide services and information to users. Such programs and applications may be, for example, designed to manage different subscription models, to identify faults and bottlenecks in game programs, to prepare for high gaming demands, to track past game service performance, to provide gaming assistance, to support social networking service, etc., and/or to perform other beneficial functions or tasks.

FIG.3illustrates an abstracted functional view300of the extensible devices and services platform200ofFIG.2, with reference to a processing engine206as well as devices of the online gaming environment100, in accordance with some implementations. Even though devices situated in online gaming environments will have a wide variety of different individual capabilities and limitations, the devices may be thought of as sharing common characteristics in that each device is a data consumer302(DC), a data source304(DS), a services consumer306(SC), and a services source308(SS). Advantageously, in addition to providing game information (e.g., gaming content and context information concerning other players in the same game scene/instance) used by the client devices102to achieve their gaming objectives, the extensible devices and services platform200may also be configured to use the large amount of game-related data that is generated by these client devices102during their gaming sessions. In addition to implementing or optimizing the actual operation of the devices themselves with respect to their immediate game functions, the extensible devices and services platform200may be directed to “repurpose” that data in a variety of automated, extensible, flexible, and/or scalable ways to achieve a variety of useful objectives. These objectives may be predefined or adaptively identified based on, e.g., usage patterns, device efficiency, and/or user input (e.g., requesting specific functionality).

The processing engine206includes a number of processing paradigms310. In some implementations, the processing engine206includes a managed gameplay services paradigm310athat monitors and manages gameplay functions of the client devices102. The gameplay functions may include ensuring proper multiplayer gameplay associated with a game instance given user inputs, troubleshooting problems that occur during gameplay, monitoring and analyzing performance of gaming sessions, and other game-related functions. In some implementations, the processing engine206includes an advertising/communication paradigm310bthat estimates characteristics (e.g., demographic information), desires and/or features of interest of a user based on gameplay activities. Services, promotions, products or upgrades may then be offered or automatically provided to the user. In some implementations, the processing engine206includes a social paradigm310cthat uses information from a social network, provides information to a social network (for example, based on gameplay activities), and/or processes data associated with user, gaming session and/or interactions with the social network platform. For example, a user's status or level in a gaming title is reported and updated to their trusted contacts on the social network. As another example, a user may be able to share gameplay tricks with other users. In yet another example, a user may arrange a gameplay competition related to a specific gaming title on the social network.

In some implementations, the processing engine206includes a challenges/rules/compliance/rewards paradigm310dthat informs a user of challenges, competitions, rules, compliance regulations and/or rewards and/or that uses operation data to determine whether a challenge has been met, a rule or regulation has been complied with and/or a reward has been earned. The challenges, rules, and/or regulations may relate to a specific gaming title. For example, one challenge may involve participants reaching an advanced skill level of a specific gaming title within a predetermined period of time (e.g., within one week). Those game players who successfully complete the challenge are rewarded, such as with coupons, virtual currency, virtual equipment or weapon, virtual game territory, status, etc.

In some implementations, the processing engine206integrates or otherwise uses extrinsic information312from extrinsic sources to improve the functioning of one or more processing paradigms. The extrinsic information312may be used to interpret data collected in association with a gaming session, to determine game states, to evaluate hardware and software performance, to identify a social network or social-network information, to determine contact information of a gaming content provider, to collect information concerning game tricks and trends associated with the gaming title, and so forth.

FIG.4is an example online interactive gaming environment400supported by a server system114that includes a plurality of servers in accordance with some implementations. The server system114includes at least one or more game servers118, and is configured to host one or more gaming titles. That said, the server system114is communicatively coupled to a plurality of client devices102over one or more communication networks112, and executes a plurality of online gaming sessions corresponding to a gaming title in parallel, thereby allowing each of the plurality of client devices102to participate in gameplay under the gaming title via a respective online gaming session. The game servers118receive session requests from the plurality of client devices102, and initiate, join or terminate the online gaming sessions accordingly. In response to each session request, one of the game servers118is assigned to execute a corresponding online gaming session. During the course of executing each online gaming session, the game server118assigned to execute the respective online gaming session renders a plurality of image frames (i.e., a session data stream) for a client device that has requested the online gaming session. In some situations, the plurality of image frames are rendered in response to receiving a user input (e.g., a command to jump, shoot or turn) from any of a subset of online gaming sessions that share a game instance or scene of the gaming title.

In some implementations, the server system includes a game library server422configured to provide executable gaming programs (e.g., game applications628inFIG.6A) that correspond to one or more gaming titles loaded to the game server118for executing the gaming titles. The server system114also includes a gaming content server450configured to provide gaming content to the client devices102and the game servers118to initiate an online gaming session and render the image frames of the online gaming session. Optionally, in some situations, the gaming content server450is managed by a third party distinct from the server system114, and provides third party gaming content to the server system114. Examples of third party gaming content include, but are not limited to sports games, racing games, role playing games (RPG) and first person shooter (FPS) games. Different gaming titles may have widely varying cloud hardware requirements and network (e.g., to ensure an optimal user gaming experience—consistent in some instances with different subscription performance tiers) based on different associated latency requirements and expectations, output video resolution, and gaming server computational workload and video encoding/streaming resources, and network bandwidth.

Optionally, the game server118assigned to execute an online gaming session also generates a review media stream, and provides it directly or indirectly, via the media streaming server120, to a spectator who wants to review the gameplay in real time. In some implementations, the media streaming server120is configured to host a media streaming website or application, and the review media stream is provided via the media streaming website or application to the spectator concurrently with gameplay.

In some implementations, the game servers118manage user accounts associated with the client devices102, e.g., subscriptions to membership of one or more online interactive games by the user accounts. Alternatively, in some implementations, the server system114includes a frontend server402configured to manage the user accounts associated with the client devices102. Optionally, the frontend server402is owned by a third party entity. After the client devices102log onto their respective user accounts and start/join their online gaming sessions116, the game servers118set up and manage the gaming sessions116by obtaining gaming contents from content servers450, sending the gaming contents to the game applications executed on the client devices102or designated media devices106, identifying user requests or actions, rendering gameplay outputs for the client devices102in response to the user requests or actions, and storing game state data during the respective gaming session116.

In some implementations, the game servers118dynamically allocate cloud gaming hardware resources (e.g., GPU406and encoder408) and monitor and utilize network bandwidth available to individual end users to provide optimal cloud gaming experience. In some implementations, the game servers118provide multiple performance tiers, including a tier that supports high performance, real time gaming sessions with high definition video/media streams and another tier that supports review media streams that has correspond with little or no latency to one or more actual gaming streams. Optionally, such review media streams are provided via a media streaming site, such as YouTube, to one or more users, and the media streaming site is optionally managed by the media streaming server120.

Each game server118includes one or more processing units (e.g., CPU404, GPU406and encoder408), main dynamic memory410, and one or more local caches412. The main dynamic memory410is configured to store programs and data for use by the CPU404and the GPU406during execution of the programs. Optionally, each game server118has its own main dynamic memory410. Optionally, a subset or all of the game servers118share a main dynamic memory410. The local caches412are configured to temporarily store data, e.g., media content that has completed a vertex post-processing and waits for a primitive assembly in the GPU406, and media content that is generated by the GPU406and will be provided to the encoder408for further encoding.

In some implementations, the server system114further includes a static memory pool414in addition to the main memory410and the caches412. The static memory pool414is shared among a subset or all of the game servers118(e.g.,118-1,118-2, . . .118-n), and configured to store one or more static data items used in the plurality of online gaming sessions of the same gaming title. The server system114further includes a memory controller416configured to manage the static memory pool414. In some implementations, each of the plurality of gaming sessions is associated with a specific static data item and requires a specific graphics operation executable by a respective GPU406using the specific static data item, and the specific static data item is stored in the static memory pool414. The respective GPU406is configured to access the static memory pool414via the memory controller416in order to extract the specific static data item.

It is noted that in some implementations, the static data items stored in the static memory pool414can be stored in the main dynamic memory410. The main dynamic memory410is accessed by the GPU406at a faster rate than the static memory pool414. However, the main dynamic memory410needs to be refreshed or given a new electronic charge every few milliseconds to compensate for charge leaks from the memory, and therefore is not efficient in power performance. If the static data items are moved to the static memory pool414, part of the main dynamic memory410in the game servers118can be freed up and used to store more programs and dynamic data, or a smaller main dynamic memory410can be used to reduce power consumption.

The server system114has a multilevel storage scheme including the local caches412, the main dynamic memory410and the static memory pool414. The local caches412includes at least one cache level (e.g., L1, L2). Compared with the main dynamic memory410and the static memory pool414, the local caches412have a smaller size, is physically closer to the CPUs and GPUs, and can be accessed at a faster rate. The local caches412stores copies of instructions and data retrieved from the main dynamic memory410or the static memory pool414for direct use by the CPUs404and GPUs406. The main dynamic memory410is also called primary memory. Every program executed by the CPUs404and GPUs406is loaded into the main dynamic memory410, and moved to the caches on a block or instruction basis prior to execution. The size of the main dynamic memory410is crucial because it determines how many programs can be executed at one time and how much data can be readily available to a program. The static memory pool414is additional to the caches412and the main dynamic memory410. In some implementations, the static memory pool414is also called a level 4 cache that can be accessed by both the CPUs404and GPUs406, but is shared among the plurality of game servers118. The static memory pool414has a larger latency than the local caches412and the main dynamic memory410, and therefore, is used to store the static data items that are less frequently used in the online gaming sessions. In the server system114, this multilevel storage scheme enables multiple online gaming sessions of a gaming title to be executed efficiently with respect to at least power consumption.

In some implementations, gaming content corresponding to a gaming title are rendered by the GPUs406for online gaming sessions according to a graphics pipeline process (e.g., an OpenGL rendering pipeline). The graphics pipeline process includes an ordered sequence of graphics operations, and each graphics operation is implemented based on one or more user-defined shader programs. A shader library server418is coupled to the game servers118, and configure to provide standard shader programs as needed by different gaming titles. If a shader program is selected for use by a gaming title, it is retrieved from the shader library server418, compiled by the game server118, and stored locally in the game servers118for use to render the plurality of image frames for each online gaming session. In some implementations, each image frame of an online gaming session is rendered by an ordered sequence of compiled shaders (e.g., including a vertex shader, a tessellation shader, a geometry shader and a fragment shader) each of which implements one or more specific image rendering operations during the course of creating the respective image frame.

In some implementations, while the GPUs execute the plurality of online gaming sessions, a shader pipeline optimizer optimizes a sequence of shader programs used to implement the ordered sequence of graphics operations. Stated another way, the CPUs404allocate an unused subset of the CPUs and GPUs in the server system114to identify an alternative sequence of shaders or another version of the ordered sequence of shaders with respect to one or more performance criteria. Optionally, the game servers118includes both the unused subset of the CPUs and GPUs allocated for shader pipeline optimization and the active CPUs and GPUs that execute the online gaming sessions. Optionally, a pipeline optimizer server420, distinct from the game servers118, includes the subset of the CPUs and GPUs allocated for shader pipeline optimization.

In some implementations of this application, the server system114further includes a profiling server430and an analytics server440that are optionally separated as two distinct servers or integrated in one server. The profiling server430and analytics server440are configured to collect data samples associated with a subset of game instances of a gaming title and analyze the collected data samples, respectively. Specifically, the profiling server430sends a game profiling instruction (e.g., instruction538inFIG.5) to the game servers118to collect a plurality of data samples for the subset of game instances corresponding to the gaming title, while the game servers118are executing a plurality of game instances. In response to the game profiling instruction, the game servers118identify the subset of the plurality of game instances, collect the plurality of data samples including at least game state data and system performance data associated with the one or more synchronous gaming sessions corresponding to each of the subset of game instances, and return the plurality of data samples to the profiling server440. The profiling server430then provides the plurality of data samples to the analytics server440, thereby allowing the analytics server440to analyze the plurality of data samples to improve gaming performance and/or user experience associated with the gaming title. As such, data collection and analysis is conducted at the profiling server430and analytics server440concurrently with execution of the online gaming sessions at the game servers118.

FIG.5is an example memory system500having a multilevel storage scheme that is configured to store data in a server system114in accordance with some implementations. In accordance with the multilevel storage scheme, the memory system500of the server system114includes a plurality of caches502that are disposed directly within each individual server (e.g., a game server118). More specifically, each of the plurality of caches502is directly associated with a processor of an individual server, e.g., a CPU404and a GPU406of a game server118, and configured to provide both instructions that are executed by the processor of the individual server and data that are used during execution of the instructions. The memory system500of the server system114further includes a main dynamic memory410that is configured to store programs executed by one or more processors of a server (e.g., game server118) and data used during execution of the programs. Each program includes a plurality of instructions, and during execution, the instructions are sequentially loaded to the caches502of a corresponding processor and wait to be executed by the respective processor. In some implementations, a subset of or all servers of the server system114share a main dynamic memory410. A data bus couples the subset of or all servers of the server system114to the main dynamic memory410to enable write and read accesses to the main dynamic memory410. Optionally, the data bus includes a high speed data bus via which the main dynamic memory410is accessed with a substantially small latency less than a threshold bus latency. In some implementations, each server of the server system114has a respective main dynamic memory410accessible only to one or more processors of the respective server. For example, a game server118includes a main dynamic memory410accessible to both the CPU404and the GPU406of the game server118. The main dynamic memory410of the game server118stores instructions504configured to execute one or more gaming titles, e.g., those of a server-side game module626inFIG.6A, and collect data samples during execution of the gaming titles.

The main dynamic memory410of the game server118also stores dynamic data506associated with online gaming sessions of each game instance, including game state data508(e.g., first gaming session data, second gaming session data) and system performance data522. Specifically, the game state data508for each gaming session optionally include game context data508A (e.g., a game level, a weapon type and a costume, objects in a corresponding game scene), game settings508B (e.g., a sliding window frame rate, frame resolution), and image frame data508C (e.g., texture pages, vertex page, constant pages and frame data) for a corresponding online gaming session of the respective gaming title. Further, in some implementations, the system performance data522is stored in at least one of CPU performance counters, GPU performance counters, a stack trace, and architectural counters. Also, each of the game state data508and the system performance data522is associated with a timestamp indicating an instant of time at which the corresponding data sample is collected.

In accordance with the multilevel storage scheme, the memory system500of the server system114further includes a static memory pool414that is centralized for and shared by the one or more game servers118in the server system114. The static memory pool414is configures to store one or more static data items used to render image frames of online gaming sessions of a gaming title. Examples of the one or more static data items include, but are not limited to, texture objects512, vertex data buffer objects514, and constant data objects516. When the one or more game servers118are assigned to execute a plurality of online gaming sessions, each game server118obtains one or more static data items from the static memory pool414, and one or more dynamic data items506from its corresponding main dynamic memory410.

Further, in some implementations, a game server118further includes a shader cache518coupled to its GPU406. The shader cache518is configured to store at least one sequence of compiled shaders such that instructions of the compile shaders can be directly loaded to the GPU406to render gaming content via a graphics pipeline process. In some implementations, the server system114includes a plurality of compiled shader sequences520each of which is configured according to a performance criterion, and each compiled shader sequence520optionally includes a plurality of versions corresponding to different user input conditions. Optionally, the plurality of compiled shader sequences520are stored in the shader cache518. Optionally, the plurality of compiled shader sequences520are stored in the static memory pool414, and moved to the shader cache518or directly to a cache of the GPU406if selected for use in gaming content rendering.

Further, in accordance with the multilevel storage scheme, the memory system500of the server system114includes a hard drive524that stores one or more executable gaming programs corresponding to one or more gaming titles. During the course of configuring the server system114to host a gaming title, the CPUs404of the server system sends a program request526for an executable gaming program528corresponding to the gaming title. In response to the program request526, the hard drive524returns the executable gaming program528corresponding to the gaming title to the CPUs404. In some implementations, the hard drive524does not store the requested executable gaming program528, and the CPUs404send the program request526to request the executable gaming program528from a remoter server (e.g., a game library server422, a content server450or another third party game provider). The CPUs404store the executable gaming program528, returned from the hard drive524or the remote server, into the main dynamic memory410as a server-side game module626.

Further, in some implementations, the hard drive524also stores static data items of one or more gaming titles. During the course of loading a gaming title, the CPUs404of the server system sends a data request530for the static data items of the gaming title. In response to the data request526, the hard drive524returns a subset or all of the static data items532corresponding to the gaming title to the CPUs404. In some implementations, the hard drive524does not store a specific static data item, and the CPUs404send a data request530to request the specific static data item from a remoter server (e.g., a content server450or a shader library server418). One or more static data items532A returned from the hard drive524or the remote server are stored in the static memory pool414. Alternatively, in some implementations, one or more static data items532B are stored in the main dynamic memory410for direct access by the CPUs404and GPUs406of the game servers118.

Additionally, the CPUs404also send a shader request to the shader library server to obtain one or more standard shader programs536used in association with the gaming title, and the returned shader programs536are stored locally in the shader cache518. While executing online gaming sessions corresponding to the gaming title, the CPUs404and GPUs406generate a plurality of dynamic data items506including at least game state data508and system performance data522corresponding to the online gaming sessions. The plurality of dynamic data items506are stored into the main dynamic memory410of the server system114. In some implementations, the game state data508and system performance data522are selectively collected and stored in the main dynamic memory410in accordance with a game profiling request538sent by a profiling server430. The game state data508and system performance data522are returned to the profiling server430periodically, according to a predefined schedule or in response to a query, and forwarded to the analytics server440for further analysis.

In some implementations, the server system114includes a dynamic memory controller416A and a static memory controller416B that are configured to manage write/read accesses to the main dynamic memory410and the static memory pool414, respectively. Alternatively, in some implementations, the server system114includes a single memory controller416configured to manage write and read accesses to the main dynamic memory410and the static memory pool414. The main dynamic memory410includes a directory540that identifies locations of the instructions204, dynamic data items506and static data items532B (if any) in the main dynamic memory410. In some implementations, the static memory pool414includes a hash table542that identifies locations of the static data items532A in the static memory pool414. Optionally, the static data items532are identified based on context information. In an example, the static memory pool414stores a plurality of compiled shader sequences520, and their addresses in the static memory pool414are identified using performance criteria, user inputs and/or usage statistics of a corresponding graphics pipeline process in the hash table. Optionally, such context based mapping in the hash table542is applied to other static data items as well. The hash table protects the static memory pool414from storing a static data item repeatedly in association with different situations, thereby causing a waste of its memory resources.

In some implementations, each CPU404includes a plurality of CPU cores. Each CPU core has a cache storing both instructions executed by the respective CPU core and data used during execution of the instructions. The cache is directly coupled with the respective CPU core and can provide the instructions and data immediately and directly to the respective CPU core. In some implementations, the CPU core further includes a CPU cache that is shared by the plurality of CPU cores of the CPU404. The CPU cache stores data that are used during execution of the instructions stored in the CPU cores. The CPU cache feeds its data to the caches352directly coupled with the CPU cores.

Similarly, in some implementations, each GPU406includes a plurality of GPU cores. Each GPU core has a cache storing both instructions executed by the respective GPU core and data used during execution of the instructions. The cache is directly coupled with the respective GPU core and can provide the instructions and data immediately and directly to the respective GPU core. In some implementations, the GPU core further includes a GPU cache that is shared by the plurality of GPU cores of the GPU406. The GPU cache stores data that are used during execution of the instructions stored in the GPU cores. The GPU cache feeds its data to the caches directly coupled with the GPU cores. To render gaming content of a gaming title, writes and reads in the memory system300are managed by page (i.e., by an array of image pixels corresponding to an image frame of gaming content). Page-sized reads are cached in the local GPU memory hierarchy (L1-L3 caches).

Server System, Client Device and Media Device in Online Gaming Environment

FIG.6Ais a block diagram illustrating a server system114in accordance with some implementations, andFIG.6Bis a block diagram illustrating a server-side game module626that is executed at a game server118of the server system114in accordance with some implementations.FIG.6Cis a block diagram illustrating data stored in the server system114in accordance with some implementations. The server system114, typically, includes one or more processing units (e.g., CPU(s)602, GPU(s)406and encoder408), one or more network interfaces604, memory606, and one or more communication buses608for interconnecting these components (sometimes called a chipset). The server system114may optionally include one or more input devices610that facilitate user input, such as a keyboard, a mouse, a voice-command input unit or microphone, a touch screen display, a touch-sensitive input pad, a gesture capturing camera, or other input buttons or controls. Furthermore, the server system114may use a microphone and voice recognition or a camera and gesture recognition to supplement or replace the keyboard. In some implementations, the server system114optionally includes one or more cameras, scanners, or photo sensor units for capturing images, for example, of graphic series codes printed on electronic devices. The server system114may also include one or more output devices612that enable presentation of user interfaces and display content, including one or more speakers and/or one or more visual displays.

Memory606includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid state memory devices; and, optionally, includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid state storage devices. Memory606, optionally, includes one or more storage devices remotely located from one or more processing units. Memory606, or alternatively the non-volatile memory within memory606, includes a non-transitory computer readable storage medium. In some implementations, memory606includes a subset or all of a main dynamic memory410, a static memory pool414and a shader cache518. In some implementations, memory606, or the non-transitory computer readable storage medium of memory606, stores the following programs, modules, and data structures, or a subset or superset thereof:Operating system616including procedures for handling various basic system services and for performing hardware dependent tasks;Network communication module618for connecting the server system114(e.g., a game server118) to other devices (e.g., other servers in the server system114, the client device102, and the media device106) via one or more network interfaces604(wired or wireless) and one or more communication networks112, such as the Internet, other wide area networks, local area networks, metropolitan area networks, and so on;User interface module620for enabling presentation of information (e.g., a graphical user interface for presenting application(s), widgets, websites and web pages thereof, and/or games, audio and/or video content, text, etc.) at the client device102and/or104;Media device module622(optional) that is executed to provide server-side functionalities for device provisioning, device control, and user account management associated with media device(s)106;Location determination module624for determining the location or proximity of the client devices102based on location or proximity information of any of the client devices102and the media device106;Server-side game module626for providing server-side functionalities associated with one or more gaming titles at one or more game servers118, including but not limited to setting up and authorizing gaming sessions via one or more game applications628, assigning a GPU to execute a subset of sessions, storing session state data and other game-related data, processing gameplay inputs from the client devices102, rendering gameplay outputs (e.g., sequences of image frames) in response to the gameplay inputs, and optimizing game rendering, where the server-side game module626includes at least a local profiling orchestrator module630for receiving a game profiling instruction from a profiling server430and a collection module632for collecting a plurality of data samples corresponding to the gaming sessions in response to the game profiling instruction while gaming sessions corresponding to a game instance are being executed;Media streaming server module634for hosting a media streaming site at a media streaming server120, receiving concurrent supplemental media streams associated with an online gaming session, and providing the concurrent media streams to a client device102or a media device106for concurrent display with the online gaming session that is being executed on the game applications732of a client device102(optionally, the same client device102or a distinct client device102);Game frontend module636executable at the gaming frontend server402for managing user accounts associated with the client devices102that have subscribed to gaming service of one or more gaming titles, where in some implementations, the game frontend module636further includes a game application platform658for providing a user interface that consolidates accesses to multiple game applications628that execute different gaming titles based on different gaming content (including third party gaming content);Gaming content module638executable at content servers450for providing gaming content to the client devices102and the game servers118to initiate online gaming sessions associated with one or more gaming titles and render image frames of the online gaming sessions;Game library module640executable at a game library server422for providing an executable gaming program (e.g., one or more modules in the server-side game module626) and static data items required for execution of the executable gaming program;Shader library module642executable at a shader library server418for providing standard shader programs as needed by different gaming titles during the course of executing online gaming sessions associated with each gaming title;Pipeline optimizer module644executable at a pipeline optimizer server420for providing server-side optimization of compiled shaders used in a graphics pipeline process for rendering image frames of online gaming sessions associated with a gaming title, optionally concurrently while the server-side game module626is rendering gameplay outputs in response to gameplay inputs for the corresponding online gaming sessions;Game profiling module646executable at a profiling server430for executing a game profiling application and providing a user interface to receive user inputs defining a game profiling request, sending out the game profiling request to instruct the game servers118to collect a plurality of data samples for a subset of game instances executed by the game servers118in association with a gaming title, and collecting the plurality of data samples from the game servers118, where the profiling server module646includes a global profiling orchestrator module648configured to perform game profiling functions in conjunction with the local profiling orchestrator module630executed at the game servers118;Game analytics module650executable at an analytics server440for analyzing the collected data samples and enabling display of analysis results on a user interface;Device/network assessment module652for assessing device and network capabilities of client device102, including but not limited to assessing network bandwidth of the connection to the client device102and assessing whether the client device102has the needed module or application to play a game;Data transmission module654for providing data (e.g., game controller configurations688, software updates, etc.) to client devices102; andServer system data656.

Referring toFIG.6B, in addition to the game applications628, the local profiling orchestrator module630and the collection module632, the server-side game module626includes the following programs, modules, or a subset or superset thereof:GPU time sharing module660for allocating each processing slice of the GPUs406to a respective subset of online gaming sessions and dynamically sharing a duty cycle of a processing slice among two or more gaming sessions (i.e., executing the two or more gaming sessions simultaneously and in parallel during the duty cycle) according to real-time data processing need of the first and second gaming sessions;Multi-stream generation module662for encoding both a low latency stream and a normal latency stream within time intervals corresponding to a predefined frame rate;Parallel encoding module664for allocating encoder cores of the encoder408to process image tiles of an image and enabling the encoder cores to encode each individual block in a respective image tile as soon as data of the individual block and its corresponding boundary information are available;Dynamic frame generation module666for dynamically generating a variable number of frames in response to a user command based on a predefined frame rate and corresponding communication and processing latencies associated with the user command during an online gaming session; andResource assignment module668for determining target quality parameters of an online gaming session based on corresponding device and connection capabilities, associating resource of the server systems114(e.g., virtual machines therein) with the online gaming session based on the target quality parameters, and managing the online gaming session according to a resource profile of the associated resource.Intent determination module670for comparing user input transit times (e.g., between the client device102and the server system114) with display transit times (e.g., between the media device106and the server system114), and determining the user's intent behind particular inputs by matching input events with respective trigger frames;Latency adjustment module672for determining a number of intermediate frames for the GPU406to insert between (i) a current frame being processed at the time a user input is received and (ii) a response frame showing a result of the received input;Resource tuning module674for determining latency tolerances for particular online gaming sessions;Simulation module676for simulating gameplay inputs (e.g., in accordance with a particular player's in-game behavior) during network disruptions between a player's gaming controller102and the server system114;Disruption detection module678for detection disruptions in or of a network connection between the server system114and the game controller102and/or the media device106, the network connection supporting a gaming session between or involving one or more of the aforementioned components; andState reconciliation module680for processing recovered inputs upon resumption of a network connection between the server system114and the game controller102, and reconciling simulation-affected game states with user-intended game states.

Referring toFIG.6C, in some implementations, the server system data656includes the following data items, or a subset or superset thereof:Client device settings682for storing information associated with the client devices102, including common device settings (e.g., service tier, device model, storage capacity, processing capabilities, communication capabilities, etc.);Media device settings684for storing information associated with user accounts of the media device module622, including one or more of account access information and information for device settings (e.g., service tier, device model, storage capacity, processing capabilities, communication capabilities, etc.);Location/proximity data686including information associated with the presence, proximity or location of any of the client device102and the media device106;Game controller configurations688for storing controller configurations for various gaming titles; andUser information690for storing information associated with user accounts of each of one or more gaming titles (e.g., game application628,FIG.6) that are hosted on the server system114, including for example user account information (e.g., identification and passwords), membership type, preference, and activity history; andGame state data508for storing data associated with gaming sessions, including for example data for a first gaming session and data for a second gaming session, where the game state data for each gaming session includes, but is not limited to, game context data508A (e.g., a game level, a weapon type and a costume, objects in a corresponding game scene, identifications of related sessions), game settings508B (e.g., a sliding window frame rate, a frame resolution, a rendering specification, latency requirement, information of GPU allocation, information of encoder allocation), and image frame data508C (e.g., texture pages, vertex page, constant pages and frame data) for a corresponding online gaming session;Static data items692for storing data shared by a plurality of online gaming sessions, where the static data items692include, but are not limited to, one or more of texture objects, a vertex data buffer object, a constant data object, and sequences of compiled shaders that are optimized based on usage statistics698to satisfy different performance criteria;Sequences of compiled shaders694for storing shaders that have been compiled and ready for execution in a graphics pipeline process, where the sequences of compiled shaders include, but are not limited to a default sequence of compiled shader that is used to render gaming content by default and multiple sequences of compiled shaders that are optimized based on usage statistics698to satisfy different performance criteria696;Performance criteria696and usage statistics698that are stored in association with the sequences of compiled shaders that are optimized, wherein each of the performance criteria696corresponds to one or more of total power consumption, a code size, an average latency, total cache usage, and total memory usage of a sequence of compiled shaders during the course of executing a plurality of online gaming sessions, and the usage statistics698characterize execution of each of a plurality of online gaming sessions;Game instance traces699stored for each of a plurality of game instances corresponding to one or more gaming titles executed by the game servers118, where the game instance traces699includes a sequence of ordered gaming events for which data samples are recorded; andSystem performance data522for storing data associated with performance of a processor, a memory, an operating system, one or more input/output ports and a data bus of the game servers118, which are assigned to execute one or more gaming sessions of a game instance, where the system performance data522is stored in at least one of CPU performance counters522A, GPU performance counters522B, a stack trace, and architectural counters in association with timestamps522C each of which indicates an instant of time when the corresponding data sample is collected.

In some implementations, the server system114further includes one or more memory controllers416that are coupled to the CPUs, GPUs, the main dynamic memory410and the static memory pool414using one or more communication buses608. The one or more memory controller416optionally include a first memory controller416A configured to manage the main dynamic memory410and a second memory controller416B configured to manage the static memory pool414. As shown above, a subset of elements616-699(e.g., game state data508and system performance data522) are stored in the main dynamic memory410, and are accessible via the first memory controller416A. Static data items692are stored in the static memory pool414, and accessible via the second memory controller416B.

FIG.7is a block diagram illustrating an example client device102or104of the gaming environment100in accordance with some implementations. Examples of the client device102or104include, but are not limited to, a mobile phone, a tablet computer, a laptop computer, a desktop computer, a wearable personal device and a dedicated game controller device. The client device102or104includes one or more processing units (CPUs)702, one or more network interfaces704, memory706, and one or more communication buses708for interconnecting these components (sometimes called a chipset). The client device102or104includes one or more input devices710that facilitate user input, such as a keyboard, a mouse, a voice-command input unit or microphone, a touch screen display, a touch-sensitive input pad, a gesture capturing camera, or other input buttons or controls. Furthermore, the client device102or104uses a microphone and voice recognition or a camera and gesture recognition to supplement or replace the keyboard. In some implementations, the client device102or104includes one or more cameras, scanners, or photo sensor units for capturing images, for example, of graphic series codes printed on electronic devices. The client device102or104also includes one or more output devices712that enable presentation of user interfaces and display content, including one or more speakers and/or one or more visual displays. Optionally, the client device102or104includes a location detection device, such as a GPS (global positioning satellite) or other geo-location receiver, for determining the location of the client device102or104. The client device102or104may also include a proximity detection unit714, e.g., an IR sensor, for determining a proximity of a media device106and/or of another client device. The client device102may also include one or more sensors760(e.g., accelerometer, gyroscope, etc.) for sensing motion, orientation, and other parameters of the client device102or104, which may be used as input.

Memory706includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid state memory devices; and, optionally, includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid state storage devices. Memory706, optionally, includes one or more storage devices remotely located from one or more processing units702. Memory706, or alternatively the non-volatile memory within memory706, includes a non-transitory computer readable storage medium. In some implementations, memory706, or the non-transitory computer readable storage medium of memory706, stores the following programs, modules, and data structures, or a subset or superset thereof:Operating system716including procedures for handling various basic system services and for performing hardware dependent tasks;Network communication module718for connecting the client device102or104to other devices (e.g., the server system114, the media device106, and other client device102or104) via one or more network interfaces704(wired or wireless) and the local network110and one or more networks112, such as the Internet, other wide area networks, local area networks, metropolitan area networks, and so on;User interface module720for enabling presentation of information (e.g., a graphical user interface for presenting applications, widgets, websites and web pages thereof, and/or games, audio and/or video content, text, etc.) at the client device102via one or more output devices712(e.g., displays, speakers, etc.);Input processing module722for detecting one or more user inputs or interactions from one of the one or more input devices710and interpreting the detected input or interaction;Web browser module724for navigating, requesting (e.g., via HTTP), and displaying websites and web pages thereof, including a web interface for joining the session116;Media device application726for interacting with a media device106, including logging into a user account associated with the media device106, controlling the media device106if associated with the user account, and editing and reviewing settings and data associated with the media device106;Client-side game module728for providing client-side functionalities associated with one or more gaming titles in a client device102, including but not limited to setting up gaming sessions, locally storing session state data and other game-related data, receiving gameplay inputs from the input device710, and providing gameplay outputs (e.g., sequences of image frames) in response to the gameplay inputs, where in some implementations the client-side game module728further includes a game controller application730that configures the client device to a game controller device, one or more game applications732corresponding to one or more gaming titles, and a third party game interface734that enables gameplay based on content provided by gaming content providers independent from a third party game platform;Game profiling and analysis application736implemented at a client device104of an administrator or a game developer for receiving a game profiling or analysis request, sending the game profiling request to the profiling server430, receiving analysis results from the analytics server440, and presenting the results on the client device104while one or more gaming titles are being executed on client devices102.Data download module738for downloading data (e.g., game controller configurations748, client-side game module728and other applications, updates to modules and applications and data in memory706) from server system114and other content hosts and providers; andclient data740storing at least data associated with the game application732and other applications/modules, including:Client device settings742for storing information associated with the client device102itself, including common device settings (e.g., service tier, device model, storage capacity, processing capabilities, communication capabilities, etc.);Media device settings744for storing information associated with user accounts of the media device application726, including one or more of account access information, and information for device settings (e.g., service tier, device model, storage capacity, processing capabilities, communication capabilities, etc.);Game application(s) settings746for storing information associated with user accounts of the game application(s), including one or more of account access information, in-game user preferences, gameplay history data, and information on other players;Game controller configuration(s)748for storing information associated with configurations (e.g., received configurations from game controller configurations688,FIG.6) of client-side game module728for game application(s)732;Location data750including information associated with the presence, proximity or location of any of the client devices102and the media devices106; andLocal gaming session data752including information associated with online gaming sessions that are executed on the client device102in association with different gaming titles, e.g., latest status information, texture pages, vertex pages, constant pages, and frame data of one or more image frames associated with each of the online gaming sessions.

In some implementations not shown inFIG.7, the client-side game module728includes a commands module for transmitting messages (e.g., setup commands) between the client device102and the media device106in a menu mode, a streaming module for transmitting gameplay commands between the client device102and the server system114(e.g., to the media streaming server120) in a gameplay mode, and a controller module for providing a gameplay input interface to the gaming application. In some implementations, the client-side game module728is a part (e.g., a sub-module) of the media device application726or another application in memory706. In some implementations, the client-side game module728is a part of the operating system716. In some implementations, the client-side game module728is a distinct module or application.

In some implementations of the client device102, the media device application726(and correspondingly media device settings744) and game application732(and correspondingly game application settings746) are optional. Depending on the particular game to which the client device102is invited to join, the media device application726and the game application732are not required to play. If any of these applications are needed for playing the game (e.g., the game uses a client-side game module within the media device application726), and if the application is not in memory706, the client device may be prompted to download the application.

In some implementations, the client device102is a dedicated game controller device. The game controller device includes one or more input devices710that facilitate user input, such as microphones (or a voice-command input unit), buttons, directional pad(s), and joystick(s). In some implementations, the game controller102further includes gesture recognition features. The game controller device102also includes one or more output devices that facilitate audio output and/or visual output, including speaker(s), LED(s), and/or optionally, a display.

FIG.8is a block diagram illustrating an example media device106of the gaming environment100in accordance with some implementations. The media device106, typically, includes one or more processing units (CPUs)802, one or more network interfaces804, memory806, and one or more communication buses808for interconnecting these components (sometimes called a chipset). Optionally, the media device106includes a proximity/location detection unit810, such as an IR sensor, for determining the proximity of a client device102.

Further, in some implementations, the media device106includes radios that enable one or more communication networks and allow the media device106to communicate with other devices (e.g., with the client device102). In some implementations, the radios are capable of data communications using any of a variety of custom or standard wireless protocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.5A, WirelessHART, MiWi, Whispernet, Bluetooth low energy etc.), custom or standard wired protocols (e.g., Ethernet, HomePlug etc.), and/or other suitable communication protocol, including communication protocols not yet developed as of the filing date of this document.

Memory806includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid state memory devices; and, optionally, includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid state storage devices. Memory806, optionally, includes one or more storage devices remotely located from one or more processing units802. Memory806, or alternatively the non-volatile memory within memory806, includes a non-transitory computer readable storage medium. In some implementations, memory806, or the non-transitory computer readable storage medium of memory806, stores the following programs, modules, and data structures, or a subset or superset thereof:Operating system816including procedures for handling various basic system services and for performing hardware dependent tasks;Network communication module818for connecting the media device106to other computers or systems (e.g., the server system114, and the client device102) via one or more network interfaces804(wired or wireless) and one or more networks112, such as the Internet, other wide area networks, local area networks, metropolitan area networks, cable television systems, satellite television systems, IPTV systems, and so on;Content decoding module820for decoding content signals received from one or more content sources (e.g., the server system114that provides game output from the gaming session116) and outputting content in the decoded signals to an output device108coupled to the media device106, where in some implementations, the outputted content includes two or more media streams130and132intended for use in gameplay and review, respectively;Proximity determination module822for determining the proximity of the client device102and based on proximity related information that is detected by the proximity detection unit810or provided by the server system114;Media display module824for controlling media display; andGaming applications826for controlling the lifecycle of all the gaming components on the media device106, receiving messages (e.g., commands) from a game controller102, and for relaying game controller inputs to the server system114;Gaming user interface module828for accessing user content (e.g., profile, avatar, purchased games, game catalog, friends, messaging) and optimized images for display, and for receiving inputs from a client device102during gameplay;Interface control module830for interfacing communications between the gaming application826and the gaming user interface module828, and for initiating a session pairing request with the server system114during a game launch;Server interface module832for communications with the server system114and an authentication system (if any), including:Authentication sub-module834for communicating with the authentication system to authenticate a linked user of the game controller;Session sub-module836for receiving gaming session information (e.g., session id, game server provisioning IP address, port etc.) from the server system114, and for invoking a callback on the interface control module830providing this information; andStreaming sub-module838for receiving and displaying, in real time during gameplay, gaming streams transmitted from the media streaming server120of the server system114; andMedia device data840storing at least data including:Media device settings842for storing information associated with user accounts of a media device application, including one or more of account access information and information for device settings (e.g., service tier, device model, storage capacity, processing capabilities, communication capabilities, etc.);Location data844including information associated with the presence, proximity or location of any of the client device102and/or104, and the media device106; andLocal gaming session data846including information associated with online gaming sessions that are executed on the client device102and/or104in association with different gaming titles, e.g., latest status information, texture pages, vertex pages, constant pages, and frame data of one or more image frames associated with each of the online gaming sessions.

Distributed Real-Time Game Profiling and Analysis

FIG.9Ais an example flow chart showing a process900of executing an online gaming session on a server system114in accordance with some implementations.FIG.9Bis an example flow chart showing a real-time game profiling and analysis process950implemented collaboratively by one or more game servers118, a profiling server430and/or an analytics server440of a server system114in accordance with some implementations. The game servers118include one or more CPUs404and a plurality of GPUs406, and employ a multilevel storage scheme (e.g.,FIG.5), and are configured to host one or more gaming titles for users of client devices102. The CPUs receives (422) a plurality of session requests to initiate a plurality of online gaming sessions under a gaming title selected from the plurality of client devices102. Each client device102executes a real time user-interactive gaming application corresponding to the gaming title, and a respective session request is sent (902) from the gaming application of the respective client device102. The respective session request includes user information of the requesting client device102. After receiving the user information of the respective client device102, the CPUs404authorize (904) a corresponding online gaming session based on the user information, and allocate (906) one of the GPUs406to execute the corresponding online gaming session. In some implementations, two or more of the plurality of online gaming sessions are allocated to the one of the GPUs406. Stated another way, the two or more online gaming sessions have to share computational and storage resources of the one of the GPUs based on time-division multiplexing (TDM), spatial multiplexing or a combination thereof. More details of GPU allocation in gaming content rendering are explained in U.S. Provisional Patent Application No. 62/646,824, titled “Methods and Systems for Rendering and Encoding Content for Online Interactive Gaming Sessions,” filed on Mar. 22, 2018, which is incorporated by reference herein in its entirety.

In response to each session request, the CPUs404copy (908) instructions of an executable gaming program of the gaming title from the main dynamic memory410to the local caches of the CPUs404, and executes the instructions one by one. Likewise, if instructed by the CPUs404, the respective GPU assigned to execute the respective online gaming session loads (908) related game rendering instructions in the executable gaming program to its local caches, and executes the game rendering instructions one by one.

In some implementations, each of the plurality of online gaming sessions is constantly refreshed, e.g., at a rate of 60 frames per second (fps). In some implementations, each of the plurality of online gaming sessions is refreshed in response to a user action inputted by the respective session itself or by a related session. For example, a subset of online gaming sessions are related and synchronized when they are in the same gaming scene of the gaming title, and gaming content delivered to each of the subset of online gaming sessions is updated at least when any one of the subset of online gaming sessions inputs a user action.

Specifically, for a specific online gaming session, the CPUs404receives (910) a user input entered via a gaming application of a first client device that is engaged in the online gaming session or another distinct gaming session sharing the same gaming scene with the online gaming session. In response, the CPUs404execute (912) the instructions of the executable gaming program corresponding to the gaming title to instruct (918) a GPU406assigned to execute the online gaming session to render a sequence of image frames according to the user input. During execution of this online gaming session, dynamic data items associated with this online gaming session are stored into or extracted from (914) the main dynamic memory410. The assigned GPU406then executes (920) game rendering instructions received from the CPU404to render the image frames of the online gaming session based on the user input. The image frames rendered by the GPU406are encoded by the encoder408, and streamed (926) to the client device102executing the online gaming session. In some implementations, the image frames rendered by the GPU406(e.g., of a normal latency stream intended for spectator review) are streamed (928) to the client device102via a media streaming server120.

While the game servers118are executing the online gaming sessions sharing the same gaming scene of the gaming title in parallel, the game servers118are also instructed by the profiling server430to collect data samples related to the online gaming sessions. Some or all of the dynamic data items stored into the main dynamic memory410in association with the online gaming sessions have been stored in response to a game profiling request issued by the profiling server430. That said, during execution of an online gaming session, game state data associated with the online gaming sessions are stored into or extracted from (930A) the main dynamic memory410, and system performance data associated with the online gaming sessions are stored into or extracted from (930B) the main dynamic memory410. More specifically, the game servers execute a plurality of game instances for the gaming title in parallel, and each of the plurality of game instances includes one or more synchronous gaming sessions. While the game servers118are executing the plurality of game instances, the profiling server430receives a game profiling request (932) for collecting a plurality of data samples for each of a subset of the game instances. In some implementations, a client device104of an administrator or game developer executes (934) a game profiling application736, and the profiling server430enables display of a user interface on the game profiling application736to receive a user input of the game profiling request that defines the subset of game instances corresponding to the gaming title. The profiling server430includes a global profiling orchestrator module648that generates a game profiling instruction based on the received game profiling request and sends (936) the game profiling instruction to the game servers118to collect the plurality data samples in real time while the game servers118executes the plurality of game instances.

The game servers118include a local profiling orchestrator module630and a collection module632. After receiving the game profiling instruction, the local profiling orchestrator module630instructs (938) the collection module632to collect the requested data samples while a game application628executes the corresponding game instances. The local profiling orchestrator630then sends (940) the collected data samples to the profiling server430periodically, according to a schedule or in response to a query. Locally, the collected data samples are stored as part of the dynamic data items that are dynamically stored in the main dynamic memory410in association with the online gaming sessions prior to being sent to the profiling server430. The collected data samples include at least game state data508and system performance data522associated with the online gaming sessions corresponding to each of the subset of game instances that are identified in the game profiling request. In some implementations, the profiling server430receives the collected data samples from the game servers118and creates a game profile based on these data samples collected for each game instance in the identified subset of game instances.

In some implementations, after receiving the data samples collected by the game servers118, the profiling server430performs analysis functions on the collected data samples by itself. Alternatively, in some implementations, the profiling server430sends (944) the collected data samples to the analytics server440that is distinct from the profiling server430, and the analytics server440is configured to perform (946) analysis functions on the collected data samples and provide (948) analysis results to a client device104of an administrator or game developer. Examples of the analysis functions include, but are not limited to, hotspot analysis, processor profiling, game data visualization, and alert generation. The client device104of the administrator or game developer executes a game analysis application on the profiling server430or analytics server440, which enables a user interface of the game analysis application on the client device for visualizing the collected data samples, receiving (952) user inputs of game analysis requests, and/or presenting analysis results in response to the game analysis requests. Optionally, the game profiling application and the game analysis application are integrated as a single application (e.g., game profiling and analysis application736inFIG.7) or executed independently from each other as two applications. The game profiling application and the game analysis application, if independent from each other, are optionally running on the same client device104or two distinct client devices104. That said, the game profiling instruction and the game analysis request are optionally sent from the same application running on the same client device104or from two distinct applications running on the same client device104or two distinct client devices104.

In some implementations, the plurality of data samples collected by the collection module632of the game servers118are stored locally until receiving a query (e.g., in a game analysis request) from the analytics server440. In response to the query, a subset or all of the plurality of data samples are sent (944′) to the analytics server440for further analysis.

FIGS.10A-10Care example graphics user interfaces (GUIs)1000enabled by a game profiling and analysis application736that is executed on a client device104in accordance with some implementations. The game profiling and analysis application736is a combination of a game profiling application and a game analysis application. Alternatively, in some implementations, the game profiling application and the game analysis application are executed as two independent applications. The game profiling and analysis application736performs both profiling and analysis functions that are available in the game profiling application and the game analysis application. Optionally, the GUIs1000of the game profiling and analysis application736have a plurality of tabs1004-1014including a first subset of tabs (e.g., a profiling request tab1004) and a second subset of tabs (e.g., analysis tabs1006-1014) that correspond to game profiling functions and game analysis functions, respectively. The game profiling and analysis application736is configured to receive user inputs defining a game profiling request on the first subset of tabs, generate the game profiling request for data samples of a subset of game instances, and send the game profiling request to the profiling server430. The game profiling and analysis application736is also configured to receive user inputs defining a game analysis request on the second subset of tabs, generate the game analysis request, and present analysis results to a user of the game analysis application (e.g., a game administrator or developer) in response to the game analysis request. It is noted that in some implementations, the GUIs1000of the game profiling and analysis application736has a single page or multiple pages used for both game data collection and analysis and that each page are activated using a mechanism distinct from tab selection.

The profiling server430and the analytics server440are configured to perform the game profiling and analysis functions associated with the game profiling and analysis application736. Thus, the GUIs1000of the game profiling and analysis application736are enabled by a server-side game profiling module646and a server-side gaming analysis module650that are provided by the profiling server430and the analytics server440. It is noted that in some implementations, the server-side game profiling module646and the server-side gaming analysis module650are not implemented on the game servers118for the purposes of conserving computational, storage and communication capabilities for executing one or more gaming titles and rendering image frames for gaming sessions related to the gaming titles.

Referring toFIG.10A, the GUIs1000of the game profiling and analysis application736are configured to receive user inputs that define the game profiling request on the profiling request tab1004. Optionally, the profiling request tab1004of the GUIs1000includes a first field1016for selecting a gaming title from a plurality of gaming titles executed by the game servers118, a second field1018for identifying a subset of game instances of the selected gaming title, and a third field1020for defining a sampling scheme for collecting data samples of the identified subset of game instances. After the gaming title is specified in the first field1016, the profiling server430identifies a plurality of game instances running with the gaming title, and allows the user to define the subset of game instances for data sample collection on the GUI1000. In some implementations, the subset of game instances is defined by a user-defined percentage1018a(e.g., 20%), a user-defined number1018b(e.g., 5), or a user-defined instance pitch1018c(e.g., one out of every five) of the plurality of game instances running with the gaming title. In an example, the user-defined percentage or number of game instances are randomly selected from the plurality of game instances. In some implementations, the subset of game instances is defined in the field1018dto include all of the plurality of game instances running with the gaming title. Alternatively, in some implementations, the subset of game instances is defined according to a characteristic1018ewhich is optionally selected from a list of pre-predetermined characteristics (e.g., having more than 100 players, having 50% players reaching an advanced level, running a test game scene or feature, and being among the top 5 most power-consuming game instances).

In some implementations, the sampling scheme for collecting data samples of the identified subset of game instances is defined as a default scheme1020a, e.g., periodic sampling at a sampling frequency of one sample per minute. It is noted that one data sample includes at least a set of game state data and system performance data captured at a specific instant of time. In some implementations, the sampling scheme is periodic sampling with a user-defined sampling rate, and the GUIs1000includes a field1020bconfigured to receive a user input of the sampling rate. In some implementations, the sampling scheme is timed sampling implemented during a user-defined time window and/or according to a user-defined schedule, and the GUIs1000includes fields1020cand1020dconfigured to receive user inputs of the time window and the schedule. Alternatively, in some situations, the user is interested in a targeted profiling condition, because performance of the corresponding gaming title is bottlenecked or desirable at the targeted profiling condition. The GUIs1000includes a list of targeted profiling conditions1020efrom which the user is prompted to select one or more targeted profiling conditions, such that data samples can be collected when the one or more targeted profiling conditions occur to the selected subset of game instances corresponding to the gaming title. The GUIs1000optionally includes a field (not shown) to define a custom condition into the list of targeted profiling conditions. In a specific example related to a Tetris Game, the list of targeted profiling conditions includes at least occurrence of a L block, occurrence of a T block, a frame drop, and >95% CPU utilization.

After the gaming title, the subset of instances and the sampling scheme are defined according to the user inputs in the fields1016-1020, the game profiling and analysis application736sends to the profiling server430the game profiling request generated based on the gaming title, the subset of instances and the sampling scheme. Specifically, the game profiling and analysis application includes a user affordance1022, and the game profiling request is generated and sent to the profiling server430in response to receiving a user action on the user affordance1022. In this example, the user defines a game profiling request to collect data samples for all game instances experiencing a target profiling condition related to L blocks (i.e., when L blocks drop) in a gaming title of Tetris. When the user of the game profiling and analysis application736clicks on the user affordance1022, this specific game profiling request is generated based on the user inputs in the fields1016-1020, and sent to the profiling server430.

The game profiling request includes information concerning one or more types of the plurality of data samples to be collected by the profiling server430from the game servers118. In some implementations, the information concerning the types of the plurality of data samples is pre-defined, and for example, includes all data in CPU performance counters, GPU performance counters, a stack trace, and architectural counters. In some implementations not shown here, the GUIs1000includes a field configured to receive user inputs that define the information concerning the types of the plurality of data samples. The user inputs may select a subset or all data in the CPU performance counters, the GPU performance counters, the stack trace, and the architectural counters.

Referring toFIG.10B, the GUIs1000of the game profiling and analysis application736are configured to receive user inputs for defining a game analysis request on the analysis tab1006, and present analysis results to the user of the corresponding game analysis application1002in response to the game analysis request on the analysis tab1006. In some implementations not shown, the GUIs1000includes two separate tabs for defining the game analysis request and presenting the analysis results, respectively. At the time of loading the analysis tab1006, a plurality of data samples has been collected from the game servers118by the profiling server430. In some situations, information of the plurality of data samples are displayed on the analysis tab1006to allow the user to select a subset of the data samples for visualization or analysis. The analytics server440enables the GUIs1000to visualize the plurality of data samples collected by the profiling server430directly. In some situations, after the information of the plurality of data samples is analyzed, analysis results are visualized on the analysis tab1006, and the user is prompted to select one or more results from a plurality of analysis results for visualization and further analysis. Conversely, in the background, the analytics server440receives the game analysis request from the client device, analyzes the plurality of data samples remotely in the background, and sends the analysis results to the client device for display on the GUIs1000. Specifically, the analytics server440is configured to conduct hotspot analysis of processors of the game servers118, profile performance of the processors, and profile the plurality of game instances and the one or more synchronous gaming sessions corresponding to each game instance.

The profiling server430stores data samples collected for different gaming titles, instances and sessions in a game profiling database. The game analysis request identifies one or more of a gaming title, a game instance, a gaming session, and a subset of the plurality of data samples, and the subset of the plurality of data samples associated with the gaming title, instance and/or session are extracted from the game profiling database for visualization or analysis. The GUIs1000includes a title field1042, an instance field1044and a session field1046to receive user selections or inputs of the gaming title, instance and session, respectively. In some situations, the instance field1044and the session field1046are left blank, when general analysis results is presented and compared for the subset of instances for which data samples are collected. In some situations, the instance field1044is filled and the session field1046is left blank, when general analysis results is presented and compared for online gaming sessions corresponding to an instance in the instance field. Further, the GUIs1000includes a list of data samples1048, e.g., CPU idle rate, GPU resource assignment, PCI switch rate, OS response time, memory access rate, frame drop rate and average game latency. A subset of the data samples1048can be selected for visualization on the GUIs1000. The game profiling and analysis application includes a user affordance1050, and the game analysis request is generated and sent to the analytics server430in response to receiving a user action on the user affordance1050. In this example, the user defines a game analysis request to visualize the GPU resource assignment and the average game latency corresponding to the subset of instances for which data samples are collected in a gaming title of Tetris, a temporal variation of the L block rate, and a temporal variation of the frame drop rate. When the user of the game profiling and analysis application736clicks on the user affordance1050, this specific game analysis request is generated based on the user inputs in the fields1042-1048, and sent to the analytics server430.

Alternatively, in some implementations, the plurality of data samples collected by the collection module632of the game servers118are stored locally until the game servers118receive a query (e.g., in a game analysis request) from the analytics server440directly or indirectly. That said, only after the user action on the user affordance1050, a subset or all of the plurality of data samples are obtained from the game servers according to the selection in the fields1042-1048, and sent to the analytics server440for further analysis.

The plurality of data samples or the analysis results are presented on the GUIs in one of a temporal diagram, an instance diagram, or a session diagram. In accordance with the temporal diagram, the plurality of data samples or analysis results are plotted in a time domain. For example, a frame drop rate is plotted in a time domain in a temporal diagram1052, and a L block rate (a percentage of sessions that are operating on L blocks) is plotted in a time domain in a temporal diagram1054. In accordance with the instance diagram, the plurality of data samples or analysis results are plotted with respect to the subset of instances for which the plurality of data samples are collected. For example, an average game latency is obtained and plotted for a subset of instances selected for Tetris Game in the instance diagram1056, and a GPU resource assignment is plotted for the subset of instances in the instance diagram1058. In accordance with the session diagram, one of the subset of game instances is selected, e.g., in a field1048, and the plurality of data samples or analysis results are plotted for all sessions of the selected one of the subset of game instances.

Referring toFIG.10C, the GUIs1000of the game profiling and analysis application736visualizes a set of data samples (e.g., GPU resource assignment, an L blocks rate, a frame drop rate and a game latency) for a specific game instance, instance #22, in the temporal domain. The set of data samples includes two data samples that are captured at two distinct instants of time t1and t2. Each data sample includes one or more of a GPU resource assignment, an L blocks rate, a frame drop rate and a game latency corresponding to its respective instant t1or t2. In some implementations, the analytics server440receives the set of data samples from the profiling server430, and compares each data sample with a respective threshold level. In accordance with a comparison result, the analytics server440identifies an anomalous profile event (e.g., a frame drop rate larger than a threshold value of 50%) and generates an alert message indicating occurrence of the anomalous profile event. The alert message1062is sent to the client device and presented on the GUIs1000to remind the user of the client device of the anomalous profile event.

Alternatively, in some implementations, the analytics server440receives the set of data samples from the profiling server430, and compares the two data samples captured at the instants t1and t2. In accordance with a comparison result, the analytics server440identifies an anomalous profile event (e.g., a frame drop rate increase of 50%) and generates an alert message indicating occurrence of the anomalous profile event. The alert message1062is sent to the client device104and presented on the GUIs1000to remind the user of the client device104of the anomalous profile event. In this specific example, a frame drop rate increase is identified based on game state data of the two data samples corresponding to the instants t1and t2. A processor error (e.g., a GPU resource assignment drop of >50%) is determined based on the system performance data of the two data samples that correspond to the online gaming sessions of the corresponding game instance. The frame drop rate increase is associated with the processor error. Thus, it is determined that the root cause of the anomalous profile event is the processor error.

In another example not shown, an increase of a frame drop rate and a game action rate of a specific game action are identified at an instant t2based on the game state data of the two data samples, and the increase of the frame drop rate is associated with the specific game action as being probably caused by the specific game action. For example, in the Tetris game, the increase of the frame drop rate is determined to be caused by L blocks, because a corresponding L block rate is substantially high (e.g., a percentage of sessions that are operating on L blocks exceeds a block rate threshold).

In some implementations not shown, the analytics server440compares values of one of the plurality of data samples (e.g., values of one of the game state data and system performance data in the plurality of data samples) at the first instant t1and the second instant t2subsequent to the first instant t1. The analytics server440determines and/or visualizes a difference between the values of the one of the plurality of data samples at the first and second instants. Further, in some implementations, a game profiling request is sent to set forth a profile criterion at a third instant t3that is subsequent to the first time t1and preceding to the second instant t2. The determined/visualized difference is thereby associated with the profile criterion. Under some circumstances, the profile criterion includes a change of type, compilation, or version of a shader program used to render the one or more synchronous gaming sessions corresponding to the respective game instance.

In some implementations, after one or more of the plurality of data samples are displayed on the GUI1000, the game profiling and analysis application736receives from the GPU a user input identifying an anomalous profile event in the subset of the plurality of game instances from which the data samples are collected. For example, the GUI1000may receive a user input at a peak1064of the average latency to indicate an anomalous profile event. Further, in some implementations, a first game profiling instruction enables a periodic data sample collection, and a subset of the plurality of collected data samples are visualized inFIG.10C. An anomalous profile event (e.g., the peak1064of the average latency) is identified in the plurality of data samples collected in response to the game profiling instruction that enables the periodic data collection. Then, the profiling server430obtains a game profiling request including a target profiling condition associated with the anomalous profile event, and generates a second game profiling instruction to collect a second plurality of data samples according to the target profiling condition. Optionally, the profiling server430automatically determines the target profiling condition based on the anomalous profile event and generates the second game profiling instruction. Optionally, the GUIs1000receives a first user input in the field1020eto define the target profiling condition based on the anomalous profile event, and a second user input on the user affordance1022to generate the game profiling request.

FIG.11is an example flow chart showing a game profiling and analysis process1100in accordance with some implementations. In accordance with the game profiling and analysis process1100, while the game servers118are executing a plurality of game instances of a gaming title, a game profiling and analysis application is executed concurrently on a client device104of a game administrator or developer to receive (1102) user inputs corresponding to a first game profiling request on the GUIs1000for collecting data samples of a subset of the game instances of the gaming title periodically. The first game profiling request for periodic data sampling is sent to the profiling server430, which has a global profiling orchestrator module648. The global profiling orchestrator module648then generates and sends (1104), to the game servers118, a first game profiling instruction for collecting a plurality of data samples for each of the subset of the game instances at the game servers118concurrently while the game servers118are executing the plurality of game instances. In some implementations, the first game profiling instruction is received by a local profiling orchestrator module630of the game servers118, and sent to a collection module632for collecting the plurality of data samples for each of the subset of the game instances. In response to the first game profiling instruction, the collection module632of the game servers118collects the plurality of data samples for each of the subset of the game instances periodically. The collected data samples are optionally sent (1106) to the profiling server430or the analytics server440periodically, according to a predefined schedule or in response to a query (e.g., in a game analysis request).

In some implementations, the analytics server440enables the game profiling and analysis application736to visualize (1108) a subset of the collected data samples on the GUIs1000. The administrator or game developer identifies (1110) the anomalous profile event from the visualized subset of the data samples, e.g., the peak1064of the average latency inFIG.10C. Alternatively, in some implementations, the analytics server440identifies (1112) an anomalous profile event (e.g., a frame drop rate larger than a threshold value of 50%) and generates (1114) an alert message indicating occurrence of the anomalous profile event. The alert message is reported to the client device and presented on the GUIs1000to remind the user of the client device of the anomalous profile event. Stated another way, the analytics server440identifies one or more abnormal game, processor, or system states based on the plurality of data samples, and associates the target profiling condition with the identified one or more game, processor, or system states. The alert message is generated to indicate the target profiling condition has occurred and is caused by the one or more game, processor, or system states.

In some circumstances, after the anomalous profile event is identified in the plurality of data samples collected in response to the game profiling instruction that enables the periodic data collection, the profiling server430receives (1116) a second game profiling request including a target profiling condition associated with the anomalous profile event, and generates (1118) a second game profiling instruction to collect a second plurality of data samples according to the target profiling condition. The second game profiling request is generated by the game profiling and analysis application736when user inputs are received on the GUIs1000defining the targeted profiling condition and the subset of game instances of the gaming title. In some implementations, the target profiling condition includes one or more of a frame drop, a register overflow, a drop of a frame rate, an increase in a game latency, an increase in power consumption of a processor, an increase in a memory occupancy rate, and a change of utilization rate of a processor.

In some implementations, the second game profiling instruction is received by the local profiling orchestrator module630of the game servers118, and sent to the collection module632for collecting a second plurality of data samples for each of the subset of the game instances in accordance with the targeted profiling condition. In response to the second game profiling instruction, the collection module632of the game servers118collects the second plurality of data samples for each of the subset of the game instances according to the targeted profiling condition, and returns (1120) the collected data samples to the profiling server430or the analytics server440periodically, according to a schedule or in response to a query. Specifically, while the game servers118are executing the plurality of game instances, the local orchestrator module of the game servers118automatically detects the target profiling condition occurs to a first game instance, and sends a command to the collection module632to collect the plurality of data samples in association with the target profiling condition. The collection module632of the game servers118collects the second plurality of data samples corresponding to the detected target profiling condition. Each of the second plurality of data samples includes at least game state data and system performance data associated with the one or more synchronous gaming sessions corresponding to each of the subset of the game instances.

FIG.12is an example trace1200recorded for a game instance in accordance with some implementations. In some implementations, while the game servers118execute the plurality of game instances for the gaming title in parallel, the game servers118creates a trace of gaming events for each of the plurality of game instances. The trace of gaming events1200includes a sequence of ordered gaming events occurring to the one or more synchronous gaming sessions corresponding to the respective game instance. The trace1200is configured to include a plurality of timestamps each of which indicating an instant of time at which game state data and system performance data are recorded. In this example, the trace1200records two data samples, three data samples and four data samples for three consecutive game events including the first game event (Game Event 1), the second game event (Game Event 2) and the third game event (Game Event 3), respectively. Examples of the game events includes a user action (e.g., a jump by a game player), a change of game settings (e.g., a costume change of a game player), and a change of a shader program used to render image frames of the online gaming sessions of the game instance.

More specifically, a first data sample1202corresponding to Game Event 1 includes game state data (Game State 1 and Game State 2) recorded for two of the one or more synchronous online gaming sessions of the corresponding game instance at an instant corresponding to a first instant t1(Timestamp 1). This game instance may include other online gaming sessions in addition to the two online gaming sessions corresponding to Game States 1 and 2. For brevity, game state data of the other online gaming sessions of this game instance are not included inFIG.12. Also, the first data sample1202corresponding to Game Event 1 includes two system performance data (System Performance 1 and System Performance 2) recorded for two of a plurality of system characteristics at the first instant t1. This game instance may include other system performance data in addition to System Performances 1 and 2. For brevity, the other system performance data of this game instance are not included inFIG.12. In some implementations, the first data sample1202is recorded when the first game event is triggered or initiated.

Further, in some implementations, the game servers118receives a game profiling instruction corresponding to a plurality of target profiling conditions (e.g., those corresponding to Tags 1, 2 and 3) from the profiling server430. The game profiling instruction includes a profiling metadata call configured to add tags (e.g., Tags 1, 2 and 3) indicating the plurality of target profiling conditions in the trace1200of game events when the trace1200of game events is recorded at the time of executing the game instance. Examples of the plurality of target profiling conditions include, but are not limited to, an occurrence of an L block (Tag 1), an occurrence of a T block, a GPU idle state (Tag 2) and a frame drop (Tag 3). Optionally, a target profiling condition is consistent with a game event. Optionally, a target profiling condition is associated with a value of one of the game state data and the system performance data recorded for the corresponding gaming instant. In response to the predefined profiling metadata call, a subset of the plurality of data samples in the trace1200is recorded and tagged when the plurality of target profiling conditions occur during execution of the game instance. For example, referring toFIG.12, data samples1202,1204and1206are tagged with Tag 1, Tag 2 and Tag 3 to indicate that they are associated with three different predefined target profiling conditions.

Subsequently, in response to a query (e.g., in a game analysis request), the game servers118identifies in the trace a subset of the recorded data samples (e.g., data sample1206) corresponding to the subset of the target profiling conditions based on one or more corresponding tags (e.g., Tag 3), and provides the subset of data samples to the profiling server430or analysis server440. In some implementations, in response to the query, a single data sample (e.g., data sample1206) corresponding to each tag indicating a respective one of the subset of the target profiling conditions is provided to the profiling server430or analysis server440. Further, in some implementations, in response to the query, a single data sample (e.g., data sample1206) corresponding to each tag indicating a respective one of the subset of the target profiling conditions is provided to the profiling server430or analysis server440with a predefined first number of data samples preceding to the respective tag (e.g., data sample1208) and/or a predefined second number of data samples following the respective tag (e.g., data sample1210).

As explained above, in some implementations, the game state and system performance data include a plurality of hardware properties of the game server, a game version of the gaming title, and a kernel/driver version running on the game server. Specifically, the game state data corresponding to an online gaming session includes game context data, e.g., a game level, a weapon type, a costume of a game roll and a number of objects in a corresponding game scene (e.g., a number of leaves on a tree), and is optionally stored as metadata of the online gaming session. The game state data optionally includes one of more of game settings, e.g., a sliding window frame rate and a frame resolution. Further, for each game instance, the system performance data includes performance data of a CPU404, a GPU406, a memory (e.g., a cache or register), an operating system, one or more input/output ports and/or a data bus, which are assigned to execute each of the one or more gaming sessions of the respective game instance. The performance data of the CPU404or GPU406includes a processor utilization rate and power consumption at a time corresponding to each of the plurality of data samples. In some implementations, the system performance data is stored in at least one of CPU performance counters, GPU performance counters, a stack trace, and architectural counters. In some implementations, the system performance data further includes performance information concerning a PCI express switch. The CPU performance counters includes a set of special-purpose registers built into the CPUs404to store counts of hardware-related activities within the CPUs404. The GPU performance counters includes a set of special-purpose registers built into the GPUs406to store counts of hardware-related activities within the GPUs406. The CPU or GPU performance counters record processor utilization rate (e.g., 10% usage) and power consumption. The stack trace includes a report of active stack frames at an instant of time during the execution of the game instance.

In an example, game metadata is interleaved with CPU counters according to a timeline. The analytics server440determines that a number of CPU is idle when a T block come based on the game metadata and CPU counters, and therefore, infers that the CPUs are underload when a scene includes T or L blocks. Thus, the GPUs cannot keep up with the CPUs, and additional processing power is needed from the GPUs. In another example, the analytics server440determines that a high-speed serial computer expansion bus (e.g., a bus complying with a peripheral component interconnect (PCI) express) functions properly while a memory block coupled to the bus has an issue (e.g., a register overflow).

FIG.13is a flow diagram illustrating an example method1300for profiling and analyzing a gaming title at a distributed real-time game profiling system in accordance with some implementations. In some implementations, a distributed real-time game profiling system includes a cloud collection orchestrator located at a profiling server430, and a local collection daemon located at one or more game servers118. The cloud collection orchestrator tells (1302) the local collection daemon running on each cloud-based game instance whether and/or when to collect data related to the respective game instance. The cloud collection orchestrator asks for collection of data samples on a subset of or all cloud-based game instances of a running gaming title (e.g., Tetris and Super Mario). Each data sample collected by the local daemon is a snapshot of state of a corresponding game instance at a corresponding moment. The snapshot includes CPU state (e.g., program counter; stack trace and architectural counters), system state (e.g., PCI express switch data), and GPU state variables. A frequency of sampling for capturing the data samples depends on a tolerance (e.g. whether sample capturing causes any momentary frame stutter) and the amount of data to collect. These data samples are then streamed (possibly batched first) to a collection server/backend (e.g., the profiling server). In some implementations, the collected data samples are also tagged with information about the corresponding game instance (e.g. hardware properties, game version, kernel/driver version). The collected data samples can then be queried and merged across one or more game instances to constitute a program profile.

Symbol data is applied to convert (1304) the captured program counter and stack traces into human-readable data. In some implementations, data samples are captured on a guest side, and the captured data samples is stored and symbolized on the guest side. Alternatively, in some implementations, data samples are captured (1306) and available on the host side, for example, at an operating system of a host (e.g., a server system) in a virtualized system, allowing runtime spent on the host to be inspected. Memory addresses on the host side are converted back to addresses on the guest side in order to determine where a program counter is for a given data sample.

While the game instance is being executed, the server system stores (1308) metadata of interest (e.g. a game level, a weapon type, a costume worn by a character) in a location in-memory that is captured as part of a sample event. The server system uses the same mechanism (or a data-tuned implementation) to store engine-level metrics (e.g., a sliding window frame rate, an instantaneous frame time, a number of objects in a scene, a number of leaves on trees). In some situations, upon sampling, a callback is triggered to fetch relevant key value pairs. Optionally, a callback state is captured alongside the CPU, GPU and System data at each data sample. Optionally, the callback state is captured at a slower sample rate if a corresponding sampling rate is very high and data changes slowly. In some implementations, game state or game metrics are captured without corresponding hardware-related data samples (e.g., CPU performance data) to monitor overall game health in the fleet. A game developer can subsequently increase hardware performance sampling across the game instances for specific game-states to zoom in on a target problem area if detected.

In some implementations, the collection orchestrator daemon of the game server supports (1310) filters the captured data samples in accordance with (a) which game instances are of interest and (b) which game state is of interest. Such pre-collection filtering avoids collecting data samples from uninteresting game instances, and reduces costs for data collection and processing. Optionally, a game state filter prevents the collection orchestrator daemon from capturing an undesirable sample. Optionally, the undesirable sample is captured, e.g., to minimize impact on run-time of the game, and, discarded prior to being transmitted to the profiling server. For example, a game developer learns chatter on a web forum concerning problems on Level 27 of Version 1.2 of a gaming title, and decides to increase a data collection rate of the game instances executed on Level 27 of Version 1.2 of the gaming title for the purposes of obtaining a fine-grained resolution on the CPU profiles.

In some implementations, post-collection filtering is applied (1312) after the collected data samples are obtained by the profiling server and/or the analytics server. The collected data samples are optionally combined based on a number of variables. A user may explore the collected data samples and determine trends. For example, in accordance with filtering on game state and game metrics, the collected data samples are zoomed into one or more subsets of the data samples to facilitate resulting hotspot analysis and CPU profiling.

In some implementations, a subset of collected data samples are visualized (1314). The subset of collected data samples is identified based on one or more sample criteria. Differences among the subset of collected data samples are visualized in real-time. A game profiling application presents a GUI on a game administrator's client device104, and the GUI visualizes the difference between two program profiles captured at different times. The two program profiles are optionally associated with slightly different shader compilations or game versions, which requires best-effort matching between the run-time program profiles. The two program profiles are optionally associated with two different profile criteria, and for example, the two program profiles compare hotspot analysis between Level 27 and all Levels to see what aspect of run-time has increased.

In some implementations, sensitivity analysis is implemented (1316) on the collected data samples by a visualization tool or ingestion pipeline. In an example, variables are analyzed to automatically flag which game-state conditions correlate most with degradation of a frame rate. In another example, game-state variables are analyzed and/or visualized to identify one of the game-state variables has the greatest impact on a given function.

In some implementations, alerts and triggers are generated (1318) based on the collected data samples. A game publisher or developer sets alerts for one or more anomalous profile events, thereby allowing the game publisher or developer to proactively identify client-side glitches in the game similar to what they often monitor on the server system (e.g., in multiplayer games). Alerts for one or more anomalous profile events allows the game publisher or developer to attach a trace-based profiler to a game instance of interest and/or trigger more complete logging/streaming of data from one or more of the participating game instances.

In some implementations, the collected data samples are used to handle (1320) heterogeneous hardware. In an example, a set of Android phones are coupled to game servers with differing hardware A pre-filter is optionally used to target a set of equivalent devices. A post-collection filter is optionally executed to scale/adjust the collected data samples according to hardware settings.

In some implementations, sample-based profiling is executed on consoles of the game servers (1322). The collected samples are stored locally in a memory of the game servers, filtered locally based on criteria sent from the cloud collection orchestrator, de-duped/compressed, and then sent/bundled when a corresponding network is quiescent or the corresponding game instance is finished. Optionally, the amount of data that could be collected is reduced.

In some implementations, data sample collection is adjusted (1324) according to a power level of a client device, e.g., to minimize impact on gameplay on the client device. For example, data samples are collected for a gaming session of a game instance in accordance with a determination that a battery level of a mobile device is beyond 50% or that the mobile device is plugged onto a power supply.

In summary, sampled data collection of both game state data and system performance data enables powerful analysis of resulting data samples. Such sample-based profiling and analysis is implemented not only during a game development phase, but also in real time while the corresponding game instance is being executed. A game developer can therefore readily determine how world-state, player-state, and new additions/changes to the corresponding game instance impact performance and experience of game players.

FIGS.14A-14Eare a flow diagram illustrating another example method1400for profiling and analyzing a gaming title at a distributed real-time game profiling system in accordance with some implementations. Method1400is, optionally, governed by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a distributed real-time game profiling system. Each of the operations shown inFIGS.14A-14Emay correspond to instructions stored in a computer memory or non-transitory computer readable storage medium (e.g., memory606of a server system114inFIG.6A). The computer readable storage medium may include a magnetic or optical disk storage device, solid state storage devices such as Flash memory, or other non-volatile memory device or devices. The instructions stored on the computer readable storage medium may include one or more of: source code, assembly language code, object code, or other instruction format that is interpreted by one or more processors. Some operations in method1100may be combined and/or the order of some operations may be changed.

In the distributed real-time game profiling system, a game server118executes (1402) a plurality of game instances for a gaming title in parallel (e.g., by game application628inFIG.6A). Each of the plurality of game instances includes (1404) one or more synchronous gaming sessions, and each of the one or more gaming sessions is executed at a game application of a client device.

In some implementations, while the game server118is executing (1406) the plurality of game instances, a profiling server430enables (1408) display of a GUI on a game profiling and analysis application736implemented at a client device104of an administrator or a game developer. The GUI is configured to receive from a user a game profiling request, send the game profiling request to the profiling server430, receive analysis results from the analytics server440, and present the results on the GUI of the client device104of the administrator or game developer.

The profiling server430generates (1410) a game profiling instruction to collect a plurality of data samples for each of a subset of the plurality of game instances. The profiling server430is coupled to the game server118over one or more wireless communication networks. The profiling server430sends (1412) the game profiling instruction to the game server118. Optionally, the game profiling instruction includes a data type of the plurality of data samples to be collected. Optionally, the game profiling instruction defines one or more of a data collection schedule, a target profiling condition and a time window. In an example, the game profiling instruction is generated to collect data samples related to a frame drop for the next hour or to collect data samples periodically at a sampling frequency. Optionally, the game profiling instruction selects the subset of the plurality of game instances randomly, according to an instance pitch, or according to one or more instance characteristics.

In response to the game profiling instruction (1414), a local orchestrator module of the game server118identifies (1416) the subset of the plurality of game instances according to the game profiling instruction, and a collection module632of the game server118collects (1418) the plurality of data samples associated with each game instance in the identified subset of the plurality of game instances. Each of the plurality of data samples includes at least game state data and system performance data associated with the one or more synchronous gaming sessions corresponding to the respective game instance. In some implementations, the game profiling instruction further includes a target profiling condition (e.g., a frame drop, a drop of a frame rate, an increase in a game latency, an increase in power consumption of a processor, an increase in a memory occupancy rate, and a change of utilization rate of a processor). In some situations, the profiling server430receives a user input defining the target profiling condition. While the game server118is executing (1414) the plurality of game instances, the local orchestrator module630of the game server118automatically detects (1422) that the target profiling condition occurs to the subset of game instances. The plurality of data samples collected by the collection module632of the game server118correspond to the detected target profiling condition.

In some implementations, while the game server118is executing (1414) the plurality of game instances, the game server118creates (1424) a trace of gaming events for each of the plurality of game instances, and the trace of gaming events includes an ordered sequence of gaming events occurring to the one or more synchronous gaming sessions corresponding to the respective game instance. Further, in some implementations, the game profiling request includes a predefined profiling metadata call, and in response to the predefined profiling metadata call, the game server118records and tags (1426) the plurality of data samples in the trace corresponding to each instance. Then, in response to a query, the game server118identifies (1428) a subset of the plurality of data samples in the trace based on one or more corresponding tags. More details on recording and tagging a trace are discussed above with reference toFIG.12.

After collecting the plurality of data samples, the game server118sends (1430) the plurality of data samples associated with each game instance in the identified subset of game instances to the profiling server320. In some implementations, for a subset of the plurality of data samples, the game server118obtains (1434) a respective difference between each of the subset of the plurality of data samples and a reference data sample, and sends (1436) the respective difference to the profiling server430to represent each of the subset of the plurality of data samples. Optionally, the plurality of data samples include the reference data sample, and the reference data sample is sent to the profiling server320, such that an analytics server may recover each of the subset of the plurality of data samples by combining the respective difference and the reference data sample. Alternatively, in some implementations, the plurality of data samples includes a first data sample and a second data sample that is collected subsequently to the first data sample, and the second data sample corresponds (1438) to a delta data that is equal to a difference between the first and second data samples. The first data sample and the delta data are sent to the profiling server430while the second data sample is kept at the game server118.

In some implementations, the collected plurality of data samples are tagged with additional information that is not included in the system performance data, e.g., game version, operating system version. These data samples can be queried and merged across one or more game instances, and constitute a game profile. That said, in some situations, the profiling server430is configured to create a game file based on the plurality of data samples collected for each game instance in the identified subset of plurality of game instances.

The profiling server430provides (1440) the plurality of data samples to an analytics server440, thereby allowing the analytics server to analyze the plurality of data samples to improve gaming performance and/or user experience associated with the gaming title. Optionally, the analytics server440is located remotely from the game server118and the profiling server430. Optionally, the analytics server440is configured to operate offline. In some implementations, the analytics server440visualizes (1442) a subset of the game state data and system performance data in the plurality of data samples.

In some implementations, the game profiling instruction enables a periodic data sample collection. The analytics server440identifies (1444) an anomalous profile event in the plurality of data samples collected in response to the game profiling instruction that enables the periodic data collection. The profiling server430receives (1448) an updated profiling request including a target profiling condition associated with the anomalous profile event, and generates (1450) a second game profiling instruction to collect a second plurality of data samples according to the target profiling condition. Specifically, in some implementations, the analytics server440enables display of the plurality of data samples on a GUI of a game profiling and analysis application implemented at a client device104of an administrator or game developer. The analytics server440receives from the GUI a user input identifying an anomalous profile event in the subset of the plurality of game instances. Alternatively, in some implementations, the analytics server440automatically identifies the anomalous profile event.

In some implementations, the analytics server440compares (1452) values of one of the plurality of data samples at a first time and a second time subsequent to the first time, determines a difference between the values of the one of the plurality of data samples at the first and second times, and visualizes the difference between the two of the plurality of data samples. Further, in some situations, the game profiling instruction includes a profile criterion configured to take effect at a third time that is subsequent to the first time and preceding to the second time, and the profile criterion (1454) is then associated with the determined and visualized difference. An example of the profile criterion includes a change of type, compilation, or version of a shader program used to render the one or more synchronous gaming sessions corresponding to the respective game instance.

In some implementations, the game server118includes a plurality of processors. Generally, the analytics server440is configured to implement at least one operation of conducting (1456) hotspot analysis of the plurality of processors based on the plurality of data samples, profiling (1458) the plurality of processors based on the plurality of data samples, and profiling (1460) the plurality of game instances and the one or more synchronous gaming sessions corresponding to each game instance.

In some implementations, the analytics server440compares (1462) two of the plurality of data samples corresponding to a first time and a second time subsequent to the first time, and identifies an anomalous profile event in accordance with a comparison result. An alert message is generated to indicate an occurrence of the anomalous profile event. Further, in some implementations, the analytics server440identifies (1464) a frame drop rate increase based on the game state data of the two of the plurality of data samples, and a processor error based on the system performance data of the two of the plurality of data samples. A root cause of the frame drop rate increase is associated with the processor error. Additionally, the analytics server440identities (1466) an increase of a frame drop rate and a game action rate of a specific game action based on the game state data of the two of the plurality of data samples, and associates the increase of the frame drop rate with the specific game action. For example, the gaming title includes a Tetris game. The analytics server440determines (1468) that the increase of the frame drop rate is caused by L blocks in the Tetris game based on the plurality of data samples.

In some implementations, the plurality of data samples are collected in response to a game profiling instruction including a target profiling condition. The analytics server440determines that one or more game, processor or operating system states are abnormal based on the plurality of data samples, and associates the target profiling condition with the determined one or more game, processor or operating system states. The analytics server440generates an alert message indicating the target profiling condition has occurred and is caused by the one or more game, processor or operating system states.

In some implementations of this application, the distributed real-time game profiling system includes a plurality of servers having distinct response times, and the plurality of servers are assigned to game management, profiling orchestration and data analysis based on the distinct response times. The game server118corresponds to a first game latency, and the profiling server430is configured to respond to the game profiling instruction with a profiling latency that is larger than the first game latency. The analytics server440is configured to respond to a game analysis request with an analysis latency that is larger than the profiling latency. A majority of computational, storage and communication resources of the game server118can be conserved for latency-sensitive game rendering, thereby providing desirable user experience with substantially low game latencies to multiple remote game players.

The distributed real-time game profiling system dedicates the profiling server430and the analytics server440to game profiling and analysis in real time while one or more corresponding gaming titles are being executed by the game server118. Once a defect is identified in a hardware unit (e.g., a CPU or GPU) or in a software program (e.g., an OS program or a shader program), the hardware unit can be reassigned, and the software program can be debugged in real time without interrupting the execution of the gaming titles. In addition, the profiling server430and the analytics server440provide a unified solution (e.g., a unified game profiling and analysis GUI) to profile and analyze performance of different gaming titles, thereby allowing game profiling and analysis to be implemented efficiently without being limited to different gaming titles.

It should be understood that the particular order in which the operations inFIGS.14A-14Ehave been described are merely exemplary and are not intended to indicate that the described order is the only order in which the operations could be performed. One of ordinary skill in the art would recognize various ways to collect data from game servers118as described herein. Additionally, it should be noted that details of other processes described herein with respect to method1300(e.g.,FIG.13) are also applicable in an analogous manner to method1400described above with respect toFIGS.14A-4E. For brevity, these details are not repeated here.

Gaming API Platform Supporting Third-Party Content

Implementations described in this specification are directed to providing a gaming application programming interface (API) and cloud platform to enable efficient, portable, low latency hosting of third party gaming content. Some implementations dynamically allocate cloud gaming hardware resources and monitor and utilize network bandwidth available to individual end users to provide an optimal cloud gaming experience. Some implementations provide multiple performance tiers, including a tier that supports high performance, real time gaming sessions with high definition video/media output/end user streams (e.g., gameplay streams130and spectator review streams132). Some implementations support different subscription models and/or are configured to provide one or more concurrent real time gameplay and/or instructional media streams that correspond with little or no latency to one or more actual gaming streams (e.g., a video stream output to a client device102of a user participating in an online/cloud gaming session via either a mobile app or a browser-based program). In some implementations, concurrent gameplay and/or instructional videos are provided with little or no latency via a media streaming site, such as YouTube, to one or more users, in particular, using a multi-stream generation module662.

Some implementations enable the hosting of public events in conjunction with gaming competitions. For example, in conjunction with a multi-player gaming event or competition based on a hosted game, in some implementations a cloud gaming site can broadcast or stream to specific spectators: (a) one or more concurrent ancillary or supplemental media streams, including associated commentary tracks/streams, (b) gaming streams from different competitor points of view, a highlights stream showing particularly compelling gaming action based on cloud server analysis and/or scoring of multiple gaming sessions associated with the gaming event, (c) one or more game point of view streams reflecting gameplay sessions of one or more active gamers, and/or (d) instructional tracks from one or more active gamers and/or commentators, possibly including real time picture-in-picture (PIP) video sent by the active gamers to the cloud gaming server system114along with their corresponding gameplay responses.

Some implementations provide content hosting and/or end user gameplay API's that enable developers of gaming content to focus on new gameplay features without the need to address programming related to infrastructure tuning issues, such as network and server performance optimization, that ensure an optimal gaming experience for end users. This also enables developers of gaming content to take advantage without additional programming of advances in hardware and/or network resources that are provided by the hosting organization.

Examples of third party content that can be effectively hosted in a cloud-based gaming system/environment in accordance with some implementations include, without limitation, sports games, racing games, role playing games (RPG) and first person shooter (FPS) games. Different instances of these games may have widely varying cloud hardware requirements and network (e.g., to ensure an optimal user gaming experience—consistent in some instances with different subscription performance tiers) based on different associated latency requirements and expectations, output video resolution, and gaming server computational workload and video encoding/streaming resources, and network bandwidth.

Some implementations provide account management APIs and/or software modules that monitor gameplay activity and related requests of subscribers (e.g., requests by end users to invite other players to participate in a gaming session, upgrade their in-game tools, and/or gaming performance) and transmit or make available by APIs associated information to the third party content provider to enable the content provider to track settings (including but not limited to billing information, in-game credits, subscription level, etc.) of their subscribers and/or followers. In some implementations, a content provider of hosted content can provide via the same hosting platform one or more different subscription models for the hosted content. In some implementations, a user (e.g., a subscriber to a gaming service) is granted unlimited access and gameplay to all games offered by the content provider on the hosting platform. In some implementations, a user (e.g., a subscriber to a gaming service) is granted unlimited access and gameplay to one or more specific gaming franchises (e.g., a specific football or first person shooter franchise) offered by the content provider on the hosting platform. In some implementations, the subscriptions are for limited participation by a user—where the participation can be limited based on gameplay time, level of hardware resources committed to the end user, or end user device type/location. The account APIs and modules provided in some implementations configure and monitor gameplay sessions, and enable the content providers to track, gaming activity of respective subscribers in accordance with their most current subscription information—even during active gameplay.

The gaming API platform includes cloud features that allow a user to move around, e.g., suspending a first game stream of a first gaming session executed on a first client device102, and restarting the first game stream on a second gaming session of a second client device102to continue the first game session. The gaming API platform also supports multiple players on a massive scale, and provides richer more persistent cloud-based worlds. The gaming API platform uses a cloud-based system to store information related to different gaming sessions of the same user, or different gaming sessions of different users.

The gaming API platform renders gaming content on a plurality of client devices102, including but not limited to, mobile phones, tablet computers, desktop computers, and televisions. Optionally, the gaming content is dynamically adjusted to comply with the specifications of these client devices102. In some implementations, the client devices102have a limited or no storage capability.

In addition to gaming content, the gaming API platform also streams to the client devices add-on content, e.g., new league rosters, statistics, and preview access to early titles, which is optionally updated regularly (e.g., readily updated, upgraded every day or every hour). In some implementations, the add-on content includes search result of an Internet search or a database search.

In some implementations, the gaming API platform includes a live online community. Users (e.g., subscribers of a service) participate in live events, tournaments or activities on the gaming API platform throughout the day. Examples of the live events, tournaments or activities include spectating live gaming sessions played by other users, posting your accomplishments to a public domain (e.g., YouTube), and getting live tips and coaching videos. For example, in response to a user action, the gaming API platform provides two or more live streams. While keeping a first gaming stream on a first gaming session of the first client device for a game player, the gaming API platform also broadcasts a second live YouTube stream to one or more other client devices102(e.g., of subscribers). The second live stream allows the user to share gaming experience with audience. Optionally, the second live stream is a reproduction of a screen of the first client device of the player. The gaming API platform may capture an audio stream with the player explaining the first gaming session, or a video stream of the player playing and explaining the first gaming session. The audio stream is optionally played for the audience while the second live YouTube stream is played for the audience. The video stream is optionally played in an embedded window in the second live YouTube stream.

In some implementations, a method is implemented at a server system114for providing media streams (e.g., a first gaming stream and a second live YouTube stream that are optionally related to an online gaming session) concurrently. Encoders of the server system114identifies frame data of an image frame associated with an online interactive session that optionally includes high definition media content, and generates a plurality of media streams corresponding to the online interactive session. The plurality of media streams include a low latency stream and a normal latency stream. The low latency stream has a faster “response rate” and lower transmission latency than the normal latency stream. The server identifies a predefined frame rate for encoding a sequence of image frames of the low latency stream. For each image frame in the first sequence of image frames of the low latency stream, the server determines a first time interval corresponding to the predefined frame rate, and encodes the image frame during the first time interval. After encoding the image frame, the server determines a remaining time left in the first time interval. In accordance with a determination that the remaining time left in the first time interval satisfies a normal latency requirement associated with the normal latency stream, the server generates a subset of the second sequence of image frames during the remaining time of the first time interval. Alternatively, in accordance with a determination that the remaining time left in the first time interval does not satisfy the normal latency requirement, the server aborts generating any image frame of the second sequence of image frames during the remaining time of the first time interval.

Some implementations provide true on-the-go gaming, allowing the user to take their most desired games to any location or client device. For example, a user can start a gaming session on a mobile device on his or her commute, then seamlessly resume the gaming session at his or her destination on a laptop. Also, in some implementations, based on the different client device resources available to a user as the gaming session is handed off between different devices, the gaming/hosting server114can dynamically deploy a different set of hardware resources to optimize the user's gaming experience based on the different end user current device resources (e.g., client hardware capability and network bandwidth).

Some implementations provide Instant access and require no or little user device storage. For example, a user can start playing in 5 seconds and save approximately 250 GB of console hard drive space.

Some implementations provide regular content drops (e.g., DLC downloadable content), roster updates, live events, tournaments and more gaming exclusives to platform provider integrated with gaming content provider.

The flexibility inherent to implementations of an API-based game hosting platform described herein also provides at least one or more of the following technical advantages and features to content developers who port their content to the platform and the content provider's associated users, potential users or fans:

In some implementations, the gaming platform is configured to add new games over time (particularly, new front-line games), thereby increasing value to content providers.

In some implementations, the gaming platform is configured to provide “Viral Trial” invitations to users in accordance with directions from a content provider. For example, gaming content provider subscribers might receive5invitations per month included in their subscription (monitored by the gaming platform provider and/or the content provider via the account API and/or modules). In some implementations, these invitations can provide one or more of the following technical features to recipients and/or senders of invitations and content and platform providers. Senders can send to any friend on the internet a quick link to a game session. That user (i.e., invitation recipient) can play a multiplayer game with their friend.

In some implementations, at the end of an invite-triggered gaming session, the invited user (invitee) is offered the opportunity to subscribe to either the gaming platform as a whole and/or to content offered by the content provider on the provider's gaming platform.

Depending on available hardware capacity/gaming loads and/or policies defined by the platform provider and/or content provider, trial invites could be restricted to certain time of day or specific trial events (e.g., trial events planned in conjunction with public competitions, new releases for particular franchises or games offered by the content provider, or updates to the gaming platform and associated end user gaming application).

In some implementations, the gaming platform can configure its service according to a predetermined criterion and/or a subscription. For example, for whole service users, the gaming platform offered free access to a certain game for a weekend.

In some implementations, the gaming platform provides Live events integrated with YouTube or another media streaming and/or broadcast platform. For example:

Specific live community events (e.g., a Field Goal challenge associated with a specific content provider football game franchise) programmed to coincide in time with specific events broadcast on YouTube (e.g., the Super Bowl).

In some implementations, the live events are hosted on the gaming platform and/or on an online platform provided by the content provider, and affordances are provided on user portals for those platforms/hosted events to allow view participation or to convert viewers to subscribers (i.e., to allow viewers to participate in the live event if they are subscribers or to subscribe to the gaming platform (including different subscription levels and/or content packages) and/or to hosted content offered by the content provider ((including different subscription levels and/or content packages).

FIG.15is an example flow chart of a method1500of executing a gaming session on a gaming API platform in accordance with some implementations. As shown inFIG.4, the gaming API platform is created by/hosted at least by an API Platform server (e.g., game servers118) and enables the gaming session in conjunction with a gaming frontend server402and content servers450. The gaming frontend server402is configured to provide service to a user of the gaming session, and to manage accounts for users. Optionally, the users have to subscribe to a gaming service via the gaming frontend server402. The content servers450provide gaming content related to the gaming session. The API Platform server (e.g., game servers118) executes (1502) gaming applications to enable the gaming session based on the gaming content. In some implementations, the gaming frontend server402and the API Platform server (e.g., game servers118) are owned and managed by distinct entities. Specifically, an online gaming application is executed for displaying a gaming user interface on a first client or media device (e.g., a television). A user selection is received (1502) at the user interface for subscribing to gaming service. In relation to the user's request to subscribe to the gaming service, the server system114receives (1504) user information, thereby allowing a user of the first client device102to register for one of different types of services (e.g., a free trial service or a regular subscribed service) for each individual gaming title. On the user interface, the user of the client device102may browse (1506) through a series of rich hubs for each content domain (also called content franchise). The user interface is configured to display game content, add-on content (e.g., news, statistics) and live events (e.g., gaming broadcast from another user, live chat messages).

In some implementations, a user selection of a game (e.g., a live football game) is received (1508) at the user interface. In response to the selection, the game application732is loaded (1508) for the first client device102within a predetermined delay time, and a gaming environment is displayed for the user. Alternatively, in some implementations, in response to a user action, the server system114provides (1510) live streams to the user of the first client device102. For example, the server system114provides a first gaming stream130on the first gaming session of the first client device102, and concurrently broadcasts (1510) a second live YouTube stream132to one or more other client devices102. The second live stream132allows the user to share gaming experience with audience. Optionally, the second live stream includes an audio feed or a video feed provided by the user of the first client device102while the user is playing the gaming based on the first gaming stream.

In some implementations, the user of the first client device102may need to use a distinct client device102(e.g., due to moving to a different location. In response to a user action to pause the first game session, the server system114suspends (1512) the first game stream of the first gaming session executed on the first client device102, and records latest status information concerning the first gaming session. The user of the first client device102may restart (1512) the first game stream on a second gaming session of a second client device102to continue the first game session. The latest status information (e.g., stored in the game state data508) is transferred to the second gaming session to allow seamless gaming experience for the user of the client device102.

Referring toFIG.15, a server system114includes a frontend server402and a gaming platform server (e.g., game servers118) each of which may have a respective user account system. In an example, the user account system for the frontend server402is used to manage subscription to specific gaming content and service, and the user account system for the gaming platform (e.g., a YouTube or Google account) is used for managing gaming experience (e.g., rendering gaming content to satisfy specific gaming criteria) and many other purposes. In some implementations, these two user account systems share customer and usage data (e.g., social, friends, presence, authentication, account information, billing information). Information of the user account system is stored in user information690in association with one or more game applications628to which each user has subscribed.

In some implementations, the content servers450provide a service layer that sits on top of a technology layer provided by the gaming API Platform server (e.g., game servers118). In some implementations, gaming content server(s)450manage additional user account systems for accessing their content. Optionally, the additional user account systems for gaming content are integrated with the user account system for the gaming frontend server402that manages user subscription (e.g., using a game application platform658in game frontend module636).

In some implementations, the gaming platform provider maintains cloud platform, infrastructure, streaming reliability, etc. (e.g., the software modules, data structures and storage, and computing infrastructure shown onFIGS.6A-C) and allowed the content provider to focus on gaming-related features of particular hosted gaming titles.

In some implementations, the cloud gaming platform provides APIs, data structures and modules (i.e., software modules) that support cross-play between cloud and PC versions for select gaming titles (e.g., a game that is executing on a server system114as shown inFIGS.6A-6Cor a client device102as shown inFIG.7). A user can transition between his or her PC version of a particular gaming title offered by a content provider; or a subscriber to an online version of a title who also owns the PC version of the same title can play the PC version and concurrently interact with an online session for the same title, or an online subscriber can interact with PC players of the same title (perhaps subject to special invitation to the PC gamers if they are not already subscribers).

Technical advantages to content providers of hosting content on gaming platforms (and in some cases to providers of hosted gaming platforms) in accordance with implementations described herein include but are not limited to:providing content providers with an optimized, hosted gaming platform on which optimization and gaming resources and programming are managed by the platform provider;shifting programming tasks related to content hosting and optimization from content providers to gaming platform providers in hosted gaming platform implementations;enabling content providers to offer multiple multi-franchise subscriptions and multiple subscription tiers based on APIs and modules provided by the platform provider;enabling party studio relationships;enabling exclusive games to be offered;providing a technology/engine partnership between content providers and platform providers; andenabling a previously unprecedented level of live programming in association with hosted gaming services—with very little or no programming burden for the content provider.

In some implementations, the application/gaming session is associated with an online, multi-player game developed by a content provider that is hosted in a cloud gaming environment provided by a gaming platform provider as described herein. The gaming session is configured in some implementations to execute on the gaming platform and interact with end users in accordance with hosting modules and APIs provided by the gaming platform provider. In some implementations, the gaming platform and/or APIs described herein enable different hosted online games to be optimized for gameplay on different hardware and in view of different client device resources and network bandwidth with minimal programming requirements on the part of the content provider. In some implementations, the gaming user is a subscriber to gaming content offered by the content provider. As described herein, in some implementations, the gaming platform provider provides account management modules and APIs that support multiple subscription tiers. For example, in some implementations the account management modules and APIs monitor and collect gameplay statistics/information for individual users based on associated subscription information for that user, enable relevant subscriber gameplay information to be provided to the content provider, enforce (sometimes in conjunction with the content provider) subscription limits associated with individual users (e.g., limits on subscribed content, level of play, time of play, etc.). In some implementations, the gaming platform also provides social and public services related to multiplayer or individual player gaming events or competitions—for example, the gaming platform can be configured to provide (e.g., stream or broadcast) with no programming burden on content providers one or more of: real time gameplay streams, instructional streams, and/or commentary streams, in real or near-real time based on concurrent active gaming sessions. Any of the features described herein for providing cloud-based gaming services can be implemented in conjunction with content provided by content providers that is hosted on a gaming platform provided by a gaming platform provider.

In some implementations of this application, the gaming API platform created by the server system114enables real-time game profiling and analysis for third-party content. As explained above, the server system114includes a profiling sever430and an analytics server440. While a game server118is executing a plurality of game instances for a gaming title (which is enabled using third party content provided by a third party content server450) in parallel, the profiling server430generates a game profiling instruction to collect a plurality of data samples for each of a subset of the plurality of game instances, and sends the game profiling instruction to the game server118. In response to the game profiling instruction, a local orchestrator module630of the game server118identifies the subset of the plurality of game instances according to the game profiling instruction, and a collection module632of the game server118collects the plurality of data samples associated with each game instance in the identified subset. Each of the plurality of data samples includes at least game state data and system performance data associated with the one or more synchronous gaming sessions corresponding to the respective game instance. The game server118sends the plurality of data samples associated with each game instance in the identified subset to the profiling server430. The profiling server430provides the plurality of data samples to the analytics server440, thereby allowing the analytics server to analyze the plurality of data samples to improve gaming performance and/or user experience associated with the gaming title supported by third party game content.

FIG.16is an example flow chart of another distributed method1600of hosting electronic games on a gaming API platform in accordance with some implementations. The method1600is implemented at an Internet-connected game server (e.g., game server118inFIG.4) including one or more processing units and memory storing programs that are executable by the game server. The programs include a plurality of game applications (e.g.,628inFIG.6A) and a plurality of gaming platform modules (e.g., modules630-655and658inFIG.6Aand modules660-680inFIG.6B) that manage execution of the game applications. In some implementations, each of the plurality of game applications is associated with a user frontend, e.g., a third-party developer interface that is optionally enabled by a developer interface module655. That said, the third-party developer interface is provided to a third-party game developer to develop a game application without being heavily involved in gaming implementation and resource management, because the plurality of gaming platform modules are provided by the Internet-connected game server to provide these services in the background. The game server obtains (1602) a first game application (e.g., a game application628) that includes first gameplay instructions that are executed by the game server to perform gameplay of the first game application and one or more API calls to host routines that are implemented by the gaming platform modules630-655and658-680. Specifically, in an example, the first game application is obtained from the third-party developer interface655associated with the third-party developer. In some implementations, the host routines include instructions that, when executed by the game server, implement one or more performance management features (e.g., GPU time sharing, multi-stream generation and parallel game encoding enabled by the modules660,662and664, respectively) of the game server transparently to the first game application. In some situations, the instructions implement the management features of the game server on behalf of the first game application.

The game server receives (1604), e.g., via the user frontend636, a first user request to initiate a first gameplay session for the first game application, and in response to the first user request, initiates (1606) the first gameplay session and manages execution and context of the first gameplay session under control of the gaming platform modules. It is noted that “gameplay session” is also called “gaming session” in this application. In some implementations, a game application platform658is implemented by a game frontend module636for providing a user interface that consolidates accesses to multiple game applications628that execute different gaming titles based on different gaming content (including third party gaming content). Optionally, the first gameplay session is a multiple gameplay session, the game server can provide profile information for the first game application based on services requested by the first user. To manage execution and context of the first gameplay session, the game server receives (1608) first gameplay inputs, e.g., via the user frontend, executes (1610) the first gameplay instructions of the first game application and updates a first context of the first gameplay session in the game state data508in response to the first gameplay inputs. The first context reflects interactions of a first user with the first game application, for example, from a point of view of the first user. Optionally, the first context includes one or more of status in the first game application of the first user, scene/environment of the first game application and interactions with other participants in the gaming session.

The game server generates (1612) and transmits to the first user, e.g., via the user front end, an output gaming stream130that reflects the updated first context of the first gaming session, which has been stored in the game state data508. In some implementations, the output gaming stream is transmitted with parallel social streams132, e.g. an instructional video or audio stream and a social media message stream. In some implementations, the output gaming stream130is provided in near real-time to reflect the updated first context of the first gaming session. In some implementations, the gaming stream130is displayable by a user display device108.

The game server manages (1614) performance of the first gaming session in accordance with session management routines implemented by the platform modules and/or first API calls associated with the first game application, e.g., using a latency adjustment module672, a resource tuning module674, a disruption detection module678and a state reconciliation module680.

In some implementations, the first user request to initiate the first gameplay session is received at the game server via a first developer interface655that is configured to enable users to initiate gaming sessions with one or more game applications628, including the first game application, provided by a first developer of the first game application628for execution on the game server. In some implementations, the first developer is associated with an organization different from the organization that operates and implements the game server.

In some implementations, the first developer is logged onto the game server via the first developer interface655. Further, in some implementations, an online marketplace (e.g., a game application platform658) is provided to users to purchase one-time and/or subscribe access to one or more second game applications provided by a plurality of distinct third party game developers, including the first developer, for execution on the game server. The game server further provides to each of the second game applications access to the API calls to host routines implemented by the gaming platform modules630-655and658-680.

Optionally, the first developer interface is implemented on the game server118. Optionally, the first developer interface is implemented on a third-party server402that is distinct from and communicatively coupled to the game server118.

In some implementations, the game server provides login services to subscribers to the one or more game applications provided by the first developer of the first game application for execution on the game server.

In some implementations, the game server obtains hosting specifications from the first developer for the first game application, wherein the hosting specifications relate to one or more of: quality of service of hosted gameplay sessions for the first game application, logging and/or reporting of user interactions with the first game application, multi-player capabilities of gameplay sessions for the first gaming application, collection and/or reporting of performance profiling data for the first gaming application, gameplay levels of the first gameplay application, and social features for gameplay sessions of the first game application. The social features include availability of streams for viewers of the gameplay sessions, and in some implementations, the availability of streams includes permission of gaming users whose streams are shared to a social site, such as YouTube. Further, in some implementations, the quality of service comprises one or more of: maximum output latency of output gaming streams delivered to gaming users, a predefined range of display resolutions of the output gaming streams, a predefined range of output frame rates of the output gaming streams, a maximum frame drop rate of the output gaming streams, and maximum display skew for multiplayer game sessions. The above hosting specifications are optionally stored in game context508A and game settings508B of the game state data508in association with the hosted gameplay sessions.

In some implementations, the game server provides to the first developer game play information of the first user (e.g., user information690and game state data508) sufficient to enable the first developer to manage account information of the first user.

In some implementations, the game server collects performance information for the first game application (e.g., usage statistics698, game instance traces699and system performance data522), and reports the performance information for execution of the first game application to the first developer. Further, in some implementations, the performance information includes one or more performance parameters of game server hardware and the game application collected concurrently and associated with a gameplay state of the first game sessions, wherein the gameplay state includes one or more of gameplay actions of the first user and associated gameplay context of the first game session. More details on game related information generated and stored by the game server are discussed above with reference to at leastFIGS.5and6C.

In some implementations, the game server obtains game performance specifications for the first game application, and manages interactions of the first user with the first game application in accordance with performance specifications of the first game application. Further, in some implementations, the game performance specifications include gaming performance options, at least some of which are indicated via the API calls. In some implementations, the game performance specifications further include branding guidelines associated with a first developer of the first game application.

In some implementations, managing the performance of the first game application includes one or more of: managing GPU time sharing, managing multi-stream generation, managing parallel encoding of video streams and gaming streams, managing dynamic frame generation, and managing assignment of computing resources of the game server to gaming sessions for the first game application. Each of the gaming sessions is associated with one or more users who are playing the first gaming application. Referring toFIG.6B, the modules660,662,662and668are configured to implemented the aforementioned performance management functions.

In some implementations, the game server is part of a gaming system associated with a plurality of gaming users, each having a gaming controller (e.g.,102B inFIG.1), a network connection to the gaming system, and a display unit (e.g.,108inFIG.1) that enable each gaming user to play the electronic games that are hosted by the gaming system (e.g., based on a media stream130). The gaming system includes a developer backend, a user frontend, a game server (e.g.,118inFIG.4), and a streaming server (e.g.,120inFIG.4). Further, in some implementations, the game server is associated with a plurality of spectator users each having a gaming controller, a network connection to the gaming system, and a display unit that enable each spectator user to view in near-real time video streams132corresponding to the hosted electronic games being played by the gaming users. The game applications show game play of gaming, and optionally provide one or more of account management services (e.g., by game application platform658), application repository (e.g., by game applications628) and gaming profiling services (e.g., by local and global profiling orchestrator module630and648).

Referring toFIG.5, in some implementations, the obtaining the first game application further includes receiving via a developer interface that is accessible to third-party developers a hosting request526to host the first game application628, and obtaining in conjunction with the hosting request executable code528for the first game application and performance specifications. The performance specifications are selected from a plurality of predefined game performance options provided by the gaming platform and relate to one or more of a plurality of performance parameters managed by the gaming platform. The operation of obtaining the first game application further includes configuring the first game application for execution by the game server in accordance with the performance specifications provided via the third party developer interface. Optionally, the selected performance specifications are stored in the user information690in association with the first game application.

In some implementations, in conjunction with the generating and transmitting to the first user the output gaming stream130, the game server generates in parallel one or more social streams132for the gameplay of the first application, and each of the social streams is viewable by spectators who are not playing the first game application628.

In some implementations, the game server executes a plurality of game instances for a gaming title in parallel. Each of the plurality of game instances includes one or more synchronous gaming sessions, and each of the one or more gaming sessions is executed at a game application of a client device102. Further, in some implementations, while the game server is executing the plurality of game instances, a profiling server430generates a game profiling instruction538to collect a plurality of data samples for each of a subset of the plurality of game instances, and sends the game profiling instruction538to the game server. The profiling server430is coupled to the game server118over one or more wireless communication networks. Specifically, in response to the game profiling instruction538, a local orchestrator module630of the game server identifies the subset of the plurality of game instances according to the game profiling instruction538, and a collection module632of the game server collects the plurality of data samples associated with each game instance in the identified subset. Each of the plurality of data samples includes at least game state data508and system performance data522associated with the one or more synchronous gaming sessions corresponding to the respective game instance. The game server sends the plurality of data samples associated with each game instance in the identified subset to the profiling server430, and provides the plurality of data samples to an analytics server440, thereby allowing the analytics server440to analyze the plurality of data samples to improve gaming performance and/or user experience associated with the gaming title.

In some implementations, the output display stream130delivered to the first user represents one or more of: point of view of the first user, status in the first game application of the first user, scene/environment of the first game application, and interactions with other participants in the gaming session.

A distributed real-time game hosting system includes one or more processors (e.g.,406,408and602), and memory (e.g.,606) coupled to the one or more processors. The memory stores one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing any of the above operations as described with reference toFIG.16. A non-transitory computer-readable storage medium (e.g.,606) stores one or more programs. The one or more programs include instructions, which when executed by a distributed real-time game hosting system, cause the system to perform any of the above operations as described with reference toFIG.16.

Reference have been made in detail to various implementations, examples of which are illustrated in the accompanying drawings. In the above detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention and the described implementations. However, the invention may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the implementations.

It will be understood that, although the terms “first,” “second,” etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first device could be termed a second device, and, similarly, a second device could be termed a first device, without changing the meaning of the description, so long as all occurrences of the first device are renamed consistently and all occurrences of the second device are renamed consistently. The first device and the second device are both device, but they are not the same device.