Patent Description:
These and other problems exist regarding streaming quality. <CIT> relates to techniques for stream data encoding. An encoding module determines an encoding profile that specifies hardware encoding devices, input sources, and software encoding capabilities of a system. The encoding module determines an initial encoding workflow based on the encoding profile and initial streaming parameters, wherein the initial workflow specifies an assignment of at least one encoding process to at least one hardware encoding device of the encoding profile, by using one of a statistical analysis process and a machine learning process. Responsive to an input capture request received via an API from a program engine, data specified by the input capture request is encoded in accordance with the initial encoding workflow. Responsive to streaming feedback information received from a streaming destination and CPU utilization data received from an Operating System of the system, the encoding workflow is incrementally updated by applying one of the machine learning process and the statistical analysis process to the feedback information, the CPU utilization data, and the encoding profile, within a time window, and the streaming is updated by using the updated encoding workflow. <NPL>, proposes redirecting enhanced Deep Q-learning toward DASH video QoE (RDQ), a QoE-oriented rate adaptation framework based on enhanced deep Q-learning. First, a chunkwise subjective QoE model is established and it is utilized as a reward function in reinforcement learning so that the strategy can converge toward the direction of maximizing the subjective QoE score. Then, several effective improvements of deep Q-learning are applied to the RDQ agent's neural network architecture and learning mechanism to achieve faster convergence and higher average reward than other learning-based techniques.

One example implementation relates to a server. The server may include a memory to store data and instructions; at least one processor operable to communicate with the memory, wherein the at least one processor is operable to: establish a connection with a client device to stream a game to the client device; receive context information from the client device; send, to a reinforcement learning system, a rank and reward call for a recommendation for channel parameters for the connection, wherein the rank and reward call includes the context information, a user vector, and an item vector; receive, from the reinforcement learning system, the recommendation for the channel parameters in response to the rank and reward call; and use the recommendation to set a value of the channel parameters to stream the game to the client device.

Another example implementation relates to a method. The method may include establishing a connection between a game server and a client device to stream a game to the client device. The method may include receiving context information from the client device. The method may include sending, to a reinforcement learning system, a rank and reward call for a recommendation for channel parameters for the connection, wherein the rank and reward call includes the context information, a user vector, and an item vector. The method may include receiving, from the reinforcement learning system, the recommendation for the channel parameters in response to the rank and reward call. The method may include using the recommendation to set a value of the channel parameters to stream the game to the client device.

Another example implementation relates to a computer-readable medium storing instructions executable by a computer device. The computer-readable medium may include at least one instruction for causing the computer device to establish a connection with a client device to stream a game to the client device. The computer-readable medium may include at least one instruction for causing the computer device to receive context information from the client device. The computer-readable medium may include at least one instruction for causing the computer device to send, to a reinforcement learning system, a rank and reward call for a recommendation for channel parameters for the connection, wherein the rank and reward call includes the context information, a user vector, and an item vector. The computer-readable medium may include at least one instruction for causing the computer device to receive, from the reinforcement learning system, the recommendation for the channel parameters in response to the rank and reward call. The computer-readable medium may include at least one instruction for causing the computer device to use the recommendation to set a value of the channel parameters to stream the game to the client device.

Features and advantages of the disclosure may be realized and obtained by means of the instruments and combinations particularly pointed out in the appended claims. Features of the present disclosure will become more fully apparent from the following description and appended claims, or may be learned by the practice of the disclosure as set forth hereinafter.

This disclosure generally relates to providing personalized settings for streaming channels. In the context of game streaming and/or other streaming scenarios requiring streaming channel configuration, optimizing the streaming channel configuration by dynamically switching between streaming channel parameters (e.g., bitrate, forward error correction (FEC), and/or video resolution) in response to user preferences is not straightforward. It is difficult to receive feedback to drive the streaming channel optimization to achieve a feeling of the user that the content is playing well while being streamed. Playing well may include, for example, content streaming on a mobile device at the same quality level as on a gaming console. Each user may have a different idea of what playing well means. For example, one user may be okay with higher jitter during content streaming but not okay with high latency. While another user may be okay with high latency during content streaming but not okay with high jitter and/or low resolution. As such, the feeling a user may have that the content is playing well may be personal to the user. In addition, it is difficult to receive feedback from users regarding user preferences. Best in class user experience needs to be provided in consonance with real time user feedback. However, capturing user feedback in a high input scenario, such as, gaming, is not trivial. In order to achieve the feeling that the content is playing well while streaming requires dynamic channel parameter switching with sub second latency.

The present disclosure includes several practical applications that provide benefits and/or solve problems associated with personalization of streaming channels. The devices and methods may personalize the streaming channel to a client device of a user via a triangulation of a user interaction history, real time user dynamic context features, and real time adjustments of streaming channel parameters to maximize a reward function that captures the plays well feeling for the user. The devices and methods may collect, or otherwise acquire, the subjective user preferences (e.g., the plays well feeling) via streaming channel configuration dynamism without having to explicitly ask the user for feedback.

The devices and methods may emulate the user feedback by learning game signature functions for each game in the system. The game signature functions may be learned per game per user and/or per a cohort of users. The game signature functions may be mathematical functions that encapsulate a plays well feeling per game, per user cohort, if applicable.

The devices and methods use a cloud computing system to stream games to client devices so that users may play a game on any client device. A cloud computing system refers to a collection of computing devices capable of providing remote services and resources. The devices and methods may dynamically adjust the channel parameters configuration continuously for a connection used to stream the game from a game server to the client device such that the error between an actual play function and the game signature function is continuously minimized as the game play progresses.

The devices and methods aid the dynamic adjustments to the channel configuration with the triangulation of a user interaction history, real time user dynamic context features, and real time adjustments to streaming channel parameters to maximize a reward function that captures the plays well feeling for the user. User interaction history may be captured in terms of a low dimensional latent embedding vector and made available globally via a globally distributed cache.

The dynamic features of the game play may be relayed from a gaming device of the user to a game streaming server via application program interface (API) calls. The API calls may be augmented on the gaming server with features from the latent embedding vector and fed into a contextual bandit that continuously learns a policy to maximize the reward function of plays well using streaming channel parameters. Each actual game play curve may feedback and update into the mathematical function being optimized for that game play session, thereby tracking shifts (if any) in user preferences over time.

The methods and devices may continuously learn the reward function for capturing subjective user preferences without requiring human intervention. The reward functions may be used alongside dynamic user context features to feed a contextual bandit aided with latent user vectors and/or item vectors capturing interaction history over past few months and/or years to help the end to end system continuously learn an optimal policy for providing best in class plays well experiences to the end user via dynamic streaming channel configuration. As such, the method and devices may identify and/or otherwise collect changing user preferences over time without asking the user.

The methods and devices may personalize the channel parameters by dynamically changing the configuration of the streaming parameters for each user while streaming the game to the client device. By personalizing the channel parameters, the present disclosure may achieve a feeling that the game is playing well on the client device for each individual user based on learned user preferences. Personalizing the channel parameters may result in increase user satisfaction with the game system and/or higher user ratings of the game system. Moreover, the gaming system may retain users longer and increase user engagement with the system when users are happy with the performance of the game system. In addition, revenue may increase through increased user engagement with the game system.

Referring now to <FIG>, illustrated is an example environment <NUM> including a cloud computing system <NUM> for use with providing personalized settings for streaming channels when streaming one or more games <NUM> from game servers <NUM> on the cloud computing system <NUM> to client devices <NUM>. Cloud computing system <NUM> may include a plurality of game servers <NUM> in communication with a plurality of client devices <NUM>. Providing personalized settings for streaming channels may include, but is not limited to, dynamically adjusting channel parameters <NUM> of the channel or connection <NUM> used to stream games <NUM> to client devices <NUM> in response to learned user preferences for game play and/or channel conditions. By personalizing the settings for the streaming channels users may play games <NUM> on client devices <NUM> and have a feeling that game <NUM> is playing well on client devices <NUM>.

A user may open a game application <NUM> on client device <NUM> to select one or more games <NUM> to play on client device <NUM>. Game application <NUM> may communicate with cloud computing system <NUM> and may provide a list of available games <NUM> from cloud computing system <NUM>. When the user selects a game <NUM> to play from the list of available games, game application <NUM> may send a request to cloud computing system <NUM> with the selected game <NUM>. Cloud computing system <NUM> may initialize a game server <NUM> to provide game <NUM> to client device <NUM>. A connection <NUM> may be established between game server <NUM> and client device <NUM> via a network so that game <NUM> may be streamed to client device <NUM> using connection <NUM> and the user may play game <NUM> using client device <NUM>.

The client devices <NUM> may refer to various types of computing devices. For example, one or more of the client devices <NUM> may include a mobile device such as a mobile telephone, a smart phone, a personal digital assistant (PDA), a tablet, or a laptop. Additionally, or alternatively, the client devices <NUM> may include one or more non-mobile devices such as a desktop computer, server device, or other non-portable device. The client device <NUM> may refer to dedicated gaming devices (e.g., handheld gaming devices) or a video game console in communication with a display device. In one or more implementations, one or more of the client devices <NUM> include graphical user interfaces thereon (e.g., a screen of a mobile device). In addition, or as an alternative, one or more of the client devices <NUM> may be communicatively coupled (e.g., wired or wirelessly) to a display device having a graphical user interface thereon for providing a display of application content. The game server(s) <NUM> may similarly refer to various types of computing devices. Each of the devices and/or components of the environment <NUM> may include features and functionality described below in connection with <FIG>.

Cloud computing system <NUM> may also include a globally distributed cache <NUM> in communication with game servers <NUM>. Globally distributed cache <NUM> may be a datastore that stores a plurality of user vectors <NUM>. The plurality of user vectors <NUM> may provide a mathematical representation of the gaming history for the users of cloud computing system <NUM>. Each user of the gaming system may have an associated user vector <NUM>. User vectors <NUM> may be a multidimensional array of numbers that provides a condensed representation of the game play for the user for every game <NUM> played by the user. For example, user vectors <NUM> may embed one or more user actions during game play. In addition, user vectors <NUM> may provide a variety of information about the user, such as, but not limited to, a level of play for the user, an amount of money the user spends on games, an amount of time a user plays game, types of games the user plays, and/or a speed of play of the user. In addition, the one or more user vectors <NUM> may illustrate similarities between groups of users. The one or more user vectors <NUM> may also illustrate differences between groups of users. As such, user vectors <NUM> may provide a summary of the total gaming history for each user of the cloud computing system <NUM>.

In addition, globally distributed cache <NUM> may include one or more item vectors <NUM>. Item vectors <NUM> may provide additional game information and/or game features associated with one or more games <NUM>. For example, item vectors <NUM> may embed one or more user actions during game play. In addition, item vectors <NUM> may relate to other items of interest relating to games <NUM>. Item vectors <NUM> may be a multidimensional array of numbers that provide a condensed representation of the game information and/or game sessions for every game <NUM> of cloud computing system <NUM>. Each game <NUM> of the gaming system may have an associated item vector <NUM>. In an implementation, games <NUM> may have a plurality of item vectors <NUM> associated with the users. The one or more item vectors <NUM> may illustrate similarities between games <NUM> and/or gaming sessions. In addition, the one or more item vectors <NUM> may illustrate differences between games <NUM> and/or gaming sessions. The item vectors <NUM> may provide a summary of game information for each game <NUM> of the cloud computing system <NUM>.

In an implementation, user vectors <NUM> and/or item vectors <NUM> may be generated offline through a deep machine learning network. Examples of deep machine learning networks may include, but are not limited to, wide and deep learning and/or Extreme Deep Factorization Machine (xDeepFM). A game streaming history datastore <NUM> may communicate with cloud computing system <NUM> and may contain game data <NUM> for cloud computing system <NUM>. Game data <NUM> may include information acquired or otherwise collected for all games <NUM> in cloud computing system <NUM> and/or all users in cloud computing system <NUM>. Examples of game data <NUM> may include, but are not limited to, game names, game content, an amount of time users are playing games, an amount of money spent on games, and/or actual game play of users. The game data <NUM> may be continuously updated as games <NUM> are created and/or played by users of cloud computing system <NUM>.

Game streaming history datastore <NUM> may apply an interaction featurizer <NUM> that may be trained to analyze game data <NUM> and generate user vectors <NUM> for each user of the cloud computing system <NUM> and/or item vectors <NUM> for each game <NUM> of cloud computing system <NUM>. In an implementation, the interaction featurizer <NUM> may be a deep learning machine model. The interaction featurizer <NUM> may run at a predetermined time period to ensure user vectors <NUM> and/or item vectors <NUM> reflect the latest gaming information received for the users. For example, the interaction featurizer <NUM> may run every four hours. Another example may include the interaction featurizer <NUM> running once a day.

In an implementation, interaction featurizer <NUM> may also be trained to analyze game data <NUM> and provide one or more game recommendations in response to the analysis. One example game recommendation may include providing a top list of recommended games for a user to play. For example, the interaction featurizer <NUM> may use the user information and/or user game interaction data stored in game streaming history datastore <NUM> to narrow down the entire catalogue of games to a recommended list of games for the user to play (e.g., the top ten recommended games out of <NUM> games).

The one or more user vectors <NUM> and/or item vectors <NUM> may be distributed to globally distributed cache <NUM>. As new game data <NUM> is received from game servers <NUM> and/or game data <NUM> is updated, the interaction featurizer <NUM> may be applied to game data <NUM>. User vectors <NUM> and/or item vectors <NUM> may be updated and/or revised as necessary as game data <NUM> is updated. As the user interaction history changes, user vectors <NUM> may be updated to reflect the changes to the user interaction history changes. In addition, as the game data <NUM> changes, item vectors <NUM> may be updated to reflect the changes to the game information. As such, the globally distributed cache <NUM> may include continuously updated user vectors <NUM> that provide a game history summary for each user of cloud computing system <NUM> and/or item vectors <NUM> that provide a summary of the games <NUM> of cloud computing system <NUM>.

In addition, cloud computing system <NUM> may include a globally distributed cache <NUM> in communication with game servers <NUM>. Globally distributed cache <NUM> may be a datastore for one or more game signature functions <NUM> for each game <NUM>. Game signature functions <NUM> may identify an optimal game play curve for the game under good network conditions. An optimal game play curve may represent over a plurality of game sessions how a user (e.g., an expert player, a new gamer, and/or a casual gamer) provides input via a game controller that may best achieve the mission or end result of the game the user is playing. The optimal game play curve may use a plurality of factors in gauging an optimal game play curve. In one example, the optimal game play curve may be represented by plotting the input frame and bytes count of the input provided via the game controller by the user plotted against time.

In an implementation, game signature functions <NUM> may be generated offline through deep machine learning. The game signature functions <NUM> may be generated from the game data <NUM> in game streaming history datastore <NUM>. A trained machine learning model may mine the game data <NUM> and may learn the equation of the curve for one or more game signature functions <NUM> for each game <NUM>. The trained machine learning model may continuously update the game signature functions <NUM> for each game <NUM> as the gaming data <NUM> changes.

The trained machine learning model may split the user population into different cohorts and may learn one or more game signature functions <NUM> differently for each cohort of users. Example cohorts may include, but are not limited to, casual gamers, powerhouse gamers, new gamers, and/or gamers in different geographic regions. For example, the trained machine learning model may analyze the game data <NUM> for users identified in the cohorts to derive the game signature function <NUM> for each game <NUM> for each cohort. The machine learning model may identify similarities in the game data <NUM> and may determine an optimal game play curve based on the analysis of the game data <NUM>. Different game signature functions <NUM> may be determined for each game <NUM> and/or a plurality of games <NUM>.

For example, one game signature function <NUM> for game <NUM> may be for casual gamers (e.g., users who play games infrequently), while a different game signature function <NUM> may be used for game <NUM> for powerhouse gamers (e.g., users who play games frequently). As such, the machine learning model may identify a group of users identified as casual gamers and may identify patterns in the game play of the casual gamers to derive the optimal game play for casual gamers for game <NUM>, while identifying a different group of users identified as powerhouse gamers (e.g., gamers with a high number of playing hours and/or gamers that have been identified as good players) and analyzing the game play of the powerhouse users to derive the optimal game play for powerhouse gamers for game <NUM>. Another example may include using different game signature functions <NUM> for different geographic regions. For example, one game signature function <NUM> may be derived from game data <NUM> for users in North America, while a different game signature function <NUM> may be derived from game data <NUM> for users in Europe. Another example may include using a game signature function <NUM> for new gamers. For example, the game signature function <NUM> may be derived by analyzing the game play of new gamers in general and/or new gamers to the game <NUM>. Another example may include using the same game signature function <NUM> for a plurality of similar games <NUM>. For example, if games <NUM> included similar features and/or game play, the same optimal game signature function <NUM> may be used for the similar games <NUM>. As such, the game signature functions <NUM> may be developed at any level of granularity for the games <NUM>.

The game signature functions <NUM> may be distributed to globally distributed cache <NUM>. As the game data <NUM> changes and/or is updated, the game signature functions <NUM> may change. As such, the game signature functions <NUM> may reflect any shifts in user preferences over time for games <NUM>.

Globally distributed caches <NUM>, <NUM> may allow user vectors <NUM>, item vectors <NUM>, and/or the game signature functions <NUM> to be available to all game servers <NUM> within cloud computing system <NUM>. As game servers <NUM> are initialized, each game servers <NUM> may have access in real time or near real time to the most recent copies of user vectors <NUM>, item vectors <NUM>, and/or the game signature functions <NUM> available in the globally distributed caches <NUM>, <NUM>. Thus, regardless of the geographic locations of game servers <NUM>, each game server <NUM> may be able to access user vectors <NUM>, item vectors <NUM>, and/or the game signature functions <NUM>. Moreover, more than one game server <NUM> may be able to receive a copy of user vectors <NUM>, item vectors <NUM>, and/or the game signature functions <NUM> at the same time. In another implementation, user vectors <NUM>, item vectors <NUM>, and/or the game signature functions <NUM> may be stored in a single globally distributed cache in communication with game servers <NUM>.

Upon initialization of game server <NUM> in response to a user selecting game <NUM> to play, game server <NUM> may access a copy of user vector <NUM> for the user from globally distributed cache <NUM>. For example, game server <NUM> may identify the user of client device <NUM> using a client identification and/or user account information associated with the user and may access the corresponding user vector <NUM> for the user. Game server <NUM> may store user vector <NUM> in a local cache <NUM>. Game server <NUM> may also identify one or more item vectors <NUM> associated with game <NUM> and may save a copy of the one or more item vectors <NUM> in local cache <NUM>. In addition, game server <NUM> may identify one or more game signature functions <NUM> for the requested game <NUM> in the globally distributed cache <NUM> and may save the one or more game signature functions <NUM> for game <NUM> in local cache <NUM>. In an implementation, game server <NUM> may save the game signature function <NUM> for a cohort associated with the user. For example, if the user is a causal gamer, game server <NUM> may save the game signature function <NUM> for causal gamers for game <NUM> in local cache <NUM>.

Game server <NUM> may include a channel manager <NUM> that may establish a connection <NUM> via a network to stream game <NUM> to client device <NUM>. Channel manager <NUM> may receive context information <NUM> from client device <NUM> and may use the context information <NUM> in establishing connection <NUM>. Context information <NUM> may provide the current context of client device <NUM> and/or the user. Context information <NUM> may include, for example, a network type for connection <NUM> (e.g., Wi-Fi or cellular), a network speed for connection <NUM>, bandwidth constraints for connection <NUM>, a geographic location of client device <NUM>, whether client device <NUM> is in motion, and/or an amount of remaining battery power for client device <NUM>. Context information <NUM> may dynamically change as the context of client device <NUM> and/or the user changes. For example, the network may go down or the user may move geographic locations, resulting in changes to the context information <NUM>. In an implementation, client device <NUM> may send context information <NUM> to game server <NUM> using a rank and reward call.

Channel manager <NUM> may use the initial context information <NUM> received from client device <NUM> in establishing connection <NUM>. Channel manager <NUM> may communicate with a reinforcement learning system <NUM> in determining initial values for channel parameters <NUM> for connection <NUM>. Channel parameters <NUM> may include, but are not limited to, video bitrate, resolution, bandwidth, forward error correction (FEC), smooth rendering, rendering, one or more types of jitter, and/or one or more types of latency (e.g., input latency or decode latency) used when streaming game <NUM> to client device <NUM>. Channel manager <NUM> may send a rank and reward call <NUM> with the context information <NUM>, user vector <NUM>, and/or item vector <NUM> to reinforcement learning system <NUM>.

In an implementation, channel manager <NUM> may receive other recommendations for channel parameters <NUM> for connection <NUM> from other systems operating on server <NUM>. For example, the other recommendations for channel parameters <NUM> for connection <NUM> may be based on heuristics applied to the context information <NUM>. However, the other systems may be unable to address learnability and/or may unable to personalize the recommendations. The rank and reward call <NUM> may include the other recommendations for channel parameters <NUM>.

Cloud computing system <NUM> may also include a reinforcement learning system <NUM>. Reinforcement learning system <NUM> may be a machine learning system that may provide one or more recommendations <NUM> for values for the channel parameters <NUM> in response to learning from feedback from previous recommendations <NUM>. One example of reinforcement learning system <NUM> may include a contextual bandit. The components of the reinforcement learning system <NUM> may include hardware, software, or both. For example, the reinforcement learning system <NUM> may include one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices. When executed by the one or more processors, the computer-executable instructions of one or more computing devices (e.g., reinforcement learning system <NUM>) can perform one or more methods described herein. Alternatively, the components of the reinforcement learning system <NUM> may include hardware, such as a special purpose processing device to perform a certain function or group of functions. Additionally, or alternatively, the components of the reinforcement learning system <NUM> may include a combination of computer-executable instructions and hardware.

Reinforcement learning system <NUM> may use the information provided in the rank and reward call <NUM> from game server <NUM> (e.g., context information <NUM>, user information provided with user vector <NUM>, game information provided with item vector <NUM>, and/or any other recommendations for channel parameters) in determining a recommendation <NUM> for the initial values for the channel parameters <NUM> for connection <NUM>. For example, the recommendation <NUM> may include initial values for the bit rate, the resolution, a level for FEC, and whether to use short-term or long-term rendering. Another example may include providing one recommendation <NUM> for targeting one type of latency and/or jitter and a different recommendation <NUM> for targeting a different type of latency and/or jitter. Yet another example may include providing a recommendation <NUM> to target only one type of latency and/or jitter.

Channel manager <NUM> may establish connection <NUM> using the recommended values for channel parameters <NUM> and game server <NUM> may stream game <NUM> to client device <NUM> using the values for channel parameters <NUM>. Game server <NUM> may receive game play <NUM> of the user while the user is playing game <NUM>. For example, the game play <NUM> may collect the actions performed by the user while playing game <NUM>.

Game server <NUM> may include a reward function manager <NUM> that may build a game play curve <NUM> using the game play <NUM> received for the user. The game play curve <NUM> may build a graph representing a current game play <NUM> for the user. For example, the game play curve <NUM> may illustrate the actions performed by the user for game <NUM> while playing game <NUM>. A game play curve <NUM> may be generated each time a user plays game <NUM>. As such, users may have different game play curves <NUM> for different gaming sessions.

Reward function manager <NUM> may also determine a reward function <NUM> to use in aiding the reinforcement learning system <NUM> in understanding a success of the recommendation <NUM> for the channel parameters <NUM> for connection <NUM>. In an implementation, the reward function <NUM> may include comparing the game play curve <NUM> to the game signature function <NUM> and determining a reward function error <NUM> in response to the comparison. The reward function error <NUM> may include the distance between the game play curve <NUM> and the game signature function <NUM>. As such, the game signature function <NUM> may be a segue between the user preferences and the channel configurations for connection <NUM>. Reward function manager <NUM> may continuously learn the reward function <NUM> as the game play <NUM> progresses for the user and may use the reward function <NUM> for capturing subjective user preferences without requiring human intervention. As such, each user may have different preferences that may be learned through using the reward function <NUM>.

The reward function error <NUM> may be included in the rank and reward calls <NUM> to reinforcement learning system <NUM> and may be used by reinforcement learning system <NUM> to modify and/or change the recommendation for the channel parameters <NUM>. The reinforcement learning system <NUM> may continue to provide recommendations for adjusting the channel parameters <NUM> until the reward function error <NUM> is within a threshold level. In an implementation, the threshold level may be a positive number. For example, if the reward function error <NUM> is negative (e.g., there is a large distance between the game play curve <NUM> and the game signature function <NUM>), the reinforcement learning system <NUM> may use this feedback to continue to recommend adjustments to the channel parameters <NUM> to attempt to lower the distance between the game play curve <NUM> and the game signature function <NUM> to move the reward function error <NUM> towards a positive number. By switching the channel conditions, reinforcement learning system <NUM> may improve the streaming conditions of game <NUM> to the user. Thus, the reinforcement learning system <NUM> may use the changes in the reward function error <NUM> as positive and/or negative feedback on the recommended changes for the channel parameters and may modify a next round of recommendations in response to the feedback provided by the reward function error <NUM>.

Client device <NUM> may periodically send the context information <NUM> to game server <NUM> at a predetermined time interval (e.g., every five or ten seconds) while game <NUM> is being streamed to client device <NUM>. As such, client device <NUM> may provide updated context information <NUM> for client device <NUM> throughout the duration of the game play <NUM>. In addition, client device <NUM> may be triggered, or otherwise forced, to send the context information <NUM> to game server <NUM> in response to changes in the context. For example, a user switching from a Wi-Fi network to cellular network may trigger the sending of context information <NUM> to game server <NUM>.

Channel manager <NUM> may receive the context information <NUM> from client device <NUM> and may send a new rank and reward call <NUM> to reinforcement learning system <NUM> at a predetermined time interval for a new recommendation <NUM> for the channel parameters <NUM> for connection <NUM>. A plurality of rank and reward calls <NUM> may be sent to reinforcement learning system <NUM> throughout the duration of game play <NUM> at predetermined time intervals in response to a predetermined frequency and/or triggers for sending the rank and reward calls <NUM>. For example, in a one hour game, new rank and reward calls <NUM> may be sent every ten seconds. The new rank and reward call <NUM> may include the context information <NUM>, user vector <NUM>, item vector <NUM>, and the reward function error <NUM>.

Reinforcement learning system <NUM> may use the information provided in the rank and reward call <NUM> to make a new recommendation <NUM> for the channel parameters <NUM> for connection <NUM>. The new recommendation <NUM> may include changing the values of the channel parameters <NUM> and/or maintaining the values of the channel parameters <NUM>. Reinforcement learning system <NUM> may reply to the rank and reward call <NUM> with the new recommendation <NUM> for the values of the channel parameters <NUM>.

For example, when the context information <NUM> changes, the recommendation <NUM> may include new values for the channel parameters <NUM>. The context information <NUM> may indicate spikes in the network jitter, as such, the reinforcement learning system <NUM> may recommend switching from a short-term rendering mode to a long-term rendering mode. Another example may include the context information <NUM> remaining the same while the reward function error <NUM> is high, the recommendation <NUM> may include new values for the channel parameters <NUM> in response to the high value for the reward function error <NUM>. In one example, the reinforcement learning system <NUM> may recommend changing the video bitrate from <NUM> mbps to <NUM> mbps and may recommend keeping the same values for resolution, the levels of FEC, and using the same rendering. In another example, the reinforcement learning system <NUM> may recommend lowering the resolution value to below <NUM> p and change the rendering from a long-term mode to a short-term mode, while keeping the bit rate and the FEC values the same. As such, the reinforcement learning system <NUM> may recommend changing all values of the channel parameters <NUM> and/or may recommend changing a subset of the values of the channel parameters <NUM> in response to the information and/or feedback provided to the reinforcement learning system <NUM> through the rank and reward calls <NUM>.

In an implementation, reinforcement learning system <NUM> may use the reward function error <NUM> and/or the information received to continuously learn so that the recommendations <NUM> for the channel parameters <NUM> for connection <NUM> may improve in response to learning from outcomes from previous recommendations <NUM>. In addition, reinforcement learning system <NUM> may use information learned from other users in cloud computing system <NUM> in generating recommendations <NUM>. A plurality of game servers <NUM> may communicate with reinforcement learning system <NUM> and reinforcement learning system <NUM> may use the information learned from each game server <NUM> in generating recommendations <NUM>.

Channel manager <NUM> may receive the recommendation <NUM> from reinforcement learning system <NUM> and may use the values for the channel parameters <NUM> to modify the channel parameters <NUM> of connection <NUM>. As such, channel manager <NUM> may continue to adjust the channel parameters <NUM> of connection <NUM> in response to recommendations <NUM> provided by reinforcement learning system <NUM> until the reward function error <NUM> is within a threshold level and/or a request to stop game play <NUM> is received.

When a request to stop game play <NUM> is received, the game play data collected by game server <NUM> during the playing of game <NUM> may be transmitted to game streaming history datastore <NUM>. For example, the game play data may be sent at the end of the game play <NUM> or may be transmitted at the end of the day so that all the game play data received for the day may be transmitted at once to the game streaming history datastore <NUM>. The game streaming history datastore <NUM> may be continuously updated with new game data <NUM> collected by game servers <NUM> on the cloud computing system <NUM>.

As such, environment <NUM> may continuously learn the reward function <NUM> for capturing subjective user preferences without requiring human intervention. The reward function <NUM> may be used alongside dynamic user context information <NUM> to feed a reinforcement learning system <NUM> aided with latent user vectors <NUM> capturing user interaction history over the past few months and/or years to help the end to end system continuously learn an optimal policy for providing best in class plays well experiences to the end user via dynamic streaming channel configuration. Thus, environment <NUM> may use the triangulation of the reward function <NUM>, the dynamic user context information <NUM>, game information provided in the item vectors <NUM>, and the gaming history summary of the user provided in user vectors <NUM> to learn the subjectivity of the user preferences and dynamically adjust the channel conditions for the game streaming to personalize the gaming experience for the user.

Each of the components (e.g., game <NUM>, channel manager <NUM>, reward function manager <NUM>, and/or local cache <NUM>) of the game server <NUM> may be in communication with each other using any suitable communication technologies. In addition, while the components of the game server <NUM> are shown to be separate in <FIG>, any of the components or subcomponents may be combined into fewer components, such as into a single component, or divided into more components as may serve a particular implementation.

Moreover, the components of the game server <NUM> may include hardware, software, or both. For example, the components of the game server <NUM> may include one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices. When executed by the one or more processors, the computer-executable instructions of one or more computing devices (e.g., server device(s) <NUM>) can perform one or more methods described herein. Alternatively, the components of the game server <NUM> may include hardware, such as a special purpose processing device to perform a certain function or group of functions. Additionally, or alternatively, the components of the game server <NUM> may include a combination of computer-executable instructions and hardware.

Referring now to <FIG> and <FIG>, an example method <NUM> may be used by game server <NUM> (<FIG>) for personalizing channel parameters <NUM> (<FIG>) for streaming content to client device <NUM> (<FIG>). While method <NUM> is discussed in connection with streaming a game <NUM> to a client device <NUM>, it should be appreciated that method <NUM> may be used by any server device to stream any form of content to a client device <NUM>. The actions of method <NUM> may be discussed below with reference to the architecture of <FIG>.

At <NUM>, method <NUM> may include initializing a game server to stream a game to a client device. Cloud computing system <NUM> may initialize a game server <NUM> to stream game <NUM> to client device <NUM> in response to a user selecting game <NUM> to play from a list of available games in a game application <NUM>.

At <NUM>, method <NUM> may include establishing a connection between the game server and the client device. Game server <NUM> may include a channel manager <NUM> that may establish the connection <NUM> via a network to stream game <NUM> to client device <NUM> so that game <NUM> may be streamed to client device <NUM>.

At <NUM>, method <NUM> may include receiving initial context information for the client device. Channel manager <NUM> may receive initial context information <NUM> from client device <NUM> and may use the initial context information <NUM> in establishing connection <NUM>. Context information <NUM> may provide the current context of client device <NUM> and/or the user. Context information <NUM> may include, for example, a network type for connection <NUM> (e.g., Wi-Fi or cellular), a network speed for connection <NUM>, bandwidth constraints for connection <NUM>, a geographic location of client device <NUM>, whether client device <NUM> is in motion, and/or an amount of remaining battery power for client device <NUM>. In an implementation, client device <NUM> may send context information <NUM> to game server <NUM> using a rank and reward call.

At <NUM>, method <NUM> may include sending a rank and reward call with the initial context information, a user vector, and an item vector. Channel manager <NUM> may communicate with a reinforcement learning system <NUM> in determining initial values for channel parameters <NUM> for connection <NUM>. Channel parameters <NUM> may include, but are not limited to, video bitrate, resolution, bandwidth, forward error correction (FEC), smooth rendering, rendering, one or more types of jitter, and/or one or more types of latency (e.g., input latency or decode latency) used when streaming game <NUM> to client device <NUM>.

Channel manager <NUM> may send a rank and reward call <NUM> to reinforcement learning system <NUM>. Reinforcement learning system <NUM> may be a machine learning system that may provide one or more recommendations <NUM> for values for the channel parameters <NUM> in response to learning from feedback from previous recommendations <NUM>. One example of reinforcement learning system <NUM> may include a contextual bandit.

The rank and reward call <NUM> may include the context information <NUM>, user vector <NUM>, and item vector <NUM>. Upon initialization of game server <NUM> in response to a user selecting game <NUM> to play, game server <NUM> may access a copy of user vector <NUM> for the user and/or a copy of one or more item vectors <NUM> for game <NUM> from globally distributed cache <NUM>. For example, game server <NUM> may identify the user of client device <NUM> using a client identification and/or user account information associated with the user and may access the corresponding user vector <NUM> for the user. In addition, game server <NUM> may access a corresponding item vector <NUM> for game <NUM> from the globally distributed cache <NUM>. Game server <NUM> may store user vector <NUM> and/or item vector <NUM> in a local cache <NUM>.

User vectors <NUM> may be a multidimensional array of numbers that provides a condensed representation of the game play for the user for every game <NUM> played by the user. One example multidimensional array may include a forty dimensional array. For example, user vectors <NUM> may embed one or more user actions during game play. In addition, user vectors <NUM> may provide a variety of information about the user, such as, but not limited to, a level of play for the user, an amount of money the user spends on games, an amount of time a user plays game, types of games the user plays, and/or a speed of play of the user. In addition, the one or more user vectors <NUM> may illustrate similarities between groups of users. As such, user vectors <NUM> may provide a summary of the total gaming history for each user of the cloud computing system <NUM>.

Item vectors <NUM> may provide additional game information and/or game features associated with one or more games <NUM>. For example, item vectors <NUM> may embed one or more user actions during game play. In addition, item vectors <NUM> may relate to other items of interest relating to games <NUM>. Item vectors <NUM> may be a multidimensional array of numbers that provide a condensed representation of the game information and/or game sessions for every game <NUM> of cloud computing system <NUM>. One example multidimensional array of numbers may include a fifty dimensional array of numbers. The one or more item vectors <NUM> may illustrate similarities between games <NUM> and/or gaming sessions. In addition, the one or more item vectors <NUM> may illustrate differences between games <NUM> and/or gaming sessions. The item vectors <NUM> may provide a summary of game information for each game <NUM> of the cloud computing system <NUM>.

At <NUM>, method <NUM> may include receiving a recommendation for channel parameters for the connection in response to the rank and reward call. Reinforcement learning system <NUM> may use the information provided in the rank and reward call <NUM> from game server <NUM> (e.g., context information <NUM>, user information provided with user vector <NUM>, item vector <NUM>, and/or any other recommendations for channel parameters) in determining a recommendation <NUM> for the initial values for the channel parameters <NUM> for connection <NUM>. For example, the recommendation <NUM> may include initial values for the bit rate, the resolution, a level for FEC, and whether to use short-term or long-term rendering. Reinforcement learning system <NUM> may reply to the rank and reward call <NUM> with the recommendation <NUM> for the values of the channel parameters <NUM>.

At <NUM>, method <NUM> may include using the recommendation to set a value of the channel parameters to stream the game to the client device. Channel manager <NUM> may establish connection <NUM> using the recommended values for channel parameters <NUM> and game server <NUM> may stream game <NUM> to client device <NUM> using the values for channel parameters <NUM>.

At <NUM>, method <NUM> may include determining whether a request to end the game has been received. Game server <NUM> may determine whether receive a request to stop game play <NUM> is received from client device <NUM>.

At <NUM>, method <NUM> may include ending the game session in response to receiving a request to end the game. Game server <NUM> may end the game session for game <NUM> and may remove the connection <NUM> between game server <NUM> and the client device <NUM>.

At <NUM>, method may optionally include transmitting the game data to a game streaming history data store. The game play data collected by game server <NUM> during the playing of game <NUM> may be transmitted to game streaming history datastore <NUM> at the end of the game play. For example, the game play data may be sent at the end of the game play <NUM> or may be transmitted at the end of the day so that all the game play data received for the day may be transmitted at once to the game streaming history datastore <NUM>. As such, the game streaming history datastore <NUM> may be continuously updated with new game data <NUM> collected by game servers <NUM> on the cloud computing system <NUM>.

At <NUM>, method <NUM> may include determining a reward function error for the channel parameters in response to an indication for the continuation of game play. Game server <NUM> may include a reward function manager <NUM> that may use the current game play <NUM> to determine a reward function error <NUM> for the values selected for the channel parameters <NUM>.

As such, method <NUM> may personalize the streaming channel to a client device <NUM> of a user via a triangulation of a user interaction history, real time user dynamic context features, and real time adjustments of streaming channel parameters <NUM> to maximize a reward function that captures the plays well feeling for the user.

Referring now to <FIG>, an example method <NUM> may be used by game server <NUM> for determining a reward function error <NUM> (<FIG>). The actions of method <NUM> may be discussed below with reference to the architectures of <FIG>.

At <NUM>, method <NUM> may include receiving game play of the user. Game server <NUM> may receive game play <NUM> of the user while the user is playing game <NUM>. For example, the game play <NUM> may collect the actions performed by the user while playing game <NUM>.

At <NUM>, method <NUM> may include building a game play curve for the user using the received game play. Game server <NUM> may include a reward function manager <NUM> that may build a game play curve <NUM> using the game play <NUM> received for the user. The game play curve <NUM> may build a graph representing a current game play <NUM> for the user. For example, the game play curve <NUM> may illustrate the actions performed by the user for game <NUM> while playing game <NUM>. A game play curve <NUM> may be generated each time a user plays game <NUM>. As such, users may have different game play curves <NUM> for different gaming sessions.

At <NUM>, method <NUM> may include accessing a game signature function for the game. Game server <NUM> access a game signature functions <NUM> for the requested game <NUM> from the local cache <NUM>. Game signature function <NUM> may identify an optimal game play curve for game <NUM> under good network conditions. An optimal game play curve may represent over a plurality of game sessions how a user (e.g., an expert player, a new gamer, and/or a casual gamer) provides input via a game controller that best achieves the mission or end result of the game the user is playing. The optimal game play curve may use a plurality of factors in gauging an optimal game play curve. In one example, the optimal game play curve may be represented by plotting the input frame and bytes count of the input provided via the game controller by the user plotted against time on an X-axis. In an implementation, game server <NUM> may save the game signature function <NUM> for a cohort associated with the user. For example, if the user is a causal gamer, game server <NUM> may save the game signature function <NUM> for causal gamers for game <NUM> in local cache <NUM>.

At <NUM>, method <NUM> may include calculating the reward function error by determining a distance between the game play curve and the game signature function. Reward function manager <NUM> may also determine a reward function <NUM> to use in aiding the reinforcement learning system <NUM> in understanding a success of the recommendation <NUM> for the channel parameters <NUM> for connection <NUM>. In an implementation, the reward function <NUM> may include comparing the game play curve <NUM> to the game signature function <NUM> and determining a reward function error <NUM> in response to the comparison. The reward function error <NUM> may include the distance between the game play curve <NUM> and the game signature function <NUM>.

As such, the game signature function <NUM> may be a segue between the user preferences and the channel configurations for connection <NUM>. Reward function manager <NUM> may continuously learn the reward function <NUM> as the game play <NUM> progresses for the user and may use the reward function <NUM> for capturing subjective user preferences without requiring human intervention. Each user may have different preferences that may be learned through using the reward function <NUM>.

Referring to <FIG>, the reward function error <NUM> may be included in the rank and reward calls <NUM> to reinforcement learning system <NUM> and may be used by reinforcement learning system <NUM> to modify and/or change the recommendation for the channel parameters <NUM>.

At <NUM>, method <NUM> may include receiving updated context information for the client device. Context information <NUM> may dynamically change as the context of client device <NUM> and/or the user changes. For example, the network may go down or the user may move geographic locations, resulting in changes to the context information <NUM>. Client device <NUM> may periodically send the context information <NUM> to game server <NUM> at a predetermined time interval (e.g., every five or ten seconds) while game <NUM> is being streamed to client device <NUM>. As such, client device <NUM> may provide updated context information <NUM> for client device <NUM> throughout the duration of the game play <NUM> in response to a predefined frequency and/or triggers for sending the updated context information <NUM>. In an implementation, the updated context information <NUM> may be sent using rank and reward calls. In addition, client device <NUM> may be triggered, or otherwise forced, to send the context information <NUM> to game server <NUM> in response to changes in the context. For example, a user switching from a Wi-Fi network to cellular network may trigger the sending of context information <NUM> to game server <NUM>.

At <NUM>, method <NUM> may include sending a new rank and reward call with the updated context information, the user vector, item vector <NUM>, and the reward function error. Channel manager <NUM> may receive the context information <NUM> from client device <NUM> and may send a new rank and reward call <NUM> to reinforcement learning system <NUM> at a predetermined time interval for a new recommendation <NUM> for the channel parameters <NUM> for connection <NUM>. A plurality of rank and reward calls <NUM> may be sent to reinforcement learning system <NUM> throughout the duration of game play <NUM> at predetermined time intervals in response to a predefined frequency and/or triggers for sending the rank and reward calls <NUM>. For example, in a one-hour game, new rank and reward calls <NUM> may be sent every ten seconds. The new rank and reward call <NUM> may include the context information <NUM>, the user vector <NUM>, item vector <NUM>, and the reward function error <NUM>.

At <NUM>, method <NUM> may include receiving a recommendation for the channel parameters for the connection. Reinforcement learning system <NUM> may use the information provided in the rank and reward call <NUM> to make a new recommendation <NUM> for the channel parameters <NUM> for connection <NUM>. The new recommendation <NUM> may include changing the values of the channel parameters <NUM> and/or maintaining the values of the channel parameters <NUM>. Reinforcement learning system <NUM> may reply to the rank and reward call <NUM> with the new recommendation <NUM> for the values of the channel parameters <NUM>.

At <NUM>, method <NUM> may include determining whether to change the channel parameters. Channel manager <NUM> may receive the recommendation <NUM> from reinforcement learning system <NUM> and may use the recommendation to determine whether to modify the channel parameters. If the recommendation recommends maintaining the values, method <NUM> may proceed to <NUM>.

At <NUM>, method <NUM> may include modifying the value of the channel parameters. If the recommendation recommends adjusting or otherwise modifying the values for the channel parameters <NUM>, channel manager <NUM> may modify the channel parameters <NUM> of connection <NUM> in response to the recommended values.

At <NUM>, method <NUM> may include using the modified channel parameters to stream the game. Channel manager <NUM> may continue to adjust the channel parameters <NUM> of connection <NUM> in response to recommendations <NUM> provided by reinforcement learning system <NUM> and use the modified channel parameters to stream game <NUM> until the reward function error <NUM> is within a threshold level and/or a request to stop game play <NUM> is received.

As such, method <NUM> may be used to personalize.

Referring now to <FIG>, an example graph <NUM> illustrates an overlay of an example game signature function curve <NUM> to a game play curve <NUM>. For example, the values in the game signature function curve <NUM> may be derived across all power gamers before being used in the game signature function curve <NUM>. The y-axis <NUM> may illustrate the input responsiveness of the user actions during game play. For example, the y-axis <NUM> may track an optimum mode value for input frame count or an average combined mode value for bytes and the input frame received count. The x-axis <NUM> may track the duration of game play. For example, the x-axis <NUM> may track the duration of game play in minutes. While the game signature function curve <NUM> and the game play curve <NUM> are illustrated as a line, it should be appreciated that any shape or type of curves may be used to represent the game signature function curve <NUM> and/or the game play curve <NUM>.

The distance <NUM> between the game signature function curve <NUM> and the game play curve <NUM> may illustrate the reward function error <NUM> (<FIG>) calculated by reward function manager <NUM> (<FIG>). The distance <NUM> between the game signature function curve <NUM> and the game play curve <NUM> at time <NUM> minutes may be used by the reinforcement learning system <NUM> to understand that the recommended adjustments to the channel parameters <NUM> did not lower the distance <NUM>, but instead, increased the distance <NUM> from a previous game time duration (e.g., time <NUM> minutes). As such, the reinforcement learning system <NUM> may use the information provided about the distance <NUM> to recommend different adjustments to the channel parameters <NUM> and attempt to lower the distance <NUM> between the game play curve <NUM> and the game signature function curve <NUM> with the next recommended adjustments to the channel parameters <NUM>.

Referring now to <FIG>, an example method <NUM> may be used by reinforcement learning system <NUM> for determining a recommendation <NUM> (<FIG>) for the channel parameters <NUM> (<FIG>). The actions of method <NUM> may be discussed below with reference to the architectures of <FIG>.

At <NUM>, method <NUM> may include receiving a rank and reward call with context information, the user vector, the item vector, and a reward function error. Reinforcement learning system <NUM> may use the information provided in the rank and reward call <NUM> to make a recommendation <NUM> for the channel parameters <NUM> for connection <NUM>. User vector <NUM> and/or item vector <NUM> may be used to provide more user information and/or game features to reinforcement learning system <NUM>, and thus, improving the functioning of reinforcement learning system <NUM> by providing more features for reinforcement learning system <NUM> to use in personalizing the channel parameters <NUM>.

At <NUM>, method <NUM> may include determining whether the context information has changed. Reinforcement learning system <NUM> may determine whether the context information <NUM> for the client device <NUM> and/or user has changed. For example, the rank and reward call <NUM> may include updated context information <NUM> for the client device <NUM> and/or the user.

At <NUM>, method <NUM> may include sending a recommendation to modify the channel parameters. When the context information <NUM> changes, reinforcement learning system <NUM> may provide a recommendation <NUM> with new values for the channel parameters <NUM>. For example, the updated context information <NUM> may indicate spikes in the network jitter, as such, the reinforcement learning system <NUM> may recommend switching from a short-term rendering mode to a long-term rendering mode.

The reinforcement learning system <NUM> may recommend changing all values of the channel parameters <NUM> and/or may recommend changing a subset of the values of the channel parameters <NUM> in response to the information and/or feedback provided to the reinforcement learning system <NUM> through the rank and reward calls <NUM>.

At <NUM>, method <NUM> may include determining whether the reward function error is within a threshold level. Reinforcement learning system <NUM> may continue to provide recommendations <NUM> for adjusting the channel parameters <NUM> until the reward function error <NUM> is within a threshold level. In an implementation, the threshold level may be a positive number. For example, if the reward function error <NUM> is negative (e.g., there is a large distance between the game play curve <NUM> and the game signature function <NUM>), the reinforcement learning system <NUM> may use this feedback to continue to recommend adjustments to the channel parameters <NUM> to attempt to lower the distance between the game play curve <NUM> and the game signature function <NUM> to move the reward function error <NUM> towards a positive number. By switching the channel conditions, reinforcement learning system <NUM> may improve the streaming conditions of game <NUM> to the user. Thus, the reinforcement learning system <NUM> may use the changes in the reward function error <NUM> as positive and/or negative feedback on the recommended changes for the channel parameters and may modify a next round of recommendations in response to the feedback provided by the reward function error <NUM>.

At <NUM>, method <NUM> may include sending a recommendation to maintain the channel parameters. Reinforcement learning system <NUM> may send a recommendation <NUM> to maintain the channel parameters <NUM> in response when the context information has stayed the same and/or the reward function error is within a threshold level.

As such, method <NUM> may be used by the reinforcement learning system <NUM> to continuously learn so that the recommendations <NUM> for the channel parameters <NUM> for connection <NUM> may improve in response to learning from outcomes from previous recommendations <NUM>. As more user information and/or game features are provided to reinforcement learning system <NUM> using, for example, user vector <NUM> and/or item vector <NUM>, the recommendations <NUM> may improve. In addition, reinforcement learning system <NUM> may use information learned from other users in cloud computing system <NUM> in generating recommendations <NUM>. A plurality of game servers <NUM> may communicate with reinforcement learning system <NUM> and reinforcement learning system <NUM> may use the information learned from each game server <NUM> in generating recommendations <NUM>.

Referring now to <FIG>, an example method <NUM> may be used by a trained machine learning model for creating user vectors <NUM> (<FIG>) for the users of cloud computing system <NUM> (<FIG>). The actions of method <NUM> may be discussed below with reference to the architectures of <FIG>.

At <NUM>, method <NUM> may include accessing game data for a plurality of users in a game streaming history database. A trained machine learning model may access game data <NUM> stored in a game streaming history datastore <NUM> in communication with cloud computing system <NUM>. Game data <NUM> may include information acquired or otherwise collected for all games <NUM> in cloud computing system <NUM> and/or all users in cloud computing system <NUM>. Examples of game data <NUM> may include, but are not limited to, game names, game content, an amount of time users are playing games, an amount of money spent on games, and/or actual game play of users. For example, game data <NUM> may include information for millions of game sessions that occurred using cloud computing system <NUM>.

At <NUM>, method <NUM> may include applying an interaction featurizer to the game data. The trained machine learning model may apply an interaction featurizer <NUM> to analyze game data <NUM> and generate user vectors <NUM> for each user of the cloud computing system <NUM>. In an implementation, the interaction featurizer <NUM> may be a deep learning network. Examples of deep machine learning networks may include, but are not limited to, wide and deep learning and/or Extreme Deep Factorization Machine (xDeepFM). Another example of the machine learning model may include Convolutional Neural Networks (CNN). The interaction featurizer <NUM> may run at a predetermined time period to ensure the user vectors <NUM> reflect the latest gaming information received for the users. For example, the interaction featurizer <NUM> may run every four hours. Another example may include the interaction featurizer <NUM> running once a day.

At <NUM>, method <NUM> may include generating for each user of the plurality of users a user vector in response to the interaction featurizer. The trained machine learning model may generate user vectors <NUM> in response to applying the interaction featurizer <NUM> to the game data <NUM>.

The user vectors <NUM> may provide a mathematical representation of the gaming history for the users of cloud computing system <NUM>. Each user of the gaming system may have an associated user vector <NUM>. User vectors <NUM> may be a multidimensional array of numbers or real values that provides a condensed representation of the game play for the user for every game <NUM> played by the user. For example, user vectors <NUM> may embed one or more user actions during game play. In addition, user vectors <NUM> may provide a variety of information about the user, such as, but not limited to, a level of play for the user, an amount of money the user spends on games, an amount of time a user plays game, types of games the user plays, and/or a speed of play of the user. In addition, the one or more user vectors <NUM> may illustrate similarities between groups of users. As such, user vectors <NUM> may provide a summary of the total gaming history for each user of the cloud computing system <NUM>.

At <NUM>, method <NUM> may include transmitting the user vectors to a globally distributed cache. The one or more user vectors <NUM> may be distributed to globally distributed cache <NUM> in cloud computing system <NUM>.

At <NUM>, method <NUM> may include receiving updated game data. The game data <NUM> may be continuously updated in game streaming history datastore <NUM> as games <NUM> are created and/or played by users of cloud computing system <NUM>.

At <NUM>, method <NUM> may include applying the interaction featurizer to the updated game data. As new game data <NUM> is received from game servers <NUM> and/or game data <NUM> is updated, the interaction featurizer <NUM> may be applied to game data <NUM>. The interaction featurizer <NUM> may be applied to a large volume of newly added game data <NUM>.

At <NUM>, method <NUM> may include determining updates to the user vector in response to the interaction featurizer. The user vectors <NUM> may be updated and/or revised as necessary in response to the output of the interaction featurizer <NUM>. As the user interaction history changes, the user vectors <NUM> may be updated to reflect the changes to the user interaction history changes.

At <NUM>, method <NUM> may include transmitting the updates to the user vector to the globally distributed cache. The globally distributed cache <NUM> may include continuously updated user vectors <NUM> that provide a game history summary for each user of cloud computing system <NUM>.

Method <NUM> may be used to generate a summary describing a user interaction history of each user of the cloud computing system <NUM>.

Referring now to <FIG>, an example method <NUM> may be used by a trained machine learning model for creating item vectors <NUM> (<FIG>) for the users of cloud computing system <NUM> (<FIG>). The actions of method <NUM> may be discussed below with reference to the architectures of <FIG>.

At <NUM>, method <NUM> may include accessing game data for a plurality of games in a game streaming history database. A trained machine learning model may access game data <NUM> stored in a game streaming history datastore <NUM> in communication with cloud computing system <NUM>. Game data <NUM> may include information acquired or otherwise collected for all games <NUM> in cloud computing system <NUM> and/or all users in cloud computing system <NUM>. Examples of game data <NUM> may include, but are not limited to, game names, game content, an amount of time users are playing games, an amount of money spent on games, and/or actual game play of users. For example, game data <NUM> may include information for millions of game sessions that occurred using cloud computing system <NUM>.

At <NUM>, method <NUM> may include applying an interaction featurizer to the game data. The trained machine learning model may apply an interaction featurizer <NUM> to analyze game data <NUM> and generate item vectors <NUM> for each game <NUM> of the cloud computing system <NUM>. In an implementation, the interaction featurizer <NUM> may be a deep learning network. Examples of deep machine learning networks may include, but are not limited to, wide and deep learning and/or Extreme Deep Factorization Machine (xDeepFM). Another example of the machine learning model may include Convolutional Neural Networks (CNN). The interaction featurizer <NUM> may run at a predetermined time period to ensure the item vectors <NUM> reflect the latest gaming information received for the games. For example, the interaction featurizer <NUM> may run every four hours. Another example may include the interaction featurizer <NUM> running once a day.

At <NUM>, method <NUM> may include generating for each game of the plurality of games an item vector in response to the interaction featurizer. The trained machine learning model may generate item vectors <NUM> in response to applying the interaction featurizer <NUM> to the game data <NUM>.

Item vectors <NUM> may provide additional game information and/or game features associated with one or more games <NUM>. For example, item vectors <NUM> may embed one or more user actions during game play. In addition, item vectors <NUM> may relate to other items of interest relating to games <NUM> of cloud computing system <NUM>. Item vectors <NUM> may be a multidimensional array of numbers that provide a condensed representation of the game information and/or game sessions for every game <NUM> of cloud computing system <NUM>. Each game <NUM> of the gaming system may have an associated item vector <NUM>. In an implementation, games <NUM> may have a plurality of item vectors <NUM> associated with the users. The one or more item vectors <NUM> may illustrate similarities between games <NUM> and/or gaming sessions. In addition, the one or more item vectors <NUM> may illustrate differences between games <NUM> and/or gaming sessions. The item vectors <NUM> may provide a summary of game information for each game <NUM> of the cloud computing system <NUM>.

At <NUM>, method <NUM> may include transmitting the item vectors to a globally distributed cache. The one or more item vectors <NUM> may be distributed to globally distributed cache <NUM> in cloud computing system <NUM>.

At <NUM>, method <NUM> may include receiving updated game data. The game data <NUM> may be continuously updated in game streaming history datastore <NUM> as games <NUM> are created and/or played by users of cloud computing system <NUM>. For example, a large amount of game sessions (e.g., hundreds or thousands) may be added to the game data <NUM> daily.

At <NUM>, method <NUM> may include determining updates to the item vector in response to the interaction featurizer. The item vectors <NUM> may be updated and/or revised as necessary in response to the output of the interaction featurizer <NUM>. As the game history changes, the item vectors <NUM> may be updated to reflect the changes to the game history.

At <NUM>, method <NUM> may include transmitting the updates to the item vector to the globally distributed cache. The globally distributed cache <NUM> may include continuously updated item vectors <NUM> that provide a game summary for each game of cloud computing system <NUM>.

Method <NUM> may be used to generate a summary describing game information for each game <NUM> of cloud computing system <NUM>.

The computer system <NUM> includes a processor <NUM>. The processor <NUM> may be a general-purpose single or multi-chip microprocessor (e.g., an Advanced RISC (Reduced Instruction Set Computer) Machine (ARM)), a special purpose microprocessor (e.g., a digital signal processor (DSP)), a microcontroller, a programmable gate array, etc. The processor <NUM> may be referred to as a central processing unit (CPU). Although just a single processor <NUM> is shown in the computer system <NUM> of <FIG>, in an alternative configuration, a combination of processors (e.g., an ARM and DSP) could be used.

The computer system <NUM> also includes memory <NUM> in electronic communication with the processor <NUM>. The memory <NUM> may be any electronic component capable of storing electronic information. For example, the memory <NUM> may be embodied as random access memory (RAM), read-only memory (ROM), magnetic disk storage mediums, optical storage mediums, flash memory devices in RAM, on-board memory included with the processor, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) memory, registers, and so forth, including combinations thereof.

A computer system <NUM> may also include one or more input devices <NUM> and one or more output devices <NUM>. Some examples of input devices <NUM> include a keyboard, mouse, microphone, remote control device, button, joystick, trackball, touchpad, and lightpen. Some examples of output devices <NUM> include a speaker and a printer. One specific type of output device that is typically included in a computer system <NUM> is a display device <NUM>. Display devices <NUM> used with embodiments disclosed herein may utilize any suitable image projection technology, such as liquid crystal display (LCD), light-emitting diode (LED), gas plasma, electroluminescence, or the like. A display controller <NUM> may also be provided, for converting data <NUM> stored in the memory <NUM> into text, graphics, and/or moving images (as appropriate) shown on the display device <NUM>.

The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof, unless specifically described as being implemented in a specific manner. Any features described as modules, components, or the like may also be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a non-transitory processor-readable storage medium comprising instructions that, when executed by at least one processor, perform one or more of the methods described herein. The instructions may be organized into routines, programs, objects, components, data structures, etc., which may perform particular tasks and/or implement particular data types, and which may be combined or distributed as desired in various embodiments.

The steps and/or actions of the methods described herein may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is required for proper operation of the method that is being described, the order and/or use of specific steps and/or actions may be modified without departing from the scope of the claims.

The articles "a," "an," and "the" are intended to mean that there are one or more of the elements in the preceding descriptions. Additionally, it should be understood that references to "one implementation" or "an implementation" of the present disclosure are not intended to be interpreted as excluding the existence of additional implementations that also incorporate the recited features. For example, any element described in relation to an implementation herein may be combinable with any element of any other implementation described herein. Numbers, percentages, ratios, or other values stated herein are intended to include that value, and also other values that are "about" or "approximately" the stated value, as would be appreciated by one of ordinary skill in the art encompassed by implementations of the present disclosure. A stated value should therefore be interpreted broadly enough to encompass values that are at least close enough to the stated value to perform a desired function or achieve a desired result.

Claim 1:
A server, comprising:
a memory to store data and instructions;
at least one processor operable to communicate with the memory, wherein the at least one processor is operable to:
establish (<NUM>) a connection with a client device to stream a game to the client device;
receive (<NUM>) context information from the client device, wherein the context information provides a current context of the client device;
send (<NUM>), to a reinforcement learning system, a rank and reward call for a recommendation for channel parameters for the connection, wherein the rank and reward call includes the context information, a user vector, and an item vector, wherein the user vector is a multidimensional array of numbers that provides a summary of a gaming history of a user of the client device, and wherein the item vector is a multidimensional array of numbers that provides a summary of game information for the game;
receive (<NUM>), from the reinforcement learning system, the recommendation for the channel parameters in response to the rank and reward call;
use (<NUM>) the recommendation to set a value of the channel parameters to stream the game to the client device; and
determine (<NUM>) a reward function error for the channel parameters by calculating (<NUM>) a distance between a game play curve for the game and a game signature function,
wherein the game signature function represents optimal game play for a cohort of users for the game learned through machine learning from analyzing game data stored in a game streaming history datastore, and
wherein the game play curve represents a current game play of the user.